Reviewerly/exHarmony huggingface dataset:
Perceptual Redundancy Estimation of Screen Images via Multi-Domain Sensitivities

Visual redundancy detection is essential for image and video communication. Human visual system (HVS) is difficult to perceive the pixel magnitude change below a certain visibility threshold which is also known as just-noticeable-difference (JND). In this letter, we present an efficient JND estimation approach for screen content images by considering high-frequency sensitivity and orientation sensitivity correction. Specifically, to better quantify the visual redundancy, we investigate the visibility threshold based on the high-frequency distortion sensitivity. To obtain the orientation sensitivity correction, we divide the screen image pixels into three levels based on the oblique effect that considers the sensitive integrity of edges. Compared with several state-of-the-art JNDs, experimental results show that our method tolerates more perceptual redundancy, and delivers better visual quality under the same injected-noise energy. The implementation of the proposed method is publicly available at https://sites.google.com/site/wangmiaohui/.
Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation

Through dance, a wide range of emotions can be expressed. As virtual agents and robots continue to become part of our daily lives, the need for them to efficiently convey emotion and intent increases. When trained to dance, to what extent can AI learn to model the tacit mappings between sound and motion? Here, we explore the creative capacity of a generative model trained on 3D motion capture recordings of improvised dance. We perform a perceptual judgment experiment wherein respondents rate movement generated by our model as well as human performances. While the sound-motion mappings remain somewhat elusive, particularly when compared to examples of human dance, our study shows that in certain aspects related to perceived dance-likeness and expressivity, the model successfully mimics human dance movement. By employing a perceptual study to evaluate our generative model, we aim to further our ability to understand the affordances and limitations of creative AI.
Robustness of deep learning models on graphs: A survey

Machine learning (ML) technologies have achieved significant success in various downstream tasks, e.g., node classification, link prediction, community detection, graph classification and graph clustering. However, many studies have shown that the models built upon ML technologies are vulnerable to noises and adversarial attacks. A number of works have studied the robust models against noise or adversarial examples in image domains and text processing domains, however, it is more challenging to learn robust models in graph domains. Adding noises or perturbations on graph data will make the robustness even harder to enhance – the noises and perturbations of edges or node attributes are easy to propagate to other neighbors via the relational information on a graph. In this paper, we investigate and summarize the existing works that study the robust deep learning models against adversarial attacks or noises on graphs, namely the robust learning (models) on graphs. Specifically, we first provide some robustness evaluation metrics of model robustness on graphs. Then, we comprehensively provide a taxonomy which groups robust models on graphs into five categories: anomaly detection, adversarial training, pre-processing, attention mechanism, and certifiable robustness. Besides, we emphasize some promising future directions in learning robust models on graphs. Hopefully, our works can offer insights for the relevant researchers, thus providing assistance for their studies.
An architecture governance approach for Agile development by tailoring the Spotify model

Abstract The role of software architecture in large-scale Agile development is important because several teams need to work together to release a single software product while helping to maximise teams’ autonomy. Governing and aligning Agile architecture across autonomous squads (i.e., teams), when using the Spotify model, is a challenge because the Spotify model lacks practices for addressing Agile architecture governance. To explore how software architecture can be governed and aligned by scaling the Spotify model , we conducted a longitudinal embedded case study in a multinational FinTech organisation. Then, we developed and evaluated an approach for architectural governance by conducting an embedded case study. The collected data was analysed using Thematic Analysis and informed by selected Grounded Theory techniques such as memoing, open coding, constant comparison, and sorting. Our approach for architectural governance comprises an organisational structure change and an architecture change management process. The benefits reported by the practitioners include devolving architectural decision-making to the operational level (i.e., Architecture Owners), enhancing architectural knowledge sharing among squads, minimising wasted effort in architectural refactoring, and other benefits. The practitioners in our case study realised an improved squad autonomy by the ability to govern and align architectural decisions. We provide two key contributions in this paper. First, we present the characteristics of our proposed architectural governance approach, its evaluation, benefits, and challenges. Second, we present how the novel Heterogeneous Tailoring model was enhanced to accommodate our architectural governance approach.
Global Transport for Fluid Reconstruction with Learned Self-Supervision

We propose a novel method to reconstruct volumetric flows from sparse views via a global transport formulation. Instead of obtaining the space-time function of the observations, we reconstruct its motion based on a single initial state. In addition we introduce a learned self-supervision that constrains observations from unseen angles. These visual constraints are coupled via the transport constraints and a differentiable rendering step to arrive at a robust end-to-end reconstruction algorithm. This makes the reconstruction of highly realistic flow motions possible, even from only a single input view. We show with a variety of synthetic and real flows that the proposed global reconstruction of the transport process yields an improved reconstruction of the fluid motion.
A mixed heterogeneous factorization model for non-overlapping cross-domain recommendation

Cross-domain collaborative filtering has attracted a great deal of attention for its capability of dealing with data sparsity by transferring valuable knowledge from auxiliary domains to assist in recommendation in a target domain. State-of-the-art research has mainly focused on the scenario in which auxiliary domains share users or items with a target domain. However, such auxiliary data is rare or the acquisition of them is limited due to privacy concerns in real-world applications. We investigate a more realistic scenario in which auxiliary domains have neither users nor items overlapped with a target domain, thereby facilitating the collection of auxiliary data. In order to extract and transfer the sharing knowledge between auxiliary domains and target domain, we assume the user-item feedback in each domain consist of two parts: domain-transferrable information containing the shared knowledge and domain-reserved information reflecting the domain-specific characteristics. Correspondingly, we propose a mixed heterogeneous factorization model to capture the sharing knowledge and the domain-specific characteristics based on adapted tensor factorization and biased matrix factorization respectively and then combine them together in an accumulative way. Meanwhile, three types of domain heterogeneity including preference heterogeneity, characteristics heterogeneity and rating bias are taken into account in this model. Experimenting on four publicly available datasets across different domains, we show that our model is superior to state-of-the-art methods in rating prediction and top- N recommendation tasks. • We focus on a CDCF scenario without overlapped users/items among related domains. • We deal with domain heterogeneity comprehensively from three different aspects. • A modified PARAFAC approach is used to capture shared knowledge with heterogeneity. • We differentiate roles of auxiliary domains based on a subspace alignment technique. • Our model has superior performance in rating prediction and top-N recommendation tasks.
Towards On-Device Face Recognition in Body-worn Cameras

Face recognition technology related to recognizing identities is widely adopted in intelligence gathering, law enforcement, surveillance, and consumer applications. Recently, this technology has been ported to smartphones and body-worn cameras (BWC). Face recognition technology in body-worn cameras is used for surveillance, situational awareness, and keeping the officer safe. Only a handful of academic studies exist in face recognition using the body-worn camera. A recent study has assembled BWCFace facial image dataset acquired using a body-worn camera and evaluated the ResNet-50 model for face identification. However, for real-time inference in resource constraint body-worn cameras and privacy concerns involving facial images, on-device face recognition is required. To this end, this study evaluates lightweight MobileNet-V2, EfficientNet-BO, LightCNN-9 and LightCNN-29 models for face identification using body-worn camera. Experiments are performed on a publicly available BWCface dataset. The real-time inference is evaluated on three mobile devices. The comparative analysis is done with heavy-weight VGG-16 and ResNet-50 models along with six hand-crafted features to evaluate the trade-off between the performance and model size. Experimental results suggest the difference in maximum rank-l accuracy of lightweight LightCNN-29 over best-performing ResNet-50 is 1.85% and the reduction in model parameters is 23.49M. Most of the deep models obtained similar performances at rank-5 and rank-10. The inference time of LightCNNs is 2.1x faster than other models on mobile devices. The least performance difference of 14% is noted between LightCNN-29 and Local Phase Quantization (LPQ) descriptor at rank-l. In most of the experimental settings, lightweight LightCNN models offered the best trade-off between accuracy and the model size in comparison to most of the models.
Distributed Watchdogs Based on Blockchain for Securing Industrial Internet of Things

The Industrial Internet of Things (IIoT) could enhance automation and analytics in industrial environments. Despite the promising benefits of IIoT, securely managing software updates is a challenging problem for those critical applications. This is due to at least the intrinsic lack of software protection mechanisms in legacy industrial systems. In this paper, to address the challenges in building a secure software supply chain for industrial environments, we propose a new approach that leverages distributed watchdogs with blockchain systems in protecting software supply chains. For this purpose, we bind every entity with a unique identity in the blockchain and employ the blockchain as a delegated authenticator by mapping every reporting action to a non-fungible token transfer. Moreover, we present a detailed specification to clearly define the behavior of systems and to apply model checking.
Deep Image Compression with Latent Optimization and Piece-wise Quantization Approximation

Benefit from its capability of learning high-dimensional compact representation from raw data, the auto-encoders are widely used in various tasks of data compression. In particular, for deep image compression, auto-encoders generally take the responsibility of mapping original images to the latent representation to be coded. In this paper, we propose a new framework for deep image compression by devising a loss function for latent optimization, and adopting the differentiable approximation of quantization. In our experiments, both subjective and objective results can confirm the effectiveness of our contributions.
Adversarial Training on Point Clouds for Sim-to-Real 3D Object Detection

In this work we address the problem of 3D object detection from point clouds in data-limited environments. Training with simulated data is a common approach in such scenarios; however a sim-to-real gap exists between clean and crisp simulated clouds and noisy real clouds. Previous sim-to-real approaches for processing point cloud scenes have compressed clouds into 2D and used 2D transfer techniques. However, this may compress useful 3D information and does not effectively reason about the unstructured nature of point cloud data. We thus propose a 3D adversarial training architecture that leverages an adaptive sampling module to reason about the unstructured nature of point cloud data. Our approach encourages the 3D feature encoder to extract features that are invariant across simulated and real scenes. We validate our approach in the context of the DARPA Subterranean Challenge and demonstrate that our 3D adversarial training architecture improves 3D object detection performance by up to 15% depending on the data representation.
SLA-aware Workload Scheduling Using Hybrid Cloud Services

Cloud services have an auto-scaling feature for load balancing to meet the performance requirements of an application. Existing auto-scaling techniques are based on upscaling and downscaling cloud resources to distribute the dynamically varying workloads. However, bursty workloads pose many challenges for auto-scaling and sometimes result in Service Level Agreement (SLA) violations. Furthermore, over-provisioning or under-provisioning cloud resources to address dynamically evolving workloads results in performance degradation and cost escalation.
Interpretability of Deep Learning: A Survey

The interpretability research of deep learning is closely related to engineering, machine learning, mathematics, cognitive psychology and other disciplines. It has important theoretical research significance and practical application value in many fields such as information push, medical research, unmanned driving and information security. Past research has made some contributions to the black box problem of deep learning, but we still face a variety of challenges. For this reason, this paper first summarizes the history and related work of deep learning interpretability research. The present research status is introduced from three aspects: visual analysis, robust perturbation analysis and sensitivity analysis. This paper introduces the research on the construction of interpretable deep learning model from four aspects: model agent, logical reasoning, network node association analysis and traditional machine learning model. In addition, this paper also analyzes and discusses the shortcomings of the existing methods. Finally, the typical applications of interpretable deep learning are listed, and the possible future research directions in this field are prospected, and corresponding suggestions are put forward.
Graph-Based Multiobject Tracking with Embedded Particle Flow

Seamless situational awareness provided by modern radar systems relies on effective methods for multiobject tracking (MOT). This paper presents a graph-based Bayesian method for nonlinear and high-dimensional MOT problems that embeds particle flow. To perform operations on the graph effectively, particles are migrated towards regions of high likelihood based on the solution of a partial differential equation. This makes it possible to obtain good object detection and tracking performance with a relatively small number of particles even if object states are high dimensional and sensor measurements are very informative. Simulation results demonstrate reduced computational complexity and memory requirements as well as favorable detection and estimation accuracy in a challenging 3-D MOT scenario.
DNB: A Joint Learning Framework for Deep Bayesian Nonparametric Clustering

Clustering algorithms based on deep neural networks have been widely studied for image analysis. Most existing methods require partial knowledge of the true labels, namely, the number of clusters, which is usually not available in practice. In this article, we propose a Bayesian nonparametric framework, deep nonparametric Bayes (DNB), for jointly learning image clusters and deep representations in a doubly unsupervised manner. In doubly unsupervised learning, we are dealing with the problem of "unknown unknowns," where we estimate not only the unknown image labels but also the unknown number of labels as well. The proposed algorithm alternates between generating a potentially unbounded number of clusters in the forward pass and learning the deep networks in the backward pass. With the help of the Dirichlet process mixtures, the proposed method is able to partition the latent representations space without specifying the number of clusters a priori. An important feature of this work is that all the estimation is realized with an end-to-end solution, which is very different from the methods that rely on post hoc analysis to select the number of clusters. Another key idea in this article is to provide a principled solution to the problem of "trivial solution" for deep clustering, which has not been much studied in the current literature. With extensive experiments on benchmark datasets, we show that our doubly unsupervised method achieves good clustering performance and outperforms many other unsupervised image clustering methods.
Contrastive Learning for Natural Language-Based Vehicle Retrieval

AI City Challenge 2021 Task 5: The Natural Language-Based Vehicle Tracking is a Natural Language-based Vehicle Retrieval task, which requires retrieving a single-camera track using a set of three natural language descriptions of the specific targets. In this paper, we present our methods to tackle the difficulties of the provided task. Experiments with our approaches on the competitive dataset from AICity Challenge 2021 show that our techniques achieve Mean Reciprocal Rank score of 0.1701 on the public test dataset and 0.1571 on the private test dataset.
Security Analysis of End-to-End Encryption for Zoom Meetings

In the wake of the global COVID-19 pandemic, video conference systems have become essential for not only business purposes, but also private, academic, and educational uses.Among the various systems, Zoom is the most widely deployed video conference system.In October 2020, Zoom Video Communications rolled out their end-to-end encryption (E2EE) to protect conversations in a meeting from even insiders, namely, the service provider Zoom.In this study, we conduct thorough security evaluations of the E2EE of Zoom (version 2.3.1) by analyzing their cryptographic protocols.We discover several attacks more powerful than those expected by Zoom according to their whitepaper.Specifically, if insiders collude with meeting participants, they can impersonate any Zoom user in target meetings, whereas Zoom indicates that they can impersonate only the current meeting participants.Besides, even without relying on malicious participants, insiders can impersonate any Zoom user in target meetings though they cannot decrypt meeting streams.In addition, we demonstrate several impersonation attacks by meeting participants or insiders colluding with meeting participants.Although these attacks may be beyond the scope of the security claims made by Zoom or may be already mentioned in the whitepaper, we reveal the details of the attack procedures and their feasibility in the real-world setting and propose effective countermeasures in this paper.Our findings are not an immediate threat to the E2EE of Zoom; however, we believe that these security evaluations are of value for deeply understanding the security of E2EE of Zoom.
Design Considerations for Energy-efficient Inference on Edge Devices

The emergence of low-power accelerators has enabled deep learning models to be executed on mobile or embedded edge devices without relying on cloud resources. The energy-constrained nature of these devices requires a judicious choice of a deep learning model and system configuration parameter to meet application needs while optimizing energy used during deep learning inference.
Monocular Depth Estimation via Listwise Ranking using the Plackett-Luce Model

In many real-world applications, the relative depth of objects in an image is crucial for scene understanding. Recent approaches mainly tackle the problem of depth prediction in monocular images by treating the problem as a regression task. Yet, being interested in an order relation in the first place, ranking methods suggest themselves as a natural alternative to regression, and indeed, ranking approaches leveraging pairwise comparisons as training information ("object A is closer to the camera than B") have shown promising performance on this problem. In this paper, we elaborate on the use of so-called listwise ranking as a generalization of the pairwise approach. Our method is based on the Plackett-Luce (PL) model, a probability distribution on rankings, which we combine with a state-of-the-art neural network architecture and a simple sampling strategy to reduce training complexity. Moreover, taking advantage of the representation of PL as a random utility model, the proposed predictor offers a natural way to recover (shift-invariant) metric depth information from ranking-only data provided at training time. An empirical evaluation on several benchmark datasets in a "zero-shot" setting demonstrates the effectiveness of our approach compared to existing ranking and regression methods.
The Vulnerabilities in Smart Contracts: A Survey

Blockchain has attracted widespread attention since its inception and one of the special technologies is smart contracts. Smart contracts are programs on blockchain that act as trusted intermediary between the users and are widely used in variety of industry (e.g., IoT, supply chain management). Smart contracts can store or manipulate valuable assets which may cause huge economic losses. Unlike traditional computer programs, the code of a smart contract cannot be modified after it is deployed on the blockchain. Hence, the security analysis and vulnerability detection of the smart contract must be performed before its deployment. In this survey, we considered 15 security vulnerabilities in smart contracts and introduced the vulnerable areas and the causes of vulnerabilities. According to the methods used, we introduced the existing smart contract analysis methods and vulnerability detection tools from three aspects of static analysis, dynamic analysis and formal verification. Finally, by considering the analysis tools and security vulnerabilities, we found that a new attack cannot be detected by existing detection tools if the vulnerability without pre-defined. We recommend using machine learning methods to analyze smart contracts in combination with traditional program vulnerabilities, and find vulnerabilities that have not yet been discovered in smart contracts. In addition, many detection tools require too much resources or are too complex, so it is necessary to introduce new detection methods.
Fast and precise certification of transformers

We present DeepT, a novel method for certifying Transformer networks based on abstract interpretation. The key idea behind DeepT is our new Multi-norm Zonotope abstract domain, an extension of the classical Zonotope designed to handle ℓ1 and ℓ2-norm bound perturbations. We introduce all Multi-norm Zonotope abstract transformers necessary to handle these complex networks, including the challenging softmax function and dot product. Our evaluation shows that DeepT can certify average robustness radii that are 28× larger than the state-of-the-art, while scaling favorably. Further, for the first time, we certify Transformers against synonym attacks on long sequences of words, where each word can be replaced by any synonym. DeepT achieves a high certification success rate on sequences of words where enumeration-based verification would take 2 to 3 orders of magnitude more time.