Reviewerly/exHarmony huggingface dataset:
Graph-Based Multimodal Sequential Embedding for Sign Language Translation

Sign language translation (SLT) is a challenging weakly supervised task without word-level annotations. An effective method of SLT is to leverage multimodal complementarity and to explore implicit temporal cues. In this work, we propose a graph-based multimodal sequential embedding network (MSeqGraph), in which multiple sequential modalities are densely correlated. Specifically, we build a graph structure to realize the intra-modal and inter-modal correlations. First, we design a graph embedding unit (GEU), which embeds a parallel convolution with channel-wise and temporal-wise learning into the graph convolution to learn the temporal cues in each modal sequence and cross-modal complementarity. Then, a hierarchical GEU stacker with a pooling-based skip connection is proposed. Unlike the state-of-the-art methods, to obtain a compact and informative representation of multimodal sequences, the GEU stacker gradually compresses the channel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$d$</tex-math></inline-formula> with multi-modalities <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$m$</tex-math></inline-formula> rather than the temporal dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$t$</tex-math></inline-formula> . Finally, we adopt the connectionist temporal decoding strategy to explore the entire video’s temporal transition and translate the sentence. Extensive experiments on the USTC-CSL and BOSTON-104 datasets demonstrate the effectiveness of the proposed method.
Improving Human–Robot Interaction by Enhancing NAO Robot Awareness of Human Facial Expression

An intriguing challenge in the human-robot interaction field is the prospect of endowing robots with emotional intelligence to make the interaction more genuine, intuitive, and natural. A crucial aspect in achieving this goal is the robot's capability to infer and interpret human emotions. Thanks to its design and open programming platform, the NAO humanoid robot is one of the most widely used agents for human interaction. As with person-to-person communication, facial expressions are the privileged channel for recognizing the interlocutor's emotional expressions. Although NAO is equipped with a facial expression recognition module, specific use cases may require additional features and affective computing capabilities that are not currently available. This study proposes a highly accurate convolutional-neural-network-based facial expression recognition model that is able to further enhance the NAO robot' awareness of human facial expressions and provide the robot with an interlocutor's arousal level detection capability. Indeed, the model tested during human-robot interactions was 91% and 90% accurate in recognizing happy and sad facial expressions, respectively; 75% accurate in recognizing surprised and scared expressions; and less accurate in recognizing neutral and angry expressions. Finally, the model was successfully integrated into the NAO SDK, thus allowing for high-performing facial expression classification with an inference time of 0.34 ± 0.04 s.
Diverse Image Style Transfer via Invertible Cross-Space Mapping

Image style transfer aims to transfer the styles of artworks onto arbitrary photographs to create novel artistic images. Although style transfer is inherently an underdetermined problem, existing approaches usually assume a deterministic solution, thus failing to capture the full distribution of possible outputs. To address this limitation, we propose a Diverse Image Style Transfer (DIST) framework which achieves significant diversity by enforcing an invertible cross-space mapping. Specifically, the framework consists of three branches: disentanglement branch, inverse branch, and stylization branch. Among them, the disentanglement branch factorizes artworks into content space and style space; the inverse branch encourages the invertible mapping between the latent space of input noise vectors and the style space of generated artistic images; the stylization branch renders the input content image with the style of an artist. Armed with these three branches, our approach is able to synthesize significantly diverse stylized images without loss of quality. We conduct extensive experiments and comparisons to evaluate our approach qualitatively and quantitatively. The experimental results demonstrate the effectiveness of our method.
Parallel random matrix particle swarm optimization scheduling algorithms with budget constraints on cloud computing systems

Nowadays, increasing number of Internet of Things and mobile Internet application services are migrated to cloud computing systems. One of the most important cloud challenges for this business is to optimize services cost. The efficient way to deal with this challenge is to improve the performance of resource management and task scheduling in cloud systems. However, this kind of service scheduling is a typical combinatorial optimization problem, and it is difficult to obtain the optimal solution. In this study, we first formalize this cloud services on virtual machines (VMs) with budget constraints scheduling problem. Then, we propose a random matrix particle swarm optimization scheduling algorithm (RMPSO), which uses the random integer matrix to represent its position and a feasible task scheduling scheme, to achieve the optimal total cost of cloud services. However, the drawback of this solution for large-scale systems is its high time complexity. Therefore, we propose two parallel RMPSO algorithms: CPU parallel algorithm (M-RMPSO) on multi-core system with shared memory and manycore GPU-accelerated strategy (G-RMPSO) to reduce its time complexity. Finally, the rigorous performance evaluation results clearly show that our proposed G-RMPSO outperforms M-RMPSO and existing FMPSO, HYBRID (MPSO+MCSO) in terms of cloud services total cost and algorithm execution time. Therefore, our proposed GPU-accelerated G-RMPSO algorithm is very suitable for cloud service scheduling.
VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models

Machine learning (ML) is increasingly applied to Electronic Health Records (EHRs) to solve clinical prediction tasks. Although many ML models perform promisingly, issues with model transparency and interpretability limit their adoption in clinical practice. Directly using existing explainable ML techniques in clinical settings can be challenging. Through literature surveys and collaborations with six clinicians with an average of 17 years of clinical experience, we identified three key challenges, including clinicians' unfamiliarity with ML features, lack of contextual information, and the need for cohort-level evidence. Following an iterative design process, we further designed and developed VBridge, a visual analytics tool that seamlessly incorporates ML explanations into clinicians' decision-making workflow. The system includes a novel hierarchical display of contribution-based feature explanations and enriched interactions that connect the dots between ML features, explanations, and data. We demonstrated the effectiveness of VBridge through two case studies and expert interviews with four clinicians, showing that visually associating model explanations with patients' situational records can help clinicians better interpret and use model predictions when making clinician decisions. We further derived a list of design implications for developing future explainable ML tools to support clinical decision-making.
Learning to Regress Bodies from Images using Differentiable Semantic Rendering

Learning to regress 3D human body shape and pose (e.g. SMPL parameters) from monocular images typically exploits losses on 2D keypoints, silhouettes, and/or part-segmentation when 3D training data is not available. Such losses, however, are limited because 2D keypoints do not supervise body shape and segmentations of people in clothing do not match projected minimally-clothed SMPL shapes. To exploit richer image information about clothed people, we introduce higher-level semantic information about clothing to penalize clothed and non-clothed regions of the human body differently. To do so, we train a body regressor using a novel "Differentiable Semantic Rendering (DSR)" loss. For Minimally-Clothed (MC) regions, we define the DSR-MC loss, which encourages a tight match between a rendered SMPL body and the minimally-clothed regions of the image. For clothed regions, we define the DSR-C loss to encourage the rendered SMPL body to be inside the clothing mask. To ensure end-to-end differentiable training, we learn a semantic clothing prior for SMPL vertices from thousands of clothed human scans. We perform extensive qualitative and quantitative experiments to evaluate the role of clothing semantics on the accuracy of 3D human pose and shape estimation. We outperform all previous state-of-the-art methods on 3DPW and Human3.6M and obtain on par results on MPI-INF-3DHP. Code and trained models are available for research at https://dsr.is.tue.mpg.de/.
Integrated Task Allocation and Path Coordination for Large-Scale Robot Networks With Uncertainties

Artificial intelligence-enhanced autonomous unmanned systems, such as large-scale autonomous robot networks, are widely used in logistic and industrial applications. In this article, we address the integrated task assignment, path planning, and coordination problem applied for large-scale robot networks with the existence of uncertainties. In particular, a novel generalized conflict graph is designed which encodes the traveling time cost of the subsequent path planning result of each task-robot assignment and also includes the predicted path conflicts of each two assignments. An integrated optimization problem which aims to minimize the total traveling cost and potential path conflicts simultaneously is first formulated and then transformed into a linear programming instance to obtain the optimal solution. In particular, to satisfy the real-time requirement in large-scale systems, a greedy solution is presented which has the near-optimal performance but can decrease the computational complexity by orders of magnitude. The optimality, scalability, robustness, and efficiency of our approach are demonstrated by comprehensive comparisons with existing state-of-the-art approaches. Note to Practitioners—With the development of artificial intelligence techniques, large-scale autonomous robot networks are increasingly used in the logistic warehouses, unmanned container terminals, and intelligence transportation systems. This article considers the large-scale networks with hundreds or even thousands of unmanned robots which are implemented in lifelong transportation systems with uncertainties existed in practical execution process. Our main concept is to simultaneously minimize the total time cost of all the tasks and the potential motion conflicts among all the robots in the subsequent execution stage, thus alleviating robot congestions, balancing traffic distributions, increasing system efficiency, and improving the robustness and scalability. Lifelong simulations with thousand robots illustrate that our approach can reduce more than 30% of the time steps consumed for coordinating robot motion conflicts, and in the meantime, the throughput, and overall system efficiency are improved. However, simulation results show that the system improvement decreases in the presence of extreme high uncertainties (such as temporary motion and communication failures of the robot), due to the inaccurate conflict prediction in the integrated optimization stage. Our future work includes the deep-learning-based traffic evolution prediction and the online reallocation and planning in highly dynamic scenarios.
PixelSynth: Generating a 3D-Consistent Experience from a Single Image

Recent advancements in differentiable rendering and 3D reasoning have driven exciting results in novel view synthesis from a single image. Despite realistic results, methods are limited to relatively small view change. In order to synthesize immersive scenes, models must also be able to extrapolate. We present an approach that fuses 3D reasoning with autoregressive modeling to outpaint large view changes in a 3D-consistent manner, enabling scene synthesis. We demonstrate considerable improvement in single-image large-angle view synthesis results compared to a variety of methods and possible variants across simulated and real datasets. In addition, we show increased 3D consistency compared to alternative accumulation methods.
Multi-Expert Adversarial Attack Detection in Person Re-identification Using Context Inconsistency

The success of deep neural networks (DNNs) has promoted the widespread applications of person re-identification (ReID). However, ReID systems inherit the vulnerability of DNNs to malicious attacks of visually in-conspicuous adversarial perturbations. Detection of adversarial attacks is, therefore, a fundamental requirement for robust ReID systems. In this work, we propose a Multi-Expert Adversarial Attack Detection (MEAAD) approach to achieve this goal by checking context inconsistency, which is suitable for any DNN-based ReID systems. Specifically, three kinds of context inconsistencies caused by adversarial attacks are employed to learn a detector for distinguishing the perturbed examples, i.e., a) the embedding distances between a perturbed query person image and its top-K retrievals are generally larger than those between a benign query image and its top-K retrievals, b) the embedding distances among the top-K retrievals of a perturbed query image are larger than those of a benign query image, c) the top-K retrievals of a benign query image obtained with multiple expert ReID models tend to be consistent, which is not preserved when attacks are present. Extensive experiments on the Market1501 and DukeMTMC-ReID datasets show that, as the first adversarial attack detection approach for ReID, MEAAD effectively detects various adversarial attacks and achieves high ROC-AUC (over 97.5%).
Performance of a Kinetic Inductance Traveling-Wave Parametric Amplifier at 4 Kelvin: Toward an Alternative to Semiconductor Amplifiers

Most microwave readout architectures in quantum computing or sensing rely on a semiconductor amplifier at 4 K, typically a high-electron mobility transistor (HEMT). Despite its remarkable noise performance, a conventional HEMT dissipates several milliwatts of power, posing a practical challenge to scale up the number of qubits or sensors addressed in these architectures. As an alternative, we present an amplification chain consisting of a kinetic inductance traveling-wave parametric amplifier (KITWPA) placed at 4 K, followed by a HEMT placed at 70 K, and demonstrate a chain-added noise TΣ=6.3±0.5K between 3.5 and 5.5 GHz. While, in principle, any parametric amplifier can be quantum limited even at 4 K, in practice we find the performance of the KITWPA to be limited by the temperature of its inputs and by an excess of noise Tex=1.9K. The dissipation of the rf pump of the KITWPA constitutes the main power load at 4 K and is about 1% that of a HEMT. These combined noise and power dissipation values pave the way for the use of the KITWPA as a replacement for semiconductor amplifiers.
A comparison of machine learning algorithms on design smell detection using balanced and imbalanced dataset: A study of God class

Design smell detection has proven to be a significant activity that has an aim of not only enhancing the software quality but also increasing its life cycle. This work investigates whether machine learning approaches can effectively be leveraged for software design smell detection. Additionally, this paper provides a comparatively study, focused on using balanced datasets, where it checks if avoiding dataset balancing can be of any influence on the accuracy and behavior during design smell detection. A set of experiments have been conducted-using 28 Machine Learning classifiers aimed at detecting God classes. This experiment was conducted using a dataset formed from 12,587 classes of 24 software systems, in which 1,958 classes were manually validated. Ultimately, most classifiers obtained high performances,-with Cat Boost showing a higher performance. Also, it is evident from the experiments conducted that data balancing does not have any significant influence on the accuracy of detection. This reinforces the application of machine learning in real scenarios where the data is usually imbalanced by the inherent nature of design smells. Machine learning approaches can effectively be used as a leverage for God class detection. While in this paper we have employed SMOTE technique for data balancing, it is worth noting that there exist other methods of data balancing and with other design smells. Furthermore, it is also important to note that application of those other methods may improve the results, in our experiments SMOTE did not improve God class detection. The results are not fully generalizable because only one design smell is studied with projects developed in a single programming language, and only one balancing technique is used to compare with the imbalanced case. But these results are promising for the application in real design smells detection scenarios as mentioned above and the focus on other measures, such as Kappa, ROC, and MCC, have been used in the assessment of the classifier behavior.
Pose-guided Inter- and Intra-part Relational Transformer for Occluded Person Re-Identification

Person Re-Identification (Re-Id) in occlusion scenarios is a challenging problem because a pedestrian can be partially occluded. The use of local information for feature extraction and matching is still necessary. Therefore, we propose a Pose-guided inter- and intra-part relational transformer (Pirt) for occluded person Re-Id, which builds part-aware long-term correlations by introducing transformer. In our framework, we firstly develop a pose-guided feature extraction module with regional grouping and mask construction for robust feature representations. The positions of a pedestrian in the image under surveillance scenarios are relatively fixed, hence we propose intra-part and inter-part relational transformer. The intra-part module creates local relations with mask-guided features, while the inter-part relationship builds correlations with transformers, to develop cross relationships between part nodes. With the collaborative learning inter- and intra-part relationships, experiments reveal that our proposed Pirt model achieves a new state of the art on the public occluded dataset, and further extensions on standard non-occluded person Re-Id datasets also reveal our comparable performances.
Active Learning for Network Traffic Classification: A Technical Study

Network Traffic Classification (NTC) has become an important feature in various network management operations, e.g., Quality of Service (QoS) provisioning and security services. Machine Learning (ML) algorithms as a popular approach for NTC can promise reasonable accuracy in classification and deal with encrypted traffic. However, ML-based NTC techniques suffer from the shortage of labeled traffic data which is the case in many real-world applications. This study investigates the applicability of an active form of ML, called Active Learning (AL), in NTC. AL reduces the need for a large number of labeled examples by actively choosing the instances that should be labeled. The study first provides an overview of NTC and its fundamental challenges along with surveying the literature on ML-based NTC methods. Then, it introduces the concepts of AL, discusses it in the context of NTC, and review the literature in this field. Further, challenges and open issues in AL-based classification of network traffic are discussed. Moreover, as a technical survey, some experiments are conducted to show the broad applicability of AL in NTC. The simulation results show that AL can achieve high accuracy with a small amount of data.
APNN-TC

Over the years, accelerating neural networks with quantization has been widely studied. Unfortunately, prior efforts with diverse precisions (e.g., 1-bit weights and 2-bit activations) are usually restricted by limited precision support on GPUs (e.g., int1 and int4). To break such restrictions, we introduce the first Arbitrary Precision Neural Network framework (APNN-TC)1 to fully exploit quantization benefits on Ampere GPU Tensor Cores. Specifically, APNN-TC first incorporates a novel emulation algorithm to support arbitrary short bit-width computation with int1 compute primitives and XOR/AND Boolean operations. Second, APNN-TC integrates arbitrary precision layer designs to efficiently map our emulation algorithm to Tensor Cores with novel batching strategies and specialized memory organization. Third, APNN-TC embodies a novel arbitrary precision NN design to minimize memory access across layers and further improve performance. Extensive evaluations show that APNN-TC can achieve significant speedup over CUTLASS kernels and various NN models, such as ResNet and VGG.
Recycling Waste Classification Using Vision Transformer on Portable Device

Recycling resources from waste can effectively alleviate the threat of global resource strain. Due to the wide variety of waste, relying on manual classification of waste and recycling recyclable resources would be costly and inefficient. In recent years, automatic recyclable waste classification based on convolutional neural network (CNN) has become the mainstream method of waste recycling. However, due to the receptive field limitation of the CNN, the accuracy of classification has reached a bottleneck, which restricts the implementation of relevant methods and systems. In order to solve the above challenges, in this study, a deep neural network architecture only based on self-attention mechanism, named Vision Transformer, is proposed to improve the accuracy of automatic classification. Experimental results on TrashNet dataset show that the proposed method can achieve the highest accuracy of 96.98%, which is better than the existing CNN-based method. By deploying the well-trained model on the server and using a portable device to take pictures of waste in order to upload to the server, automatic waste classification can be expediently realized on the portable device, which broadens the scope of application of automatic waste classification and is of great significance with respect to resource conservation and recycling.