Reviewerly/exHarmony huggingface dataset:
Skeleton-Aware Neural Sign Language Translation

As an essential communication way for deaf-mutes, sign languages are expressed by human actions. To distinguish human actions for sign language understanding, the skeleton which contains position information of human pose can provide an important cue, since different actions usually correspond to different poses/skeletons. However, skeleton has not been fully studied for Sign Language Translation (SLT), especially for end-to-end SLT. Therefore, in this paper, we propose a novel end-to-end Skeleton-Aware neural Network (SANet) for video-based SLT. Specifically, to achieve end-to-end SLT, we design a self-contained branch for skeleton extraction. To efficiently guide the feature extraction from video with skeletons, we concatenate the skeleton channel and RGB channels of each frame for feature extraction. To distinguish the importance of clips, we construct a skeleton-based Graph Convolutional Network (GCN) for feature scaling, i.e., giving importance weight for each clip. The scaled features of each clip are then sent to a decoder module to generate spoken language. In our SANet, a joint training strategy is designed to optimize skeleton extraction and sign language translation jointly. Experimental results on two large scale SLT datasets demonstrate the effectiveness of our approach, which outperforms the state-of-the-art methods. Our code is available at https://github.com/SignLanguageCode/SANet.
View-Invariant Human Action Recognition Via View Transformation Network (VTN)

Since the human body is non-rigid, actions captured in different views always involve action occlusion and information loss. Recently, view-variation-related human action recognition is still a challenging problem. To address the problem, we propose a View Transformation Network (VTN) that realizes the view normalization by transforming arbitrary-view action samples to a base view to seek for a view-invariant representation. an attention learning module is designed to learn a co-attention for action samples of different views, that contributes to output a similar feature representation to erase the view diversity in different views. Extensive and fair evaluations are performed on the UESTC varying-view RGB-D dataset, the NTU RGB-D 60 dataset, and the NTU RGB-D 120 dataset, where three evaluation types, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> X-subject, X-view, and A-view recognition, are performed. Experiments illustrate that our VTN model achieves outstanding performance.
Multiple graph regularized semi-supervised nonnegative matrix factorization with adaptive weights for clustering

Traditional multiple graph regularized nonnegative matrix factorization (NMF) techniques have shown good performance in image clustering applications. However, existing multiple graph regularized NMF methods are unsupervised learning methods which fail to take full advantage of priori information. To solve this issue, this paper develops a novel multiple graph regularized NMF method, namely the multiple graph regularized semi-supervised NMF with adaptive weights (MSNMF), to capture the discriminative data representation. Specifically, the MSNMF method combines the limited supervised information in the form of pairwise constraints, into multiple graph regularization, and propagates the pairwise constraints from the constrained data samples to the unconstrained data samples. Moreover, convergence, connection with the gradient descent method, and computational cost of the proposed method are studied. The relationships between MSNMF and some typical NMF methods are also discussed. Experimental results on eight practical image datasets have shown that the MSNMF method can obtain better clustering results than several related NMF methods.
Deep ladder reconstruction-classification network for unsupervised domain adaptation

Unsupervised Domain Adaptation aims to learn a classifier for an unlabeled target domain by transferring knowledge from a labeled source domain. Most existing approaches learn domain-invariant features by adapting the entire information of each image. However, forcing adaptation of domain-specific components can undermine the effectiveness of learned features. We propose a novel architecture called Deep Ladder Reconstruction-Classification Network (DLaReC) which is designed to learn cross-domain shared contents by suppressing domain-specific variations. The DLaReC adopts an encoder with cross-domain sharing and a target-domain reconstruction decoder. The encoder and decoder are connected with residual shortcuts at each intermediate layer. By this means, the domain-specific components are directly fed to the decoder for reconstruction, relieving the pressure to learn domain-specific variations at later layers of the shared encoder. Therefore, DLaReC allows the encoder to focus on learning cross-domain shared representations and ignore domain-specific variations. DLaReC is implemented by jointly learning three tasks: supervised classification of the source domain, unsupervised reconstruction of the target domain and cross-domain shared representation adaptation. Extensive experiments on Digit, Office31, ImageCLEF-DA and Office-Home datasets demonstrate the DLaReC outperforms state-of-the-art methods on the whole. The average accuracy on the Digit datasets, for instance, is improved from 95.6% to 96.9%. In addition, the result on Amazon → Webcam obtains significant improvement, i.e., from 91.1% to 94.7%.
Performance Modeling and FPGA Acceleration of Homomorphic Encrypted Convolution

Privacy of data is a critical concern when applying Machine Learning (ML) techniques to domains with sensitive data. Homomorphic Encryption (HE), by enabling computations on encrypted data, has emerged as a promising approach to perform inference on ML models such as Convolution Neural Network (CNN) in a privacy preserving manner. A significant portion of the total inference latency is in performing convolution over homomorphic encrypted data (HE-Convolution). For performing convolution over plaintext data, low latency accelerator designs have been proposed using algorithms such as im2col, frequency domain convolution, etc. However, developing accelerators for the HE versions of these algorithms is non-trivial. In this work, we develop a unified FPGA design that enables low latency execution of both im2col and frequency domain HE-Convolution. To enable selection of the efficient algorithm for each convolution layer of a CNN, we develop a performance model that takes the parameters about the encryption and convolution layer as input and outputs the computation and resource requirements of the two algorithms for that layer. We use the performance model to select convolution algorithm for each layer of ResNet-50 and obtain the first low latency batch-1 inference accelerator for CNN inference with HE-Convolution targeting FPGAs using HLS. We compare our design against prior techniques on CPUs and show that our accelerator achieves speedups in the range of $3.4\times\sim 6.7\times$ in latency.
Cross-modal Joint Prediction and Alignment for Composed Query Image Retrieval

In this paper, we focus on the composed query image retrieval task, namely retrieving the target images that are similar to a composed query, in which a modification text is combined with a query image to describe a user's accurate search intention. Previous methods usually focus on learning the joint image-text representations, but rarely consider the intrinsic relationship among the query image, the target image and the modification text. To address this problem, we propose a new cross-modal joint prediction and alignment framework for composed query image retrieval. In our framework, the modification text is regarded as an implicit transformation between the query image and the target image. Motivated by that, not only the combination of the query image and modification text should be similar to the target image, but also the modification text should be predicted according to the query image and the target image. We devote to aligning this relationship by a novel Joint Prediction Module (JPM). Our proposed framework can seamlessly incorporate the JPM into the existing methods to effectively improve the discrimination and robustness of visual and textual representations. The experiments on three public datasets demonstrate the effectiveness of our proposed framework, proving that our proposed JPM can be simply incorporated with the existing methods while effectively improving the performance.
Achieving change requirements of feature models by an evolutionary approach

Feature models are a widely used modeling notation for variability and commonality management in software product line (SPL) engineering. In order to keep an SPL and its feature model aligned, feature models must be changed by including/excluding new features and products, either because faults in the model are found or to reflect the normal evolution of the SPL. The modification of the feature model to be made to satisfy these change requirements can be complex and error-prone. In this paper, we present a method that is able to automatically update a feature model in order to satisfy a given update request. The method is based on an evolutionary algorithm that iteratively applies structure-preserving mutations to the original model, until the model is completely updated or some other termination condition occurs. Among all the possible models achieving the update request, the method privileges those structurally simpler. We evaluate the approach on real-world feature models; although it does not guarantee to completely update all the possible feature models, empirical analysis shows that, on average, around 89% of requested changes are applied.
Quantum Graph Convolutional Neural Networks

At present, there are a large number of quantum neural network models to deal with Euclidean spatial data, while little research have been conducted on non-Euclidean spatial data. In this paper, we propose a novel quantum graph convolutional neural network (QGCN) model based on quantum parametric circuits and utilize the computing power of quantum systems to accomplish graph classification tasks in traditional machine learning. The proposed QGCN model has a similar architecture as the classical graph convolutional neural networks, which can illustrate the topology of the graph type data and efficiently learn the hidden layer representation of node features as well. Numerical simulation results on a graph dataset demonstrate that the proposed model can be effectively trained and has good performance in graph level classification tasks.
An approach combining a new weight initialization method and constructive algorithm to configure a single Feedforward Neural Network for multi-class classification

This paper presents a new method for initializing weights in a Feedforward Neural Network (FNN) with a single hidden layer combined with a constructive approach to define the number of hidden units associated with the best classification performance. The strategy consists of defining an initial number of hidden units according to the classification problem, the linearization of the whole network around an equilibrium point and the determination of the initial weights and bias through the maximum approximation of the linearized model to the Optimal Linear Classifier (OLC) whose solution can be obtained analytically. The constructive algorithm comprises a gradual increase in the number of hidden units in such a way that at each training only the weights and bias associated with the new hidden units are initiated randomly while the weights and bias obtained from previous training are used as initial guesses. Additionally, the constructive algorithm seeks to ensure that the loss function of the trained networks decreases with the successive additions of hidden units. The proposed approach (Weight Initialization based on the Linearization of the Whole Neural Network combined with a new Constructive Algorithm, WILWNN-CA) is applied to synthetic and real datasets widely used as benchmark for multi-class classification problems. The comparison with conventional random weight initialization and other approaches involving different network topologies (and initialization strategies) shows that the proposed method is efficient and capable of providing success rates (correct classification rates) higher or similar to those achieved with existing methods.
On the Autoregressive Time Series Model Using Real and Complex Analysis

The autoregressive model is a tool used in time series analysis to describe and model time series data. Its main structure is a linear equation using the previous values to compute the next time step; i.e., the short time relationship is the core component of the autoregressive model. Therefore, short-term effects can be modeled in an easy way, but the global structure of the model is not obvious. However, this global structure is a crucial aid in the model selection process in data analysis. If the global properties are not reflected in the data, a corresponding model is not compatible. This helpful knowledge avoids unsuccessful modeling attempts. This article analyzes the global structure of the autoregressive model through the derivation of a closed form. In detail, the closed form of an autoregressive model consists of the basis functions of a fundamental system of an ordinary differential equation with constant coefficients; i.e., it consists of a combination of polynomial factors with sinusoidal, cosinusoidal, and exponential functions. This new insight supports the model selection process.
TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images

Today’s touchscreen devices commonly detect the coordinates of user input through capacitive sensing. Yet, these coordinates are the mere 2D manifestations of the more complex 3D configuration of the whole hand—a sensation that touchscreen devices so far remain oblivious to. In this work, we introduce the problem of reconstructing a 3D hand skeleton from capacitive images, which encode the sparse observations captured by touch sensors. These low-resolution images represent intensity mappings that are proportional to the distance to the user’s fingers and hands.
Generative Adversarial Graph Convolutional Networks for Human Action Synthesis

Synthesising the spatial and temporal dynamics of the human body skeleton remains a challenging task, not only in terms of the quality of the generated shapes, but also of their diversity, particularly to synthesise realistic body movements of a specific action (action conditioning). In this paper, we propose Kinetic-GAN, a novel architecture that leverages the benefits of Generative Adversarial Networks and Graph Convolutional Networks to synthesise the kinetics of the human body. The proposed adversarial architecture can condition up to 120 different actions over local and global body movements while improving sample quality and diversity through latent space disentanglement and stochastic variations. Our experiments were carried out in three well-known datasets, where Kinetic-GAN notably surpasses the state-of-the-art methods in terms of distribution quality metrics while having the ability to synthesise more than one order of magnitude regarding the number of different actions. Our code and models are publicly available at https://github.com/DegardinBruno/Kinetic-GAN.
SAC: Semantic Attention Composition for Text-Conditioned Image Retrieval

The ability to efficiently search for images is essential for improving the user experiences across various products. Incorporating user feedback, via multi-modal inputs, to navigate visual search can help tailor retrieved results to specific user queries. We focus on the task of text-conditioned image retrieval that utilizes support text feedback alongside a reference image to retrieve images that concurrently satisfy constraints imposed by both inputs. The task is challenging since it requires learning composite image-text features by incorporating multiple cross-granular semantic edits from text feedback and then applying the same to visual features. To address this, we propose a novel framework SAC which resolves the above in two major steps: "where to see" (Semantic Feature Attention) and "how to change" (Semantic Feature Modification). We systematically show how our architecture streamlines the generation of text-aware image features by removing the need for various modules required by other state-of-art techniques. We present extensive quantitative, qualitative analysis, and ablation studies, to show that our architecture SAC outperforms existing techniques by achieving state-of-the-art performance on 3 benchmark datasets: FashionIQ, Shoes, and Birds-to-Words, while supporting natural language feedback of varying lengths.
FOCAS

Super-resolution (SR) is a well-studied technique for reconstructing high-resolution (HR) images from low-resolution (LR) ones. SR holds great promise for video streaming since an LR video segment can be transmitted from the video server to the client that then reconstructs the HR version using SR, resulting in a significant reduction in network bandwidth. However, SR is seldom used in practice for real-time video streaming, because the computational overhead of frame reconstruction results in large latency and low frame rate.
Robotic-Based Well-Being Monitoring and Coaching System for the Elderly in Their Daily Activities

The increasingly ageing population and the tendency to live alone have led science and engineering researchers to search for health care solutions. In the COVID 19 pandemic, the elderly have been seriously affected in addition to suffering from isolation and its associated and psychological consequences. This paper provides an overview of the RobWell (Robotic-based Well-Being Monitoring and Coaching System for the Elderly in their Daily Activities) system. It is a system focused on the field of artificial intelligence for mood prediction and coaching. This paper presents a general overview of the initially proposed system as well as the preliminary results related to the home automation subsystem, autonomous robot navigation and mood estimation through machine learning prior to the final system integration, which will be discussed in future works. The main goal is to improve their mental well-being during their daily household activities. The system is composed of ambient intelligence with intelligent sensors, actuators and a robotic platform that interacts with the user. A test smart home system was set up in which the sensors, actuators and robotic platform were integrated and tested. For artificial intelligence applied to mood prediction, we used machine learning to classify several physiological signals into different moods. In robotics, it was concluded that the ROS autonomous navigation stack and its autodocking algorithm were not reliable enough for this task, while the robot's autonomy was sufficient. Semantic navigation, artificial intelligence and computer vision alternatives are being sought.
Using a Hybrid-Classification Method to Analyze Twitter Data During Critical Events

In this paper, sentiment analysis of two critical events is presented using machine learning (ML) techniques. COVID-19 has put immense pressure across the globe and sentiment analysis of data from Twitter using ML techniques has become a hot topic. We extract the COVID-19 and Expo2020 data from twitter. First, we evaluate the Twitter data of these two significant events for sentiment analysis and then use the classification algorithm to find out the usefulness of the proposed methodology. A hybrid approach that uses supervised learning model Support Vector Machine (SVM) combined with Bayes Factor Tree Augmented Naive Bayes (BFTAN) technique is proposed to accurately classify the input tweet while keeping in mind the different challenges of sentiment analysis. Our study has four main contributions: a) hybrid classification techniques are thoroughly explored for sentiment analysis, b) a novel hybrid classification approach is proposed for sentiment analysis, c) a new Twitter dataset related to COVID-19 that can be used for future research, d) empirical study to show that the hybrid-classification approach can achieve comparable performance in improving accuracy, identifying the polarity of comparative sentences, distinguishing the intensity of opinion words, considering negative words, and handling sarcasm as well. The experimental results show that the proposed approach is robust in producing correct classification results with the tradeoff of poor time efficiency. Also, the accuracy of the proposed model is comparable to other classifiers, which is encouraging. Class distribution of each dataset demonstrates that more than 60% of tweets are negative.
Big data-driven scheduling optimization algorithm for Cyber–Physical Systems based on a cloud platform

In this paper, we study big data-driven Cyber–Physical Systems (CPS) through cloud platforms and design scheduling optimization algorithms to improve the efficiency of the system. A task scheduling scheme for large-scale factory access under cloud–edge collaborative computing architecture is proposed. The method firstly merges the directed acyclic graphs on cloud-side servers and edge-side servers; secondly, divide the tasks using a critical path-based partitioning strategy to effectively improve the allocation accuracy; then achieves load balancing through reasonable processor allocation, and finally compares and analyses the proposed task scheduling algorithm through simulation experiments. The experimental system is thoroughly analysed, hierarchically designed, and modelled, simulated, and the experimental data analysed and compared with related methods. The experimental results prove the effectiveness and correctness of the worst-case execution time analysis method and the idea of big data-driven CPS proposed in this paper and show that big data knowledge can help improve the accuracy of worst-case execution time analysis. This paper implements a big data-driven scheduling optimization algorithm for Cyber–Physical Systems based on a cloud platform, which improves the accuracy and efficiency of the algorithm by about 15% compared to other related studies.
Design and development of machine learning based resume ranking system

Finding acceptable applicants for a vacant job might be a difficult process, especially when there are many prospects. The manual process of screening resumes could stymie the team's efforts to locate the right individual at the right moment. The laborious screening may be greatly aided by an automated technique for screening and ranking applicants. In our work, the top applicants might be rated using content-based suggestion, which uses cosine similarity to find the curriculum vitae that are the most comparable to the job description supplied and KNN algorithm is used to pick and rank Curriculum Vitaes (CV) based on job descriptions in huge quantities. Experimental results indicate the performance of the proposed system as an average text parsing accuracy of 85% and a ranking accuracy of 92%.
GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor

A large, generative language model's output can be influenced through well-designed prompts, or text-based inputs that establish textual patterns that the model replicates in its output [6]. These capabilities create new opportunities for novel interactions with large, generative language models. We present a macro system with two tools that allow users to invoke language model prompts as macros in a code editor. GenLine allows users to execute macros inline as they write code in the editor (e.g., "Make an OK button" produces the equivalent HTML). GenForm provides a form-like interface where the user provides input that is then transformed into multiple pieces of output at the same time (e.g., a description of web code is transformed into HTML, CSS, and JavaScript).