Reviewerly/exHarmony huggingface dataset:
Active Learning for Video Classification with Frame Level Queries

Deep learning algorithms have pushed the boundaries of computer vision research and have depicted commendable performance in a variety of applications. However, training a robust deep neural network necessitates a large amount of labeled training data, acquiring which involves significant time and human effort. This problem is even more serious for an application like video classification, where a human annotator has to watch an entire video end-to-end to furnish a label. Active learning algorithms automatically identify the most informative samples from large amounts of unlabeled data; this tremendously reduces the human annotation effort in inducing a machine learning model, as only the few samples that are identified by the algorithm, need to be labeled manually. In this paper, we propose a novel active learning framework for video classification, with the goal of further reducing the labeling onus on the human annotators. Our framework identifies a batch of exemplar videos, together with a set of informative frames for each video; the human annotator needs to merely review the frames and provide a label for each video. This involves much less manual work than watching the complete video to come up with a label. We formulate a criterion based on uncertainty and diversity to identify the informative videos and exploit representative sampling techniques to extract a set of exemplar frames from each video. To the best of our knowledge, this is the first research effort to develop an active learning framework for video classification, where the annotators need to inspect only a few frames to produce a label, rather than watching the end-to-end video. Our extensive empirical analyses corroborate the potential of our method to substantially reduce human annotation effort in applications like video classification, where annotating a single data instance can be extremely tedious.
FPGA Implementation of a Chaotic Boltzmann Machine Annealer

Ising machines are attracting attention for their ability to solve large-scale combinatorial optimization problems because these problems are difficult to solve. To accelerate the computing of Ising machines, implementation of Ising machines with digital circuits such as simulated annealing (SA) machines is in progress. However, these Ising machines on digital circuits require random number generators, which are implemented with large circuit resources. This work focuses on chaotic Boltzmann machines (CBMs), which imitate the stochastic behavior of Boltzmann machines (BMs) with deterministic chaotic dynamics. CBMs are one of the models that work as chaotic simulated annealing (CSA) machines within Ising machines. Therefore, we can implement the Ising machines without random number generators by using CBMs. In conventional work, CSA machines using CBMs (CBM-CSAs) are implemented with some hardware-oriented algorithms, but the CBM-CSA circuit is not optimized for these hardware-oriented algorithms. In the conventional CBM-CSA circuit, memory circuits are implemented separately, which prevents making the CBM-CSA from larger, and neuron circuits require the reset of accumulated values, which causes the increase in the calculation time. To solve these problems, we implement only one large memory circuit to make the CBM-CSA larger and improve the neuron circuits to allow dynamic changes of inputs to arithmetic circuits to inhibit the increase in the calculation time. As a result, we implement a CBM-CSA with 4096 nodes on an FPGA (Alveo U250), and the CBM-CSA can control 16-bit width weights and run at 100MHz. We evaluate the implemented CBM-CSA by solving K4000, max-cut problem, which is one of the combinatorial optimization problems. The best solution of CBM-CSA is comparable to that of the SA on the central processing unit (CPU). Moreover, the CBM-CSA is approximately 600 times as fast as the SA on the CPU and approximately twice as fast as the conventional Ising machine on an FPGA based on the improvements in this work. Furthermore, this work implements one of the highest-performance Ising machines on a single FPGA.
Feature Stylization Adversarial Domain Generalization

Although deep learning (DL) systems have achieved remarkable performance in many fields, the problem of severe performance slippage caused by domain shift still needs to be solved. To overcome this problem, domain generalization (DG) aims to learn a generalized model for arbitrary unseen domains leveraging data from multiple source domains. In this paper, we propose a novel DG approach, FSADG. FSADG consists of two components for DG: a domain discriminator and a set of feature style randomization modules. The feature style randomization modules aim to learn the latent distribution of image styles and generate diverse stylized features. Furthermore, adversarial training is conducted between the feature extractor of the target model and the domain discriminator for domain invariant representation learning. This paper also introduces applying FSADG for a more challenging task, DG with the data sparsity problem. We evaluate our method on PACS and OfficeHome datasets on image classification tasks. The experimental results demonstrate the effectiveness of FSADG.
BSDGAN: Balancing Sensor Data Generative Adversarial Networks for Human Activity Recognition

The development of IoT technology has enabled various sensors to be integrated into mobile devices. Human Activity Recognition (HAR) based on sensor data has become an important part of the field of machine learning and ubiquitous computing. However, human activities are not evenly distributed because of the rarely performed activities. The dataset is extremely imbalanced. In this paper, we propose Balancing Sensor Data Generative Adversarial Networks (BSDGAN) to generate sensor data for rarely performed human activities. To address extreme imbalances in human activity dataset, an autoencoder is employed to initialize the training process of BSDGAN, which to ensure the data features of each activity can be learned. Add the generated activity data to the original data to generate a new dataset that increases the proportion of rarely performed human activities, which makes the dataset balanced. We deployed multiple human activity recognition models on two publicly available imbalanced human activity datasets, WISDM and UNIMIB. Experimental results show that the proposed BSDGAN can effectively capture the data features of real human activity sensor data, and generate realistic synthetic sensor data. Meanwhile, the balanced activity dataset is effective for the activity recognition model to improve the recognition accuracy.
Schatten p-norm based Image-to-Video Adaptation for Video Action Recognition

Human action recognition has been receiving extensive interest among researchers from the computer vision community. Numerous successful action recognition techniques have demonstrated the effectiveness of learning action knowledge from still images or motion videos. The relevant action information learned for the same action via various media types, such as images or videos, may be correlated. Nonetheless, less attention has been paid to adapting the action knowledge from images to videos to enhance action recognition performance in videos. Furthermore, most existing video action recognition methods suffer from insufficient labeled training videos. Overfitting could be an issue in these circumstances; hence, action recognition performance can be inhibited. This paper proposes an adaptation framework to transfer knowledge from images to videos for action recognition. A multi-task learning framework is designed to optimize the image and video domain classifiers jointly. The general Schatten p-norm is applied to the classifiers to mine the shared knowledge between these two domains. In this way, our framework can learn the correlated action semantics by leveraging the shared components of labeled images and videos. Our proposed approach can fully use the action knowledge from images and performs better in the case of poor and limited video data compared with the existing state-of-the-art action recognition techniques.
Distance Functions and Normalization Under Stream Scenarios

Data normalization is an essential task when modeling a classification system. When dealing with data streams, data normalization becomes especially challenging since we may not know in advance the properties of the features, such as their minimum/maximum values, and these properties may change over time. We compare the accuracies generated by eight well-known distance functions in data streams without normalization, normalized considering the statistics of the first batch of data received, and considering the previous batch received. We argue that experimental protocols for streams that consider the full stream as normalized are unrealistic and can lead to biased and poor results. Our results indicate that using the original data stream without applying normalization, and the Canberra distance, can be a good combination when no information about the data stream is known beforehand.
Self-Supervised Adversarial Imitation Learning

Behavioural cloning is an imitation learning technique that teaches an agent how to behave via expert demonstrations. Recent approaches use self-supervision of fully-observable unlabelled snapshots of the states to decode state pairs into actions. However, the iterative learning scheme employed by these techniques is prone to get trapped into bad local minima. Previous work uses goal-aware strategies to solve this issue. However, this requires manual intervention to verify whether an agent has reached its goal. We address this limitation by incorporating a discriminator into the original framework, offering two key advantages and directly solving a learning problem previous work had. First, it disposes of the manual intervention requirement. Second, it helps in learning by guiding function approximation based on the state transition of the expert's trajectories. Third, the discriminator solves a learning issue commonly present in the policy model, which is to sometimes perform a ‘no action’ within the environment until the agent finally halts.
Enhanced Named Entity Recognition through Joint Dependency Parsing

Named entity recognition (NER) is the task of identifying and classifying named entities from texts. NER can benefit from linguistic dependency information, yet existing NER models can only utilize such information on datasets where dependency annotations are readily available. Dependency parsing (DP) models can be used to generate annotations, which are trained independent of the NER task and can cause error propagation to NER. In this paper, we propose a joint NER and DP model through multi-task learning, which allows the NER and DP modules to benefit from the joint training and provides an end-to-end solution to dependency-guided NER. Our model JOINDER uses a shared contextualized embedder, a word encoder, a biaffine dependency classifier, and a multi-hop dependency-guided NER. Experiments on several standard datasets in four languages show the effectiveness of joint learning and the outstanding performance of JOINDER compared to existing models. Moreover, our model can transfer dependency knowledge to other datasets with no dependency annotat.
Data Augmentation via Back-translation for Aspect Term Extraction

We tackle Aspect Term Extraction (ATE), a task that automatically recognizes aspect terms conditioned on the under-standing of word-level semantics. Due to the capacity of enriching linguistic phenomena for learning, data augmentation contributes to the establishment of robust ATE models. In this paper, we propose to leverage back translation to augment the training data for ATE. It is grounded on the potential advantages that the back-translated instances generally appear as paraphrases, providing diverse pragmatic modes for learning when semantics remains unchanged. This helps to enhance ATE models in recognizing aspect terms when varied contexts and morphologically-different words occur during test. In our experiments, we apply an off-the-shelf Neural Machine Translation (NMT) model for back translation, using French, Chinese and German as interlanguages, respectively. Besides, word alignment is conducted to designate aspect terms in the back-translated cases. Experimental results on SemEval benchmarks show that retraining with the augmented data produces substantial improvements, reaching up to 3.46% at best. In addition, the experiments suggest that 1) family languages are more beneficial than non-family for the aforementioned data augmentation, and 2) selective sampling produces positive effects in the low-resource settings, It is noteworthy that back translation has been explored for data augmentation in other fields, with the aim to enhance neural language modeling. Nevertheless, it hasn't yet been systematically studied towards the ATE task. Although a vest-pocket method is provided in this paper, the comprehensive analysis is conducted, including that on interlanguage selection, low-resource application, as well as compatibility with both conventional and pretrained neural models, besides that in the common comparison and ablation experiments. All the models and codes in the experiments will be made publicly available to support reproducible research.
HES: Edge Sampling for Heterogeneous Graphs

In light of the success of graph neural networks (GNNs), recent years have seen significant developments in modeling graphstructured data. Heterogeneous graphs have been widely adopted to model complex systems for various ML tasks. However, although some researchers have proposed methods for heterogeneous graphs, they merely focus on node features while neglecting the effectiveness of edge features. Besides, some research projects attempted incorporating edges into GNNs, but most regarded edge features as shared weights between node pairs. In this paper, we propose a two-stage method HES to learn the correlations among edge neighbors: (1) Graph Trans-formation: We convert the original heterogeneous graph into an undirected graph while preserving the orientation information, (2) Group Edge Sampling: To reduce the computation cost for the edge sampling in a heterogeneous graph, we propose to sample the most important edges over a group of edge neighbors instead of the whole graph, which leverages the edge features based on the one-hot encodings to describe the mutual influences between any adjacent edges. Finally, the experimental results on multiple public datasets show that HES outperforms existing state-of-the-art (SOTA) graph sampling methods. We further apply our approach to some existing GNN models as a pre-training process, demonstrating that HES can augment GNN-based models effectively.
Attack Assessment and Augmented Identity Recognition for Human Skeleton Data

Machine learning models trained on small data sets for security applications are especially vulnerable to adversarial attacks. Person identification from LiDAR based skeleton data requires time consuming and expensive data acquisition for each subject identity. Recently, Assessment and Augmented Identity Recognition for Skeletons (AAIRS) has been used to train Hierarchical Co-occurrence Networks for Person Identification (HCN-ID) with small LiDAR based skeleton data sets. However, AAIRS does not evaluate robustness of HCN-ID to adversarial attacks or inoculate the model to defend against such attacks. Popular perturbation-based approaches to generating adversarial attacks are constrained to targeted perturbations added to real training samples, which is not ideal for inoculating models with small training sets. Thus, we propose Attack-AAIRS, a novel addition to the AAIRS framework. Attack-AAIRS leverages a small real data set and a generative adversarial network (GAN) generated synthetic data set to assess and improve model robustness against unseen adversarial attacks. Rather than being constrained to perturbations of limited real training samples, the GAN learns the distribution of adversarial attack samples that exploit weaknesses in HCN-ID. Attack samples drawn from this distribution augment training for inoculation of the HCN-ID to improve robustness. Ten-fold cross validation of Attack-AAIRS yields noticeable increase in robustness to unseen attacks-including Fast Gradient Sign Method, Projected Gradient Descent, Additive Gaussian Noise, Momentum Iterative Fast Gradient Sign Method, and Basic Iterative Method. The HCN-ID Synthetic Data Quality Score for Attack-AAIRS indicates that generated attack samples are of similar quality to the original benign synthetic samples generated by AAIRS. Furthermore, inoculated models show consistent final test accuracy with the original model trained on real data, demonstrating that our method improves robustness to adversarial attacks without reducing testing performance on real data.
Building Decision Forest via Deep Reinforcement Learning

Ensemble learning methods whose base classifier is a decision tree usually belong to the bagging or boosting. It is widely used in all aspects of machine learning and has made great achievements in classification problems. However, no previous work has ever built the ensemble classifier by maximizing long-term returns to the best of our knowledge. This paper proposes a decision forest building method called MA-H-SAC-DF (Multi-agent Hybrid Soft Actor Critic based Decision Forest) for binary classification via deep reinforcement learning. First, the building process is modeled as a decentralized partial observable Markov decision process, and a set of cooperative agents jointly constructs all base classifiers. Second, the global state and local observations are defined based on information of the parent node and the current location. Last, the state-of-the-art deep reinforcement method Hybrid SAC (Hybrid Soft Actor Critic) with hybrid action space is extended to a multi-agent system under the CTDE (centralized training decentralized execution) architecture to find an optimal decision forest building policy. The experiments indicate that MA-H-SAC-DF has the same performance as random forest, Adaboost, and GBDT (Gradient Boosting Decision Tree) on balanced datasets and outperforms state-of-the-art ensemble learning algorithms on imbalanced datasets.
SAR: Self-Supervised Anti-Distortion Representation for End-To-End Speech Model

In recent Text-to-Speech (TTS) systems, a neural vocoder often generates speech samples by solely conditioning on acoustic features predicted from an acoustic model. However, there are always distortions existing in the predicted acoustic features, compared to those of the groundtruth, especially in the common case of poor acoustic modeling due to low-quality training data. To overcome such limits, we propose a Self-supervised learning framework to learn an Anti-distortion acoustic Representation (SAR) to replace human-crafted acoustic features by introducing distortion prior to an auto-encoder pre-training process. The learned acoustic representation from the proposed framework is proved anti-distortion compared to the most commonly used mel-spectrogram through both objective and subjective evaluation.
MEGA: Explaining Graph Neural Networks with Network Motifs

Graph Neural Networks (GNNs) are powerful tools for graph representation. However, GNNs have remained black boxes, leading to the lack of explainability. As a consequence, the application of GNNs has been severely limited. Existing methods focus on generating an explanation with important nodes and edges but pay less attention to high-order structures (e.g., network motif), which are more intuitive and important for graph data. The explanations generated by the explainer will thus ignore the high-order information contained in multi-node neighbours, resulting in a decrease in the human comprehensibility of the explanation subgraph. In this paper, we propose a motif-aware GNNs explainer (MEGA), wherein a motif-aware subgraph generation module and a counterfactual optimization layer are employed. MEGA can provide high-quality counterfactual explanations for GNNs while focusing on high-order features of graph data. We justify the effectiveness of the proposed MEGA on both synthetic and real-world datasets. Experimental results show that MEGA outperforms state-of-the-art baselines while keeping explanations at a smaller level.
Deep Multi - Resolution Network for Real- Time Semantic Segmentation in Street Scenes

Information at different resolutions plays distinct roles in computer vision tasks. Although the research on the utilization of different resolution information in semantic segmentation has made some achievements, the research on the utilization of different resolution information in real-time semantic segmentation needs to be improved. To address this issue, we propose Deep Multi-Resolution Network (DMRNet), a lightweight model using different resolution information for real-time semantic segmentation. This model consists of several branches with different resolutions, and information is fused between neighbouring branches after convolution operations. At the end of the lowest resolution branch, we designed an enhanced semantic information module, Amplify Aggregate Pyramid Pooling Module (AAPPM), to balance the extraction of semantic information with the speed of inference. In addition, at the end of all branches, we propose a multi-resolution fusion module (MRFM) to guide the information fusion in different branches, which helps to improve the problem of spatial details being covered by semantic information. On CityScapes and Camvid, the most widely-used datasets in the field of semantic segmentation, our method strikes a balance between network accuracy and inference speed. On a single 2080Ti GPU, DMRNet achieves 77.6 % and 74.7 % accuracy at inference speeds of 68.7 FPS and 91.6 FPS, respectively.
Novel Automatic Deep Learning Feature Extractor with Target Class Specific Feature Explanations

Deep-learning models are popular machine learning models that have gained their popularity in various fields of computer vision, natural language processing etc, due to their excellent predictive accuracy and ability to automatically extract good features from raw input data. Though these models have these significant advantages, most deep learning models are notoriously black-box models. The features learned by these models are not explainable. The need to explain these predictions is important in many high stakes fields. In this work, we propose a method that uses convolutional layers like in Convolutional Neural Networks (CNNs) that automatically learns features from raw data. In our proposed method we extract feature representation that is unique to each target class in the given data. Visualizations of this unique representation explain the feature learned by the model for a specific target class. To classify an input data we use two approaches. The first approach uses the similarity scores to match the features learned for a particular instance with the extracted target class-specific features to make prediction. The second approach uses an intrinsically explainable model like logistic regression to make predictions. Thus our proposed method in addition to having good predictive accuracy, identifies features that are class-specific. The proposed method achieved an accuracy of 99.31 % on MNIST data set, 90.45% on CIFAR-10 data set and 96.45% accuracy on Assira data set.