Reviewerly/exHarmony huggingface dataset:
Real-time classification of handball game situations

During the broadcast of sporting events, certain situations such as a penalty or a time-out occur, for which a specific action is required. In traditional broadcasting, many people are implied in making decisions based on what is happening at any given moment. To broadcast quality and entirely automatically matches it is necessary to be able to classify the important situations and then make decisions based on them. This paper presents a solution based on deep learning which is able to classify the main states of a handball match. The generated model has been trained using 127,600 images of 13 local team matches. On a test set of 118,129 images of other 7 matches, it is able to classify these situations with an accuracy of 98.6% in only 4 milliseconds, allowing to analyze the state of the game in real time. The full pipeline takes only 34.04 milliseconds using GPU acceleration, processing more than 25 frames per seconds.
A Stable Spatial Pooling Algorithm for Hierarchical Temporal Memory Network

Hierarchical Temporal Memory (HTM) is an emerging neural network technology which is inspired by the structure and the working principles of human neocortex. HTM uses a set of active columns to represent the input and inhibits frequently activated columns by a boosting mechanism. However, the boosting mechanism may cause an instable representation and thus affects the HTM performance. To address this issue, we propose a stable spatial pooling algorithm for HTM, where the column loadability is introduced, and a new column activation strategy based on the loadability is presented. We implement the proposed algorithm and carry out various experiments on both the synthetic and real-world sequential datasets. The results show that our HTM can generate more stable internal representation for the input data. The prediction accuracy of the proposed HTM outperforms that of the conventional HTM and the popular LSTM networks.
A Cross-Modal Object-Aware Transformer for Vision-and-Language Navigation

Vision-and-language navigation (VLN) combines cross-modal object references and scene descriptions to provide a breadcrumb trail to a goal location. Whereas existing VLN approaches often do not take full advantage of cross-modal object information, this work proposes a transformer network with perceptual cross-modal object data that fuses and aligns the two cue features of object reference to help agents capture object features. In our method, linguistic object processing provides semantic-level contextual information for visual object features. With this design, our model is able to leverage object features to assist the agent in substantially improving performance on the R2R and R4R benchmarks. Through extensive experiments on R2R and R4R, we demonstrate the effectiveness of the proposed model, and our method improves the absolute 1.6% in SPL on R2R and 2.1% in CLS on R4R. Our analysis shows that the network performs better when focusing on longer heavily object-referenced navigation instructions, which also indicates that our approach is better able to use object features and align them to references in the instructions.
Block-term Dropout For Robust Adversarial Defense

Deep neural networks (DNNs) have lately shown tremendous performance in various applications. However, along-side their superiority in these tasks, recent studies have demon-strated that DNNs are easily fooled by adversarial attacks. To guard against adversarial examples, we provide a new solution to hardening DNNs through Block-term Dropout (BT-Dropout), an adversarial defense technique that leverages a latent high-order factorization of the network. Specifically, we impose low-rank block-term tensor structure on the weights of fully-connected layer to obtain compact networks, and then apply BT-Dropout in the latent subspace without pruning the weights directly. Meanwhile, for activation tensor fed into fully-connected layer, Tucker Dropout which can be viewed as a special case of BT-Dropout is introduced to preserve multilinear structure of activations. Furthermore, we show that BT-Dropout implicitly regularizes the tensor decomposition. Comprehensive experiments have demonstrated the effectiveness of our proposed method to improve the adversarial robustness for the models on standard image classification benchmarks.
Transfer Learning for Regression through Adaptive Gaussian Process

Extracting knowledge from closely-related domains to solve a new problem has become an advanced methodology in machine learning, and is called transfer learning. Conspicuous among existing regression methods are those built around Gaussian process (GP) because they consider variance in the predicted values, although they also require that the similarity between the source domain and the target domain needs to fall within a certain range. To overcome this limitation in GP methods and improve transfer learning performance, this study proposes an adaptive Gaussian process (AGP) for regression. The AGP method broadens the range of acceptable similarity in current GP methods by developing a new transfer kernel. The results of experiments with transfer regression problems on both synthetic and real-world datasets indicate that this AGP method signiticantly improves prediction accuracy.
Concession-First Learning and Coarse-to-Fine Retrieval for Open-Domain Conversational Question Answering

We tackle Open-Domain Conversational Question Answering (abbr., ODCQA), a task of answering questions in multi-turn conversations by mining clues from a large passage collection. Recent progress in deep learning and large-scale Pre-trained Language Model (PLM) is driving fast-paced advances in ODCQA. However, the typical retriever-reranker-reader (3R) pipeline framework of the existing ODCQA systems suffers from a potential flaw, which is referred to the unavoidable interference from the reader to reranker. Briefly, there are a large number of shareable parameters between the reader and reranker because they share the same PLM-based encoder. During the gradient backpropagation, their directions of updating PLM parameters are most probably inconsistent, which leads to interference in seeking for the optimal solution. In addition, the recent retriever in 3R framework merely utilizes dense representation. Though, dense retrievers are generally weaker than sparse retrievers in lexical matching for rare entities. To address the aforementioned two issues, we first propose to utilize a weak reader to alleviate the interference, and explore three different methods to pursue the goal, including (1) masking gradient, (2) dropping inputs and (3) early stopping. Besides, we propose to hybridize a semantic-sensitive dense retriever and a keyword -sensitive sparse retriever, so as to enhance the robustness of retriever in dealing with rare entities. We conduct extensive experiments on the benchmark corpus OR-QUAC. Experimental results show that our approach outperforms the State-of-The-Art (SoTA) models, yeilding an improvement of 6.1 % F1-score.
Exploring Heterogeneous Feature Representation for Document Layout Understanding

There are increasing interests in document layout representation learning and understanding. Transformer, with its great power, has become the mainstream model architecture and achieved promising results in this area. As elements in a document layout consist of multi-modal and multi-dimensional features such as position, size, and its text content, prior works represent each element by summing all feature embeddings into one unified vector in the input layer, which is then fed into the self-attention for element-wise interaction. However, this simple summation would potentially raise mixed correlations among heterogeneous features and bring noise to the representation learning. In this paper, we propose a novel two-step disentangled attention mechanism to allow more flexible feature interactions in the self-attention. Furthermore, inspired by the principles of document design (e.g., contrast, proximity), we propose an unsupervised learning objective to constrain the layout representations. We verify our approach on two layout understanding tasks, namely element role labeling and image captioning. Experiment results show that our approach achieves state-of-the-art performances. Moreover, we conduct extensive studies and observe better interpretability using our approach.
Multi-Instance Active Learning for Relation Extraction

Achieving high-accuracy relation extraction usually requires a large well-annotated corpus for model training, which is extremely time-consuming and labor-intensive. Consequently, distant supervision, implemented as a multi-instance learning paradigm, was proposed to build models from bags using automated labeled corpora, where massive noises are unavoidably introduced in the bag labels, deteriorating the model performance. To deal with the above problem, this paper proposes novel multi-instance active learning for relation extraction, which aims to improve the classification performance of the learned models under the premise of using as few labeled training samples as possible. The proposed active learning strategies for bag selection are based on the similarity between bags, the improved Fisher matrix, and the combination of the two (hybrid), respectively. Extensive experiment results on two real-world datasets under three base learners consistently show that our proposed active learning methods archive better performance for relation extraction, compared with three baseline methods. Furthermore, the hybrid bag selection strategy performs the best.
Reinforcement Learning Based Incentive Mechanism for Federated Meta Learning: A Game-Theoretic Perspective

Federated learning (FL) is a novel decentralized machine learning mechanism, which bridges data silos to train a global model by utilizing data and computation power of local clients in a privacy-preserving way. Moreover, to handle heterogeneous data across domains, tasks and partities, federated meta-learning (FML) has been proposed, which leverages the fast adaptation of meta-learning for transferable and customizable models. Nevertheless, most of the existing studies focus on providing personalized models for different users, leaving other important issues not well solved, especially, incentive mechanisms, which are used as a common foundation for FML to attrack and maintain high-quality and reputable clients. To fill the gap, this paper proposes a learning-based incentive mechanism for FML to motivate local clients to participate in the data federation. First, we propose to reward clients according to the amount of data they contribute to the model training. Then, to analyze the behaviors of model owner and local clients, we formulate the incentivized training task as a Stackelberg game, and design a method based on reinforcement learning (RL) to learn optimal pricing and participating strategies for the task publisher and the local clients, respectively. Lastly, extensive experiments are conducted to demonstrate the efficiency and effectiveness of the proposed RL-based incentive mechanism, which assists multiple parties to reach the equilibrium of the game designed for FML.
Federated Grouping Panoptic Segmentation for LiDAR Point Cloud Based on Convolutional Network

Panoptic segmentation of LiDAR point cloud in autonomous driving as a complete perception task has large data demand and does not facilitate data sharing when seeking collaboration. As a distributed collaborative learning method, Federated Learning (FL) is playing an increasingly important role in several fields. This paper is the first attempt to apply FL to LiDAR panoptic segmentation task. We propose a novel Panoptic Convolutional Fusion Network (PCF-Net), which outputs instance-level panoptic predictions directly through a learnable convolutional module. Additionally, we design the Dynamic Grouping Federated Learning (DGFL) algorithm, which transfers parameters based on grouping before global average. It maximizes the sharing times of model parameters while ensuring communication efficiency. Combining these two approaches, we propose a DGFL-based LiDAR panoptic segmentation framework. To demonstrate the effectiveness and generalizability of the framework, we reorganize and split two open-source datasets, namely SemanticKITTI and nuScenes. We conduct experiments on them under various conditions, and our method archives state-of-the-art performance. Our work provides new insights into the application of FL in autonomous driving.
Domain-adaptive Graph based on Post-hoc Explanation for Cross-domain Hate Speech Detection

Hate speech detection is hampered by the scarcity and topical and lexical biases of annotated data, leading to poor generalization. It is imperative to devise a cross-domain approach to solve this problem. The ability to learn transferable knowledge is critical for cross-domain hate speech detection. In this work, We propose a domain-adaptive dependency graph method based on post-hoc explanation (DPDG). We extract post-hoc explanations from fine-tuned BERT classifiers as the importance score for hate representation. Based on these, we construct in-domain graph and cross-domain graph to better learn in-domain hate representation and adapt to the target domain respectively. Finally, we use interactive GCN blocks to interactively and adaptively learn and adjust the domain adaptive graph representation. The results of cross-domain experiments on multiple domains show that our proposed model outperforms competitive baselines in cross-domain hate speech detection.
EMAS: Efficient Meta Architecture Search for Few-Shot Learning

With the progress of few-shot learning, it has been scaled to many domains in the real world which have few labeled data, such as image classification and object detection. Many efforts for data embedding and feature combination have been made by designing a fixed neural architecture that can also be extended to variable and adaptive neural architectures for better performance. Recent works leverage neural architecture search technique to automatically design networks for few-shot learning but it requires vast computation costs and GPU memory requirements. This work introduces EMAS, an efficient method to speed up the searching process for few-shot learning. Specifically, we build a supernet to combine all candidate operations and then adopt gradient-based methods to search. Instead of training the whole supernet, we adopt Gumbel reparameterization technique to sample and activate a small subset of operations. EMAS handles a single path in a novel task adapted with just a few steps and time. A novel task only needs to learn fewer parameters and compute less content. During meta-testing, the task can well adapt to the network architecture although only with a few iterations. Empirical results show that EMAS yields a fair improvement in accuracy on the standard few-shot classification benchmark and is five times smaller in time.
Expand-Extract: A Parallel Corpus Mining Framework from Comparable Corpora for English-Myanmar Machine Translation

High-quality neural machine translation (NMT) systems rely on the availability of large-scale and reliable parallel data. Since Myanmar language is a low-resource language, the parallel corpus of English-Myanmar language pair is sparse in volume. In this paper, we present a simple yet effective framework to create a parallel corpus from the available comparable corpora. Our proposed system first uses self-training and back-translation approaches together with the denoising-based automatic post-editing (DbAPE) system for augmenting synthetic datasets that are used to expand the size of existing comparable corpora. Then, LaBSE-based sentence embeddings and the proposed scoring function are applied to extract parallel sentences from the expanded comparable corpora. The extracted parallel sentences can be used to supplement parallel corpus when training the low-resource English-Myanmar NMT systems. We investigate the effectiveness of our methods by evaluating the NMT systems trained on the concatenation of parallel data created by our framework and an existing dataset. We show that the proposed framework is capable of creating a reliable parallel corpus, and that the created corpus substantially increases translation quality of MT systems trained on the existing parallel data, as measured by automatic evaluation metrics.
Pyramid Transformer: A Multi-size Object Detection Model with Limited Device Requirements for the Nursing Robot

Multi-size object detection is a technical difficulty which impeding the development of the intelligent nursing robot. To cope with the problem, this paper proposes a Pyramid Transformer model to detect the objects with different sizes in nursing scenario. Pyramid Transformer consists of three parts including Transformer Module, Pyramid Structure and Convolution Module. Transformer Module can improve the performance of large object detection with Multi-head Attention mechanism, and Pyramid Structure enables the model to make prediction with feature maps of different sizes which benefits the detection of small objects. Convolution Module is employed to reduce hardware requirements, and it makes Pyramid Transformer could run and implement on a single graphics card. The experiments show that the mean average precision reaches 72.7% which makes improvement over other models. This shows that the proposed Pyramid Transformer model is practical and effective for object detection of the nursing robot. The dataset can be got at https://github.com/NotFar1997/NSI-dataset.
Enhancing Neural Aspect Term Extraction Using Part-Of-Speech and Syntactic Dependency Features

We tackle Aspect Term Extraction (ATE), an important natural language processing task that automatically identifies words of domain-specific aspects. Recently, a variety of sophisticated neural models and learning strategies have been explored for enhancing ATE, and significant improvements have been obtained. We intend to strengthen the neural models by involving features of Part-Of-Speech (POS) and syntactic dependency into the encoding process. It is motivated by the empirical findings that features of linguistic structure help to refine the understanding of semantics. Accordingly, we propose a stepwise encoding approach, where POS and syntactic dependency are successively leveraged step by step, including 1) joint encoding over both word and POS sequences using a pretrained language model; 2) BiGRU-based representational refinement conditioned on semantics-aware POS information and POS-aware semantic information; 3) representational augmentation by convolutional encoding of dependency graph. We conduct experiments on the four benchmark datasets of Semantic Evaluation (SemEval) for ATE. Experimental results show that our method obtains substantial improvements on all the considered datasets, and the performance ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$F1$</tex> -score) reaches 87.69%, 89.76%, 77.94% and 83.96% for L-14, R-14, R-15 and R-16, respectively. All the models and source codes in the experiments will be made publicly available to support reproducible research.
RGB-D Saliency Detection with 3D Cross-modal Fusion and Mid-level Integration

In recent years, many salient object detection (SOD) methods introduce depth cues to boost detection performance in challenging scenes, named as RGB-D SOD. However, how to effectively fuse cross-modal features with various properties (i.e., RGB and depth) has become a key issue that is hard to be avoided. Most existing methods employ simple operations, such as concatenation or summation, for cross-modal fusion, ignoring the negative effects of low-quality depth maps, thus yielding poor performance. In this paper, we design a simple yet effective fusion method, which utilizes 3D convolution to extract modality-specific and modality-shared information respectively for sufficient cross-modal fusion, and combines modality weights to mitigate the interference of invalid information. In addition, we propose a novel multi-level feature integration strategy in the decoder, which explicitly incorporates the low-level detail information and high-level semantic information into the mid-level to generate accurate saliency maps. Extensive experiments on six public datasets show that our method achieves competitive results compared to 17 state-of-the-art methods.
Mining Influential Training Data by Tracing Influence on Hard Validation Samples

The ever-growing deep learning model size is constantly driven by the ever-growing dataset size. Mining the influential training data has significant payoff of either reducing the training time, model complexity as well as potentially increasing the model accuracy. In this paper, we propose a few approaches, e.g. classifying the validation dataset into easy, medium and hard levels, introducing influence value by calculating each training data on the hard validation data, to co-prune the validation dataset and the training dataset. Empirically we conclude that the portion of the hard validation data could be used to mine the most influential training data, whereby reducing the training dataset size by 50% without losing accuracy in our experiments.
Extending a Refinement Acting Engine for Fleet Management: Concurrency and Resources

Recent years have seen an important increase in the complexity of deployed robotic systems, both in terms of the number of robots involved, and scale of the tackled problems. The key challenge in this context is to allow the design of fleet control systems that, on the one hand, allow flexible and reactive operation of individual robots and, on the other hand, enable the system to optimize the global behavior of the fleet in order to increase its effectiveness and efficiency. To approach this problem, we propose to extend the Refinement Acting Engine (RAE) that has been used to program the behavior of autonomous agents through a hierarchical decomposition of high-level tasks into primitive commands, and is the subject of active research in order to guide its decisions with planning and scheduling techniques. The core of our proposal is to provide first-hand support for concurrency in the RAE procedure, allowing a natural representation for concurrent systems by reasoning on resource allocation. The resulting acting engine exploits a custom language that is designed to ease its integration with planning engines, both through its simple and orthogonal core constructs as well as in the explicit identification of decision points in the system operation. We provide an initial validation of the system in simulation on a logistic problem involving a fleet of robots.
Towards Query-limited Adversarial Attacks on Graph Neural Networks

Graph Neural Network (GNN) is a graph representation learning approach for graph-structured data, which has witnessed a remarkable progress in the past few years. As a counterpart, the robustness of such a model has also received considerable attention. Previous studies show that the performance of a well-trained GNN can be faded by black-box adversarial examples significantly. In practice, the attacker can only query the target model with very limited counts, yet the existing methods require hundreds of thousand queries to extend attacks, leading the attacker to be exposed easily. To perform a step forward in addressing this issue, in this paper, we propose a novel attack methods, namely Graph Query-limited Attack (GQA), in which we generate adversarial examples on the surrogate model to fool the target model. Specifically, in GQA, we use contrastive learning to fit the feature extraction layers of the surrogate model in a query-free manner, which can reduce the need of queries. Furthermore, in order to utilize query results sufficiently, we obtain a series of queries with rich information by changing the input iteratively, and storing them in a buffer for recycling usage. Experiments show that GQA can decrease the accuracy of the target model by 4.8%, with only 1% edges modified and 100 queries performed.