Reviewerly/exHarmony huggingface dataset:
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

We generalize deep self-attention distillation in MINILM (Wang et al., 2020) by only using self-attention relation distillation for taskagnostic compression of pretrained Transformers.In particular, we define multi-head selfattention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module.Then we employ the above relational knowledge to train the student model.Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student.Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer.In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MINILM.We conduct extensive experiments on compressing both monolingual and multilingual pre-trained models.Experimental results demonstrate that our models 1 distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting. Code, results, and videos are anonymously available at https://sites.google.com/view/sac-ae/home.
A comparison of satellite hyperspectral and multispectral remote sensing imagery for improved classification and mapping of vegetation

In recent years the use of remote sensing imagery to classify and map vegetation over different spatial scales has gained wide acceptance in the research community.Many national and regional datasets have been derived using remote sensing data.However, much of this research was undertaken using multispectral remote sensing datasets.With advances in remote sensing technologies, the use of hyperspectral sensors which produce data at a higher spectral resolution is being investigated.The aim of this study was to compare the classification of selected vegetation types using both hyperspectral and multispectral satellite remote sensing data.Several statistical classifiers including maximum likelihood, minimum distance, mahalanobis distance, spectral angular mapper and parallelepiped methods of classification were used.Classification using mahalanobis distance and maximum likelihood methods with an optimal set of hyperspectral and multispectral bands produced overall accuracies greater than 80%.
A Novel Hybrid Algorithm for Feature Selection Based on Whale Optimization Algorithm

Feature selection enhances classification accuracy by removing irrelevant and redundant feature.Feature selection plays an important role in data mining and pattern recognition.In this paper, we propose a hybrid feature subset selection algorithm called the maximum Pearson maximum distance improved whale optimization algorithm (MPMDIWOA).First, based on Pearson's correlation coefficient and correlation distance, a filter algorithm is proposed named maximum Pearson maximum distance (MPMD).Two parameters are proposed in MPMD to adjust the weights of the relevance and redundancy.Second, the modified whale optimization algorithm can act as a wrapper algorithm.After introducing the maximum value without change (MVWC) and the threshold, the filter algorithm and the wrapper algorithm are combined to form an algorithm called MPMDIWOA.In MPMDIWOA, the filter algorithm and wrapper algorithm are called different times according to changes in MVWC and threshold.Finally, the optimal classification accuracy was found.The proposed method is tested on 10 benchmark datasets from UCI machine learning databases.The experimental results show that the classification accuracy of the proposed algorithm is significantly higher than that of the other three wrapper algorithms and one hybrid algorithm.
Multi-Order Attentive Ranking Model for Sequential Recommendation

In modern e-commerce, the temporal order behind users’ transactions implies the importance of exploiting the transition dependency among items for better inferring what a user prefers to interact in “near future”. The types of interaction among items are usually divided into individual-level interaction that can stand out the transition order between a pair of items, or union-level relation between a set of items and single one. However, most of existing work only captures one of them from a single view, especially on modeling the individual-level interaction. In this paper, we propose a Multi-order Attentive Ranking Model (MARank) to unify both individual- and union-level item interaction into preference inference model from multiple views. The idea is to represent user’s short-term preference by embedding user himself and a set of present items into multi-order features from intermedia hidden status of a deep neural network. With the help of attention mechanism, we can obtain a unified embedding to keep the individual-level interactions with a linear combination of mapped items’ features. Then, we feed the aggregated embedding to a designed residual neural network to capture union-level interaction. Thorough experiments are conducted to show the features of MARank under various component settings. Furthermore experimental results on several public datasets show that MARank significantly outperforms the state-of-the-art baselines on different evaluation metrics. The source code can be found at https://github.com/voladorlu/MARank.
Deep Short Text Classification with Knowledge Powered Attention

Short text classification is one of important tasks in Natural Language Processing (NLP). Unlike paragraphs or documents, short texts are more ambiguous since they have not enough contextual information, which poses a great challenge for classification. In this paper, we retrieve knowledge from external knowledge source to enhance the semantic representation of short texts. We take conceptual information as a kind of knowledge and incorporate it into deep neural networks. For the purpose of measuring the importance of knowledge, we introduce attention mechanisms and propose deep Short Text Classification with Knowledge powered Attention (STCKA). We utilize Concept towards Short Text (CST) attention and Concept towards Concept Set (C-CS) attention to acquire the weight of concepts from two aspects. And we classify a short text with the help of conceptual information. Unlike traditional approaches, our model acts like a human being who has intrinsic ability to make decisions based on observation (i.e., training data for machines) and pays more attention to important knowledge. We also conduct extensive experiments on four public datasets for different tasks. The experimental results and case studies show that our model outperforms the state-of-the-art methods, justifying the effectiveness of knowledge powered attention.
XBully

Over the last decade, research has revealed the high prevalence of cyberbullying among youth and raised serious concerns in society. Information on the social media platforms where cyberbullying is most prevalent (e.g., Instagram, Facebook, Twitter) is inherently multi-modal, yet most existing work on cyberbullying identification has focused solely on building generic classification models that rely exclusively on text analysis of online social media sessions (e.g., posts). Despite their empirical success, these efforts ignore the multi-modal information manifested in social media data (e.g., image, video, user profile, time, and location), and thus fail to offer a comprehensive understanding of cyberbullying. Conventionally, when information from different modalities is presented together, it often reveals complementary insights about the application domain and facilitates better learning performance. In this paper, we study the novel problem of cyberbullying detection within a multi-modal context by exploiting social media data in a collaborative way. This task, however, is challenging due to the complex combination of both cross-modal correlations among various modalities and structural dependencies between different social media sessions, and the diverse attribute information of different modalities. To address these challenges, we propose XBully, a novel cyberbullying detection framework, that first reformulates multi-modal social media data as a heterogeneous network and then aims to learn node embedding representations upon it. Extensive experimental evaluations on real-world multi-modal social media datasets show that the XBully framework is superior to the state-of-the-art cyberbullying detection models.
Multi-Source Causal Feature Selection

Causal feature selection has attracted much attention in recent years, as the causal features selected imply the causal mechanism related to the class attribute, leading to more reliable prediction models built using them. Currently there is a need of developing multi-source feature selection methods, since in many applications data for studying the same problem has been collected from various sources, such as multiple gene expression datasets obtained from different experiments for studying the causes of the same disease. However, the state-of-the-art causal feature selection methods generally tackle a single dataset, and a direct application of the methods to multiple datasets will result in unreliable results as the datasets may have different distributions. To address the challenges, by utilizing the concept of causal invariance in causal inference, we first formulate the problem of causal feature selection with multiple datasets as a search problem for an invariant set across the datasets, then give the upper and lower bounds of the invariant set, and finally we propose a new Multi-source Causal Feature Selection algorithm, MCFS. Using synthetic and real world datasets and 16 feature selection methods, the extensive experiments have validated the effectiveness of MCFS.
Data sharing and how it can benefit your scientific career

Open science can lead to greater collaboration, increased confidence in findings and goodwill between researchers. Open science can lead to greater collaboration, increased confidence in findings and goodwill between researchers.
Operation and intrinsic error budget of a two-qubit cross-resonance gate

A relevant quantum gate for building a superconducting quantum computer is extensively analyzed. The limitations for achieving high gate fidelities due to several physical mechanisms are established, which may promote further experimental explorations.
T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion

Story completion is a very challenging task of generating the missing plot for an incomplete story, which requires not only understanding but also inference of the given contextual clues. In this paper, we present a novel conditional variational autoencoder based on Transformer for missing plot generation. Our model uses shared attention layers for encoder and decoder, which make the most of the contextual clues, and a latent variable for learning the distribution of coherent story plots. Through drawing samples from the learned distribution, diverse reasonable plots can be generated. Both automatic and manual evaluations show that our model generates better story plots than state-of-the-art models in terms of readability, diversity and coherence.
MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting

We study the task of image inpainting, where an image with missing region is recovered with plausible context. Recent approaches based on deep neural networks have exhibited potential for producing elegant detail and are able to take advantage of background information, which gives texture information about missing region in the image. These methods often perform pixel/patch level replacement on the deep feature maps of missing region and therefore enable the generated content to have similar texture as background region. However, this kind of replacement is a local strategy and often performs poorly when the background information is misleading. To this end, in this study, we propose to use a multi-scale image contextual attention learning (MUSICAL) strategy that helps to flexibly handle richer background information while avoid to misuse of it. However, such strategy may not promising in generating context of reasonable style. To address this issue, both of the style loss and the perceptual loss are introduced into the proposed method to achieve the style consistency of the generated image. Furthermore, we have also noticed that replacing some of the down sampling layers in the baseline network with the stride 1 dilated convolution layers is beneficial for producing sharper and fine-detailed results. Experiments on the Paris Street View, Places, and CelebA datasets indicate the superior performance of our approach compares to the state-of-the-arts.
Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model

The scarcity of labeled training data across many languages is a significant roadblock for multilingual neural language processing.We approach the lack of in-language training data using sentence embeddings that map text written in different languages, but with similar meanings, to nearby embedding space representations.The representations are produced using a dual-encoder based model trained to maximize the representational similarity between sentence pairs drawn from parallel data.The representations are enhanced using multitask training and unsupervised monolingual corpora.The effectiveness of our multilingual sentence embeddings are assessed on a comprehensive collection of monolingual, cross-lingual, and zeroshot/few-shot learning tasks.
Label-Specific Document Representation for Multi-Label Text Classification

Multi-label text classification (MLTC) aims to tag most relevant labels for the given document.In this paper, we propose a Label-Specific Attention Network (LSAN) to learn the new document representation.LSAN takes advantage of label semantic information to determine the semantic connection between labels and document for constructing labelspecific document representation.Meanwhile, the self-attention mechanism is adopted to identify the label-specific document representation from document content information.In order to seamlessly integrate the above two parts, an adaptive fusion strategy is designed, which can effectively output the comprehensive document representation to build multilabel text classifier.Extensive experimental results on four benchmark datasets demonstrate that LSAN consistently outperforms the stateof-the-art methods, especially on the prediction of low-frequency labels.The code and hyper-parameter settings are released to facilitate other researchers 1 .
Global Reasoning over Database Structures for Text-to-SQL Parsing

State-of-the-art semantic parsers rely on autoregressive decoding, emitting one symbol at a time.When tested against complex databases that are unobserved at training time (zeroshot), the parser often struggles to select the correct set of database constants in the new database, due to the local nature of decoding.In this work, we propose a semantic parser that globally reasons about the structure of the output query to make a more contextuallyinformed selection of database constants.We use message-passing through a graph neural network to softly select a subset of database constants for the output query, conditioned on the question.Moreover, we train a model to rank queries based on the global alignment of database constants to question words.We apply our techniques to the current state-ofthe-art model for SPIDER, a zero-shot semantic parsing dataset with complex databases, increasing accuracy from 39.4% to 47.4%.
Identification of Encrypted Traffic Through Attention Mechanism Based Long Short Term Memory

Network traffic classification has become an important part of network management, which is beneficial for achieving intelligent network operation and maintenance, enhancing the network quality of service (QoS), and for network security. Given the rapid development of various applications and protocols, more and more encrypted traffic has emerged in networks. Traditional traffic classification methods exhibited the unsatisfied performance since the encrypted traffic is no longer in plain text. In this work, we modeled the time-series network traffic by the recurrent neural network (RNN). Moreover, the attention mechanism was introduced for assisting network traffic classification in the form of the following two models, the attention aided long short term memory (LSTM) as well as the hierarchical attention network (HAN). Finally, relying on the ISCX VPN-NonVPN dataset, extensive experiments were conducted, showing that the proposed methods achieved 91.2 percent in accuracy while the highest accuracy of other methods was 89.8 percent relying on the same dataset.
Language2Pose: Natural Language Grounded Pose Forecasting

Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations. In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language-to-Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-to-end using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and human-annotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.
On the use of BERT for Neural Machine Translation

Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently.In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation.We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality.We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments.In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness.
Probing Natural Language Inference Models through Semantic Fragments

Do state-of-the-art models for language understanding already have, or can they easily learn, abilities such as boolean coordination, quantification, conditionals, comparatives, and monotonicity reasoning (i.e., reasoning about word substitutions in sentential contexts)? While such phenomena are involved in natural language inference (NLI) and go beyond basic linguistic understanding, it is unclear the extent to which they are captured in existing NLI benchmarks and effectively learned by models. To investigate this, we propose the use of semantic fragments—systematically generated datasets that each target a different semantic phenomenon—for probing, and efficiently improving, such capabilities of linguistic models. This approach to creating challenge datasets allows direct control over the semantic diversity and complexity of the targeted linguistic phenomena, and results in a more precise characterization of a model's linguistic behavior. Our experiments, using a library of 8 such semantic fragments, reveal two remarkable findings: (a) State-of-the-art models, including BERT, that are pre-trained on existing NLI benchmark datasets perform poorly on these new fragments, even though the phenomena probed here are central to the NLI task; (b) On the other hand, with only a few minutes of additional fine-tuning—with a carefully selected learning rate and a novel variation of “inoculation”—a BERT-based model can master all of these logic and monotonicity fragments while retaining its performance on established NLI benchmarks.
Spatial Attention Fusion for Obstacle Detection Using MmWave Radar and Vision Sensor

For autonomous driving, it is important to detect obstacles in all scales accurately for safety consideration. In this paper, we propose a new spatial attention fusion (SAF) method for obstacle detection using mmWave radar and vision sensor, where the sparsity of radar points are considered in the proposed SAF. The proposed fusion method can be embedded in the feature-extraction stage, which leverages the features of mmWave radar and vision sensor effectively. Based on the SAF, an attention weight matrix is generated to fuse the vision features, which is different from the concatenation fusion and element-wise add fusion. Moreover, the proposed SAF can be trained by an end-to-end manner incorporated with the recent deep learning object detection framework. In addition, we build a generation model, which converts radar points to radar images for neural network training. Numerical results suggest that the newly developed fusion method achieves superior performance in public benchmarking. In addition, the source code will be released in the GitHub.
A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods

Real-time sensing and modeling of the human body, especially the hands, is an important research endeavor for various applicative purposes such as in natural human computer interactions. Hand pose estimation is a big academic and technical challenge due to the complex structure and dexterous movement of human hands. Boosted by advancements from both hardware and artificial intelligence, various prototypes of data gloves and computer-vision-based methods have been proposed for accurate and rapid hand pose estimation in recent years. However, existing reviews either focused on data gloves or on vision methods or were even based on a particular type of camera, such as the depth camera. The purpose of this survey is to conduct a comprehensive and timely review of recent research advances in sensor-based hand pose estimation, including wearable and vision-based solutions. Hand kinematic models are firstly discussed. An in-depth review is conducted on data gloves and vision-based sensor systems with corresponding modeling methods. Particularly, this review also discusses deep-learning-based methods, which are very promising in hand pose estimation. Moreover, the advantages and drawbacks of the current hand gesture estimation methods, the applicative scope, and related challenges are also discussed.