OpenRaiser/Intern-Atlas huggingface dataset:
Sequence-to-Sequence Learning for Indonesian Automatic Question Generator
Venue: arXiv cs.CL 2020
Year: 2020

Automatic question generation is defined as the task of automating the creation of question given a various of textual data. Research in automatic question generator (AQG) has been conducted for more than 10 years, mainly focused on factoid question. In all these studies, the state-of-the-art is attained using sequence-to-sequence approach. However, AQG system for Indonesian has not ever been researched intensely. In this work we construct an Indonesian automatic question generator, adapting the architecture from some previous works. In summary, we used sequence-to-sequence approach using BiGRU, BiLSTM, and Transformer with additional linguistic features, copy mechanism, and coverage mechanism. Since there is no public large dan popular Indonesian dataset for question generation, we translated SQuAD v2.0 factoid question answering dataset, with additional Indonesian TyDiQA dev set for testing. The system achieved BLEU1, BLEU2, BLEU3, BLEU4, and ROUGE-L score at 38,35, 20,96, 10,68, 5,78, and 43,4 for SQuAD, and 39.9, 20.78, 10.26, 6.31, 44.13 for TyDiQA, respectively. The system performed well when the expected answers are named entities and are syntactically close with the context explaining them. Additionally, from native Indonesian perspective, the best questions generated by our best models on their best cases are acceptable and reasonably useful.
Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
Venue: arXiv cs.LG 2020
Year: 2020

Context, the embedding of previous collected trajectories, is a powerful construct for Meta-Reinforcement Learning (Meta-RL) algorithms. By conditioning on an effective context, Meta-RL policies can easily generalize to new tasks within a few adaptation steps. We argue that improving the quality of context involves answering two questions: 1. How to train a compact and sufficient encoder that can embed the task-specific information contained in prior trajectories? 2. How to collect informative trajectories of which the corresponding context reflects the specification of tasks? To this end, we propose a novel Meta-RL framework called CCM (Contrastive learning augmented Context-based Meta-RL). We first focus on the contrastive nature behind different tasks and leverage it to train a compact and sufficient context encoder. Further, we train a separate exploration policy and theoretically derive a new information-gain-based objective which aims to collect informative trajectories in a few steps. Empirically, we evaluate our approaches on common benchmarks as well as several complex sparse-reward environments. The experimental results show that CCM outperforms state-of-the-art algorithms by addressing previously mentioned problems respectively.
Utterance-level Dialogue Understanding: An Empirical Study
Venue: arXiv cs.CL 2020
Year: 2020

The recent abundance of conversational data on the Web and elsewhere calls for effective NLP systems for dialog understanding. Complete utterance-level understanding often requires context understanding, defined by nearby utterances. In recent years, a number of approaches have been proposed for various utterance-level dialogue understanding tasks. Most of these approaches account for the context for effective understanding. In this paper, we explore and quantify the role of context for different aspects of a dialogue, namely emotion, intent, and dialogue act identification, using state-of-the-art dialog understanding methods as baselines. Specifically, we employ various perturbations to distort the context of a given utterance and study its impact on the different tasks and baselines. This provides us with insights into the fundamental contextual controlling factors of different aspects of a dialogue. Such insights can inspire more effective dialogue understanding models, and provide support for future text generation approaches. The implementation pertaining to this work is available at https://github.com/declare-lab/dialogue-understanding.
Performance Modeling of Streaming Kernels and Sparse Matrix-Vector Multiplication on A64FX
Venue: arXiv cs.PF 2020
Year: 2020

The A64FX CPU powers the current number one supercomputer on the Top500 list. Although it is a traditional cache-based multicore processor, its peak performance and memory bandwidth rival accelerator devices. Generating efficient code for such a new architecture requires a good understanding of its performance features. Using these features, we construct the Execution-Cache-Memory (ECM) performance model for the A64FX processor in the FX700 supercomputer and validate it using streaming loops. We also identify architectural peculiarities and derive optimization hints. Applying the ECM model to sparse matrix-vector multiplication (SpMV), we motivate why the CRS matrix storage format is inappropriate and how the SELL-C-sigma format with suitable code optimizations can achieve bandwidth saturation for SpMV.
Towards Intelligent Risk-based Customer Segmentation in Banking
Venue: arXiv cs.CY 2020
Year: 2020

Business Processes, i.e., a set of coordinated tasks and activities to achieve a business goal, and their continuous improvements are key to the operation of any organization. In banking, business processes are increasingly dynamic as various technologies have made dynamic processes more prevalent. For example, customer segmentation, i.e., the process of grouping related customers based on common activities and behaviors, could be a data-driven and knowledge-intensive process. In this paper, we present an intelligent data-driven pipeline composed of a set of processing elements to move customers' data from one system to another, transforming the data into the contextualized data and knowledge along the way. The goal is to present a novel intelligent customer segmentation process which automates the feature engineering, i.e., the process of using (banking) domain knowledge to extract features from raw data via data mining techniques, in the banking domain. We adopt a typical scenario for analyzing customer transaction records, to highlight how the presented approach can significantly improve the quality of risk-based customer segmentation in the absence of feature engineering.As result, our proposed method is able to achieve accuracy of 91% compared to classical approaches in terms of detecting, identifying and classifying transaction to the right classification.
Tackling unsupervised multi-source domain adaptation with optimism and consistency
Venue: arXiv cs.LG 2020
Year: 2020

It has been known for a while that the problem of multi-source domain adaptation can be regarded as a single source domain adaptation task where the source domain corresponds to a mixture of the original source domains. Nonetheless, how to adjust the mixture distribution weights remains an open question. Moreover, most existing work on this topic focuses only on minimizing the error on the source domains and achieving domain-invariant representations, which is insufficient to ensure low error on the target domain. In this work, we present a novel framework that addresses both problems and beats the current state of the art by using a mildly optimistic objective function and consistency regularization on the target samples.
A Prototype-Based Generalized Zero-Shot Learning Framework for Hand Gesture Recognition
Venue: arXiv cs.CV 2020
Year: 2020

Hand gesture recognition plays a significant role in human-computer interaction for understanding various human gestures and their intent. However, most prior works can only recognize gestures of limited labeled classes and fail to adapt to new categories. The task of Generalized Zero-Shot Learning (GZSL) for hand gesture recognition aims to address the above issue by leveraging semantic representations and detecting both seen and unseen class samples. In this paper, we propose an end-to-end prototype-based GZSL framework for hand gesture recognition which consists of two branches. The first branch is a prototype-based detector that learns gesture representations and determines whether an input sample belongs to a seen or unseen category. The second branch is a zero-shot label predictor which takes the features of unseen classes as input and outputs predictions through a learned mapping mechanism between the feature and the semantic space. We further establish a hand gesture dataset that specifically targets this GZSL task, and comprehensive experiments on this dataset demonstrate the effectiveness of our proposed approach on recognizing both seen and unseen gestures.
Wireless Powered Cooperative Relaying Systems with Non-orthogonal Multiple Access
Venue: arXiv cs.IT 2020
Year: 2020

Non-orthogonal multiple access (NOMA) and the cooperative relaying systems are two of the promising techniques to meet requirements of future wireless networks such as high spectral efficiency and wide coverage area. On the other hand, the energy efficiency has also high priority in the applications with limited energy such as sensor networks and/or internet of things (IoTs). To this end, in this paper, we propose wireless powered cooperative relaying system with NOMA thereby increasing spectral and energy efficiency. We consider three different energy harvesting (EH) protocols (i.e., power sharing (PS), time sharing (TS) and ideal) and for all three EH protocols, we derive achievable rate for the considered system model. We validate the analysis with computer simulations and present the effectiveness of wireless powered system compared to the benchmark.
Selective Cascade of Residual ExtraTrees
Venue: arXiv cs.LG 2020
Year: 2020

We propose a novel tree-based ensemble method named Selective Cascade of Residual ExtraTrees (SCORE). SCORE draws inspiration from representation learning, incorporates regularized regression with variable selection features, and utilizes boosting to improve prediction and reduce generalization errors. We also develop a variable importance measure to increase the explainability of SCORE. Our computer experiments show that SCORE provides comparable or superior performance in prediction against ExtraTrees, random forest, gradient boosting machine, and neural networks; and the proposed variable importance measure for SCORE is comparable to studied benchmark methods. Finally, the predictive performance of SCORE remains stable across hyper-parameter values, suggesting potential robustness to hyperparameter specification.
A Survey on Deep Learning Techniques for Video Anomaly Detection
Venue: arXiv cs.CV 2020
Year: 2020

Anomaly detection in videos is a problem that has been studied for more than a decade. This area has piqued the interest of researchers due to its wide applicability. Because of this, there has been a wide array of approaches that have been proposed throughout the years and these approaches range from statistical-based approaches to machine learning-based approaches. Numerous surveys have already been conducted on this area but this paper focuses on providing an overview on the recent advances in the field of anomaly detection using Deep Learning. Deep Learning has been applied successfully in many fields of artificial intelligence such as computer vision, natural language processing and more. This survey, however, focuses on how Deep Learning has improved and provided more insights to the area of video anomaly detection. This paper provides a categorization of the different Deep Learning approaches with respect to their objectives. Additionally, it also discusses the commonly used datasets along with the common evaluation metrics. Afterwards, a discussion synthesizing all of the recent approaches is made to provide direction and possible areas for future research.
Modeling and Testing Multi-Agent Traffic Rules within Interactive Behavior Planning
Venue: arXiv cs.RO 2020
Year: 2020

Autonomous vehicles need to abide by the same rules that humans follow. Some of these traffic rules may depend on multiple agents or time. Especially in situations with traffic participants that interact densely, the interactions with other agents need to be accounted for during planning. To study how multi-agent and time-dependent traffic rules shall be modeled, a framework is needed that restricts the behavior to rule-conformant actions during planning, and that can eventually evaluate the satisfaction of these rules. This work presents a method to model the conformance to traffic rules for interactive behavior planning and to test the ramifications of the traffic rule formulations on metrics such as collision, progress, or rule violations. The interactive behavior planning problem is formulated as a dynamic game and solved using Monte Carlo Tree Search, for which we contribute a new method to integrate history-dependent traffic rules into a decision tree. To study the effect of the rules, we treat it as a multi-objective problem and apply a relaxed lexicographical ordering to the vectorized rewards. We demonstrate our approach in a merging scenario. We evaluate the effect of modeling and combining traffic rules to the eventual compliance in simulation. We show that with our approach, interactive behavior planning while satisfying even complex traffic rules can be achieved. Moving forward, this gives us a generic framework to formalize traffic rules for autonomous vehicles.
Towards decolonising computational sciences
Venue: arXiv cs.CY 2020
Year: 2020

This article sets out our perspective on how to begin the journey of decolonising computational fields, such as data and cognitive sciences. We see this struggle as requiring two basic steps: a) realisation that the present-day system has inherited, and still enacts, hostile, conservative, and oppressive behaviours and principles towards women of colour (WoC); and b) rejection of the idea that centering individual people is a solution to system-level problems. The longer we ignore these two steps, the more "our" academic system maintains its toxic structure, excludes, and harms Black women and other minoritised groups. This also keeps the door open to discredited pseudoscience, like eugenics and physiognomy. We propose that grappling with our fields' histories and heritage holds the key to avoiding mistakes of the past. For example, initiatives such as "diversity boards" can still be harmful because they superficially appear reformatory but nonetheless center whiteness and maintain the status quo. Building on the shoulders of many WoC's work, who have been paving the way, we hope to advance the dialogue required to build both a grass-roots and a top-down re-imagining of computational sciences -- including but not limited to psychology, neuroscience, cognitive science, computer science, data science, statistics, machine learning, and artificial intelligence. We aspire for these fields to progress away from their stagnant, sexist, and racist shared past into carving and maintaining an ecosystem where both a diverse demographics of researchers and scientific ideas that critically challenge the status quo are welcomed.
TEST_POSITIVE at W-NUT 2020 Shared Task-3: Joint Event Multi-task Learning for Slot Filling in Noisy Text
Venue: arXiv cs.CL 2020
Year: 2020

The competition of extracting COVID-19 events from Twitter is to develop systems that can automatically extract related events from tweets. The built system should identify different pre-defined slots for each event, in order to answer important questions (e.g., Who is tested positive? What is the age of the person? Where is he/she?). To tackle these challenges, we propose the Joint Event Multi-task Learning (JOELIN) model. Through a unified global learning framework, we make use of all the training data across different events to learn and fine-tune the language model. Moreover, we implement a type-aware post-processing procedure using named entity recognition (NER) to further filter the predictions. JOELIN outperforms the BERT baseline by 17.2% in micro F1.
Toward Privacy and Utility Preserving Image Representation
Venue: arXiv cs.CV 2020
Year: 2020

Face images are rich data items that are useful and can easily be collected in many applications, such as in 1-to-1 face verification tasks in the domain of security and surveillance systems. Multiple methods have been proposed to protect an individual's privacy by perturbing the images to remove traces of identifiable information, such as gender or race. However, significantly less attention has been given to the problem of protecting images while maintaining optimal task utility. In this paper, we study the novel problem of creating privacy-preserving image representations with respect to a given utility task by proposing a principled framework called the Adversarial Image Anonymizer (AIA). AIA first creates an image representation using a generative model, then enhances the learned image representations using adversarial learning to preserve privacy and utility for a given task. Experiments were conducted on a publicly available data set to demonstrate the effectiveness of AIA as a privacy-preserving mechanism for face images.
Multiple Word Embeddings for Increased Diversity of Representation
Venue: arXiv cs.CL 2020
Year: 2020

Most state-of-the-art models in natural language processing (NLP) are neural models built on top of large, pre-trained, contextual language models that generate representations of words in context and are fine-tuned for the task at hand. The improvements afforded by these "contextual embeddings" come with a high computational cost. In this work, we explore a simple technique that substantially and consistently improves performance over a strong baseline with negligible increase in run time. We concatenate multiple pre-trained embeddings to strengthen our representation of words. We show that this concatenation technique works across many tasks, datasets, and model types. We analyze aspects of pre-trained embedding similarity and vocabulary coverage and find that the representational diversity between different pre-trained embeddings is the driving force of why this technique works. We provide open source implementations of our models in both TensorFlow and PyTorch.
A law of robustness for two-layers neural networks
Venue: arXiv cs.LG 2020
Year: 2020

We initiate the study of the inherent tradeoffs between the size of a neural network and its robustness, as measured by its Lipschitz constant. We make a precise conjecture that, for any Lipschitz activation function and for most datasets, any two-layers neural network with $k$ neurons that perfectly fit the data must have its Lipschitz constant larger (up to a constant) than $\sqrt{n/k}$ where $n$ is the number of datapoints. In particular, this conjecture implies that overparametrization is necessary for robustness, since it means that one needs roughly one neuron per datapoint to ensure a $O(1)$-Lipschitz network, while mere data fitting of $d$-dimensional data requires only one neuron per $d$ datapoints. We prove a weaker version of this conjecture when the Lipschitz constant is replaced by an upper bound on it based on the spectral norm of the weight matrix. We also prove the conjecture in the high-dimensional regime $n \approx d$ (which we also refer to as the undercomplete case, since only $k \leq d$ is relevant here). Finally we prove the conjecture for polynomial activation functions of degree $p$ when $n \approx d^p$. We complement these findings with experimental evidence supporting the conjecture.
Towards Improved Model Design for Authorship Identification: A Survey on Writing Style Understanding
Venue: arXiv cs.CL 2020
Year: 2020

Authorship identification tasks, which rely heavily on linguistic styles, have always been an important part of Natural Language Understanding (NLU) research. While other tasks based on linguistic style understanding benefit from deep learning methods, these methods have not behaved as well as traditional machine learning methods in many authorship-based tasks. With these tasks becoming more and more challenging, however, traditional machine learning methods based on handcrafted feature sets are already approaching their performance limits. Thus, in order to inspire future applications of deep learning methods in authorship-based tasks in ways that benefit the extraction of stylistic features, we survey authorship-based tasks and other tasks related to writing style understanding. We first describe our survey results on the current state of research in both sets of tasks and summarize existing achievements and problems in authorship-related tasks. We then describe outstanding methods in style-related tasks in general and analyze how they are used in combination in the top-performing models. We are optimistic about the applicability of these models to authorship-based tasks and hope our survey will help advance research in this field.
Neural RST-based Evaluation of Discourse Coherence
Venue: arXiv cs.CL 2020
Year: 2020

This paper evaluates the utility of Rhetorical Structure Theory (RST) trees and relations in discourse coherence evaluation. We show that incorporating silver-standard RST features can increase accuracy when classifying coherence. We demonstrate this through our tree-recursive neural model, namely RST-Recursive, which takes advantage of the text's RST features produced by a state of the art RST parser. We evaluate our approach on the Grammarly Corpus for Discourse Coherence (GCDC) and show that when ensembled with the current state of the art, we can achieve the new state of the art accuracy on this benchmark. Furthermore, when deployed alone, RST-Recursive achieves competitive accuracy while having 62% fewer parameters.
Impact and Calibration of Nonlinear Reciprocity Mismatch in Massive MIMO
Venue: arXiv cs.IT 2020
Year: 2020

Time-division-duplexing massive multiple-input multiple-output (MIMO) systems estimate the channel state information (CSI) by leveraging the uplink-downlink channel reciprocity, which is no longer valid when the mismatch arises from the asymmetric uplink and downlink radio frequency (RF) chains. Existing works treat the reciprocity mismatch as constant for simplicity. However, the practical RF chain consists of nonlinear components, which leads to nonlinear reciprocity mismatch. In this work, we examine the impact and the calibration approach of the nonlinear reciprocity mismatch in massive MIMO systems. To evaluate the impact of the nonlinear mismatch, we first derive the closed-form expression of the ergodic achievable rate. Then, we analyze the performance loss caused by the nonlinear mismatch to show that the impact of the mismatch at the base station (BS) side is much larger than that at the user equipment side. Therefore, we propose a calibration method for the BS. During the calibration, polynomial function is applied to approximate the nonlinear mismatch factor, and over-the-air training is employed to estimate the polynomial coefficients. After that, the calibration coefficients are computed by maximizing the downlink achievable rate. Simulation results are presented to verify the analytical results and to show the performance of the proposed calibration approach.