OpenRaiser/Intern-Atlas huggingface dataset:
How Chaotic Are Recurrent Neural Networks?
Venue: arXiv cs.CL 2020
Year: 2020

Recurrent neural networks (RNNs) are non-linear dynamic systems. Previous work believes that RNN may suffer from the phenomenon of chaos, where the system is sensitive to initial states and unpredictable in the long run. In this paper, however, we perform a systematic empirical analysis, showing that a vanilla or long short term memory (LSTM) RNN does not exhibit chaotic behavior along the training process in real applications such as text generation. Our findings suggest that future work in this direction should address the other side of non-linear dynamics for RNN.
Template-based Question Answering using Recursive Neural Networks
Venue: arXiv cs.CL 2020
Year: 2020

We propose a neural network-based approach to automatically learn and classify natural language questions into its corresponding template using recursive neural networks. An obvious advantage of using neural networks is the elimination of the need for laborious feature engineering that can be cumbersome and error-prone. The input question is encoded into a vector representation. The model is trained and evaluated on the LC-QuAD dataset (Large-scale Complex Question Answering Dataset). The LC-QuAD queries are annotated based on 38 unique templates that the model attempts to classify. The resulting model is evaluated against both the LC-QuAD dataset and the 7th Question Answering Over Linked Data (QALD-7) dataset. The recursive neural network achieves template classification accuracy of 0.828 on the LC-QuAD dataset and an accuracy of 0.618 on the QALD-7 dataset. When the top-2 most likely templates were considered the model achieves an accuracy of 0.945 on the LC-QuAD dataset and 0.786 on the QALD-7 dataset. After slot filling, the overall system achieves a macro F-score 0.419 on the LC-QuAD dataset and a macro F-score of 0.417 on the QALD-7 dataset.
DARE: Data Augmented Relation Extraction with GPT-2
Venue: arXiv cs.CL 2020
Year: 2020

Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues. In this work, we present Data Augmented Relation Extraction(DARE), a simple method to augment training data by properly fine-tuning GPT-2 to generate examples for specific relation types. The generated training data is then used in combination with the gold dataset to train a BERT-based RE classifier. In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points against a strong base-line. Also, DARE achieves new state of the art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F1 points on average.
Character-level Japanese Text Generation with Attention Mechanism for Chest Radiography Diagnosis
Venue: arXiv cs.CL 2020
Year: 2020

Chest radiography is a general method for diagnosing a patient's condition and identifying important information; therefore, radiography is used extensively in routine medical practice in various situations, such as emergency medical care and medical checkup. However, a high level of expertise is required to interpret chest radiographs. Thus, medical specialists spend considerable time in diagnosing such huge numbers of radiographs. In order to solve these problems, methods for generating findings have been proposed. However, the study of generating chest radiograph findings has primarily focused on the English language, and to the best of our knowledge, no studies have studied Japanese data on this subject. There are two challenges involved in generating findings in the Japanese language. The first challenge is that word splitting is difficult because the boundaries of Japanese word are not clear. The second challenge is that there are numerous orthographic variants. For deal with these two challenges, we proposed an end-to-end model that generates Japanese findings at the character-level from chest radiographs. In addition, we introduced the attention mechanism to improve not only the accuracy, but also the interpretation ability of the results. We evaluated the proposed method using a public dataset with Japanese findings. The effectiveness of the proposed method was confirmed using the Bilingual Evaluation Understudy score. And, we were confirmed from the generated findings that the proposed method was able to consider the orthographic variants. Furthermore, we confirmed via visual inspection that the attention mechanism captures the features and positional information of radiographs.
The Explanation Game: Towards Prediction Explainability through Sparse Communication
Venue: arXiv cs.CL 2020
Year: 2020

Explainability is a topic of growing importance in NLP. In this work, we provide a unified perspective of explainability as a communication problem between an explainer and a layperson about a classifier's decision. We use this framework to compare several prior approaches for extracting explanations, including gradient methods, representation erasure, and attention mechanisms, in terms of their communication success. In addition, we reinterpret these methods at the light of classical feature selection, and we use this as inspiration to propose new embedded methods for explainability, through the use of selective, sparse attention. Experiments in text classification, natural language entailment, and machine translation, using different configurations of explainers and laypeople (including both machines and humans), reveal an advantage of attention-based explainers over gradient and erasure methods. Furthermore, human evaluation experiments show promising results with post-hoc explainers trained to optimize communication success and faithfulness.
A Swarm of Simple Robots Constructing Planar Shapes
Venue: arXiv cs.RO 2020
Year: 2020

We present a new version of our previously proposed algorithm enabling a swarm of robots to construct a desired shape from objects in the plane. We also describe a hardware realization for this system which makes use of simple and readily sourced components. We refer to the task as planar construction which is the gathering of ambient objects into some desired shape. As an example application, a swarm of robots could use this algorithm to not only gather waste material into a pile, but shape that pile into a line for easy collection. The shape is specified by an image known as the scalar field. The scalar field serves an analogous role to the template pheromones that guide the construction of complex natural structures such as termite mounds. In addition to describing the algorithm and hardware platform, we develop some performance insights using a custom simulation environment and present experimental results on physical robots.
Hierarchy-Based Algorithms for Minimizing Makespan under Precedence and Communication Constraints
Venue: arXiv cs.DS 2020
Year: 2020

We consider the classic problem of scheduling jobs with precedence constraints on a set of identical machines to minimize the makespan objective function. Understanding the exact approximability of the problem when the number of machines is a constant is a well-known question in scheduling theory. Indeed, an outstanding open problem from the classic book of Garey and Johnson asks whether this problem is NP-hard even in the case of 3 machines and unit-length jobs. In a recent breakthrough, Levey and Rothvoss gave a $(1+ε)$-approximation algorithm, which runs in nearly quasi-polynomial time, for the case when job have unit lengths. However, a substantially more difficult case where jobs have arbitrary processing lengths has remained open. We make progress on this more general problem. We show that there exists a $(1+ε)$-approximation algorithm (with similar running time as that of Levey and Rothvoss) for the non-migratory setting: when every job has to be scheduled entirely on a single machine, but within a machine the job need not be scheduled during consecutive time steps. Further, we also show that our algorithmic framework generalizes to another classic scenario where, along with the precedence constraints, the jobs also have communication delay constraints. Both of these fundamental problems are highly relevant to the practice of datacenter scheduling.
Interactive Rainbow Score: A Visual-centered Multimodal Flute Tutoring System
Venue: arXiv cs.HC 2020
Year: 2020

Learning to play an instrument is intrinsically multimodal, and we have seen a trend of applying visual and haptic feedback in music games and computer-aided music tutoring systems. However, most current systems are still designed to master individual pieces of music; it is unclear how well the learned skills can be generalized to new pieces. We aim to explore this question. In this study, we contribute Interactive Rainbow Score, an interactive visual system to boost the learning of sight-playing, the general musical skill to read music and map the visual representations to performance motions. The key design of Interactive Rainbow Score is to associate pitches (and the corresponding motions) with colored notation and further strengthen such association via real-time interactions. Quantitative results show that the interactive feature on average increases the learning efficiency by 31.1%. Further analysis indicates that it is critical to apply the interaction in the early period of learning.
DS-FACTO: Doubly Separable Factorization Machines
Venue: arXiv cs.LG 2020
Year: 2020

Factorization Machines (FM) are powerful class of models that incorporate higher-order interaction among features to add more expressive power to linear models. They have been used successfully in several real-world tasks such as click-prediction, ranking and recommender systems. Despite using a low-rank representation for the pairwise features, the memory overheads of using factorization machines on large-scale real-world datasets can be prohibitively high. For instance on the criteo tera dataset, assuming a modest $128$ dimensional latent representation and $10^{9}$ features, the memory requirement for the model is in the order of $1$ TB. In addition, the data itself occupies $2.1$ TB. Traditional algorithms for FM which work on a single-machine are not equipped to handle this scale and therefore, using a distributed algorithm to parallelize the computation across a cluster is inevitable. In this work, we propose a hybrid-parallel stochastic optimization algorithm DS-FACTO, which partitions both the data as well as parameters of the factorization machine simultaneously. Our solution is fully de-centralized and does not require the use of any parameter servers. We present empirical results to analyze the convergence behavior, predictive power and scalability of DS-FACTO.
Directed Graph Convolutional Network
Venue: arXiv cs.LG 2020
Year: 2020

Graph Convolutional Networks (GCNs) have been widely used due to their outstanding performance in processing graph-structured data. However, the undirected graphs limit their application scope. In this paper, we extend spectral-based graph convolution to directed graphs by using first- and second-order proximity, which can not only retain the connection properties of the directed graph, but also expand the receptive field of the convolution operation. A new GCN model, called DGCN, is then designed to learn representations on the directed graph, leveraging both the first- and second-order proximity information. We empirically show the fact that GCNs working only with DGCNs can encode more useful information from graph and help achieve better performance when generalized to other models. Moreover, extensive experiments on citation networks and co-purchase datasets demonstrate the superiority of our model against the state-of-the-art methods.
Knowledgeable Dialogue Reading Comprehension on Key Turns
Venue: arXiv cs.CL 2020
Year: 2020

Multi-choice machine reading comprehension (MRC) requires models to choose the correct answer from candidate options given a passage and a question. Our research focuses dialogue-based MRC, where the passages are multi-turn dialogues. It suffers from two challenges, the answer selection decision is made without support of latently helpful commonsense, and the multi-turn context may hide considerable irrelevant information. This work thus makes the first attempt to tackle those two challenges by extracting substantially important turns and utilizing external knowledge to enhance the representation of context. In this paper, the relevance of each turn to the question are calculated to choose key turns. Besides, terms related to the context and the question in a knowledge graph are extracted as external knowledge. The original context, question and external knowledge are encoded with the pre-trained language model, then the language representation and key turns are combined together with a will-designed mechanism to predict the answer. Experimental results on a DREAM dataset show that our proposed model achieves great improvements on baselines.
Counting of Grapevine Berries in Images via Semantic Segmentation using Convolutional Neural Networks
Venue: arXiv cs.CV 2020
Year: 2020

The extraction of phenotypic traits is often very time and labour intensive. Especially the investigation in viticulture is restricted to an on-site analysis due to the perennial nature of grapevine. Traditionally skilled experts examine small samples and extrapolate the results to a whole plot. Thereby different grapevine varieties and training systems, e.g. vertical shoot positioning (VSP) and semi minimal pruned hedges (SMPH) pose different challenges. In this paper we present an objective framework based on automatic image analysis which works on two different training systems. The images are collected semi automatic by a camera system which is installed in a modified grape harvester. The system produces overlapping images from the sides of the plants. Our framework uses a convolutional neural network to detect single berries in images by performing a semantic segmentation. Each berry is then counted with a connected component algorithm. We compare our results with the Mask-RCNN, a state-of-the-art network for instance segmentation and with a regression approach for counting. The experiments presented in this paper show that we are able to detect green berries in images despite of different training systems. We achieve an accuracy for the berry detection of 94.0% in the VSP and 85.6% in the SMPH.
Automatically Identifying Gender Issues in Machine Translation using Perturbations
Venue: arXiv cs.CL 2020
Year: 2020

The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. While previous studies have identified issues using synthetic examples, we develop a novel technique to mine examples from real world data to explore challenges for deployed systems. We use our method to compile an evaluation benchmark spanning examples for four languages from three language families, which we publicly release to facilitate research. The examples in our benchmark expose where model representations are gendered, and the unintended consequences these gendered representations can have in downstream application.
FitChat: Conversational Artificial Intelligence Interventions for Encouraging Physical Activity in Older Adults
Venue: arXiv cs.HC 2020
Year: 2020

Delivery of digital behaviour change interventions which encourage physical activity has been tried in many forms. Most often interventions are delivered as text notifications, but these do not promote interaction. Advances in conversational AI have improved natural language understanding and generation, allowing AI chatbots to provide an engaging experience with the user. For this reason, chatbots have recently been seen in healthcare delivering digital interventions through free text or choice selection. In this work, we explore the use of voice-based AI chatbots as a novel mode of intervention delivery, specifically targeting older adults to encourage physical activity. We co-created "FitChat", an AI chatbot, with older adults and we evaluate the first prototype using Think Aloud Sessions. Our thematic evaluation suggests that older adults prefer voice-based chat over text notifications or free text entry and that voice is a powerful mode for encouraging motivation.
DeCoRIC: Decentralized Connected Resilient IoT Clustering
Venue: arXiv cs.NI 2020
Year: 2020

Maintaining peer-to-peer connectivity with low energy overhead is a key requirement for several emerging Internet of Things (IoT) applications. It is also desirable to develop such connectivity solutions for non-static network topologies, so that resilience to device failures can be fully realized. Decentralized clustering has emerged as a promising technique to address this critical challenge. Clustering of nodes around cluster heads (CHs) provides an energy-efficient two-tier framework for peer-to-peer communication. At the same time, decentralization ensures that the framework can quickly adapt to a dynamically changing network topology. Although some decentralized clustering solutions have been proposed in the literature, they either lack guarantees on connectivity or incur significant energy overhead to maintain the clusters. In this paper, we present Decentralized Connected Resilient IoT Clustering (DeCoRIC), an energy-efficient clustering scheme that is self-organizing and resilient to network changes while guaranteeing connectivity. Using experiments implemented on the Contiki simulator, we show that our clustering scheme adapts itself to node faults in a time-bound manner. Our experiments show that DeCoRIC achieves 100% connectivity among all nodes while improving the power efficiency of nodes in the system compared to the state-of-the-art techniques BEEM and LEACH by up to 110% and 70%, respectively. The improved power efficiency also translates to longer lifetime before first node death with a best-case of 109% longer than BEEM and 42% longer than LEACH.
Compilation of Coordinated Choice
Venue: arXiv cs.PL 2020
Year: 2020

Recently, we have proposed coordinated choices, which are nondeterministic choices equipped with names. The main characteristic of coordinated choices is that they synchronize nondeterministic decision among choices of the same name. The motivation of the synchronization mechanism is to solve a theoretical problem. So, as a practical programming language, we still want to use coordinated choices like standard ones. In other words, we want to avoid synchronization. Now, there are two problems: (i) practically, it is a bit complicated work to write a program using coordinated choices in which execution synchronization never happens; and (ii) theoretically, it is unknown whether any programs using standard choices can be written by using only coordinated ones. In this paper, we define two simply typed lambda calculi called $λ^\parallel$ equipped with standard choices and $λ^{\parallelω}$ equipped with coordinated choices, and give compilation rules from the former into the latter. The challenge is to show the correctness of the compilation because behavioral correspondence between expressions before and after compiling cannot be defined directly by the compilation rules. For the challenge, we give an effect system for $λ^{\parallelω}$ that characterizes expressions in which execution synchronization never happens. Then, we show that all compiled expressions can be typed by the effect system. As a result, we can easily show the correctness because the main concern of the correctness is whether synchronization happens or not.
Dense Steiner problems: Approximation algorithms and inapproximability
Venue: arXiv cs.DS 2020
Year: 2020

The Steiner Tree problem is a classical problem in combinatorial optimization: the goal is to connect a set $T$ of terminals in a graph $G$ by a tree of minimum size. Karpinski and Zelikovsky (1996) studied the $δ$-dense version of {\sc Steiner Tree}, where each terminal has at least $δ|V(G)\setminus T|$ neighbours outside $T$, for a fixed $δ> 0$. They gave a PTAS for this problem. We study a generalization of pairwise $δ$-dense {\sc Steiner Forest}, which asks for a minimum-size forest in $G$ in which the nodes in each terminal set $T_1,\dots,T_k$ are connected, and every terminal in $T_i$ has at least $δ|T_j|$ neighbours in $T_j$, and at least $δ|S|$ nodes in $S = V(G)\setminus (T_1\cup\dots\cup T_k)$, for each $i, j$ in $\{1,\dots, k\}$ with $i\neq j$. Our first result is a polynomial-time approximation scheme for all $δ> 1/2$. Then, we show a $(\frac{13}{12}+\varepsilon)$-approximation algorithm for $δ= 1/2$ and any $\varepsilon > 0$. We also consider the $δ$-dense Group Steiner Tree problem as defined by Hauptmann and show that the problem is $\mathsf{APX}$-hard.
Combining Word Embeddings and N-grams for Unsupervised Document Summarization
Venue: arXiv cs.CL 2020
Year: 2020

Graph-based extractive document summarization relies on the quality of the sentence similarity graph. Bag-of-words or tf-idf based sentence similarity uses exact word matching, but fails to measure the semantic similarity between individual words or to consider the semantic structure of sentences. In order to improve the similarity measure between sentences, we employ off-the-shelf deep embedding features and tf-idf features, and introduce a new text similarity metric. An improved sentence similarity graph is built and used in a submodular objective function for extractive summarization, which consists of a weighted coverage term and a diversity term. A Transformer based compression model is developed for sentence compression to aid in document summarization. Our summarization approach is extractive and unsupervised. Experiments demonstrate that our approach can outperform the tf-idf based approach and achieve state-of-the-art performance on the DUC04 dataset, and comparable performance to the fully supervised learning methods on the CNN/DM and NYT datasets.
BERT Fine-tuning For Arabic Text Summarization
Venue: arXiv cs.CL 2020
Year: 2020

Fine-tuning a pretrained BERT model is the state of the art method for extractive/abstractive text summarization, in this paper we showcase how this fine-tuning method can be applied to the Arabic language to both construct the first documented model for abstractive Arabic text summarization and show its performance in Arabic extractive summarization. Our model works with multilingual BERT (as Arabic language does not have a pretrained BERT of its own). We show its performance in English corpus first before applying it to Arabic corpora in both extractive and abstractive tasks.