OpenRaiser/Intern-Atlas huggingface dataset:
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models
Venue: arXiv cs.CL 2020
Year: 2020

LSTM-based recurrent neural networks are the state-of-the-art for many natural language processing (NLP) tasks. Despite their performance, it is unclear whether, or how, LSTMs learn structural features of natural languages such as subject-verb number agreement in English. Lacking this understanding, the generality of LSTM performance on this task and their suitability for related tasks remains uncertain. Further, errors cannot be properly attributed to a lack of structural capability, training data omissions, or other exceptional faults. We introduce *influence paths*, a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network. The approach refines the notion of influence (the subject's grammatical number has influence on the grammatical number of the subsequent verb) into a set of gate or neuron-level paths. The set localizes and segments the concept (e.g., subject-verb agreement), its constituent elements (e.g., the subject), and related or interfering elements (e.g., attractors). We exemplify the methodology on a widely-studied multi-layer LSTM language model, demonstrating its accounting for subject-verb number agreement. The results offer both a finer and a more complete view of an LSTM's handling of this structural aspect of the English language than prior results based on diagnostic classifiers and ablation.
Graph Homomorphism Convolution
Venue: arXiv cs.LG 2020
Year: 2020

In this paper, we study the graph classification problem from the graph homomorphism perspective. We consider the homomorphisms from $F$ to $G$, where $G$ is a graph of interest (e.g. molecules or social networks) and $F$ belongs to some family of graphs (e.g. paths or non-isomorphic trees). We show that graph homomorphism numbers provide a natural invariant (isomorphism invariant and $\mathcal{F}$-invariant) embedding maps which can be used for graph classification. Viewing the expressive power of a graph classifier by the $\mathcal{F}$-indistinguishable concept, we prove the universality property of graph homomorphism vectors in approximating $\mathcal{F}$-invariant functions. In practice, by choosing $\mathcal{F}$ whose elements have bounded tree-width, we show that the homomorphism method is efficient compared with other methods.
Probabilistic Analysis of RRT Trees
Venue: arXiv cs.RO 2020
Year: 2020

This thesis presents analysis of the properties and run-time of the Rapidly-exploring Random Tree (RRT) algorithm. It is shown that the time for the RRT with stepsize $ε$ to grow close to every point in the $d$-dimensional unit cube is $Θ\left(\frac1{ε^d} \log \left(\frac1ε\right)\right)$. Also, the time it takes for the tree to reach a region of positive probability is $O\left(ε^{-\frac32}\right)$. Finally, a relationship is shown to the Nearest Neighbour Tree (NNT). This relationship shows that the total Euclidean path length after $n$ steps is $O(\sqrt n)$ and the expected height of the tree is bounded above by $(e + o(1)) \log n$.
Generalized Reinforcement Meta Learning for Few-Shot Optimization
Venue: arXiv cs.LG 2020
Year: 2020

We present a generic and flexible Reinforcement Learning (RL) based meta-learning framework for the problem of few-shot learning. During training, it learns the best optimization algorithm to produce a learner (ranker/classifier, etc) by exploiting stable patterns in loss surfaces. Our method implicitly estimates the gradients of a scaled loss function while retaining the general properties intact for parameter updates. Besides providing improved performance on few-shot tasks, our framework could be easily extended to do network architecture search. We further propose a novel dual encoder, affinity-score based decoder topology that achieves additional improvements to performance. Experiments on an internal dataset, MQ2007, and AwA2 show our approach outperforms existing alternative approaches by 21%, 8%, and 4% respectively on accuracy and NDCG metrics. On Mini-ImageNet dataset our approach achieves comparable results with Prototypical Networks. Empirical evaluations demonstrate that our approach provides a unified and effective framework.
A New Data Normalization Method to Improve Dialogue Generation by Minimizing Long Tail Effect
Venue: arXiv cs.CL 2020
Year: 2020

Recent neural models have shown significant progress in dialogue generation. Most generation models are based on language models. However, due to the Long Tail Phenomenon in linguistics, the trained models tend to generate words that appear frequently in training datasets, leading to a monotonous issue. To address this issue, we analyze a large corpus from Wikipedia and propose three frequency-based data normalization methods. We conduct extensive experiments based on transformers and three datasets respectively collected from social media, subtitles, and the industrial application. Experimental results demonstrate significant improvements in diversity and informativeness (defined as the numbers of nouns and verbs) of generated responses. More specifically, the unigram and bigram diversity are increased by 2.6%-12.6% and 2.2%-18.9% on the three datasets, respectively. Moreover, the informativeness, i.e. the numbers of nouns and verbs, are increased by 4.0%-7.0% and 1.4%-12.1%, respectively. Additionally, the simplicity and effectiveness enable our methods to be adapted to different generation models without much extra computational cost.
Foraging-based Optimization of Menu Systems
Venue: arXiv cs.HC 2020
Year: 2020

Computational design of menu systems has been solved in limited cases such as the linear menu (list) as an assignment task, where commands are assigned to menu positions while optimizing for for users selection performance and distance of associated items. We show that this approach falls short with larger, hierarchically organized menu systems, where one must also take into account how users navigate hierarchical structures. This paper presents a novel integer programming formulation that models hierarchical menus as a combination of the exact set covering problem and the assignment problem. It organizes commands into ordered groups of ordered groups via a novel objective function based on information foraging theory. It minimizes, on the one hand, the time required to select a command whose location is known from previous usage and, on the other, the time wasted in irrelevant parts of the menu while searching for commands whose location is not known. The convergence of these two factors yields usable, well-ordered command hierarchies from a single model. In generated menus, the lead (first) elements of a group or tab are good indicators of the remaining contents, thereby facilitating the search process. In a controlled usability evaluation, the performance of computationally designed menus was 25 faster than existing commercial designs with respect to selection time. The algorithm is efficient for large, representative instances of the problem. We further show applications in personalization and adaptation of menu systems.
Building Proactive Voice Assistants: When and How (not) to Interact
Venue: arXiv cs.HC 2020
Year: 2020

Voice assistants have recently achieved remarkable commercial success. However, the current generation of these devices is typically capable of only reactive interactions. In other words, interactions have to be initiated by the user, which somewhat limits their usability and user experience. We propose, that the next generation of such devices should be able to proactively provide the right information in the right way at the right time, without being prompted by the user. However, achieving this is not straightforward, since there is the danger it could interrupt what the user is doing too much, resulting in it being distracting or even annoying. Furthermore, it could unwittingly, reveal sensitive/private information to third parties. In this report, we discuss the challenges of developing proactively initiated interactions, and suggest a framework for when it is appropriate for the device to intervene. To validate our design assumptions, we describe firstly, how we built a functioning prototype and secondly, a user study that was conducted to assess users' reactions and reflections when in the presence of a proactive voice assistant. This pre-print summarises the state, ideas and progress towards a proactive device as of autumn 2018.
DoQA -- Accessing Domain-Specific FAQs via Conversational QA
Venue: arXiv cs.CL 2020
Year: 2020

The goal of this work is to build conversational Question Answering (QA) interfaces for the large body of domain-specific information available in FAQ sites. We present DoQA, a dataset with 2,437 dialogues and 10,917 QA pairs. The dialogues are collected from three Stack Exchange sites using the Wizard of Oz method with crowdsourcing. Compared to previous work, DoQA comprises well-defined information needs, leading to more coherent and natural conversations with less factoid questions and is multi-domain. In addition, we introduce a more realistic information retrieval(IR) scenario where the system needs to find the answer in any of the FAQ documents. The results of an existing, strong, system show that, thanks to transfer learning from a Wikipedia QA dataset and fine tuning on a single FAQ domain, it is possible to build high quality conversational QA systems for FAQs without in-domain training data. The good results carry over into the more challenging IR scenario. In both cases, there is still ample room for improvement, as indicated by the higher human upperbound.
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Venue: arXiv cs.CL 2020
Year: 2020

Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones. Overall, humans are correct more often than out-of-the-box models, and the models are sometimes right for the wrong reasons. Finally, we show that fine-tuning on a large, task-specific dataset can offer a solution to these issues.
Knowledge Graph Validation
Venue: arXiv cs.DB 2020
Year: 2020

Knowledge graphs (KGs) have shown to be an important asset of large companies like Google and Microsoft. KGs play an important role in providing structured and semantically rich information, making them available to people and machines, and supplying accurate, correct and reliable knowledge. To do so a critical task is knowledge validation, which measures whether statements from KGs are semantically correct and correspond to the so-called "real" world. In this paper, we provide an overview and review of the state-of-the-art approaches, methods and tools on knowledge validation for KGs, as well as an evaluation of them. As a result, we demonstrate a lack of reproducibility of tools results, give insights, and state our future research direction.
Automata Tutor v3
Venue: arXiv cs.FL 2020
Year: 2020

Computer science class enrollments have rapidly risen in the past decade. With current class sizes, standard approaches to grading and providing personalized feedback are no longer possible and new techniques become both feasible and necessary. In this paper, we present the third version of Automata Tutor, a tool for helping teachers and students in large courses on automata and formal languages. The second version of Automata Tutor supported automatic grading and feedback for finite-automata constructions and has already been used by thousands of users in dozens of countries. This new version of Automata Tutor supports automated grading and feedback generation for a greatly extended variety of new problems, including problems that ask students to create regular expressions, context-free grammars, pushdown automata and Turing machines corresponding to a given description, and problems about converting between equivalent models - e.g., from regular expressions to nondeterministic finite automata. Moreover, for several problems, this new version also enables teachers and students to automatically generate new problem instances. We also present the results of a survey run on a class of 950 students, which shows very positive results about the usability and usefulness of the tool.
Automated eye disease classification method from anterior eye image using anatomical structure focused image classification technique
Venue: arXiv cs.CV 2020
Year: 2020

This paper presents an automated classification method of infective and non-infective diseases from anterior eye images. Treatments for cases of infective and non-infective diseases are different. Distinguishing them from anterior eye images is important to decide a treatment plan. Ophthalmologists distinguish them empirically. Quantitative classification of them based on computer assistance is necessary. We propose an automated classification method of anterior eye images into cases of infective or non-infective disease. Anterior eye images have large variations of the eye position and brightness of illumination. This makes the classification difficult. If we focus on the cornea, positions of opacified areas in the corneas are different between cases of the infective and non-infective diseases. Therefore, we solve the anterior eye image classification task by using an object detection approach targeting the cornea. This approach can be said as "anatomical structure focused image classification". We use the YOLOv3 object detection method to detect corneas of infective disease and corneas of non-infective disease. The detection result is used to define a classification result of a image. In our experiments using anterior eye images, 88.3% of images were correctly classified by the proposed method.
Neuromorphic AI Empowered Root Cause Analysis of Faults in Emerging Networks
Venue: arXiv cs.NI 2020
Year: 2020

Mobile cellular network operators spend nearly a quarter of their revenue on network maintenance and management. A significant portion of that budget is spent on resolving faults diagnosed in the system that disrupt or degrade cellular services. Historically, the operations to detect, diagnose and resolve issues were carried out by human experts. However, with diversifying cell types, increased complexity and growing cell density, this methodology is becoming less viable, both technically and financially. To cope with this problem, in recent years, research on self-healing solutions has gained significant momentum. One of the most desirable features of the self-healing paradigm is automated fault diagnosis. While several fault detection and diagnosis machine learning models have been proposed recently, these schemes have one common tenancy of relying on human expert contribution for fault diagnosis and prediction in one way or another. In this paper, we propose an AI-based fault diagnosis solution that offers a key step towards a completely automated self-healing system without requiring human expert input. The proposed solution leverages Random Forests classifier, Convolutional Neural Network and neuromorphic based deep learning model which uses RSRP map images of faults generated. We compare the performance of the proposed solution against state-of-the-art solution in literature that mostly use Naive Bayes models, while considering seven different fault types. Results show that neuromorphic computing model achieves high classification accuracy as compared to the other models even with relatively small training data
Robotic Self-Assessment of Competence
Venue: arXiv cs.RO 2020
Year: 2020

In robotics, one of the main challenges is that the on-board Artificial Intelligence (AI) must deal with different or unexpected environments. Such AI agents may be incompetent there, while the underlying model itself may not be aware of this (e.g., deep learning models are often overly confident). This paper proposes two methods for the online assessment of the competence of the AI model, respectively for situations when nothing is known about competence beforehand, and when there is prior knowledge about competence (in semantic form). The proposed method assesses whether the current environment is known. If not, it asks a human for feedback about its competence. If it knows the environment, it assesses its competence by generalizing from earlier experience. Results on real data show the merit of competence assessment for a robot moving through various environments in which it sometimes is competent and at other times it is not competent. We discuss the role of the human in robot's self-assessment of its competence, and the challenges to acquire complementary information from the human that reinforces the assessments.
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Venue: arXiv cs.LG 2020
Year: 2020

In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.
Learning the Associations of MITRE ATT&CK Adversarial Techniques
Venue: arXiv cs.CR 2020
Year: 2020

The MITRE ATT&CK Framework provides a rich and actionable repository of adversarial tactics, techniques, and procedures (TTP). However, this information would be highly useful for attack diagnosis (i.e., forensics) and mitigation (i.e., intrusion response) if we can reliably construct technique associations that will enable predicting unobserved attack techniques based on observed ones. In this paper, we present our statistical machine learning analysis on APT and Software attack data reported by MITRE ATT&CK to infer the technique clustering that represents the significant correlation that can be used for technique prediction. Due to the complex multidimensional relationships between techniques, many of the traditional clustering methods could not obtain usable associations. Our approach, using hierarchical clustering for inferring attack technique associations with 95% confidence, provides statistically significant and explainable technique correlations. Our analysis discovers 98 different technique associations (i.e., clusters) for both APT and Software attacks. Our evaluation results show that 78% of the techniques associated by our algorithm exhibit significant mutual information that indicates reasonably high predictability.
Fast and Robust Unsupervised Contextual Biasing for Speech Recognition
Venue: arXiv cs.CL 2020
Year: 2020

Automatic speech recognition (ASR) system is becoming a ubiquitous technology. Although its accuracy is closing the gap with that of human level under certain settings, one area that can further improve is to incorporate user-specific information or context to bias its prediction. A common framework is to dynamically construct a small language model from the provided contextual mini corpus and interpolate its score with the main language model during the decoding process. Here we propose an alternative approach that does not entail explicit contextual language model. Instead, we derive the bias score for every word in the system vocabulary from the training corpus. The method is unique in that 1) it does not require meta-data or class-label annotation for the context or the training corpus. 2) The bias score is proportional to the word's log-probability, thus not only would it bias the provided context, but also robust against irrelevant context (e.g. user mis-specified or in case where it is hard to quantify a tight scope). 3) The bias score for the entire vocabulary is pre-determined during the training stage, thereby eliminating computationally expensive language model construction during inference. We show significant improvement in recognition accuracy when the relevant context is available. Additionally, we also demonstrate that the proposed method exhibits high tolerance to false-triggering errors in the presence of irrelevant context.
ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents
Venue: arXiv cs.CL 2020
Year: 2020

We present ADVISER - an open-source, multi-domain dialog system toolkit that enables the development of multi-modal (incorporating speech, text and vision), socially-engaged (e.g. emotion recognition, engagement level prediction and backchanneling) conversational agents. The final Python-based implementation of our toolkit is flexible, easy to use, and easy to extend not only for technically experienced users, such as machine learning researchers, but also for less technically experienced users, such as linguists or cognitive scientists, thereby providing a flexible platform for collaborative research. Link to open-source code: https://github.com/DigitalPhonetics/adviser
Preventing Time Synchronization in NTP's Broadcast Mode
Venue: arXiv cs.CR 2020
Year: 2020

Network Time Protocol (NTP) is used by millions of hosts in Internet today to synchronize their clocks. Clock synchronization is necessary for many network applications to function correctly. Unsynchronized clock may lead to failure of various core Internet services including DNS and RPKI based interdomain routing and opens path for more sophisticated attacks. In this paper, we describe a new attack which can prevent a client configured in NTP's broadcast mode from synchronizing its clock with the server. We test the attack in real networks and show that it is effective in both authenticated and unauthenticated broadcast/multicast modes of NTP. We also perform experiments to measure the overall attack surface by scanning the entire IPv4 address space and show that NTP broadcast mode is being used in the wild by several low stratum (highly accurate) hosts. We also suggest few countermeasures to mitigate the proposed attack.
Data Science as a Route to AI for Middle- and High-School Students
Venue: arXiv cs.CY 2020
Year: 2020

The Bootstrap Project's Data Science curriculum has trained about 100 teachers who are using it around the country. It is specifically designed to aid adoption at a wide range of institutions. It emphasizes valuable curricular goals by drawing on both the education literature and on prior experience with other computing outreach projects. It embraces "three P's" of data-oriented thinking: the promise, pitfalls, and perils. This paper briefly describes the curriculum's design, content, and outcomes, and explains its value on the road to AI curricula.