OpenRaiser/Intern-Atlas huggingface dataset:
Improving Attention Mechanism with Query-Value Interaction
Venue: arXiv cs.CL 2020
Year: 2020

Attention mechanism has played critical roles in various state-of-the-art NLP models such as Transformer and BERT. It can be formulated as a ternary function that maps the input queries, keys and values into an output by using a summation of values weighted by the attention weights derived from the interactions between queries and keys. Similar with query-key interactions, there is also inherent relatedness between queries and values, and incorporating query-value interactions has the potential to enhance the output by learning customized values according to the characteristics of queries. However, the query-value interactions are ignored by existing attention methods, which may be not optimal. In this paper, we propose to improve the existing attention mechanism by incorporating query-value interactions. We propose a query-value interaction function which can learn query-aware attention values, and combine them with the original values and attention weights to form the final output. Extensive experiments on four datasets for different tasks show that our approach can consistently improve the performance of many attention-based models by incorporating query-value interactions.
An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference
Venue: arXiv cs.CL 2020
Year: 2020

The prior work on natural language inference (NLI) debiasing mainly targets at one or few known biases while not necessarily making the models more robust. In this paper, we focus on the model-agnostic debiasing strategies and explore how to (or is it possible to) make the NLI models robust to multiple distinct adversarial attacks while keeping or even strengthening the models' generalization power. We firstly benchmark prevailing neural NLI models including pretrained ones on various adversarial datasets. We then try to combat distinct known biases by modifying a mixture of experts (MoE) ensemble method and show that it's nontrivial to mitigate multiple NLI biases at the same time, and that model-level ensemble method outperforms MoE ensemble method. We also perform data augmentation including text swap, word substitution and paraphrase and prove its efficiency in combating various (though not all) adversarial attacks at the same time. Finally, we investigate several methods to merge heterogeneous training data (1.35M) and perform model ensembling, which are straightforward but effective to strengthen NLI models.
Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines
Venue: arXiv cs.AI 2020
Year: 2020

Text-based games have emerged as an important test-bed for Reinforcement Learning (RL) research, requiring RL agents to combine grounded language understanding with sequential decision making. In this paper, we examine the problem of infusing RL agents with commonsense knowledge. Such knowledge would allow agents to efficiently act in the world by pruning out implausible actions, and to perform look-ahead planning to determine how current actions might affect future world states. We design a new text-based gaming environment called TextWorld Commonsense (TWC) for training and evaluating RL agents with a specific kind of commonsense knowledge about objects, their attributes, and affordances. We also introduce several baseline RL agents which track the sequential context and dynamically retrieve the relevant commonsense knowledge from ConceptNet. We show that agents which incorporate commonsense knowledge in TWC perform better, while acting more efficiently. We conduct user-studies to estimate human performance on TWC and show that there is ample room for future improvement.
Mapping of Real World Problems to Nature Inspired Algorithm using Goal based Classification and TRIZ
Venue: arXiv cs.NE 2020
Year: 2020

The technologies and algorithms are growing at an exponential rate. The technologies are capable enough to solve technically challenging and complex problems which seemed impossible task. However, the trending methods and approaches are facing multiple challenges on various fronts of data, algorithms, software, computational complexities, and energy efficiencies. Nature also faces similar challenges. Nature has solved those challenges and formulation of those are available as Nature Inspired Algorithms (NIA), which are derived based on the study of nature. A novel method based on TRIZ to map the real-world problems to nature problems is explained here.TRIZ is a Theory of inventive problem solving. Using the proposed framework, best NIA can be identified to solve the real-world problems. For this framework to work, a novel classification of NIA based on the end goal that nature is trying to achieve is devised. The application of the this framework along with examples is also discussed.
Strategies for Integrating Controls Flows in Software-Defined In-Vehicle Networks and Their Impact on Network Security
Venue: arXiv cs.NI 2020
Year: 2020

Current In-Vehicle Networks (IVNs) connect Electronic Control Units (ECUs) via domain busses. A gateway forwards messages between these domains. Automotive Ethernet emerges as a flat, high-speed backbone technology for IVNs that carries the various control flows within Ethernet frames. Recently, Software-Defined-Networking (SDN) has been identified as a useful building block of the vehicular domain, as it allows the differentiation of packets based on all header fields and thus can isolate unrelated control flows. In this work, we systematically explore the different strategies for integrating automotive control flows in switched Ether-networks and analyze their security impact for a software-defined IVN. We discuss how control flow identifiers can be embedded on different layers resulting in a range of solutions from fully exposed embedding to deep encapsulation. We evaluate these strategies in a realistic IVN based on the communication matrix of a production grade vehicle, which we map into a modern Ethernet topology. We find that visibility of automotive control flows within packet headers is essential for the network infrastructure to enable isolation and access control. With an exposed embedding, the SDN backbone can establish and survey trust zones within the IVN and largely reduce the attack surface of connected cars. An exposed embedding strategy also minimizes communication expenses.
A practical guide towards agile test-driven development for scientific software projects
Venue: arXiv cs.SE 2020
Year: 2020

Software testing has received much attention over the last years and has reached such critical importance that agile software development practices put software testing at its core. Agile software development is successfully applied in large-scale industrial software developments but due to its granular responsibilities with roles assigned to various members of the development team, these practices may not be applicable to scientific code development, especially in an academic environment, where it is not uncommon that the codebase is developed, maintained and used by a single person. Even for collaborative scientific software development, financed through external grants, the end-users are typically still part of the development team. This is in contrast to how software is developed in many industries, where the development team and end-users are two separate entities. There are, however, many good code development practices that can be adopted for scientific software projects. Specifically, the intention of this article is to take the centrepiece of agile software development and tailor it to scientific and academic, single-user code development. In this study, a c++ starter project is developed and made available, based on the meson build system, which provides native support for software testing. It is used to show how a simple linear algebra application, found in many scientific and academic applications, can be developed and how simple unit, integration and system tests can be created that are managed through the meson build system. In this way, we are able to minimise software defects and reduce the risk to interpret incorrect data generated by erroneous software that may result in the wrong conclusions to be drawn. Each layer of testing presents one additional layer of protection and we will explore how these may be incorporated with minimum overhead to produce bug-free software.
Population Based Training for Data Augmentation and Regularization in Speech Recognition
Venue: arXiv cs.CL 2020
Year: 2020

Varying data augmentation policies and regularization over the course of optimization has led to performance improvements over using fixed values. We show that population based training is a useful tool to continuously search those hyperparameters, within a fixed budget. This greatly simplifies the experimental burden and computational cost of finding such optimal schedules. We experiment in speech recognition by optimizing SpecAugment this way, as well as dropout. It compares favorably to a baseline that does not change those hyperparameters over the course of training, with an 8% relative WER improvement. We obtain 5.18% word error rate on LibriSpeech's test-other.
An Experimental Analysis of Indoor Spatial Queries: Modeling, Indexing, and Processing
Venue: arXiv cs.DB 2020
Year: 2020

Indoor location-based services (LBS), such as POI search and routing, are often built on top of typical indoor spatial queries. To support such queries and indoor LBS, multiple techniques including model/indexes and search algorithms have been proposed. In this work, we conduct an extensive experimental study on existing proposals for indoor spatial queries. We survey five model/indexes, compare their algorithmic characteristics, and analyze their space and time complexities. We also design an in-depth benchmark with real and synthetic datasets, evaluation tasks and performance metrics. Enabled by the benchmark, we obtain and report the performance results of all model/indexes under investigation. By analyzing the results, we summarize the pros and cons of all techniques and suggest the best choice for typical scenarios.
Predicting Typological Features in WALS using Language Embeddings and Conditional Probabilities: ÚFAL Submission to the SIGTYP 2020 Shared Task
Venue: arXiv cs.CL 2020
Year: 2020

We present our submission to the SIGTYP 2020 Shared Task on the prediction of typological features. We submit a constrained system, predicting typological features only based on the WALS database. We investigate two approaches. The simpler of the two is a system based on estimating correlation of feature values within languages by computing conditional probabilities and mutual information. The second approach is to train a neural predictor operating on precomputed language embeddings based on WALS features. Our submitted system combines the two approaches based on their self-estimated confidence scores. We reach the accuracy of 70.7% on the test data and rank first in the shared task.
VECMAtk: A Scalable Verification, Validation and Uncertainty Quantification Toolkit for Scientific Simulations
Venue: arXiv cs.MS 2020
Year: 2020

We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantification (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year, we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail.
Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms
Venue: arXiv cs.LG 2020
Year: 2020

Understanding and removing bias from the decisions made by machine learning models is essential to avoid discrimination against unprivileged groups. Despite recent progress in algorithmic fairness, there is still no clear answer as to which bias-mitigation approaches are most effective. Evaluation strategies are typically use-case specific, rely on data with unclear bias, and employ a fixed policy to convert model outputs to decision outcomes. To address these problems, we performed a systematic comparison of a number of popular fairness algorithms applicable to supervised classification. Our study is the most comprehensive of its kind. It utilizes three real and four synthetic datasets, and two different ways of converting model outputs to decisions. It considers fairness, predictive-performance, calibration quality, and speed of 28 different modelling pipelines, corresponding to both fairness-unaware and fairness-aware algorithms. We found that fairness-unaware algorithms typically fail to produce adequately fair models and that the simplest algorithms are not necessarily the fairest ones. We also found that fairness-aware algorithms can induce fairness without material drops in predictive power. Finally, we found that dataset idiosyncracies (e.g., degree of intrinsic unfairness, nature of correlations) do affect the performance of fairness-aware approaches. Our results allow the practitioner to narrow down the approach(es) they would like to adopt without having to know in advance their fairness requirements.
Reversible Computation in Cyclic Petri Nets
Venue: arXiv cs.LO 2020
Year: 2020

Petri nets are a mathematical language for modeling and reasoning about distributed systems. In this paper we propose an approach to Petri nets for embedding reversibility, i.e., the ability of reversing an executed sequence of operations at any point during operation. Specifically, we introduce machinery and associated semantics to support the three main forms of reversibility namely, backtracking, causal reversing, and out-of-causal-order reversing in a variation of cyclic Petri nets where tokens are persistent and are distinguished from each other by an identity. Our formalism is influenced by applications in biochemistry but the methodology can be applied to a wide range of problems that feature reversibility. In particular, we demonstrate the applicability of our approach with a model of the ERK signalling pathway, an example that inherently features reversible behavior.
DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates
Venue: arXiv cs.LG 2020
Year: 2020

CPU simulators are useful tools for modeling CPU execution behavior. However, they suffer from inaccuracies due to the cost and complexity of setting their fine-grained parameters, such as the latencies of individual instructions. This complexity arises from the expertise required to design benchmarks and measurement frameworks that can precisely measure the values of parameters at such fine granularity. In some cases, these parameters do not necessarily have a physical realization and are therefore fundamentally approximate, or even unmeasurable. In this paper we present DiffTune, a system for learning the parameters of x86 basic block CPU simulators from coarse-grained end-to-end measurements. Given a simulator, DiffTune learns its parameters by first replacing the original simulator with a differentiable surrogate, another function that approximates the original function; by making the surrogate differentiable, DiffTune is then able to apply gradient-based optimization techniques even when the original function is non-differentiable, such as is the case with CPU simulators. With this differentiable surrogate, DiffTune then applies gradient-based optimization to produce values of the simulator's parameters that minimize the simulator's error on a dataset of ground truth end-to-end performance measurements. Finally, the learned parameters are plugged back into the original simulator. DiffTune is able to automatically learn the entire set of microarchitecture-specific parameters within the Intel x86 simulation model of llvm-mca, a basic block CPU simulator based on LLVM's instruction scheduling model. DiffTune's learned parameters lead llvm-mca to an average error that not only matches but lowers that of its original, expert-provided parameter values.
Precise Task Formalization Matters in Winograd Schema Evaluations
Venue: arXiv cs.CL 2020
Year: 2020

Performance on the Winograd Schema Challenge (WSC), a respected English commonsense reasoning benchmark, recently rocketed from chance accuracy to 89% on the SuperGLUE leaderboard, with relatively little corroborating evidence of a correspondingly large improvement in reasoning ability. We hypothesize that much of this improvement comes from recent changes in task formalization---the combination of input specification, loss function, and reuse of pretrained parameters---by users of the dataset, rather than improvements in the pretrained model's reasoning ability. We perform an ablation on two Winograd Schema datasets that interpolates between the formalizations used before and after this surge, and find (i) framing the task as multiple choice improves performance by 2-6 points and (ii) several additional techniques, including the reuse of a pretrained language modeling head, can mitigate the model's extreme sensitivity to hyperparameters. We urge future benchmark creators to impose additional structure to minimize the impact of formalization decisions on reported results.
Ensemble Machine Learning Methods for Modeling COVID19 Deaths
Venue: arXiv cs.LG 2020
Year: 2020

Using a hybrid of machine learning and epidemiological approaches, we propose a novel data-driven approach in predicting US COVID-19 deaths at a county level. The model gives a more complete description of the daily death distribution, outputting quantile-estimates instead of mean deaths, where the model's objective is to minimize the pinball loss on deaths reported by the New York Times coronavirus county dataset. The resulting quantile estimates accurately forecast deaths at an individual-county level for a variable-length forecast period, and the approach generalizes well across different forecast period lengths. We won the Caltech-run modeling competition out of 50+ teams, and our aggregate is competitive with the best COVID-19 modeling systems (on root mean squared error).
3D Object Detection and Pose Estimation of Unseen Objects in Color Images with Local Surface Embeddings
Venue: arXiv cs.CV 2020
Year: 2020

We present an approach for detecting and estimating the 3D poses of objects in images that requires only an untextured CAD model and no training phase for new objects. Our approach combines Deep Learning and 3D geometry: It relies on an embedding of local 3D geometry to match the CAD models to the input images. For points at the surface of objects, this embedding can be computed directly from the CAD model; for image locations, we learn to predict it from the image itself. This establishes correspondences between 3D points on the CAD model and 2D locations of the input images. However, many of these correspondences are ambiguous as many points may have similar local geometries. We show that we can use Mask-RCNN in a class-agnostic way to detect the new objects without retraining and thus drastically limit the number of possible correspondences. We can then robustly estimate a 3D pose from these discriminative correspondences using a RANSAC- like algorithm. We demonstrate the performance of this approach on the T-LESS dataset, by using a small number of objects to learn the embedding and testing it on the other objects. Our experiments show that our method is on par or better than previous methods.
BERTering RAMS: What and How Much does BERT Already Know About Event Arguments? -- A Study on the RAMS Dataset
Venue: arXiv cs.CL 2020
Year: 2020

Using the attention map based probing frame-work from (Clark et al., 2019), we observe that, on the RAMS dataset (Ebner et al., 2020), BERT's attention heads have modest but well above-chance ability to spot event arguments sans any training or domain finetuning, vary-ing from a low of 17.77% for Place to a high of 51.61% for Artifact. Next, we find that linear combinations of these heads, estimated with approx 11% of available total event argument detection supervision, can push performance well-higher for some roles - highest two being Victim (68.29% Accuracy) and Artifact(58.82% Accuracy). Furthermore, we investigate how well our methods do for cross-sentence event arguments. We propose a procedure to isolate "best heads" for cross-sentence argument detection separately of those for intra-sentence arguments. The heads thus estimated have superior cross-sentence performance compared to their jointly estimated equivalents, albeit only under the unrealistic assumption that we already know the argument is present in an-other sentence. Lastly, we seek to isolate to what extent our numbers stem from lexical frequency based associations between gold arguments and roles. We propose NONCE, a scheme to create adversarial test examples by replacing gold arguments with randomly generated "nonce" words. We find that learnt linear combinations are robust to NONCE, though individual best heads can be more sensitive.
Deep SVBRDF Estimation on Real Materials
Venue: arXiv cs.CV 2020
Year: 2020

Recent work has demonstrated that deep learning approaches can successfully be used to recover accurate estimates of the spatially-varying BRDF (SVBRDF) of a surface from as little as a single image. Closer inspection reveals, however, that most approaches in the literature are trained purely on synthetic data, which, while diverse and realistic, is often not representative of the richness of the real world. In this paper, we show that training such networks exclusively on synthetic data is insufficient to achieve adequate results when tested on real data. Our analysis leverages a new dataset of real materials obtained with a novel portable multi-light capture apparatus. Through an extensive series of experiments and with the use of a novel deep learning architecture, we explore two strategies for improving results on real data: finetuning, and a per-material optimization procedure. We show that adapting network weights to real data is of critical importance, resulting in an approach which significantly outperforms previous methods for SVBRDF estimation on real materials. Dataset and code are available at https://lvsn.github.io/real-svbrdf
Hybrid Beamforming in 5G mmWave Networks: a Full-stack Perspective
Venue: arXiv cs.IT 2020
Year: 2020

This paper studies the cross-layer challenges and performance of Hybrid Beamforming (HBF) and Multi-User Multiple-Input Multiple-Output (MU-MIMO) in 5G millimeter wave (mmWave) cellular networks with full-stack TCP/IP traffic and MAC scheduling. While previous research on HBF and MU-MIMO has focused on link-level analysis of full-buffer transmissions, this work reveals the interplay between HBF techniques and the higher layers of the protocol stack. To this aim, prior work on full stack simulation of mmWave cellular network has been extended by including the modeling of MU-MIMO and HBF. Our results reveal novel relations between the networking layers and the HBF MU-MIMO performance in the physical layer. Particularly, throughput can be increased in 5G networks by means of Spatial Division Multiple Access (SDMA). However, in order to achieve such benefits it is necessary to take into account certain trade-offs and the implementation complexity of a full-stack HBF solution.