OpenRaiser/Intern-Atlas huggingface dataset:
Adversarial Multi-Source Transfer Learning in Healthcare: Application to Glucose Prediction for Diabetic People
Venue: arXiv cs.CV 2020
Year: 2020

Deep learning has yet to revolutionize general practices in healthcare, despite promising results for some specific tasks. This is partly due to data being in insufficient quantities hurting the training of the models. To address this issue, data from multiple health actors or patients could be combined by capitalizing on their heterogeneity through the use of transfer learning. To improve the quality of the transfer between multiple sources of data, we propose a multi-source adversarial transfer learning framework that enables the learning of a feature representation that is similar across the sources, and thus more general and more easily transferable. We apply this idea to glucose forecasting for diabetic people using a fully convolutional neural network. The evaluation is done by exploring various transfer scenarios with three datasets characterized by their high inter and intra variability. While transferring knowledge is beneficial in general, we show that the statistical and clinical accuracies can be further improved by using of the adversarial training methodology, surpassing the current state-of-the-art results. In particular, it shines when using data from different datasets, or when there is too little data in an intra-dataset situation. To understand the behavior of the models, we analyze the learnt feature representations and propose a new metric in this regard. Contrary to a standard transfer, the adversarial transfer does not discriminate the patients and datasets, helping the learning of a more general feature representation. The adversarial training framework improves the learning of a general feature representation in a multi-source environment, enhancing the knowledge transfer to an unseen target. The proposed method can help improve the efficiency of data shared by different health actors in the training of deep models.
Hinting Semantic Parsing with Statistical Word Sense Disambiguation
Venue: arXiv cs.CL 2020
Year: 2020

The task of Semantic Parsing can be approximated as a transformation of an utterance into a logical form graph where edges represent semantic roles and nodes represent word senses. The resulting representation should be capture the meaning of the utterance and be suitable for reasoning. Word senses and semantic roles are interdependent, meaning errors in assigning word senses can cause errors in assigning semantic roles and vice versa. While statistical approaches to word sense disambiguation outperform logical, rule-based semantic parsers for raw word sense assignment, these statistical word sense disambiguation systems do not produce the rich role structure or detailed semantic representation of the input. In this work, we provide hints from a statistical WSD system to guide a logical semantic parser to produce better semantic type assignments while maintaining the soundness of the resulting logical forms. We observe an improvement of up to 10.5% in F-score, however we find that this improvement comes at a cost to the structural integrity of the parse
End-Effect Exploration Drive for Effective Motor Learning
Venue: arXiv cs.AI 2020
Year: 2020

Stemming on the idea that a key objective in reinforcement learning is to invert a target distribution of effects, end-effect drives are proposed as an effective way to implement goal-directed motor learning, in the absence of an explicit forward model. An end-effect model relies on a simple statistical recording of the effect of the current policy, here used as a substitute for the more resource-demanding forward models. When combined with a reward structure, it forms the core of a lightweight variational free energy minimization setup. The main difficulty lies in the maintenance of this simplified effect model together with the online update of the policy. When the prior target distribution is uniform, it provides a ways to learn an efficient exploration policy, consistently with the intrinsic curiosity principles. When combined with an extrinsic reward, our approach is finally shown to provide a faster training than traditional off-policy techniques.
Explainable 3D Convolutional Neural Networks by Learning Temporal Transformations
Venue: arXiv cs.CV 2020
Year: 2020

In this paper we introduce the temporally factorized 3D convolution (3TConv) as an interpretable alternative to the regular 3D convolution (3DConv). In a 3TConv the 3D convolutional filter is obtained by learning a 2D filter and a set of temporal transformation parameters, resulting in a sparse filter where the 2D slices are sequentially dependent on each other in the temporal dimension. We demonstrate that 3TConv learns temporal transformations that afford a direct interpretation. The temporal parameters can be used in combination with various existing 2D visualization methods. We also show that insight about what the model learns can be achieved by analyzing the transformation parameter statistics on a layer and model level. Finally, we implicitly demonstrate that, in popular ConvNets, the 2DConv can be replaced with a 3TConv and that the weights can be transferred to yield pretrained 3TConvs. pretrained 3TConvnets leverage more than a decade of work on traditional 2DConvNets by being able to make use of features that have been proven to deliver excellent results on image classification benchmarks.
Harnessing Adversarial Distances to Discover High-Confidence Errors
Venue: arXiv cs.LG 2020
Year: 2020

Given a deep neural network image classification model that we treat as a black box, and an unlabeled evaluation dataset, we develop an efficient strategy by which the classifier can be evaluated. Randomly sampling and labeling instances from an unlabeled evaluation dataset allows traditional performance measures like accuracy, precision, and recall to be estimated. However, random sampling may miss rare errors for which the model is highly confident in its prediction, but wrong. These high-confidence errors can represent costly mistakes, and therefore should be explicitly searched for. Past works have developed search techniques to find classification errors above a specified confidence threshold, but ignore the fact that errors should be expected at confidence levels anywhere below 100\%. In this work, we investigate the problem of finding errors at rates greater than expected given model confidence. Additionally, we propose a query-efficient and novel search technique that is guided by adversarial perturbations to find these mistakes in black box models. Through rigorous empirical experimentation, we demonstrate that our Adversarial Distance search discovers high-confidence errors at a rate greater than expected given model confidence.
Using an Automated Heterogeneous Robotic System for Radiation Surveys
Venue: arXiv cs.RO 2020
Year: 2020

During missions involving radiation exposure, unmanned robotic platforms may embody a valuable tool, especially thanks to their capability of replacing human operators in certain tasks to eliminate the health risks associated with such an environment. Moreover, rapid development of the technology allows us to increase the automation rate, making the human operator generally less important within the entire process. This article presents a multi-robotic system designed for highly automated radiation mapping and source localization. Our approach includes a three-phase procedure comprising sequential deployment of two diverse platforms, namely, an unmanned aircraft system (UAS) and an unmanned ground vehicle (UGV), to perform aerial photogrammetry, aerial radiation mapping, and terrestrial radiation mapping. The central idea is to produce a sparse dose rate map of the entire study site via the UAS and, subsequently, to perform detailed UGV-based mapping in limited radiation-contaminated regions. To accomplish these tasks, we designed numerous methods and data processing algorithms to facilitate, for example, digital elevation model (DEM)-based terrain following for the UAS, automatic selection of the regions of interest, obstacle map-based UGV trajectory planning, and source localization. The overall usability of the multi-robotic system was demonstrated by means of a one-day, authentic experiment, namely, a fictitious car accident including the loss of several radiation sources. The ability of the system to localize radiation hotspots and individual sources has been verified.
Measuring Memorization Effect in Word-Level Neural Networks Probing
Venue: arXiv cs.CL 2020
Year: 2020

Multiple studies have probed representations emerging in neural networks trained for end-to-end NLP tasks and examined what word-level linguistic information may be encoded in the representations. In classical probing, a classifier is trained on the representations to extract the target linguistic information. However, there is a threat of the classifier simply memorizing the linguistic labels for individual words, instead of extracting the linguistic abstractions from the representations, thus reporting false positive results. While considerable efforts have been made to minimize the memorization problem, the task of actually measuring the amount of memorization happening in the classifier has been understudied so far. In our work, we propose a simple general method for measuring the memorization effect, based on a symmetric selection of comparable sets of test words seen versus unseen in training. Our method can be used to explicitly quantify the amount of memorization happening in a probing setup, so that an adequate setup can be chosen and the results of the probing can be interpreted with a reliability estimate. We exemplify this by showcasing our method on a case study of probing for part of speech in a trained neural machine translation encoder.
Legal Risks of Adversarial Machine Learning Research
Venue: arXiv cs.CY 2020
Year: 2020

Adversarial Machine Learning is booming with ML researchers increasingly targeting commercial ML systems such as those used in Facebook, Tesla, Microsoft, IBM, Google to demonstrate vulnerabilities. In this paper, we ask, "What are the potential legal risks to adversarial ML researchers when they attack ML systems?" Studying or testing the security of any operational system potentially runs afoul the Computer Fraud and Abuse Act (CFAA), the primary United States federal statute that creates liability for hacking. We claim that Adversarial ML research is likely no different. Our analysis show that because there is a split in how CFAA is interpreted, aspects of adversarial ML attacks, such as model inversion, membership inference, model stealing, reprogramming the ML system and poisoning attacks, may be sanctioned in some jurisdictions and not penalized in others. We conclude with an analysis predicting how the US Supreme Court may resolve some present inconsistencies in the CFAA's application in Van Buren v. United States, an appeal expected to be decided in 2021. We argue that the court is likely to adopt a narrow construction of the CFAA, and that this will actually lead to better adversarial ML security outcomes in the long term.
Shared vs Private Randomness in Distributed Interactive Proofs
Venue: arXiv cs.DC 2020
Year: 2020

In distributed interactive proofs, the nodes of a graph G interact with a powerful but untrustable prover who tries to convince them, in a small number of rounds and through short messages, that G satisfies some property. This series of interactions is followed by a phase of distributed verification, which may be either deterministic or randomized, where nodes exchange messages with their neighbors. The nature of this last verification round defines the two types of interactive protocols. We say that the protocol is of Arthur-Merlin type if the verification round is deterministic. We say that the protocol is of Merlin-Arthur type if, in the verification round, the nodes are allowed to use a fresh set of random bits. In the original model introduced by Kol, Oshman, and Saxena [PODC 2018], the randomness was private in the sense that each node had only access to an individual source of random coins. Crescenzi, Fraigniaud, and Paz [DISC 2019] initiated the study of the impact of shared randomness (the situation where the coin tosses are visible to all nodes) in the distributed interactive model. In this work, we continue that research line by showing that the impact of the two forms of randomness is very different depending on whether we are considering Arthur-Merlin protocols or Merlin-Arthur protocols. While private randomness gives more power to the first type of protocols, shared randomness provides more power to the second. Our results also connect shared randomness in distributed interactive proofs with distributed verification, and new lower bounds are obtained.
Liquid Resource Types
Venue: arXiv cs.PL 2020
Year: 2020

This article presents liquid resource types, a technique for automatically verifying the resource consumption of functional programs. Existing resource analysis techniques trade automation for flexibility -- automated techniques are restricted to relatively constrained families of resource bounds, while more expressive proof techniques admitting value-dependent bounds rely on handwritten proofs. Liquid resource types combine the best of these approaches, using logical refinements to automatically prove precise bounds on a program's resource consumption. The type system augments refinement types with potential annotations to conduct an amortized resource analysis. Importantly, users can annotate data structure declarations to indicate how potential is allocated within the type, allowing the system to express bounds with polynomials and exponentials, as well as more precise expressions depending on program values. We prove the soundness of the type system, provide a library of flexible and reusable data structures for conducting resource analysis, and use our prototype implementation to automatically verify resource bounds that previously required a manual proof.
Universal linguistic inductive biases via meta-learning
Venue: arXiv cs.CL 2020
Year: 2020

How do learners acquire languages from the limited data available to them? This process must involve some inductive biases - factors that affect how a learner generalizes - but it is unclear which inductive biases can explain observed patterns in language acquisition. To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases. This framework disentangles universal inductive biases, which are encoded in the initial values of a neural network's parameters, from non-universal factors, which the neural network must learn from data in a given language. The initial state that encodes the inductive biases is found with meta-learning, a technique through which a model discovers how to acquire new languages more easily via exposure to many possible languages. By controlling the properties of the languages that are used during meta-learning, we can control the inductive biases that meta-learning imparts. We demonstrate this framework with a case study based on syllable structure. First, we specify the inductive biases that we intend to give our model, and then we translate those inductive biases into a space of languages from which a model can meta-learn. Finally, using existing analysis techniques, we verify that our approach has imparted the linguistic inductive biases that it was intended to impart.
Handling Missing Data in Decision Trees: A Probabilistic Approach
Venue: arXiv cs.LG 2020
Year: 2020

Decision trees are a popular family of models due to their attractive properties such as interpretability and ability to handle heterogeneous data. Concurrently, missing data is a prevalent occurrence that hinders performance of machine learning models. As such, handling missing data in decision trees is a well studied problem. In this paper, we tackle this problem by taking a probabilistic approach. At deployment time, we use tractable density estimators to compute the "expected prediction" of our models. At learning time, we fine-tune parameters of already learned trees by minimizing their "expected prediction loss" w.r.t.\ our density estimators. We provide brief experiments showcasing effectiveness of our methods compared to few baselines.
SeMPE: Secure Multi Path Execution Architecture for Removing Conditional Branch Side Channels
Venue: arXiv cs.CR 2020
Year: 2020

One of the most prevalent source of side channel vulnerabilities is the secret-dependent behavior of conditional branches (SDBCB). The state-of-the-art solution relies on Constant-Time Expressions, which require high programming effort and incur high performance overheads. In this paper, we propose SeMPE, an approach that relies on architecture support to eliminate SDBCB without requiring much programming effort while incurring low performance overheads. The key idea is that when a secret-dependent branch is encountered, the SeMPE microarchitecture fetches, executes, and commits both paths of the branch, preventing the adversary from inferring secret values from the branching behavior of the program. To enable that, SeMPE relies on an architecture that is capable of safely executing both branch paths sequentially. Through microbenchmarks and an evaluation of a real-world library, we show that SeMPE incurs near ideal execution time overheads, which is the sum of the execution time of all branch paths of secret-dependent branches. SeMPE outperforms code generated by FaCT, a constant-time expression language, by up to a factor of 18x.
Reading Between the Demographic Lines: Resolving Sources of Bias in Toxicity Classifiers
Venue: arXiv cs.CY 2020
Year: 2020

The censorship of toxic comments is often left to the judgment of imperfect models. Perspective API, a creation of Google technology incubator Jigsaw, is perhaps the most widely used toxicity classifier in industry; the model is employed by several online communities including The New York Times to identify and filter out toxic comments with the goal of preserving online safety. Unfortunately, Google's model tends to unfairly assign higher toxicity scores to comments containing words referring to the identities of commonly targeted groups (e.g., "woman,'' "gay,'' etc.) because these identities are frequently referenced in a disrespectful manner in the training data. As a result, comments generated by marginalized groups referencing their identities are often mistakenly censored. It is important to be cognizant of this unintended bias and strive to mitigate its effects. To address this issue, we have constructed several toxicity classifiers with the intention of reducing unintended bias while maintaining strong classification performance.
Ideal Membership Problem for Boolean Minority
Venue: arXiv cs.CC 2020
Year: 2020

The Ideal Membership Problem (IMP) tests if an input polynomial $f\in \mathbb{F}[x_1,\dots,x_n]$ with coefficients from a field $\mathbb{F}$ belongs to a given ideal $I \subseteq \mathbb{F}[x_1,\dots,x_n]$. It is a well-known fundamental problem with many important applications, though notoriously intractable in the general case. In this paper we consider the IMP for polynomial ideals encoding combinatorial problems and where the input polynomial $f$ has degree at most $d=O(1)$ (we call this problem IMP$_d$). A dichotomy result between ``hard'' (NP-hard) and ``easy'' (polynomial time) IMPs was recently achieved for Constraint Satisfaction Problems over finite domains [Bulatov FOCS'17, Zhuk FOCS'17] (this is equivalent to IMP$_0$) and IMP$_d$ for the Boolean domain [Mastrolilli SODA'19], both based on the classification of the IMP through functions called polymorphisms. The complexity of the IMP$_d$ for five polymorphisms has been solved in [Mastrolilli SODA'19] whereas for the ternary minority polymorphism it was incorrectly declared to have been resolved by a previous result. As a matter of fact the complexity of the IMP$_d$ for the ternary minority polymorphism is open. In this paper we provide the missing link by proving that the IMP$_d$ for Boolean combinatorial ideals whose constraints are closed under the minority polymorphism can be solved in polynomial time. This result, along with the results in [Mastrolilli SODA'19], completes the identification of the precise borderline of tractability for the IMP$_d$ for constrained problems over the Boolean domain. This paper is motivated by the pursuit of understanding the issue of bit complexity of Sum-of-Squares proofs raised by O'Donnell [ITCS'17]. Raghavendra and Weitz [ICALP'17] show how the IMP$_d$ tractability for combinatorial ideals implies bounded coefficients in Sum-of-Squares proofs.
Learning to Read through Machine Teaching
Venue: arXiv cs.LG 2020
Year: 2020

Learning to read words aloud is a major step towards becoming a reader. Many children struggle with the task because of the inconsistencies of English spelling-sound correspondences. Curricula vary enormously in how these patterns are taught. Children are nonetheless expected to master the system in limited time (by grade 4). We used a cognitively interesting neural network architecture to examine whether the sequence of learning trials could be structured to facilitate learning. This is a hard combinatorial optimization problem even for a modest number of learning trials (e.g., 10K). We show how this sequence optimization problem can be posed as optimizing over a time varying distribution i.e., defining probability distributions over words at different steps in training. We then use stochastic gradient descent to find an optimal time-varying distribution and a corresponding optimal training sequence. We observed significant improvement on generalization accuracy compared to baseline conditions (random sequences; sequences biased by word frequency). These findings suggest an approach to improving learning outcomes in domains where performance depends on ability to generalize beyond limited training experience.
Graph Neural Networks for Leveraging Industrial Equipment Structure: An application to Remaining Useful Life Estimation
Venue: arXiv cs.LG 2020
Year: 2020

Automated equipment health monitoring from streaming multisensor time-series data can be used to enable condition-based maintenance, avoid sudden catastrophic failures, and ensure high operational availability. We note that most complex machinery has a well-documented and readily accessible underlying structure capturing the inter-dependencies between sub-systems or modules. Deep learning models such as those based on recurrent neural networks (RNNs) or convolutional neural networks (CNNs) fail to explicitly leverage this potentially rich source of domain-knowledge into the learning procedure. In this work, we propose to capture the structure of a complex equipment in the form of a graph, and use graph neural networks (GNNs) to model multi-sensor time-series data. Using remaining useful life estimation as an application task, we evaluate the advantage of incorporating the graph structure via GNNs on the publicly available turbofan engine benchmark dataset. We observe that the proposed GNN-based RUL estimation model compares favorably to several strong baselines from literature such as those based on RNNs and CNNs. Additionally, we observe that the learned network is able to focus on the module (node) with impending failure through a simple attention mechanism, potentially paving the way for actionable diagnosis.