OpenRaiser/Intern-Atlas huggingface dataset:
Multiplierless and Sparse Machine Learning based on Margin Propagation Networks
Venue: arXiv cs.LG 2020
Year: 2020

The new generation of machine learning processors have evolved from multi-core and parallel architectures that were designed to efficiently implement matrix-vector-multiplications (MVMs). This is because at the fundamental level, neural network and machine learning operations extensively use MVM operations and hardware compilers exploit the inherent parallelism in MVM operations to achieve hardware acceleration on GPUs and FPGAs. However, many IoT and edge computing platforms require embedded ML devices close to the network in order to compensate for communication cost and latency. Hence a natural question to ask is whether MVM operations are even necessary to implement ML algorithms and whether simpler hardware primitives can be used to implement an ultra-energy-efficient ML processor/architecture. In this paper we propose an alternate hardware-software codesign of ML and neural network architectures where instead of using MVM operations and non-linear activation functions, the architecture only uses simple addition and thresholding operations to implement inference and learning. At the core of the proposed approach is margin-propagation (MP) based computation that maps multiplications into additions and additions into a dynamic rectifying-linear-unit (ReLU) operations. This mapping results in significant improvement in computational and hence energy cost. In this paper, we show how the MP network formulation can be applied for designing linear classifiers, shallow multi-layer perceptrons and support vector networks suitable fot IoT platforms and tiny ML applications. We show that these MP based classifiers give comparable results to that of their traditional counterparts for benchmark UCI datasets, with the added advantage of reduction in computational complexity enabling an improvement in energy efficiency.
Covariance-free Partial Least Squares: An Incremental Dimensionality Reduction Method
Venue: arXiv cs.CV 2020
Year: 2020

Dimensionality reduction plays an important role in computer vision problems since it reduces computational cost and is often capable of yielding more discriminative data representation. In this context, Partial Least Squares (PLS) has presented notable results in tasks such as image classification and neural network optimization. However, PLS is infeasible on large datasets, such as ImageNet, because it requires all the data to be in memory in advance, which is often impractical due to hardware limitations. Additionally, this requirement prevents us from employing PLS on streaming applications where the data are being continuously generated. Motivated by this, we propose a novel incremental PLS, named Covariance-free Incremental Partial Least Squares (CIPLS), which learns a low-dimensional representation of the data using a single sample at a time. In contrast to other state-of-the-art approaches, instead of adopting a partially-discriminative or SGD-based model, we extend Nonlinear Iterative Partial Least Squares (NIPALS) -- the standard algorithm used to compute PLS -- for incremental processing. Among the advantages of this approach are the preservation of discriminative information across all components, the possibility of employing its score matrices for feature selection, and its computational efficiency. We validate CIPLS on face verification and image classification tasks, where it outperforms several other incremental dimensionality reduction techniques. In the context of feature selection, CIPLS achieves comparable results when compared to state-of-the-art techniques.
Empirical Coordination with Multiple Descriptions
Venue: arXiv cs.IT 2020
Year: 2020

We extend the framework of empirical coordination to a distributed setup where for a given action by nature, multiple descriptions of the action of the decoder are available. We adopt the coding strategy applied by El Gamal and Cover in \cite{gamal:1982} to get a lower bound of the coordination region. Then, we improve this region by applying the coding scheme applied by Zhang and Berger in \cite{zhang:1987}.
Fast Detection of Outliers in Data Streams with the $Q_n$ Estimator
Venue: arXiv cs.DS 2020
Year: 2020

We present FQN (Fast $Q_n$), a novel algorithm for fast detection of outliers in data streams. The algorithm works in the sliding window model, checking if an item is an outlier by cleverly computing the $Q_n$ scale estimator in the current window. We thoroughly compare our algorithm for online $Q_n$ with the state of the art competing algorithm by Nunkesser et al, and show that FQN (i) is faster, (ii) its computational complexity does not depend on the input distribution and (iii) it requires less space. Extensive experimental results on synthetic datasets confirm the validity of our approach.
Differential Privacy-enabled Federated Learning for Sensitive Health Data
Venue: arXiv cs.LG 2020
Year: 2020

Leveraging real-world health data for machine learning tasks requires addressing many practical challenges, such as distributed data silos, privacy concerns with creating a centralized database from person-specific sensitive data, resource constraints for transferring and integrating data from multiple sites, and risk of a single point of failure. In this paper, we introduce a federated learning framework that can learn a global model from distributed health data held locally at different sites. The framework offers two levels of privacy protection. First, it does not move or share raw data across sites or with a centralized server during the model training process. Second, it uses a differential privacy mechanism to further protect the model from potential privacy attacks. We perform a comprehensive evaluation of our approach on two healthcare applications, using real-world electronic health data of 1 million patients. We demonstrate the feasibility and effectiveness of the federated learning framework in offering an elevated level of privacy and maintaining utility of the global model.
Deep Evidential Regression
Venue: arXiv cs.LG 2020
Year: 2020

Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order to learn both aleatoric and epistemic uncertainty. We accomplish this by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution. We additionally impose priors during training such that the model is regularized when its predicted evidence is not aligned with the correct output. Our method does not rely on sampling during inference or on out-of-distribution (OOD) examples for training, thus enabling efficient and scalable uncertainty learning. We demonstrate learning well-calibrated measures of uncertainty on various benchmarks, scaling to complex computer vision tasks, as well as robustness to adversarial and OOD test samples.
Controllable Sentence Simplification
Venue: arXiv cs.CL 2020
Year: 2020

Text simplification aims at making a text easier to read and understand by simplifying grammar and structure while keeping the underlying information identical. It is often considered an all-purpose generic task where the same simplification is suitable for all; however multiple audiences can benefit from simplified text in different ways. We adapt a discrete parametrization mechanism that provides explicit control on simplification systems based on Sequence-to-Sequence models. As a result, users can condition the simplifications returned by a model on attributes such as length, amount of paraphrasing, lexical complexity and syntactic complexity. We also show that carefully chosen values of these attributes allow out-of-the-box Sequence-to-Sequence models to outperform their standard counterparts on simplification benchmarks. Our model, which we call ACCESS (as shorthand for AudienCe-CEntric Sentence Simplification), establishes the state of the art at 41.87 SARI on the WikiLarge test set, a +1.42 improvement over the best previously reported score.
Semi-Analytical Model for Design and Analysis of On-Orbit Servicing Architecture
Venue: arXiv cs.PF 2020
Year: 2020

Robotic on-orbit servicing (OOS) is expected to be a key technology and concept for future sustainable space exploration. This paper develops a semi-analytical model for OOS systems analysis, responding to the growing needs and ongoing trend of robotic OOS. An OOS infrastructure system is considered whose goal is to provide responsive services to the random failures of a set of customer modular satellites distributed in space (e.g., at the geosynchronous equatorial orbit). The considered OOS architecture is comprised of a servicer that travels and provides module-replacement services to the customer satellites, an on-orbit depot to store the spares, and a series of launch vehicles to replenish the depot. The OOS system performance is analyzed by evaluating the mean waiting time before service completion for a given failure and its relationship with the depot capacity. Leveraging the queueing theory and inventory management methods, the developed semi-analytical model is capable of analyzing the OOS system performance without relying on computationally costly simulations. The effectiveness of the proposed model is demonstrated using a case study compared with simulation results. This paper is expected to provide a critical step to push the research frontier of analytical/semi-analytical models development for complex space systems design.
MIM: Mutual Information Machine
Venue: arXiv cs.LG 2020
Year: 2020

We introduce the Mutual Information Machine (MIM), a probabilistic auto-encoder for learning joint distributions over observations and latent variables. MIM reflects three design principles: 1) low divergence, to encourage the encoder and decoder to learn consistent factorizations of the same underlying distribution; 2) high mutual information, to encourage an informative relation between data and latent variables; and 3) low marginal entropy, or compression, which tends to encourage clustered latent representations. We show that a combination of the Jensen-Shannon divergence and the joint entropy of the encoding and decoding distributions satisfies these criteria, and admits a tractable cross-entropy bound that can be optimized directly with Monte Carlo and stochastic gradient descent. We contrast MIM learning with maximum likelihood and VAEs. Experiments show that MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and data log likelihood comparable to VAE, while avoiding posterior collapse.
Multilevel Modelling and Domain-Specific Languages
Venue: arXiv cs.SE 2020
Year: 2020

Modern software engineering deals with demanding problems that yield large and complex software. The area of Model-Driven Software Engineering tackles this issue by using models during the development process, but it does not address some of the communication problems among different stakeholders. Domain-Specific Modelling Languages (DSML) aim at involving domain experts with non-technical profiles in that process. DSMLs define concepts with different levels of abstraction, but traditional modelling does not allow enough flexibility to organise them adequately. Multilevel Modelling (MLM) approaches provide an unbounded number of levels of abstraction, plus other features that perfectly fit DSMLs. Their development can also benefit from Model Transformations (MT), especially when these encode the behaviour of DSMLs. MTs can be exploited by MLM, becoming a precise and reusable definition of behaviour. This thesis presents a MLM and Multilevel MT approach which tackles open issues in the field and compares it with the state of the art through literature review and experiments, providing its formalisation and its implementation in the tool MultEcore, together with case studies.
Greedy Convex Ensemble
Venue: arXiv cs.LG 2020
Year: 2020

We consider learning a convex combination of basis models, and present some new theoretical and empirical results that demonstrate the effectiveness of a greedy approach. Theoretically, we first consider whether we can use linear, instead of convex, combinations, and obtain generalization results similar to existing ones for learning from a convex hull. We obtain a negative result that even the linear hull of very simple basis functions can have unbounded capacity, and is thus prone to overfitting; on the other hand, convex hulls are still rich but have bounded capacities. Secondly, we obtain a generalization bound for a general class of Lipschitz loss functions. Empirically, we first discuss how a convex combination can be greedily learned with early stopping, and how a convex combination can be non-greedily learned when the number of basis models is known a priori. Our experiments suggest that the greedy scheme is competitive with or better than several baselines, including boosting and random forests. The greedy algorithm requires little effort in hyper-parameter tuning, and also seems able to adapt to the underlying complexity of the problem. Our code is available at https://github.com/tan1889/gce.
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Venue: arXiv cs.CL 2020
Year: 2020

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{https://github.com/huggingface/transformers}.
Entanglement-Enabled Communication
Venue: arXiv cs.IT 2020
Year: 2020

We introduce and analyse a multiple-access channel with two senders and one receiver, in the presence of i.i.d. noise coming from the environment. Partial side information about the environmental states allows the senders to modulate their signals accordingly. An adversarial jammer with its own access to information on environmental states and the modulation signals can jam a fraction of the transmissions. Our results show that for many choices of the system parameters, entanglement shared between the two senders allows them to communicate at non-zero rates with the receiver, while for the same parameters the system forbids any communication without entanglement-assistance, even if the senders have access to common randomness (local correlations). A simplified model displaying a similar behaviour but with a compound channel instead of a jammer is outlined to introduce basic aspects of the modeling. We complement these results by demonstrating that there even exist model parameters for which entanglement-assisted communication is no longer possible, but a hypothetical use of nonlocal no-signalling correlations between Alice and Bob could enable them to communicate to Charlie again.
Interpreting Deep Learning-Based Networking Systems
Venue: arXiv cs.NI 2020
Year: 2020

While many deep learning (DL)-based networking systems have demonstrated superior performance, the underlying Deep Neural Networks (DNNs) remain blackboxes and stay uninterpretable for network operators. The lack of interpretability makes DL-based networking systems prohibitive to deploy in practice. In this paper, we propose Metis, a framework that provides interpretability for two general categories of networking problems spanning local and global control. Accordingly, Metis introduces two different interpretation methods based on decision tree and hypergraph, where it converts DNN policies to interpretable rule-based controllers and highlight critical components based on analysis over hypergraph. We evaluate Metis over several state-of-the-art DL-based networking systems and show that Metis provides human-readable interpretations while preserving nearly no degradation in performance. We further present four concrete use cases of Metis, showcasing how Metis helps network operators to design, debug, deploy, and ad-hoc adjust DL-based networking systems.
BrokenStrokes: On the (in)Security of Wireless Keyboards
Venue: arXiv cs.CR 2020
Year: 2020

Wireless devices resorting to event-triggered communications have been proved to suffer critical privacy issues, due to the intrinsic leakage associated with radio-frequency (RF) emissions. In this paper, we move the attack frontier forward by proposing BrokenStrokes: an inexpensive, easy to implement, efficient, and effective attack able to detect the typing of a pre-defined keyword by only eavesdropping the communication channel used by the wireless keyboard. BrokenStrokes proves itself to be a particularly dreadful attack: it achieves its goal when the eavesdropping antenna is up to 15 meters from the target keyboard, regardless of the encryption scheme, the communication protocol, the presence of radio noise, and the presence of physical obstacles. While we detail the attack in three current scenarios and discuss its striking performance--its success probability exceeds 90% in normal operating conditions--, we also provide some suggestions on how to mitigate it. The data utilized in this paper have been released as open-source to allow practitioners, industries, and academia to verify our claims and use them as a basis for further developments.
Improved Res2Net model for Person re-identification
Venue: arXiv cs.CV 2020
Year: 2020

Person re-identification has become a very popular research topic in the computer vision community owing to its numerous applications and growing importance in visual surveillance. Person re-identification remains challenging due to occlusion, illumination and significant intra-class variations across different cameras. In this paper, we propose a multi-task network base on an improved Res2Net model that simultaneously computes the identification loss and verification loss of two pedestrian images. Given a pair of pedestrian images, the system predicts the identities of the two input images and whether they belong to the same identity. In order to obtain deeper feature information of pedestrians, we propose to use the latest Res2Net model for feature extraction of each input image. Experiments on several large-scale person re-identification benchmark datasets demonstrate the accuracy of our approach. For example, rank-1 accuracies are 83.18% (+1.38) and 93.14% (+0.84) for the DukeMTMC and Market-1501 datasets, respectively. The proposed method shows encouraging improvements compared with state-of-the-art methods.
BIAS: Transparent reporting of biomedical image analysis challenges
Venue: arXiv cs.CV 2020
Year: 2020

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical I mage Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.
Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methods
Venue: arXiv cs.CV 2020
Year: 2020

Recent approaches for predicting layouts from 360 panoramas produce excellent results. These approaches build on a common framework consisting of three steps: a pre-processing step based on edge-based alignment, prediction of layout elements, and a post-processing step by fitting a 3D layout to the layout elements. Until now, it has been difficult to compare the methods due to multiple different design decisions, such as the encoding network (e.g. SegNet or ResNet), type of elements predicted (e.g. corners, wall/floor boundaries, or semantic segmentation), or method of fitting the 3D layout. To address this challenge, we summarize and describe the common framework, the variants, and the impact of the design decisions. For a complete evaluation, we also propose extended annotations for the Matterport3D dataset [3], and introduce two depth-based evaluation metrics.
Deciding Differential Privacy for Programs with Finite Inputs and Outputs
Venue: arXiv cs.CR 2020
Year: 2020

Differential privacy is a de facto standard for statistical computations over databases that contain private data. The strength of differential privacy lies in a rigorous mathematical definition that guarantees individual privacy and yet allows for accurate statistical results. Thanks to its mathematical definition, differential privacy is also a natural target for formal analysis. A broad line of work uses logical methods for proving privacy. However, these methods are not complete, and only partially automated. A recent and complementary line of work uses statistical methods for finding privacy violations. However, the methods only provide statistical guarantees (but no proofs). We propose the first decision procedure for checking the differential privacy of a non-trivial class of probabilistic computations. Our procedure takes as input a program P parametrized by a privacy budget $ε$, and either proves differential privacy for all possible values of $ε$ or generates a counterexample. In addition, our procedure applies both to $ε$-differential privacy and $(ε,δ)$-differential privacy. Technically, the decision procedure is based on a novel and judicious encoding of the semantics of programs in our class into a decidable fragment of the first-order theory of the reals with exponentiation. We implement our procedure and use it for (dis)proving privacy bounds for many well-known examples, including randomized response, histogram, report noisy max and sparse vector.
Explaining image classifiers by removing input features using generative models
Venue: arXiv cs.LG 2020
Year: 2020

Perturbation-based explanation methods often measure the contribution of an input feature to an image classifier's outputs by heuristically removing it via e.g. blurring, adding noise, or graying out, which often produce unrealistic, out-of-samples. Instead, we propose to integrate a generative inpainter into three representative attribution methods to remove an input feature. Our proposed change improved all three methods in (1) generating more plausible counterfactual samples under the true data distribution; (2) being more accurate according to three metrics: object localization, deletion, and saliency metrics; and (3) being more robust to hyperparameter changes. Our findings were consistent across both ImageNet and Places365 datasets and two different pairs of classifiers and inpainters.