OpenRaiser/Intern-Atlas huggingface dataset:
Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
Venue: arXiv cs.CL 2020
Year: 2020

Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.
Intermediate problems in modular circuits satisfiability
Venue: arXiv cs.CC 2020
Year: 2020

In arXiv:1710.08163 a generalization of Boolean circuits to arbitrary finite algebras had been introduced and applied to sketch P versus NP-complete borderline for circuits satisfiability over algebras from congruence modular varieties. However the problem for nilpotent (which had not been shown to be NP-hard) but not supernilpotent algebras (which had been shown to be polynomial time) remained open. In this paper we provide a broad class of examples, lying in this grey area, and show that, under the Exponential Time Hypothesis and Strong Exponential Size Hypothesis (saying that Boolean circuits need exponentially many modular counting gates to produce boolean conjunctions of any arity), satisfiability over these algebras have intermediate complexity between $Ω(2^{c\log^{h-1} n})$ and $O(2^{c\log^h n})$, where $h$ measures how much a nilpotent algebra fails to be supernilpotent. We also sketch how these examples could be used as paradigms to fill the nilpotent versus supernilpotent gap in general. Our examples are striking in view of the natural strong connections between circuits satisfiability and Constraint Satisfaction Problem for which the dichotomy had been shown by Bulatov and Zhuk.
Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree
Venue: arXiv cs.SE 2020
Year: 2020

Code clones are semantically similar code fragments pairs that are syntactically similar or different. Detection of code clones can help to reduce the cost of software maintenance and prevent bugs. Numerous approaches of detecting code clones have been proposed previously, but most of them focus on detecting syntactic clones and do not work well on semantic clones with different syntactic features. To detect semantic clones, researchers have tried to adopt deep learning for code clone detection to automatically learn latent semantic features from data. Especially, to leverage grammar information, several approaches used abstract syntax trees (AST) as input and achieved significant progress on code clone benchmarks in various programming languages. However, these AST-based approaches still can not fully leverage the structural information of code fragments, especially semantic information such as control flow and data flow. To leverage control and data flow information, in this paper, we build a graph representation of programs called flow-augmented abstract syntax tree (FA-AST). We construct FA-AST by augmenting original ASTs with explicit control and data flow edges. Then we apply two different types of graph neural networks (GNN) on FA-AST to measure the similarity of code pairs. As far as we have concerned, we are the first to apply graph neural networks on the domain of code clone detection. We apply our FA-AST and graph neural networks on two Java datasets: Google Code Jam and BigCloneBench. Our approach outperforms the state-of-the-art approaches on both Google Code Jam and BigCloneBench tasks.
Computationally Tractable Riemannian Manifolds for Graph Embeddings
Venue: arXiv cs.LG 2020
Year: 2020

Representing graphs as sets of node embeddings in certain curved Riemannian manifolds has recently gained momentum in machine learning due to their desirable geometric inductive biases, e.g., hierarchical structures benefit from hyperbolic geometry. However, going beyond embedding spaces of constant sectional curvature, while potentially more representationally powerful, proves to be challenging as one can easily lose the appeal of computationally tractable tools such as geodesic distances or Riemannian gradients. Here, we explore computationally efficient matrix manifolds, showcasing how to learn and optimize graph embeddings in these Riemannian spaces. Empirically, we demonstrate consistent improvements over Euclidean geometry while often outperforming hyperbolic and elliptical embeddings based on various metrics that capture different graph properties. Our results serve as new evidence for the benefits of non-Euclidean embeddings in machine learning pipelines.
Neural Network Compression Framework for fast model inference
Venue: arXiv cs.CV 2020
Year: 2020

In this work we present a new framework for neural networks compression with fine-tuning, which we called Neural Network Compression Framework (NNCF). It leverages recent advances of various network compression methods and implements some of them, such as sparsity, quantization, and binarization. These methods allow getting more hardware-friendly models which can be efficiently run on general-purpose hardware computation units (CPU, GPU) or special Deep Learning accelerators. We show that the developed methods can be successfully applied to a wide range of models to accelerate the inference time while keeping the original accuracy. The framework can be used within the training samples, which are supplied with it, or as a standalone package that can be seamlessly integrated into the existing training code with minimal adaptations. Currently, a PyTorch version of NNCF is available as a part of OpenVINO Training Extensions at https://github.com/openvinotoolkit/nncf.
Weighted Epsilon-Nets
Venue: arXiv cs.CG 2020
Year: 2020

Motivated by recent work of Bukh and Nivasch on one-sided $\varepsilon$-approximants, we introduce the notion of \emph{weighted $\varepsilon$-nets}. It is a geometric notion of approximation for point sets in $\mathbb{R}^d$ similar to $\varepsilon$-nets and $\varepsilon$-approximations, where it is stronger than the former and weaker than the latter. The main idea is that small sets can contain many points, whereas large sets must contain many points of the weighted $\varepsilon$-net. In this paper, we analyze weak weighted $\varepsilon$-nets with respect to convex sets and axis-parallel boxes and give upper and lower bounds on $\varepsilon$ for weighted $\varepsilon$-nets of size two and three. Some of these bounds apply to classical $\varepsilon$-nets as well.
Deep Energy Autoencoder for Noncoherent Multicarrier MU-SIMO Systems
Venue: arXiv cs.IT 2020
Year: 2020

We propose a novel deep energy autoencoder (EA) for noncoherent multicarrier multiuser single-input multipleoutput (MU-SIMO) systems under fading channels. In particular, a single-user noncoherent EA-based (NC-EA) system, based on the multicarrier SIMO framework, is first proposed, where both the transmitter and receiver are represented by deep neural networks (DNNs), known as the encoder and decoder of an EA. Unlike existing systems, the decoder of the NC-EA is fed only with the energy combined from all receive antennas, while its encoder outputs a real-valued vector whose elements stand for the subcarrier power levels. Using the NC-EA, we then develop two novel DNN structures for both uplink and downlink NC-EA multiple access (NC-EAMA) schemes, based on the multicarrier MUSIMO framework. Note that NC-EAMA allows multiple users to share the same sub-carriers, thus enables to achieve higher performance gains than noncoherent orthogonal counterparts. By properly training, the proposed NC-EA and NC-EAMA can efficiently recover the transmitted data without any channel state information estimation. Simulation results clearly show the superiority of our schemes in terms of reliability, flexibility and complexity over baseline schemes.
Object 6D Pose Estimation with Non-local Attention
Venue: arXiv cs.CV 2020
Year: 2020

In this paper, we address the challenging task of estimating 6D object pose from a single RGB image. Motivated by the deep learning based object detection methods, we propose a concise and efficient network that integrate 6D object pose parameter estimation into the object detection framework. Furthermore, for more robust estimation to occlusion, a non-local self-attention module is introduced. The experimental results show that the proposed method reaches the state-of-the-art performance on the YCB-video and the Linemod datasets.
Combining Partial Specifications using Alternating Interface Automata
Venue: arXiv cs.FL 2020
Year: 2020

To model real-world software systems, modelling paradigms should support a form of compositionality. In interface theory and model-based testing with inputs and outputs, conjunctive operators have been introduced: the behaviour allowed by composed specification s1 $\wedge$ s2 is the behaviour allowed by both partial models s1 and s2. The models at hand are non-deterministic interface automata, but the interaction between non-determinism and conjunction is not yet well understood. On the other hand, in the theory of alternating automata, conjunction and non-determinism are core aspects. Alternating automata have not been considered in the context of inputs and outputs, making them less suitable for modelling software interfaces. In this paper, we combine the two modelling paradigms to define alternating interface automata (AIA). We equip these automata with an observational, trace-based semantics, and define testers, to establish correctness of black-box interfaces with respect to an AIA specification.
The Age of Information in a Discrete Time Queue: Stationary Distribution and Non-linear Age Mean Analysis
Venue: arXiv cs.NI 2020
Year: 2020

In this work, we investigate information freshness in a status update communication system consisting of a source-destination link. Initially, we study the properties of a sample path of the age of information (AoI) process at the destination. We obtain a general formula of the stationary distribution of the AoI, under the assumption of ergodicity. We relate this result to a discrete time queueing system and provide a general expression of the generating function of AoI in relation with the system time and the peak age of information (PAoI) metric. Furthermore, we consider three different single-server system models and we obtain closed-form expressions of the generating functions and the stationary distributions of the AoI and the PAoI. The first model is a first-come-first-served (FCFS) queue, the second model is a preemptive last-come-first-served (LCFS) queue, and the last model is a bufferless system with packet dropping. We build upon these results to provide a methodology for analyzing general non-linear age functions for this type of systems, using representations of functions as power series.
Knowledge and simplicial complexes
Venue: arXiv cs.DC 2020
Year: 2020

Simplicial complexes are a versatile and convenient paradigm on which to build all the tools and techniques of the logic of knowledge, on the assumption that initial epistemic models can be described in a distributed fashion. Thus, we can define: knowledge, belief, bisimulation, the group notions of mutual, distributed and common knowledge, and also dynamics in the shape of simplicial action models. We give a survey on how to interpret all such notions on simplicial complexes, building upon the foundations laid in prior work by Goubault and others.
Compositional Neural Machine Translation by Removing the Lexicon from Syntax
Venue: arXiv cs.CL 2020
Year: 2020

The meaning of a natural language utterance is largely determined from its syntax and words. Additionally, there is evidence that humans process an utterance by separating knowledge about the lexicon from syntax knowledge. Theories from semantics and neuroscience claim that complete word meanings are not encoded in the representation of syntax. In this paper, we propose neural units that can enforce this constraint over an LSTM encoder and decoder. We demonstrate that our model achieves competitive performance across a variety of domains including semantic parsing, syntactic parsing, and English to Mandarin Chinese translation. In these cases, our model outperforms the standard LSTM encoder and decoder architecture on many or all of our metrics. To demonstrate that our model achieves the desired separation between the lexicon and syntax, we analyze its weights and explore its behavior when different neural modules are damaged. When damaged, we find that the model displays the knowledge distortions that aphasics are evidenced to have.
Application of Pre-training Models in Named Entity Recognition
Venue: arXiv cs.CL 2020
Year: 2020

Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities from unstructured data. The previous methods for NER were based on machine learning or deep learning. Recently, pre-training models have significantly improved performance on multiple NLP tasks. In this paper, firstly, we introduce the architecture and pre-training tasks of four common pre-training models: BERT, ERNIE, ERNIE2.0-tiny, and RoBERTa. Then, we apply these pre-training models to a NER task by fine-tuning, and compare the effects of the different model architecture and pre-training tasks on the NER task. The experiment results showed that RoBERTa achieved state-of-the-art results on the MSRA-2006 dataset.
Deep Learning-Based Feature Extraction in Iris Recognition: Use Existing Models, Fine-tune or Train From Scratch?
Venue: arXiv cs.CV 2020
Year: 2020

Modern deep learning techniques can be employed to generate effective feature extractors for the task of iris recognition. The question arises: should we train such structures from scratch on a relatively large iris image dataset, or it is better to fine-tune the existing models to adapt them to a new domain? In this work we explore five different sets of weights for the popular ResNet-50 architecture to find out whether iris-specific feature extractors perform better than models trained for non-iris tasks. Features are extracted from each convolutional layer and the classification accuracy achieved by a Support Vector Machine is measured on a dataset that is disjoint from the samples used in training of the ResNet-50 model. We show that the optimal training strategy is to fine-tune an off-the-shelf set of weights to the iris recognition domain. This approach results in greater accuracy than both off-the-shelf weights and a model trained from scratch. The winning, fine-tuned approach also shows an increase in performance when compared to previous work, in which only off-the-shelf (not fine-tuned) models were used in iris feature extraction. We make the best-performing ResNet-50 model, fine-tuned with more than 360,000 iris images, publicly available along with this paper.
Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
Venue: arXiv cs.LG 2020
Year: 2020

Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood. In practice, augmentation policies are often chosen using heuristics of either distribution shift or augmentation diversity. Inspired by these, we seek to quantify how data augmentation improves model generalization. To this end, we introduce interpretable and easy-to-compute measures: Affinity and Diversity. We find that augmentation performance is predicted not by either of these alone but by jointly optimizing the two.
Comparing recurrent and convolutional neural networks for predicting wave propagation
Venue: arXiv cs.LG 2020
Year: 2020

Dynamical systems can be modelled by partial differential equations and numerical computations are used everywhere in science and engineering. In this work, we investigate the performance of recurrent and convolutional deep neural network architectures to predict the surface waves. The system is governed by the Saint-Venant equations. We improve on the long-term prediction over previous methods while keeping the inference time at a fraction of numerical simulations. We also show that convolutional networks perform at least as well as recurrent networks in this task. Finally, we assess the generalisation capability of each network by extrapolating in longer time-frames and in different physical settings.
Total tessellation cover and quantum walk
Venue: arXiv cs.DM 2020
Year: 2020

We propose the total staggered quantum walk model and the total tessellation cover of a graph. This model uses the concept of total tessellation cover to describe the motion of the walker who is allowed to hop both to vertices and edges of the graph, in contrast with previous models in which the walker hops either to vertices or edges. We establish bounds on $T_t(G)$, which is the smallest number of tessellations required in a total tessellation cover of $G$. We highlight two of these lower bounds $T_t(G) \geq ω(G)$ and $T_t(G)\geq is(G)+1$, where $ω(G)$ is the size of a maximum clique and $is(G)$ is the number of edges of a maximum induced star subgraph. Using these bounds, we define the good total tessellable graphs with either $T_t(G)=ω(G)$ or $T_t(G)=is(G)+1$. The $k$-total tessellability problem aims to decide whether a given graph $G$ has $T_t(G) \leq k$. We show that $k$-total tessellability is in $\mathcal{P}$ for good total tessellable graphs. We establish the $\mathcal{NP}$-completeness of the following problems when restricted to the following classes: ($is(G)+1$)-total tessellability for graphs with $ω(G) = 2$; $ω(G)$-total tessellability for graphs $G$ with $is(G)+1 = 3$; $k$-total tessellability for graphs $G$ with $\max\{ω(G), is(G)+1\}$ far from $k$; and $4$-total tessellability for graphs $G$ with $ω(G) = is(G)+1 = 4$. As a consequence, we establish hardness results for bipartite graphs, line graphs of triangle-free graphs, universal graphs, planar graphs, and $(2,1)$-chordal graphs.
How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance
Venue: arXiv cs.SE 2020
Year: 2020

With modern requirements, there is an increasing tendency of considering multiple objectives/criteria simultaneously in many Software Engineering (SE) scenarios. Such a multi-objective optimization scenario comes with an important issue -- how to evaluate the outcome of optimization algorithms, which typically is a set of incomparable solutions (i.e., being Pareto non-dominated to each other). This issue can be challenging for the SE community, particularly for practitioners of Search-Based SE (SBSE). On one hand, multi-objective optimization could still be relatively new to SE/SBSE researchers, who may not be able to identify the right evaluation methods for their problems. On the other hand, simply following the evaluation methods for general multi-objective optimization problems may not be appropriate for specific SE problems, especially when the problem nature or decision maker's preferences are explicitly/implicitly available. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading use of evaluation methods. In this paper, we first carry out a systematic and critical review of quality evaluation for multi-objective optimization in SBSE. We survey 717 papers published between 2009 and 2019 from 36 venues in seven repositories, and select 95 prominent studies, through which we identify five important but overlooked issues in the area. We then conduct an in-depth analysis of quality evaluation indicators/methods and general situations in SBSE, which, together with the identified issues, enables us to codify a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.
Distributed No-Regret Learning in Multi-Agent Systems
Venue: arXiv cs.GT 2020
Year: 2020

In this tutorial article, we give an overview of new challenges and representative results on distributed no-regret learning in multi-agent systems modeled as repeated unknown games. Four emerging game characteristics---dynamicity, incomplete and imperfect feedback, bounded rationality, and heterogeneity---that challenge canonical game models are explored. For each of the four characteristics, we illuminate its implications and ramifications in game modeling, notions of regret, feasible game outcomes, and the design and analysis of distributed learning algorithms.
Unsupervised Pre-trained, Texture Aware And Lightweight Model for Deep Learning-Based Iris Recognition Under Limited Annotated Data
Venue: arXiv cs.CV 2020
Year: 2020

In this paper, we present a texture aware lightweight deep learning framework for iris recognition. Our contributions are primarily three fold. Firstly, to address the dearth of labelled iris data, we propose a reconstruction loss guided unsupervised pre-training stage followed by supervised refinement. This drives the network weights to focus on discriminative iris texture patterns. Next, we propose several texture aware improvisations inside a Convolution Neural Net to better leverage iris textures. Finally, we show that our systematic training and architectural choices enable us to design an efficient framework with upto 100X fewer parameters than contemporary deep learning baselines yet achieve better recognition performance for within and cross dataset evaluations.
Adapted Center and Scale Prediction: More Stable and More Accurate
Venue: arXiv cs.CV 2020
Year: 2020

Pedestrian detection benefits from deep learning technology and gains rapid development in recent years. Most of detectors follow general object detection frame, i.e. default boxes and two-stage process. Recently, anchor-free and one-stage detectors have been introduced into this area. However, their accuracies are unsatisfactory. Therefore, in order to enjoy the simplicity of anchor-free detectors and the accuracy of two-stage ones simultaneously, we propose some adaptations based on a detector, Center and Scale Prediction(CSP). The main contributions of our paper are: (1) We improve the robustness of CSP and make it easier to train. (2) We propose a novel method to predict width, namely compressing width. (3) We achieve the second best performance on CityPersons benchmark, i.e. 9.3% log-average miss rate(MR) on reasonable set, 8.7% MR on partial set and 5.6% MR on bare set, which shows an anchor-free and one-stage detector can still have high accuracy. (4) We explore some capabilities of Switchable Normalization which are not mentioned in its original paper.