OpenRaiser/Intern-Atlas huggingface dataset:
Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings
Venue: arXiv cs.CL 2019
Year: 2019

In this paper, we formulate keyphrase extraction from scholarly articles as a sequence labeling task solved using a BiLSTM-CRF, where the words in the input text are represented using deep contextualized embeddings. We evaluate the proposed architecture using both contextualized and fixed word embedding models on three different benchmark datasets (Inspec, SemEval 2010, SemEval 2017) and compare with existing popular unsupervised and supervised techniques. Our results quantify the benefits of (a) using contextualized embeddings (e.g. BERT) over fixed word embeddings (e.g. Glove); (b) using a BiLSTM-CRF architecture with contextualized word embeddings over fine-tuning the contextualized word embedding model directly, and (c) using genre-specific contextualized embeddings (SciBERT). Through error analysis, we also provide some insights into why particular models work better than others. Lastly, we present a case study where we analyze different self-attention layers of the two best models (BERT and SciBERT) to better understand the predictions made by each for the task of keyphrase extraction.
Machine Learning for AC Optimal Power Flow
Venue: arXiv cs.LG 2019
Year: 2019

We explore machine learning methods for AC Optimal Powerflow (ACOPF) - the task of optimizing power generation in a transmission network according while respecting physical and engineering constraints. We present two formulations of ACOPF as a machine learning problem: 1) an end-to-end prediction task where we directly predict the optimal generator settings, and 2) a constraint prediction task where we predict the set of active constraints in the optimal solution. We validate these approaches on two benchmark grids.
LSTM-Assisted Evolutionary Self-Expressive Subspace Clustering
Venue: arXiv cs.LG 2019
Year: 2019

Massive volumes of high-dimensional data that evolves over time is continuously collected by contemporary information processing systems, which brings up the problem of organizing this data into clusters, i.e. achieve the purpose of dimensional deduction, and meanwhile learning its temporal evolution patterns. In this paper, a framework for evolutionary subspace clustering, referred to as LSTM-ESCM, is introduced, which aims at clustering a set of evolving high-dimensional data points that lie in a union of low-dimensional evolving subspaces. In order to obtain the parsimonious data representation at each time step, we propose to exploit the so-called self-expressive trait of the data at each time point. At the same time, LSTM networks are implemented to extract the inherited temporal patterns behind data in an overall time frame. An efficient algorithm has been proposed based on MATLAB. Next, experiments are carried out on real-world datasets to demonstrate the effectiveness of our proposed approach. And the results show that the suggested algorithm dramatically outperforms other known similar approaches in terms of both run time and accuracy.
Downlink Performance of Dense Antenna Deployment: To Distribute or Concentrate?
Venue: arXiv cs.NI 2019
Year: 2019

Massive multiple-input multiple-output (massive MIMO) and small cell densification are complementary key 5G enablers. Given a fixed number of the entire base-station antennas per unit area, this paper fairly compares (i) to deploy few base stations (BSs) and concentrate many antennas on each of them, i.e. massive MIMO, and (ii) to deploy more BSs equipped with few antennas, i.e. small cell densification. We observe that small cell densification always outperforms for both signal-to-interference ratio (SIR) coverage and energy efficiency (EE), when each BS serves multiple users via L number of sub-bands (multi-carrier transmission). Moreover, we also observe that larger L increases SIR coverage while decreasing EE, thus urging the necessity of optimal 5G network design. These two observations are based on our novel closed-form SIR coverage probability derivation using stochastic geometry, also validated via numerical simulations.
On Self-Orthogonality and Self-Duality of Matrix-Product Codes over Commutative Rings
Venue: arXiv cs.IT 2019
Year: 2019

Let $R$ be a commutative ring with identity. The paper studies the problem of self-orthogonality and self-duality matrix-product codes (MPCs) over $R$. Some methods as well as special matrices are introduced for the construction of such MPCs. A characterization of such codes (in a special case) is also given. Some concrete examples are presented throughout the paper.
Visually Exploring Software Maintenance Activities
Venue: arXiv cs.SE 2019
Year: 2019

Lehman's Laws teach us that a software system will become progressively less satisfying to its users over time, unless it is continually adapted to meet new needs. A line of previous works sought to better understand software maintenance by studying how commits can be classified into three main software maintenance activities. Corrective: fault fixing; Perfective: system improvements; Adaptive: new feature introduction. In this work we suggest visualizations for exploring software maintenance activities in both project and individual developer scopes. We demonstrate our approach using a prototype we have built using the Shiny R framework. In addition, we have also published our prototype as an online demo. This demo allows users to explore the maintenance activities of a number of popular open source projects. We believe that the visualizations we provide can assist practitioners in monitoring and maintaining the health of software projects. In particular, they can be useful for identifying general imbalances, peaks, deeps and other anomalies in projects' and developers' maintenance activities.
Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network
Venue: arXiv cs.CV 2019
Year: 2019

User Interface (UI) prototyping is a necessary step in the early stages of application development. Transforming sketches of a Graphical User Interface (UI) into a coded UI application is an uninspired but time-consuming task performed by a UI designer. An automated system that can replace human efforts for straightforward implementation of UI designs will greatly speed up this procedure. The works that propose such a system primarily focus on using UI wireframes as input rather than hand-drawn sketches. In this paper, we put forward a novel approach wherein we employ a Deep Neural Network that is trained on our custom database of such sketches to detect UI elements in the input sketch. Detection of objects in sketches is a peculiar visual recognition task that requires a specific solution that our deep neural network model attempts to provide. The output from the network is a platform-independent UI representation object. The UI representation object is a dictionary of key-value pairs to represent the UI elements recognized along with their properties. This is further consumed by our UI parser which creates code for different platforms. The intrinsic platform-independence allows the model to create a UI prototype for multiple platforms with single training. This two-step approach without the need for two trained models improves over other methods giving time-efficient results (average time: 129 ms) with good accuracy.
Online Bagging for Anytime Transfer Learning
Venue: arXiv cs.LG 2019
Year: 2019

Transfer learning techniques have been widely used in the reality that it is difficult to obtain sufficient labeled data in the target domain, but a large amount of auxiliary data can be obtained in the relevant source domain. But most of the existing methods are based on offline data. In practical applications, it is often necessary to face online learning problems in which the data samples are achieved sequentially. In this paper, We are committed to applying the ensemble approach to solving the problem of online transfer learning so that it can be used in anytime setting. More specifically, we propose a novel online transfer learning framework, which applies the idea of online bagging methods to anytime transfer learning problems, and constructs strong classifiers through online iterations of the usefulness of multiple weak classifiers. Further, our algorithm also provides two extension schemes to reduce the impact of negative transfer. Experiments on three real data sets show that the effectiveness of our proposed algorithms.
Towards Further Understanding of Sparse Filtering via Information Bottleneck
Venue: arXiv cs.LG 2019
Year: 2019

In this paper we examine a formalization of feature distribution learning (FDL) in information-theoretic terms relying on the analytical approach and on the tools already used in the study of the information bottleneck (IB). It has been conjectured that the behavior of FDL algorithms could be expressed as an optimization problem over two information-theoretic quantities: the mutual information of the data with the learned representations and the entropy of the learned distribution. In particular, such a formulation was offered in order to explain the success of the most prominent FDL algorithm, sparse filtering (SF). This conjecture was, however, left unproven. In this work, we aim at providing preliminary empirical support to this conjecture by performing experiments reminiscent of the work done on deep neural networks in the context of the IB research. Specifically, we borrow the idea of using information planes to analyze the behavior of the SF algorithm and gain insights on its dynamics. A confirmation of the conjecture about the dynamics of FDL may provide solid ground to develop information-theoretic tools to assess the quality of the learning process in FDL, and it may be extended to other unsupervised learning algorithms.
Worst-Case Polylog Incremental SPQR-trees: Embeddings, Planarity, and Triconnectivity
Venue: arXiv cs.DS 2019
Year: 2019

We show that every labelled planar graph $G$ can be assigned a canonical embedding $ϕ(G)$, such that for any planar $G'$ that differs from $G$ by the insertion or deletion of one edge, the number of local changes to the combinatorial embedding needed to get from $ϕ(G)$ to $ϕ(G')$ is $O(\log n)$. In contrast, there exist embedded graphs where $Ω(n)$ changes are necessary to accommodate one inserted edge. We provide a matching lower bound of $Ω(\log n)$ local changes, and although our upper bound is worst-case, our lower bound hold in the amortized case as well. Our proof is based on BC trees and SPQR trees, and we develop \emph{pre-split} variants of these for general graphs, based on a novel biased heavy-path decomposition, where the structural changes corresponding to edge insertions and deletions in the underlying graph consist of at most $O(\log n)$ basic operations of a particularly simple form. As a secondary result, we show how to maintain the pre-split trees under edge insertions in the underlying graph deterministically in worst case $O(\log^3 n)$ time. Using this, we obtain deterministic data structures for incremental planarity testing, incremental planar embedding, and incremental triconnectivity, that each have worst case $O(\log^3 n)$ update and query time, answering an open question by La Poutré and Westbrook from 1998.
Multi-User MABs with User Dependent Rewards for Uncoordinated Spectrum Access
Venue: arXiv cs.IT 2019
Year: 2019

Multi-user multi-armed bandits have emerged as a good model for uncoordinated spectrum access problems. In this paper we consider the scenario where users cannot communicate with each other. In addition, the environment may appear differently to different users, ${i.e.}$, the mean rewards as observed by different users for the same channel may be different. With this setup, we present a policy that achieves a regret of $O (\log{T})$. This paper has been accepted at Asilomar Conference on Signals, Systems, and Computers 2019.
Perception-Distortion Trade-off with Restricted Boltzmann Machines
Venue: arXiv cs.LG 2019
Year: 2019

In this work, we introduce a new procedure for applying Restricted Boltzmann Machines (RBMs) to missing data inference tasks, based on linearization of the effective energy function governing the distribution of observations. We compare the performance of our proposed procedure with those obtained using existing reconstruction procedures trained on incomplete data. We place these performance comparisons within the context of the perception-distortion trade-off observed in other data reconstruction tasks, which has, until now, remained unexplored in tasks relying on incomplete training data.
Generating Mock Skeletons for Lightweight Web-Service Testing
Venue: arXiv cs.SE 2019
Year: 2019

Modern application development allows applications to be composed using lightweight HTTP services. Testing such an application requires the availability of services that the application makes requests to. However, access to dependent services during testing may be restrained. Simulating the behaviour of such services is, therefore, useful to address their absence and move on application testing. This paper examines the appropriateness of Symbolic Machine Learning algorithms to automatically synthesise HTTP services' mock skeletons from network traffic recordings. These skeletons can then be customised to create mocks that can generate service responses suitable for testing. The mock skeletons have human-readable logic for key aspects of service responses, such as headers and status codes, and are highly accurate.
Constructing Privacy Channels from Information Channels
Venue: arXiv cs.CR 2019
Year: 2019

Data privacy protection studies how to query a dataset while preserving the privacy of individuals whose sensitive information is contained in the dataset. The information privacy model protects the privacy of an individual by using a noisy channel, called privacy channel, to filter out most information of the individual from the query's output. This paper studies how to construct privacy channels, which is challenging since it needs to evaluate the maximal amount of disclosed information of each individual contained in the query's output, called individual channel capacity. Our main contribution is an interesting result which can transform the problem of evaluating a privacy channel's individual channel capacity, which equals the problem of evaluating the capacities of an infinite number of channels, into the problem of evaluating the capacities of a finite number of channels. This result gives us a way to utilize the results in the information theory to construct privacy channels. As some examples, it is used to construct several basic privacy channels, such as the random response privacy channel, the exponential privacy channel and the Gaussian privacy channel, which are respective counterparts of the random response mechanism, the exponential mechanism and the Gaussian mechanism of differential privacy.
Recovering Localized Adversarial Attacks
Venue: arXiv cs.LG 2019
Year: 2019

Deep convolutional neural networks have achieved great successes over recent years, particularly in the domain of computer vision. They are fast, convenient, and -- thanks to mature frameworks -- relatively easy to implement and deploy. However, their reasoning is hidden inside a black box, in spite of a number of proposed approaches that try to provide human-understandable explanations for the predictions of neural networks. It is still a matter of debate which of these explainers are best suited for which situations, and how to quantitatively evaluate and compare them. In this contribution, we focus on the capabilities of explainers for convolutional deep neural networks in an extreme situation: a setting in which humans and networks fundamentally disagree. Deep neural networks are susceptible to adversarial attacks that deliberately modify input samples to mislead a neural network's classification, without affecting how a human observer interprets the input. Our goal with this contribution is to evaluate explainers by investigating whether they can identify adversarially attacked regions of an image. In particular, we quantitatively and qualitatively investigate the capability of three popular explainers of classifications -- classic salience, guided backpropagation, and LIME -- with respect to their ability to identify regions of attack as the explanatory regions for the (incorrect) prediction in representative examples from image classification. We find that LIME outperforms the other explainers.
Batch Face Alignment using a Low-rank GAN
Venue: arXiv cs.CV 2019
Year: 2019

This paper studies the problem of aligning a set of face images of the same individual into a normalized image while removing the outliers like partial occlusion, extreme facial expression as well as significant illumination variation. Our model seeks an optimal image domain transformation such that the matrix of misaligned images can be decomposed as the sum of a sparse matrix of noise and a rank-one matrix of aligned images. The image transformation is learned in an unsupervised manner, which means that ground-truth aligned images are unnecessary for our model. Specifically, we make use of the remarkable non-linear transforming ability of generative adversarial network(GAN) and guide it with low-rank generation as well as sparse noise constraint to achieve the face alignment. We verify the efficacy of the proposed model with extensive experiments on real-world face databases, demonstrating higher accuracy and efficiency than existing methods.
A Note on a Unifying Proof of the Undecidability of Several Diagrammatic Properties of Term Rewriting Systems
Venue: arXiv cs.LO 2019
Year: 2019

In this note we give a simple unifying proof of the undecidability of several diagrammatic properties of term rewriting systems that include: local confluence, strong confluence, diamond property, subcommutative property, and the existence of successor. The idea is to code configurations of Turing Machines into terms, and then define a suitable relation on those terms such that the termination of the Turing Machine becomes equivalent to the satisfiability of the diagrammatic property.
Multi-Band Multi-Resolution Fully Convolutional Neural Networks for Singing Voice Separation
Venue: arXiv cs.SD 2019
Year: 2019

Deep neural networks with convolutional layers usually process the entire spectrogram of an audio signal with the same time-frequency resolutions, number of filters, and dimensionality reduction scale. According to the constant-Q transform, good features can be extracted from audio signals if the low frequency bands are processed with high frequency resolution filters and the high frequency bands with high time resolution filters. In the spectrogram of a mixture of singing voices and music signals, there is usually more information about the voice in the low frequency bands than the high frequency bands. These raise the need for processing each part of the spectrogram differently. In this paper, we propose a multi-band multi-resolution fully convolutional neural network (MBR-FCN) for singing voice separation. The MBR-FCN processes the frequency bands that have more information about the target signals with more filters and smaller dimentionality reduction scale than the bands with less information. Furthermore, the MBR-FCN processes the low frequency bands with high frequency resolution filters and the high frequency bands with high time resolution filters. Our experimental results show that the proposed MBR-FCN with very few parameters achieves better singing voice separation performance than other deep neural networks.
A mathematical model of asynchronous data flow in parallel computers
Venue: arXiv cs.DC 2019
Year: 2019

We present a simplified model of data flow on processors in a high performance computing framework involving computations necessitating inter-processor communications. From this ordinary differential model, we take its asymptotic limit, resulting in a model which treats the computer as a continuum of processors and data flow as an Eulerian fluid governed by a conservation law. We derive a Hamilton-Jacobi equation associated with this conservation law for which the existence and uniqueness of solutions can be proven. We then present the results of numerical experiments for both discrete and continuum models; these show a qualitative agreement between the two and the effect of variations in the computing environment's processing capabilities on the progress of the modeled computation.
The Lean mathematical library
Venue: arXiv cs.LO 2019
Year: 2019

This paper describes mathlib, a community-driven effort to build a unified library of mathematics formalized in the Lean proof assistant. Among proof assistant libraries, it is distinguished by its dependently typed foundations, focus on classical mathematics, extensive hierarchy of structures, use of large- and small-scale automation, and distributed organization. We explain the architecture and design decisions of the library and the social organization that has led us here.
Gradient Boosted Decision Tree Neural Network
Venue: arXiv cs.LG 2019
Year: 2019

In this paper we propose a method to build a neural network that is similar to an ensemble of decision trees. We first illustrate how to convert a learned ensemble of decision trees to a single neural network with one hidden layer and an input transformation. We then relax some properties of this network such as thresholds and activation functions to train an approximately equivalent decision tree ensemble. The final model, Hammock, is surprisingly simple: a fully connected two layers neural network where the input is quantized and one-hot encoded. Experiments on large and small datasets show this simple method can achieve performance similar to that of Gradient Boosted Decision Trees.
Approximate Sampling using an Accelerated Metropolis-Hastings based on Bayesian Optimization and Gaussian Processes
Venue: arXiv cs.LG 2019
Year: 2019

Markov Chain Monte Carlo (MCMC) methods have a drawback when working with a target distribution or likelihood function that is computationally expensive to evaluate, specially when working with big data. This paper focuses on Metropolis-Hastings (MH) algorithm for unimodal distributions. Here, an enhanced MH algorithm is proposed that requires less number of expensive function evaluations, has shorter burn-in period, and uses a better proposal distribution. The main innovations include the use of Bayesian optimization to reach the high probability region quickly, emulating the target distribution using Gaussian processes (GP), and using Laplace approximation of the GP to build a proposal distribution that captures the underlying correlation better. The experiments show significant improvement over the regular MH. Statistical comparison between the results from two algorithms is presented.
Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling
Venue: arXiv cs.CL 2019
Year: 2019

Word2Vec is the most popular model for word representation and has been widely investigated in literature. However, its noise distribution for negative sampling is decided by empirical trials and the optimality has always been ignored. We suggest that the distribution is a sub-optimal choice, and propose to use a sub-sampled unigram distribution for better negative sampling. Our contributions include: (1) proposing the concept of semantics quantification and deriving a suitable sub-sampling rate for the proposed distribution adaptive to different training corpora; (2) demonstrating the advantages of our approach in both negative sampling and noise contrastive estimation by extensive evaluation tasks; and (3) proposing a semantics weighted model for the MSR sentence completion task, resulting in considerable improvements. Our work not only improves the quality of word vectors but also benefits current understanding of Word2Vec.