OpenRaiser/Intern-Atlas huggingface dataset:
Variational Self-attention Model for Sentence Representation
Venue: arXiv cs.CL 2020
Year: 2020

This paper proposes a variational self-attention model (VSAM) that employs variational inference to derive self-attention. We model the self-attention vector as random variables by imposing a probabilistic distribution. The self-attention mechanism summarizes source information as an attention vector by weighted sum, where the weights are a learned probabilistic distribution. Compared with conventional deterministic counterpart, the stochastic units incorporated by VSAM allow multi-modal attention distributions. Furthermore, by marginalizing over the latent variables, VSAM is more robust against overfitting. Experiments on the stance detection task demonstrate the superiority of our method.
Regularized Binary Network Training
Venue: arXiv cs.LG 2020
Year: 2020

There is a significant performance gap between Binary Neural Networks (BNNs) and floating point Deep Neural Networks (DNNs). We propose to improve the binary training method, by introducing a new regularization function that encourages training weights around binary values. In addition, we add trainable scaling factors to our regularization functions. Additionally, an improved approximation of the derivative of the sign activation function in the backward computation. These modifications are based on linear operations that are easily implementable into the binary training framework. Experimental results on ImageNet shows our method outperforms the traditional BNN method and XNOR-net.
Elimination of All Bad Local Minima in Deep Learning
Venue: arXiv cs.LG 2020
Year: 2020

In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function, under practical assumptions. At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network. The effects of the added neurons are proven to automatically vanish at every local minimum. Moreover, we provide a novel theoretical characterization of a failure mode of eliminating suboptimal local minima via an additional theorem and several examples. This paper also introduces a novel proof technique based on the perturbable gradient basis (PGB) necessary condition of local minima, which provides new insight into the elimination of local minima and is applicable to analyze various models and transformations of objective functions beyond the elimination of local minima.
Plugin Networks for Inference under Partial Evidence
Venue: arXiv cs.CV 2020
Year: 2020

In this paper, we propose a novel method to incorporate partial evidence in the inference of deep convolutional neural networks. Contrary to the existing, top performing methods, which either iteratively modify the input of the network or exploit external label taxonomy to take the partial evidence into account, we add separate network modules ("Plugin Networks") to the intermediate layers of a pre-trained convolutional network. The goal of these modules is to incorporate additional signal, ie information about known labels, into the inference procedure and adjust the predicted output accordingly. Since the attached plugins have a simple structure, consisting of only fully connected layers, we drastically reduced the computational cost of training and inference. At the same time, the proposed architecture allows to propagate information about known labels directly to the intermediate layers to improve the final representation. Extensive evaluation of the proposed method confirms that our Plugin Networks outperform the state-of-the-art in a variety of tasks, including scene categorization, multi-label image annotation, and semantic segmentation.
Optical Fringe Patterns Filtering Based on Multi-Stage Convolution Neural Network
Venue: arXiv cs.CV 2020
Year: 2020

Optical fringe patterns are often contaminated by speckle noise, making it difficult to accurately and robustly extract their phase fields. To deal with this problem, we propose a filtering method based on deep learning, called optical fringe patterns denoising convolutional neural network (FPD-CNN), for directly removing speckle from the input noisy fringe patterns. Regularization technology is integrated into the design of deep architecture. Specifically, the FPD-CNN method is divided into multiple stages, each stage consists of a set of convolutional layers along with batch normalization and leaky rectified linear unit (Leaky ReLU) activation function. The end-to-end joint training is carried out using the Euclidean loss. Extensive experiments on simulated and experimental optical fringe patterns,especially finer ones with high-density regions, show that the proposed method is competitive with some state-of-the-art denoising techniques in spatial or transform domains, efficiently preserving main features of fringe at a fairly fast speed.
From exploration to control: learning object manipulation skills through novelty search and local adaptation
Venue: arXiv cs.RO 2020
Year: 2020

Programming a robot to deal with open-ended tasks remains a challenge, in particular if the robot has to manipulate objects. Launching, grasping, pushing or any other object interaction can be simulated but the corresponding models are not reversible and the robot behavior thus cannot be directly deduced. These behaviors are hard to learn without a demonstration as the search space is large and the reward sparse. We propose a method to autonomously generate a diverse repertoire of simple object interaction behaviors in simulation. Our goal is to bootstrap a robot learning and development process with limited information about what the robot has to achieve and how. This repertoire can be exploited to solve different tasks in reality thanks to a proposed adaptation method or could be used as a training set for data-hungry algorithms. The proposed approach relies on the definition of a goal space and generates a repertoire of trajectories to reach attainable goals, thus allowing the robot to control this goal space. The repertoire is built with an off-the-shelf simulation thanks to a quality diversity algorithm. The result is a set of solutions tested in simulation only. It may result in two different problems: (1) as the repertoire is discrete and finite, it may not contain the trajectory to deal with a given situation or (2) some trajectories may lead to a behavior in reality that differs from simulation because of a reality gap. We propose an approach to deal with both issues by using a local linearization of the mapping between the motion parameters and the observed effects. Furthermore, we present an approach to update the existing solutions repertoire with the tests done on the real robot. The approach has been validated on two different experiments on the Baxter robot: a ball launching and a joystick manipulation tasks.
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
Venue: arXiv cs.DC 2020
Year: 2020

We study federated machine learning (ML) at the wireless edge, where power- and bandwidth-limited wireless devices with local datasets carry out distributed stochastic gradient descent (DSGD) with the help of a remote parameter server (PS). Standard approaches assume separate computation and communication, where local gradient estimates are compressed and transmitted to the PS over orthogonal links. Following this digital approach, we introduce D-DSGD, in which the wireless devices employ gradient quantization and error accumulation, and transmit their gradient estimates to the PS over a multiple access channel (MAC). We then introduce a novel analog scheme, called A-DSGD, which exploits the additive nature of the wireless MAC for over-the-air gradient computation, and provide convergence analysis for this approach. In A-DSGD, the devices first sparsify their gradient estimates, and then project them to a lower dimensional space imposed by the available channel bandwidth. These projections are sent directly over the MAC without employing any digital code. Numerical results show that A-DSGD converges faster than D-DSGD thanks to its more efficient use of the limited bandwidth and the natural alignment of the gradient estimates over the channel. The improvement is particularly compelling at low power and low bandwidth regimes. We also illustrate for a classification problem that, A-DSGD is more robust to bias in data distribution across devices, while D-DSGD significantly outperforms other digital schemes in the literature. We also observe that both D-DSGD and A-DSGD perform better by increasing the number of devices (while keeping the total dataset size constant), showing their ability in harnessing the computation power of edge devices.
Cascade Codes For Distributed Storage Systems
Venue: arXiv cs.IT 2020
Year: 2020

A novel coding scheme for exact repair-regenerating codes is presented in this paper. The codes proposed in this work can trade between the repair bandwidth of nodes (number of downloaded symbols from each surviving node in a repair process) and the required storage overhead of the system. These codes work for general system parameters $(n,k,d)$, which are the total number of nodes, the number of nodes suffice for data recovery, and the number of helper nodes in a repair process, respectively. The proposed construction offers a unified scheme to develop exact-repair regenerating codes for the entire trade-off, including the MBR and MSR points. We conjecture that the new storage-vs.-bandwidth trade-off achieved by the proposed codes is optimum. Some other key features of this code include: the construction is linear; the required field size is only $Θ(n)$; and the code parameters and in particular sub-packetization level is at most $(d-k+1)^k$; which is independent of the number of the parity nodes. Moreover, the proposed repair mechanism is \emph{helper-independent}, that is the data sent from each helper only depends on the identity of the helper and failed nodes, but independent of the identity of other helper nodes participating in the repair process.
Joint Source-Channel Coding for the Transmission of Correlated Sources over Two-Way Channels
Venue: arXiv cs.IT 2020
Year: 2020

A joint source-channel coding (JSCC) scheme based on hybrid digital/analog coding is proposed for the transmission of correlated sources over discrete-memoryless two-way channels (DM-TWCs). The scheme utilizes the correlation between the sources in generating channel inputs, thus enabling the users to coordinate their transmission to combat channel noise. The hybrid scheme also subsumes prior coding methods such as rate-one separate source-channel coding and uncoded schemes for two-way lossy transmission, as well as the correlation-preserving coding scheme for (almost) lossless transmission. Moreover, we derive a distortion outer bound for the source-channel system using a genie-aided argument. A complete JSSC theorem for a class of correlated sources and DM-TWCs whose capacity region cannot be enlarged via interactive adaptive coding is also established. Examples that illustrate the theorem are given.
CAE-RLSM: Consistent and Efficient Redundant Line Segment Merging for Online Feature Map Building
Venue: arXiv cs.RO 2020
Year: 2020

In order to obtain a compact line segment-based map representation for localization and planning of mobile robots, it is necessary to merge redundant line segments which physically represent the same part of the environment in different scans. In this paper, a consistent and efficient redundant line segment merging approach (CAE-RLSM) is proposed for online feature map building. The proposed CAE-RLSM is composed of two newly proposed modules: one-to-many incremental line segment merging (OTM-ILSM) and multi-processing global map adjustment (MP-GMA). Different from state-of-the-art offline merging approaches, the proposed CAE-RLSM can achieve real-time mapping performance, which not only reduces the redundancy of incremental merging with high efficiency, but also solves the problem of global map adjustment after loop closing to guarantee global consistency. Furthermore, a new correlation-based evaluation metric is proposed for the quality evaluation of line segment maps. This evaluation metric does not require manual measurement of the environmental metric information, instead it makes full use of globally consistent laser scans obtained by simultaneous localization and mapping (SLAM) systems to compare the performance of different line segment-based mapping approaches in an objective and fair manner. Comparative experimental results with respect to a mean shift-based offline redundant line segment merging approach (MS-RLSM) and an offline version of one-to-one incremental line segment merging approach (O$^2$TO-ILSM) on both public data sets and self-recorded data set are presented to show the superior performance of CAE-RLSM in terms of efficiency and map quality in different scenarios.
Locally Repairable Convolutional Codes with Sliding Window Repair
Venue: arXiv cs.IT 2020
Year: 2020

Locally repairable convolutional codes (LRCCs) for distributed storage systems (DSSs) are introduced in this work. They enable local repair, for a single node erasure (or more generally, $ \partial - 1 $ erasures per local group), and sliding-window global repair, which can correct erasure patterns with up to $ {\rm d}^c_j - 1 $ erasures in every window of $ j+1 $ consecutive blocks of $ n $ nodes, where $ {\rm d}^c_j $ is the $ j $th column distance of the code. The parameter $ j $ can be adjusted, for a fixed LRCC, according to different catastrophic erasure patterns, requiring only to contact $ n(j+1) - {\rm d}^c_j + 1 $ nodes, plus less than $ μn $ other nodes, in the storage system, where $ μ$ is the memory of the code. A Singleton-type bound is provided for $ {\rm d}^c_j $. If it attains such a bound, an LRCC can correct the same number of catastrophic erasures in a window of length $ n(j+1) $ as an optimal locally repairable block code of the same rate and locality, and with block length $ n(j+1) $. In addition, the LRCC is able to perform the flexible and somehow local sliding-window repair by adjusting $ j $. Furthermore, by adjusting and/or sliding the window, the LRCC can potentially correct more erasures in the original window of $ n(j+1) $ nodes than an optimal locally repairable block code of the same rate and locality, and length $ n(j+1) $. Finally, the concept of partial maximum distance profile (partial MDP) codes is introduced. Partial MDP codes can correct all information-theoretically correctable erasure patterns for a given locality, local distance and information rate. An explicit construction of partial MDP codes whose column distances attain the provided Singleton-type bound, up to certain parameter $ j=L $, is obtained based on known maximum sum-rank distance convolutional codes.
K-Core Minimization: A Game Theoretic Approach
Venue: arXiv cs.SI 2020
Year: 2020

K-cores are maximal induced subgraphs where all vertices have degree at least k. These dense patterns have applications in community detection, network visualization and protein function prediction. However, k-cores can be quite unstable to network modifications, which motivates the question: How resilient is the k-core structure of a network, such as the Web or Facebook, to edge deletions? We investigate this question from an algorithmic perspective. More specifically, we study the problem of computing a small set of edges for which the removal minimizes the $k$-core structure of a network. This paper provides a comprehensive characterization of the hardness of the k-core minimization problem (KCM), including innaproximability and fixed-parameter intractability. Motivated by such a challenge in terms of algorithm design, we propose a novel algorithm inspired by Shapley value -- a cooperative game-theoretic concept -- that is able to leverage the strong interdependencies in the effects of edge removals in the search space. As computing Shapley values is also NP-hard, we efficiently approximate them using a randomized algorithm with probabilistic guarantees. Our experiments, using several real datasets, show that the proposed algorithm outperforms competing solutions in terms of k-core minimization while being able to handle large graphs. Moreover, we illustrate how KCM can be applied in the analysis of the k-core resilience of networks.
Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments
Venue: arXiv cs.CL 2020
Year: 2020

Recent work on evaluating grammatical knowledge in pretrained sentence encoders gives a fine-grained view of a small number of phenomena. We introduce a new analysis dataset that also has broad coverage of linguistic phenomena. We annotate the development set of the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018) for the presence of 13 classes of syntactic phenomena including various forms of argument alternations, movement, and modification. We use this analysis set to investigate the grammatical knowledge of three pretrained encoders: BERT (Devlin et al., 2018), GPT (Radford et al., 2018), and the BiLSTM baseline from Warstadt et al. We find that these models have a strong command of complex or non-canonical argument structures like ditransitives (Sue gave Dan a book) and passives (The book was read). Sentences with long distance dependencies like questions (What do you think I ate?) challenge all models, but for these, BERT and GPT have a distinct advantage over the baseline. We conclude that recent sentence encoders, despite showing near-human performance on acceptability classification overall, still fail to make fine-grained grammaticality distinctions for many complex syntactic structures.
OMG-VR: Open-source Mudra Gloves for Manipulating Molecular Simulations in VR
Venue: arXiv cs.HC 2020
Year: 2020

As VR finds increasing application in scientific research domains like nanotechnology and biochemistry, we are beginning to better understand the domains in which it brings the most benefit, as well as the gestures and form factors that are most useful for specific applications. Here we describe Open-source Mudra Gloves for Virtual Reality (OMG-VR): etextile gloves designed to facilitate research scientists and students carrying out detailed and complex manipulation of simulated 3d molecular objects in VR. The OMG-VR is designed to sense when a user pinches together their thumb and index finger, or thumb and middle finger, forming a "mudra" position. Tests show that they provide good positional tracking of the point at which a pinch takes place, require no calibration, and are sufficiently accurate and robust to enable scientists to accomplish a range of tasks that involve complex spatial manipulation of molecules. The open source design offers a promising alternative to existing controllers and more costly commercial VR data gloves.
Power Control in Cellular Massive MIMO with Varying User Activity: A Deep Learning Solution
Venue: arXiv cs.IT 2020
Year: 2020

This paper considers the sum spectral efficiency (SE) optimization problem in multi-cell Massive MIMO systems with a varying number of active users. This is formulated as a joint pilot and data power control problem. Since the problem is non-convex, we first derive a novel iterative algorithm that obtains a stationary point in polynomial time. To enable real-time implementation, we also develop a deep learning solution. The proposed neural network, PowerNet, only uses the large-scale fading information to predict both the pilot and data powers. The main novelty is that we exploit the problem structure to design a single neural network that can handle a dynamically varying number of active users; hence, PowerNet is simultaneously approximating many different power control functions with varying number inputs and outputs. This is not the case in prior works and thus makes PowerNet an important step towards a practically useful solution. Numerical results demonstrate that PowerNet only loses $2\%$ in sum SE, compared to the iterative algorithm, in a nine-cell system with up to $90$ active users per in each coherence interval, and the runtime was only $0.03$ ms on a graphics processing unit (GPU). When good data labels are selected for the training phase, PowerNet can yield better sum SE than by solving the optimization problem with one initial point.
The Bayesian Prophet: A Low-Regret Framework for Online Decision Making
Venue: arXiv cs.DS 2020
Year: 2020

We develop a new framework for designing online policies given access to an oracle providing statistical information about an offline benchmark. Having access to such prediction oracles enables simple and natural Bayesian selection policies, and raises the question as to how these policies perform in different settings. Our work makes two important contributions towards this question: First, we develop a general technique we call *compensated coupling* which can be used to derive bounds on the expected regret (i.e., additive loss with respect to a benchmark) for any online policy and offline benchmark. Second, using this technique, we show that a natural greedy policy, which we call *the Bayes Selector*, has constant expected regret (i.e., independent of the number of arrivals and resource levels) for a large class of problems we refer to as Online Allocation with finite types, which includes widely-studied Online Packing and Online Matching problems. Our results generalize and simplify several existing results for Online Packing and Online Matching, and suggest a promising pathway for obtaining oracle-driven policies for other online decision-making settings.
Fully Memristive Spiking-Neuron Learning Framework and its Applications on Pattern Recognition and Edge Detection
Venue: arXiv cs.ET 2020
Year: 2020

Fully memristive spiking-neuron learning framework, which uses drift and diffusion memristor models as axon and dendrite respectively, becomes a hot topic recently with the development of memristor devices. Normally, some other devices like resistor or capacitor are still necessary on recent works of fully memristive learning framework. However, theoretically, one neuron needs axon and dendrite only, which makes technique process simpler and learning framework more similar to biologic brain. In this paper, a fully memristive spiking-neuron learning framework is introduced, in which a neuron structure is just built of one drift and one diffusion memristive models. To verify it merits, a feedforward neural network for pattern recognition and a cellular neural network for edge detection are designed. Experiment results show that compared to other memristive neural networks, our framework's the processing speed is much faster and the hardware resource is saved in pattern recognition due to its simple structure. Further due to the dynamic filtering function of diffusion memristor model in our learning framework, its peak signal noise ratio (PSNR) is much higher than traditional algorithms in edge detection.