OpenRaiser/Intern-Atlas huggingface dataset:
Robustness for Non-Parametric Classification: A Generic Attack and Defense
Venue: arXiv cs.LG 2020
Year: 2020

Adversarially robust machine learning has received much recent attention. However, prior attacks and defenses for non-parametric classifiers have been developed in an ad-hoc or classifier-specific basis. In this work, we take a holistic look at adversarial examples for non-parametric classifiers, including nearest neighbors, decision trees, and random forests. We provide a general defense method, adversarial pruning, that works by preprocessing the dataset to become well-separated. To test our defense, we provide a novel attack that applies to a wide range of non-parametric classifiers. Theoretically, we derive an optimally robust classifier, which is analogous to the Bayes Optimal. We show that adversarial pruning can be viewed as a finite sample approximation to this optimal classifier. We empirically show that our defense and attack are either better than or competitive with prior work on non-parametric classifiers. Overall, our results provide a strong and broadly-applicable baseline for future work on robust non-parametrics. Code available at https://github.com/yangarbiter/adversarial-nonparametrics/ .
DiCENet: Dimension-wise Convolutions for Efficient Networks
Venue: arXiv cs.CV 2020
Year: 2020

We introduce a novel and generic convolutional unit, DiCE unit, that is built using dimension-wise convolutions and dimension-wise fusion. The dimension-wise convolutions apply light-weight convolutional filtering across each dimension of the input tensor while dimension-wise fusion efficiently combines these dimension-wise representations; allowing the DiCE unit to efficiently encode spatial and channel-wise information contained in the input tensor. The DiCE unit is simple and can be seamlessly integrated with any architecture to improve its efficiency and performance. Compared to depth-wise separable convolutions, the DiCE unit shows significant improvements across different architectures. When DiCE units are stacked to build the DiCENet model, we observe significant improvements over state-of-the-art models across various computer vision tasks including image classification, object detection, and semantic segmentation. On the ImageNet dataset, the DiCENet delivers 2-4% higher accuracy than state-of-the-art manually designed models (e.g., MobileNetv2 and ShuffleNetv2). Also, DiCENet generalizes better to tasks (e.g., object detection) that are often used in resource-constrained devices in comparison to state-of-the-art separable convolution-based efficient networks, including neural search-based methods (e.g., MobileNetv3 and MixNet. Our source code in PyTorch is open-source and is available at https://github.com/sacmehta/EdgeNets/
Cross-view Semantic Segmentation for Sensing Surroundings
Venue: arXiv cs.CV 2020
Year: 2020

Sensing surroundings plays a crucial role in human spatial perception, as it extracts the spatial configuration of objects as well as the free space from the observations. To facilitate the robot perception with such a surrounding sensing capability, we introduce a novel visual task called Cross-view Semantic Segmentation as well as a framework named View Parsing Network (VPN) to address it. In the cross-view semantic segmentation task, the agent is trained to parse the first-view observations into a top-down-view semantic map indicating the spatial location of all the objects at pixel-level. The main issue of this task is that we lack the real-world annotations of top-down-view data. To mitigate this, we train the VPN in 3D graphics environment and utilize the domain adaptation technique to transfer it to handle real-world data. We evaluate our VPN on both synthetic and real-world agents. The experimental results show that our model can effectively make use of the information from different views and multi-modalities to understanding spatial information. Our further experiment on a LoCoBot robot shows that our model enables the surrounding sensing capability from 2D image input. Code and demo videos can be found at \url{https://view-parsing-network.github.io}.
Optimal Energy Efficiency with Delay Constraints for Multi-layer Cooperative Fog Computing Networks
Venue: arXiv cs.NI 2020
Year: 2020

We develop a joint offloading and resource allocation framework for a multi-layer cooperative fog computing network, aiming to minimize the total energy consumption of multiple mobile devices subject to their service delay requirements. The resulting optimization involves both binary (offloading decisions) and real variables (resource allocations), making it an NP-hard and computationally intractable problem. To tackle it, we first propose an improved branch-and-bound algorithm (IBBA) that is implemented in a centralized manner. However, due to the large size of the cooperative fog computing network, the computational complexity of the proposed IBBA is relatively high. To speed up the optimal solution searching as well as to enable its distributed implementation, we then leverage the unique structure of the underlying problem and the parallel processing at fog nodes. To that end, we propose a distributed framework, namely feasibility finding Benders decomposition (FFBD), that decomposes the original problem into a master problem for the offloading decision and subproblems for resource allocation. The master problem (MP) is then equipped with powerful cutting-planes to exploit the fact of resource limitation at fog nodes. The subproblems (SP) for resource allocation can find their closed-form solutions using our fast solution detection method. These (simpler) subproblems can then be solved in parallel at fog nodes. The numerical results show that the FFBD always returns the optimal solution of the problem with significantly less computation time (e.g., compared with the centralized IBBA approach). The FFBD with the fast solution detection method, namely FFBD-F, can reduce up to $60\%$ and $90\%$ of computation time, respectively, compared with those of the conventional FFBD, namely FFBD-S, and IBBA.
Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
Venue: arXiv cs.LG 2020
Year: 2020

We improve the over-parametrization size over two beautiful results [Li and Liang' 2018] and [Du, Zhai, Poczos and Singh' 2019] in deep learning theory.
Out-of-Vocabulary Embedding Imputation with Grounded Language Information by Graph Convolutional Networks
Venue: arXiv cs.CL 2020
Year: 2020

Due to the ubiquitous use of embeddings as input representations for a wide range of natural language tasks, imputation of embeddings for rare and unseen words is a critical problem in language processing. Embedding imputation involves learning representations for rare or unseen words during the training of an embedding model, often in a post-hoc manner. In this paper, we propose an approach for embedding imputation which uses grounded information in the form of a knowledge graph. This is in contrast to existing approaches which typically make use of vector space properties or subword information. We propose an online method to construct a graph from grounded information and design an algorithm to map from the resulting graphical structure to the space of the pre-trained embeddings. Finally, we evaluate our approach on a range of rare and unseen word tasks across various domains and show that our model can learn better representations. For example, on the Card-660 task our method improves Pearson's and Spearman's correlation coefficients upon the state-of-the-art by 11% and 17.8% respectively using GloVe embeddings.
Stochastic Neural Network with Kronecker Flow
Venue: arXiv cs.LG 2020
Year: 2020

Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and better than the baselines.
Gait modeling and optimization for the perturbed Stokes regime
Venue: arXiv cs.RO 2020
Year: 2020

Many forms of locomotion, both natural and artificial, are dominated by viscous friction in the sense that without power expenditure they quickly come to a standstill. From geometric mechanics, it is known that for swimming at the "Stokesian" (viscous; zero Reynolds number) limit, the motion is governed by a reduced order "connection" model that describes how body shape change produces motion for the body frame with respect to the world. In the "perturbed Stokes regime" where inertial forces are still dominated by viscosity, but are not negligible (low Reynolds number), we show that motion is still governed by a functional relationship between shape velocity and body velocity, but this function is no longer linear in shape change rate. We derive this model using results from singular perturbation theory, and the theory of noncompact normally hyperbolic invariant manifolds (NHIMs). Using the theoretical properties of this reduced-order model, we develop an algorithm that estimates an approximation to the dynamics near a cyclic body shape change (a "gait") directly from observational data of shape and body motion. This extends our previous work which assumed kinematic "connection" models. To compare the old and new algorithms, we analyze simulated swimmers over a range of inertia to damping ratios. Our new class of models performs well on the Stokesian regime, and over several orders of magnitude outside it into the perturbed Stokes regime, where it gives significantly improved prediction accuracy compared to previous work. In addition to algorithmic improvements, we thereby present a new class of models that is of independent interest. Their application to data-driven modeling improves our ability to study the optimality of animal gaits, and our ability to use hardware-in-the-loop optimization to produce gaits for robots.
Stability and Metastability of Traffic Dynamics in Uplink Random Access Networks
Venue: arXiv cs.IT 2020
Year: 2020

We characterize the stability, metastability, and the stationary regime of traffic dynamics in a single-cell uplink wireless system. The traffic is represented in terms of spatial birth-death processes, in which users arrive as a Poisson point process in time and space, each with a file to transmit to the base station. The service rate of each user is based on its signal to interference plus noise ratio, where the interference is from other active users in the cell. Once the file is fully transmitted, the user leaves the cell. We derive the necessary and sufficient condition for network stability, which is independent of the specific bounded path loss function. A novel observation is that for a certain range of arrival rates, the network appears stable for a possibly long time, and then suddenly exhibits instability. This property, which is known in statistical physics but rarely observed in wireless communication, is called metastability. Finally, we propose two heuristic characterizations based on mean-field interpretation, of the network steady-state regime when it exists. The first-order approximation is very simple to compute, but loose in some regimes, whereas the second-order approximation is more sophisticated but tight for the whole range of arrival rates.
The Prolog debugger and declarative programming
Venue: arXiv cs.PL 2020
Year: 2020

Logic programming is a declarative programming paradigm. Programming language Prolog makes logic programming possible, at least to a substantial extent. However the Prolog debugger works solely in terms of the operational semantics. So it is incompatible with declarative programming. This report discusses this issue and tries to find how the debugger may be used from the declarative point of view. The results are rather not encouraging. Also, the box model of Byrd, used by the debugger, is explained in terms of SLD-resolution.
Statistical guarantees for local graph clustering
Venue: arXiv cs.LG 2020
Year: 2020

Local graph clustering methods aim to find small clusters in very large graphs. These methods take as input a graph and a seed node, and they return as output a good cluster in a running time that depends on the size of the output cluster but that is independent of the size of the input graph. In this paper, we adopt a statistical perspective on local graph clustering, and we analyze the performance of the l1-regularized PageRank method~(Fountoulakis et. al.) for the recovery of a single target cluster, given a seed node inside the cluster. Assuming the target cluster has been generated by a random model, we present two results. In the first, we show that the optimal support of l1-regularized PageRank recovers the full target cluster, with bounded false positives. In the second, we show that if the seed node is connected solely to the target cluster then the optimal support of l1-regularized PageRank recovers exactly the target cluster. We also show empirically that l1-regularized PageRank has a state-of-the-art performance on many real graphs, demonstrating the superiority of the method. From a computational perspective, we show that the solution path of l1-regularized PageRank is monotonic. This allows for the application of the forward stagewise algorithm, which approximates the solution path in running time that does not depend on the size of the whole graph. Finally, we show that l1-regularized PageRank and approximate personalized PageRank (APPR), another very popular method for local graph clustering, are equivalent in the sense that we can lower and upper bound the output of one with the output of the other. Based on this relation, we establish for APPR similar results to those we establish for l1-regularized PageRank.
Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
Venue: arXiv cs.LG 2020
Year: 2020

The outcome of Jacobian singular values regularization was studied for supervised learning problems. It also was shown that Jacobian conditioning regularization can help to avoid the ``mode-collapse'' problem in Generative Adversarial Networks. In this paper, we try to answer the following question: Can information about policy conditioning help to shape a more stable and general policy of reinforcement learning agents? To answer this question, we conduct a study of Jacobian conditioning behavior during policy optimization. To the best of our knowledge, this is the first work that research condition number in reinforcement learning agents. We propose a conditioning regularization algorithm and test its performance on the range of continuous control tasks. Finally, we compare algorithms on the CoinRun environment with separated train end test levels to analyze how conditioning regularization contributes to agents' generalization.
Near-Optimal Glimpse Sequences for Improved Hard Attention Neural Network Training
Venue: arXiv cs.LG 2020
Year: 2020

Hard visual attention is a promising approach to reduce the computational burden of modern computer vision methodologies. Hard attention mechanisms are typically non-differentiable. They can be trained with reinforcement learning but the high-variance training this entails hinders more widespread application. We show how hard attention for image classification can be framed as a Bayesian optimal experimental design (BOED) problem. From this perspective, the optimal locations to attend to are those which provide the greatest expected reduction in the entropy of the classification distribution. We introduce methodology from the BOED literature to approximate this optimal behaviour, and use it to generate `near-optimal' sequences of attention locations. We then show how to use such sequences to partially supervise, and therefore speed up, the training of a hard attention mechanism. Although generating these sequences is computationally expensive, they can be reused by any other networks later trained on the same task.
ATRW: A Benchmark for Amur Tiger Re-identification in the Wild
Venue: arXiv cs.CV 2020
Year: 2020

Monitoring the population and movements of endangered species is an important task to wildlife conversation. Traditional tagging methods do not scale to large populations, while applying computer vision methods to camera sensor data requires re-identification (re-ID) algorithms to obtain accurate counts and moving trajectory of wildlife. However, existing re-ID methods are largely targeted at persons and cars, which have limited pose variations and constrained capture environments. This paper tries to fill the gap by introducing a novel large-scale dataset, the Amur Tiger Re-identification in the Wild (ATRW) dataset. ATRW contains over 8,000 video clips from 92 Amur tigers, with bounding box, pose keypoint, and tiger identity annotations. In contrast to typical re-ID datasets, the tigers are captured in a diverse set of unconstrained poses and lighting conditions. We demonstrate with a set of baseline algorithms that ATRW is a challenging dataset for re-ID. Lastly, we propose a novel method for tiger re-identification, which introduces precise pose parts modeling in deep neural networks to handle large pose variation of tigers, and reaches notable performance improvement over existing re-ID methods. The dataset is public available at https://cvwc2019.github.io/ .
Robust Regression for Safe Exploration in Control
Venue: arXiv cs.LG 2020
Year: 2020

We study the problem of safe learning and exploration in sequential control problems. The goal is to safely collect data samples from operating in an environment, in order to learn to achieve a challenging control goal (e.g., an agile maneuver close to a boundary). A central challenge in this setting is how to quantify uncertainty in order to choose provably-safe actions that allow us to collect informative data and reduce uncertainty, thereby achieving both improved controller safety and optimality. To address this challenge, we present a deep robust regression model that is trained to directly predict the uncertainty bounds for safe exploration. We derive generalization bounds for learning, and connect them with safety and stability bounds in control. We demonstrate empirically that our robust regression approach can outperform the conventional Gaussian process (GP) based safe exploration in settings where it is difficult to specify a good GP prior.
What is Stablecoin?: A Survey on Its Mechanism and Potential as Decentralized Payment Systems
Venue: arXiv cs.CR 2020
Year: 2020

Our study provides a survey on how existing stablecoins-- cryptocurrencies aiming at price stabilization-- peg their value to other assets, from the perspective of Decentralized Payment Systems (DPSs). This attempt is important because there has been no preceding surveys focusing on the stablecoin as DPSs, i.e., the one aiming at not only price stabilization but also decentralization. Specifically, we first classified existing stablecoins into four types according to their collaterals (fiat, commodity, crypto, and non-collateralized) and pointed out the high potential of non-collateralized stablecoins as DPSs; then, we further classified existing non-collateralized stablecoins into two types according to their intervention layers (protocol, application) and confirmed details of their representative mechanisms. Utilizing concepts such as Quantity Theory of Money (QTM), Tobin tax, and speculative attack, our survey revealed the status quo where, despite the high potential of non-collateralized stablecoins, they have no standard mechanism to achieve the stablecoin for practical DPSs.
Cross-cultural data shows musical scales evolved to maximise imperfect fifths
Venue: arXiv cs.SD 2020
Year: 2020

Musical scales are used throughout the world, but the question of how they evolved remains open. Some suggest that scales based on the harmonic series are inherently pleasant, while others propose that scales are chosen that are easy to communicate. However, testing these theories has been hindered by the sparseness of empirical evidence. Here, we assimilate data from diverse ethnomusicological sources into a cross-cultural database of scales. We generate populations of scales based on multiple theories and assess their similarity to empirical distributions from the database. Most scales tend to include intervals which are close in size to perfect fifths (``imperfect fifths''), and packing arguments explain the salient features of the distributions. Scales are also preferred if their intervals are compressible, which may facilitate efficient communication and memory of melodies. While scales appear to evolve according to various selection pressures, the simplest h imperfect-fifths packing model best fits the empirical data.
Membership Privacy for Machine Learning Models Through Knowledge Transfer
Venue: arXiv cs.LG 2020
Year: 2020

Large capacity machine learning (ML) models are prone to membership inference attacks (MIAs), which aim to infer whether the target sample is a member of the target model's training dataset. The serious privacy concerns due to the membership inference have motivated multiple defenses against MIAs, e.g., differential privacy and adversarial regularization. Unfortunately, these defenses produce ML models with unacceptably low classification performances. Our work proposes a new defense, called distillation for membership privacy (DMP), against MIAs that preserves the utility of the resulting models significantly better than prior defenses. DMP leverages knowledge distillation to train ML models with membership privacy. We provide a novel criterion to tune the data used for knowledge transfer in order to amplify the membership privacy of DMP. Our extensive evaluation shows that DMP provides significantly better tradeoffs between membership privacy and classification accuracies compared to state-of-the-art MIA defenses. For instance, DMP achieves ~100% accuracy improvement over adversarial regularization for DenseNet trained on CIFAR100, for similar membership privacy (measured using MIA risk): when the MIA risk is 53.7%, adversarially regularized DenseNet is 33.6% accurate, while DMP-trained DenseNet is 65.3% accurate.
Theoretical Limitations of Self-Attention in Neural Sequence Models
Venue: arXiv cs.CL 2020
Year: 2020

Transformers are emerging as the new workhorse of NLP, showing great success across tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of self-attention, finding that it cannot model periodic finite-state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.