OpenRaiser/Intern-Atlas huggingface dataset:
Modeling the Formation of Social Conventions from Embodied Real-Time Interactions
Venue: arXiv cs.MA 2019
Year: 2019

What is the role of real-time control and learning in the formation of social conventions? To answer this question, we propose a computational model that matches human behavioral data in a social decision-making game that was analyzed both in discrete-time and continuous-time setups. Furthermore, unlike previous approaches, our model takes into account the role of sensorimotor control loops in embodied decision-making scenarios. For this purpose, we introduce the Control-based Reinforcement Learning (CRL) model. CRL is grounded in the Distributed Adaptive Control (DAC) theory of mind and brain, where low-level sensorimotor control is modulated through perceptual and behavioral learning in a layered structure. CRL follows these principles by implementing a feedback control loop handling the agent's reactive behaviors (pre-wired reflexes), along with an adaptive layer that uses reinforcement learning to maximize long-term reward. We test our model in a multi-agent game-theoretic task in which coordination must be achieved to find an optimal solution. We show that CRL is able to reach human-level performance on standard game-theoretic metrics such as efficiency in acquiring rewards and fairness in reward distribution.
Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
Venue: arXiv cs.CV 2019
Year: 2019

Strong intelligent machines powered by deep neural networks are increasingly deployed as black boxes to make decisions in risk-sensitive domains, such as finance and medical. To reduce potential risk and build trust with users, it is critical to interpret how such machines make their decisions. Existing works interpret a pre-trained neural network by analyzing hidden neurons, mimicking pre-trained models or approximating local predictions. However, these methods do not provide a guarantee on the exactness and consistency of their interpretation. In this paper, we propose an elegant closed form solution named $OpenBox$ to compute exact and consistent interpretations for the family of Piecewise Linear Neural Networks (PLNN). The major idea is to first transform a PLNN into a mathematically equivalent set of linear classifiers, then interpret each linear classifier by the features that dominate its prediction. We further apply $OpenBox$ to demonstrate the effectiveness of non-negative and sparse constraints on improving the interpretability of PLNNs. The extensive experiments on both synthetic and real world data sets clearly demonstrate the exactness and consistency of our interpretation.
Breaking the Linear-Memory Barrier in MPC: Fast MIS on Trees with Strongly Sublinear Memory
Venue: arXiv cs.DS 2019
Year: 2019

Recently, studying fundamental graph problems in the \emph{Massively Parallel Computation (MPC) framework, inspired by the MapReduce paradigm, has gained a lot of attention. An assumption common to a vast majority of approaches is to allow $\widetildeΩ(n)$ memory per machine, where $n$ is the number of nodes in the graph and $\widetildeΩ$ hides polylogarithmic factors. However, as pointed out by Karloff et al. [SODA'10] and Czumaj et al. [STOC'18], it might be unrealistic for a single machine to have linear or only slightly sublinear memory. In this paper, we thus study a more practical variant of the MPC model which only requires substantially sublinear or even subpolynomial memory per machine. In contrast to the linear-memory MPC model and also to streaming algorithms, in this low-memory MPC setting, a single machine will only see a small number of nodes in the graph. We introduce a new and strikingly simple technique to cope with this imposed locality. In particular, we show that the Maximal Independent Set (MIS) problem can be solved efficiently, that is, in $O(\log^3 \log n)$ rounds, when the input graph is a tree. This constitutes an almost exponential speed-up over the low-memory MPC algorithm in $O(\sqrt{\log n})$-algorithm in a concurrent work by Ghaffari and Uitto [SODA'19] and substantially reduces the local memory from $\widetildeΩ(n)$ required by the recent $O(\log \log n)$-round MIS algorithm of Ghaffari et al. [PODC'18] to $n^α$ for any $α>0$, without incurring a significant loss in the round complexity. Moreover, it demonstrates how to make use of the all-to-all communication in the MPC model to almost exponentially improve on the corresponding bound in the $\mathsf{LOCAL}$ and $\mathsf{PRAM}$ models by Lenzen and Wattenhofer [PODC'11].
Devon: Deformable Volume Network for Learning Optical Flow
Venue: arXiv cs.CV 2019
Year: 2019

State-of-the-art neural network models estimate large displacement optical flow in multi-resolution and use warping to propagate the estimation between two resolutions. Despite their impressive results, it is known that there are two problems with the approach. First, the multi-resolution estimation of optical flow fails in situations where small objects move fast. Second, warping creates artifacts when occlusion or dis-occlusion happens. In this paper, we propose a new neural network module, Deformable Cost Volume, which alleviates the two problems. Based on this module, we designed the Deformable Volume Network (Devon) which can estimate multi-scale optical flow in a single high resolution. Experiments show Devon is more suitable in handling small objects moving fast and achieves comparable results to the state-of-the-art methods in public benchmarks.
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Venue: arXiv cs.LG 2019
Year: 2019

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.
Multiscale Planar Graph Generation
Venue: arXiv cs.SI 2019
Year: 2019

The study of network representations of physical, biological, and social phenomena can help us better understand the structural and functional dynamics of their networks and formulate predictive models of these phenomena. However, due to the scarcity of real-world network data owing to factors such as cost and effort required in collection of network data and the sensitivity of this data towards theft and misuse, engineers and researchers often rely on synthetic data for simulations, hypothesis testing, decision making, and algorithm engineering. An important characteristic of infrastructure networks such as roads, water distribution and other utility systems is that they can be embedded in a plane, therefore to simulate these system we need realistic networks which are also planar. While the currently-available synthetic network generators can model networks that exhibit realism, they do not guarantee or achieve planarity. Therefore, in this paper we present a flexible algorithm that can synthesize realistic networks that are planar. The method follows a multi-scale randomized editing approach generating a hierarchy of coarsened networks of a given planar graph and introducing edits at various levels in the hierarchy. The method preserves the structural properties with minimal bias including the planarity of the network, while introducing realistic variability at multiple scales.
Robust GANs against Dishonest Adversaries
Venue: arXiv cs.LG 2019
Year: 2019

Robustness of deep learning models is a property that has recently gained increasing attention. We explore a notion of robustness for generative adversarial models that is pertinent to their internal interactive structure, and show that, perhaps surprisingly, the GAN in its original form is not robust. Our notion of robustness relies on a perturbed discriminator, or noisy, adversarial interference with its feedback. We explore, theoretically and empirically, the effect of model and training properties on this robustness. In particular, we show theoretical conditions for robustness that are supported by empirical evidence. We also test the effect of regularization. Our results suggest variations of GANs that are indeed more robust to noisy attacks and have more stable training behavior, requiring less regularization in general. Inspired by our theoretical results, we further extend our framework to obtain a class of models related to WGAN, with good empirical performance. Overall, our results suggest a new perspective on understanding and designing GAN models from the viewpoint of their internal robustness.
Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
Venue: arXiv cs.CV 2019
Year: 2019

The problem of $L_p$-norm constrained coding is to convert signal into code that lies inside an $L_p$-ball and most faithfully reconstructs the signal. Previous works under the name of sparse coding considered the cases of $L_0$ and $L_1$ norms. The cases with $p>1$ values, i.e. non-sparse coding studied in this paper, remain a difficulty. We propose an interpretable deep structure namely Frank-Wolfe Network (F-W Net), whose architecture is inspired by unrolling and truncating the Frank-Wolfe algorithm for solving an $L_p$-norm constrained problem with $p\geq 1$. We show that the Frank-Wolfe solver for the $L_p$-norm constraint leads to a novel closed-form nonlinear unit, which is parameterized by $p$ and termed $pool_p$. The $pool_p$ unit links the conventional pooling, activation, and normalization operations, making F-W Net distinct from existing deep networks either heuristically designed or converted from projected gradient descent algorithms. We further show that the hyper-parameter $p$ can be made learnable instead of pre-chosen in F-W Net, which gracefully solves the non-sparse coding problem even with unknown $p$. We evaluate the performance of F-W Net on an extensive range of simulations as well as the task of handwritten digit recognition, where F-W Net exhibits strong learning capability. We then propose a convolutional version of F-W Net, and apply the convolutional F-W Net into image denoising and super-resolution tasks, where F-W Net all demonstrates impressive effectiveness, flexibility, and robustness.
FFT Multichannel Interpolation and Application to Image Super-resolution
Venue: arXiv cs.IT 2019
Year: 2019

This paper presents an innovative set of tools to support a methodology for the multichannel interpolation (MCI) of a discrete signal. It is shown that a bandlimited signal $f$ can be exactly reconstructed from finite samples of $g_k$ ($1\leq k\leq M$) which are the responses of $M$ linear systems with input $f$. The proposed interpolation can also be applied to approximate non-bandlimited signals. Quantitative error is analyzed to ensure its effectiveness in approximating non-bandlimited signals and its Hilbert transform. Based on the FFT technique, a fast algorithm which brings high computational efficiency and reliability for MCI is presented. The standout performance of MCI is illustrated by several simulations. Additionally, the proposed interpolation is applied to the single image super-resolution (SISR). Its superior performance in accuracy and speed of SISR is demonstrated by the experimental studies. Our results are compared qualitatively and quantitatively with the state-of-the-art methods in image upsampling and reconstruction by using the standard measurement criteria.
Toll Caps in Privatized Road Networks
Venue: arXiv cs.GT 2019
Year: 2019

We consider a network pricing game on a parallel network with congestion effects in which link owners set tolls for travel so as to maximize profit. A central authority is able to regulate this competition by means of a (uniform) price cap. The first question we want to answer is how such a cap should be designed in order to minimize the total congestion. We provide an algorithm that finds an optimal price cap for networks with affine latency functions and a full support Wardrop equilibrium. Second, we consider the induced network performance at an optimal price cap. We show that for two link networks with affine latency functions, the congestion costs at the optimal price cap are at most $8/7$ times the optimal congestion costs. For more general latency functions, this bound goes up to 2 under the assumption that an uncapped Nash equilibrium exists. However, in general such an equilibrium need not exist and this can be used to show that optimal price caps can induce arbitrarily inefficient flows.
Exactly Robust Kernel Principal Component Analysis
Venue: arXiv cs.LG 2019
Year: 2019

Robust principal component analysis (RPCA) can recover low-rank matrices when they are corrupted by sparse noises. In practice, many matrices are, however, of high-rank and hence cannot be recovered by RPCA. We propose a novel method called robust kernel principal component analysis (RKPCA) to decompose a partially corrupted matrix as a sparse matrix plus a high or full-rank matrix with low latent dimensionality. RKPCA can be applied to many problems such as noise removal and subspace clustering and is still the only unsupervised nonlinear method robust to sparse noises. Our theoretical analysis shows that, with high probability, RKPCA can provide high recovery accuracy. The optimization of RKPCA involves nonconvex and indifferentiable problems. We propose two nonconvex optimization algorithms for RKPCA. They are alternating direction method of multipliers with backtracking line search and proximal linearized minimization with adaptive step size. Comparative studies in noise removal and robust subspace clustering corroborate the effectiveness and superiority of RKPCA.
Semi-Supervised Online Structure Learning for Composite Event Recognition
Venue: arXiv cs.AI 2019
Year: 2019

Online structure learning approaches, such as those stemming from Statistical Relational Learning, enable the discovery of complex relations in noisy data streams. However, these methods assume the existence of fully-labelled training data, which is unrealistic for most real-world applications. We present a novel approach for completing the supervision of a semi-supervised structure learning task. We incorporate graph-cut minimisation, a technique that derives labels for unlabelled data, based on their distance to their labelled counterparts. In order to adapt graph-cut minimisation to first order logic, we employ a suitable structural distance for measuring the distance between sets of logical atoms. The labelling process is achieved online (single-pass) by means of a caching mechanism and the Hoeffding bound, a statistical tool to approximate globally-optimal decisions from locally-optimal ones. We evaluate our approach on the task of composite event recognition by using a benchmark dataset for human activity recognition, as well as a real dataset for maritime monitoring. The evaluation suggests that our approach can effectively complete the missing labels and eventually, improve the accuracy of the underlying structure learning system.
Deep Learning for Signal Authentication and Security in Massive Internet of Things Systems
Venue: arXiv cs.CR 2019
Year: 2019

Secure signal authentication is arguably one of the most challenging problems in the Internet of Things (IoT) environment, due to the large-scale nature of the system and its susceptibility to man-in-the-middle and eavesdropping attacks. In this paper, a novel deep learning method is proposed for dynamic authentication of IoT signals to detect cyber attacks. The proposed learning framework, based on a long short-term memory (LSTM) structure, enables the IoT devices (IoTDs) to extract a set of stochastic features from their generated signal and dynamically watermark these features into the signal. This method enables the cloud, which collects signals from the IoT devices, to effectively authenticate the reliability of the signals. Moreover, in massive IoT scenarios, since the cloud cannot authenticate all the IoTDs simultaneously due to computational limitations, a game-theoretic framework is proposed to improve the cloud's decision making process by predicting vulnerable IoTDs. The mixed-strategy Nash equilibrium (MSNE) for this game is derived and the uniqueness of the expected utility at the equilibrium is proven. In the massive IoT system, due to a large set of available actions for the cloud, it is shown that analytically deriving the MSNE is challenging and, thus, a learning algorithm proposed that converges to the MSNE. Moreover, in order to cope with the incomplete information case in which the cloud cannot access the state of the unauthenticated IoTDs, a deep reinforcement learning algorithm is proposed to dynamically predict the state of unauthenticated IoTDs and allow the cloud to decide on which IoTDs to authenticate. Simulation results show that, with an attack detection delay of under 1 second the messages can be transmitted from IoT devices with an almost 100% reliability.
Subspace Tracking and Least Squares Approaches to Channel Estimation in Millimeter Wave Multiuser MIMO
Venue: arXiv cs.IT 2019
Year: 2019

The problem of MIMO channel estimation at millimeter wave frequencies, both in a single-user and in a multi-user setting, is tackled in this paper. Using a subspace approach, we develop a protocol enabling the estimation of the right (resp. left) singular vectors at the transmitter (resp. receiver) side; then, we adapt the projection approximation subspace tracking with deflation and the orthogonal Oja algorithms to our framework and obtain two channel estimation algorithms. We also present an alternative algorithm based on the least squares approach. The hybrid analog/digital nature of the beamformer is also explicitly taken into account at the algorithm design stage. In order to limit the system complexity, a fixed analog beamformer is used at both sides of the communication links. The obtained numerical results, showing the accuracy in the estimation of the channel matrix dominant singular vectors, the system achievable spectral efficiency, and the system bit-error-rate, prove that the proposed algorithms are effective, and that they compare favorably, in terms of the performance-complexity trade-off, with respect to several competing alternatives.
Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification
Venue: arXiv cs.CV 2019
Year: 2019

The increased availability of X-ray image archives (e.g. the ChestX-ray14 dataset from the NIH Clinical Center) has triggered a growing interest in deep learning techniques. To provide better insight into the different approaches, and their applications to chest X-ray classification, we investigate a powerful network architecture in detail: the ResNet-50. Building on prior work in this domain, we consider transfer learning with and without fine-tuning as well as the training of a dedicated X-ray network from scratch. To leverage the high spatial resolution of X-ray data, we also include an extended ResNet-50 architecture, and a network integrating non-image data (patient age, gender and acquisition type) in the classification process. In a concluding experiment, we also investigate multiple ResNet depths (i.e. ResNet-38 and ResNet-101). In a systematic evaluation, using 5-fold re-sampling and a multi-label loss function, we compare the performance of the different approaches for pathology classification by ROC statistics and analyze differences between the classifiers using rank correlation. Overall, we observe a considerable spread in the achieved performance and conclude that the X-ray-specific ResNet-38, integrating non-image data yields the best overall results. Furthermore, class activation maps are used to understand the classification process, and a detailed analysis of the impact of non-image features is provided.
Efficient Synchronization of State-based CRDTs
Venue: arXiv cs.DC 2019
Year: 2019

To ensure high availability in large scale distributed systems, Conflict-free Replicated Data Types (CRDTs) relax consistency by allowing immediate query and update operations at the local replica, with no need for remote synchronization. State-based CRDTs synchronize replicas by periodically sending their full state to other replicas, which can become extremely costly as the CRDT state grows. Delta-based CRDTs address this problem by producing small incremental states (deltas) to be used in synchronization instead of the full state. However, current synchronisation algorithms for Delta-based CRDTs induce redundant wasteful delta propagation, performing worse than expected, and surprisingly, no better than State-based. In this paper we: 1) identify two sources of inefficiency in current synchronization algorithms for delta-based CRDTs; 2) bring the concept of join decomposition to state-based CRDTs; 3) exploit join decompositions to obtain optimal deltas and 4) improve the efficiency of synchronization algorithms; and finally, 5) evaluate the improved algorithms.
Generating Artificial Data for Private Deep Learning
Venue: arXiv cs.LG 2019
Year: 2019

In this paper, we propose generating artificial data that retain statistical properties of real data as the means of providing privacy with respect to the original dataset. We use generative adversarial network to draw privacy-preserving artificial data samples and derive an empirical method to assess the risk of information disclosure in a differential-privacy-like way. Our experiments show that we are able to generate artificial data of high quality and successfully train and validate machine learning models on this data while limiting potential privacy loss.
Learning Rules-First Classifiers
Venue: arXiv cs.LG 2019
Year: 2019

Complex classifiers may exhibit "embarassing" failures in cases where humans can easily provide a justified classification. Avoiding such failures is obviously of key importance. In this work, we focus on one such setting, where a label is perfectly predictable if the input contains certain features, or rules, and otherwise it is predictable by a linear classifier. We define a hypothesis class that captures this notion and determine its sample complexity. We also give evidence that efficient algorithms cannot achieve this sample complexity. We then derive a simple and efficient algorithm and show that its sample complexity is close to optimal, among efficient algorithms. Experiments on synthetic and sentiment analysis data demonstrate the efficacy of the method, both in terms of accuracy and interpretability.