OpenRaiser/Intern-Atlas huggingface dataset:
Library network, a possible path to explainable neural networks
Venue: arXiv cs.LG 2020
Year: 2020

Deep neural networks (DNNs) may outperform human brains in complex tasks, but the lack of transparency in their decision-making processes makes us question whether we could fully trust DNNs with high stakes problems. As DNNs' operations rely on a massive number of both parallel and sequential linear/nonlinear computations, predicting their mistakes is nearly impossible. Also, a line of studies suggests that DNNs can be easily deceived by adversarial attacks, indicating that their decisions can easily be corrupted by unexpected factors. Such vulnerability must be overcome if we intend to take advantage of DNNs' efficiency in high stakes problems. Here, we propose an algorithm that can help us better understand DNNs' decision-making processes. Our empirical evaluations suggest that this algorithm can effectively trace DNNs' decision processes from one layer to another and detect adversarial attacks.
Deep k-NN Defense against Clean-label Data Poisoning Attacks
Venue: arXiv cs.LG 2020
Year: 2020

Targeted clean-label data poisoning is a type of adversarial attack on machine learning systems in which an adversary injects a few correctly-labeled, minimally-perturbed samples into the training data, causing a model to misclassify a particular test sample during inference. Although defenses have been proposed for general poisoning attacks, no reliable defense for clean-label attacks has been demonstrated, despite the attacks' effectiveness and realistic applications. In this work, we propose a simple, yet highly-effective Deep k-NN defense against both feature collision and convex polytope clean-label attacks on the CIFAR-10 dataset. We demonstrate that our proposed strategy is able to detect over 99% of poisoned examples in both attacks and remove them without compromising model performance. Additionally, through ablation studies, we discover simple guidelines for selecting the value of k as well as for implementing the Deep k-NN defense on real-world datasets with class imbalance. Our proposed defense shows that current clean-label poisoning attack strategies can be annulled, and serves as a strong yet simple-to-implement baseline defense to test future clean-label poisoning attacks. Our code is available at https://github.com/neeharperi/DeepKNNDefense
Lattice PUF: A Strong Physical Unclonable Function Provably Secure against Machine Learning Attacks
Venue: arXiv cs.CR 2020
Year: 2020

We propose a strong physical unclonable function (PUF) provably secure against machine learning (ML) attacks with both classical and quantum computers. Its security is derived from cryptographic hardness of learning decryption functions of public-key cryptosystems. Our design compactly realizes the decryption function of the learning-with-errors (LWE) cryptosystem. Due to the fundamental connection of LWE to lattice problems, we call the construction the lattice PUF. Lattice PUF is constructed using a physically obfuscated key (POK), an LWE decryption function block, and a linear-feedback shift register (LFSR) as a pseudo-random number generator. The POK provides the secret key of the LWE decryption function; its stability is ensured by a fuzzy extractor (FE). To reduce the challenge size, we exploit distributional relaxations of space-efficient LWEs. That allows only a small challenge-seed to be transmitted with the full-length challenge generated by the LFSR, resulting in a 100X reduction of communication cost. To prevent an active challenge-manipulation attack, a self-incrementing counter is embedded into the challenge seed. We prototyped the lattice PUF with 2^136 challenge-response pairs (CRPs) on a Spartan 6 FPGA, which required 45 slices for the PUF logic proper and 233 slices for the FE. Simulation-based evaluation shows the mean (std) of uniformity to be 49.98% (1.58%), of uniqueness to be 50.00% (1.58%), and of reliability to be 1.26% (2.88%). The LWE concrete hardness estimator guarantees that a successful ML attack of the lattice PUF will require the infeasible 2^128 CPU operations. Several classes of empirical ML attacks, including support vector machine, logistic regression, and deep neural networks, are used: in all attacks, the prediction error remains above 49.76% after 1 million training CRPs.
Random Bias Initialization Improves Quantized Training
Venue: arXiv cs.LG 2020
Year: 2020

Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training and binary training. We bring some insights about why this accuracy drop exists and call for a better understanding of binary network geometry. We start with analyzing full-precision neural networks with ReLU activation and compare it with its binarized version. This comparison suggests to initialize networks with random bias, a counter-intuitive remedy.
Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
Venue: arXiv cs.LG 2020
Year: 2020

We consider a deep ReLU / Leaky ReLU student network trained from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). The student network is \emph{over-realized}: at each layer $l$, the number $n_l$ of student nodes is more than that ($m_l$) of teacher. Under mild conditions on dataset and teacher network, we prove that when the gradient is small at every data sample, each teacher node is \emph{specialized} by at least one student node \emph{at the lowest layer}. For two-layer network, such specialization can be achieved by training on any dataset of \emph{polynomial} size $\mathcal{O}( K^{5/2} d^3 ε^{-1})$. until the gradient magnitude drops to $\mathcal{O}(ε/K^{3/2}\sqrt{d})$. Here $d$ is the input dimension, $K = m_1 + n_1$ is the total number of neurons in the lowest layer of teacher and student. Note that we require a specific form of data augmentation and the sample complexity includes the additional data generated from augmentation. To our best knowledge, we are the first to give polynomial sample complexity for student specialization of training two-layer (Leaky) ReLU networks with finite depth and width in teacher-student setting, and finite complexity for the lowest layer specialization in multi-layer case, without parametric assumption of the input (like Gaussian). Our theory suggests that teacher nodes with large fan-out weights get specialized first when the gradient is still large, while others are specialized with small gradient, which suggests inductive bias in training. This shapes the stage of training as empirically observed in multiple previous works. Experiments on synthetic and CIFAR10 verify our findings. The code is released in https://github.com/facebookresearch/luckmatters.
Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis
Venue: arXiv cs.LG 2020
Year: 2020

In this paper we explore the richness of information captured by the latent space of a vision-based generative model. The model combines unsupervised generative learning with a task-based performance predictor to learn and to exploit task-relevant object affordances given visual observations from a reaching task, involving a scenario and a stick-like tool. While the learned embedding of the generative model captures factors of variation in 3D tool geometry (e.g. length, width, and shape), the performance predictor identifies sub-manifolds of the embedding that correlate with task success. Within a variety of scenarios, we demonstrate that traversing the latent space via backpropagation from the performance predictor allows us to imagine tools appropriate for the task at hand. Our results indicate that affordances-like the utility for reaching-are encoded along smooth trajectories in latent space. Accessing these emergent affordances by considering only high-level performance criteria (such as task success) enables an agent to manipulate tool geometries in a targeted and deliberate way.
Min-Max Optimization without Gradients: Convergence and Applications to Adversarial ML
Venue: arXiv cs.LG 2020
Year: 2020

In this paper, we study the problem of constrained robust (min-max) optimization ina black-box setting, where the desired optimizer cannot access the gradients of the objective function but may query its values. We present a principled optimization framework, integrating a zeroth-order (ZO) gradient estimator with an alternating projected stochastic gradient descent-ascent method, where the former only requires a small number of function queries and the later needs just one-step descent/ascent update. We show that the proposed framework, referred to as ZO-Min-Max, has a sub-linear convergence rate under mild conditions and scales gracefully with problem size. From an application side, we explore a promising connection between black-box min-max optimization and black-box evasion and poisoning attacks in adversarial machine learning (ML). Our empirical evaluations on these use cases demonstrate the effectiveness of our approach and its scalability to dimensions that prohibit using recent black-box solvers.
Optimal Differential Privacy Composition for Exponential Mechanisms and the Cost of Adaptivity
Venue: arXiv cs.CR 2020
Year: 2020

Composition is one of the most important properties of differential privacy (DP), as it allows algorithm designers to build complex private algorithms from DP primitives. We consider precise composition bounds of the overall privacy loss for exponential mechanisms, one of the fundamental classes of mechanisms in DP. We give explicit formulations of the optimal privacy loss for both the adaptive and non-adaptive settings. For the non-adaptive setting in which each mechanism has the same privacy parameter, we give an efficiently computable formulation of the optimal privacy loss. Furthermore, we show that there is a difference in the privacy loss when the exponential mechanism is chosen adaptively versus non-adaptively. To our knowledge, it was previously unknown whether such a gap existed for any DP mechanisms with fixed privacy parameters, and we demonstrate the gap for a widely used class of mechanism in a natural setting. We then improve upon the best previously known upper bounds for adaptive composition of exponential mechanisms with efficiently computable formulations and show the improvement.
Deep Coordination Graphs
Venue: arXiv cs.LG 2020
Year: 2020

This paper introduces the deep coordination graph (DCG) for collaborative multi-agent reinforcement learning. DCG strikes a flexible trade-off between representational capacity and generalization by factoring the joint value function of all agents according to a coordination graph into payoffs between pairs of agents. The value can be maximized by local message passing along the graph, which allows training of the value function end-to-end with Q-learning. Payoff functions are approximated with deep neural networks that employ parameter sharing and low-rank approximations to significantly improve sample efficiency. We show that DCG can solve predator-prey tasks that highlight the relative overgeneralization pathology, as well as challenging StarCraft II micromanagement tasks.
Towards Standardization of Millimeter Wave Vehicle-to-Vehicle Networks: Open Challenges and Performance Evaluation
Venue: arXiv cs.NI 2020
Year: 2020

IEEE 802.11bd and 3GPP NR V2X represent the new specifications for next generation vehicular networks, exploiting new communication technologies and new spectrum, such as the millimeter wave (mmWave) band, to improve throughput and reduce latency. In this paper, we specifically focus on the challenges that mmWaves introduce for Vehicle-to-Vehicle (V2V) networking, by reviewing the latest standard developments and the issues that 802.11bd and NR V2X will have to address for V2V operations at mmWaves. To the best of our knowledge, our work is the first that considers a full-stack, end-to-end approach for the design of mmWave V2V networks, discussing open issues that span from the physical to the higher layers, and reporting the results of an end-to-end performance evaluation that highlight the potential of mmWaves for V2V communications.
Analysis of error dependencies on NewHope
Venue: arXiv cs.CR 2020
Year: 2020

Among many submissions to the NIST post-quantum cryptography (PQC) project, NewHope is a promising key encapsulation mechanism (KEM) based on the Ring-Learning with errors (Ring-LWE) problem. Since NewHope is an indistinguishability (IND)-chosen ciphertext attack secure KEM by applying the Fujisaki-Okamoto transform to an IND-chosen plaintext attack secure public key encryption, accurate calculation of decryption failure rate (DFR) is required to guarantee resilience against attacks that exploit decryption failures. However, the current upper bound of DFR on NewHope is rather loose because the compression noise, the effect of encoding/decoding of NewHope, and the approximation effect of centered binomial distribution are not fully considered. Furthermore, since NewHope is a Ring-LWE based cryptosystem, there is a problem of error dependency among error coefficients, which makes accurate DFR calculation difficult. In this paper, we derive much tighter upper bound on DFR than the current upper bound using constraint relaxation and union bound. Especially, the above-mentioned factors are all considered in derivation of new upper bound and the centered binomial distribution is not approximated to subgaussian distribution. In addition, since the error dependency is considered, the new upper bound is much closer to the real DFR than the previous upper bound. Furthermore, the new upper bound is parameterized by using Chernoff-Cramer bound in order to facilitate calculation of new upper bound for the parameters of NewHope. Since the new upper bound is much lower than the DFR requirement of PQC, this DFR margin is used to improve the security and bandwidth efficiency of NewHope. As a result, the security level of NewHope is improved by 7.2 % or bandwidth efficiency is improved by 5.9 %.
PubChain: A Decentralized Open-Access Publication Platform with Participants Incentivized by Blockchain Technology
Venue: arXiv cs.CR 2020
Year: 2020

We design and implement Publication Chain (PubChain), a decentralized open-access publication platform built on decentralized and distributed technologies of blockchain and IPFS peer-to-peer file sharing systems. The existing publication platforms have some severe drawbacks. First, instead of promoting widespread knowledge sharing, access to publications on the platforms owned by publishers is often on a fee basis. This drawback of pay wall prevents researchers from "standing on the shoulders of giants". Moreover, the peer review process on most all existing publication platforms (including both open-access and publisher platforms) is prone to be ineffective, since there is no proper incentive to reviewers for performing high-qualified reviews. PubChain is an alternative platform to the existing publication venues aiming to address their drawbacks. No central third-party owns the contents (i.e., papers and reviews) of PubChain. Exploiting blockchain technology, we devise an elaborate incentive scheme on PubChain to incentivize key stakeholders (i.e., authors, readers and reviewers) to participate publication activities on PubChain in a substantive manner by earning credits and rewards through self-motivated interactions. We have performed simulations to investigate the robustness of our proposed incentive scheme against fraudulent publications and reviews. We also have implemented a prototype of PubChain to demonstrate its key concepts.
Pre- and post-processing in quantum-computational hydrologic inverse analysis
Venue: arXiv cs.ET 2020
Year: 2020

It was recently shown that certain subsurface hydrological inverse problems -- here framed as determining the composition of an aquifer from pressure readings -- can be solved on a quantum annealer. However, the quantum annealer performance suffered when solving problems where the aquifer was composed of materials with vastly different permeability, which is often encountered in practice. In this paper, we study why this regime is difficult and use several pre- and post-processing tools to address these issues. This study has three benefits: it improves quantum annealing performance for real-world problems in hydrology, it studies the scaling behavior for these problems (which were previously studied at a fixed size), and it elucidates a challenging class of problems that are amenable to quantum annealers.
Potential Key Technologies for 6G Mobile Communications
Venue: arXiv cs.IT 2020
Year: 2020

The standard development of 5G wireless communication culminated between 2017 and 2019, followed by the worldwide deployment of 5G networks, which is expected to result in very high data rate for enhanced mobile broadband, support ultra-reliable and low-latency services and accommodate massive number of connections. Research attention is shifting to future generation of wireless communications, for instance, beyond 5G or 6G. Unlike previous papers, which discussed the use cases, deployment scenarios, or new network architectures of 6G in depth, this paper focuses on a few potential technologies for 6G wireless communications, all of which represent certain fundamental breakthrough at the physical layer-technical hardcore of any new generation of wireless communications. Some of them, such as holographic radio, terahertz communication, large intelligent surface, and orbital angular momentum, are of revolutionary nature and many related studies are still at their scientific exploration stage. Several technical areas, such as advanced channel coding and modulation, visible light communication, and advanced duplex, while having been studied, may find more opportunities in 6G.
Beamspace Channel Estimation for Massive MIMO mmWave Systems: Algorithm and VLSI Design
Venue: arXiv cs.IT 2020
Year: 2020

Millimeter-wave (mmWave) communication in combination with massive multiuser multiple-input multiple-output (MU-MIMO) enables high-bandwidth data transmission to multiple users in the same time-frequency resource. The strong path loss of wave propagation at such high frequencies necessitates accurate channel state information to ensure reliable data transmission. We propose a novel channel estimation algorithm called BEAmspace CHannel EStimation (BEACHES), which leverages the fact that wave propagation at mmWave frequencies is predominantly directional. BEACHES adaptively denoises the channel vectors in the beamspace domain using an adaptive shrinkage procedure that relies on Stein's unbiased risk estimator (SURE). Simulation results for line-of-sight (LoS) and non-LoS mmWave channels reveal that BEACHES performs on par with state-of-the-art channel estimation methods while requiring orders-of-magnitude lower complexity. To demonstrate the effectiveness of BEACHES in practice, we develop a very large-scale integration (VLSI) architecture and provide field-programmable gate array (FPGA) implementation results. Our results show that adaptive channel denoising can be performed at high throughput and in a hardware-friendly manner for massive MU-MIMO mmWave systems with hundreds of antennas.
Efficient Graph Generation with Graph Recurrent Attention Networks
Venue: arXiv cs.LG 2020
Year: 2020

We propose a new family of efficient and expressive deep generative models of graphs, called Graph Recurrent Attention Networks (GRANs). Our model generates graphs one block of nodes and associated edges at a time. The block size and sampling stride allow us to trade off sample quality for efficiency. Compared to previous RNN-based graph generative models, our framework better captures the auto-regressive conditioning between the already-generated and to-be-generated parts of the graph using Graph Neural Networks (GNNs) with attention. This not only reduces the dependency on node ordering but also bypasses the long-term bottleneck caused by the sequential nature of RNNs. Moreover, we parameterize the output distribution per block using a mixture of Bernoulli, which captures the correlations among generated edges within the block. Finally, we propose to handle node orderings in generation by marginalizing over a family of canonical orderings. On standard benchmarks, we achieve state-of-the-art time efficiency and sample quality compared to previous models. Additionally, we show our model is capable of generating large graphs of up to 5K nodes with good quality. To the best of our knowledge, GRAN is the first deep graph generative model that can scale to this size. Our code is released at: https://github.com/lrjconan/GRAN.
SharPer: Sharding Permissioned Blockchains Over Network Clusters
Venue: arXiv cs.DB 2020
Year: 2020

Scalability is one of the main roadblocks to business adoption of blockchain systems. Despite recent intensive research on using sharding techniques to enhance the scalability of blockchain systems, existing solutions do not efficiently address cross-shard transactions. In this paper, we introduce SharPer, a permissioned blockchain system that improves scalability by clustering (partitioning) the nodes and assigning different data shards to different clusters where each data shard is replicated on the nodes of a cluster. SharPer supports both intra-shard and cross-shard transactions and processes intra-shard transactions of different clusters as well as cross-shard transactions with non-overlapping clusters simultaneously. In SharPer, the blockchain ledger is formed as a directed acyclic graph where each cluster maintains only a view of the ledger. SharPer also incorporates a flattened protocol to establish consensus among clusters on the order of cross-shard transactions. The experimental results reveal the efficiency of SharPer in terms of performance and scalability especially in workloads with a low percentage of cross-shard transactions.
An Algebraic Attack on Rank Metric Code-Based Cryptosystems
Venue: arXiv cs.CR 2020
Year: 2020

The Rank metric decoding problem is the main problem considered in cryptography based on codes in the rank metric. Very efficient schemes based on this problem or quasi-cyclic versions of it have been proposed recently, such as those in the submissions ROLLO and RQC currently at the second round of the NIST Post-Quantum Cryptography Standardization Process. While combinatorial attacks on this problem have been extensively studied and seem now well understood, the situation is not as satisfactory for algebraic attacks, for which previous work essentially suggested that they were ineffective for cryptographic parameters. In this paper, starting from Ourivski and Johansson's algebraic modelling of the problem into a system of polynomial equations, we show how to augment this system with easily computed equations so that the augmented system is solved much faster via Groebner bases. This happens because the augmented system has solving degree $r$, $r+1$ or $r+2$ depending on the parameters, where $r$ is the rank weight, which we show by extending results from Verbel et al. (PQCrypto 2019) on systems arising from the MinRank problem; with target rank $r$, Verbel et al. lower the solving degree to $r+2$, and even less for some favorable instances that they call superdetermined. We give complexity bounds for this approach as well as practical timings of an implementation using Magma. This improves upon the previously known complexity estimates for both Groebner basis and (non-quantum) combinatorial approaches, and for example leads to an attack in 200 bits on ROLLO-I-256 whose claimed security was 256 bits.