OpenRaiser/Intern-Atlas huggingface dataset:
Bayesian experimental design using regularized determinantal point processes
Venue: arXiv cs.LG 2019
Year: 2019

In experimental design, we are given $n$ vectors in $d$ dimensions, and our goal is to select $k\ll n$ of them to perform expensive measurements, e.g., to obtain labels/responses, for a linear regression task. Many statistical criteria have been proposed for choosing the optimal design, with popular choices including A- and D-optimality. If prior knowledge is given, typically in the form of a $d\times d$ precision matrix $\mathbf A$, then all of the criteria can be extended to incorporate that information via a Bayesian framework. In this paper, we demonstrate a new fundamental connection between Bayesian experimental design and determinantal point processes, the latter being widely used for sampling diverse subsets of data. We use this connection to develop new efficient algorithms for finding $(1+ε)$-approximations of optimal designs under four optimality criteria: A, C, D and V. Our algorithms can achieve this when the desired subset size $k$ is $Ω(\frac{d_{\mathbf A}}ε + \frac{\log 1/ε}{ε^2})$, where $d_{\mathbf A}\leq d$ is the $\mathbf A$-effective dimension, which can often be much smaller than $d$. Our results offer direct improvements over a number of prior works, for both Bayesian and classical experimental design, in terms of algorithm efficiency, approximation quality, and range of applicable criteria.
SymNet: Symmetrical Filters in Convolutional Neural Networks
Venue: arXiv cs.CV 2019
Year: 2019

Symmetry is present in nature and science. In image processing, kernels for spatial filtering possess some symmetry (e.g. Sobel operators, Gaussian, Laplacian). Convolutional layers in artificial feed-forward neural networks have typically considered the kernel weights without any constraint. In this paper, we propose to investigate the impact of a symmetry constraint in convolutional layers for image classification tasks, taking our inspiration from the processes involved in the primary visual cortex and common image processing techniques. The goal is to assess the extent to which it is possible to enforce symmetrical constraints on the filters throughout the training process of a convolutional neural network (CNN) by modifying the weight update preformed during the backpropagation algorithm and to evaluate the change in performance. The main hypothesis of this paper is that the symmetrical constraint reduces the number of free parameters in the network, and it is able to achieve near identical performance to the modern methodology of training. In particular, we address the following cases: x/y-axis symmetry, point reflection, and anti-point reflection. The performance has been evaluated on four databases of images. The results support the conclusion that while random weights offer more freedom to the model, the symmetry constraint provides a similar level of performance while decreasing substantially the number of free parameters in the model. Such an approach can be valuable in phase-sensitive applications that require a linear phase property throughout the feature extraction process.
On the LoRa Modulation for IoT: Waveform Properties and Spectral Analysis
Venue: arXiv cs.NI 2019
Year: 2019

An important modulation technique for Internet of Things (IoT) is the one proposed by the LoRa allianceTM. In this paper we analyze the M-ary LoRa modulation in the time and frequency domains. First, we provide the signal description in the time domain, and show that LoRa is a memoryless continuous phase modulation. The cross-correlation between the transmitted waveforms is determined, proving that LoRa can be considered approximately an orthogonal modulation only for large M. Then, we investigate the spectral characteristics of the signal modulated by random data, obtaining a closed-form expression of the spectrum in terms of Fresnel functions. Quite surprisingly, we found that LoRa has both continuous and discrete spectra, with the discrete spectrum containing exactly a fraction 1/M of the total signal power.
Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices
Venue: arXiv cs.LG 2019
Year: 2019

Training deep learning models on mobile devices recently becomes possible, because of increasing computation power on mobile hardware and the advantages of enabling high user experiences. Most of the existing work on machine learning at mobile devices is focused on the inference of deep learning models (particularly convolutional neural network and recurrent neural network), but not training. The performance characterization of training deep learning models on mobile devices is largely unexplored, although understanding the performance characterization is critical for designing and implementing deep learning models on mobile devices. In this paper, we perform a variety of experiments on a representative mobile device (the NVIDIA TX2) to study the performance of training deep learning models. We introduce a benchmark suite and tools to study performance of training deep learning models on mobile devices, from the perspectives of memory consumption, hardware utilization, and power consumption. The tools can correlate performance results with fine-grained operations in deep learning models, providing capabilities to capture performance variance and problems at a fine granularity. We reveal interesting performance problems and opportunities, including under-utilization of heterogeneous hardware, large energy consumption of the memory, and high predictability of workload characterization. Based on the performance analysis, we suggest interesting research directions.
Artificial Noisy MIMO Systems under Correlated Scattering Rayleigh Fading -- A Physical Layer Security Approach
Venue: arXiv cs.IT 2019
Year: 2019

The existing investigations on artificial noise (AN) security systems assumed that only null spaces is used to send AN signals, and all eigen-subchannels should be used to transmit messages. Our previous work proposed an AN scheme that allocates some of eigen-subchannels to transmit AN signals for improving secrecy rates. Nevertheless, our previous work considered only uncorrelated MIMO Rayleigh fading channels. In fact, the correlations among antennas exist in realistic scattering channel environments. In this paper, we extend our previous AN scheme to spatially correlated Rayleigh fading channels at both legitimate receiver- and eavesdropper-sides and derive an exact theoretical expression for the ergodic secrecy rate of the AN scheme, along with an approximate analysis. Both numerical and simulation results show that the proposed AN scheme offers a higher ergodic secrecy rate than the existing schemes, revealing a fact that the correlation among eavesdropper's antennas can potentially improve the secrecy rate of an MIMO system.
Associative Convolutional Layers
Venue: arXiv cs.LG 2019
Year: 2019

Motivated by the necessity for parameter efficiency in distributed machine learning and AI-enabled edge devices, we provide a general and easy to implement method for significantly reducing the number of parameters of Convolutional Neural Networks (CNNs), during both the training and inference phases. We introduce a simple auxiliary neural network which can generate the convolutional filters of any CNN architecture from a low dimensional latent space. This auxiliary neural network, which we call "Convolutional Slice Generator" (CSG), is unique to the network and provides the association between its convolutional layers. During the training of the CNN, instead of training the filters of the convolutional layers, only the parameters of the CSG and their corresponding "code vectors" are trained. This results in a significant reduction of the number of parameters due to the fact that the CNN can be fully represented using only the parameters of the CSG, the code vectors, the fully connected layers, and the architecture of the CNN. We evaluate our approach by applying it to ResNet and DenseNet models when trained on CIFAR-10 and ImageNet datasets. While reducing the number of parameters by $\approx 2 \times$ on average, the accuracies of these networks remain within 1$\%$ of their original counterparts and in some cases there is an increase in the accuracy.
General Linear Group Action on Tensors: A Candidate for Post-Quantum Cryptography
Venue: arXiv cs.CR 2019
Year: 2019

Starting from the one-way group action framework of Brassard and Yung (Crypto '90), we revisit building cryptography based on group actions. Several previous candidates for one-way group actions no longer stand, due to progress both on classical algorithms (e.g., graph isomorphism) and quantum algorithms (e.g., discrete logarithm). We propose the general linear group action on tensors as a new candidate to build cryptography based on group actions. Recent works (Futorny--Grochow--Sergeichuk, Lin. Alg. Appl., 2019) suggest that the underlying algorithmic problem, the tensor isomorphism problem, is the hardest one among several isomorphism testing problems arising from areas including coding theory, computational group theory, and multivariate cryptography. We present evidence to justify the viability of this proposal from comprehensive study of the state-of-art heuristic algorithms, theoretical algorithms, and hardness results, as well as quantum algorithms. We then introduce a new notion called pseudorandom group actions to further develop group-action based cryptography. Briefly speaking, given a group $G$ acting on a set $S$, we assume that it is hard to distinguish two distributions of $(s, t)$ either uniformly chosen from $S\times S$, or where $s$ is randomly chosen from $S$ and $t$ is the result of applying a random group action of $g\in G$ on $s$. This subsumes the classical decisional Diffie-Hellman assumption when specialized to a particular group action. We carefully analyze various attack strategies that support the general linear group action on tensors as a candidate for this assumption. Finally, we establish the quantum security of several cryptographic primitives based on the one-way group action assumption and the pseudorandom group action assumption.
PAN: Projective Adversarial Network for Medical Image Segmentation
Venue: arXiv cs.CV 2019
Year: 2019

Adversarial learning has been proven to be effective for capturing long-range and high-level label consistencies in semantic segmentation. Unique to medical imaging, capturing 3D semantics in an effective yet computationally efficient way remains an open problem. In this study, we address this computational burden by proposing a novel projective adversarial network, called PAN, which incorporates high-level 3D information through 2D projections. Furthermore, we introduce an attention module into our framework that helps for a selective integration of global information directly from our segmentor to our adversarial network. For the clinical application we chose pancreas segmentation from CT scans. Our proposed framework achieved state-of-the-art performance without adding to the complexity of the segmentor.
Reinforcement Learning for Channel Coding: Learned Bit-Flipping Decoding
Venue: arXiv cs.IT 2019
Year: 2019

In this paper, we use reinforcement learning to find effective decoding strategies for binary linear codes. We start by reviewing several iterative decoding algorithms that involve a decision-making process at each step, including bit-flipping (BF) decoding, residual belief propagation, and anchor decoding. We then illustrate how such algorithms can be mapped to Markov decision processes allowing for data-driven learning of optimal decision strategies, rather than basing decisions on heuristics or intuition. As a case study, we consider BF decoding for both the binary symmetric and additive white Gaussian noise channel. Our results show that learned BF decoders can offer a range of performance-complexity trade-offs for the considered Reed-Muller and BCH codes, and achieve near-optimal performance in some cases. We also demonstrate learning convergence speed-ups when biasing the learning process towards correct decoding decisions, as opposed to relying only on random explorations and past knowledge.
Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer
Venue: arXiv cs.LG 2019
Year: 2019

We focus on the problem of teaching a robot to solve tasks presented sequentially, i.e., in a continual learning scenario. The robot should be able to solve all tasks it has encountered, without forgetting past tasks. We provide preliminary work on applying Reinforcement Learning to such setting, on 2D navigation tasks for a 3 wheel omni-directional robot. Our approach takes advantage of state representation learning and policy distillation. Policies are trained using learned features as input, rather than raw observations, allowing better sample efficiency. Policy distillation is used to combine multiple policies into a single one that solves all encountered tasks.
Optimizing Pipelined Computation and Communication for Latency-Constrained Edge Learning
Venue: arXiv cs.LG 2019
Year: 2019

Consider a device that is connected to an edge processor via a communication channel. The device holds local data that is to be offloaded to the edge processor so as to train a machine learning model, e.g., for regression or classification. Transmission of the data to the learning processor, as well as training based on Stochastic Gradient Descent (SGD), must be both completed within a time limit. Assuming that communication and computation can be pipelined, this letter investigates the optimal choice for the packet payload size, given the overhead of each data packet transmission and the ratio between the computation and the communication rates. This amounts to a tradeoff between bias and variance, since communicating the entire data set first reduces the bias of the training process but it may not leave sufficient time for learning. Analytical bounds on the expected optimality gap are derived so as to enable an effective optimization, which is validated in numerical results.
BasisConv: A method for compressed representation and learning in CNNs
Venue: arXiv cs.LG 2019
Year: 2019

It is well known that Convolutional Neural Networks (CNNs) have significant redundancy in their filter weights. Various methods have been proposed in the literature to compress trained CNNs. These include techniques like pruning weights, filter quantization and representing filters in terms of a basis functions. Our approach falls in this latter class of strategies, but is distinct in that that we show both compressed learning and representation can be achieved without significant modifications of popular CNN architectures. Specifically, any convolution layer of the CNN is easily replaced by two successive convolution layers: the first is a set of fixed filters (that represent the knowledge space of the entire layer and do not change), which is followed by a layer of one-dimensional filters (that represent the learned knowledge in this space). For the pre-trained networks, the fixed layer is just the truncated eigen-decompositions of the original filters. The 1D filters are initialized as the weights of linear combination, but are fine-tuned to recover any performance loss due to the truncation. For training networks from scratch, we use a set of random orthogonal fixed filters (that never change), and learn the 1D weight vector directly from the labeled data. Our method substantially reduces i) the number of learnable parameters during training, and ii) the number of multiplication operations and filter storage requirements during implementation. It does so without requiring any special operators in the convolution layer, and extends to all known popular CNN architectures. We apply our method to four well known network architectures trained with three different data sets. Results show a consistent reduction in i) the number of operations by up to a factor of 5, and ii) number of learnable parameters by up to a factor of 18, with less than 3% drop in performance on the CIFAR100 dataset.
Automatic Multi-Sensor Extrinsic Calibration for Mobile Robots
Venue: arXiv cs.RO 2019
Year: 2019

In order to fuse measurements from multiple sensors mounted on a mobile robot, it is needed to express them in a common reference system through their relative spatial transformations. In this paper, we present a method to estimate the full 6DoF extrinsic calibration parameters of multiple heterogeneous sensors (Lidars, Depth and RGB cameras) suitable for automatic execution on a mobile robot. Our method computes the 2D calibration parameters (x, y, yaw) through a motion-based approach, while for the remaining 3 parameters (z, pitch, roll) it requires the observation of the ground plane for a short period of time. What set this proposal apart from others is that: i) all calibration parameters are initialized in closed form, and ii) the scale ambiguity inherent to motion estimation from a monocular camera is explicitly handled, enabling the combination of these sensors and metric ones (Lidars, stereo rigs, etc.) within the same optimization framework. %Additionally, outlier observations arising from local sensor drift are automatically detected and removed from the calibration process. We provide a formal definition of the problem, as well as of the contributed method, for which a C++ implementation has been made publicly available. The suitability of the method has been assessed in simulation an with real data from indoor and outdoor scenarios. Finally, improvements over state-of-the-art motion-based calibration proposals are shown through experimental evaluation.
Variance-reduced $Q$-learning is minimax optimal
Venue: arXiv cs.LG 2019
Year: 2019

We introduce and analyze a form of variance-reduced $Q$-learning. For $γ$-discounted MDPs with finite state space $\mathcal{X}$ and action space $\mathcal{U}$, we prove that it yields an $ε$-accurate estimate of the optimal $Q$-function in the $\ell_\infty$-norm using $\mathcal{O} \left(\left(\frac{D}{ ε^2 (1-γ)^3} \right) \; \log \left( \frac{D}{(1-γ)} \right) \right)$ samples, where $D = |\mathcal{X}| \times |\mathcal{U}|$. This guarantee matches known minimax lower bounds up to a logarithmic factor in the discount complexity. In contrast, our past work shows that ordinary $Q$-learning has worst-case quartic scaling in the discount complexity.
Measuring and exploiting the cloud consolidation of the Web
Venue: arXiv cs.NI 2019
Year: 2019

We present measurements showing that the top one million most popular Web domains are reachable within 13ms (in the median) from a collective of just 12 cloud data centers. We explore the consequences of this Web "consolidation", focusing on its potential for speeding up the evolution of the Web. That most popular services reside in or near a small number of data centers implies that new application and transport technologies can be rapidly deployed for these Web services, without the involvement of their operators. We show how this may be achieved by orchestrating a handful of reverse proxies deployed in the same data centers, with new technologies deployed at these proxies being nearly as effective as deploying them directly to the Web servers. We present early measurements of this approach, demonstrating a >50% reduction in Web page load times for users with high latencies to Web servers. We also show that this model, using a small number of proxies, can be surprisingly competitive with extensive CDN deployments, especially in geographies with high last-mile latencies.
Second-best Beam-Alignment via Bayesian Multi-Armed Bandits
Venue: arXiv cs.IT 2019
Year: 2019

Millimeter-wave (mm-wave) systems rely on narrow-beams to cope with the severe signal attenuation in the mm-wave frequency band. However, susceptibility to beam mis-alignment due to mobility or blockage requires the use of beam-alignment schemes, with huge cost in terms of overhead and use of system resources. In this paper, a beam-alignment scheme is proposed based on Bayesian multi-armed bandits, with the goal to maximize the alignment probability and the data-communication throughput. A Bayesian approach is proposed, by considering the state as a posterior distribution over angles of arrival (AoA) and of departure (AoD), given the history of feedback signaling and of beam pairs scanned by the base-station (BS) and the user-end (UE). A simplified sufficient statistics for optimal control is derived, in the form of preference of BS-UE beam pairs. By bounding a value function, the second-best preference policy is derived, which strikes an optimal balance between exploration and exploitation by selecting the beam pair with the current second-best preference. Through Monte-Carlo simulation with analog beamforming, the superior performance of the second-best preference policy is demonstrated in comparison to existing schemes based on first-best preference, linear Thompson sampling, and upper confidence bounds, with up to 7%, 10% and 30% improvements in alignment probability, respectively.
Enhancing the Vertical Mobility of a Robot Hexapod Using Microspines
Venue: arXiv cs.RO 2019
Year: 2019

Modern climbing robots have risen to great heights, but mechanisms meant to scale cliffs often locomote slowly and over-cautiously on level ground. Here we introduce T-RHex, an iteration on the classic cockroach-inspired hexapod that has been augmented with microspine feet for climbing. T-RHex is a mechanically intelligent platform capable of efficient locomotion on ground with added climbing abilities. The legs integrate the compliance required for the microspines with the compliance required for locomotion in order to simplify the design and reduce mass. The microspine fabrication is simplified by embedding the spines during an additive manufacturing process. We present results that show that the addition of microspines to the T-RHex platform greatly increases the maximum slope that the robot is able to statically hang on (up to a 45 degree overhang) and ascend (up to 55 degrees) without sacrificing ground mobility.
Weakly-supervised Compositional FeatureAggregation for Few-shot Recognition
Venue: arXiv cs.CV 2019
Year: 2019

Learning from a few examples is a challenging task for machine learning. While recent progress has been made for this problem, most of the existing methods ignore the compositionality in visual concept representation (e.g. objects are built from parts or composed of semantic attributes), which is key to the human ability to easily learn from a small number of examples. To enhance the few-shot learning models with compositionality, in this paper we present the simple yet powerful Compositional Feature Aggregation (CFA) module as a weakly-supervised regularization for deep networks. Given the deep feature maps extracted from the input, our CFA module first disentangles the feature space into disjoint semantic subspaces that model different attributes, and then bilinearly aggregates the local features within each of these subspaces. CFA explicitly regularizes the representation with both semantic and spatial compositionality to produce discriminative representations for few-shot recognition tasks. Moreover, our method does not need any supervision for attributes and object parts during training, thus can be conveniently plugged into existing models for end-to-end optimization while keeping the model size and computation cost nearly the same. Extensive experiments on few-shot image classification and action recognition tasks demonstrate that our method provides substantial improvements over recent state-of-the-art methods.