OpenRaiser/Intern-Atlas huggingface dataset:
Disentangling feature and lazy training in deep neural networks
Venue: arXiv cs.LG 2020
Year: 2020

Two distinct limits for deep learning have been derived as the network width $h\rightarrow \infty$, depending on how the weights of the last layer scale with $h$. In the Neural Tangent Kernel (NTK) limit, the dynamics becomes linear in the weights and is described by a frozen kernel $Θ$. By contrast, in the Mean-Field limit, the dynamics can be expressed in terms of the distribution of the parameters associated with a neuron, that follows a partial differential equation. In this work we consider deep networks where the weights in the last layer scale as $αh^{-1/2}$ at initialization. By varying $α$ and $h$, we probe the crossover between the two limits. We observe the previously identified regimes of lazy training and feature training. In the lazy-training regime, the dynamics is almost linear and the NTK barely changes after initialization. The feature-training regime includes the mean-field formulation as a limiting case and is characterized by a kernel that evolves in time, and learns some features. We perform numerical experiments on MNIST, Fashion-MNIST, EMNIST and CIFAR10 and consider various architectures. We find that (i) The two regimes are separated by an $α^*$ that scales as $h^{-1/2}$. (ii) Network architecture and data structure play an important role in determining which regime is better: in our tests, fully-connected networks perform generally better in the lazy-training regime, unlike convolutional networks. (iii) In both regimes, the fluctuations $δF$ induced on the learned function by initial conditions decay as $δF\sim 1/\sqrt{h}$, leading to a performance that increases with $h$. The same improvement can also be obtained at an intermediate width by ensemble-averaging several networks. (iv) In the feature-training regime we identify a time scale $t_1\sim\sqrt{h}α$, such that for $t\ll t_1$ the dynamics is linear.
A Grasping-centered Analysis for Cloth Manipulation
Venue: arXiv cs.RO 2020
Year: 2020

Compliant and soft hands have gained a lot of attention in the past decade because of their ability to adapt to the shape of the objects, increasing their effectiveness for grasping. However, when it comes to grasping highly flexible objects such as textiles, we face the dual problem: it is the object that will adapt to the shape of the hand or gripper. In this context, the classic grasp analysis or grasping taxonomies are not suitable for describing textile objects grasps. This work proposes a novel definition of textile object grasps that abstracts from the robotic embodiment or hand shape and recovers concepts from the early neuroscience literature on hand prehension skills. This framework enables us to identify what grasps have been used in literature until now to perform robotic cloth manipulation, and allows for a precise definition of all the tasks that have been tackled in terms of manipulation primitives based on regrasps. In addition, we also review what grippers have been used. Our analysis shows how the vast majority of cloth manipulations have relied only on one type of grasp, and at the same time we identify several tasks that need more variety of grasp types to be executed successfully. Our framework is generic, provides a classification of cloth manipulation primitives and can inspire gripper design and benchmark construction for cloth manipulation.
A Strong Baseline and Batch Normalization Neck for Deep Person Re-identification
Venue: arXiv cs.CV 2020
Year: 2020

This study explores a simple but strong baseline for person re-identification (ReID). Person ReID with deep neural networks has progressed and achieved high performance in recent years. However, many state-of-the-art methods design complex network structures and concatenate multi-branch features. In the literature, some effective training tricks briefly appear in several papers or source codes. The present study collects and evaluates these effective training tricks in person ReID. By combining these tricks, the model achieves 94.5% rank-1 and 85.9% mean average precision on Market1501 with only using the global features of ResNet50. The performance surpasses all existing global- and part-based baselines in person ReID. We propose a novel neck structure named as batch normalization neck (BNNeck). BNNeck adds a batch normalization layer after global pooling layer to separate metric and classification losses into two different feature spaces because we observe they are inconsistent in one embedding space. Extended experiments show that BNNeck can boost the baseline, and our baseline can improve the performance of existing state-of-the-art methods. Our codes and models are available at: https://github.com/michuanhaohao/reid-strong-baseline.
Complex phase retrieval from subgaussian measurements
Venue: arXiv cs.IT 2020
Year: 2020

Phase retrieval refers to the problem of reconstructing an unknown vector $x_0 \in \mathbb{C}^n$ or $x_0 \in \mathbb{R}^n $ from $m$ measurements of the form $y_i = \big\vert \langle ξ^{\left(i\right)}, x_0 \rangle \big\vert^2 $, where $ \left\{ ξ^{\left(i\right)} \right\}^m_{i=1} \subset \mathbb{C}^m $ are known measurement vectors. While Gaussian measurements allow for recovery of arbitrary signals provided the number of measurements scales at least linearly in the number of dimensions, it has been shown that ambiguities may arise for certain other classes of measurements $ \left\{ ξ^{\left(i\right)} \right\}^{m}_{i=1}$ such as Bernoulli measurements or Fourier measurements. In this paper, we will prove that even when a subgaussian vector $ ξ^{\left(i\right)} \in \mathbb{C}^m $ does not fulfill a small-ball probability assumption, the PhaseLift method is still able to reconstruct a large class of signals $x_0 \in \mathbb{R}^n$ from the measurements. This extends recent work by Krahmer and Liu from the real-valued to the complex-valued case. However, our proof strategy is quite different and we expect some of the new proof ideas to be useful in several other measurement scenarios as well. We then extend our results $x_0 \in \mathbb{C}^n $ up to an additional assumption which, as we show, is necessary.
Improving the robustness of ImageNet classifiers using elements of human visual cognition
Venue: arXiv cs.CV 2020
Year: 2020

We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale. As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models against small-norm adversarial perturbations under some threat models. It does not, however, improve the robustness against more natural, and typically larger, perturbations. Learning more robust features during training appears to be necessary for robustness in this second sense. We show that features derived from a model that was encouraged to learn global, shape-based representations (Geirhos et al., 2019) do not only improve the robustness against natural perturbations, but when used in conjunction with an episodic memory, they also provide additional robustness against adversarial perturbations. Finally, we address three important design choices for the episodic memory: memory size, dimensionality of the memories and the retrieval method. We show that to make the episodic memory more compact, it is preferable to reduce the number of memories by clustering them, instead of reducing their dimensionality.
Predicting Motion of Vulnerable Road Users using High-Definition Maps and Efficient ConvNets
Venue: arXiv cs.RO 2020
Year: 2020

Following detection and tracking of traffic actors, prediction of their future motion is the next critical component of a self-driving vehicle (SDV) technology, allowing the SDV to operate safely and efficiently in its environment. This is particularly important when it comes to vulnerable road users (VRUs), such as pedestrians and bicyclists. These actors need to be handled with special care due to an increased risk of injury, as well as the fact that their behavior is less predictable than that of motorized actors. To address this issue, in the current study we present a deep learning-based method for predicting VRU movement, where we rasterize high-definition maps and actor's surroundings into a bird's-eye view image used as an input to deep convolutional networks. In addition, we propose a fast architecture suitable for real-time inference, and perform an ablation study of various rasterization approaches to find the optimal choice for accurate prediction. The results strongly indicate benefits of using the proposed approach for motion prediction of VRUs, both in terms of accuracy and latency.
A New Achievable Rate-Distortion Region for Distributed Source Coding
Venue: arXiv cs.IT 2020
Year: 2020

In this work, lossy distributed compression of pairs of correlated sources is considered. Conventionally, Shannon's random coding arguments -- using randomly generated unstructured codebooks whose blocklength is taken to be asymptotically large -- are used to derive achievability results. However, it was recently observed that in various multi-terminal communications scenarios, using random codes with constant finite blocklength may lead to improved achievable regions compared to the conventional approach. In other words, in some network communication scenarios, there is a finite optimal value in the blocklength of the randomly generated code used for distributed processing of information sources. Motivated by this, a coding scheme is proposed which consists of two codebook layers: i) the primary codebook which has constant finite blocklength, and ii) the secondary codebook whose blocklength is taken to be asymptotically large. The achievable region is analyzed in two steps. In the first step, a characterization of the achievable region is derived using information measures which are functions of multi-letter probability distributions. In the next step, a computable single-letter inner-bound to the achievable region is derived. It is shown through several examples that the resulting rate-distortion region is strictly larger than the Berger Tung achievable region.
LPD Communication: A Sequential Change-Point Detection Perspective
Venue: arXiv cs.IT 2020
Year: 2020

In this paper, we establish a framework for low probability of detection (LPD) communication from a sequential change-point detection (SCPD) perspective, where a transmitter, Alice, wants to hide her signal transmission to a receiver, Bob, under the surveillance of an adversary, Willie. The new framework facilitates to model LPD communication and further evaluate its performance under the condition that Willie has no prior knowledge on when the transmission from Alice starts and that Willie wants to detect the existence of the communication as quickly as possible in real-time manner. We consider three different sequential tests for Willie, i.e., the Shewhart test, the cumulative sum (CUSUM) test, and the Shiryaev-Roberts (SR) test, to model the detection procedure. Communication is said to be covert if it stops before detection by Willie with high probability. Covert probability defined as the probability that Willie is not alerted during the communication procedure is investigated. We formulate an optimization problem aimed at finding the transmit power and transmission duration such that the total amount of information that can be transmitted is maximized subject to a high covert probability. Under Shewhart test, closed-form approximations of the optimal transmit power and transmission duration are derived, which well approximate the solutions obtained from exhaustive search. As for CUSUM and SR tests, we provide an effective algorithm to search the optimal solution. Numeric results are presented to show the performance of LPD communication.
A Fourier Perspective on Model Robustness in Computer Vision
Venue: arXiv cs.LG 2020
Year: 2020

Achieving robustness to distributional shift is a longstanding and challenging goal of computer vision. Data augmentation is a commonly used approach for improving robustness, however robustness gains are typically not uniform across corruption types. Indeed increasing performance in the presence of random noise is often met with reduced performance on other corruptions such as contrast change. Understanding when and why these sorts of trade-offs occur is a crucial step towards mitigating them. Towards this end, we investigate recently observed trade-offs caused by Gaussian data augmentation and adversarial training. We find that both methods improve robustness to corruptions that are concentrated in the high frequency domain while reducing robustness to corruptions that are concentrated in the low frequency domain. This suggests that one way to mitigate these trade-offs via data augmentation is to use a more diverse set of augmentations. Towards this end we observe that AutoAugment, a recently proposed data augmentation policy optimized for clean accuracy, achieves state-of-the-art robustness on the CIFAR-10-C benchmark.
Acute Lymphoblastic Leukemia Classification from Microscopic Images using Convolutional Neural Networks
Venue: arXiv cs.CV 2020
Year: 2020

Examining blood microscopic images for leukemia is necessary when expensive equipment for flow cytometry is unavailable. Automated systems can ease the burden on medical experts for performing this examination and may be especially helpful to quickly screen a large number of patients. We present a simple, yet effective classification approach using a ResNeXt convolutional neural network with Squeeze-and-Excitation modules. The approach was evaluated in the C-NMC online challenge and achieves a weighted F1-score of 88.91% on the test set. Code is available at https://github.com/jprellberg/isbi2019cancer
Simple Mechanisms for Profit Maximization in Multi-item Auctions
Venue: arXiv cs.GT 2020
Year: 2020

We study a classical Bayesian mechanism design problem where a seller is selling multiple items to multiple buyers. We consider the case where the seller has costs to produce the items, and these costs are private information to the seller. How can the seller design a mechanism to maximize her profit? Two well-studied problems, revenue maximization in multi-item auctions and signaling in ad auctions, are special cases of our problem. We show that there exists a simple mechanism whose profit is at least $\frac{1}{44}$ of the optimal profit for multiple buyers with matroid-rank valuation functions. When there is a single buyer, the approximation factor is $11$ for general constraint-additive valuations and $6$ for additive valuations. Our result holds even when the seller's costs are correlated across items. We introduce a new class of mechanisms called permit-selling mechanisms. For single buyer case these mechanisms are quite simple: there are two stages. For each item $j$, we create a separate permit that allows the buyer to purchase the item in the second stage. In the first stage, we sell the permits without revealing any information about the costs. In the second stage, the seller reveals all the costs, and the buyer can buy item $j$ by paying the item price if the buyer has purchased the permit for item $j$ in the first stage. We show that either selling the permits separately or as a grand bundle suffices to achieve the constant factor approximation to the optimal profit (6 for additive, and 11 for constrained additive). For multiple buyers, we sell the permits sequentially and obtain the constant factor approximation. Our proof is enabled by constructing a benchmark for the optimal profit by combining a novel dual solution with the existing ex-ante relaxation technique.
Models of Continuous-Time Networks with Tie Decay, Diffusion, and Convection
Venue: arXiv cs.SI 2020
Year: 2020

The study of temporal networks in discrete time has yielded numerous insights into time-dependent networked systems in a wide variety of applications. For many complex systems, however, it is useful to develop continuous-time models of networks and to compare them to associated discrete models. In this paper, we study several continuous-time network models and examine discrete approximations of them both numerically and analytically. To consider continuous-time networks, we associate each edge in a graph with a time-dependent tie strength that can take continuous non-negative values and decays in time after the most recent interaction. We investigate how the mean tie strength evolves with time in several models, and we explore -- both numerically and analytically -- criteria for the emergence of a giant connected component in some of these models. We also briefly examine the effects of interaction patterns of our continuous-time networks on contagion dynamics in a susceptible-infected-recovered model of an infectious disease.
TopoLines: Topological Smoothing for Line Charts
Venue: arXiv cs.HC 2020
Year: 2020

Line charts are commonly used to visualize a series of data values. When the data are noisy, smoothing is applied to make the signal more apparent. Conventional methods used to smooth line charts, e.g., using subsampling or filters, such as median, Gaussian, or low-pass, each optimize for different properties of the data. The properties generally do not include retaining peaks (i.e., local minima and maxima) in the data, which is an important feature for certain visual analytics tasks. We present TopoLines, a method for smoothing line charts using techniques from Topological Data Analysis. The design goal of TopoLines is to maintain prominent peaks in the data while minimizing any residual error. We evaluate TopoLines for 2 visual analytics tasks by comparing to 5 popular line smoothing methods with data from 4 application domains.
A Neural Vocoder with Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis
Venue: arXiv cs.SD 2020
Year: 2020

This paper presents a neural vocoder named HiNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra hierarchically. Different from existing neural vocoders such as WaveNet, SampleRNN and WaveRNN which directly generate waveform samples using single neural networks, the HiNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a simple DNN model which predicts log amplitude spectra (LAS) from acoustic features. The predicted LAS are sent into the PSP for phase recovery. Considering the issue of phase warping and the difficulty of phase modeling, the PSP is constructed by concatenating a neural source-filter (NSF) waveform generator with a phase extractor. We also introduce generative adversarial networks (GANs) into both ASP and PSP. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by short-time Fourier synthesis. Since there are no autoregressive structures in both predictors, the HiNet vocoder can generate speech waveforms with high efficiency. Objective and subjective experimental results show that our proposed HiNet vocoder achieves better naturalness of reconstructed speech than the conventional STRAIGHT vocoder, a 16-bit WaveNet vocoder using open source implementation and an NSF vocoder with similar complexity to the PSP and obtains similar performance with a 16-bit WaveRNN vocoder. We also find that the performance of HiNet is insensitive to the complexity of the neural waveform generator in PSP to some extend. After simplifying its model structure, the time consumed for generating 1s waveforms of 16kHz speech using a GPU can be further reduced from 0.34s to 0.19s without significant quality degradation.
Towards a Blockchain based digital identity verification, record attestation and record sharing system
Venue: arXiv cs.CR 2020
Year: 2020

The Covid-19 pandemic has made individuals and organizations to rethink the way of handling identity verification and credentials sharing particularly in quarantined situations. In this study, we investigate the inefficiencies of traditional identity systems, and discuss how a proper implementation of Blockchain technology would result in safer, more secure, privacy respecting and remote friendly identity systems. As a result, we propose a Blockchain based framework for digital identity verification, record attestation and record sharing, and we explain the framework in details with certain use cases. Our proposed framework promotes individuals to fully control their identity data and govern the level of the identity data sharing.
Foolproof Cooperative Learning
Venue: arXiv cs.GT 2020
Year: 2020

This paper extends the notion of learning equilibrium in game theory from matrix games to stochastic games. We introduce Foolproof Cooperative Learning (FCL), an algorithm that converges to a Tit-for-Tat behavior. It allows cooperative strategies when played against itself while being not exploitable by selfish players. We prove that in repeated symmetric games, this algorithm is a learning equilibrium. We illustrate the behavior of FCL on symmetric matrix and grid games, and its robustness to selfish learners.
Hard Pixel Mining for Depth Privileged Semantic Segmentation
Venue: arXiv cs.CV 2020
Year: 2020

Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depthaware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on two benchmark datasets, and even outperforms the methods which need depth input during testing.
Evolving Robust Neural Architectures to Defend from Adversarial Attacks
Venue: arXiv cs.NE 2020
Year: 2020

Neural networks are prone to misclassify slightly modified input images. Recently, many defences have been proposed, but none have improved the robustness of neural networks consistently. Here, we propose to use adversarial attacks as a function evaluation to search for neural architectures that can resist such attacks automatically. Experiments on neural architecture search algorithms from the literature show that although accurate, they are not able to find robust architectures. A significant reason for this lies in their limited search space. By creating a novel neural architecture search with options for dense layers to connect with convolution layers and vice-versa as well as the addition of concatenation layers in the search, we were able to evolve an architecture that is inherently accurate on adversarial samples. Interestingly, this inherent robustness of the evolved architecture rivals state-of-the-art defences such as adversarial training while being trained only on the non-adversarial samples. Moreover, the evolved architecture makes use of some peculiar traits which might be useful for developing even more robust ones. Thus, the results here confirm that more robust architectures exist as well as opens up a new realm of feasibilities for the development and exploration of neural networks. Code available at http://bit.ly/RobustArchitectureSearch.