Miracle-dz/UniFAR-FTUs huggingface dataset:
Classifying Images with Few Spikes per Neuron

Spiking neural networks (SNNs) promise to provide AI implementations with a drastically reduced energy budget in comparison with standard artificial neural networks (ANNs). Besides recurrent SNN modules that can be efficiently trained on-chip, many AI applications require the use of feedforward convolutional neural networks (CNNs) as preprocessors for visual or other sensory inputs. The standard solution has been to train a CNN consisting of non-spiking neurons, typically using the rectified linear ReLU function as activation function, and then to translate these CNNs with ReLU neurons via rate coding into SNNs. However this produces SNNs with long latency and small throughput, since the number of spikes that a neuron has to emit is on the order of the number N of output values of the corresponding CNN gate which subsequent layers need to be able to distinguish. We introduce a new ANN-SNN conversion - called FS-conversion - that needs only log N many time steps for that, which is optimal from the perspective of information theory. This can be achieved with a simple variation of the spiking neuron model that has no membrane leak but an exponentially decreasing firing threshold. We show that for the classification of images from ImageNet and CIFAR10 this new conversion reduces latency and drastically increases the throughput compared with rate-based conversion, while achieving almost the same classification performance as the ANN.
Model Extraction Attacks against Recurrent Neural Networks

Model extraction attacks are a kind of attacks in which an adversary obtains a new model, whose performance is equivalent to that of a target model, via query access to the target model efficiently, i.e., fewer datasets and computational resources than those of the target model. Existing works have dealt with only simple deep neural networks (DNNs), e.g., only three layers, as targets of model extraction attacks, and hence are not aware of the effectiveness of recurrent neural networks (RNNs) in dealing with time-series data. In this work, we shed light on the threats of model extraction attacks against RNNs. We discuss whether a model with a higher accuracy can be extracted with a simple RNN from a long short-term memory (LSTM), which is a more complicated and powerful RNN. Specifically, we tackle the following problems. First, in a case of a classification problem, such as image recognition, extraction of an RNN model without final outputs from an LSTM model is presented by utilizing outputs halfway through the sequence. Next, in a case of a regression problem. such as in weather forecasting, a new attack by newly configuring a loss function is presented. We conduct experiments on our model extraction attacks against an RNN and an LSTM trained with publicly available academic datasets. We then show that a model with a higher accuracy can be extracted efficiently, especially through configuring a loss function and a more complex architecture different from the target model.
Analysis of Deep Feature Loss based Enhancement for Speaker Verification

Data augmentation is conventionally used to inject robustness in Speaker Verification systems. Several recently organized challenges focus on handling novel acoustic environments. Deep learning based speech enhancement is a modern solution for this. Recently, a study proposed to optimize the enhancement network in the activation space of a pre-trained auxiliary network. This methodology, called deep feature loss, greatly improved over the state-of-the-art conventional x-vector based system on a children speech dataset called BabyTrain. This work analyzes various facets of that approach and asks few novel questions in that context. We first search for optimal number of auxiliary network activations, training data, and enhancement feature dimension. Experiments reveal the importance of Signal-to-Noise Ratio filtering that we employ to create a large, clean, and naturalistic corpus for enhancement network training. To counter the "mismatch" problem in enhancement, we find enhancing front-end (x-vector network) data helpful while harmful for the back-end (Probabilistic Linear Discriminant Analysis (PLDA)). Importantly, we find enhanced signals contain complementary information to original. Established by combining them in front-end, this gives ~40% relative improvement over the baseline. We also do an ablation study to remove a noise class from x-vector data augmentation and, for such systems, we establish the utility of enhancement regardless of whether it has seen that noise class itself during training. Finally, we design several dereverberation schemes to conclude ineffectiveness of deep feature loss enhancement scheme for this task.
Classification with Rejection: Scaling Generative Classifiers with Supervised Deep Infomax

Deep Infomax (DIM) is an unsupervised representation learning framework by maximizing the mutual information between the inputs and the outputs of an encoder, while probabilistic constraints are imposed on the outputs. In this paper, we propose Supervised Deep InfoMax (SDIM), which introduces supervised probabilistic constraints to the encoder outputs. The supervised probabilistic constraints are equivalent to a generative classifier on high-level data representations, where class conditional log-likelihoods of samples can be evaluated. Unlike other works building generative classifiers with conditional generative models, SDIMs scale on complex datasets, and can achieve comparable performance with discriminative counterparts. With SDIM, we could perform classification with rejection. Instead of always reporting a class label, SDIM only makes predictions when test samples’ largest class conditional surpass some pre-chosen thresholds, otherwise they will be deemed as out of the data distributions, and be rejected. Our experiments show that SDIM with rejection policy can effectively reject illegal inputs, including adversarial examples and out-of-distribution samples.
Effective Diversity in Population-Based Reinforcement Learning

Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.
The generalized scalar auxiliary variable approach (G-SAV) for gradient flows

We establish a general framework for developing, efficient energy stable numerical schemes for gradient flows and develop three classes of generalized scalar auxiliary variable approaches (G-SAV). Numerical schemes based on the G-SAV approaches are as efficient as the original SAV schemes \cite{SXY19,cheng2018multiple} for gradient flows, i.e., only require solving linear equations with constant coefficients at each time step, can be unconditionally energy stable. But G-SAV approaches remove the definition restriction that auxiliary variables can only be square root function. The definition form of auxiliary variable is applicable to any reversible function for G-SAV approaches . Ample numerical results for phase field models are presented to validate the effectiveness and accuracy of the proposed G-SAV numerical schemes.
Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems

In this paper we study the smooth convex-concave saddle point problem. Specifically, we analyze the last iterate convergence properties of the Extragradient (EG) algorithm. It is well known that the ergodic (averaged) iterates of EG converge at a rate of $O(1/T)$ (Nemirovski, 2004). In this paper, we show that the last iterate of EG converges at a rate of $O(1/\sqrt{T})$. To the best of our knowledge, this is the first paper to provide a convergence rate guarantee for the last iterate of EG for the smooth convex-concave saddle point problem. Moreover, we show that this rate is tight by proving a lower bound of $\Omega(1/\sqrt{T})$ for the last iterate. This lower bound therefore shows a quadratic separation of the convergence rates of ergodic and last iterates in smooth convex-concave saddle point problems.
Ambient backscatters-friendly 5G networks: creating hot spots for tags and good spots for readers

In this paper, we present an ambient backscatters-friendly 5G network that creates locations with large power (hot spots) for tags and good reception locations (quiet spots or coherent spots) for readers. Tags are devices that communicate information to readers by backscattering ambient radiofrequency signals. The massive multiple input multiple output (M-MIMO) antenna and beamforming capability of the 5G network is used as follows. In a first step, a training device (separate from tags and readers) is used to send pilots and train the 5G network to perform focusing and/or nulling onto marked locations. In a second step, tags and readers are positioned onto the marked locations. The robustness of M-MIMO beamforming to slight changes in the environment is exploited. Our initial simulation results, in a multipath propagation channel environment, show that creating a hot spot on a tag improves the tag-reader range however with a low probability of detection. Creating a hot spot on a tag and a good reception spot on a reader at the same time, improves the tag-reader range with 99% probability of detection. The study also shows that beamforming does not degrade the performance of a legacy 5G communication.
A Relaxation Argument for Optimization in Neural Networks and Non-Convex Compressed Sensing

It has been observed in practical applications and in theoretical analysis that over-parametrization helps to find good minima in neural network training. Similarly, in this article we study widening and deepening neural networks by a relaxation argument so that the enlarged networks are rich enough to run $r$ copies of parts of the original network in parallel, without necessarily achieving zero training error as in over-parametrized scenarios. The partial copies can be combined in $r^\theta$ possible ways for layer width $\theta$. Therefore, the enlarged networks can potentially achieve the best training error of $r^\theta$ random initializations, but it is not immediately clear if this can be realized via gradient descent or similar training methods. The same construction can be applied to other optimization problems by introducing a similar layered structure. We apply this idea to non-convex compressed sensing, where we show that in some scenarios we can realize the $r^\theta$ times increased chance to obtain a global optimum by solving a convex optimization problem of dimension $r\theta$.
Deep Self-Supervised Representation Learning for Free-Hand Sketch

In this paper, we tackle for the first time, the problem of self-supervised representation learning for free-hand sketches. This importantly addresses a common problem faced by the sketch community – that annotated supervisory data are difficult to obtain. This problem is very challenging in which sketches are highly abstract and subject to different drawing styles, making existing solutions tailored for photos unsuitable. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs: (i) we propose a set of pretext tasks specifically designed for sketches that mimic different drawing styles, and (ii) we further exploit the use of the textual convolution network (TCN) together with the convolutional neural network (CNN) in a dual-branch architecture for sketch feature learning, as means to accommodate the sequential stroke nature of sketches. We demonstrate the superiority of our sketch-specific designs through two sketch-related applications (retrieval and recognition) on a million-scale sketch dataset, and show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods, and significantly narrows the performance gap between with supervised representation learning.11PyTorch code of this work is available at https://github.com/zzz1515151/self-supervised_learning_sketch.
Randomized Spectral Clustering in Large-Scale Stochastic Block Models

Abstract Spectral clustering has been one of the widely used methods for community detection in networks. However, large-scale networks bring computational challenges to the eigenvalue decomposition therein. In this paper, we study the spectral clustering using randomized sketching algorithms from a statistical perspective, where we typically assume the network data are generated from a stochastic block model that is not necessarily of full rank. To do this, we first use the recently developed sketching algorithms to obtain two randomized spectral clustering algorithms, namely, the random projection-based and the random sampling-based spectral clustering. Then we study the theoretical bounds of the resulting algorithms in terms of the approximation error for the population adjacency matrix, the misclassification error, and the estimation error for the link probability matrix. It turns out that, under mild conditions, the randomized spectral clustering algorithms lead to the same theoretical bounds as those of the original spectral clustering algorithm. We also extend the results to degree-corrected stochastic block models. Numerical experiments support our theoretical findings and show the efficiency of randomized methods. A new R package called Rclust is developed and made available to the public. Supplementary materials for this article are available online.
Foreground object segmentation in RGB-D data implemented on GPU

This paper presents a GPU implementation of two foreground object segmentation algorithms: Gaussian Mixture Model (GMM) and Pixel Based Adaptive Segmenter (PBAS) modified for RGB-D data support. The simultaneous use of colour (RGB) and depth (D) data allows to improve segmentation accuracy, especially in case of colour camouflage, illumination changes and occurrence of shadows. Three GPUs were used to accelerate calculations: embedded NVIDIA Jetson TX2 (Maxwell architecture), mobile NVIDIA GeForce GTX 1050m (Pascal architecture) and efficient NVIDIA RTX 2070 (Turing architecture). Segmentation accuracy comparable to previously published works was obtained. Moreover, the use of a GPU platform allowed to get real-time image processing. In addition, the system has been adapted to work with two RGB-D sensors: RealSense D415 and D435 from Intel.
Automatic structured variational inference

The aim of probabilistic programming is to automatize every aspect of probabilistic inference in arbitrary probabilistic models (programs) so that the user can focus her attention on modeling, without dealing with ad-hoc inference methods. Gradient based automatic differentiation stochastic variational inference offers an attractive option as the default method for (differentiable) probabilistic programming as it combines high performance with high computational efficiency. However, the performance of any (parametric) variational approach depends on the choice of an appropriate variational family. Here, we introduced a fully automatic method for constructing structured variational families inspired to the closed-form update in conjugate models. These pseudo-conjugate families incorporate the forward pass of the input probabilistic program and can capture complex statistical dependencies. Pseudo-conjugate families have the same space and time complexity of the input probabilistic program and are therefore tractable in a very large class of models. We validate our automatic variational method on a wide range of high dimensional inference problems including deep learning components.
Topological valley transport of gapped Dirac magnons in bilayer ferromagnetic insulators

Bilayer Heisenberg ferromagnetic insulators hold degenerate terahertz Dirac magnon modes associated with two opposite valleys of the hexagonal Brillouin zone. We show that this energy degeneracy can be removed by breaking of the inversion symmetry ($\mathcal{I}$), leading to a topological magnon valley current. We show furthermore that this current leads to valley Seebeck effect for magnons and is thereby detectable. We perform calculations in the specific example of bilayer ${\mathrm{CrBr}}_{3}$, where $\mathcal{I}$ can be broken by electrostatic doping.
A Novel Incremental Cross-Modal Hashing Approach

Cross-modal retrieval deals with retrieving relevant items from one modality, when provided with a search query from another modality. Hashing techniques, where the data is represented as binary bits have specifically gained importance due to the ease of storage, fast computations and high accuracy. In real world, the number of data categories is continuously increasing, which requires algorithms capable of handling this dynamic scenario. In this work, we propose a novel incremental cross-modal hashing algorithm termed "iCMH", which can adapt itself to handle incoming data of new categories. The proposed approach consists of two sequential stages, namely, learning the hash codes and training the hash functions. At every stage, a small amount of old category data termed "exemplars" is is used so as not to forget the old data while trying to learn for the new incoming data, i.e. to avoid catastrophic forgetting. In the first stage, the hash codes for the exemplars is used, and simultaneously, hash codes for the new data is computed such that it maintains the semantic relations with the existing data. For the second stage, we propose both a non-deep and deep architectures to learn the hash functions effectively. Extensive experiments across a variety of cross-modal datasets and comparisons with state-of-the-art cross-modal algorithms shows the usefulness of our approach.
A Novel Approach to Classify Natural Grasp Actions by Estimating Muscle Activity Patterns from EEG Signals

Developing electroencephalogram (EEG) based brain-computer interface (BCI) systems is challenging. In this study, we analyzed natural grasp actions from EEG. Ten healthy subjects participated in this experiment. They executed and imagined three sustained grasp actions. We proposed a novel approach which estimates muscle activity patterns from EEG signals to improve the overall classification accuracy. For implementing, we have recorded EEG and electromyogram (EMG) simultaneously. Using the similarity of the estimated pattern from EEG signals compare to the activity pattern from EMG signals showed higher classification accuracy than competitive methods. As a result, we obtained the average classification accuracy of 63.89±7.54% for actual movement and 46.96±15.30% for motor imagery. These are 21.59% and 5.66% higher than the result of the competitive model, respectively. This result is encouraging, and the proposed method could potentially be used in future applications, such as a BCI -driven robot control for handling various daily use objects.
Characterization of H-Brownian Gibbsian line
ensembles

In this paper we show that an $H$-Brownian Gibbsian line ensemble is completely characterized by the finite-dimensional marginals of its lowest indexed curve for a large class of interaction Hamiltonians $H$. A particular consequence of our result is that the KPZ line ensemble is the unique line ensemble that satisfies the $H_1$-Brownian Gibbs property with $H_1(x) = e^x$ and whose lowest indexed curve is equal to the Hopf-Cole solution to the narrow wedge KPZ equation.
Image denoising with less artefacts: Novel non-linear filtering on fast patch reorderings.

Leading denoising methods such as 3D block matching (BM3D) are patch-based. However, they can suffer from frequency domain artefacts and require to specify explicit noise models. We present a patch-based method that avoids these drawbacks. It combines a simple and fast patch reordering with a non-linear smoothing. The smoothing rewards both patch and pixel similarities in a multiplicative way. We perform experiments on real world images with additive white Gaussian noise (AWGN), and on electron microscopy data with a more general additive noise model. Our filter outperforms BM3D in 77% of the experiments, with improvements of up to 29% with respect to the mean squared error.
Distinct Topological Surface States on the Two Terminations of 
MnBi4Te7

The recent discovered intrinsic magnetic topological insulator MnBi2Te4 have been met with unusual success in hosting emergent phenomena such as the quantum anomalous Hall effect and the axion insulator states. However, the surface-bulk correspondence of the Mn-Bi-Te family, composed by the superlattice-like MnBi2Te4/(Bi2Te3)n (n = 0, 1, 2, 3 .) layered structure, remains intriguing but elusive. Here, by using scanning tunneling microscopy (STM) and angle-resolved photoemission spectroscopy (ARPES) techniques, we unambiguously assign the two distinct surface states of MnBi4Te7 (n = 1) to the quintuple-layer (QL) Bi2Te3 termination and the septuple-layer (SL) MnBi2Te4 termination, respectively. A comparison of the experimental observations with theoretical calculations reveals the diverging topological behaviors, especially the hybridization effect between magnetic and nonmagnetic layers, on the two terminations: a gap on the QL termination originating from the topological surface states of the QL hybridizing with the bands of the beneath SL, and a gapless Dirac-cone band structure on the SL termination with time-reversal symmetry. The quasi-particle interference patterns further confirm the topological nature of the surface states for both terminations, continuing far above the Fermi energy. The QL termination carries a spin-helical Dirac state with hexagonal warping, while at the SL termination, a strongly canted helical state from the surface lies between a pair of Rashba-split states from its neighboring layer. Our work elucidates an unprecedented hybridization effect between the building blocks of the topological surface states, and also reveals the termination-dependent time-reversal symmetry breaking in a magnetic topological insulator, rendering an ideal platform to realize the half-integer quantum Hall effect and relevant quantum phenomena.