OpenRaiser/Intern-Atlas huggingface dataset:
Boosting Unconstrained Face Recognition with Auxiliary Unlabeled Data
Venue: arXiv cs.CV 2021
Year: 2021

In recent years, significant progress has been made in face recognition, which can be partially attributed to the availability of large-scale labeled face datasets. However, since the faces in these datasets usually contain limited degree and types of variation, the resulting trained models generalize poorly to more realistic unconstrained face datasets. While collecting labeled faces with larger variations could be helpful, it is practically infeasible due to privacy and labor cost. In comparison, it is easier to acquire a large number of unlabeled faces from different domains, which could be used to regularize the learning of face representations. We present an approach to use such unlabeled faces to learn generalizable face representations, where we assume neither the access to identity labels nor domain labels for unlabeled images. Experimental results on unconstrained datasets show that a small amount of unlabeled data with sufficient diversity can (i) lead to an appreciable gain in recognition performance and (ii) outperform the supervised baseline when combined with less than half of the labeled data. Compared with the state-of-the-art face recognition methods, our method further improves their performance on challenging benchmarks, such as IJB-B, IJB-C and IJB-S.
Adversarial Transferability in Wearable Sensor Systems
Venue: arXiv cs.LG 2021
Year: 2021

Machine learning is used for inference and decision making in wearable sensor systems. However, recent studies have found that machine learning algorithms are easily fooled by the addition of adversarial perturbations to their inputs. What is more interesting is that adversarial examples generated for one machine learning system is also effective against other systems. This property of adversarial examples is called transferability. In this work, we take the first stride in studying adversarial transferability in wearable sensor systems from the following perspectives: 1) transferability between machine learning systems, 2) transferability across subjects, 3) transferability across sensor body locations, and 4) transferability across datasets. We found strong untargeted transferability in most cases. Targeted attacks were less successful with success scores from $0\%$ to $80\%$. The transferability of adversarial examples depends on many factors such as the inclusion of data from all subjects, sensor body position, number of samples in the dataset, type of learning algorithm, and the distribution of source and target system dataset. The transferability of adversarial examples decreases sharply when the data distribution of the source and target system becomes more distinct. We also provide guidelines for the community for designing robust sensor systems.
Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
Venue: arXiv cs.LG 2021
Year: 2021

Learning continuous representations of discrete objects such as text, users, movies, and URLs lies at the heart of many applications including language and user modeling. When using discrete objects as input to neural networks, we often ignore the underlying structures (e.g., natural groupings and similarities) and embed the objects independently into individual vectors. As a result, existing methods do not scale to large vocabulary sizes. In this paper, we design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix. We call our method Anchor & Transform (ANT) as the embeddings of discrete objects are a sparse linear combination of the anchors, weighted according to the transformation matrix. ANT is scalable, flexible, and end-to-end trainable. We further provide a statistical interpretation of our algorithm as a Bayesian nonparametric prior for embeddings that encourages sparsity and leverages natural groupings among objects. By deriving an approximate inference algorithm based on Small Variance Asymptotics, we obtain a natural extension that automatically learns the optimal number of anchors instead of having to tune it as a hyperparameter. On text classification, language modeling, and movie recommendation benchmarks, we show that ANT is particularly suitable for large vocabulary sizes and demonstrates stronger performance with fewer parameters (up to 40x compression) as compared to existing compression baselines.
Integer Ring Sieve for Constructing Compact QC-LDPC Codes with Girths 8, 10, and 12
Venue: arXiv cs.IT 2021
Year: 2021

This paper proposes a new method of constructing compact fully-connected Quasi-Cyclic Low Density Parity Check (QC-LDPC) codes with girth g = 8, 10, and 12. The originality of the proposed method is to impose constraints on the exponent matrix P to reduce the search space drastically. For a targeted lifting degree of N, the first step of the method is to sieve the integer ring Z_N to make a particular sub-group with specific properties to construct the second column of P (the first column being filled with zeros). The remaining columns of P are determined recursively as multiples of the second column by adapting the sequentially multiplied column (SMC) method whereby a controlled greedy search is applied at each step. The codes constructed with the proposed semi-algebraic method show lengths that can be significantly shorter than their best counterparts in the literature.
Heterogeneous Millimeter Wave Wireless Power Transfer With Poisson Cluster Processes
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, we analyze the energy coverage performance of heterogeneous millimeter wave (mmWave) wireless power transfer (WPT) networks, where macro base stations (MBSs) are distributed according to a Poisson point process (PPP), the location of power beacons (PBs) is modeled as a $k$-tier Poisson cluster process (PCP), and energy users (EUs) are clustered around the centers of PB clusters. Moreover, the cosine antenna gain model is adopted instead of the prevalent flap-top gain model, which is simpler in derivation but less accurate. Based on the generalized exponential distribution approximation, we propose a new technique of deriving the energy coverage probability of randomly deployed mmWave WPT networks. Specifically, taking the Thomas cluster process (TCP) for instance, we derive the energy coverage probabilities with two PB association strategies, i.e., the random PB association and the nearest PB association. Through Monte-Carlo simulations, our theoretical results are verified and the impact of system parameters, such as the array antenna size, energy threshold or average number of PBs in a cluster, are also investigated.
Grant-Free Coexistence of Critical and Non-Critical IoT Services in Two-Hop Satellite and Terrestrial Networks
Venue: arXiv cs.IT 2021
Year: 2021

Terrestrial and satellite communication networks often rely on two-hop wireless architectures with an access channel followed by backhaul links. Examples include Cloud-Radio Access Networks (C-RAN) and Low-Earth Orbit (LEO) satellite systems. Furthermore, communication services characterized by the coexistence of heterogeneous requirements are emerging as key use cases. This paper studies the performance of critical service (CS) and non-critical service (NCS) for Internet of Things (IoT) systems sharing a grant-free channel consisting of radio access and backhaul segments. On the radio access segment, IoT devices send packets to a set of non-cooperative access points (APs) using slotted ALOHA (SA). The APs then forward correctly received messages to a base station over a shared wireless backhaul segment adopting SA. We study first a simplified erasure channel model, which is well suited for satellite applications. Then, in order to account for terrestrial scenarios, the impact of fading is considered. Among the main conclusions, we show that orthogonal inter-service resource allocation is generally preferred for NCS devices, while non-orthogonal protocols can improve the throughput and packet success rate of CS devices for both terrestrial and satellite scenarios.
SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
Venue: arXiv cs.LG 2021
Year: 2021

Adversarial training (AT) has become a popular choice for training robust networks. However, it tends to sacrifice clean accuracy heavily in favor of robustness and suffers from a large generalization error. To address these concerns, we propose Smooth Adversarial Training (SAT), guided by our analysis on the eigenspectrum of the loss Hessian. We find that curriculum learning, a scheme that emphasizes on starting "easy" and gradually ramping up on the "difficulty" of training, smooths the adversarial loss landscape for a suitably chosen difficulty metric. We present a general formulation for curriculum learning in the adversarial setting and propose two difficulty metrics based on the maximal Hessian eigenvalue (H-SAT) and the softmax probability (P-SA). We demonstrate that SAT stabilizes network training even for a large perturbation norm and allows the network to operate at a better clean accuracy versus robustness trade-off curve compared to AT. This leads to a significant improvement in both clean accuracy and robustness compared to AT, TRADES, and other baselines. To highlight a few results, our best model improves normal and robust accuracy by 6% and 1% on CIFAR-100 compared to AT, respectively. On Imagenette, a ten-class subset of ImageNet, our model outperforms AT by 23% and 3% on normal and robust accuracy respectively.
A Framework for Generating Explanations from Temporal Personal Health Data
Venue: arXiv cs.CL 2021
Year: 2021

Whereas it has become easier for individuals to track their personal health data (e.g., heart rate, step count, food log), there is still a wide chasm between the collection of data and the generation of meaningful explanations to help users better understand what their data means to them. With an increased comprehension of their data, users will be able to act upon the newfound information and work towards striving closer to their health goals. We aim to bridge the gap between data collection and explanation generation by mining the data for interesting behavioral findings that may provide hints about a user's tendencies. Our focus is on improving the explainability of temporal personal health data via a set of informative summary templates, or "protoforms." These protoforms span both evaluation-based summaries that help users evaluate their health goals and pattern-based summaries that explain their implicit behaviors. In addition to individual users, the protoforms we use are also designed for population-level summaries. We apply our approach to generate summaries (both univariate and multivariate) from real user data and show that our system can generate interesting and useful explanations.
Towards Automatic Bayesian Optimization: A first step involving acquisition functions
Venue: arXiv cs.AI 2021
Year: 2021

Bayesian Optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression nor its gradients, they are expensive to evaluate and its evaluation is noisy. The most popular application of bayesian optimization is the automatic hyperparameter tuning of machine learning algorithms, where we obtain the best configuration of machine learning algorithms by optimizing the estimation of the generalization error of these algorithms. Despite being applied with success, bayesian optimization methodologies also have hyperparameters that need to be configured such as the probabilistic surrogate model or the acquisition function used. A bad decision over the configuration of these hyperparameters implies obtaining bad quality results. Typically, these hyperparameters are tuned by making assumptions of the objective function that we want to evaluate but there are scenarios where we do not have any prior information about the objective function. In this paper, we propose a first attempt over automatic bayesian optimization by exploring several heuristics that automatically tune the acquisition function of bayesian optimization. We illustrate the effectiveness of these heurisitcs in a set of benchmark problems and a hyperparameter tuning problem of a machine learning algorithm.
Lower Bounds on the Running Time of Two-Way Quantum Finite Automata and Sublogarithmic-Space Quantum Turing Machines
Venue: arXiv cs.CC 2021
Year: 2021

The two-way finite automaton with quantum and classical states (2QCFA), defined by Ambainis and Watrous, is a model of quantum computation whose quantum part is extremely limited; however, as they showed, 2QCFA are surprisingly powerful: a 2QCFA with only a single-qubit can recognize the language $L_{pal}=\{w \in \{a,b\}^*:w \text{ is a palindrome}\}$ with bounded error in expected time $2^{O(n)}$, on inputs of length $n$. We prove that their result essentially cannot be improved upon: a 2QCFA (of any size) cannot recognize $L_{pal}$ with bounded error in expected time $2^{o(n)}$. To our knowledge, this is the first example of a language that can be recognized with bounded error by a 2QCFA in exponential time but not in subexponential time. Moreover, we prove that a quantum Turing machine (QTM) running in space $o(\log n)$ and expected time $2^{n^{1-Ω(1)}}$ cannot recognize $L_{pal}$ with bounded error; again, this is the first lower bound of its kind. Far more generally, we establish a lower bound on the running time of any 2QCFA or $o(\log n)$-space QTM that recognizes any language $L$ in terms of a natural "hardness measure" of $L$. This allows us to exhibit a large family of languages for which we have asymptotically matching lower and upper bounds on the running time of any such 2QCFA or QTM recognizer.
Bayesian Models Applied to Cyber Security Anomaly Detection Problems
Venue: arXiv cs.CR 2021
Year: 2021

Cyber security is an important concern for all individuals, organisations and governments globally. Cyber attacks have become more sophisticated, frequent and dangerous than ever, and traditional anomaly detection methods have been proved to be less effective when dealing with these new classes of cyber threats. In order to address this, both classical and Bayesian models offer a valid and innovative alternative to the traditional signature-based methods, motivating the increasing interest in statistical research that it has been observed in recent years. In this review we provide a description of some typical cyber security challenges, typical types of data and statistical methods, paying special attention to Bayesian approaches for these problems.
DevSecOps in Robotics
Venue: arXiv cs.RO 2021
Year: 2021

Quality in software is often understood as "execution according to design purpose" whereas security means that "software will not put data or computing systems at risk of unauthorized access." There seems to be a connection between these two aspects but, how do we integrate both of them in the robotics development cycle? In this article we introduce DevSecOps in Robotics, a set of best practices designed to help roboticists implant security deep in the heart of their development and operations processes. First, we briefly describe DevOps, introduce the value added with DevSecOps and describe and illustrate how these practices may be implemented in the robotics field. We finalize with a discussion on the relationship between security, quality and safety, open problems and future research questions.
Symmetry & critical points for a model shallow neural network
Venue: arXiv cs.LG 2021
Year: 2021

We consider the optimization problem associated with fitting two-layer ReLU networks with $k$ hidden neurons, where labels are assumed to be generated by a (teacher) neural network. We leverage the rich symmetry exhibited by such models to identify various families of critical points and express them as power series in $k^{-\frac{1}{2}}$. These expressions are then used to derive estimates for several related quantities which imply that not all spurious minima are alike. In particular, we show that while the loss function at certain types of spurious minima decays to zero like $k^{-1}$, in other cases the loss converges to a strictly positive constant. The methods used depend on symmetry, the geometry of group actions, bifurcation, and Artin's implicit function theorem.
Spatio-Temporal Graph Convolution for Resting-State fMRI Analysis
Venue: arXiv cs.LG 2021
Year: 2021

The Blood-Oxygen-Level-Dependent (BOLD) signal of resting-state fMRI (rs-fMRI) records the temporal dynamics of intrinsic functional networks in the brain. However, existing deep learning methods applied to rs-fMRI either neglect the functional dependency between different brain regions in a network or discard the information in the temporal dynamics of brain activity. To overcome those shortcomings, we propose to formulate functional connectivity networks within the context of spatio-temporal graphs. We train a spatio-temporal graph convolutional network (ST-GCN) on short sub-sequences of the BOLD time series to model the non-stationary nature of functional connectivity. Simultaneously, the model learns the importance of graph edges within ST-GCN to gain insight into the functional connectivities contributing to the prediction. In analyzing the rs-fMRI of the Human Connectome Project (HCP, N=1,091) and the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA, N=773), ST-GCN is significantly more accurate than common approaches in predicting gender and age based on BOLD signals. Furthermore, the brain regions and functional connections significantly contributing to the predictions of our model are important markers according to the neuroscience literature.
Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders
Venue: arXiv cs.LG 2021
Year: 2021

Detecting out of distribution (OOD) samples is of paramount importance in all Machine Learning applications. Deep generative modeling has emerged as a dominant paradigm to model complex data distributions without labels. However, prior work has shown that generative models tend to assign higher likelihoods to OOD samples compared to the data distribution on which they were trained. First, we propose Adversarial Mirrored Autoencoder (AMA), a variant of Adversarial Autoencoder, which uses a mirrored Wasserstein loss in the discriminator to enforce better semantic-level reconstruction. We also propose a latent space regularization to learn a compact manifold for in-distribution samples. The use of AMA produces better feature representations that improve anomaly detection performance. Second, we put forward an alternative measure of anomaly score to replace the reconstruction-based metric which has been traditionally used in generative model-based anomaly detection methods. Our method outperforms the current state-of-the-art methods for anomaly detection on several OOD detection benchmarks.
Pipelined Backpropagation at Scale: Training Large Models without Batches
Venue: arXiv cs.LG 2021
Year: 2021

New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparameters for traditional training without additional tuning.
Towards Backward-Compatible Representation Learning
Venue: arXiv cs.CV 2021
Year: 2021

We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions and are learned via different neural network architectures and loss functions. Compatible means that, if such features are used to compare images, then "new" features can be compared directly to "old" features, so they can be used interchangeably. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models, a process known as backfilling. Backward compatibility is critical to quickly deploy new embedding models that leverage ever-growing large-scale training datasets and improvements in deep learning architectures and training methods. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning. In experiments on learning embeddings for face recognition, models trained with BCT successfully achieve backward compatibility without sacrificing accuracy, thus enabling backfill-free model updates of visual embeddings.
Bayes-Optimal Convolutional AMP
Venue: arXiv cs.IT 2021
Year: 2021

This paper proposes Bayes-optimal convolutional approximate message-passing (CAMP) for signal recovery in compressed sensing. CAMP uses the same low-complexity matched filter (MF) for interference suppression as approximate message-passing (AMP). To improve the convergence property of AMP for ill-conditioned sensing matrices, the so-called Onsager correction term in AMP is replaced by a convolution of all preceding messages. The tap coefficients in the convolution are determined so as to realize asymptotic Gaussianity of estimation errors via state evolution (SE) under the assumption of orthogonally invariant sensing matrices. An SE equation is derived to optimize the sequence of denoisers in CAMP. The optimized CAMP is proved to be Bayes-optimal for all orthogonally invariant sensing matrices if the SE equation converges to a fixed-point and if the fixed-point is unique. For sensing matrices with low-to-moderate condition numbers, CAMP can achieve the same performance as high-complexity orthogonal/vector AMP that requires the linear minimum mean-square error (LMMSE) filter instead of the MF.
Large independent sets on random $d$-regular graphs with fixed degree $d$
Venue: arXiv cs.DS 2021
Year: 2021

This paper presents a linear prioritized local algorithm that computes large independent sets on a random $d$-regular graph with small and fixed degree $d$. We studied experimentally the independence ratio obtained by the algorithm when $ d \in [3,100]$. For all $d \in [5,100]$, our results are larger than lower bounds calculated by exact methods, thus providing improved estimates of lower bounds.
Using the Split Bregman Algorithm to Solve the Self-repelling Snake Model
Venue: arXiv cs.CV 2021
Year: 2021

Preserving contour topology during image segmentation is useful in many practical scenarios. By keeping the contours isomorphic, it is possible to prevent over-segmentation and under-segmentation, as well as to adhere to given topologies. The Self-repelling Snake model (SR) is a variational model that preserves contour topology by combining a non-local repulsion term with the geodesic active contour model (GAC). The SR is traditionally solved using the additive operator splitting (AOS) scheme. In our paper, we propose an alternative solution to the SR using the Split Bregman method. Our algorithm breaks the problem down into simpler sub-problems to use lower-order evolution equations and a simple projection scheme rather than re-initialization. The sub-problems can be solved via fast Fourier transform (FFT) or an approximate soft thresholding formula which maintains stability, shortening the convergence time, and reduces the memory requirement. The Split Bregman and AOS algorithms are compared theoretically and experimentally.