OpenRaiser/Intern-Atlas huggingface dataset:
Continual Learning Using Bayesian Neural Networks
Venue: arXiv cs.LG 2020
Year: 2020

Continual learning models allow to learn and adapt to new changes and tasks over time. However, in continual and sequential learning scenarios in which the models are trained using different data with various distributions, neural networks tend to forget the previously learned knowledge. This phenomenon is often referred to as catastrophic forgetting. The catastrophic forgetting is an inevitable problem in continual learning models for dynamic environments. To address this issue, we propose a method, called Continual Bayesian Learning Networks (CBLN), which enables the networks to allocate additional resources to adapt to new tasks without forgetting the previously learned tasks. Using a Bayesian Neural Network, CBLN maintains a mixture of Gaussian posterior distributions that are associated with different tasks. The proposed method tries to optimise the number of resources that are needed to learn each task and avoids an exponential increase in the number of resources that are involved in learning multiple tasks. The proposed method does not need to access the past training data and can choose suitable weights to classify the data points during the test time automatically based on an uncertainty criterion. We have evaluated our method on the MNIST and UCR time-series datasets. The evaluation results show that our method can address the catastrophic forgetting problem at a promising rate compared to the state-of-the-art models.
Hardness of Minimum Barrier Shrinkage and Minimum Installation Path
Venue: arXiv cs.CC 2020
Year: 2020

In the Minimum Installation Path problem, we are given a graph $G$ with edge weights $w(.)$ and two vertices $s,t$ of $G$. We want to assign a non-negative power $p(v)$ to each vertex $v$ of $G$ so that the edges $uv$ such that $p(u)+p(v)$ is at least $w(uv)$ contain some $s$-$t$-path, and minimize the sum of assigned powers. In the Minimum Barrier Shrinkage problem, we are given a family of disks in the plane and two points $x$ and $y$ lying outside the disks. The task is to shrink the disks, each one possibly by a different amount, so that we can draw an $x$-$y$ curve that is disjoint from the interior of the shrunken disks, and the sum of the decreases in the radii is minimized. We show that the Minimum Installation Path and the Minimum Barrier Shrinkage problems (or, more precisely, the natural decision problems associated with them) are weakly NP-hard.
Improving Password Guessing via Representation Learning
Venue: arXiv cs.CR 2020
Year: 2020

Learning useful representations from unstructured data is one of the core challenges, as well as a driving force, of modern data-driven approaches. Deep learning has demonstrated the broad advantages of learning and harnessing such representations. In this paper, we introduce a deep generative model representation learning approach for password guessing. We show that an abstract password representation naturally offers compelling and versatile properties that can be used to open new directions in the extensively studied, and yet presently active, password guessing field. These properties can establish novel password generation techniques that are neither feasible nor practical with the existing probabilistic and non-probabilistic approaches. Based on these properties, we introduce:(1) A general framework for conditional password guessing that can generate passwords with arbitrary biases; and (2) an Expectation Maximization-inspired framework that can dynamically adapt the estimated password distribution to match the distribution of the attacked password set.
Online Learning Using Only Peer Prediction
Venue: arXiv cs.LG 2020
Year: 2020

This paper considers a variant of the classical online learning problem with expert predictions. Our model's differences and challenges are due to lacking any direct feedback on the loss each expert incurs at each time step $t$. We propose an approach that uses peer prediction and identify conditions where it succeeds. Our techniques revolve around a carefully designed peer score function $s()$ that scores experts' predictions based on the peer consensus. We show a sufficient condition, that we call \emph{peer calibration}, under which standard online learning algorithms using loss feedback computed by the carefully crafted $s()$ have bounded regret with respect to the unrevealed ground truth values. We then demonstrate how suitable $s()$ functions can be derived for different assumptions and models.
Visual Indeterminacy in GAN Art
Venue: arXiv cs.CV 2020
Year: 2020

This paper explores visual indeterminacy as a description for artwork created with Generative Adversarial Networks (GANs). Visual indeterminacy describes images which appear to depict real scenes, but, on closer examination, defy coherent spatial interpretation. GAN models seem to be predisposed to producing indeterminate images, and indeterminacy is a key feature of much modern representational art, as well as most GAN art. It is hypothesized that indeterminacy is a consequence of a powerful-but-imperfect image synthesis model that must combine general classes of objects, scenes, and textures.
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
Venue: arXiv cs.CV 2020
Year: 2020

Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation
Venue: arXiv cs.LG 2020
Year: 2020

We consider stochastic second-order methods for minimizing smooth and strongly-convex functions under an interpolation condition satisfied by over-parameterized models. Under this condition, we show that the regularized subsampled Newton method (R-SSN) achieves global linear convergence with an adaptive step-size and a constant batch-size. By growing the batch size for both the subsampled gradient and Hessian, we show that R-SSN can converge at a quadratic rate in a local neighbourhood of the solution. We also show that R-SSN attains local linear convergence for the family of self-concordant functions. Furthermore, we analyze stochastic BFGS algorithms in the interpolation setting and prove their global linear convergence. We empirically evaluate stochastic L-BFGS and a "Hessian-free" implementation of R-SSN for binary classification on synthetic, linearly-separable datasets and real datasets under a kernel mapping. Our experimental results demonstrate the fast convergence of these methods, both in terms of the number of iterations and wall-clock time.
Central Server Free Federated Learning over Single-sided Trust Social Networks
Venue: arXiv cs.LG 2020
Year: 2020

Federated learning has become increasingly important for modern machine learning, especially for data privacy-sensitive scenarios. Existing federated learning mostly adopts the central server-based architecture or centralized architecture. However, in many social network scenarios, centralized federated learning is not applicable (e.g., a central agent or server connecting all users may not exist, or the communication cost to the central server is not affordable). In this paper, we consider a generic setting: 1) the central server may not exist, and 2) the social network is unidirectional or of single-sided trust (i.e., user A trusts user B but user B may not trust user A). We propose a central server free federated learning algorithm, named Online Push-Sum (OPS) method, to handle this challenging but generic scenario. A rigorous regret analysis is also provided, which shows very interesting results on how users can benefit from communication with trusted users in the federated learning scenario. This work builds upon the fundamental algorithm framework and theoretical guarantees for federated learning in the generic social network scenario.
Noise as a Resource for Learning in Knowledge Distillation
Venue: arXiv cs.LG 2020
Year: 2020

While noise is commonly considered a nuisance in computing systems, a number of studies in neuroscience have shown several benefits of noise in the nervous system from enabling the brain to carry out computations such as probabilistic inference as well as carrying additional information about the stimuli. Similarly, noise has been shown to improve the performance of deep neural networks. In this study, we further investigate the effect of adding noise in the knowledge distillation framework because of its resemblance to collaborative subnetworks in the brain regions. We empirically show that injecting constructive noise at different levels in the collaborative learning framework enables us to train the model effectively and distill desirable characteristics in the student model. In doing so, we propose three different methods that target the common challenges in deep neural networks: minimizing the performance gap between a compact model and large model (Fickle Teacher), training high performance compact adversarially robust models (Soft Randomization), and training models efficiently under label noise (Messy Collaboration). Our findings motivate further study in the role of noise as a resource for learning in a collaborative learning framework.
Fairness in Clustering with Multiple Sensitive Attributes
Venue: arXiv cs.LG 2020
Year: 2020

A clustering may be considered as fair on pre-specified sensitive attributes if the proportions of sensitive attribute groups in each cluster reflect that in the dataset. In this paper, we consider the task of fair clustering for scenarios involving multiple multi-valued or numeric sensitive attributes. We propose a fair clustering method, \textit{FairKM} (Fair K-Means), that is inspired by the popular K-Means clustering formulation. We outline a computational notion of fairness which is used along with a cluster coherence objective, to yield the FairKM clustering method. We empirically evaluate our approach, wherein we quantify both the quality and fairness of clusters, over real-world datasets. Our experimental evaluation illustrates that the clusters generated by FairKM fare significantly better on both clustering quality and fair representation of sensitive attribute groups compared to the clusters from a state-of-the-art baseline fair clustering method.
Near-Optimal Massively Parallel Graph Connectivity
Venue: arXiv cs.DS 2020
Year: 2020

Identifying the connected components of a graph, apart from being a fundamental problem with countless applications, is a key primitive for many other algorithms. In this paper, we consider this problem in parallel settings. Particularly, we focus on the Massively Parallel Computations (MPC) model, which is the standard theoretical model for modern parallel frameworks such as MapReduce, Hadoop, or Spark. We consider the truly sublinear regime of MPC for graph problems where the space per machine is $n^δ$ for some desirably small constant $δ\in (0, 1)$. We present an algorithm that for graphs with diameter $D$ in the wide range $[\log^ε n, n]$, takes $O(\log D)$ rounds to identify the connected components and takes $O(\log \log n)$ rounds for all other graphs. The algorithm is randomized, succeeds with high probability, does not require prior knowledge of $D$, and uses an optimal total space of $O(m)$. We complement this by showing a conditional lower-bound based on the widely believed TwoCycle conjecture that $Ω(\log D)$ rounds are indeed necessary in this setting. Studying parallel connectivity algorithms received a resurgence of interest after the pioneering work of Andoni et al. [FOCS 2018] who presented an algorithm with $O(\log D \cdot \log \log n)$ round-complexity. Our algorithm improves this result for the whole range of values of $D$ and almost settles the problem due to the conditional lower-bound. Additionally, we show that with minimal adjustments, our algorithm can also be implemented in a variant of the (CRCW) PRAM in asymptotically the same number of rounds.
Landmarks-assisted Collaborative Deep Framework for Automatic 4D Facial Expression Recognition
Venue: arXiv cs.CV 2020
Year: 2020

We propose a novel landmarks-assisted collaborative end-to-end deep framework for automatic 4D FER. Using 4D face scan data, we calculate its various geometrical images, and afterwards use rank pooling to generate their dynamic images encapsulating important facial muscle movements over time. As well, the given 3D landmarks are projected on a 2D plane as binary images and convolutional layers are used to extract sequences of feature vectors for every landmark video. During the training stage, the dynamic images are used to train an end-to-end deep network, while the feature vectors of landmark images are used train a long short-term memory (LSTM) network. The finally improved set of expression predictions are obtained when the dynamic and landmark images collaborate over multi-views using the proposed deep framework. Performance results obtained from extensive experimentation on the widely-adopted BU-4DFE database under globally used settings prove that our proposed collaborative framework outperforms the state-of-the-art 4D FER methods and reach a promising classification accuracy of 96.7% demonstrating its effectiveness.
Quantification of the Leakage in Federated Learning
Venue: arXiv cs.CR 2020
Year: 2020

With the growing emphasis on users' privacy, federated learning has become more and more popular. Many architectures have been raised for a better security. Most architecture work on the assumption that data's gradient could not leak information. However, some work, recently, has shown such gradients may lead to leakage of the training data. In this paper, we discuss the leakage based on a federated approximated logistic regression model and show that such gradient's leakage could leak the complete training data if all elements of the inputs are either 0 or 1.
Model-based process design of a ternary protein separation using multi-step gradient ion-exchange SMB chromatography
Venue: arXiv cs.CE 2020
Year: 2020

Model-based process design of ion-exchange simulated moving bed (IEX-SMB) chromatography for center-cut separation of proteins is studied. Use of nonlinear binding models that describe more accurate adsorption behaviours of macro-molecules could make it impossible to utilize triangle theory to obtain operating parameters. Moreover, triangle theory provides no rules to design salt profiles in IEX-SMB. In the modelling study here, proteins (i.e., ribonuclease, cytochrome and lysozyme) on the chromatographic columns packed with strong cation-exchanger SP Sepharose FF is used as an example system. The general rate model with steric mass-action kinetics was used; two closed-loop IEX-SMB network schemes were investigated (i.e., cascade and eight-zone schemes). Performance of the IEX-SMB schemes was examined with respect to multi-objective indicators (i.e., purity and yield) and productivity, and compared to a single column batch system with the same amount of resin utilized. A multi-objective sampling algorithm, Markov Chain Monte Carlo (MCMC), was used to generate samples for constructing the Pareto optimal fronts. MCMC serves on the sampling purpose, which is interested in sampling the Pareto optimal points as well as those near Pareto optimal. Pareto fronts of the three schemes provide the full information of trade-off between the conflicting indicators of purity and yield. The results indicate the cascade IEX-SMB scheme and the integrated eight-zone IEX-SMB scheme have the similar performance that both outperforms the single column batch system.
A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels
Venue: arXiv cs.CV 2020
Year: 2020

The major driving force behind the immense success of deep learning models is the availability of large datasets along with their clean labels. Unfortunately, this is very difficult to obtain, which has motivated research on the training of deep models in the presence of label noise and ways to avoid over-fitting on the noisy labels. In this work, we build upon the seminal work in this area, Co-teaching and propose a simple, yet efficient approach termed mCT-S2R (modified co-teaching with self-supervision and re-labeling) for this task. First, to deal with significant amount of noise in the labels, we propose to use self-supervision to generate robust features without using any labels. Next, using a parallel network architecture, an estimate of the clean labeled portion of the data is obtained. Finally, using this data, a portion of the estimated noisy labeled portion is re-labeled, before resuming the network training with the augmented data. Extensive experiments on three standard datasets show the effectiveness of the proposed framework.
Byzantine Generalized Lattice Agreement
Venue: arXiv cs.DC 2020
Year: 2020

The paper investigates the Lattice Agreement (LA) problem in asynchronous systems. In LA each process proposes an element $e$ from a predetermined lattice, and has to decide on an element $e'$ of the lattice such that $e \leq e'$. Moreover, decisions of different processes have to be comparable (no two processes can decide two elements $e$ and $e'$ such that $(e \not\leq e') \land (e' \not\leq e)$). It has been shown that Generalized LA (i.e., a version of LA proposing and deciding on sequences of values) can be used to build a Replicated State Machine (RSM) with commutative update operations. The key advantage of LA and Generalized LA is that they can be solved in asynchronous systems prone to crash-failures (this is not the case with standard Consensus). In this paper we assume Byzantine failures. We propose the Wait Till Safe (WTS) algorithm for LA, and we show that its resilience to $f \leq (n-1)/3$ Byzantines is optimal. We then generalize WTS obtaining a Generalized LA algorithm, namely GWTS. We use GWTS to build a RSM with commutative updates. Our RSM works in asynchronous systems and tolerates $f \leq (n-1)/3$ malicious entities. All our algorithms use the minimal assumption of authenticated channels. When the more powerful public signatures are available, we discuss how to improve the message complexity of our results (from quadratic to linear, when $f={\cal O}(1)$). At the best of our knowledge this is the first paper proposing a solution for Byzantine LA that works on any possible lattice, and it is the first work proposing a Byzantine tolerant RSM built on it.
Manifold Embedded Knowledge Transfer for Brain-Computer Interfaces
Venue: arXiv cs.HC 2020
Year: 2020

Transfer learning makes use of data or knowledge in one problem to help solve a different, yet related, problem. It is particularly useful in brain-computer interfaces (BCIs), for coping with variations among different subjects and/or tasks. This paper considers offline unsupervised cross-subject electroencephalogram (EEG) classification, i.e., we have labeled EEG trials from one or more source subjects, but only unlabeled EEG trials from the target subject. We propose a novel manifold embedded knowledge transfer (MEKT) approach, which first aligns the covariance matrices of the EEG trials in the Riemannian manifold, extracts features in the tangent space, and then performs domain adaptation by minimizing the joint probability distribution shift between the source and the target domains, while preserving their geometric structures. MEKT can cope with one or multiple source domains, and can be computed efficiently. We also propose a domain transferability estimation (DTE) approach to identify the most beneficial source domains, in case there are a large number of source domains. Experiments on four EEG datasets from two different BCI paradigms demonstrated that MEKT outperformed several state-of-the-art transfer learning approaches, and DTE can reduce more than half of the computational cost when the number of source subjects is large, with little sacrifice of classification accuracy.
FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence
Venue: arXiv cs.DS 2020
Year: 2020

This paper presents a new distributed-memory algorithm called FastSV for finding connected components in an undirected graph. Our algorithm simplifies the classic Shiloach-Vishkin algorithm and employs several novel and efficient hooking strategies for faster convergence. We map different steps of FastSV to linear algebraic operations and implement them with the help of scalable graph libraries. FastSV uses sparse operations to avoid redundant work and optimized MPI communication to avoid bottlenecks. The resultant algorithm shows high-performance and scalability as it can find the connected components of a hyperlink graph with over 134B edges in 30 seconds using 262K cores on a Cray XC40 supercomputer. FastSV outperforms the state-of-the-art algorithm by an average speedup of 2.21x (max 4.27x) on a variety of real-world graphs.
A Collaborative Framework for High-Definition Mapping
Venue: arXiv cs.RO 2020
Year: 2020

For connected vehicles to have a substantial effect on road safety, it is required that accurate positions and trajectories can be shared. To this end, all vehicles must be accurately geolocalized in a common frame. This can be achieved by merging GNSS (Global Navigation Satellite System) information and visual observations matched with a map of geo-positioned landmarks. Building such a map remains a challenge, and current solutions are facing strong cost-related limitations. We present a collaborative framework for high-definition mapping, in which vehicles equipped with standard sensors, such as a GNSS receiver and a mono-visual camera, update a map of geolocalized landmarks. Our system is composed of two processing blocks: the first one is embedded in each vehicle, and aims at geolocalizing the vehicle and the detected feature marks. The second is operated on cloud servers, and uses observations from all the vehicles to compute updates for the map of geo-positioned landmarks. As the map's landmarks are detected and positioned by more and more vehicles, the accuracy of the map increases, eventually converging in probability towards a null error. The landmarks geo-positions are estimated in a stable and scalable way, enabling to provide dynamic map updates in an automatic manner.