Miracle-dz/UniFAR-FTUs huggingface dataset:
Adaptive Random Walk Gradient Descent for Decentralized Optimization

In this paper, we study the adaptive step size random walk gradient descent with momentum for decentralized optimization, in which the training samples are drawn dependently with each other. We establish theoretical convergence rates of the adaptive step size random walk gradient descent with momentum for both convex and nonconvex settings. In particular, we prove that adaptive random walk algorithms perform as well as the non-adaptive method for dependent data in general cases but achieve acceleration when the stochastic gradients are “sparse”. Moreover, we study the zeroth-order version of adaptive random walk gradient descent and provide corresponding convergence results. All assumptions used in this paper are mild and general, making our results applicable to many machine learning problems.
Fractal Structures and Magnetic Footprints in a Divertor Tokamak

Fractal structures appear very often in open Hamiltonian systems, and can be identiﬁed in the deposition of chaotic magnetic ﬁeld line on the plates of a tokamak divertor. Indeed, tokamaks with divertors are used to control the magnetic conﬁnement of plasmas, such that the ﬁeld lines created by electric currents have escape channels, through which plasma particles can be diverted out of the tokamak wall and redirected to divertor plates. In this work, we use a symplectic map to investigate the deposition patterns on the plates of a divertor. We show that the pattern of magnetic footprints on divertor plates (deposition patterns) underlying the chaotic orbits involves a number of fractal structures related to the existence of a nonattractive invariant chaotic set. In order to investigate the fractal characteristic of magnetic footprints, we analyze quantitatively the degree of stability of the fractal structure by calculating the entropy of the basin. These numerical analyses indicate that the fractal pattern on the divertor plates depends sensibly on the magnetic ﬁeld structure. We show qualitative evidences of the Wada property.
Variational Inference for Infinitely Deep Neural Networks

We introduce the unbounded depth neural network (UDN), an infinitely deep probabilistic model that adapts its complexity to the training data. The UDN contains an infinite sequence of hidden layers and places an unbounded prior on a truncation L, the layer from which it produces its data. Given a dataset of observations, the posterior UDN provides a conditional distribution of both the parameters of the infinite neural network and its truncation. We develop a novel variational inference algorithm to approximate this posterior, optimizing a distribution of the neural network weights and of the truncation depth L, and without any upper limit on L. To this end, the variational family has a special structure: it models neural network weights of arbitrary depth, and it dynamically creates or removes free variational parameters as its distribution of the truncation is optimized. (Unlike heuristic approaches to model search, it is solely through gradient-based optimization that this algorithm explores the space of truncations.) We study the UDN on real and synthetic data. We find that the UDN adapts its posterior depth to the dataset complexity; it outperforms standard neural networks of similar computational complexity; and it outperforms other approaches to infinite-depth neural networks.
Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding

Semi-supervised learning (SSL) has proven to be successful in overcoming labeling difﬁculties by leveraging unlabeled data. Previous SSL algorithms typically assume a balanced class distribution. However, real-world datasets are usually class-imbalanced, causing the performance of existing SSL algorithms to be seriously decreased. One essential reason is that pseudo-labels for unlabeled data are selected based on a ﬁxed conﬁ-dence threshold, resulting in low performance on minority classes. In this paper, we develop a simple yet effective framework, which only involves adaptive thresholding for different classes in SSL algorithms, and achieves remarkable performance improvement on more than twenty imbalance ra-tios. Speciﬁcally, we explicitly optimize the number of pseudo-labels for each class in the SSL objective, so as to simultaneously obtain adaptive thresholds and minimize empirical risk. More-over, the determination of the adaptive threshold can be efﬁciently obtained by a closed-form solu-tion. Extensive experimental results demonstrate the effectiveness of our proposed algorithms.
Kill a Bird with Two Stones: Closing the Convergence Gaps in Non-Strongly Convex Optimization by Directly Accelerated SVRG with Double Compensation and Snapshots

Recently, some accelerated stochastic variance reduction algorithms such as Katyusha and ASVRG-ADMM achieve faster convergence than non-accelerated methods such as SVRG and SVRG-ADMM. However, there are still some gaps between the oracle complexities and their lower bounds. To ﬁll in these gaps, this paper propos-es a novel Directly Accelerated stochastic Variance reductIon algorithm with two Snapshots (DAVIS) for non-strongly convex (non-SC) unconstrained problems. Our theoretical results show that DAVIS achieves the optimal convergence rate O (1 / ( nS 2 )) and optimal gradient complexity O ( n + (cid:112) nL/(cid:15) ) , which is identical to its lower bound. To the best of our knowledge, this is the ﬁrst directly accelerated algorithm that attains the lower bound and improves the convergence rate from O (1 /S 2 ) to O (1 / ( nS 2 )) . More-over, we extend DAVIS and theoretical results to non-SC problems with an equality constraint, and prove that the proposed DAVIS-ADMM algorithm with double snapshots for each variable also attains the optimal convergence rate O (1 / ( nS )) and optimal oracle complexity O (cid:0) n + L/(cid:15) (cid:1) for such problems, and it is at least by a factor n/S faster than existing accelerated stochastic algorithms,
On Non-local Convergence Analysis of Deep Linear Networks

In this paper, we study the non-local convergence properties of deep linear networks. Specifically, under the quadratic loss, we consider optimizing deep linear networks in which there is at least a layer with only one neuron. We describe the convergent point of trajectories with an arbitrary balanced starting point under gradient flow, including the paths which converge to one of the saddle points. We also show specific convergence rates of trajectories that converge to the global minimizers by stages. We conclude that the rates vary from polynomial to linear. As far as we know, our results are the first to give a non-local analysis of deep linear neural networks with arbitrary balanced initialization, rather than the lazy training regime which has dominated the literature of neural networks or the restricted benign initialization.
Asymptotically-Optimal Gaussian Bandits with Side Observations

We study the problem of Gaussian bandits with general side information, as first introduced by Wu, Szepesv´ari, and Gy¨orgy. In this setting, the play of an arm reveals information about other arms, according to an arbitrary a priori known side information matrix: each element of this matrix encodes the fidelity of the information that the “row” arm reveals about the “column” arm. In the case of Gaussian noise, this model subsumes standard bandits, full-feedback, and graph-structured feedback as special cases. In this work, we first construct an LP-based asymptotic instance-dependent lower bound on the regret. The LP optimizes the cost (regret) required to reliably estimate the suboptimality gap of each arm. This LP lower bound motivates our main contribution: the first known asymptotically optimal algorithm for this general setting.
Mitigating Gender Bias in Face Recognition using the von Mises-Fisher Mixture Model

In spite of the high performance and reliability of deep learning algorithms in a wide range of everyday applications, many investigations tend to show that a lot of models exhibit biases, discriminating against specific subgroups of the population (e.g. gender, ethnicity). This urges the practitioner to develop fair systems with a uniform/comparable performance across sensitive groups. In this work, we investigate the gender bias of deep Face Recognition networks. In order to measure this bias, we introduce two new metrics, $\mathrm{BFAR}$ and $\mathrm{BFRR}$, that better reflect the inherent deployment needs of Face Recognition systems. Motivated by geometric considerations, we mitigate gender bias through a new post-processing methodology which transforms the deep embeddings of a pre-trained model to give more representation power to discriminated subgroups. It consists in training a shallow neural network by minimizing a Fair von Mises-Fisher loss whose hyperparameters account for the intra-class variance of each gender. Interestingly, we empirically observe that these hyperparameters are correlated with our fairness metrics. In fact, extensive numerical experiments on a variety of datasets show that a careful selection significantly reduces gender bias. The code used for the experiments can be found at https://github.com/JRConti/EthicalModule_vMF.
Entropic Causal Inference: Graph Identifiability

Entropic causal inference is a recent framework for learning the causal graph between two variables from observational data by finding the information-theoretically simplest structural explanation of the data, i.e., the model with smallest entropy. In our work, we first extend the causal graph identifiability result in the two-variable setting under relaxed assumptions. We then show the first identifiability result using the entropic approach for learning causal graphs with more than two nodes. Our approach utilizes the property that ancestrality between a source node and its descendants can be determined using the bivariate entropic tests. We provide a sound sequential peeling algorithm for general graphs that relies on this property. We also propose a heuristic al-gorithm for small graphs that shows strong empirical performance. We rigorously evaluate the performance of our algorithms on synthetic data generated from a variety of models, observing improvement over prior work. Finally we test our algorithms on real-world datasets.
Hybrid beamforming for massive MIMO systems.

Massive multiple-input multiple-output (MIMO) technology is considered as one of the enabling technologies to scale up the data rates for the future communication systems. Traditional MIMO systems employ digital beamforming where each antenna element is equipped with one radio frequency (RF) chain. When the number of the antennas are scaled up, the cost and power consumption of massive MIMO systems also increase significantly. Recently, hybrid analog-and-digital beamformers have attracted a lot of attention as a cost effective approach to benefit from the advantages of massive MIMO. In hybrid structure, a small number of RF chains are connected to a large number of antennas through a network of phase shifters. The optimal hybrid beamforming problem is a complex nonconvex optimization due to the nonconvex constraint imposed by phase shifters. The overall objective of this thesis is to provide simple and effective hybrid beamforming solutions for narrowband point-to-point and multiuser massive MIMO scenarios. Firstly, hybrid beamforming problem for a point-to-point communication system with perfect channel state information (CSI) is investigated, and an effective codebook based hybrid beamforming with low resolution phase shifters is proposed which is suitable for sparse scattering channels. Then, by leveraging the properties of massive MIMO, an asymptotically optimal hybrid beamforming solution as well as its closed-form formula will be presented. It will be shown that the proposed method is effective in both sparse and rich scattering propagation environments. In addition, the closed-form expression and lower-bounds for the achievable rates are derived when analog and digital phase shifters are employed. Secondly, hybrid beamforming problem to maximise the total sum-rate for the downlink of multiuser MIMO is investigated, and an effective solution as well as its closed-form expression for this system is proposed. The presented solutions for the single-antenna and multiantenna scenarios are shown to be effective as they can achieve a similar sum-rate as digital beamforming can reach. In addition, it is shown that the proposed technique with low-cost low resolution phase shifters at the RF beamformer demonstrates a comparable performance to that of a hybrid beamformer with an expensive analog beamformer. Finally, two novel hybrid beamforming techniques are proposed to reduce the power consumption at the RF beamformer. Defining a threshold level, it is shown that half of the phase shifters could be turned off without a performance loss when the wireless channel matrix is modeled by Rayleigh fading. Then, we reduce the number of the phase shifters by using a combination of phase shifters and switches at the RF beamformer. The proposed methods can significantly reduce the power consumption as switches, in general, have lower power consumption compared to phase shifters. It is noted that the presented algorithms and the closed-form expressions of their performance are derived by using the asymptotic properties of the elements of the singular vectors for the rich scattering channel matrix.
Learning Pseudometric-based Action Representations for Offline Reinforcement Learning

Offline reinforcement learning is a promising approach for practical applications since it does not require interactions with real-world environments. However, existing offline RL methods only work well in environments with continuous or small discrete action spaces. In environments with large and discrete action spaces, such as recommender systems and dialogue systems, the performance of existing methods decreases drastically because they suffer from inaccurate value estimation for a large proportion of out-of-distribution (o.o.d.) actions. While recent works have demonstrated that online RL benefits from incorporating semantic information in action representations, unfortunately, they fail to learn reasonable relative distances between action representations, which is key to offline RL to reduce the influence of o.o. d. actions. This paper proposes an action representation learning framework for offline RL based on a pseudometric, which measures both the behavioral relation and the data-distributional relation between actions. We provide theoretical analysis on the continuity of the expected Q-values and the offline policy improvement using the learned action representations. Experimental results show that our methods significantly improve the performance of two typical offline RL methods in environments with large and discrete action spaces.
Validating Causal Inference Methods

The fundamental challenge of drawing causal inference is that counterfactual outcomes are not fully observed for any unit. Furthermore, in observational studies, treatment assignment is likely to be confounded. Many statistical methods have emerged for causal inference under unconfoundedness conditions given pre-treatment covariates, including propensity score-based methods, prognostic score-based methods, and doubly robust methods. Unfortunately for applied researchers, there is no `one-size-fits-all' causal method that can perform optimally universally. In practice, causal methods are primarily evaluated quantitatively on handcrafted simulated data. Such data-generative procedures can be of limited value because they are typically stylized models of reality. They are simplified for tractability and lack the complexities of real-world data. For applied researchers, it is critical to understand how well a method performs for the data at hand. Our work introduces a deep generative model-based framework, Credence, to validate causal inference methods. The framework's novelty stems from its ability to generate synthetic data anchored at the empirical distribution for the observed sample, and therefore virtually indistinguishable from the latter. The approach allows the user to specify ground truth for the form and magnitude of causal effects and confounding bias as functions of covariates. Thus simulated data sets are used to evaluate the potential performance of various causal estimation methods when applied to data similar to the observed sample. We demonstrate Credence's ability to accurately assess the relative performance of causal estimation techniques in an extensive simulation study and two real-world data applications from Lalonde and Project STAR studies.
Spectral Representation of Robustness Measures for Optimization Under Input Uncertainty

We study the inference of mean-variance robustness measures to quantify input uncertainty under the Gaussian Process (GP) framework. These measures are widely used in applications where the robustness of the solution is of interest, for example, in engineering design. While the variance is commonly used to characterize the robustness, Bayesian inference of the variance using GPs is known to be challenging. In this paper, we propose a Spectral Representation of Robustness Measures based on the GP’s spectral representation, i.e., an analytical approach to approximately infer both robustness measures for normal and uniform input uncertainty distributions. We present two approximations based on different Fourier features and compare their accuracy numerically. To demonstrate their utility and efficacy in robust Bayesian Optimization, we integrate the analytical robustness measures in three standard acquisition functions for various robust optimization formulations. We show their competitive performance on numerical benchmarks and real-life applications.
Fast Provably Robust Decision Trees and Boosting

Learning with adversarial robustness has been a challenge in contemporary machine learning, and recent years have witnessed increasing attention on robust decision trees and ensembles, mostly working with high computational complexity or without guarantees of provable robustness. This work proposes the Fast Provably Robust Decision Tree (FPRDT) with the smallest computational complexity O ( n log n ) , a tradeoff between global and local optimizations over the adversarial 0 / 1 loss. We further develop the Provably Robust AdaBoost (PRAdaBoost) according to our robust decision trees, and present convergence analysis for training adversarial 0 / 1 loss. We conduct extensive experiments to support our approaches; in particular, our approaches are superior to those unprovably robust methods, and achieve better or comparable performance to those provably robust methods yet with the smallest running time.
G2CN: Graph Gaussian Convolution Networks with Concentrated Graph Filters

Recently, linear GCNs have shown competitive performance against non-linear ones with less computation cost, and the key lies in their propagation layers. Spectral analysis has been widely adopted in designing and analyzing existing graph propagations. Nevertheless, we notice that existing spectral analysis fails to explain why existing graph propagations with the same global tendency, such as low-pass or high-pass, still yield very different results. Motivated by this situation, we develop a new framework for spectral analysis in this paper called concentration analysis. In particular, we propose three attributes: concentration centre, maximum response, and bandwidth for our analysis. Through a dissection of the limitations of existing graph propagations via the above analysis, we propose a new kind of propagation layer, Graph Gaussian Convolution Networks (G 2 CN), in which the three properties are decoupled and the whole structure becomes more ﬂexible and applicable to different kinds of graphs. Extensive experiments show that we can obtain state-of-the-art performance on heterophily and homophily datasets with our proposed G 2 CN.
Greedy when Sure and Conservative when Uncertain about the Opponents

We develop a new approach, named Greedy when Sure and Conservative when Uncertain (GSCU), to competing online against unknown and non-stationary opponents. GSCU improves in four aspects: 1) introduces a novel way of learning opponent policy embeddings offline; 2) trains offline a single best response (conditional additionally on our opponent policy embedding) instead of a finite set of separate best responses against any opponent; 3) computes online a posterior of the current opponent policy embedding, without making the discrete and ineffective decision which type the current opponent belongs to; and 4) selects online between a real-time greedy policy and a fixed conservative policy via an adversarial bandit algorithm, gaining a theoretically better regret than adhering to either. Experimental studies on popular benchmarks demonstrate GSCU’s superiority over the state-of-the-art meth-ods. The code is available online at https: //github.com/YeTianJHU/GSCU .
Neurocoder: General-Purpose Computation Using Stored Neural Programs

Artiﬁcial Neural Networks are functionally equivalent to special-purpose computers. Their inter-neuronal connection weights represent the learnt Neural Program that instructs the networks on how to compute the data. However, without storing Neural Programs, they are restricted to only one, overwriting learnt programs when trained on new data. Here we design Neurocoder, a new class of general-purpose neural networks in which the neural network “codes” itself in a data-responsive way by composing relevant programs from a set of shareable, modular programs stored in external memory. This time, a Neural Program is efﬁciently treated as data in memory. Integrating Neurocoder into current neural architectures, we demonstrate new capacity to learn modular programs, reuse simple programs to build complex ones, handle pattern shifts and remember old programs as new ones are learnt, and show substantial performance improvement in solving object recognition, playing video games and continual learning tasks.
Sequential Covariate Shift Detection Using Classifier Two-Sample Tests

A standard assumption in supervised learning is that the training data and test data are from the same distribution. However, this assumption often fails to hold in practice, which can cause the learned model to perform poorly. We consider the problem of detecting covariate shift , where the covariate distribution shifts but the conditional distribution of labels given covariates remains the same. This problem can naturally be solved using a two-sample test— i.e. , test whether the current test distribution of covariates equals the training distribution of covariates. Our algorithm builds on classifier tests , which train a discriminator to distinguish train and test covariates, and then use the accuracy of this discriminator as a test statistic. A key challenge is that classifier tests assume given a fixed set of test covariates. In practice, test covariates often arrive sequentially over time—e.g., a self-driving car observes a stream of images while driving. Furthermore, covariate shift can occur multiple times— i.e. , shift and then shift back later or gradually shift over time. To address these challenges, our algorithm trains the discriminator online. Additionally, it evaluates test accuracy using each new covariate before taking a gradient step; this strategy avoids constructing a held-out test set, which can improve sample efficiency. We prove that this optimization preserves the correctness— i.e. , our algorithm achieves a desired bound on the false positive rate. In our experiments, we show that our algorithm efficiently detects covariate shifts on multiple datasets—ImageNet, IWildCam, and Py150.