GenAI4ELab/papercli-papers huggingface dataset:
Triple Component Matrix Factorization: Untangling Global, Local, and Noisy Components
Authors: Naichen Shi; Salar Fattahi; Raed Al Kontar
Venue: JMLR 2024 (main)

Abstract: In this work, we study the problem of common and unique feature extraction from noisy data. When we have $N$ observation matrices from $N$ different and associated sources corrupted by sparse and potentially gross noise, can we recover the common and unique components from these noisy observations? This is a challenging task as the number of parameters to estimate is approximately thrice the number of observations. Despite the difficulty, we propose an intuitive alternating minimization algorithm called triple component matrix factorization (TCMF) to recover the three components exactly. TCMF is distinguished from existing works in literature thanks to two salient features. First, TCMF is a principled method to separate the three components given noisy observations provably. Second, the bulk of the computation in TCMF can be distributed.  On the technical side, we formulate the problem as a constrained nonconvex nonsmooth optimization problem. Despite the intricate nature of the problem, we provide a Taylor series characterization of its solution by solving the corresponding KarushâKuhnâTucker conditions. Using this characterization, we can show that the alternating minimization algorithm makes significant progress at each iteration and converges into the ground truth at a linear rate. Numerical experiments in video segmentation and anomaly detection highlight the superior feature extraction abilities of TCMF.
Open-Source Conversational AI with SpeechBrain 1.0
Authors: Mirco Ravanelli; Titouan Parcollet; Adel Moumen; Sylvain de Langen; Cem Subakan; Peter Plantinga; Yingzhi Wang; Pooneh Mousavi; Luca Della Libera; Artem Ploujnikov; Francesco Paissan; Davide Borra; Salah Zaiem; Zeyu Zhao; Shucong Zhang; Georgios Karakasidis; Sung-Lin Yeh; Pierre Champion; Aku Rouhe; Rudolf Braun; Florian Mai; Juan Zuluaga-Gomez; Seyed Mahed Mousavi; Andreas Nautsch; Ha Nguyen; Xuechen Liu; Sangeet Sagar; Jarod Duret; Salima Mdhaffar; GaÃ«lle LaperriÃ¨re; Mickael Rouvier; Renato De Mori; Yannick EstÃ¨ve
Venue: JMLR 2024 (main)

Abstract: SpeechBrain is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more.
It promotes transparency and replicability by releasing both the pre-trained models and the complete recipes of code and algorithms required for training them.
This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face.
SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.
On the Optimality of Gaussian Kernel Based Nonparametric Tests against Smooth Alternatives
Authors: Tong Li; Ming Yuan
Venue: JMLR 2024 (main)

Abstract: Nonparametric tests via kernel embedding of distributions have witnessed a great deal of practical successes in recent years. However, statistical properties of these tests are largely unknown beyond consistency against a fixed alternative. To fill in this void, we study here the asymptotic properties of goodness-of-fit, homogeneity and independence tests using Gaussian kernels, arguably the most popular and successful among such tests. Our results provide theoretical justifications for this common practice by showing that tests using a Gaussian kernel with an appropriately chosen scaling parameter are minimax optimal against smooth alternatives in all three settings. In addition, our analysis also pinpoints the importance of choosing a diverging scaling parameter when using Gaussian kernels and suggests a data-driven choice of the scaling parameter that yields tests optimal, up to an iterated logarithmic factor, over a wide range of smooth alternatives. Numerical experiments are also presented to further demonstrate the practical merits of the methodology.
ENNS: Variable Selection, Regression, Classification, and Deep Neural Network for High-Dimensional Data
Authors: Kaixu Yang; Arkaprabha Ganguli; Tapabrata Maiti
Venue: JMLR 2024 (main)

Abstract: High-dimensional, low-sample-size (HDLSS) data have been attracting people's attention for a long time. Many studies have proposed different approaches to dealing with this situation, among which variable selection is a significant idea. However, neural networks have been used to model complicated relationships. This paper discusses current variable selection techniques with neural networks. We showed that the stage-wise algorithm with the neural network suffers from some disadvantages, such as that the variables entering the model later may not be consistent. We also proposed an ensemble method to achieve better variable selection and proved that it has a probability tending to zero that a false variable will be selected. Moreover, we discussed further regularization to deal with over-fitting. Simulations and examples of real data are given to support the theory.
On the Convergence of Projected Alternating Maximization for Equitable and Optimal Transport
Authors: Minhui Huang; Shiqian Ma; Lifeng Lai
Venue: JMLR 2024 (main)

Abstract: This paper studies the equitable and optimal transport (EOT) problem, which has many applications such as fair division problems and optimal transport with multiple agents etc. In the discrete distributions case, the EOT problem can be formulated as a linear program (LP). Since this LP is prohibitively large for general LP solvers, (Scetbon et al., 2021) suggests to perturb the problem by adding an entropy regularization. They proposed a projected alternating maximization algorithm (PAM) to solve the dual of the entropy regularized EOT. In this paper, we provide the first convergence analysis of PAM. A novel rounding procedure is proposed to help construct the primal solution for the original EOT problem. We also propose a variant of PAM by incorporating the extrapolation technique that can numerically improve the performance of PAM. Results in this paper may shed lights on block coordinate (gradient) descent methods for general optimization problems.
Inference on High-dimensional Single-index Models with Streaming Data
Authors: Dongxiao Han; Jinhan Xie; Jin Liu; Liuquan Sun; Jian Huang; Bei Jiang; Linglong Kong
Venue: JMLR 2024 (main)

Abstract: Traditional statistical methods are faced with new challenges due to streaming data. The major challenge is the rapidly growing volume and velocity of data, which makes storing such huge data sets in memory impossible. The paper presents an online inference framework for regression parameters in high-dimensional semiparametric single-index models with unknown link functions. The proposed online procedure updates only the current data batch and summary statistics of historical data instead of re-accessing the entire raw data set. At the same time, we do not need to estimate the unknown link function, which is a highly challenging task. In addition, a generalized convex loss function is used in the proposed inference procedure. To illustrate the proposed method, we use the Huber loss function and the negative log-likelihood  of the logistic regression model. In this study, the asymptotic normality of the proposed online debiased Lasso estimators and the bounds of the proposed online Lasso estimators are investigated. To evaluate the performance of the proposed method, extensive simulation studies have been conducted. We provide applications to Nasdaq stock prices and financial distress data sets.
Transfer Learning with Uncertainty Quantification: Random Effect Calibration of Source to Target (RECaST)
Authors: Jimmy Hickey; Jonathan P. Williams; Emily C. Hector
Venue: JMLR 2024 (main)

Abstract: Transfer learning uses a data model, trained to make predictions or inferences on data from one population, to make reliable predictions or inferences on data from another population. Most existing transfer learning approaches are based on fine-tuning pre-trained neural network models, and fail to provide crucial uncertainty quantification. We develop a statistical framework for model predictions based on transfer learning, called RECaST. The primary mechanism is a Cauchy random effect that recalibrates a source model to a target population; we mathematically and empirically demonstrate the validity of our RECaST approach for transfer learning between linear models, in the sense that prediction sets will achieve their nominal stated coverage, and we numerically illustrate the method's robustness to asymptotic approximations for nonlinear models. Whereas many existing techniques are built on particular source models, RECaST is agnostic to the choice of source model, and does not require access to source data.  For example, our RECaST transfer learning approach can be applied to a continuous or discrete data model with linear or logistic regression, deep neural network architectures, etc.  Furthermore, RECaST provides uncertainty quantification for predictions, which is mostly absent in the literature.  We examine our method's performance in a simulation study and in an application to real hospital data.
Bayesian Structural Learning with Parametric Marginals for Count Data: An Application to Microbiota Systems
Authors: Veronica Vinciotti; Pariya Behrouzi; Reza Mohammadi
Venue: JMLR 2024 (main)

Abstract: High dimensional and heterogeneous count data are collected in various applied fields. In this paper, we look closely at high-resolution sequencing data on the microbiome, which have enabled researchers to study the genomes of entire microbial communities. Revealing the underlying interactions between these communities is of vital importance to learn how microbes influence human health. To perform structural learning from multivariate count data such as these, we develop a novel Gaussian copula graphical model with two key elements. Firstly, we employ parametric regression to characterize the marginal distributions. This step is crucial for accommodating the impact of external covariates. Neglecting this adjustment could potentially introduce distortions in the inference of the underlying network of dependences. Secondly, we advance a Bayesian structure learning framework, based on a computationally efficient search algorithm that is suited to high dimensionality. The approach returns simultaneous inference of the marginal effects and of the dependence structure, including graph uncertainty estimates. A simulation study and a real data analysis of microbiome data highlight the applicability of the proposed approach at inferring networks from multivariate count data in general, and its relevance to microbiome analyses in particular. The proposed method is implemented in the R package BDgraph.
Lower Bounds on the Bayesian Risk via Information Measures
Authors: Amedeo Roberto Esposito; Adrien Vandenbroucque; Michael Gastpar
Venue: JMLR 2024 (main)

Abstract: This paper focuses on parameter estimation and introduces a new method for lower bounding the Bayesian risk. The method allows for the use of virtually any information measure, including RÃ©nyi's $\alpha$, $\varphi$-divergences, and Sibson's $\alpha$-Mutual Information. 
The approach considers divergences as functionals of measures and exploits the duality between spaces of measures and spaces of functions. In particular, we show that one can lower bound the risk with any information measure by upper bounding its dual via Markov's inequality.
We are thus able to provide estimator-independent impossibility results thanks to the Data-Processing Inequalities that divergences satisfy.
The results are then applied to settings of interest involving both discrete and continuous parameters, including the âHide-and-Seekâ problem, and compared to the state-of-the-art techniques. An important observation is that the behaviour of the lower bound in the number of samples is influenced by the choice of the information measure. We leverage this by introducing a new divergence inspired by the âHockey-Stickâ divergence, which is demonstrated empirically to provide the largest lower bound across all considered settings. If the observations are subject to privatisation, stronger impossibility results can be obtained via Strong Data-Processing Inequalities. The paper also discusses some generalisations and alternative directions.
Sample Complexity of Variance-Reduced Distributionally Robust Q-Learning
Authors: Shengbo Wang; Nian Si; Jose Blanchet; Zhengyuan Zhou
Venue: JMLR 2024 (main)

Abstract: Dynamic decision-making under distributional shifts is of fundamental interest in theory and applications of reinforcement learning:  The distribution of the environment in which the data is collected can differ from that of the environment in which the model is deployed. This paper presents two novel model-free algorithms, namely the distributionally robust Q-learning and its variance-reduced counterpart, that can effectively learn a robust policy despite distributional shifts. These algorithms are designed to efficiently approximate the $q$-function of an infinite-horizon $\gamma$-discounted robust Markov decision process with Kullback-Leibler ambiguity set to an entry-wise $\epsilon$-degree of precision.  Further, the variance-reduced distributionally robust Q-learning combines the synchronous Q-learning with variance-reduction techniques to enhance its performance. Consequently, we establish that it attains a minimax sample complexity upper bound of $\tilde O(|\mathbf{S}||\mathbf{A}|(1-\gamma)^{-4}\epsilon^{-2})$, where $\mathbf{S}$ and $\mathbf{A}$ denote the state and action spaces.  This is the first complexity result that is independent of the ambiguity size $\delta$, thereby providing new complexity theoretic insights. Additionally, a series of numerical experiments confirm the theoretical findings and the efficiency of the algorithms in handling distributional shifts.
A Characterization of Multioutput Learnability
Authors: Vinod Raman; Unique Subedi; Ambuj Tewari
Venue: JMLR 2024 (main)

Abstract: We consider the problem of learning multioutput function classes in the batch and online settings. In both settings, we show that a multioutput function class is learnable if and only if each single-output restriction of the function class is learnable. This provides a complete characterization of the learnability of multilabel classification and multioutput regression in both batch and online settings. As an extension, we also consider multilabel learnability in the bandit feedback setting and show a similar characterization as in the full-feedback setting.
A Note on Entrywise Consistency for Mixed-data Matrix Completion
Authors: Yunxiao Chen; Xiaoou Li
Venue: JMLR 2024 (main)

Abstract: This note studies matrix completion for a partially observed $n$ by $p$ data matrix involving mixed types of variables (e.g., continuous, binary, ordinal). A general family of non-linear factor models is considered, under which the matrix completion problem becomes the estimation of  an $n$ by $p$ low-rank matrix ${\mathbf M}$. For existing methods in the literature, estimation consistency is established by showing $\Vert \hat {\mathbf M} - {\mathbf M}^*\Vert_F/\sqrt{np}$, the scaled Frobenius norm of the difference between the estimated and true ${\mathbf M}$ matrices, converges to zero in probability as $n$ and $p$ grow to infinity. However, this notion of consistency does not guarantee the convergence of each individual entry and, thus, may not be sufficient when specific data entries or the worst-case scenario is of interest. To address this issue, we consider the notion of entrywise consistency based on $\Vert \hat {\mathbf M} - {\mathbf M}^* \Vert_{\mbox{max}}$, the max norm of the estimation error matrix. We propose refinement procedures that turn estimators, which are consistent in the Frobenius norm sense, into entrywise estimators through a one-step refinement.
Tight probabilistic error bounds are derived for the proposed estimators. The proposed methods are evaluated by simulation studies and real-data applications for collaborative filtering and large-scale educational assessment.
Lower Complexity Adaptation for Empirical Entropic Optimal Transport
Authors: Michel Groppe; Shayan Hundrieser
Venue: JMLR 2024 (main)

Abstract: Entropic optimal transport (EOT) presents an effective and computationally viable alternative to unregularized optimal transport (OT), offering diverse applications for large-scale data analysis. In this work, we derive novel statistical bounds for empirical plug-in estimators of the EOT cost and show that their statistical performance in the entropy regularization parameter $\varepsilon$ and the sample size $n$ only depends on the simpler of the two probability measures. For instance, under sufficiently smooth costs this yields the parametric rate $n^{-1/2}$ with factor $\varepsilon^{-d/2}$, where $d$ is the minimum dimension of the two population measures. This confirms that empirical EOT also adheres to the lower complexity adaptation principle, a hallmark feature only recently identified for unregularized OT. As a consequence of our theory, we show that the empirical entropic Gromov-Wasserstein distance and its unregularized version for measures on Euclidean spaces also obey this principle. Additionally, we comment on computational aspects and complement our findings with Monte Carlo simulations. Our technique employs empirical process theory and relies on a dual formulation of EOT over a single function class. Central to our analysis is the observation that the entropic cost-transformation of a function class does not increase its uniform metric entropy by much.
Causal effects of intervening variables in settings with unmeasured confounding
Authors: Lan Wen; Aaron Sarvet; Mats Stensrud
Venue: JMLR 2024 (main)

Abstract: We present new results on average causal effects in settings with unmeasured exposure-outcome confounding. Our results are motivated by a class of estimands, e.g., frequently of interest in medicine and public health, that are currently not targeted by standard approaches for average causal effects. We recognize these estimands as queries about the average causal effect of an intervening variable. We anchor our introduction of these estimands in an investigation of the role of chronic pain and opioid prescription patterns, and illustrate how conventional approaches will lead to non-replicable estimates with ambiguous policy implications. We argue that our alternative effects are replicable and have clear policy implications, and furthermore are non-parametrically identified by the classical frontdoor formula. As an independent contribution, we derive a new semiparametric efficient estimator of the frontdoor formula with a uniform sample boundedness guarantee. This property is unique among previously-described estimators in its class, and we demonstrate superior performance in finite-sample settings. The theoretical results are applied to data from the National Health and Nutrition Examination Survey.
PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates
Authors: Zachary Frangella; Pratik Rathore; Shipu Zhao; Madeleine Udell
Venue: JMLR 2024 (main)

Abstract: Ill-conditioned problems are ubiquitous in large-scale machine learning: as a data set grows to include more and more features correlated with the labels, the condition number increases. Yet traditional stochastic gradient methods converge slowly on these ill-conditioned problems, even with careful hyperparameter tuning. This paper introduces PROMISE (Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates), a suite of sketching-based preconditioned stochastic gradient algorithms that deliver fast convergence on ill-conditioned large-scale convex optimization problems arising in machine learning. PROMISE includes preconditioned versions of SVRG, SAGA, and Katyusha; each algorithm comes with a strong theoretical analysis and effective default hyperparameter values. Empirically, we verify the superiority of the proposed algorithms by showing that, using default hyperparameter values, they outperform or match popular tuned stochastic gradient optimizers on a test bed of 51 ridge and logistic regression problems assembled from benchmark machine learning repositories. On the theoretical side, this paper introduces the notion of quadratic regularity in order to establish linear convergence of all proposed methods even when the preconditioner is updated infrequently. The speed of linear convergence is determined by the quadratic regularity ratio, which often provides a tighter bound on the convergence rate compared to the condition number, both in theory and in practice, and explains the fast global linear convergence of the proposed methods.
Memorization With Neural Nets: Going Beyond the Worst Case
Authors: Sjoerd Dirksen; Patrick Finke; Martin Genzel
Venue: JMLR 2024 (main)

Abstract: In practice, deep neural networks are often able to easily interpolate their training data. To understand this phenomenon, many works have aimed to quantify the memorization capacity of a neural network architecture: the largest number of points such that the architecture can interpolate any placement of these points with any assignment of labels. For real-world data, however, one intuitively expects the presence of a benign structure so that interpolation already occurs at a smaller network size than suggested by memorization capacity. In this paper, we investigate interpolation by adopting an instance-specific viewpoint. We introduce a simple randomized algorithm that, given a fixed finite data set with two classes, with high probability constructs an interpolating three-layer neural network in polynomial time. The required number of parameters is linked to geometric properties of the two classes and their mutual arrangement. As a result, we obtain guarantees that are independent of the number of samples and hence move beyond worst-case memorization capacity bounds. We verify our theoretical result with numerical experiments and additionally investigate the effectiveness of the algorithm on MNIST and CIFAR-10.
Hamiltonian Monte Carlo for efficient Gaussian sampling: long and random steps
Authors: Simon Apers; Sander Gribling; DÃ¡niel SzilÃ¡gyi
Venue: JMLR 2024 (main)

Abstract: Hamiltonian Monte Carlo (HMC) is a Markov chain algorithm for sampling from a high-dimensional distribution with density $e^{-f(x)}$, given access to the gradient of $f$. A particular case of interest is that of a $d$-dimensional Gaussian distribution with covariance matrix $\Sigma$, in which case $f(x) = x^\top \Sigma^{-1} x$. We show that Metropolis-adjusted HMC can sample from a distribution that is $\varepsilon$-close to a Gaussian in total variation distance using $\widetilde{O}(\sqrt{\kappa} d^{1/4} \log(1/\varepsilon))$ gradient queries, where $\varepsilon>0$ and $\kappa$ is the condition number of $\Sigma$.

Our algorithm uses long and random integration times for the Hamiltonian dynamics, and it creates a warm start by first running HMC without a Metropolis adjustment. This contrasts with (and was motivated by) recent results that give an $\widetilde\Omega(\kappa d^{1/2})$ query lower bound for HMC with a fixed integration times or from a cold start, even for the Gaussian case.