Miracle-dz/UniFAR-FTUs huggingface dataset:
Adaptive risk bounds in univariate total variation denoising and trend filtering

We study trend filtering, a relatively recent method for univariate nonparametric regression. For a given positive integer $r$, the $r$-th order trend filtering estimator is defined as the minimizer of the sum of squared errors when we constrain (or penalize) the sum of the absolute $r$-th order discrete derivatives of the fitted function at the design points. For $r=1$, the estimator reduces to total variation regularization which has received much attention in the statistics and image processing literature. In this paper, we study the performance of the trend filtering estimator for every positive integer $r$, both in the constrained and penalized forms. Our main results show that in the strong sparsity setting when the underlying function is a (discrete) spline with few "knots", the risk (under the global squared error loss) of the trend filtering estimator (with an appropriate choice of the tuning parameter) achieves the parametric $n^{-1}$ rate, up to a logarithmic (multiplicative) factor. Our results therefore provide support for the use of trend filtering, for every $r$, in the strong sparsity setting.
The reliability of a nutritional meta-analysis study

Background: Many researchers have studied the relationship between diet and health. There are papers showing an association between the consumption of sugar-sweetened beverages and Type 2 diabetes. Many meta-analyses use individual studies that do not adjust for multiple testing or multiple modeling and thus provide biased estimates of effect. Hence the claims reported in a meta-analysis paper may be unreliable if the primary papers do not ensure unbiased estimates of effect. Objective: Determine the statistical reliability of 10 papers and indirectly the reliability of the meta-analysis study. Method: Ten primary papers used in a meta-analysis paper and counted the numbers of outcomes, predictors, and covariates. We estimated the size of the potential analysis search space available to the authors of these papers; i.e. the number of comparisons and models available. Since we noticed that there were differences between predictors and covariates cited in the abstract and in the text, we applied this formula to information found in the abstracts, Space A, as well as the text, Space T, of each primary paper. Results: The median and range of the number of comparisons possible across the primary papers are 6.5 and (2-12,288) for abstracts, and 196,608 and (3,072-117,117,952) the texts. Note that the median of 6.5 for Space A is misleading as each primary study has 60-165 foods not mentioned in the abstract. Conclusion: Given that testing is at the 0.05 level and the number of comparisons is very large, nominal statistical significance is very weak support for a claim. The claims in these papers are not statistically supported and hence are unreliable. Thus, the claims of the meta-analysis paper lack evidentiary confirmation.
Bayesian Change Point Analysis of Linear Models on Graphs

Consider observations $y_1,\dots,y_n$ on nodes of a connected graph, where the $y_i$ independently come from $N(\theta_i, \sigma^2)$ distributions and an unknown partition divides the $n$ observations into blocks. One well-studied class of change point problems assumes the means $\theta_i$ are equal for all nodes within contiguous blocks of a simple graph of sequential observations; both frequentist and Bayesian approaches have been used to estimate the $\theta_i$ and the change points of the underlying partition. This paper examines a broad class of change point problems on general connected graphs in which a regression model is assumed to apply within each block of the partition of the graph. This general class also supports multivariate change point problems. We use Bayesian methods to estimate change points or block boundaries of the underlying partition. This paper presents the methodology for the general class of change point problems and develops new algorithms for implementation via Markov Chain Monte Carlo. The paper concludes with simulations and real data examples to demonstrate application of the methodology on a wide range of problems.
How to compute the barycenter of a weighted graph

Discrete structures like graphs make it possible to naturally and flexibly model complex phenomena. Since graphs that represent various types of information are increasingly available today, their analysis has become a popular subject of research. The graphs studied in the field of data science at this time generally have a large number of nodes that are not fairly weighted and connected to each other, translating a structural specification of the data. Yet, even an algorithm for locating the average position in graphs is lacking although this knowledge would be of primary interest for statistical or representation problems. In this work, we develop a stochastic algorithm for finding the Frechet mean of weighted undirected metric graphs. This method relies on a noisy simulated annealing algorithm dealt with using homogenization. We then illustrate our algorithm with two examples (subgraphs of a social network and of a collaboration and citation network).
Algebraic problems in structural equation modeling

The paper gives an overview of recent advances in structural equation modeling. A structural equation model is a multivariate statistical model that is determined by a mixed graph, also known as a path diagram. Our focus is on the covariance matrices of linear structural equation models. In the linear case, each covariance is a rational function of parameters that are associated to the edges and nodes of the graph. We statistically motivate algebraic problems concerning the rational map that parametrizes the covariance matrix. We review combinatorial tools such as the trek rule, projection to ancestral sets, and a graph decomposition due to Jin Tian. Building on these tools, we discuss advances in parameter identification, i.e., the study of (generic) injectivity of the parametrization, and explain recent results on determinantal relations among the covariances. The paper is based on lectures given at the 8th Mathematical Society of Japan Seasonal Institute.
Notes on Discrete Compound Poisson Point Process and Its Concentration Inequalities

The first part of this notes provides a new characterization for discrete compound Poisson point process (proposed by {Acz{\'e}l} [Acta~Math.~Hungar.~3(3)(1952), 219-224. ]), which extends the characterization of Poisson point process given by Copeland and Regan [Ann.~Math.~(1936): 357-362. ]. Next, we derive some concentration inequalities for discrete compound Poisson random variable and discrete compound Poisson point process (Poisson and negative binomial are the special cases). These concentration inequalities are potentially useful. In high-dimensional negative binomial regression with weighted Lasso penalty, we give the application that KKT conditions of penalized likelihood holds with high probability.
Differentially Private Significance Tests for Regression Coefficients

ABSTRACT Many data producers seek to provide users access to confidential data without unduly compromising data subjects’ privacy and confidentiality. One general strategy is to require users to do analyses without seeing the confidential data; for example, analysts only get access to synthetic data or query systems that provide disclosure-protected outputs of statistical models. With synthetic data or redacted outputs, the analyst never really knows how much to trust the resulting findings. In particular, if the user did the same analysis on the confidential data, would regression coefficients of interest be statistically significant or not? We present algorithms for assessing this question that satisfy differential privacy. We describe conditions under which the algorithms should give accurate answers about statistical significance. We illustrate the properties of the proposed methods using artificial and genuine data. Supplementary materials for this article are available online.
Adaptive Scan Gibbs Sampler for Large Scale Inference Problems

For large scale on-line inference problems the update strategy is critical for performance. We derive an adaptive scan Gibbs sampler that optimizes the update frequency by selecting an optimum mini-batch size. We demonstrate performance of our adaptive batch-size Gibbs sampler by comparing it against the collapsed Gibbs sampler for Bayesian Lasso, Dirichlet Process Mixture Models (DPMM) and Latent Dirichlet Allocation (LDA) graphical models.
Merging MCMC Subposteriors through Gaussian-Process Approximations

Markov chain Monte Carlo (MCMC) algorithms have become powerful tools for Bayesian inference. However, they do not scale well to large-data problems. Divide-and-conquer strategies, which split the data into batches and, for each batch, run independent MCMC algorithms targeting the corresponding subposterior, can spread the computational burden across a number of separate workers. The challenge with such strategies is in recombining the subposteriors to approximate the full posterior. By creating a Gaussian-process approximation for each log-subposterior density we create a tractable approximation for the full posterior. This approximation is exploited through three methodologies: firstly a Hamiltonian Monte Carlo algorithm targeting the expectation of the posterior density provides a sample from an approximation to the posterior; secondly, evaluating the true posterior at the sampled points leads to an importance sampler that, asymptotically, targets the true posterior expectations; finally, an alternative importance sampler uses the full Gaussian-process distribution of the approximation to the log-posterior density to re-weight any initial sample and provide both an estimate of the posterior expectation and a measure of the uncertainty in it.
Generalized cluster trees and singular measures

In this paper, we study the $\alpha$-cluster tree ($\alpha$-tree) under both singular and nonsingular measures. The $\alpha$-tree uses probability contents within a level set to construct a cluster tree so that it is well-defined for singular measures. We first derive the convergence rate for a density level set around critical points, which leads to the convergence rate for estimating an $\alpha$-tree under nonsingular measures. For singular measures, we study how the kernel density estimator (KDE) behaves and prove that the KDE is not uniformly consistent but pointwisely consistent after rescaling. We further prove that the estimated $\alpha$-tree fails to converge in the $L_\infty$ metric but is still consistent under the integrated distance. We also observe a new type of critical points--the dimensional critical points (DCPs)--of a singular measure. DCPs occur only at singular measures, and similar to the usual critical points, DCPs contribute to cluster tree topology as well. Building on the analysis of the KDE and DCPs, we prove the topological consistency of an estimated $\alpha$-tree.
The Second Order Linear Model

We study a fundamental class of regression models called the second order linear model (SLM). The SLM extends the linear model to high order functional space and has attracted considerable research interest recently. Yet how to efficiently learn the SLM under full generality using nonconvex solver still remains an open question due to several fundamental limitations of the conventional gradient descent learning framework. In this study, we try to attack this problem from a gradient-free approach which we call the moment-estimation-sequence (MES) method. We show that the conventional gradient descent heuristic is biased by the skewness of the distribution therefore is no longer the best practice of learning the SLM. Based on the MES framework, we design a nonconvex alternating iteration process to train a $d$-dimension rank-$k$ SLM within $O(kd)$ memory and one-pass of the dataset. The proposed method converges globally and linearly, achieves $\epsilon$ recovery error after retrieving $O[k^{2}d\cdot\mathrm{polylog}(kd/\epsilon)]$ samples. Furthermore, our theoretical analysis reveals that not all SLMs can be learned on every sub-gaussian distribution. When the instances are sampled from a so-called $\tau$-MIP distribution, the SLM can be learned by $O(p/\tau^{2})$ samples where $p$ and $\tau$ are positive constants depending on the skewness and kurtosis of the distribution. For non-MIP distribution, an addition diagonal-free oracle is necessary and sufficient to guarantee the learnability of the SLM. Numerical simulations verify the sharpness of our bounds on the sampling complexity and the linear convergence rate of our algorithm.
Forward-reverse EM algorithm for Markov chains: convergence and numerical analysis

We develop a forward-reverse EM (FREM) algorithm for estimating parameters that determine the dynamics of a discrete time Markov chain evolving through a certain measurable state space. As a key tool for the construction of the FREM method we develop forward-reverse representations for Markov chains conditioned on a certain terminal state. These representations may be considered as an extension of the earlier work Bayer and Schoenmakers [2013] on conditional diffusions. We proof almost sure convergence of our algorithm for a Markov chain model with curved exponential family structure. On the numerical side we give a complexity analysis of the forward-reverse algorithm by deriving its expected cost. Two application examples are discuss to demonstrate the scope of possible applications ranging from models based on continuous time processes to discrete time Markov chain models.
Quasi-reliable estimates of effective sample size

The efficiency of a Markov chain Monte Carlo algorithm for estimating the mean of a function of interest might be measured by the cost of generating one independent sample, or equivalently, the total cost divided by the effective sample size, defined in terms of the integrated autocorrelation time. To ensure the reliability of such an estimate, it is suggested that there be an adequate sampling of state space— to the extent that this can be determined from the available samples. A sufficient condition for adequate sampling is derived in terms of the supremum of all possible integrated autocorrelation times, which leads to a more stringent condition for adequate sampling than that simply obtained from integrated autocorrelation times for functions of interest. A method for estimating the supremum of all integrated autocorrelation times, based on approximation in a finite-dimensional subspace, is derived and evaluated empirically.
Bayesian matrix completion: prior specification and consistency

Low-rank matrix estimation from incomplete measurements recently received increased attention due to the emergence of several challenging applications, such as recommender systems; see in particular the famous Netflix challenge. While the behaviour of algorithms based on nuclear norm minimization is now well understood [SRJ05, SS05, CP09, CT09, CR09, Gro11, RT11, Klo11, KLT11], an as yet unexplored avenue of research is the behaviour of Bayesian algorithms in this context. In this paper, we briefly review the priors used in the Bayesian literature for matrix completion. A standard approach is to assign an inverse gamma prior to the singular values of a certain singular value decomposition of the matrix of interest; this prior is conjugate. However, we show that two other types of priors (again for the singular values) may be conjugate for this model: a gamma prior, and a discrete prior. Conjugacy is very convenient, as it makes it possible to implement either Gibbs sampling or Variational Bayes. Interestingly enough, the maximum a posteriori for these dierent priors is related to the nuclear norm minimization problems. Our main contribution is to prove the consistency of the posterior expectation when the discrete prior is used. We also compare all these priors on simulated datasets, and on the classical MovieLens and Netflix datasets.
Dr.VAE: Drug Response Variational Autoencoder

We present two deep generative models based on Variational Autoencoders to improve the accuracy of drug response prediction. Our models, Perturbation Variational Autoencoder and its semi-supervised extension, Drug Response Variational Autoencoder (Dr.VAE), learn latent representation of the underlying gene states before and after drug application that depend on: (i) drug-induced biological change of each gene and (ii) overall treatment response outcome. Our VAE-based models outperform the current published benchmarks in the field by anywhere from 3 to 11% AUROC and 2 to 30% AUPR. In addition, we found that better reconstruction accuracy does not necessarily lead to improvement in classification accuracy and that jointly trained models perform better than models that minimize reconstruction error independently.
Approximate Kernel PCA Using Random Features: Computational vs. Statistical Trade-off

Kernel methods are powerful learning methodologies that provide a simple way to construct nonlinear algorithms from linear ones. Despite their popularity, they suffer from poor scalability in big data scenarios. Various approximation methods, including random feature approximation have been proposed to alleviate the problem. However, the statistical consistency of most of these approximate kernel methods is not well understood except for kernel ridge regression wherein it has been shown that the random feature approximation is not only computationally efficient but also statistically consistent with a minimax optimal rate of convergence. In this paper, we investigate the efficacy of random feature approximation in the context of kernel principal component analysis (KPCA) by studying the trade-off between computational and statistical behaviors of approximate KPCA. We show that the approximate KPCA is both computationally and statistically efficient compared to KPCA in terms of the error associated with reconstructing a kernel function based on its projection onto the corresponding eigenspaces. Depending on the eigenvalue decay behavior of the covariance operator, we show that only $n^{2/3}$ features (polynomial decay) or $\sqrt{n}$ features (exponential decay) are needed to match the statistical performance of KPCA. We also investigate their statistical behaviors in terms of the convergence of corresponding eigenspaces wherein we show that only $\sqrt{n}$ features are required to match the performance of KPCA and if fewer than $\sqrt{n}$ features are used, then approximate KPCA has a worse statistical behavior than that of KPCA.
On Bayesian robust regression with diverging number of predictors

This paper concerns the robust regression model when the number of predictors and the number of observations grow in a similar rate. Theory for M-estimators in this regime has been recently developed by several authors [El Karoui et al., 2013, Bean et al., 2013, Donoho and Montanari, 2013]. Motivated by the inability of M-estimators to successfully estimate the Euclidean norm of the coefficient vector, we consider a Bayesian framework for this model. We suggest a two-component mixture of normals prior for the coefficients and develop a Gibbs sampler procedure for sampling from relevant posterior distributions, while utilizing a scale mixture of normal representation for the error distribution . Unlike M-estimators, the proposed Bayes estimator is consistent in the Euclidean norm sense. Simulation results demonstrate the superiority of the Bayes estimator over traditional estimation methods.
CONCENTRATION OF RANDOM GRAPHS AND APPLICATION TO COMMUNITY DETECTION

Random matrix theory has played an important role in recent work on statistical network analysis. In this paper, we review recent results on regimes of concentration of random graphs around their expectation, showing that dense graphs concentrate and sparse graphs concentrate after regularization. We also review relevant network models that may be of interest to probabilists considering directions for new random matrix theory developments, and random matrix theory tools that may be of interest to statisticians looking to prove properties of network algorithms. Applications of concentration results to the problem of community detection in networks are discussed in detail.
Consistent nonparametric estimation for heavy-tailed sparse graphs

We study graphons as a non-parametric generalization of stochastic block models, and show how to obtain compactly represented estimators for sparse networks in this framework. Our algorithms and analysis go beyond previous work in several ways. First, we relax the usual boundedness assumption for the generating graphon and instead treat arbitrary integrable graphons, so that we can handle networks with long tails in their degree distributions. Second, again motivated by real-world applications, we relax the usual assumption that the graphon is defined on the unit interval, to allow latent position graphs where the latent positions live in a more general space, and we characterize identifiability for these graphons and their underlying position spaces. We analyze three algorithms. The first is a least squares algorithm, which gives an approximation we prove to be consistent for all square-integrable graphons, with errors expressed in terms of the best possible stochastic block model approximation to the generating graphon. Next, we analyze a generalization based on the cut norm, which works for any integrable graphon (not necessarily square-integrable). Finally, we show that clustering based on degrees works whenever the underlying degree distribution is atomless. Unlike the previous two algorithms, this third one runs in polynomial time.
Stable and predictive functional domain selection with application to brain images

Motivated by increasing trends of relating brain images to a clinical outcome of interest, we propose a functional domain selection (FuDoS) method that effectively selects subregions of the brain associated with the outcome. View each individual's brain as a 3D functional object, the statistical aim is to distinguish the region where a regression coefficient $\beta(t)=0$ from $\beta(t)\neq0$, where $t$ denotes spatial location. FuDoS is composed of two stages of estimation. We first segment the brain into several small parts based on the correlation structure. Then, potential subsets are built using the obtained segments and their predictive performance are evaluated to select the best subset, augmented by a stability selection criterion. We conduct extensive simulations both for 1D and 3D functional data, and evaluate its effectiveness in selecting the true subregion. We also investigate predictive ability of the selected stable regions. To find the brain regions related to cognitive ability, FuDoS is applied to the ADNI's PET data. Due to the induced sparseness, the results naturally provide more interpretable information about the relations between the regions and the outcome. Moreover, the selected regions from our analysis show high associations with the expected anatomical brain areas known to have memory-related functions.
Sparse canonical correlation analysis

Canonical correlation analysis was proposed by Hotelling [6] and it measures linear relationship between two multidimensional variables. In high dimensional setting, the classical canonical correlation analysis breaks down. We propose a sparse canonical correlation analysis by adding l1 constraints on the canonical vectors and show how to solve it efficiently using linearized alternating direction method of multipliers (ADMM) and using TFOCS as a black box. We illustrate this idea on simulated data.