Miracle-dz/UniFAR-FTUs huggingface dataset:
Thermodynamic assessment of probability distribution divergencies and Bayesian model comparison

Within path sampling framework, we show that probability distribution divergences, such as the Chernoff information, can be estimated via thermodynamic integration. The Boltzmann-Gibbs distribution pertaining to different Hamiltonians is implemented to derive tempered transitions along the path, linking the distributions of interest at the endpoints. Under this perspective, a geometric approach is feasible, which prompts intuition and facilitates tuning the error sources. Additionally, there are direct applications in Bayesian model evaluation. Existing marginal likelihood and Bayes factor estimators are reviewed here along with their stepping-stone sampling analogues. New estimators are presented and the use of compound paths is introduced.
Fast and exact implementation of 3-dimensional Tukey depth regions

Tukey depth regions are important notions in nonparametric multivariate data analysis. A $\tau$-th Tukey depth region $\mathcal{D}_{\tau}$ is the set of all points that have at least depth $\tau$. While the Tukey depth regions are easily defined and interpreted as $p$-variate quantiles, their practical applications is impeded by the lack of efficient computational procedures in dimensions with $p > 2$. Feasible algorithms are available, but practically very slow. In this paper we present a new exact algorithm for 3-dimensional data. An efficient implementation is also provided. Data examples indicate that the proposed algorithm runs much faster than the existing ones.
On the estimators of autocorrelation model parameters

Estimation of autocorrelations and spectral densities is of fundamental importance in many fields of science, from identifying pulsar signals in astronomy to measuring heart beats in medicine. In circumstances where one is interested in specific autocorrelation functions that do not fit into any simple families of models, such as auto-regressive moving average (ARMA), estimating model parameters is generally approached in one of two ways: by fitting the model autocorrelation function to a non-parameteric autocorrelation estimate via regression analysis or by fitting the model autocorrelation function directly to the data via maximum likelihood. Prior literature suggests that variogram regression yields parameter estimates of comparable quality to maximum likelihood. In this letter we demonstrate that, as sample size is increases, the accuracy of the maximum-likelihood estimates (MLE) ultimately improves by orders of magnitude beyond that of variogram regression. For relatively continuous and Gaussian processes, this improvement can occur for sample sizes of less than 100. Moreover, even where the accuracy of these methods is comparable, the MLE remains almost universally better and, more critically, variogram regression does not provide reliable confidence intervals. Inaccurate regression parameter estimates are typically accompanied by underestimated standard errors, whereas likelihood provides reliable confidence intervals.
A simple scheme for the parallelization of particle filters and its application to the tracking of complex stochastic systems

We investigate the use of possibly the simplest scheme for the parallelisation of the standard particle filter, that consists in splitting the computational budget into $M$ fully independent particle filters with $N$ particles each, and then obtaining the desired estimators by averaging over the $M$ independent outcomes of the filters. This approach minimises the parallelisation overhead yet displays highly desirable theoretical properties. Under very mild assumptions, we analyse the mean square error (MSE) of the estimators of 1-dimensional statistics of the optimal filtering distribution and show explicitly the effect of parallelisation scheme on the convergence rate. Specifically, we study the decomposition of the MSE into variance and bias components, to show that the former decays as $\frac{1}{MN}$, i.e., linearly with the total number of particles, while the latter converges towards $0$ as $\frac{1}{N^2}$. Parallelisation, therefore, has the obvious advantage of dividing the running times while preserving the (asymptotic) performance of the particle filter. Following this lead, we propose a time-error index to compare schemes with different degrees of parallelisation. Finally, we provide two numerical examples. The first one deals with the tracking of a Lorenz 63 chaotic system with dynamical noise and partial (noisy) observations, while the second example involves a dynamical network of modified FitzHugh-Nagumo (FH-N) stochastic nodes. The latter is a large dimensional system ($\approx3,000$ state variables in our computer experiments) designed to numerically reproduce typical electrical phenomena observed in the atria of the human heart. In both examples, we show how the proposed parallelisation scheme attains the same approximation accuracy as a centralised particle filter with only a small fraction of the running time, using a standard multicore computer.
Parametric Sequential Causal Inference in Point Parametrization

Suppose that a sequence of treatments are assigned to influence an outcome of interest that occurs after the last treatment. Between treatments there exist time-dependent covariates that may be posttreatment variables of the earlier treatments and confounders of the subsequent treatments. In this article, we develop a parametric approach to inference of the causal effect of the treatment sequence on the outcome called the sequential causal effect. We construct a point parametrization for the conditional distribution of an outcome given all treatments and time-dependent covariates, in which the point parameters of interest are the point effects of treatments considered as single-point treatments. We (1) identify net effects of treatments by point effects of treatments, (2) express patterns of net effects of treatments by constraints on point effects of treatments, and (3) show that all sequential causal effects are determined by net effects of treatments. Accordingly we (1) estimate net effects of treatments through point effects of treatments by maximum likelihood, (2) improve the estimation by constraints on point effects of treatments and assignment conditions of treatments, and (3) use the estimates of net effects of treatments to obtain those of sequential causal effects. As a result, we obtain unbiased consistent maximum-likelihood estimates of sequential causal effects even for long treatment sequences. For illustration of our method, we study the causal effects of various sequences of recreational drugs on the CD4 count among HIV patients.
Embedding Graphs under Centrality Constraints for Network Visualization

Visual rendering of graphs is a key task in the mapping of complex network data. Although most graph drawing algorithms emphasize aesthetic appeal, certain applications such as travel-time maps place more importance on visualization of structural network properties. The present paper advocates two graph embedding approaches with centrality considerations to comply with node hierarchy. The problem is formulated first as one of constrained multi-dimensional scaling (MDS), and it is solved via block coordinate descent iterations with successive approximations and guaranteed convergence to a KKT point. In addition, a regularization term enforcing graph smoothness is incorporated with the goal of reducing edge crossings. A second approach leverages the locally-linear embedding (LLE) algorithm which assumes that the graph encodes data sampled from a low-dimensional manifold. Closed-form solutions to the resulting centrality-constrained optimization problems are determined yielding meaningful embeddings. Experimental results demonstrate the efficacy of both approaches, especially for visualizing large networks on the order of thousands of nodes.
Additive Gaussian Process Regression

Additive-interactive regression has recently been shown to offer attractive minimax error rates over traditional nonparametric multivariate regression in a wide variety of settings, including cases where the predictor count is much larger than the sample size and many of the predictors have important effects on the response, potentially through complex interactions. We present a Bayesian implementation of additive-interactive regression using an additive Gaussian process (AGP) prior and develop an efficient Markov chain sampler that extends stochastic search variable selection in this setting. Careful prior and hyper-parameter specification are developed in light of performance and computational considerations, and key innovations address difficulties in exploring a joint posterior distribution over multiple subsets of high dimensional predictor inclusion vectors. The method offers state-of-the-art support and interaction recovery while improving dramatically over competitors in terms of prediction accuracy on a diverse set of simulated and real data. Results from real data studies provide strong evidence that the additive-interactive framework is an attractive modeling platform for high-dimensional nonparametric regression.
Selective Factor Extraction in High Dimensions

This paper studies simultaneous feature selection and extraction in supervised and unsupervised learning. We propose and investigate selective reduced rank regression for constructing optimal explanatory factors from a parsimonious subset of input features. The proposed estimators enjoy sharp oracle inequalities, and with a predictive information criterion for model selection, they adapt to unknown sparsity by controlling both rank and row support of the coefficient matrix. A class of algorithms is developed that can accommodate various convex and nonconvex sparsity-inducing penalties, and can be used for rank-constrained variable screening in high-dimensional multivariate data. The paper also showcases applications in macroeconomics and computer vision to demonstrate how low-dimensional data structures can be effectively captured by joint variable selection and projection.
Constant Modulus Algorithms Using Hyperbolic Givens Rotation

We propose two new algorithms to minimize the constant modulus (CM) criterion in the context of blind source separation. The first algorithm, referred to as Givens CMA (G-CMA) uses unitary Givens rotations and proceeds in two stages: prewhitening step, which reduces the channel matrix to a unitary one followed by a separation step where the resulting unitary matrix is computed using Givens rotations by minimizing the CM criterion. However, for small sample sizes, the prewhitening does not make the channel matrix close enough to unitary and hence applying Givens rotations alone does not provide satisfactory performance. To remediate to this problem, we propose to use non-unitary Shear (Hyperbolic) rotations in conjunction with Givens rotations. This second algorithm referred to as Hyperbolic G-CMA (HG-CMA) is shown to outperform the G-CMA as well as the Analytical CMA (ACMA) in terms of separation quality. The last part of this paper is dedicated to an efficient adaptive implementation of the HG-CMA and to performance assessment through numerical experiments.
Classification and Bayesian Optimization for Likelihood-Free Inference

Abstract Some statistical models are speciﬁed via a data generating process forwhich the likelihood function cannot be computed in closed form. Standardlikelihood-based inference is then not feasible but the model parameters canbe inferred by ﬁnding the values which yield simulated data that resemblethe observed data. This approach faces at least two major diﬃculties: Theﬁrst diﬃculty is the choice of the discrepancy measure which is used tojudge whether the simulated data resemble the observed data. The seconddiﬃculty is the computationally eﬃcient identiﬁcation of regions in the pa-rameter space where the discrepancy is low. We give here an introductionto our recent work where we tackle the two diﬃculties through classiﬁcationand Bayesian optimization. 1 Introduction The likelihood function plays a central role in statistics and machine learning. It is the jointprobability of the observed data X seen as a function of the model parameters of interest θ . We may assume that the data X
Religious affiliation, education and Internet use

Using data from the General Social Survey, we measure the effect of education and Internet use on religious affiliation. We find that Internet use is associated with decreased probability of religious affiliation; for moderate use (2 or more hours per week) the odds ratio is 0.82 (CI 0.69–0.98, p = 0.01). For heavier use (7 or more hours per week) the odds ratio is 0.58 (CI 0.41–0.81, p < 0.001). In the 2010 U.S. population, Internet use could account for 5.1 million people with no religious affiliation, or 20% of the observed decrease in affiliation relative to the 1980s. Increases in college graduation between the 1980s and 2000s could account for an additional 5% of the decrease. From 1990 to 2010 the fraction of people in the United States with no religious preference increased from 8% to 18%, based on data from the General Social Survey (GSS). At the same time, the fraction of Protestants dropped from 62% to 51%; the fraction of Catholics and Jews did not change significantly; the fraction of other religions increased from 3.3% to 4.2%. During the same period, the prevalence of Internet use increased from essentially zero to nearly 80%. Figure 1 shows Internet users per 100 people and the fraction of the population with no religious preference. Also during this period, the fraction of the population graduating from college increased from 17% to 27%. In this paper we explore the effect of increases in college education and Internet use on religious affiliation. Of course there are other factors that might contribute to religious disaffiliation. Hout and Fischer investigate this trend (through 2000) and identify ∗Professor of Computer Science, Olin College of Engineering, Needham MA 02492. Email: allen.downey@olin.edu, Web: allendowney.com 1 ar X iv :1 40 3. 55 34 v1 [ st at . A P] 2 1 M ar 2 01 4 1975 198
New Accumulative Score Function Based Bound For Sparsity Level of L1 Minimization

This paper discusses a fundamental problem in compressed sensing: the sparse recoverability of L1 minimization with an arbitrary sensing matrix. We develop an new accumulative score function (ASF) to provide a lower bound for the recoverable sparsity level (SL) of a sensing matrix while preserving a low computational complexity. We first define a score function for each row of a matrix, and then ASF sums up large scores until the total score reaches 0.5. Interestingly, the number of involved rows in the summation is a reliable lower bound of SL. It is further proved that ASF provides a sharper bound for SL than coherence We also investigate the underlying relationship between the new ASF and the classical RIC and achieve a RIC-based bound for SL.
Graphical Modeling of Spatial Health Data

The literature on Gaussian graphical models (GGMs) contains two equally rich and equally significant domains of research efforts and interests. The first research domain relates to the problem of graph determination. That is, the underlying graph is unknown and needs to be inferred from the data. The second research domain dominates the applications in spatial epidemiology. In this context GGMs are typically referred to as Gaussian Markov random fields (GMRFs). Here the underlying graph is assumed to be known: the vertices correspond to geographical areas, while the edges are associated with areas that are considered to be neighbors of each other (e.g., if they share a border). We introduce multi-way Gaussian graphical models that unify the statistical approaches to inference for spatiotemporal epidemiology with the literature on general GGMs. The novelty of the proposed work consists of the addition of the G-Wishart distribution to the substantial collection of statistical tools used to model multivariate areal data. As opposed to fixed graphs that describe geography, there is an inherent uncertainty related to graph determination across the other dimensions of the data. Our new class of methods for spatial epidemiology allow the simultaneous use of GGMs to represent known spatial dependencies and to determine unknown dependencies in the other dimensions of the data. KEYWORDS: Gaussian graphical models, Gaussian Markov random fields, spatiotemporal multivariate models
Valid uncertainty quantification about the model in a linear regression setting

In scientific applications, there often are several competing models that could be fit to the observed data, so quantification of the model uncertainty is of fundamental importance. In this paper, we develop an inferential model (IM) approach for simultaneously valid probabilistic inference over a collection of assertions of interest without requiring any prior input. Our construction guarantees that the approach is optimal in the sense that it is the most efficient among those which are valid. Connections between the IM's simultaneous validity and post-selection inference are also made. We apply the general results to obtain valid uncertainty quantification about the set of predictor variables to be included in a linear regression model.
Level Sets Based Distances for Probability Measures and Ensembles with Applications

In this paper we study Probability Measures (PM) from a functional point of view: we show that PMs can be considered as functionals (generalized functions) that belong to some functional space endowed with an inner product. This approach allows us to introduce a new family of distances for PMs, based on the action of the PM functionals on `interesting' functions of the sample. We propose a specific (non parametric) metric for PMs belonging to this class, based on the estimation of density level sets. Some real and simulated data sets are used to measure the performance of the proposed distance against a battery of distances widely used in Statistics and related areas.
On the Generalization of the C-Bound to Structured Output Ensemble Methods

This paper generalizes an important result from the PAC-Bayesian literature for binary classification to the case of ensemble methods for structured outputs. We prove a generic version of the \Cbound, an upper bound over the risk of models expressed as a weighted majority vote that is based on the first and second statistical moments of the vote's margin. This bound may advantageously $(i)$ be applied on more complex outputs such as multiclass labels and multilabel, and $(ii)$ allow to consider margin relaxations. These results open the way to develop new ensemble methods for structured output prediction with PAC-Bayesian guarantees.
Building an NCAA men’s basketball predictive model and quantifying its success

Abstract Computing and machine learning advancements have led to the creation of many cutting-edge predictive algorithms, some of which have been demonstrated to provide more accurate forecasts than traditional statistical tools. In this manuscript, we provide evidence that the combination of modest statistical methods with informative data can meet or exceed the accuracy of more complex models when it comes to predicting the NCAA men’s basketball tournament. First, we describe a prediction model that merges the point spreads set by Las Vegas sportsbooks with possession based team efficiency metrics by using logistic regressions. The set of probabilities generated from this model most accurately predicted the 2014 tournament, relative to approximately 400 competing submissions, as judged by the log loss function. Next, we attempt to quantify the degree to which luck played a role in the success of this model by simulating tournament outcomes under different sets of true underlying game probabilities. We estimate that under the most optimistic of game probability scenarios, our entry had roughly a 12% chance of outscoring all competing submissions and just less than a 50% chance of finishing with one of the ten best scores.
On the exact region determined by Kendall's τ and Spearman's ρ

Using properties of shuffles of copulas and tools from combinatorics we solve the open question about the exact region Ω determined by all possible values of Kendall's τ and Spearman's ρ. In particular, we prove that the well‐known inequality established by Durbin and Stuart in 1951 is not sharp outside a countable set, give a simple analytic characterization of Ω in terms of a continuous, strictly increasing piecewise concave function and show that Ω is compact and simply connected, but not convex. The results also show that for each (x,y) ∈ Ω there are mutually completely dependent random variables X and Y whose τ‐ and ρ‐values coincide with x and y respectively.
Persistent Homology Transform for Modeling Shapes and Surfaces

In this paper we introduce a statistic, the persistent homology transform (PHT), to model surfaces in $\mathbb{R}^3$ and shapes in $\mathbb{R}^2$. This statistic is a collection of persistence diagrams - multiscale topological summaries used extensively in topological data analysis. We use the PHT to represent shapes and execute operations such as computing distances between shapes or classifying shapes. We prove the map from the space of simplicial complexes in $\mathbb{R}^3$ into the space spanned by this statistic is injective. This implies that the statistic is a sufficient statistic for probability densities on the space of piecewise linear shapes. We also show that a variant of this statistic, the Euler Characteristic Transform (ECT), admits a simple exponential family formulation which is of use in providing likelihood based inference for shapes and surfaces. We illustrate the utility of this statistic on simulated and real data.
A Class of Conjugate Priors Defined on the Unit Simplex

Dirichlet distribution and Dirichlet process as its infinite dimensional generalization are primarily used conjugate prior of categorical and multinomial distributions in Bayesian statistics. Extensions have been proposed to broaden applications for different purposes. In this article, we explore a class of prior distributions closely related to Dirichlet distribution incorporating additional information on the data generating mechanism. Examples are given to show potential use of the models.
On estimation of the noise variance in high dimensional probabilistic principal component analysis

We develop new statistical theory for probabilistic principal component analysis models in high dimensions. The focus is the estimation of the noise variance, which is an important and unresolved issue when the number of variables is large in comparison with the sample size. We first unveil the reasons for an observed downward bias of the maximum likelihood estimator of the noise variance when the data dimension is high. We then propose a bias‐corrected estimator by using random‐matrix theory and establish its asymptotic normality. The superiority of the new and bias‐corrected estimator over existing alternatives is checked by Monte Carlo experiments with various combinations of (p,n) (the dimension and sample size). Next, we construct a new criterion based on the bias‐corrected estimator to determine the number of the principal components, and a consistent estimator is obtained. Its good performance is confirmed by a simulation study and real data analysis. The bias‐corrected estimator is also used to derive new asymptotics for the related goodness‐of‐fit statistic under the high dimensional scheme.