Miracle-dz/UniFAR-FTUs huggingface dataset:
From optimal transport to generative modeling: the VEGAN cookbook

We study unsupervised generative modeling in terms of the optimal transport (OT) problem between true (but unknown) data distribution $P_X$ and the latent variable model distribution $P_G$. We show that the OT problem can be equivalently written in terms of probabilistic encoders, which are constrained to match the posterior and prior distributions over the latent space. When relaxed, this constrained optimization problem leads to a penalized optimal transport (POT) objective, which can be efficiently minimized using stochastic gradient descent by sampling from $P_X$ and $P_G$. We show that POT for the 2-Wasserstein distance coincides with the objective heuristically employed in adversarial auto-encoders (AAE) (Makhzani et al., 2016), which provides the first theoretical justification for AAEs known to the authors. We also compare POT to other popular techniques like variational auto-encoders (VAE) (Kingma and Welling, 2014). Our theoretical results include (a) a better understanding of the commonly observed blurriness of images generated by VAEs, and (b) establishing duality between Wasserstein GAN (Arjovsky and Bottou, 2017) and POT for the 1-Wasserstein distance.
A Sequential Scheme for Large Scale Bayesian Multiple Testing

The problem of large scale multiple testing arises in many contexts, including testing for pairwise interaction among large numbers of neurons. With advances in technologies, it has become common to record from hundreds of neurons simultaneously, and this number is growing quickly, so that the number of pairwise tests can be very large. It is important to control the rate at which false positives occur. In addition, there is sometimes information that affects the probability of a positive result for any given pair. In the case of neurons, they are more likely to have correlated activity when they are close together, and when they respond similarly to various stimuli. Recently a method was developed to control false positives when covariate information, such as distances between pairs of neurons, is available. This method, however, relies on computationally-intensive Markov Chain Monte Carlo (MCMC). Here we develop an alternative, based on Sequential Monte Carlo, which scales well with the size of the dataset. This scheme considers data items sequentially, with relevant probabilities being updated at each step. Simulation experiments demonstrate that the proposed algorithm delivers results as accurately as the previous MCMC method with only a single pass through the data. We illustrate the method by using it to analyze neural recordings from extrastriate cortex in a macaque monkey.
Robust Power System Dynamic State Estimator with Non-Gaussian Measurement Noise: Part I-Theory

This paper develops the theoretical framework and the equations of a new robust Generalized Maximum-likelihood-type Unscented Kalman Filter (GM-UKF) that is able to suppress observation and innovation outliers while filtering out non-Gaussian measurement noise. Because the errors of the real and reactive power measurements calculated using Phasor Measurement Units (PMUs) follow long-tailed probability distributions, the conventional UKF provides strongly biased state estimates since it relies on the weighted least squares estimator. By contrast, the state estimates and residuals of our GM-UKF are proved to be roughly Gaussian, allowing the sigma points to reliably approximate the mean and the covariance matrices of the predicted and corrected state vectors. To develop our GM-UKF, we first derive a batch-mode regression form by processing the predictions and observations simultaneously, where the statistical linearization approach is used. We show that the set of equations so derived are equivalent to those of the unscented transformation. Then, a robust GM-estimator that minimizes a convex Huber cost function while using weights calculated via Projection Statistics (PS's) is proposed. The PS's are applied to a two-dimensional matrix that consists of serially correlated predicted state and innovation vectors to detect observation and innovation outliers. These outliers are suppressed by the GM-estimator using the iteratively reweighted least squares algorithm. Finally, the asymptotic error covariance matrix of the GM-UKF state estimates is derived from the total influence function. In the companion paper, extensive simulation results will be shown to verify the effectiveness and robustness of the proposed method.
Causal inference with observational studies trimmed by the estimated propensity scores

Causal inference with observational studies often relies on the assumptions of unconfoundedness and overlap of covariate distributions in different treatment groups. The overlap assumption is violated when some units have propensity scores close to zero or one, and therefore both theoretical and practical researchers suggest dropping units with extreme estimated propensity scores. We advance the literature in three directions. First, we clarify a conceptual issue of sample trimming by defining causal parameters based on a target population without extreme propensity score. Second, we propose a procedure of smooth weighting, which approximates the existing sample trimming but has better asymptotic properties. The new weighting estimator is asymptotically linear and the bootstrap can be used to construct confidence intervals. Third, we extend the theory to the average treatment effect on the treated, suggesting trimming samples with estimated propensity scores close to one.
Meta-Analysis of Odds Ratios With Incomplete Extracted Data

A typical random effects meta-analysis of odds-ratios assumes binomially distributed numbers of events in a treatment and control group and requires the proportion of deaths to be extracted from published papers. This data is often not available in the publications due to loss to follow-up. When the Kaplan Meier survival plot is available, it is common practice to manually measure the needed information from the plot and infer the probability of survival and then to infer a best-guess of the number of deaths. Uncertainty introduced from theses guesses is not accounted for in current models. This naive approach leads to over-certain results and potentially inaccurate conclusions. We propose the Uncertain Reading-Estimated Events model to construct each study's contribution to the meta-analysis separately using the data available for extraction in the publications. We use real and simulated data to illustrate our methods. Meta-analysis based on the observed number of deaths lead to biased estimates while our proposed model does not. Our results show increases in the standard deviation of the log-odds as compared to a naive meta-analysis that assumes ideal extracted data, equivalent to a reduction of the overall sample size of 43% in our example.
The Geometric Foundations of Hamiltonian Monte Carlo

Although Hamiltonian Monte Carlo has proven an empirical success, the lack of a rigorous theoretical understanding of the algorithm has in many ways impeded both principled developments of the method and use of the algorithm in practice. In this paper we develop the formal foundations of the algorithm through the construction of measures on smooth manifolds, and demonstrate how the theory naturally identifies efficient implementations and motivates promising generalizations.
Affine Reduction of Dimensionality: An Origin-Centric Perspective

We consider statistical methods for reduction of multivariate dimensionality that have invariance and/or commutativity properties under the affine group of transformations (origin translations plus linear combinations of coordinates along initial axes). The methods discussed here differ from traditional principal component and coordinate approaches in that they are origin-centric. Because all Cartesian coordinates of the origin are zero, it is the unique fixed point for subsequent linear transformations of point scatters. Whenever visualizations allow shifting between and/or combining of Cartesian and polar coordinate representations, as in Biplots, the location of this origin is critical. Specifically, origin-centric visualizations enhance the psychology of graphical perception by yielding scatters that can be interpreted as Dyson swarms. The key factor is typically the analyst's choice of origin via an initial "centering" translation; this choice determines whether the recovered scatter will have either no points depicted as being near the origin or else one (or more) points exactly coincident with this origin.
Exchangeability, the 'Histogram Theorem', and population inference

Some practical results are derived for population inference based on a sample, under the two qualitative conditions of 'ignorability' and exchangeability. These are the 'Histogram Theorem', for predicting the outcome of a non-sampled member of the population, and its application to inference about the population, both without and with groups. There are discussions of parametric versus non-parametric models, and different approaches to marginalisation. An Appendix gives a self-contained proof of the Representation Theorem for finite exchangeable sequences.
Default Bayesian Analysis with Global-Local Shrinkage Priors

We provide a framework for assessing the default nature of a prior distribution using the property of regular variation, which we study for global-local shrinkage priors. In particular, we demonstrate the horseshoe priors, originally designed to handle sparsity, also possess regular variation and thus are appropriate for default Bayesian analysis. To illustrate our methodology, we solve a problem of non-informative priors due to Efron (1973), who showed standard flat non-informative priors in high-dimensional normal means model can be highly informative for nonlinear parameters of interest. We consider four such problems and show global-local shrinkage priors such as the horseshoe and horseshoe+ perform as Efron (1973) requires in each case. We find the reason for this lies in the ability of the global-local shrinkage priors to separate a low-dimensional signal embedded in high-dimensional noise, even for nonlinear functions.
co-BPM: a Bayesian Model for Divergence Estimation

Divergence is not only an important mathematical concept in information theory, but also applied to machine learning problems such as low-dimensional embedding, manifold learning, clustering, classification, and anomaly detection. We proposed a bayesian model---co-BPM---to characterize the discrepancy of two sample sets, i.e., to estimate the divergence of their underlying distributions. In order to avoid the pitfalls of plug-in methods that estimate each density independently, our bayesian model attempts to learn a coupled binary partition of the sample space that best captures the landscapes of both distributions, then make direct inference on their divergences. The prior is constructed by leveraging the sequential buildup of the coupled binary partitions and the posterior is sampled via our specialized MCMC. Our model provides a unified way to estimate various types of divergences and enjoys convincing accuracy. We demonstrate its effectiveness through simulations, comparisons with the \emph{state-of-the-art} and a real data example.
Sequential Importance Sampling for Two-dimensional Ising Models

In recent years, sequential importance sampling (SIS) has been well developed for sampling contingency tables with linear constraints. In this paper, we apply SIS procedure to 2-dimensional Ising models, which give observations of 0-1 tables and include both linear and quadratic constraints. We show how to compute bounds for specific cells by solving linear programming (LP) problems over cut polytopes to reduce rejections. The computational results, which includes both simulations and real data analysis, suggest that our method performs very well for sparse tables and when the 1's are spread out: the computational times are short, the acceptance rates are high, and if proper tests are used then in most cases our conclusions are theoretically reasonable.
Consider avoiding the .05 significance level

It is suggested that some shortcomings of Null Hypothesis Significance Testing (NHST), viewed from the perspective of Bayesian statistics, turn benign once the traditional threshold p value of . 05 is substituted by a sufficiently smaller value. To illustrate, the posterior probability of H0 stating P=. 5, given data that just render it rejected by NHST with a p value of . 05 (and a uniform prior), is shown here to be not much smaller than . 50 for most values of N below 100 (and even exceeds .50 for N>=100); in contrast, with a p value of . 001 posterior probability does not exceed . 06 for N<=100 (neither .25 for N<9000). Yet more interesting, posterior probability becomes quite independent of N with a p value of . 0001, hence practically satisfying the alpha postulate - set by Cornfield (1966) as the condition for p value being a measure of evidence in itself. In view of the low prospect that most researchers will soon convert to use Bayesian statistics in any form, we thus suggest that researchers who elect the conservative option of resorting to NHST be encouraged to avoid as much as possible using a p value of . 05 as a threshold for rejecting H0. The analysis presented here may be used to discuss afresh which level of threshold p value seems to be a reasonable, practical substitute.
Analysis of Nonstationary Time Series Using Locally Coupled Gaussian Processes

The analysis of nonstationary time series is of great importance in many scientific fields such as physics and neuroscience. In recent years, Gaussian process regression has attracted substantial attention as a robust and powerful method for analyzing time series. In this paper, we introduce a new framework for analyzing nonstationary time series using locally stationary Gaussian process analysis with parameters that are coupled through a hidden Markov model. The main advantage of this framework is that arbitrary complex nonstationary covariance functions can be obtained by combining simpler stationary building blocks whose hidden parameters can be estimated in closed-form. We demonstrate the flexibility of the method by analyzing two examples of synthetic nonstationary signals: oscillations with time varying frequency and time series with two dynamical states. Finally, we report an example application on real magnetoencephalographic measurements of brain activity.
Gaussian Process for Noisy Inputs with Ordering Constraints

We study the Gaussian Process regression model in the context of training data with noise in both input and output. The presence of two sources of noise makes the task of learning accurate predictive models extremely challenging. However, in some instances additional constraints may be available that can reduce the uncertainty in the resulting predictive models. In particular, we consider the case of monotonically ordered latent input, which occurs in many application domains that deal with temporal data. We present a novel inference and learning approach based on non-parametric Gaussian variational approximation to learn the GP model while taking into account the new constraints. The resulting strategy allows one to gain access to posterior estimates of both the input and the output and results in improved predictive performance. We compare our proposed models to state-of-the-art Noisy Input Gaussian Process (NIGP) and other competing approaches on synthetic and real sea-level rise data. Experimental results suggest that the proposed approach consistently outperforms selected methods while, at the same time, reducing the computational costs of learning and inference.
Marginalized Two Part Models for Generalized Gamma Family of Distributions

Positive continuous outcomes with a point mass at zero are prevalent in biomedical research. To model the point mass at zero and to provide marginalized covariate effect estimates, marginalized two part models (MTP) have been developed for outcomes with lognormal and log skew normal distributions. In this paper, we propose MTP models for outcomes from a generalized gamma (GG) family of distributions. In the proposed MTP-GG model, the conditional mean from a two-part model with a three-parameter GG distribution is parameterized to provide regression coefficients that have marginal interpretation. MTP-gamma and MTP-Weibull are developed as special cases of MTP-GG. We derive marginal covariate effect estimators from each model and through simulations assess their finite sample operating characteristics in terms of bias, standard errors, 95% coverage, and rate of convergence. We illustrate the models using data sets from The Medical Expenditure Survey (MEPS) and from a randomized trial of addictive disorders and we provide SAS code for implementation. The simulation results show that when the response distribution is unknown or mis-specified, which is usually the case in real data sets, the MTP-GG is preferable over other models.
On the Design and use of Ensembles of Multi-model Simulations for Forecasting

Probability forecasting is common in the geosciences, the finance sector, and elsewhere. It is sometimes the case that one has multiple probability-forecasts for the same target. How is the information in these multiple forecast systems best "combined"? Assuming stationary, then in the limit of a very large forecast-outcome archive, each model-based probability density function can be weighted to form a "multi-model forecast" which will, in expectation, provide the most information. In the case that one of the forecast systems yields a probability distribution which reflects the distribution from which the outcome will be drawn, then Bayesian Model Averaging will identify this model as the number of forecast-outcome pairs goes to infinity. In many applications, like those of seasonal forecasting, data are precious: the archive is often limited to fewer than $2^6$ entries. And no perfect model is in hand. In this case, it is shown that forming a single "multi-model probability forecast" can be expected to prove misleading. These issues are investigated using probability forecasts of a simple mathematical system, which allows most limiting behaviours to be quantified.
Linear Regression with Shuffled Labels

Is it possible to perform linear regression on datasets whose labels are shuffled with respect to the inputs? We explore this question by proposing several estimators that recover the weights of a noisy linear model from labels that are shuffled by an unknown permutation. We show that the analog of the classical least-squares estimator produces inconsistent estimates in this setting, and introduce an estimator based on the self-moments of the input features and labels. We study the regimes in which each estimator excels, and generalize the estimators to the setting where partial ordering information is available in the form of experiments replicated independently. The result is a framework that enables robust inference, as we demonstrate by experiments on both synthetic and standard datasets, where we are able to recover approximate weights using only shuffled labels. Our work demonstrates that linear regression in the absence of complete ordering information is possible and can be of practical interest, particularly in experiments that characterize populations of particles, such as flow cytometry.
Bayesian binary quantile regression for the analysis of Bachelor-Master transition

The multi-cycle organization of modern university systems stimulates the interest in studying the progression to higher level degree courses during the academic career. In particular, after the achievement of the first level qualification (Bachelor degree), students have to decide whether to continue their university studies, by enrolling in a second level (Master) programme, or to conclude their training experience. In this work we propose a binary quantile regression approach to analyze the Bachelor-Master transition phenomenon with the adoption of the Bayesian inferential perspective. In addition to the traditional predictors of academic outcomes, such as the personal characteristics and the field of study, different aspects of the student's performance are considered. Moreover, a new contextual variable, indicating the type of university regulations, is taken into account in the model specification. The utility of the Bayesian binary quantile regression to characterize the non-continuation decision after the first cycle studies is illustrated with an application to administrative data of Bachelor graduates at the School of Economics of Sapienza University of Rome and compared with a more conventional logistic regression approach.
Non-parametric Structural Change Detection in Multivariate Systems

Structural change detection problems are often encountered in analytics and econometrics, where the performance of a model can be significantly affected by unforeseen changes in the underlying relationships. Although these problems have a comparatively long history in statistics, the number of studies done in the context of multivariate data under nonparametric settings is still small. In this paper, we propose a consistent method for detecting multiple structural changes in a system of related regressions over a large dimensional variable space. In most applications, practitioners also do not have a priori information on the relevance of different variables, and therefore, both locations of structural changes as well as the corresponding sparse regression coefficients need to be estimated simultaneously. The method combines nonparametric energy distance minimization principle with penalized regression techniques. After showing asymptotic consistency of the model, we compare the proposed approach with competing methods in a simulation study. As an example of a large scale application, we consider structural change point detection in the context of news analytics during the recent financial crisis period.
A Prior Distribution over Directed Acyclic Graphs for Sparse Bayesian Networks

The main contribution of this article is a new prior distribution over directed acyclic graphs, which gives larger weight to sparse graphs. This distribution is intended for structured Bayesian networks, where the structure is given by an ordered block model. That is, the nodes of the graph are objects which fall into categories (or blocks); the blocks have a natural ordering. The presence of a relationship between two objects is denoted by an arrow, from the object of lower category to the object of higher category. The models considered here were introduced in Kemp et al. [4] for relational data and extended to multivariate data in Mansinghka et al. [7]. The prior over graph structures presented here has an explicit formula. The number of nodes in each layer of the graph follow a Hoppe Ewens urn model. We consider the situation where the nodes of the graph represent random variables, whose joint probability distribution factorises along the DAG. We describe Monte Carlo schemes for finding the optimal aposteriori structure given a data matrix and compare the performance with Mansinghka et al. and also with the uniform prior.