Miracle-dz/UniFAR-FTUs huggingface dataset:
Combining and Extremizing Real-Valued Forecasts

The weighted average is by far the most popular approach to combining multiple forecasts of some future outcome. This paper shows that both for probability or real-valued forecasts, a non-trivial weighted average of different forecasts is always sub-optimal. More specifically, it is not consistent with any set of information about the future outcome even if the individual forecasts are. Furthermore, weighted averaging does not behave as if it collects information from the forecasters and hence needs to be extremized, that is, systematically transformed away from the marginal mean. This paper proposes a linear extremization technique for improving the weighted average of real-valued forecasts. The resulting more extreme version of the weighted average exhibits many properties of optimal aggregation. Both this and the sub-optimality of the weighted average are illustrated with simple examples involving synthetic and real-world data.
On Principal Components Regression, Random Projections, and Column Subsampling

Principal Components Regression (PCR) is a traditional tool for dimension reduction in linear regression that has been both criticized and defended. One concern about PCR is that obtaining the leading principal components tends to be computationally demanding for large data sets. While random projections do not possess the optimality properties of the leading principal subspace, they are computationally appealing and hence have become increasingly popular in recent years. In this paper, we present an analysis showing that for random projections satisfying a Johnson-Lindenstrauss embedding property, the prediction error in subsequent regression is close to that of PCR, at the expense of requiring a slightly large number of random projections than principal components. Column sub-sampling constitutes an even cheaper way of randomized dimension reduction outside the class of Johnson-Lindenstrauss transforms. We provide numerical results based on synthetic and real data as well as basic theory revealing differences and commonalities in terms of statistical performance.
Tools for predicting rainfall from lightning records: events identification and rain prediction using a Bayesian hierarchical model

We propose a new statistical protocol for the estimation of precipitation using lightning data. We first identify rainy events using a scan statistics, then we estimate Rainfall Lighting Ratio (RLR) to convert lightning number into rain volume given the storm intensity. Then we build a hierarchical Bayesian model aiming at the prediction of 15- and 30-minutes cumulated precipitation at unobserved locations and time using information on lightning in the same area. More specifically, we build a Bayesian hierarchical model in which precipitation is modeled as function of lightning count and space time variation is handled using specific structured (random) effects. The mean component of the model relates precipitation and lightning assuming that the number of lightning recorded on a regular grid depends on the number of lightning occurring in neighboring cells. We analyze several model formulations where storms propagation speed, spatial dependence and time variation incorporates different descriptions of the phenomena at hand. The space-time variation is assumed separable. The study area is located in Central Italy, where two storms, that differ for duration and intensity, are presented.
Monte Carlo on Manifolds: Sampling Densities and Integrating Functions

We describe and analyze some Monte Carlo methods for manifolds in euclidean space defined by equality and inequality constraints. First, we give an MCMC sampler for probability distributions defined by unnormalized densities on such manifolds. The sampler uses a specific orthogonal projection to the surface that requires only information about the tangent space to the manifold, obtainable from first derivatives of the constraint functions, hence avoiding the need for curvature information or second derivatives. Second, we use the sampler to develop a multistage algorithm to compute integrals over such manifolds. We provide single‐run error estimates that avoid the need for multiple independent runs. Computational experiments on various test problems show that the algorithms and error estimates work in practice. The method is applied to compute the entropies of different sticky hard sphere systems. These predict the temperature or interaction energy at which loops of hard sticky spheres become preferable to chains. © 2018 Wiley Periodicals, Inc.
Approximate $\ell_{0}$-penalized estimation of piecewise-constant signals on graphs

We study recovery of piecewise-constant signals on graphs by the estimator minimizing an $l_0$-edge-penalized objective. Although exact minimization of this objective may be computationally intractable, we show that the same statistical risk guarantees are achieved by the $\alpha$-expansion algorithm which computes an approximate minimizer in polynomial time. We establish that for graphs with small average vertex degree, these guarantees are minimax rate-optimal over classes of edge-sparse signals. For spatially inhomogeneous graphs, we propose minimization of an edge-weighted objective where each edge is weighted by its effective resistance or another measure of its contribution to the graph's connectivity. We establish minimax optimality of the resulting estimators over corresponding edge-weighted sparsity classes. We show theoretically that these risk guarantees are not always achieved by the estimator minimizing the $l_1$/total-variation relaxation, and empirically that the $l_0$-based estimates are more accurate in high signal-to-noise settings.
On asymptotic inference in stochastic differential equations with time‐varying covariates

In this article, we introduce a system of stochastic differential equations (SDEs) consisting of time‐dependent covariates and consider both fixed and random effects. We also allow the functional part associated with the drift function to depend upon unknown parameters. For this general SDE system we establish consistency and asymptotic normality of the maximum likelihood estimator. We consider a Bayesian approach for learning about the population parameters, and prove consistency and asymptotic normality of the corresponding posterior distribution. We supplement our theoretical investigation with simulated and real data analyses, obtaining encouraging results in both cases. The Canadian Journal of Statistics 46: 635–655; 2018 © 2018 Statistical Society of Canada
A geometric approach to non-linear correlations with intrinsic scatter

We propose a new mathematical model for $n-k$-dimensional non-linear correlations with intrinsic scatter in $n$-dimensional data. The model is based on Riemannian geometry, and is naturally symmetric with respect to the measured variables and invariant under coordinate transformations. We combine the model with a Bayesian approach for estimating the parameters of the correlation relation and the intrinsic scatter. A side benefit of the approach is that censored and truncated datasets and independent, arbitrary measurement errors can be incorporated. We also derive analytic likelihoods for the typical astrophysical use case of linear relations in $n$-dimensional Euclidean space. We pay particular attention to the case of linear regression in two dimensions, and compare our results to existing methods. Finally, we apply our methodology to the well-known $M_\text{BH}$-$\sigma$ correlation between the mass of a supermassive black hole in the centre of a galactic bulge and the corresponding bulge velocity dispersion. The main result of our analysis is that the most likely slope of this correlation is $\sim 6$ for the datasets used, rather than the values in the range $\sim 4$-$5$ typically quoted in the literature for these data.
Tuning Parameter Calibration in High-dimensional Logistic Regression With Theoretical Guarantees

Feature selection is a standard approach to understanding and modeling high-dimensional classification data, but the corresponding statistical methods hinge on tuning parameters that are difficult to calibrate. In particular, existing calibration schemes in the logistic regression framework lack any finite sample guarantees. In this paper, we introduce a novel calibration scheme for penalized logistic regression. It is based on simple tests along the tuning parameter path and satisfies optimal finite sample bounds. It is also amenable to easy and efficient implementations, and it rivals or outmatches existing methods in simulations and real data applications.
Goodness of Fit of Logistic Regression Models for Random Graphs

ABSTRACT Logistic regression is a natural and simple tool to understand how covariates contribute to explain the topology of a binary network. Once the model is fitted, the practitioner is interested in the goodness of fit of the regression to check if the covariates are sufficient to explain the whole topology of the network and, if they are not, to analyze the residual structure. To address this problem, we introduce a generic model that combines logistic regression with a network-oriented residual term. This residual term takes the form of the graphon function of a W-graph. Using a variational Bayes framework, we infer the residual graphon by averaging over a series of blockwise constant functions. This approach allows us to define a generic goodness-of-fit criterion, which corresponds to the posterior probability for the residual graphon to be constant. Experiments on toy data are carried out to assess the accuracy of the procedure. Several networks from social sciences and ecology are studied to illustrate the proposed methodology. Supplementary material for this article is available online.
Dimension reduction for data of unknown cluster structure

For numerous reasons there raises a need for dimension reduction that preserves certain characteristics of data. In this work we focus on data coming from a mixture of Gaussian distributions and we propose a method that preserves distinctness of clustering structure, although the structure is assumed to be yet unknown. The rationale behind the method is the following: (i) had one known the clusters (classes) within the data, one could facilitate further analysis and reduce space dimension by projecting the data to the Fisher's linear subspace, which -- by definition -- preserves the structure of the given classes best (ii) under some reasonable assumptions, this can be done, albeit approximately, without the prior knowledge of the clusters (classes). In the paper, we show how this approach works. We present a method of preliminary data transformation that brings the directions of largest overall variability close to the directions of the best between-class separation. Hence, for the transformed data, simple PCA provides an approximation to the Fisher's subspace. We show that the transformation preserves distinctness of unknown structure in the data to a great extent.
A time-varying shared frailty model with application to infectious diseases

We propose a new parametric time-varying shared frailty model to represent changes over time in population heterogeneity, for use with bivariate current status data. The model uses a power transformation of a time-invariant frailty $U$, and is particularly convenient when $U$ is a member of the generalized gamma family. This model avoids some shortcomings of a previously suggested time-varying frailty model, notably time-dependent support. We describe some key properties of the model, including its relative frailty variance function in different settings and how the model can be fitted to data. We describe several applications to shared frailty modeling of bivariate current status data on infectious diseases, in which the frailty represents age-dependent heterogeneity in contact rates or susceptibility to infection.
Conditions for Posterior Contraction in the Sparse Normal Means Problem

The first Bayesian results for the sparse normal means problem were proven for spike-and-slab priors. However, these priors are less convenient from a computational point of view. In the meanwhile, a large number of continuous shrinkage priors has been proposed. Many of these shrinkage priors can be written as a scale mixture of normals, which makes them particularly easy to implement. We propose general conditions on the prior on the local variance in scale mixtures of normals, such that posterior contraction at the minimax rate is assured. The conditions require tails at least as heavy as Laplace, but not too heavy, and a large amount of mass around zero relative to the tails, more so as the sparsity increases. These conditions give some general guidelines for choosing a shrinkage prior for estimation under a nearly black sparsity assumption. We verify these conditions for the class of priors considered in [12], which includes the horseshoe and the normal-exponential gamma priors, and for the horseshoe+, the inverse-Gaussian prior, the normal-gamma prior, and the spike-and-slab Lasso, and thus extend the number of shrinkage priors which are known to lead to posterior contraction at the minimax estimation rate.
Is profile likelihood a true likelihood? An argument in favor

Profile likelihood is the key tool for dealing with nuisance parameters in likelihood theory. It is often asserted, however, that profile likelihood is not a true likelihood. One implication is that likelihood theory lacks the generality of e.g. Bayesian inference, wherein marginalization is the universal tool for dealing with nuisance parameters. Here we argue that profile likelihood has as much claim to being a true likelihood as a marginal probability has to being a true probability distribution. The crucial point we argue is that a likelihood function is naturally interpreted as a maxitive possibility measure: given this, the associated theory of integration with respect to maxitive measures delivers profile likelihood as the direct analogue of marginal probability in additive measure theory. Thus, given a background likelihood function, we argue that profiling over the likelihood function is as natural (or as unnatural, as the case may be) as marginalizing over a background probability measure.
Multiple changepoint detection with partial information on changepoint times

This paper proposes a new minimum description length procedure to detect multiple changepoints in time series data when some times are a priori thought more likely to be changepoints. This scenario arises with temperature time series homogenization pursuits, our focus here. Our Bayesian procedure constructs a natural prior distribution for the situation, and is shown to estimate the changepoint locations consistently, with an optimal convergence rate. Our methods substantially improve changepoint detection power when prior information is available. The methods are also tailored to bivariate data, allowing changes to occur in one or both component series.
Generative Adversarial Nets from a Density Ratio Estimation Perspective

Generative adversarial networks (GANs) are successful deep generative models. GANs are based on a two-player minimax game. However, the objective function derived in the original motivation is changed to obtain stronger gradients when learning the generator. We propose a novel algorithm that repeats the density ratio estimation and f-divergence minimization. Our algorithm offers a new perspective toward the understanding of GANs and is able to make use of multiple viewpoints obtained in the research of density ratio estimation, e.g. what divergence is stable and relative density ratio is useful.
Generalized Optimal Matching Methods for Causal Inference

We develop an encompassing framework for matching, covariate balancing, and doubly-robust methods for causal inference from observational data called generalized optimal matching (GOM). The framework is given by generalizing a new functional-analytical formulation of optimal matching, giving rise to the class of GOM methods, for which we provide a single unified theory to analyze tractability, consistency, and efficiency. Many commonly used existing methods are included in GOM and, using their GOM interpretation, can be extended to optimally and automatically trade off balance for variance and outperform their standard counterparts. As a subclass, GOM gives rise to kernel optimal matching (KOM), which, as supported by new theoretical and empirical results, is notable for combining many of the positive properties of other methods in one. KOM, which is solved as a linearly-constrained convex-quadratic optimization problem, inherits both the interpretability and model-free consistency of matching but can also achieve the $\sqrt{n}$-consistency of well-specified regression and the efficiency and robustness of doubly robust methods. In settings of limited overlap, KOM enables a very transparent method for interval estimation for partial identification and robust coverage. We demonstrate these benefits in examples with both synthetic and real data
Multiple Outlier Detection in Samples with Exponential & Pareto Tails: Redeeming the Inward Approach & Detecting Dragon Kings

We consider the detection of multiple outliers in Exponential and Pareto samples -- as well as general samples that have approximately Exponential or Pareto tails, thanks to Extreme Value Theory. It is shown that a simple "robust'' modification of common test statistics makes inward sequential testing -- formerly relegated within the literature since the introduction of outward testing -- as powerful as, and potentially less error prone than, outward tests. Moreover, inward testing does not require the complicated type 1 error control of outward tests. A variety of test statistics, employed in both block and sequential tests, are compared for their power and errors, in cases including no outliers, dispersed outliers (the classical slippage alternative), and clustered outliers (a case seldom considered). We advocate a density mixture approach for detecting clustered outliers. Tests are found to be highly sensitive to the correct specification of the main distribution (Exponential/Pareto), exposing high potential for errors in inference. Further, in five case studies -- financial crashes, nuclear power generation accidents, stock market returns, epidemic fatalities, and cities within countries -- significant outliers are detected and related to the concept of ‘Dragon King’ events, defined as meaningful outliers of unique origin.
Vanilla Lasso for sparse classification under single index models

This paper study sparse classification problems. We show that under single-index models, vanilla Lasso could give good estimate of unknown parameters. With this result, we see that even if the model is not linear, and even if the response is not continuous, we could still use vanilla Lasso to train classifiers. Simulations confirm that vanilla Lasso could be used to get a good estimation when data are generated from a logistic regression model.
Mean-Field Variational Inference for Gradient Matching with Gaussian Processes

Gradient matching with Gaussian processes is a promising tool for learning parameters of ordinary differential equations (ODE's). The essence of gradient matching is to model the prior over state variables as a Gaussian process which implies that the joint distribution given the ODE's and GP kernels is also Gaussian distributed. The state-derivatives are integrated out analytically since they are modelled as latent variables. However, the state variables themselves are also latent variables because they are contaminated by noise. Previous work sampled the state variables since integrating them out is \textit{not} analytically tractable. In this paper we use mean-field approximation to establish tight variational lower bounds that decouple state variables and are therefore, in contrast to the integral over state variables, analytically tractable and even concave for a restricted family of ODE's, including nonlinear and periodic ODE's. Such variational lower bounds facilitate "hill climbing" to determine the maximum a posteriori estimate of ODE parameters. An additional advantage of our approach over sampling methods is the determination of a proxy to the intractable posterior distribution over state variables given observations and the ODE's.
A simple and robust confidence interval for causal effects with possibly invalid instruments

Instrumental variables have been widely used to estimate the causal effect of a treatment on an outcome. Existing confidence intervals for causal effects based on instrumental variables assume that all of the putative instrumental variables are valid; a valid instrumental variable is a variable that affects the outcome only by affecting the treatment and is not related to unmeasured confounders. However, in practice, some of the putative instrumental variables are likely to be invalid. This paper presents a simple and general approach to construct a confidence interval that is robust to possibly invalid instruments. The robust confidence interval has theoretical guarantees on having the correct coverage and can also be used to assess the sensitivity of inference when instrumental variables assumptions are violated. The paper also shows that the robust confidence interval outperforms traditional confidence intervals popular in instrumental variables literature when invalid instruments are present. The new approach is applied to a developmental economics study of the causal effect of income on food expenditures.
Bayesian Functional Linear Regression with Sparse Step Functions

The functional linear regression model is a common tool to determine the relationship between a scalar outcome and a functional predictor seen as a function of time. This paper focuses on the Bayesian estimation of the support of the coefficient function. To this aim we propose a parsimonious and adaptive decomposition of the coefficient function as a step function, and a model including a prior distribution that we name Bayesian functional Linear regression with Sparse Step functions (Bliss). The aim of the method is to recover areas of time which influences the most the outcome. A Bayes estimator of the support is built with a specific loss function, as well as two Bayes estimators of the coefficient function, a first one which is smooth and a second one which is a step function. The performance of the proposed methodology is analysed on various synthetic datasets and is illustrated on a black Perigord truffle dataset to study the influence of rainfall on the production.
Distribution-free Multiple Testing

We study a stylized multiple testing problem where the test statistics are independent and assumed to have the same distribution under their respective null hypotheses. We first show that, in the normal means model where the test statistics are normal Z-scores, the well-known method of (Benjamini and Hochberg, 1995) is optimal in some asymptotic sense. We then show that this is also the case of a recent distribution-free method proposed by Foygel-Barber and Cand\`es (2015). The method is distribution-free in the sense that it is agnostic to the null distribution - it only requires that the null distribution be symmetric. We extend these optimality results to other location models with a base distribution having fast-decaying tails.