Miracle-dz/UniFAR-FTUs huggingface dataset:
A flexible procedure for mixture proportion estimation in positive‐unlabeled learning

Positive‐unlabeled (PU) learning considers two samples, a positive set P with observations from only one class and an unlabeled set U with observations from two classes. The goal is to classify observations in U. Class mixture proportion estimation (MPE) in U is a key step in PU learning. Blanchard et al. showed that MPE in PU learning is a generalization of the problem of estimating the proportion of true null hypotheses in multiple testing problems. Motivated by this idea, we propose reducing the problem to one‐dimension via construction of a probabilistic classifier trained on the P and U data sets followed by application of a one‐dimensional mixture proportion method from the multiple testing literature to the observation class probabilities. The flexibility of this framework lies in the freedom to choose the classifier and the one‐dimensional MPE method. We prove consistency of two mixture proportion estimators using bounds from empirical process theory, develop tuning parameter free implementations, and demonstrate that they have competitive performance on simulated waveform data and a protein signaling problem.
Prediction Accuracy Measures for a Nonlinear Model and for Right-Censored Time-to-Event Data

ABSTRACT This article develops a pair of new prediction summary measures for a nonlinear prediction function with right-censored time-to-event data. The first measure, defined as the proportion of explained variance by a linearly corrected prediction function, quantifies the potential predictive power of the nonlinear prediction function. The second measure, defined as the proportion of explained prediction error by its corrected prediction function, gauges the closeness of the prediction function to its corrected version and serves as a supplementary measure to indicate (by a value less than 1) whether the correction is needed to fulfill its potential predictive power and quantify how much prediction error reduction can be realized with the correction. The two measures together provide a complete summary of the predictive accuracy of the nonlinear prediction function. We motivate these measures by first establishing a variance decomposition and a prediction error decomposition at the population level and then deriving uncensored and censored sample versions of these decompositions. We note that for the least square prediction function under the linear model with no censoring, the first measure reduces to the classical coefficient of determination and the second measure degenerates to 1. We show that the sample measures are consistent estimators of their population counterparts and conduct extensive simulations to investigate their finite sample properties. A real data illustration is provided using the PBC data. Supplementary materials for this article are available online. An R package PAmeasures has been developed and made available via the CRAN R library. Supplementary materials for this article are available online.
Treatment Effects for Which Population?: Sampling Design and External Validity

Causal inference in a program evaluation setting faces the problem of external validity when the treatment effect in the target population is different from the treatment effect identified from the population of which the sample is representative. This paper focuses on a situation where such discrepancy arises by a stratified sampling design based on the individual treatment status and other characteristics. In such settings, the design probability is known from the sampling design but the target population depends on the underlying population share vector which is often unknown, and except for special cases, the treatment effect parameters are not identified. First, we propose a method of constructing confidence sets that are valid for a given range of population shares. Second, when a benchmark population share vector and a corresponding estimator of a treatment effect parameter are given, we propose a method to discover the scope of external validity with familywise error rate control. Third, we propose an optimal sampling design which minimizes the semiparametric efficiency bound given a population share associated with a target population. We provide Monte Carlo simulation results and an empirical application to demonstrate the usefulness of our proposals.
Trimmed Conformal Prediction for High-Dimensional Models

In regression, conformal prediction is a general methodology to construct prediction intervals in a distribution-free manner. Although conformal prediction guarantees strong statistical property for predictive inference, its inherent computational challenge has attracted the attention of researchers in the community. In this paper, we propose a new framework, called Trimmed Conformal Prediction (TCP), based on two stage procedure, a trimming step and a prediction step. The idea is to use a preliminary trimming step to substantially reduce the range of possible values for the prediction interval, and then applying conformal prediction becomes far more efficient. As is the case of conformal prediction, TCP can be applied to any regression method, and further offers both statistical accuracy and computational gains. For a specific example, we also show how TCP can be implemented in the sparse regression setting. The experiments on both synthetic and real data validate the empirical performance of TCP.
Data-driven modelling and validation of aircraft inbound-stream at some major European airports

This paper presents an exhaustive study on the arrivals process at eight important European airports. Using inbound traffic data, we define, compare, and contrast a data-driven Poisson and PSRA point process. Although, there is sufficient evidence that the interarrivals might follow an exponential distribution, this finding does not directly translate to evidence that the arrivals stream is Poisson. The main reason is that finite-capacity constraints impose a correlation structure to the arrivals stream, which a Poisson model cannot capture. We show the weaknesses and somehow the difficulties of using a Poisson process to model with good approximation the arrivals stream. On the other hand, our innovative non-parametric, data-driven PSRA model, predicts quite well and captures important properties of the typical arrivals stream.
An approximate empirical Bayesian method for large-scale linear-Gaussian inverse problems

We study Bayesian inference methods for solving linear inverse problems, focusing on hierarchical formulations where the prior or the likelihood function depend on unspecified hyperparameters. In practice, these hyperparameters are often determined via an empirical Bayesian method that maximizes the marginal likelihood function, i.e. the probability density of the data conditional on the hyperparameters. Evaluating the marginal likelihood, however, is computationally challenging for large-scale problems. In this work, we present a method to approximately evaluate marginal likelihood functions, based on a low-rank approximation of the update from the prior covariance to the posterior covariance. We show that this approximation is optimal in a minimax sense. Moreover, we provide an efficient algorithm to implement the proposed method, based on a combination of the randomized SVD and a spectral approximation method to compute square roots of the prior covariance matrix. Several numerical examples demonstrate good performance of the proposed method.
Beyond Gaussian approximation: Bootstrap for maxima of sums of independent random vectors

The Bonferroni adjustment, or the union bound, is commonly used to study rate optimality properties of statistical methods in high-dimensional problems. However, in practice, the Bonferroni adjustment is overly conservative. The extreme value theory has been proven to provide more accurate multiplicity adjustments in a number of settings, but only on ad hoc basis. Recently, Gaussian approximation has been used to justify bootstrap adjustments in large scale simultaneous inference in some general settings when $n \gg (\log p)^7$, where $p$ is the multiplicity of the inference problem and $n$ is the sample size. The thrust of this theory is the validity of the Gaussian approximation for maxima of sums of independent random vectors in high-dimension. In this paper, we reduce the sample size requirement to $n \gg (\log p)^5$ for the consistency of the empirical bootstrap and the multiplier/wild bootstrap in the Kolmogorov-Smirnov distance, possibly in the regime where the Gaussian approximation is not available. New comparison and anti-concentration theorems, which are of considerable interest in and of themselves, are developed as existing ones interweaved with Gaussian approximation are no longer applicable.
A Neighborhood-Assisted Hotelling's $T^2$ Test for High-Dimensional Means

This paper aims to revive the classical Hotelling's $T^2$ test in the "large $p$, small $n$" paradigm. A Neighborhood-Assisted Hotelling's $T^2$ statistic is proposed to replace the inverse of sample covariance matrix in the classical Hotelling's $T^2$ statistic with a regularized covariance estimator. Utilizing a regression model, we establish its asymptotic normality under mild conditions. We show that the proposed test is able to match the performance of the population Hotelling's $T^2$ test with a known covariance under certain conditions, and thus possesses certain optimality. Moreover, the test has the ability to attain its best power possible by adjusting a neighborhood size to unknown structures of population mean and covariance matrix. An optimal neighborhood size selection procedure is proposed to maximize the power of the Neighborhood-Assisted $T^2$ test via maximizing the signal-to-noise ratio. Simulation experiments and case studies are given to demonstrate the empirical performance of the proposed test.
Noncompliance and Missing Data in Health Economic Evaluation

Health economic evaluations face the issues of noncompliance and missing data. Here, noncompliance is defined as non-adherence to a specific treatment, and occurs within randomized controlled trials (RCTs) when participants depart from their random assignment. Missing data arises if, for example, there is loss-to-follow-up, survey non-response, or the information available from routine data sources is incomplete. Appropriate statistical methods for handling noncompliance and missing data have been developed, but they have rarely been applied in health economics studies. Here, we illustrate the issues and outline some of the appropriate methods with which to handle these with application to health economic evaluation that uses data from an RCT. In an RCT the random assignment can be used as an instrument-for-treatment receipt, to obtain consistent estimates of the complier average causal effect, provided the underlying assumptions are met. Instrumental variable methods can accommodate essential features of the health economic context such as the correlation between individuals’ costs and outcomes in cost-effectiveness studies. Methodological guidance for handling missing data encourages approaches such as multiple imputation or inverse probability weighting, which assume the data are Missing At Random, but also sensitivity analyses that recognize the data may be missing according to the true, unobserved values, that is, Missing Not at Random. Future studies should subject the assumptions behind methods for handling noncompliance and missing data to thorough sensitivity analyses. Modern machine-learning methods can help reduce reliance on correct model specification. Further research is required to develop flexible methods for handling more complex forms of noncompliance and missing data.
Hierarchical Community Detection by Recursive Partitioning

Abstract The problem of community detection in networks is usually formulated as finding a single partition of the network into some “correct” number of communities. We argue that it is more interpretable and in some regimes more accurate to construct a hierarchical tree of communities instead. This can be done with a simple top-down recursive partitioning algorithm, starting with a single community and separating the nodes into two communities by spectral clustering repeatedly, until a stopping rule suggests there are no further communities. This class of algorithms is model-free, computationally efficient, and requires no tuning other than selecting a stopping rule. We show that there are regimes where this approach outperforms K-way spectral clustering, and propose a natural framework for analyzing the algorithm’s theoretical performance, the binary tree stochastic block model. Under this model, we prove that the algorithm correctly recovers the entire community tree under relatively mild assumptions. We apply the algorithm to a gene network based on gene co-occurrence in 1580 research papers on anemia, and identify six clusters of genes in a meaningful hierarchy. We also illustrate the algorithm on a dataset of statistics papers. Supplementary materials for this article are available online.
Efficient Construction of Test Inversion Confidence Intervals Using Quantile Regression

Abstract Modern problems in statistics often include estimators of high computational complexity and with complicated distributions. Statistical inference on such estimators usually relies on asymptotic normality assumptions, however, such assumptions are often not applicable for available sample sizes, due to dependencies in the data. A common alternative is the use of resampling procedures, such as bootstrapping, but these may be computationally intensive to an extent that renders them impractical for modern problems. In this article, we develop a method for fast construction of test-inversion bootstrap confidence intervals. Our approach uses quantile regression to model the quantile of an estimator conditional on the true value of the parameter. We apply this to the Watterson estimator of mutation rate in a standard coalescent model. We demonstrate an improved efficiency of up to 40% from using quantile regression compared to state of the art methods based on stochastic approximation, as measured by the number of simulations required to achieve comparable accuracy. Supplementary materials for this article are available online.
A Statistical Perspective on Inverse and Inverse Regression Problems

Inverse problems, where in broad sense the task is to learn from the noisy response about some unknown function, usually represented as the argument of some known functional form, has received wide attention in the general scientific disciplines. How- ever, in mainstream statistics such inverse problem paradigm does not seem to be as popular. In this article we provide a brief overview of such problems from a statistical, particularly Bayesian, perspective. We also compare and contrast the above class of problems with the perhaps more statistically familiar inverse regression problems, arguing that this class of problems contains the traditional class of inverse problems. In course of our review we point out that the statistical literature is very scarce with respect to both the inverse paradigms, and substantial research work is still necessary to develop the fields.
Optimized Regression Discontinuity Designs

Abstract The increasing popularity of regression discontinuity methods for causal inference in observational studies has led to a proliferation of different estimating strategies, most of which involve first fitting nonparametric regression models on both sides of a treatment assignment boundary and then reporting plug-in estimates for the effect of interest. In applications, however, it is often difficult to tune the nonparametric regressions in a way that is well calibrated for the specific target of inference; for example, the model with the best global in-sample fit may provide poor estimates of the discontinuity parameter, which depends on the regression function at boundary points. We propose an alternative method for estimation and statistical inference in regression discontinuity designs that uses numerical convex optimization to directly obtain the finite-sample-minimax linear estimator for the regression discontinuity parameter, subject to bounds on the second derivative of the conditional response function. Given a bound on the second derivative, our proposed method is fully data driven and provides uniform confidence intervals for the regression discontinuity parameter with both discrete and continuous running variables. The method also naturally extends to the case of multiple running variables.
High dimensional independence testing with maxima of rank correlations

Testing mutual independence for high dimensional observations is a fundamental statistical challenge. Popular tests based on linear and simple rank correlations are known to be incapable of detecting non-linear, non-monotone relationships, calling for methods that can account for such dependences. To address this challenge, we propose a family of tests that are constructed using maxima of pairwise rank correlations that permit consistent assessment of pairwise independence. Built upon a newly developed Cramer-type moderate deviation theorem for degenerate U-statistics, our results cover a variety of rank correlations including Hoeffding's $D$, Blum-Kiefer-Rosenblatt's $R$, and Bergsma-Dassios-Yanagimoto's $\tau^*$. The proposed tests are distribution-free, implementable without the need for permutation, and are shown to be rate-optimal against sparse alternatives under the Gaussian copula model. As a by-product of the study, we reveal an identity between the aforementioned three rank correlation statistics, and hence make a step towards proving a conjecture of Bergsma and Dassios.
Minimax Linear Estimation of the Retargeted Mean

Weighting methods that adjust for observed covariates, such as inverse probability weighting, are widely used for causal inference and estimation with incomplete outcome data. These methods are appealingly interpretable: we observe an outcome of interest on one `observation' sample and wish to know its average over another `target' sample, so we weight the observation sample so it looks like the target and calculate the weighted average outcome. In this paper, we discuss a minimax linear estimator for this retargeted mean estimation problem. It is a weighting estimator that optimizes for similarity, as stochastic processes acting on a class of smooth functions, between the empirical measure of the covariates on the weighted observation sample and that on the target sample. This approach generalizes methods that optimize for similarity of the covariate means or marginal distributions, which correspond to taking this class of functions to be linear or additive respectively. Focusing on the case that this class is the unit ball of a reproducing kernel Hilbert space, we show that the minimax linear estimator is semiparametrically efficient under weak conditions; establish bounds attesting to the estimator's good finite sample properties; and observe promising performance on simulated data throughout a wide range of sample sizes, noise levels, and levels of overlap between the covariate distributions for the observation and target samples.
Robust functional ANOVA model with t-process

Robust estimation approaches are of fundamental importance for statistical modelling. To reduce susceptibility to outliers, we propose a robust estimation procedure with t-process under functional ANOVA model. Besides common mean structure of the studied subjects, their personal characters are also informative, especially for prediction. We develop a prediction method to predict the individual effect. Statistical properties, such as robustness and information consistency, are studied. Numerical studies including simulation and real data examples show that the proposed method performs well.
A distributed regression analysis application based on SAS software. Part I: Linear and logistic regression.

Previous work has demonstrated the feasibility and value of conducting distributed regression analysis (DRA), a privacy-protecting analytic method that performs multivariable-adjusted regression analysis with only summary-level information from participating sites. To our knowledge, there are no DRA applications in SAS, the statistical software used by several large national distributed data networks (DDNs), including the Sentinel System and PCORnet. SAS/IML is available to perform the required matrix computations for DRA in the SAS system. However, not all data partners in these large DDNs have access to SAS/IML, which is licensed separately. In this first article of a two-paper series, we describe a DRA application developed for use in Base SAS and SAS/STAT modules for linear and logistic DRA within horizontally partitioned DDNs and its successful tests.
Parallel Concatenation of Bayesian Filters: Turbo Filtering

In this manuscript a method for developing novel filtering algorithms through the parallel concatenation of two Bayesian filters is illustrated. Our description of this method, called turbo filtering, is based on a new graphical model; this allows us to efficiently describe both the processing accomplished inside each of the constituent filter and the interactions between them. This model is exploited to develop two new filtering algorithms for conditionally linear Gaussian systems. Numerical results for a specific dynamic system evidence that such filters can achieve a better complexity-accuracy tradeoff than marginalized particle filtering.
Semiparametric Regression Using Variational Approximations

ABSTRACT Semiparametric regression offers a flexible framework for modeling nonlinear relationships between a response and covariates. A prime example are generalized additive models (GAMs) where splines (say) are used to approximate nonlinear functional components in conjunction with a quadratic penalty to control for overfitting. Estimation and inference are then generally performed based on the penalized likelihood, or under a mixed model framework. The penalized likelihood framework is fast but potentially unstable, and choosing the smoothing parameters needs to be done externally using cross-validation, for instance. The mixed model framework tends to be more stable and offers a natural way for choosing the smoothing parameters, but for nonnormal responses involves an intractable integral. In this article, we introduce a new framework for semiparametric regression based on variational approximations (VA). The approach possesses the stability and natural inference tools of the mixed model framework, while achieving computation times comparable to using penalized likelihood. Focusing on GAMs, we derive fully tractable variational likelihoods for some common response types. We present several features of the VA framework for inference, including a variational information matrix for inference on parametric components, and a closed-form update for estimating the smoothing parameter. We demonstrate the consistency of the VA estimates, and an asymptotic normality result for the parametric component of the model. Simulation studies show the VA framework performs similarly to and sometimes better than currently available software for fitting GAMs. Supplementary materials for this article are available online.