Miracle-dz/UniFAR-FTUs huggingface dataset:
Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and Its Asymptotic Optimality

Abstract Tuning parameter selection is of critical importance for kernel ridge regression. To date, a data-driven tuning method for divide-and-conquer kernel ridge regression (d-KRR) has been lacking in the literature, which limits the applicability of d-KRR for large datasets. In this article, by modifying the generalized cross-validation (GCV) score, we propose a distributed generalized cross-validation (dGCV) as a data-driven tool for selecting the tuning parameters in d-KRR. Not only the proposed dGCV is computationally scalable for massive datasets, it is also shown, under mild conditions, to be asymptotically optimal in the sense that minimizing the dGCV score is equivalent to minimizing the true global conditional empirical loss of the averaged function estimator, extending the existing optimality results of GCV to the divide-and-conquer framework. Supplemental materials for this article are available online.
Multivariate Functional Principal Component Analysis for Data Observed on Different (Dimensional) Domains

ABSTRACT Existing approaches for multivariate functional principal component analysis are restricted to data on the same one-dimensional interval. The presented approach focuses on multivariate functional data on different domains that may differ in dimension, such as functions and images. The theoretical basis for multivariate functional principal component analysis is given in terms of a Karhunen–Loève Theorem. For the practically relevant case of a finite Karhunen–Loève representation, a relationship between univariate and multivariate functional principal component analysis is established. This offers an estimation strategy to calculate multivariate functional principal components and scores based on their univariate counterparts. For the resulting estimators, asymptotic results are derived. The approach can be extended to finite univariate expansions in general, not necessarily orthonormal bases. It is also applicable for sparse functional data or data with measurement error. A flexible R implementation is available on CRAN. The new method is shown to be competitive to existing approaches for data observed on a common one-dimensional domain. The motivating application is a neuroimaging study, where the goal is to explore how longitudinal trajectories of a neuropsychological test score covary with FDG-PET brain scans at baseline. Supplementary material, including detailed proofs, additional simulation results, and software is available online.
Improving the local scoring algorithm using gradient sampling

We adapt the gradient sampling algorithm to the local scoring algorithm to solve complex estimation problems based on an optimization of an objective function. This overcomes non-differentiability and non-smoothness of the objective function. The new algorithm estimates the Clarke generalized subgradient used in the local scoring, thus reducing numerical instabilities. The method is applied to quantile regression and to the peaks-over-threshold method, as two examples. Real applications are provided for a retail store and temperature data analysis.
Predictive mean matching imputation in survey sampling

Predictive mean matching imputation is popular for handling item nonresponse in survey sampling. In this article, we study the asymptotic properties of the predictive mean matching estimator of the population mean. For variance estimation, the conventional bootstrap inference for matching estimators with fixed matches has been shown to be invalid due to the nonsmoothness nature of the matching estimator. We propose asymptotically valid replication variance estimation. The key strategy is to construct replicates of the estimator directly based on linear terms, instead of individual records of variables. Extension to nearest neighbor imputation is also discussed. A simulation study confirms that the new procedure provides valid variance estimation.
Photo-z Estimation: An Example of Nonparametric Conditional Density Estimation under Selection Bias

Redshift is a key quantity for inferring cosmological model parameters. In photometric redshift estimation, cosmologists use the coarse data collected from the vast majority of galaxies to predict the redshift of individual galaxies. To properly quantify the uncertainty in the predictions, however, one needs to go beyond standard regression and instead estimate the full conditional density f(z|x) of a galaxy's redshift z given its photometric covariates x. The problem is further complicated by selection bias: usually only the rarest and brightest galaxies have known redshifts, and these galaxies have characteristics and measured covariates that do not necessarily match those of more numerous and dimmer galaxies of unknown redshift. Unfortunately, there is not much research on how to best estimate complex multivariate densities in such settings. Here we describe a general framework for properly constructing and assessing nonparametric conditional density estimators under selection bias, and for combining two or more estimators for optimal performance. We propose new improved photo-z estimators and illus- trate our methods on data from the Sloan Data Sky Survey and an application to galaxy-galaxy lensing. Although our main application is photo-z estimation, our methods are relevant to any high-dimensional regression setting with complicated asymmetric and multimodal distributions in the response variable.
A general asymptotic framework for distribution‐free graph‐based two‐sample tests

Testing equality of two multivariate distributions is a classical problem for which many non‐parametric tests have been proposed over the years. Most of the popular two‐sample tests, which are asymptotically distribution free, are based either on geometric graphs constructed by using interpoint distances between the observations (multivariate generalizations of the Wald–Wolfowitz runs test) or on multivariate data depth (generalizations of the Mann–Whitney rank test). The paper introduces a general notion of distribution‐free graph‐based two‐sample tests and provides a unified framework for analysing and comparing their asymptotic properties. The asymptotic (Pitman) efficiency of a general graph‐based test is derived, which includes tests based on geometric graphs, such as the Friedman–Rafsky test, the test based on the K‐nearest‐neighbour graph, the cross‐match test and the generalized edge count test, as well as tests based on multivariate depth functions (the Liu–Singh rank sum statistic). The results show how the combinatorial properties of the underlying graph affect the performance of the associated two‐sample test and can be used to validate and decide which tests to use in practice. Applications of the results are illustrated both on synthetic and on real data sets.
Single-index modulated multiple testing

In the context of large-scale multiple testing, hypotheses are often accompanied with certain prior information. In this paper, we present a single-index modulated (SIM) multiple testing procedure, which maintains control of the false discovery rate while incorporating prior information, by assuming the availability of a bivariate $p$-value, $(p_1,p_2)$, for each hypothesis, where $p_1$ is a preliminary $p$-value from prior information and $p_2$ is the primary $p$-value for the ultimate analysis. To find the optimal rejection region for the bivariate $p$-value, we propose a criteria based on the ratio of probability density functions of $(p_1,p_2)$ under the true null and nonnull. This criteria in the bivariate normal setting further motivates us to project the bivariate $p$-value to a single-index, $p(\theta)$, for a wide range of directions $\theta$. The true null distribution of $p(\theta)$ is estimated via parametric and nonparametric approaches, leading to two procedures for estimating and controlling the false discovery rate. To derive the optimal projection direction $\theta$, we propose a new approach based on power comparison, which is further shown to be consistent under some mild conditions. Simulation evaluations indicate that the SIM multiple testing procedure improves the detection power significantly while controlling the false discovery rate. Analysis of a real dataset will be illustrated.
Monte Carlo with determinantal point processes

We show that repulsive random variables can yield Monte Carlo methods with faster convergence rates than the typical $N^{-1/2}$, where $N$ is the number of integrand evaluations. More precisely, we propose stochastic numerical quadratures involving determinantal point processes associated with multivariate orthogonal polynomials, and we obtain root mean square errors that decrease as $N^{-(1+1/d)/2}$, where $d$ is the dimension of the ambient space. First, we prove a central limit theorem (CLT) for the linear statistics of a class of determinantal point processes, when the reference measure is a product measure supported on a hypercube, which satisfies the Nevai-class regularity condition, a result which may be of independent interest. Next, we introduce a Monte Carlo method based on these determinantal point processes, and prove a CLT with explicit limiting variance for the quadrature error, when the reference measure satisfies a stronger regularity condition. As a corollary, by taking a specific reference measure and using a construction similar to importance sampling, we obtain a general Monte Carlo method, which applies to any measure with continuously derivable density. Loosely speaking, our method can be interpreted as a stochastic counterpart to Gaussian quadrature, which, at the price of some convergence rate, is easily generalizable to any dimension and has a more explicit error term.
An Adapted Loss Function for Censored Quantile Regression

ABSTRACT In this article, we study a novel approach for the estimation of quantiles when facing potential right censoring of the responses. Contrary to the existing literature on the subject, the adopted strategy of this article is to tackle censoring at the very level of the loss function usually employed for the computation of quantiles, the so-called “check” function. For interpretation purposes, a simple comparison with the latter reveals how censoring is accounted for in the newly proposed loss function. Subsequently, when considering the inclusion of covariates for conditional quantile estimation, by defining a new general loss function the proposed methodology opens the gate to numerous parametric, semiparametric, and nonparametric modeling techniques. To illustrate this statement, we consider the well-studied linear regression under the usual assumption of conditional independence between the true response and the censoring variable. For practical minimization of the studied loss function, we also provide a simple algorithmic procedure shown to yield satisfactory results for the proposed estimator with respect to the existing literature in an extensive simulation study. From a more theoretical prospect, consistency and asymptotic normality of the estimator for linear regression are obtained using several recent results on nonsmooth semiparametric estimation equations with an infinite-dimensional nuisance parameter, while numerical examples illustrate the adequateness of a simple bootstrap procedure for inferential purposes. Lastly, an application to a real dataset is used to further illustrate the validity and finite sample performance of the proposed estimator. Supplementary materials for this article are available online.
Locally Adaptive Dynamic Networks

Our focus is on realistically modeling and forecasting dynamic networks of face-to-face contacts among individuals. Important aspects of such data that lead to problems with current methods include the tendency of the contacts to move between periods of slow and rapid changes, and the dynamic heterogeneity in the actors' connectivity behaviors. Motivated by this application, we develop a novel method for Locally Adaptive DYnamic (LADY) network inference. The proposed model relies on a dynamic latent space representation in which each actor's position evolves in time via stochastic differential equations. Using a state space representation for these stochastic processes and Polya-gamma data augmentation, we develop an efficient MCMC algorithm for posterior inference along with tractable procedures for online updating and forecasting of future networks. We evaluate performance in simulation studies, and consider an application to face-to-face contacts among individuals in a primary school.
Reduction of wind power variability through geographic diversity

The variability of wind-generated electricity can be reduced by aggregating the outputs of wind generation plants spread over a large geographic area. In this chapter we utilize Monte Carlo simulations to investigate upper bounds on the degree of achievable smoothing and clarify how the degree of smoothing depends on the number of plants and on the size of the geographic area over which they are spread. We model two distinct benefits of geographic diversity that have different behaviors: (1) increased tendency of generation level to lie near its mean and (2) decreased tendency for it to lie near its extremes. Gaussian or normal probability distributions give accurate estimates of the first but underestimate the second. The second benefit, which has particular importance for electric grid reliability, has not been widely treated before. The effect of geographic diversity on wind generation variability has been investigated by many, starting with Thomas (1945). Significant early work was carried out by Molly (1977), Justus and Mikhail (1978), Kahn (1979), Farmer, Newman, and Ashmole (1980), and Carlin and Haslett (1982). Some researchers have focused on particular geographic regions, including the U.S. Midwest (Archer and Jacobson, 2003, 2007; Fisher et al., 2013) and the Nordic countries (Holttinen, 2005). Others have investigated the effect on the frequency spectrum of the generated power (McNerney and Richardson, 1992; Beyer, Luther, and Steinberger-Willms, 1993; Nanahara et al., 2004; Katzenstein, Fertig, and Apt, 2010; Tarroja et al., 2011). Hasche (2010) has modeled how the smoothing benefit saturates as the number of generator sites within a region increases. Some recent investigations have focused on the effects of spreading arrays of wind generators over especially large distances. Kempton et al. (2010) and Dvorak et al. (2012) considered an array of wind farms
A machine learning nowcasting method based on real‐time reanalysis data

Despite marked progress over the past several decades, convective storm nowcasting remains a challenge because most nowcasting systems are based on linear extrapolation of radar reflectivity without much consideration for other meteorological fields. The variational Doppler radar analysis system (VDRAS) is an advanced convective‐scale analysis system capable of providing analysis of 3‐D wind, temperature, and humidity by assimilating Doppler radar observations. Although potentially useful, it is still an open question as to how to use these fields to improve nowcasting. In this study, we present results from our first attempt at developing a support vector machine (SVM) box‐based nowcasting (SBOW) method under the machine learning framework using VDRAS analysis data. The key design points of SBOW are as follows: (1) The study domain is divided into many position‐fixed small boxes, and the nowcasting problem is transformed into one question, i.e., will a radar echo > 35 dBZ appear in a box in 30 min? (2) Box‐based temporal and spatial features, which include time trends and surrounding environmental information, are constructed. (3) And the box‐based constructed features are used to first train the SVM classifier, and then the trained classifier is used to make predictions. Compared with complicated and expensive expert systems, the above design of SBOW allows the system to be small, compact, straightforward, and easy to maintain and expand at low cost. The experimental results show that although no complicated tracking algorithm is used, SBOW can predict the storm movement trend and storm growth with reasonable skill.
Convex and non-convex regularization methods for spatial point processes intensity estimation

This paper deals with feature selection procedures for spatial point processes intensity estimation. We consider regularized versions of estimating equations based on Campbell theorem derived from two classical functions: Poisson likelihood and logistic regression likelihood. We provide general conditions on the spatial point processes and on penalty functions which ensure consistency, sparsity and asymptotic normality. We discuss the numerical implementation and assess finite sample properties in a simulation study. Finally, an application to tropical forestry datasets illustrates the use of the proposed methods.
Estimating Animal Abundance with N-Mixture Models Using the R-INLA Package for R

Successful management of wildlife populations requires accurate estimates of abundance. Abundance estimates can be confounded by imperfect detection during wildlife surveys. N-mixture models enable quantification of detection probability and, under appropriate conditions, produce abundance estimates that are less biased. Here, we demonstrate use of the R-INLA package for R to analyze N-mixture models and compare performance of R-INLA to two other common approaches: JAGS (via the runjags package for R), which uses Markov chain Monte Carlo and allows Bayesian inference, and the unmarked package for R, which uses maximum likelihood and allows frequentist inference. We show that R-INLA is an attractive option for analyzing N-mixture models when (i) fast computing times are necessary (R-INLA is 10 times faster than unmarked and 500 times faster than JAGS), (ii) familiar model syntax and data format (relative to other R packages) is desired, (iii) survey-level covariates of detection are not essential, and (iv) Bayesian inference is preferred.
Bayesian model reduction

This paper reviews recent developments in statistical structure learning; namely, Bayesian model reduction. Bayesian model reduction is a method for rapidly computing the evidence and parameters of probabilistic models that differ only in their priors. In the setting of variational Bayes this has an analytical solution, which finesses the problem of scoring large model spaces in model comparison or structure learning. In this technical note, we review Bayesian model reduction and provide the relevant equations for several discrete and continuous probability distributions. We provide worked examples in the context of multivariate linear regression, Gaussian mixture models and dynamical systems (dynamic causal modelling). These examples are accompanied by the Matlab scripts necessary to reproduce the results. Finally, we briefly review recent applications in the fields of neuroimaging and neuroscience. Specifically, we consider structure learning and hierarchical or empirical Bayes that can be regarded as a metaphor for neurobiological processes like abductive reasoning.
Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of two stochastic momentum methods, namely stochastic heavy-ball method and the stochastic variant of Nesterov's accelerated gradient method. We hope that the basic convergence results developed in this paper can serve the reference to the convergence of stochastic momentum methods and also serve the baselines for comparison in future development of stochastic momentum methods. The novelty of convergence analysis presented in this paper is a unified framework, revealing more insights about the similarities and differences between different stochastic momentum methods and stochastic gradient method. The unified framework exhibits a continuous change from the gradient method to Nesterov's accelerated gradient method and finally the heavy-ball method incurred by a free parameter, which can help explain a similar change observed in the testing error convergence behavior for deep learning. Furthermore, our empirical results for optimizing deep neural networks demonstrate that the stochastic variant of Nesterov's accelerated gradient method achieves a good tradeoff (between speed of convergence in training error and robustness of convergence in testing error) among the three stochastic methods.
Mitigating the Influence of the Boundary on PDE-based Covariance Operators

Gaussian random fields over infinite-dimensional Hilbert spaces require the definition of appropriate covariance operators. The use of elliptic PDE operators to construct covariance operators allows to build on fast PDE solvers for manipulations with the resulting covariance and precision operators. However, PDE operators require a choice of boundary conditions, and this choice can have a strong and usually undesired influence on the Gaussian random field. We propose two techniques that allow to ameliorate these boundary effects for large-scale problems. The first approach combines the elliptic PDE operator with a Robin boundary condition, where a varying Robin coefficient is computed from an optimization problem. The second approach normalizes the pointwise variance by rescaling the covariance operator. These approaches can be used individually or can be combined. We study properties of these approaches, and discuss their computational complexity. The performance of our approaches is studied for random fields defined over simple and complex two- and three-dimensional domains.
Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder.

We consider a causal effect that is confounded by an unobserved variable, but with observed proxy variables of the confounder. We show that, with at least two independent proxy variables satisfying a certain rank condition, the causal effect is nonparametrically identified, even if the measurement error mechanism, i.e., the conditional distribution of the proxies given the confounder, may not be identified. Our result generalizes the identification strategy of Kuroki & Pearl (2014) that rests on identification of the measurement error mechanism. When only one proxy for the confounder is available, or the required rank condition is not met, we develop a strategy to test the null hypothesis of no causal effect.
Estimation of Multiple Quantiles in Dynamically Varying Data Streams

In this paper we consider the problem of estimating quantiles when data are received sequentially (data stream). For real life data streams, the distribution of the data typically varies with time making estimation of quantiles challenging. We present a method that simultaneously maintain estimates of multiple quantiles of the data stream distribution. The method is based on making incremental updates of the quantile estimates every time a new sample from the data stream is received. The method is memory and computationally efficient since it only stores one value for each quantile estimate and only performs one operation per quantile estimate when a new sample is received from the data stream. The estimates are realistic in the sense that the monotone property of quantiles is satisfied in every iteration. Experiments show that the method efficiently tracks multiple quantiles and outperforms state of the art methods.