cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.6084/m9.figshare.7887842
title: Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and Its Asymptotic Optimality

description: Tuning parameter selection is of critical importance for kernel ridge regression. To date, a data-driven tuning method for divide-and-conquer kernel ridge regression (d-KRR) has been lacking in the literature, which limits the applicability of d-KRR for large datasets. In this article, by modifying the generalized cross-validation (GCV) score, we propose a distributed generalized cross-validation (dGCV) as a data-driven tool for selecting the tuning parameters in d-KRR. Not only the proposed dGCV is computationally scalable for massive datasets, it is also shown, under mild conditions, to be asymptotically optimal in the sense that minimizing the dGCV score is equivalent to minimizing the true global conditional empirical loss of the averaged function estimator, extending the existing optimality results of GCV to the divide-and-conquer framework. Supplemental materials for this article are available online.
DOI: 10.6084/m9.figshare.7904939.v2
title: Component-Based Regularization of Multivariate Generalized Linear Mixed Models

description: We address the component-based regularization of a multivariate Generalized Linear Mixed Model (GLMM) in the framework of grouped data. A set Y of random responses is modelled with a multivariate GLMM, based on a set X of explanatory variables, a set A of additional explanatory variables, and random effects to introduce the within-group dependence of observations. Variables in X are assumed many and redundant so that regression demands regularization. This is not the case for A, which contains few and selected variables. Regularization is performed building an appropriate number of orthogonal components that both contribute to model Y and capture relevant structural information in X. To estimate the model, we propose to maximize a criterion specific to the supervised component-based generalized linear regression (SCGLR) within an adaptation of Schall’s algorithm. This extension of SCGLR is tested on both simulated and real grouped data, and compared to ridge and LASSO regularizations. Supplementary material for this article is available online.
DOI: 10.6084/m9.figshare.8063768.v2
title: Fast Generalized Linear Models by Database Sampling and One-Step Polishing

description: In this article, I show how to fit a generalized linear model to <i>N</i> observations on <i>p</i> variables stored in a relational database, using one sampling query and one aggregation query, as long as N12+δ observations can be stored in memory, for some δ&gt;0. The resulting estimator is fully efficient and asymptotically equivalent to the maximum likelihood estimator, and so its variance can be estimated from the Fisher information in the usual way. A proof-of-concept implementation uses R with MonetDB and with SQLite, and could easily be adapted to other popular databases. I illustrate the approach with examples of taxi-trip data in New York City and factors related to car color in New Zealand. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7862876
title: Damped Anderson Acceleration With Restarts and Monotonicity Control for Accelerating EM and EM-like Algorithms

description: The expectation-maximization (EM) algorithm is a well-known iterative method for computing maximum likelihood estimates in a variety of statistical problems. Despite its numerous advantages, a main drawback of the EM algorithm is its frequently observed slow convergence which often hinders the application of EM algorithms in high-dimensional problems or in other complex settings. To address the need for more rapidly convergent EM algorithms, we describe a new class of acceleration schemes that build on the Anderson acceleration technique for speeding fixed-point iterations. Our approach is effective at greatly accelerating the convergence of EM algorithms and is automatically scalable to high-dimensional settings. Through the introduction of periodic algorithm restarts and a damping factor, our acceleration scheme provides faster and more robust convergence when compared to un-modified Anderson acceleration, while also improving global convergence. Crucially, our method works as an “off-the-shelf” method in that it may be directly used to accelerate any EM algorithm without relying on the use of any model-specific features or insights. Through a series of simulation studies involving five representative problems, we show that our algorithm is substantially faster than the existing state-of-art acceleration schemes. The acceleration schemes described in this paper are implemented in the R package daarem which is available from the comprehensive R archive network (https://cran.r-project.org). Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7910549.v4
title: Adaptive Incremental Mixture Markov Chain Monte Carlo

description: We propose adaptive incremental mixture Markov chain Monte Carlo (AIMM), a novel approach to sample from challenging probability distributions defined on a general state-space. While adaptive MCMC methods usually update a parametric proposal kernel with a global rule, AIMM locally adapts a semiparametric kernel. AIMM is based on an independent Metropolis–Hastings proposal distribution which takes the form of a finite mixture of Gaussian distributions. Central to this approach is the idea that the proposal distribution adapts to the target by locally adding a mixture component when the discrepancy between the proposal mixture and the target is deemed to be too large. As a result, the number of components in the mixture proposal is not fixed in advance. Theoretically, we prove that there exists a stochastic process that can be made arbitrarily close to AIMM and that converges to the correct target distribution. We also illustrate that it performs well in practice in a variety of challenging situations, including high-dimensional and multimodal target distributions. Finally, the methodology is successfully applied to two real data examples, including the Bayesian inference of a semiparametric regression model for the Boston Housing dataset. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7959380.v3
title: A Metaheuristic Adaptive Cubature Based Algorithm to Find Bayesian Optimal Designs for Nonlinear Models

description: Finding Bayesian optimal designs for nonlinear models is a difficult task because the optimality criterion typically requires us to evaluate complex integrals before we perform a constrained optimization. We propose a hybridized method where we combine an adaptive multidimensional integration algorithm and a metaheuristic algorithm called imperialist competitive algorithm to find Bayesian optimal designs. We apply our numerical method to a few challenging design problems to demonstrate its efficiency. They include finding <i>D</i>-optimal designs for an item response model commonly used in education, Bayesian optimal designs for survival models, and Bayesian optimal designs for a four-parameter sigmoid Emax dose response model. Supplementary materials for this article are available online and they contain an R package for implementing the proposed algorithm and codes for reproducing all the results in this article.
DOI: 10.6084/m9.figshare.7869350.v4
title: Simultaneous Variable and Covariance Selection With the Multivariate Spike-and-Slab LASSO

description: We propose a Bayesian procedure for simultaneous variable and covariance selection using continuous spike-and-slab priors in multivariate linear regression models where <i>q</i> possibly correlated responses are regressed onto <i>p</i> predictors. Rather than relying on a stochastic search through the high-dimensional model space, we develop an ECM algorithm similar to the EMVS procedure of Ročková and George targeting modal estimates of the matrix of regression coefficients and residual precision matrix. Varying the scale of the continuous spike densities facilitates dynamic posterior exploration and allows us to filter out negligible regression coefficients and partial covariances gradually. Our method is seen to substantially outperform regularization competitors on simulated data. We demonstrate our method with a re-examination of data from a recent observational study of the effect of playing high school football on several later-life cognition, psychological, and socio-economic outcomes. An R package, scripts for replicating examples in this article, and results from further simulation studies are provided in the supplementary materials available online.
DOI: 10.6084/m9.figshare.7910549.v3
title: Adaptive Incremental Mixture Markov Chain Monte Carlo

description: We propose adaptive incremental mixture Markov chain Monte Carlo (AIMM), a novel approach to sample from challenging probability distributions defined on a general state-space. While adaptive MCMC methods usually update a parametric proposal kernel with a global rule, AIMM locally adapts a semiparametric kernel. AIMM is based on an independent Metropolis–Hastings proposal distribution which takes the form of a finite mixture of Gaussian distributions. Central to this approach is the idea that the proposal distribution adapts to the target by locally adding a mixture component when the discrepancy between the proposal mixture and the target is deemed to be too large. As a result, the number of components in the mixture proposal is not fixed in advance. Theoretically, we prove that there exists a stochastic process that can be made arbitrarily close to AIMM and that converges to the correct target distribution. We also illustrate that it performs well in practice in a variety of challenging situations, including high-dimensional and multimodal target distributions. Finally, the methodology is successfully applied to two real data examples, including the Bayesian inference of a semiparametric regression model for the Boston Housing dataset. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7859750
title: Incremental Mixture Importance Sampling With Shotgun Optimization

description: <b>This article proposes a general optimization strategy, which combines results from different optimization or parameter estimation methods to overcome shortcomings of a single method. Shotgun optimization is developed as a framework which employs different optimization strategies, criteria, or conditional targets to enable wider likelihood exploration. The introduced shotgun optimization approach is embedded into an incremental mixture importance sampling algorithm to produce improved posterior samples for multimodal densities and creates robustness in cases where the likelihood and prior are in disagreement. Despite using different optimization approaches, the samples are combined into samples from a single target posterior. The diversity of the framework is demonstrated on parameter estimation from differential equation models employing diverse strategies including numerical solutions and approximations thereof. Additionally the approach is demonstrated on mixtures of discrete and continuous parameters and is shown to ease estimation from synthetic likelihood models. R code of the implemented examples can be found at</b>https://github.com/BiljanaJSJ/IMIS-ShOpt<b>. Supplementary materials for this article are available online.</b>
DOI: 10.6084/m9.figshare.7862810
title: Improving Spectral Clustering Using the Asymptotic Value of the Normalized Cut

description: Spectral clustering (SC) is a popular and versatile clustering method based on a relaxation of the normalized graph cut objective. Despite its popularity, selecting the number of clusters and tuning the important scaling parameter remain challenging problems in practical applications of SC. Popular heuristics have been proposed, but corresponding theoretical results are scarce. In this article, we investigate the asymptotic value of the normalized cut for an increasing sample assumed to arise from an underlying probability distribution. Based on this, we find strong connections between spectral and density clustering. This enables us to provide recommendations for selecting the number of clusters and setting the scaling parameter in a data driven manner. An algorithm inspired by these recommendations is proposed, which we have found to exhibit strong performance in a range of applied domains. An R implementation of the algorithm is available from https://github.com/DavidHofmeyr/spuds. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7887842.v3
title: Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and Its Asymptotic Optimality

description: Tuning parameter selection is of critical importance for kernel ridge regression. To date, a data-driven tuning method for divide-and-conquer kernel ridge regression (d-KRR) has been lacking in the literature, which limits the applicability of d-KRR for large datasets. In this article, by modifying the generalized cross-validation (GCV) score, we propose a distributed generalized cross-validation (dGCV) as a data-driven tool for selecting the tuning parameters in d-KRR. Not only the proposed dGCV is computationally scalable for massive datasets, it is also shown, under mild conditions, to be asymptotically optimal in the sense that minimizing the dGCV score is equivalent to minimizing the true global conditional empirical loss of the averaged function estimator, extending the existing optimality results of GCV to the divide-and-conquer framework. Supplemental materials for this article are available online.
DOI: 10.6084/m9.figshare.7869350.v3
title: Simultaneous Variable and Covariance Selection With the Multivariate Spike-and-Slab LASSO

description: We propose a Bayesian procedure for simultaneous variable and covariance selection using continuous spike-and-slab priors in multivariate linear regression models where <i>q</i> possibly correlated responses are regressed onto <i>p</i> predictors. Rather than relying on a stochastic search through the high-dimensional model space, we develop an ECM algorithm similar to the EMVS procedure of Ročková and George targeting modal estimates of the matrix of regression coefficients and residual precision matrix. Varying the scale of the continuous spike densities facilitates dynamic posterior exploration and allows us to filter out negligible regression coefficients and partial covariances gradually. Our method is seen to substantially outperform regularization competitors on simulated data. We demonstrate our method with a re-examination of data from a recent observational study of the effect of playing high school football on several later-life cognition, psychological, and socio-economic outcomes. An R package, scripts for replicating examples in this article, and results from further simulation studies are provided in the supplementary materials available online.
DOI: 10.6084/m9.figshare.7932509
title: Projection Pursuit Based on Gaussian Mixtures and Evolutionary Algorithms

description: We propose a projection pursuit (PP) algorithm based on Gaussian mixture models (GMMs). The negentropy obtained from a multivariate density estimated by GMMs is adopted as the PP index to be maximized. For a fixed dimension of the projection subspace, the GMM-based density estimation is projected onto that subspace, where an approximation of the negentropy for Gaussian mixtures is computed. Then, genetic algorithms are used to find the optimal, orthogonal projection basis by maximizing the former approximation. We show that this semiparametric approach to PP is flexible and allows highly informative structures to be detected, by projecting multivariate datasets onto a subspace, where the data can be feasibly visualized. The performance of the proposed approach is shown on both artificial and real datasets. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7869350
title: Simultaneous Variable and Covariance Selection With the Multivariate Spike-and-Slab LASSO

description: We propose a Bayesian procedure for simultaneous variable and covariance selection using continuous spike-and-slab priors in multivariate linear regression models where <i>q</i> possibly correlated responses are regressed onto <i>p</i> predictors. Rather than relying on a stochastic search through the high-dimensional model space, we develop an ECM algorithm similar to the EMVS procedure of Ročková and George targeting modal estimates of the matrix of regression coefficients and residual precision matrix. Varying the scale of the continuous spike densities facilitates dynamic posterior exploration and allows us to filter out negligible regression coefficients and partial covariances gradually. Our method is seen to substantially outperform regularization competitors on simulated data. We demonstrate our method with a re-examination of data from a recent observational study of the effect of playing high school football on several later-life cognition, psychological, and socio-economic outcomes. An R package, scripts for replicating examples in this article, and results from further simulation studies are provided in the supplementary materials available online.
DOI: 10.6084/m9.figshare.7887842.v2
title: Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and Its Asymptotic Optimality

description: Tuning parameter selection is of critical importance for kernel ridge regression. To date, a data-driven tuning method for divide-and-conquer kernel ridge regression (d-KRR) has been lacking in the literature, which limits the applicability of d-KRR for large datasets. In this article, by modifying the generalized cross-validation (GCV) score, we propose a distributed generalized cross-validation (dGCV) as a data-driven tool for selecting the tuning parameters in d-KRR. Not only the proposed dGCV is computationally scalable for massive datasets, it is also shown, under mild conditions, to be asymptotically optimal in the sense that minimizing the dGCV score is equivalent to minimizing the true global conditional empirical loss of the averaged function estimator, extending the existing optimality results of GCV to the divide-and-conquer framework. Supplemental materials for this article are available online.
DOI: 10.6084/m9.figshare.7981790.v2
title: Variable-Domain Functional Principal Component Analysis

description: We introduce a novel method of principal component analysis for data with varying domain lengths for each functional observation. We refer to this technique as variable-domain functional principal component analysis, or vd-FPCA. We fit a trivariate smoother using penalized thin plate splines to estimate the covariance as a function of the domain length. Principal components are then calculated through eigen-decomposition of the estimated covariance matrix, conditional on the domain length. We apply vd-FPCA in two functional data settings, first to daily measures of patient wellness during a stay in the ICU, and second, to accelerometer recordings of repeated in-lab movements. In each example, vd-FPCA uses fewer principal components than typical FPCA methods to explain a greater proportion of the variation in the data. We also find the principal components provide greater flexibility in interpretation with respect to domain length than traditional approaches. These methods are easily implementable through standard statistical software and applicable to a wide variety of datasets involving continuous observations over a variable domain. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.8063765
title: Beyond Prediction: A Framework for Inference With Variational Approximations in Mixture Models

description: Variational inference is a popular method for estimating model parameters and conditional distributions in hierarchical and mixed models, which arise frequently in many settings in the health, social, and biological sciences. Variational inference in a frequentist context works by approximating intractable conditional distributions with a tractable family and optimizing the resulting lower bound on the log-likelihood. The variational objective function is typically less computationally intensive to optimize than the true likelihood, enabling scientists to fit rich models even with extremely large datasets. Despite widespread use, little is known about the general theoretical properties of estimators arising from variational approximations to the log-likelihood, which hinders their use in inferential statistics. In this article, we connect such estimators to profile <i>M</i>-estimation, which enables us to provide regularity conditions for consistency and asymptotic normality of variational estimators. Our theory also motivates three methodological improvements to variational inference: estimation of the asymptotic model-robust covariance matrix, a one-step correction that improves estimator efficiency, and an empirical assessment of consistency. We evaluate the proposed results using simulation studies and data on marijuana use from the National Longitudinal Study of Youth. Supplementary materials for this article are available online.
DOI: 10.6084/m9.figshare.7862810.v3
title: Improving Spectral Clustering Using the Asymptotic Value of the Normalized Cut

description: Spectral clustering (SC) is a popular and versatile clustering method based on a relaxation of the normalized graph cut objective. Despite its popularity, selecting the number of clusters and tuning the important scaling parameter remain challenging problems in practical applications of SC. Popular heuristics have been proposed, but corresponding theoretical results are scarce. In this article, we investigate the asymptotic value of the normalized cut for an increasing sample assumed to arise from an underlying probability distribution. Based on this, we find strong connections between spectral and density clustering. This enables us to provide recommendations for selecting the number of clusters and setting the scaling parameter in a data driven manner. An algorithm inspired by these recommendations is proposed, which we have found to exhibit strong performance in a range of applied domains. An R implementation of the algorithm is available from https://github.com/DavidHofmeyr/spuds. Supplementary materials for this article are available online.