Miracle-dz/UniFAR-FTUs huggingface dataset:
metboost: Exploratory regression analysis with hierarchically clustered data

As data collections become larger, exploratory regression analysis becomes more important but more challenging. When observations are hierarchically clustered the problem is even more challenging because model selection with mixed effect models can produce misleading results when nonlinear effects are not included into the model (Bauer and Cai, 2009). A machine learning method called boosted decision trees (Friedman, 2001) is a good approach for exploratory regression analysis in real data sets because it can detect predictors with nonlinear and interaction effects while also accounting for missing data. We propose an extension to boosted decision decision trees called metboost for hierarchically clustered data. It works by constraining the structure of each tree to be the same across groups, but allowing the terminal node means to differ. This allows predictors and split points to lead to different predictions within each group, and approximates nonlinear group specific effects. Importantly, metboost remains computationally feasible for thousands of observations and hundreds of predictors that may contain missing values. We apply the method to predict math performance for 15,240 students from 751 schools in data collected in the Educational Longitudinal Study 2002 (Ingels et al., 2007), allowing 76 predictors to have unique effects for each school. When comparing results to boosted decision trees, metboost has 15% improved prediction performance. Results of a large simulation study show that metboost has up to 70% improved variable selection performance and up to 30% improved prediction performance compared to boosted decision trees when group sizes are small
On embedded hidden Markov models and particle Markov chain Monte Carlo methods

The embedded hidden Markov model (EHMM) sampling method is a Markov chain Monte Carlo (MCMC) technique for state inference in non-linear non-Gaussian state-space models which was proposed in Neal (2003); Neal et al. (2004) and extended in Shestopaloff and Neal (2016). An extension to Bayesian parameter inference was presented in Shestopaloff and Neal (2013). An alternative class of MCMC schemes addressing similar inference problems is provided by particle MCMC (PMCMC) methods (Andrieu et al. 2009; 2010). All these methods rely on the introduction of artificial extended target distributions for multiple state sequences which, by construction, are such that one randomly indexed sequence is distributed according to the posterior of interest. By adapting the Metropolis-Hastings algorithms developed in the framework of PMCMC methods to the EHMM framework, we obtain novel particle filter (PF)-type algorithms for state inference and novel MCMC schemes for parameter and state inference. In addition, we show that most of these algorithms can be viewed as particular cases of a general PF and PMCMC framework. We compare the empirical performance of the various algorithms on low- to high-dimensional state-space models. We demonstrate that a properly tuned conditional PF with "local" MCMC moves proposed in Shestopaloff and Neal (2016) can outperform the standard conditional PF significantly when applied to high-dimensional state-space models while the novel PF-type algorithm could prove to be an interesting alternative to standard PFs for likelihood estimation in some lower-dimensional scenarios.
Blocking Collapsed Gibbs Sampler for Latent Dirichlet Allocation Models

The latent Dirichlet allocation (LDA) model is a widely-used latent variable model in machine learning for text analysis. Inference for this model typically involves a single-site collapsed Gibbs sampling step for latent variables associated with observations. The efficiency of the sampling is critical to the success of the model in practical large scale applications. In this article, we introduce a blocking scheme to the collapsed Gibbs sampler for the LDA model which can, with a theoretical guarantee, improve chain mixing efficiency. We develop two procedures, an O(K)-step backward simulation and an O(log K)-step nested simulation, to directly sample the latent variables within each block. We demonstrate that the blocking scheme achieves substantial improvements in chain mixing compared to the state of the art single-site collapsed Gibbs sampler. We also show that when the number of topics is over hundreds, the nested-simulation blocking scheme can achieve a significant reduction in computation time compared to the single-site sampler.
A Latent Trait Model for Multivariate Longitudinal Data With Two Sources of Measurement Error

Personality traits are latent variables, and as such, are impossible to measure without the use of an assessment. Responses on the assessments can be influenced by both transient (state-related) error and measurement error, obscuring the true trait levels. Typically, these assessments utilize Likert scales, which yield only discrete data. The loss of information due to the discrete nature of the data represents an additional challenge in assessing the ability of these instruments to measure the latent trait of interest. This paper is concerned with parameter estimation in a model relating a latent variable, as well transient error and measurement error components when data are longitudinal and measured using a Likert scale. Two methods for parameter estimation are detailed: correlation reconstruction, a method that uses polychoric correlations, and maximum likelihood implemented using a Stochastic EM algorithm. These methods are applied to a motivating dataset of 440 college students taking the Big Five inventory twice in a two month period.
Recovering metric from full ordinal information

Given a geodesic space (E, d), we show that full ordinal knowledge on the metric d-i.e. knowledge of the function D d : (w, x, y, z) → 1 d(w,x)≤d(y,z) , determines uniquely-up to a constant factor-the metric d. For a subspace En of n points of E, converging in Hausdorff distance to E, we construct a metric dn on En, based only on the knowledge of D d on En and establish a sharp upper bound of the Gromov-Hausdorff distance between (En, dn) and (E, d).
Improvements in the small sample efficiency of the minimum S-divergence estimators under discrete models

ABSTRACT This paper considers the problem of inliers and empty cells and the resulting issue of relative inefficiency in estimation under pure samples from a discrete population when the sample size is small. Many minimum divergence estimators in the S-divergence family, although possessing very strong outlier stability properties, often have very poor small sample efficiency in the presence of inliers and some are not even defined in the presence of a single empty cell; this limits the practical applicability of these estimators, in spite of their otherwise sound robustness properties and high asymptotic efficiency. Here, we study a penalized version of the S-divergences such that the resulting minimum divergence estimators are free from these issues, without altering their robustness properties and asymptotic efficiencies. We present a general proof for the asymptotic properties of these minimum penalized S-divergence estimators. This provides a significant addition to the literature, as the asymptotics of penalized divergences which are not finitely defined are currently unavailable in the literature. The small sample advantages of the minimum penalized S-divergence estimators are examined through an extensive simulation study and some empirical suggestions regarding the choice of the relevant underlying tuning parameters are also provided.
Detecting Serial Dependence in Binomial Time Series II: Observation Driven Models

The detection of serial dependence in binary or binomial valued time series is difficult using standard time series methods, particularly when there are regression effects to be modelled. In this paper we derive score-type tests for detecting departures from independence in the directions of the GLARMA\ and BARMA\ type observation driven models. These score tests can easily be applied using a standard logistic regression and so may have appeal to practitioners who wish to initially assess the need to incorporate serial dependence effects. To deal with the nuisance parameters in some GLARMA models a supremum type test is implemented.
On the Kendall Correlation Coefficient

In the present paper, we first discuss the Kendall rank correlation coefficient. In continuous case, we define the Kendall rank correlation coefficient in terms of the concomitants of order statistics, find the expected value of the Kendall rank correlation coefficient and show that the later is free of n. We also prove that in continuous case the Kendall correlation coefficient converges in probability to its expected value. We then propose to consider the expected value of the Kendall rank correlation coefficient as a new theoretical correlation coefficient which can be an alternative to the classical Pearson product-moment correlation coefficient. At the end of this work we analyze illustrative examples.
Leveraging Contact Network Information in Clustered Observational Studies of Contagion Processes

In an observational study, obtaining unbiased estimates of an exposure effect requires adjusting for all potential confounders. When this condition is met, leveraging additional covariates related to the outcome may produce less variable estimates of the effect of exposure. For contagion processes operating on a contact network, transmission can only occur through ties that connect exposed and unexposed individuals; the outcome of such a process is known to depend intimately on the structure of the network. In this paper, we investigate the use of contact network features as both confounders and efficiency covariates in exposure effect estimation. Using doubly-robust augmented generalized estimating equations (GEE), we estimate how gains in efficiency depend on the network structure and spread of the contagious agent or behavior. We apply this approach to estimate the effects of two distinct exposures, the proportion of leaders in a village and the proportion of households participating in a self-help program, for the spread of a microfinance program in a collection of villages in Karnataka, India. We compare these results to simulated observational trials using a stochastic compartmental contagion model on a collection of model-based contact networks and compare the bias and variance of the estimated exposure effects using an assortment of network covariate adjustment strategies.
Conflict diagnostics for evidence synthesis in a multiple testing framework

Evidence synthesis models that combine multiple datasets of varying design, to estimate quantities that cannot be directly observed, require the formulation of complex probabilistic models that can be expressed as graphical models. An assessment of whether the different datasets synthesised contribute information that is consistent with each other (and in a Bayesian context, with the prior distribution) is a crucial component of the model criticism process. However, a systematic assessment of conflict in evidence syntheses suffers from the multiple testing problem, through testing for conflict at multiple locations in a model. We demonstrate how conflict diagnostics can be employed throughout a graphical model while accounting for the multiple hypothesis tests of no conflict at each location in the graph. The method is illustrated by a network meta-analysis to estimate treatment effects in smoking cessation programs and an evidence synthesis to estimate HIV prevalence in Poland.
The Statistical Analysis of Pairwise Experiments with Qualitative Responses

Suppose an experiment is conducted on pairs of objects with outcome responses a continuous variable measuring the interactions among the pairs. Furthermore, assume the response variable is hard to measure numerically but easy to be coded into ordered categories such as low, moderate, and high levels of interaction. In this paper we estimate the unknown interaction values from the information contained in the coded data and the design structure of the experiment. The method of estimation is shown to enjoy several optimal properties such as explaining maximum variance in the responses with minimum number of parameters and for any probability distribution underlying the responses. Other properties of the method include: the interactions have the simple interpretation of correlation, size of error is estimable from the experiment, and only a single run of each pair is needed to carry out the experiment. We also explore possible applications of the technique. Three applications are presented, one on protein interaction, a second on drug combination, and the third on computer imaging. The first two applications are illustrated using real life data while for the third application the data are generated via binary coding of an image.
Coupling of Particle Filters

Particle filters provide Monte Carlo approximations of intractable quantities such as point-wise evaluations of the likelihood in state space models. In many scenarios, the interest lies in the comparison of these quantities as some parameter or input varies. To facilitate such comparisons, we introduce and study methods to couple two particle filters in such a way that the correlation between the two underlying particle systems is increased. The motivation stems from the classic variance reduction technique of positively correlating two estimators. The key challenge in constructing such a coupling stems from the discontinuity of the resampling step of the particle filter. As our first contribution, we consider coupled resampling algorithms. Within bootstrap particle filters, they improve the precision of finite-difference estimators of the score vector and boost the performance of particle marginal Metropolis--Hastings algorithms for parameter inference. The second contribution arises from the use of these coupled resampling schemes within conditional particle filters, allowing for unbiased estimators of smoothing functionals. The result is a new smoothing strategy that operates by averaging a number of independent and unbiased estimators, which allows for 1) straightforward parallelization and 2) the construction of accurate error estimates. Neither of the above is possible with existing particle smoothers.
Horseshoe Regularization for Feature Subset Selection

Feature subset selection arises in many high-dimensional applications of statistics, such as compressed sensing and genomics. The $\ell_0$ penalty is ideal for this task, the caveat being it requires the NP-hard combinatorial evaluation of all models. A recent area of considerable interest is to develop efficient algorithms to fit models with a non-convex $\ell_\gamma$ penalty for $\gamma\in (0,1)$, which results in sparser models than the convex $\ell_1$ or lasso penalty, but is harder to fit. We propose an alternative, termed the horseshoe regularization penalty for feature subset selection, and demonstrate its theoretical and computational advantages. The distinguishing feature from existing non-convex optimization approaches is a full probabilistic representation of the penalty as the negative of the logarithm of a suitable prior, which in turn enables efficient expectation-maximization and local linear approximation algorithms for optimization and MCMC for uncertainty quantification. In synthetic and real data, the resulting algorithms provide better statistical performance, and the computation requires a fraction of time of state-of-the-art non-convex solvers.
Predicting human-driving behavior to help driverless vehicles drive: random intercept Bayesian Additive Regression Trees

The development of driverless vehicles has spurred the need to predict human driving behavior to facilitate interaction between driverless and human-driven vehicles. Predicting human driving movements can be challenging, and poor prediction models can lead to accidents between the driverless and human-driven vehicles. We used the vehicle speed obtained from a naturalistic driving dataset to predict whether a human-driven vehicle would stop before executing a left turn. In a preliminary analysis, we found that BART produced less variable and higher AUC values compared to a variety of other state-of-the-art binary predictor methods. However, BART assumes independent observations, but our dataset consists of multiple observations clustered by driver. Although methods extending BART to clustered or longitudinal data are available, they lack readily available software and can only be applied to clustered continuous outcomes. We extend BART to handle correlated binary observations by adding a random intercept and used a simulation study to determine bias, root mean squared error, 95% coverage, and average length of 95% credible interval in a correlated data setting. We then successfully implemented our random intercept BART model to our clustered dataset and found substantial improvements in prediction performance compared to BART and random intercept linear logistic regression.
Multiple Influential Point Detection in High-Dimensional Spaces

Influence diagnosis is an integrated component of data analysis, but is severely under-investigated in a high-dimensional setting. One of the key challenges, even in a fixed-dimensional setting, is how to deal with multiple influential points giving rise to the masking and swamping effects. This paper proposes a novel group deletion procedure referred to as MIP by studying two extreme statistics based on a marginal correlation based influence measure. Named the Min and Max statistics, they have complimentary properties in that the Max statistic is effective for overcoming the masking effect while the Min statistic is useful for overcoming the swamping effect. Combining their strengths, we further propose an efficient algorithm that can detect influential points with a prespecified false discovery rate. The proposed influential point detection procedure is simple to implement, efficient to run, and enjoys attractive theoretical properties. Its effectiveness is verified empirically via extensive simulation study and data analysis. An R package implementing the procedure is freely available.
Convex method for selection of fixed effects in high-dimensional linear mixed models

Analysis of high-dimensional data is currently a popular field of research, thanks to many applications e.g. in genetics (DNA data in genomewide association studies), spectrometry or web analysis. At the same time, the type of problems that tend to arise in genetics can often be modelled using linear mixed models in conjunction with high-dimensional data because linear mixed models allow us to specify the covariance structure of the models. This enables us to capture relationships in data such as the population structure, family relatedness, etc. In this paper we introduce two new convex methods for variable selection in high-dimensional linear mixed models which, thanks to convexity, can handle many more variables than existing non-convex methods. Both methods are compared with existing methods and in the end we suggest an approach for a wider class of linear mixed models.
Sparse Poisson regression with penalized weighted score function

We proposed a new penalized method in this paper to solve sparse Poisson Regression problems. Being different from $\ell_1$ penalized log-likelihood estimation, our new method can be viewed as penalized weighted score function method. We show that under mild conditions, our estimator is $\ell_1$ consistent and the tuning parameter can be pre-specified, which shares the same good property of the square-root Lasso.
Functional Inverse Regression in an Enlarged Dimension Reduction Space

We consider an enlarged dimension reduction space in functional inverse regression. Our operator and functional analysis based approach facilitates a compact and rigorous formulation of the functional inverse regression problem. It also enables us to expand the possible space where the dimension reduction functions belong. Our formulation provides a unified framework so that the classical notions, such as covariance standardization, Mahalanobis distance, SIR and linear discriminant analysis, can be naturally and smoothly carried out in our enlarged space. This enlarged dimension reduction space also links to the linear discriminant space of Gaussian measures on a separable Hilbert space.
Probabilistic Bisection Converges Almost as Quickly as Stochastic Approximation

The probabilistic bisection algorithm (PBA) solves a class of stochastic root-finding problems in one dimension by successively updating a prior belief on the location of the root based on noisy responses to queries at chosen points. The responses indicate the direction of the root from the queried point, and are incorrect with a fixed probability. The fixed-probability assumption is problematic in applications, and so we extend the PBA to apply when this assumption is relaxed. The extension involves the use of a power-one test at each queried point. We explore the convergence behavior of the extended PBA, showing that it converges at a rate arbitrarily close to, but slower than, the canonical "square root" rate of stochastic approximation.
General Latent Feature Models for Heterogeneous Datasets

Latent feature modeling allows capturing the latent structure responsible for generating the observed properties of a set of objects. It is often used to make predictions either for new values of interest or missing information in the original data, as well as to perform data exploratory analysis. However, although there is an extensive literature on latent feature models for homogeneous datasets, where all the attributes that describe each object are of the same (continuous or discrete) nature, there is a lack of work on latent feature modeling for heterogeneous databases. In this paper, we introduce a general Bayesian nonparametric latent feature model suitable for heterogeneous datasets, where the attributes describing each object can be either discrete, continuous or mixed variables. The proposed model presents several important properties. First, it accounts for heterogeneous data while keeping the properties of conjugate models, which allow us to infer the model in linear time with respect to the number of objects and attributes. Second, its Bayesian nonparametric nature allows us to automatically infer the model complexity from the data, i.e., the number of features necessary to capture the latent structure in the data. Third, the latent features in the model are binary-valued variables, easing the interpretability of the obtained latent features in data exploratory analysis. We show the flexibility of the proposed model by solving both prediction and data analysis tasks on several real-world datasets. Moreover, a software package of the GLFM is publicly available for other researcher to use and improve it.