Miracle-dz/UniFAR-FTUs huggingface dataset:
A BIRTH AND DEATH PROCESS FOR BAYESIAN NETWORK STRUCTURE INFERENCE

Bayesian networks are convenient graphical expressions for high-dimensional probability distributions which represent complex relationships between a large number of random variables. They have been employed extensively in areas such as bioinformatics, artificial intelligence, diagnosis, and risk management. The recovery of the structure of a network from data is of prime importance for the purposes of modeling, analysis, and prediction. There has been a great deal of interest in recent years in the NP-hard problem of learning the structure of a Bayesian network from observed data. Typically, one assigns a score to various structures and the search becomes an optimization problem that can be approached with either deterministic or stochastic methods. In this paper, we introduce a new search strategy where one walks through the space of graphs by modeling the appearance and disappearance of edges as a birth and death process. We compare our novel approach with the popular Metropolis–Hastings search strategy and give empirical evidence that the birth and death process has superior mixing properties.
Auxiliary gradient‐based sampling algorithms

We introduce a new family of Markov chain Monte Carlo samplers that combine auxiliary variables, Gibbs sampling and Taylor expansions of the target density. Our approach permits the marginalization over the auxiliary variables, yielding marginal samplers, or the augmentation of the auxiliary variables, yielding auxiliary samplers. The well‐known Metropolis‐adjusted Langevin algorithm MALA and preconditioned Crank–Nicolson–Langevin algorithm pCNL are shown to be special cases. We prove that marginal samplers are superior in terms of asymptotic variance and demonstrate cases where they are slower in computing time compared with auxiliary samplers. In the context of latent Gaussian models we propose new auxiliary and marginal samplers whose implementation requires a single tuning parameter, which can be found automatically during the transient phase. Extensive experimentation shows that the increase in efficiency (measured as the effective sample size per unit of computing time) relative to (optimized implementations of) pCNL, elliptical slice sampling and MALA ranges from tenfold in binary classification problems to 25 fold in log‐Gaussian Cox processes to 100 fold in Gaussian process regression, and it is on a par with Riemann manifold Hamiltonian Monte Carlo sampling in an example where that algorithm has the same complexity as the aforementioned algorithms. We explain this remarkable improvement in terms of the way that alternative samplers try to approximate the eigenvalues of the target. We introduce a novel Markov chain Monte Carlo sampling scheme for hyperparameter learning that builds on the auxiliary samplers. The MATLAB code for reproducing the experiments in the paper is publicly available and an on‐line supplement to this paper contains additional experiments and implementation details.
A penalized approach to the bivariate logistic regression model for the association between ordinal responses

Bivariate ordered logistic models (BOLMs) are appealing to jointly model the marginal distribution of two ordered responses and their association, given a set of co- variates. When the number of categories of the responses increases, the number of global odds ratios (or their re- parametrizations) to be estimated also increases and estimat- ing the association structure becomes crucial for this type of data. In fact, such data could be too "rich" to be fully mod- elled with an ordinary BOLM while, sometimes, the well- known Dale's model could be too parsimonious to provide a good fit. In addition, when the cross-tabulation of the re- sponses contains some zeros, for a number of model config- urations, including the bivariate version of the partial pr o- portional odds model (PPOM), estimation of a BOLM by the Fisher-scoring algorithm may either fail or estimate a too "irregular" association structure. In this work, we propose to use a nonparametric ap- proach for the maximum likelihood estimation of a BOLM. We apply penalties to the differences between adjacent row and column effects. As a result, estimation is less demand- ing than an ordinary BOLM, permitting the fit of PPOMs and/or the smoothing of the marginal and association param- eters by polynomial curves and surfaces, with scores chosen by the data. Model selection is based on the penalized log- likelihood ratio, whose limiting distribution has been stu d- ied through simulations, and AIC. Our proposal is compared to the Goodman's model and the Dale's model, in terms of goodness-of-fit and parsimony, on a literature data set. Fi- nally, an application on an original data set of liver diseas e patients is proposed.
Solution path clustering with adaptive concave penalty

Fast accumulation of large amounts of complex data has created a need for more sophisticated statistical methodologies to discover interesting patterns and better extract information from these data. The large scale of the data often results in challenging high-dimensional estimation problems where only a minority of the data shows specific grouping patterns. To address these emerging challenges, we develop a new clustering methodology that introduces the idea of a regularization path into unsupervised learning. A regularization path for a clustering problem is created by varying the degree of sparsity constraint that is imposed on the differences between objects via the minimax concave penalty with adaptive tuning parameters. Instead of providing a single solution represented by a cluster assignment for each object, the method produces a short sequence of solutions that determines not only the cluster assignment but also a corresponding number of clusters for each solution. The optimization of the penalized loss function is carried out through an MM algorithm with block coordinate descent. The advantages of this clustering algorithm compared to other existing methods are as follows: it does not require the input of the number of clusters; it is capable of simultaneously separating irrelevant or noisy observations that show no grouping pattern, which can greatly improve data interpretation; it is a general methodology that can be applied to many clustering problems. We test this method on various simulated datasets and on gene expression data, where it shows better or competitive performance compared against several clustering methods.
Bayesian Aggregation of Rank Data with Covariates and Heterogeneous Rankers

Rank aggregation is the combining of ranking results from different sources to generate a "better" ranking list. In our applications, the rank data contain covariate information of ranked entities and incomplete ranking lists for non-overlapping subgroups. Since most existing rank aggregation methods do not handle covariate information of the ranked entities as well as the rankers' heterogeneity, we propose the Bayesian Aggregation of Rank-data with Covariates (BARC), its weighted version (BARCW), and its extension to mixture models (BARCM). All three methods employ latent variable models to account for the covariate information and heterogeneity of rankers. Specifically, BARC assumes identical opinion of all rankers with the same quality; BARCW extends it by allowing varying qualities of rankers, while BARCM clusters heterogeneous ranking opinions with a Dirichlet process mixture model. Moreover, we use a parameter-expanded Gibbs sampler to draw posterior samples, and generate aggregated ranking lists with credible intervals quantifying their uncertainty. Simulation studies show the superior performance of our methods compared with other existing methods in a variety of scenarios. Finally, we exploit our proposed method to solve two real-data problems.
Regularizing Random Points: Complementary Matérn Hard-Core Point Process

In this paper we present a tractable approach for regularizing randomly placed points, by splitting them into two subsets: the first is generated by means of the Mat\'ern hard-core point process, while the remaining points constitute the complementary Mat\'ern hard-core point process. We study the characteristics of these processes, deriving its pair-correlation functions, and the distribution of the distance to the nearest neighbour. The results have several applications in wireless communications, including the modeling of wireless sensor networks, where we investigate an example of regularizing such networks and illustrate its advantage in reducing the energy consumption of wireless nodes.
Hamiltonian Monte Carlo for Hierarchical Models

Hierarchical modeling provides a framework for modeling the complex interactions typical of problems in applied statistics. By capturing these relationships, however, hierarchical models also introduce distinctive pathologies that quickly limit the efficiency of most common methods of in- ference. In this paper we explore the use of Hamiltonian Monte Carlo for hierarchical models and demonstrate how the algorithm can overcome those pathologies in practical applications.
Simpson's Paradox and Collapsibility

Simpson's paradox and collapsibility are two closely related concepts in the context of data analysis. While the knowledge about the occurrence of Simpson's paradox helps a statistician to draw correct and meaningful conclusions, the concept of collapsibility deals with dimension-reduction aspects, when Simpson's paradox does not occur. We discuss in this paper in some detail the nature and the genesis of Simpson's paradox with respect to well-known examples and also various concepts of collapsiblity. The main aim is to bring out the close connections between these two phenomena, especially with regard to the analysis of contingency tables, regression models and a certain measure of association or a dependence function. There is a vast literature on these topics and so we focus only on certain aspects, recent developments and some important results in the above-mentioned areas.
Expanded Alternating Optimization of Nonconvex Functions with Applications to Matrix Factorization and Penalized Regression

We propose a general technique for improving alternating optimization (AO) of nonconvex functions. Starting from the solution given by AO, we conduct another sequence of searches over subspaces that are both meaningful to the optimization problem at hand and different from those used by AO. To demonstrate the utility of our approach, we apply it to the matrix factorization (MF) algorithm for recommender systems and the coordinate descent algorithm for penalized regression (PR), and show meaningful improvements using both real-world (for MF) and simulated (for PR) data sets. Moreover, we demonstrate for MF that, by constructing search spaces customized to the given data set, we can significantly increase the convergence rate of our technique.
Changepoint Detection in the Presence of Outliers

ABSTRACT Many traditional methods for identifying changepoints can struggle in the presence of outliers, or when the noise is heavy-tailed. Often they will infer additional changepoints to fit the outliers. To overcome this problem, data often needs to be preprocessed to remove outliers, though this is difficult for applications where the data needs to be analyzed online. We present an approach to changepoint detection that is robust to the presence of outliers. The idea is to adapt existing penalized cost approaches for detecting changes so that they use loss functions that are less sensitive to outliers. We argue that loss functions that are bounded, such as the classical biweight loss, are particularly suitable—as we show that only bounded loss functions are robust to arbitrarily extreme outliers. We present an efficient dynamic programming algorithm that can find the optimal segmentation under our penalized cost criteria. Importantly, this algorithm can be used in settings where the data needs to be analyzed online. We show that we can consistently estimate the number of changepoints, and accurately estimate their locations, using the biweight loss function. We demonstrate the usefulness of our approach for applications such as analyzing well-log data, detecting copy number variation, and detecting tampering of wireless devices. Supplementary materials for this article are available online.
Accelerated Nonparametric Maximum Likelihood Density Deconvolution Using Bernstein Polynomial

A new maximum likelihood method for deconvoluting a continuous density with a positive lower bound on a known compact support in additive measurement error models with known error distribution using the approximate Bernstein type polynomial model, a finite mixture of specific beta distributions, is proposed. The change-point detection method is used to choose an optimal model degree. Based on a contaminated sample of size $n$, under an assumption which is satisfied, among others, by the generalized normal error distribution, the optimal rate of convergence of the mean integrated squared error is proved to be $k^{-1}\mathcal{O}(n^{-1+1/k}\log^3 n)$ if the underlying unknown density has continuous $2k$th derivative with $k>1$. Simulation shows that small sample performance of our estimator is better than the deconvolution kernel density estimator. The proposed method is illustrated by a real data application.
Beyond Uniform Priors in Bayesian Network Structure Learning

Bayesian network structure learning is often performed in a Bayesian setting, evaluating candidate structures using their posterior probabilities for a given data set. Score-based algorithms then use those posterior probabilities as an objective function and return the maximum a posteriori network as the learned model. For discrete Bayesian networks, the canonical choice for a posterior score is the Bayesian Dirichlet equivalent uniform (BDeu) marginal likelihood with a uniform (U) graph prior, which assumes a uniform prior both on the network structures and on the parameters of the networks. In this paper, we investigate the problems arising from these assumptions, focusing on those caused by small sample sizes and sparse data. We then propose an alternative posterior score: the Bayesian Dirichlet sparse (BDs) marginal likelihood with a marginal uniform (MU) graph prior. Like U+BDeu, MU+BDs does not require any prior information on the probabilistic structure of the data and can be used as a replacement noninformative score. We study its theoretical properties and we evaluate its performance in an extensive simulation study, showing that MU+BDs is both more accurate than U+BDeu in learning the structure of the network and competitive in predicting power, while not being computationally more complex to estimate.
Impact of subsampling and pruning on random forests

Random forests are ensemble learning methods introduced by Breiman (2001) that operate by averaging several decision trees built on a randomly selected subspace of the data set. Despite their widespread use in practice, the respective roles of the different mechanisms at work in Breiman's forests are not yet fully understood, neither is the tuning of the corresponding parameters. In this paper, we study the influence of two parameters , namely the subsampling rate and the tree depth, on Breiman's forests performance. More precisely, we show that fully developed sub-sampled forests and pruned (without subsampling) forests have similar performances, as long as respective parameters are well chosen. Moreover, experiments show that a proper tuning of subsampling or pruning lead in most cases to an improvement of Breiman's original forests errors.
Generalized Alpha Investing: Definitions, Optimality Results, and Application to Public Databases

The increasing prevalence and utility of large, public databases necessitates the development of appropriate methods for controlling false discovery. Motivated by this challenge, we discuss the generic problem of testing a possibly infinite stream of null hypotheses. In this context, Foster and Stine (2008) suggested a novel method named Alpha Investing for controlling a false discovery measure known as mFDR. We develop a more general procedure for controlling mFDR, of which Alpha Investing is a special case. We show that in common, practical situations, the general procedure can be optimized to produce an expected reward optimal (ERO) version, which is more powerful than Alpha Investing. We then present the concept of quality preserving databases (QPD), originally introduced in Aharoni et al. (2011), which formalizes efficient public database management to simultaneously save costs and control false discovery. We show how one variant of generalized alpha investing can be used to control mFDR in a QPD and lead to significant reduction in costs compared to naive approaches for controlling the Family-Wise Error Rate implemented in Aharoni et al. (2011).
Multitask Learning using Task Clustering with Applications to Predictive Modeling and GWAS of Plant Varieties

Inferring predictive maps between multiple input and multiple output variables or tasks has innumerable applications in data science. Multi-task learning attempts to learn the maps to several output tasks simultaneously with information sharing between them. We propose a novel multi-task learning framework for sparse linear regression, where a full task hierarchy is automatically inferred from the data, with the assumption that the task parameters follow a hierarchical tree structure. The leaves of the tree are the parameters for individual tasks, and the root is the global model that approximates all the tasks. We apply the proposed approach to develop and evaluate: (a) predictive models of plant traits using large-scale and automated remote sensing data, and (b) GWAS methodologies mapping such derived phenotypes in lieu of hand-measured traits. We demonstrate the superior performance of our approach compared to other methods, as well as the usefulness of discovering hierarchical groupings between tasks. Our results suggest that richer genetic mapping can indeed be obtained from the remote sensing data. In addition, our discovered groupings reveal interesting insights from a plant science perspective.
Multi Level Monte Carlo methods for a class of ergodic stochastic differential equations

We develop a framework that allows the use of the multi-level Monte Carlo (MLMC) methodology (Giles 2015) to calculate expectations with respect to the invariant measures of ergodic SDEs. In that context, we study the (over-damped) Langevin equations with strongly convex potential. We show that, when appropriate contracting couplings for the numerical integrators are available, one can obtain a time-uniform estimates of the MLMC variance in stark contrast to the majority of the results in the MLMC literature. As a consequence, one can approximate expectations with respect to the invariant measure in an unbiased way without the need of a Metropolis- Hastings step. In addition, a root mean square error of $\mathcal{O}(\epsilon)$ is achieved with $\mathcal{O}(\epsilon^{-2})$ complexity on par with Markov Chain Monte Carlo (MCMC) methods, which however can be computationally intensive when applied to large data sets. Finally, we present a multilevel version of the recently introduced Stochastic Gradient Langevin (SGLD) method (Welling and Teh, 2011) built for large datasets applications. We show that this is the first stochastic gradient MCMC method with complexity $\mathcal{O}(\epsilon^{-2}|\log {\epsilon}|^{3})$, which is asymptotically an order $\epsilon$ lower than the $ \mathcal{O}(\epsilon^{-3})$ complexity of all stochastic gradient MCMC methods that are currently available. Numerical experiments confirm our theoretical findings.
Combining Forecasts Using Ensemble Learning

The problem of combining individual forecasters to produce a forecaster with improved performance is considered. The connections between probability elicitation and classification are used to pose the combining forecaster problem as that of ensemble learning. With this connection in place, a number of theoretically sound ensemble learning methods such as Bagging and Boosting are adapted for combining forecasters. It is shown that the simple yet effective method of averaging the forecasts is equivalent to Bagging. This provides theoretical insight into why the well established averaging of forecasts method works so well. Also, a nonlinear combination of forecasters can be attained through Boosting which is shown to theoretically produce combined forecasters that are both calibrated and highly refined. Finally, the proposed methods of combining forecasters are applied to the Good Judgment Project data set and are shown to outperform the individual forecasters.
Model error moment estimation via data assimilation

Using a dynamical model to make predictions about a system has many sources of error. These can include errors in how the model was initialised but also errors in the dynamics of the model itself. For many applications in data assimilation, probabilistic forecasting, or model improvement, these model errors need to be known over the timestep of the model, not over a time-averaged period. Using a forecast from a state that combines observational information as well as prior information we can gain an approximation to the statistics of the model errors on the timescale of the model that is required. Here we give bounds on the errors in the estimation of the mean and covariance of the errors in the model equations in terms of the errors made in the state estimation. This is the first time that such a result has been derived. The result shows to what extent the state estimation must constrain the analysis in order to obtain a specified error on the mean or covariance of the model errors. This is particularly useful for experimental design as it indicates the necessary information content required in observations of the dynamical system.
UNIFORM INFERENCE IN HIGH-DIMENSIONAL DYNAMIC PANEL DATA MODELS WITH APPROXIMATELY SPARSE FIXED EFFECTS

We establish oracle inequalities for a version of the Lasso in high-dimensional fixed effects dynamic panel data models. The inequalities are valid for the coefficients of the dynamic and exogenous regressors. Separate oracle inequalities are derived for the fixed effects. Next, we show how one can conduct uniformly valid inference on the parameters of the model and construct a uniformly valid estimator of the asymptotic covariance matrix which is robust to conditional heteroskedasticity in the error terms. Allowing for conditional heteroskedasticity is important in dynamic models as the conditional error variance may be nonconstant over time and depend on the covariates. Furthermore, our procedure allows for inference on high-dimensional subsets of the parameter vector of an increasing cardinality. We show that the confidence bands resulting from our procedure are asymptotically honest and contract at the optimal rate. This rate is different for the fixed effects than for the remaining parts of the parameter vector.
Laplacian Prior Variational Automatic Relevance Determination for Transmission Tomography

In the classic sparsity-driven problems, the fundamental L-1 penalty method has been shown to have good performance in reconstructing signals for a wide range of problems. However this performance relies on a good choice of penalty weight which is often found from empirical experiments. We propose an algorithm called the Laplacian variational automatic relevance determination (Lap-VARD) that takes this penalty weight as a parameter of a prior Laplace distribution. Optimization of this parameter using an automatic relevance determination framework results in a balance between the sparsity and accuracy of signal reconstruction. Our algorithm is implemented in a transmission tomography model with sparsity constraint in wavelet domain.