Miracle-dz/UniFAR-FTUs huggingface dataset:
JavaNPST: Nonparametric Statistical Tests in Java

Nonparametric statistical tests are useful procedures that can be applied in a wide range of situations, such as testing randomness or goodness of fit, one-sample, two-sample and multiple-sample analysis, association between bivariate samples or count data analysis. Their use is often preferred to parametric tests due to the fact that they require less restrictive assumptions about the population sampled. In this work, JavaNPST, an open source Java library implementing 40 nonparametric statistical tests, is presented. It can be helpful for programmers and practitioners interested in performing nonparametric statistical analyses, providing a quick and easy way of running these tests directly within any Java code. Some examples of use are also shown, highlighting some of the more remarkable capabilities of the library.
Improved detection of changes in species richness in high diversity microbial communities

Biodiversity is important for balance and function of a broad variety of ecosystems, and identifying factors that influence biodiversity can assist environmental management and maintenance. However, low abundance taxa are often missing from ecosystem surveys. These rare taxa, which may be critical to the ecosystem function, are not accounted for in existing methods for detecting changes in species richness. We introduce a model for total (observed and unobserved) biodiversity that explicitly accounts for these rare taxa. Our method permits rigorous testing for both heterogeneity and biodiversity changes, and simultaneously improves type I and II error rates compared with existing methods. To estimate model parameters we utilize the well‐developed literature of meta‐analysis. The problem of substantial low abundance taxa missing from samples is especially pronounced in microbiomes, which are the focus of our case‐studies.
Latent variable model selection for Gaussian conditional random fields

We consider the problem of learning a conditional Gaussian graphical model in the presence of latent variables. Building on recent advances in this field, we suggest a method that decomposes the parameters of a conditional Markov random field into the sum of a sparse and a low-rank matrix. We derive convergence bounds for this estimator and show that it is well-behaved in the high-dimensional regime as well as "sparsistent" (i.e. capable of recovering the graph structure). We then show how proximal gradient algorithms and semi-definite programming techniques can be employed to fit the model to thousands of variables. Through extensive simulations, we illustrate the conditions required for identifiability and show that there is a wide range of situations in which this model performs significantly better than its counterparts, for example, by accommodating more latent variables. Finally, the suggested method is applied to two datasets comprising individual level data on genetic variants and metabolites levels. We show our results replicate better than alternative approaches and show enriched biological signal.
State and Parameter Estimation of Partially Observed Linear Ordinary Differential Equations with Deterministic Optimal Control

Ordinary Differential Equations are a simple but powerful framework for modeling complex systems. Parameter estimation from times series can be done by Nonlinear Least Squares (or other classical approaches), but this can give unsatisfactory results because the inverse problem can be ill-posed, even when the differential equation is linear. Following recent approaches that use approximate solutions of the ODE model, we propose a new method that converts parameter estimation into an optimal control problem: our objective is to determine a control and a parameter that are as close as possible to the data. We derive then a criterion that makes a balance between discrepancy with data and with the model, and we minimize it by using optimization in functions spaces: our approach is related to the so-called Deterministic Kalman Filtering, but dierent from the usual statistical Kalman ltering. We show the root-n consistency and asymptotic normality of the estimators for the parameter and for the states. Experiments in a toy model and in a real case shows that our approach is generally more accurate and more reliable than Nonlinear Least Squares and Generalized Smoothing, even in misspecied cases.
A unit level small area model with misclassified covariates

Model‐based small area estimation relies on mixed effects regression models that link the small areas and borrow strength from similar domains. When the auxiliary variables that are used in the models are measured with error, small area estimators that ignore the measurement error may be worse than direct estimators. Alternative small area estimators accounting for measurement error have been proposed in the literature but only for continuous auxiliary variables. Adopting a Bayesian approach, we extend the unit level model to account for measurement error in both continuous and categorical covariates. For the discrete variables we model the misclassification probabilities and estimate them jointly with all the unknown model parameters. We test our model through a simulation study. The effect of the model proposed is emphasized through application to data from the Ethiopia Demographic and Health Survey where we focus on the women's malnutrition issue: a dramatic problem in developing countries and an important indicator of the socio‐economic progress of a country.
Why (and How) Avoid Orthogonal Procrustes in Regularized Multivariate Analysis

Multivariate Analysis (MVA) comprises a family of well-known methods for feature extraction that exploit correlations among input variables of the data representation. One important property that is enjoyed by most such methods is uncorrelation among the extracted features. Recently, regularized versions of MVA methods have appeared in the literature, mainly with the goal to gain interpretability of the solution. In these cases, the solutions can no longer be obtained in a closed manner, and it is frequent to recur to the iteration of two steps, one of them being an orthogonal Procrustes problem. This letter shows that the Procrustes solution is not optimal from the perspective of the overall MVA method, and proposes an alternative approach based on the solution of an eigenvalue problem. Our method ensures the preservation of several properties of the original methods, most notably the uncorrelation of the extracted features, as demonstrated theoretically and through a collection of selected experiments.
High-dimensional inference in misspecified linear models

We consider high-dimensional inference when the assumed linear model is misspecified. We describe some correct interpretations and corresponding sufficient assumptions for valid asymptotic inference of the model parameters, which still have a useful meaning when the model is misspecified. We largely focus on the de-sparsified Lasso procedure but we also indicate some implications for (multiple) sample splitting techniques. In view of available methods and software, our results contribute to robustness considerations with respect to model misspecification.
Universally consistent vertex classification for latent positions graphs

In this work we show that, using the eigen-decomposition of the adjacency matrix, we can consistently estimate feature maps for latent position graphs with positive definite link function $\kappa$, provided that the latent positions are i.i. d. from some distribution F. We then consider the exploitation task of vertex classification where the link function $\kappa$ belongs to the class of universal kernels and class labels are observed for a number of vertices tending to infinity and that the remaining vertices are to be classified. We show that minimization of the empirical $\varphi$-risk for some convex surrogate $\varphi$ of 0-1 loss over a class of linear classifiers with increasing complexities yields a universally consistent classifier, that is, a classification rule with error converging to Bayes optimal for any distribution F.
Nonparametric Bayesian posterior contraction rates for discretely observed scalar diffusions

We consider nonparametric Bayesian inference in a reflected diffusion model $dX_t = b (X_t)dt + \sigma(X_t) dW_t,$ with discretely sampled observations $X_0, X_\Delta, \dots, X_{n\Delta}$. We analyse the nonlinear inverse problem corresponding to the `low frequency sampling' regime where $\Delta>0$ is fixed and $n \to \infty$. A general theorem is proved that gives conditions for prior distributions $\Pi$ on the diffusion coefficient $\sigma$ and the drift function $b$ that ensure minimax optimal contraction rates of the posterior distribution over H\"older-Sobolev smoothness classes. These conditions are verified for natural examples of nonparametric random wavelet series priors. For the proofs we derive new concentration inequalities for empirical processes arising from discretely observed diffusions that are of independent interest.
Testing hypotheses via a mixture estimation model

We consider a novel paradigm for Bayesian testing of hypotheses and Bayesian model comparison. Our alternative to the traditional construction of posterior probabilities that a given hypothesis is true or that the data originates from a specific model is to consider the models under comparison as components of a mixture model. We therefore replace the original testing problem with an estimation one that focus on the probability weight of a given model within a mixture model. We analyse the sensitivity on the resulting posterior distribution on the weights of various prior modelling on the weights. We stress that a major appeal in using this novel perspective is that generic improper priors are acceptable, while not putting convergence in jeopardy. Among other features, this allows for a resolution of the Lindley-Jeffreys paradox. When using a reference Beta B(a,a) prior on the mixture weights, we note that the sensitivity of the posterior estimations of the weights to the choice of a vanishes with the sample size increasing and advocate the default choice a=0.5, derived from Rousseau and Mengersen (2011). Another feature of this easily implemented alternative to the classical Bayesian solution is that the speeds of convergence of the posterior mean of the weight and of the corresponding posterior probability are quite similar.
Linear quantile hidden Markov models for longitudinal data with informative drop-out

Linear quantile regression models aim at providing a detailed and robust picture of the (conditional) response distribution as function of a set of observed covariates. Longitudinal data represent an interesting field of application of such models; due to their peculiar features, they represent a substantial challenge, in that the standard, cross-sectional, model representation needs to be extended for dealing with such kind of data. In fact, repeated observations from the same statistical unit poses a problem of dependence; in a conditional perspective, this dependence could be ascribed to sources of unobserved, individual-specific, heterogeneity. Along these lines, quantile regression models have recently been extended to the analysis of longitudinal, continuous, responses, by modelling dependence via time-constant or time-varying random effects. In this manuscript, we introduce a general quantile regression model for longitudinal, continuous, responses where time-varying and time-constant random parameters are jointly taken into account. A further feature of longitudinal designs is the presence of partially incomplete sequences, due to some individuals leaving the study before its designed end. The missing data process may produce a selection of units which can be informative with respect to the parameters of the longitudinal data model. To deal with the case of irretrievable drop-out, we introduce a pattern mixture version of the linear quantile hidden Markov model, where we account for time-varying heterogeneity and for changes in the fixed effect vector due to differential propensities to stay in the study. The proposed models are illustrated using a well known benchmark dataset on longitudinal dynamics of CD4 cells and by means of a large scale simulation study, entailing different quantiles and both complete and partially complete (ie subject to drop-out) individual sequences.
Dealing with a large number of classes -- Likelihood, Discrimination or Ranking?

We consider training probabilistic classifiers in the case of a large number of classes. The number of classes is assumed too large to perform exact normalisation over all classes. To account for this we consider a simple approach that directly approximates the likelihood. We show that this simple approach works well on toy problems and is competitive with recently introduced alternative non-likelihood based approximations. Furthermore, we relate this approach to a simple ranking objective. This leads us to suggest a specific setting for the optimal threshold in the ranking objective.
Progress on a conjecture regarding the triangular distribution

ABSTRACT Triangular distributions are a well-known class of distributions that are often used as an elementary example of a probability model. Maximum likelihood estimation of the mode parameter of the triangular distribution over the unit interval can be performed via an order statistic based method. It had been conjectured that such a method can be conducted using only a constant number of likelihood function evaluations, on average, as the sample size becomes large. We prove two theorems that validate this conjecture. Graphical and numerical results are presented to supplement our proofs.
A U-statistic Approach to Hypothesis Testing for Structure Discovery in Undirected Graphical Models

Structure discovery in graphical models is the determination of the topology of a graph that encodes conditional independence properties of the joint distribution of all variables in the model. For some class of probability distributions, an edge between two variables is present if and only if the corresponding entry in the precision matrix is non-zero. For a finite sample estimate of the precision matrix, entries close to zero may be due to low sample effects, or due to an actual association between variables; these two cases are not readily distinguishable. %Fisher provided a hypothesis test based on a parametric approximation to the distribution of an entry in the precision matrix of a Gaussian distribution, but this may not provide valid upper bounds on $p$-values for non-Gaussian distributions. Many related works on this topic consider potentially restrictive distributional or sparsity assumptions that may not apply to a data sample of interest, and direct estimation of the uncertainty of an estimate of the precision matrix for general distributions remains challenging. Consequently, we make use of results for $U$-statistics and apply them to the covariance matrix. By probabilistically bounding the distortion of the covariance matrix, we can apply Weyl's theorem to bound the distortion of the precision matrix, yielding a conservative, but sound test threshold for a much wider class of distributions than considered in previous works. The resulting test enables one to answer with statistical significance whether an edge is present in the graph, and convergence results are known for a wide range of distributions. The computational complexities is linear in the sample size enabling the application of the test to large data samples for which computation time becomes a limiting factor. We experimentally validate the correctness and scalability of the test on multivariate distributions for which the distributional assumptions of competing tests result in underestimates of the false positive ratio. By contrast, the proposed test remains sound, promising to be a useful tool for hypothesis testing for diverse real-world problems.
Reducing estimation bias in adaptively changing monitoring networks with preferential site selection

This paper explores the topic of preferential sampling, specifically situations where monitoring sites in environmental networks are preferentially located by the designers. This means the data arising from such networks may not accurately characterize the spatio-temporal field they intend to monitor. Approaches that have been developed to mitigate the effects of preferential sampling in various contexts are reviewed and, building on these approaches, a general framework for dealing with the effects of preferential sampling in environmental monitoring is proposed. Strategies for implementation are proposed, leading to a method for improving the accuracy of official statistics used to report trends and inform regulatory policy. An essential feature of the method is its capacity to learn the preferential selection process over time and hence to reduce bias in these statistics. Simulation studies suggest dramatic reductions in bias are possible. A case study demonstrates use of the method in assessing the levels of air pollution due to black smoke in the UK over an extended period (1970-1996). In particular, dramatic reductions in the estimates of the number of sites out of compliance are observed.
A discrete bouncy particle sampler

Most Markov chain Monte Carlo methods operate in discrete time and are reversible with respect to the target probability. Nevertheless, it is now understood that the use of nonreversible Markov chains can be beneficial in many contexts. In particular, the recently proposed bouncy particle sampler leverages a continuous-time and nonreversible Markov process, and empirically shows state-of-the-art performance when used to explore certain probability densities; however, its implementation typically requires the computation of local upper bounds on the gradient of the log target density. We present the discrete bouncy particle sampler, a general algorithm based on a guided random walk, a partial refreshment of direction and a delayed-rejection step. We show that the bouncy particle sampler can be understood as a scaling limit of a special case of our algorithm. In contrast to the bouncy particle sampler, implementing the discrete bouncy particle sampler only requires pointwise evaluation of the target density and its gradient. We propose extensions of the basic algorithm for situations when the exact gradient of the target density is not available. In a Gaussian setting, we establish a scaling limit for the radial process as the dimension increases to infinity. We leverage this result to obtain the theoretical efficiency of the discrete bouncy particle sampler as a function of the partial-refreshment parameter, which leads to a simple and robust tuning criterion. A further analysis in a more general setting suggests that this tuning criterion applies more generally. Theoretical and empirical efficiency curves are then compared for different targets and algorithm variations.
New improvements in the use of dependence measures for sensitivity analysis and screening

ABSTRACT Physical phenomena are commonly modelled by time consuming numerical simulators, function of many uncertain parameters whose influences can be measured via a global sensitivity analysis. The usual variance-based indices require too many simulations, especially as the inputs are numerous. To address this limitation, we consider recent advances in dependence measures, focusing on the distance correlation and the Hilbert–Schmidt independence criterion. We study and use these indices for a screening purpose. Numerical tests reveal differences between variance-based indices and dependence measures. Then, two approaches are proposed to use the latter for a screening purpose. The first approach uses independence tests, with existing asymptotic versions and spectral extensions; bootstrap versions are also proposed. The second considers a linear model with dependence measures, coupled to a bootstrap selection method or a Lasso penalization. Numerical experiments show their potential in the presence of many non-influential inputs and give successful results for a nuclear reliability application.
Generalized linear models with low rank effects for network data

Networks are a useful representation for data on connections between units of interests, but the observed connections are often noisy and/or include missing values. One common approach to network analysis is to treat the network as a realization from a random graph model, and estimate the underlying edge probability matrix, which is sometimes referred to as network denoising. Here we propose a generalized linear model with low rank effects to model network edges. This model can be applied to various types of networks, including directed and undirected, binary and weighted, and it can naturally utilize additional information such as node and/or edge covariates. We develop an efficient projected gradient ascent algorithm to fit the model, establish asymptotic consistency, and demonstrate empirical performance of the method on both simulated and real networks.
The likelihood principle does not entail a `sure thing', `evil demon' or `determinist' hypothesis

The likelihood principle makes strong claims about the nature of statistical evidence but is controversial. Its claims are undermined by the existence of several examples that are assumed to show that it allows, with unity probability, domination of all other hypotheses by the uninteresting, determinist hypothesis that whatever happened had to happen. Such examples are generally assumed to be important obstacles to the application of the likelihood principle: they are counter-examples to the principle. A re-analysis of Birnbaum's 1969 `counter-example', demonstrates that the standardly reported analyses of such examples involves an inappropriate treatment of a nuisance parameter and that, when the nuisance parameter is adequately considered, there is no conflict between the evidential consequences of the likelihood principle and the intuitive evidential account of the problem. It also shows that the conclusion that the likelihood principle allows the determinist hypothesis to dominate with unity probability requires a misconception about the scope of the likelihood principle or an inappropriately specified statistical model. Whatever happened did \textit{not} have to happen.
Large scale multi-objective optimization: Theoretical and practical challenges

Multi-objective optimization (MOO) is a well-studied problem for several important recommendation problems. While multiple approaches have been proposed, in this work, we focus on using constrained optimization formulations (e.g., quadratic and linear programs) to formulate and solve MOO problems. This approach can be used to pick desired operating points on the trade-off curve between multiple objectives. It also works well for internet applications which serve large volumes of online traffic, by working with Lagrangian duality formulation to connect dual solutions (computed offline) with the primal solutions (computed online). We identify some key limitations of this approach -- namely the inability to handle user and item level constraints, scalability considerations and variance of dual estimates introduced by sampling processes. We propose solutions for each of the problems and demonstrate how through these solutions we significantly advance the state-of-the-art in this realm. Our proposed methods can exactly handle user and item (and other such local) constraints, achieve a $100\times$ scalability boost over existing packages in R and reduce variance of dual estimates by two orders of magnitude.