Miracle-dz/UniFAR-FTUs huggingface dataset:
Expanding Tidy Data Principles to Facilitate Missing Data Exploration, Visualization and Assessment of Imputations

Despite the large body of research on missing value distributions and imputation, there is comparatively little literature on how to make it easy to handle, explore, and impute missing values in data. This paper addresses this gap. The new methodology builds upon tidy data principles, with a goal to integrating missing value handling as an integral part of data analysis workflows. New data structures are defined along with new functions (verbs) to perform common operations. Together these provide a cohesive framework for handling, exploring, and imputing missing values. These methods have been made available in the R package naniar.
Detection of sparse positive dependence

In a bivariate setting, we consider the problem of detecting a sparse contamination or mixture component, where the effect manifests itself as a positive dependence between the variables, which are otherwise independent in the main component. We first look at this problem in the context of a normal mixture model. In essence, the situation reduces to a univariate setting where the effect is a decrease in variance. In particular, a higher criticism test based on the pairwise differences is shown to achieve the detection boundary defined by the (oracle) likelihood ratio test. We then turn to a Gaussian copula model where the marginal distributions are unknown. Standard invariance considerations lead us to consider rank tests. In fact, a higher criticism test based on the pairwise rank differences achieves the detection boundary in the normal mixture model, although not in the very sparse regime. We do not know of any rank test that has any power in that regime.
A Stochastic Trust Region Method for Non-convex Minimization

We target the problem of finding a local minimum in non-convex finite-sum minimization. Towards this goal, we first prove that the trust region method with inexact gradient and Hessian estimation can achieve a convergence rate of order $\mathcal{O}(1/{k^{2/3}})$ as long as those differential estimations are sufficiently accurate. Combining such result with a novel Hessian estimator, we propose the sample-efficient stochastic trust region (STR) algorithm which finds an $(\epsilon, \sqrt{\epsilon})$-approximate local minimum within $\mathcal{O}({\sqrt{n}}/{\epsilon^{1.5}})$ stochastic Hessian oracle queries. This improves state-of-the-art result by $\mathcal{O}(n^{1/6})$. Experiments verify theoretical conclusions and the efficiency of STR.
Implicit Manifold Learning on Generative Adversarial Networks

This paper raises an implicit manifold learning perspective in Generative Adversarial Networks (GANs), by studying how the support of the learned distribution, modelled as a submanifold $\mathcal{M}_{\theta}$, perfectly match with $\mathcal{M}_{r}$, the support of the real data distribution. We show that optimizing Jensen-Shannon divergence forces $\mathcal{M}_{\theta}$ to perfectly match with $\mathcal{M}_{r}$, while optimizing Wasserstein distance does not. On the other hand, by comparing the gradients of the Jensen-Shannon divergence and the Wasserstein distances ($W_1$ and $W_2^2$) in their primal forms, we conjecture that Wasserstein $W_2^2$ may enjoy desirable properties such as reduced mode collapse. It is therefore interesting to design new distances that inherit the best from both distances.
Improved Algorithms for Matrix Recovery from Rank-One Projections

We consider the problem of estimation of a low-rank matrix from a limited number of noisy rank-one projections. In particular, we propose two fast, non-convex \emph{proper} algorithms for matrix recovery and support them with rigorous theoretical analysis. We show that the proposed algorithms enjoy linear convergence and that their sample complexity is independent of the condition number of the unknown true low-rank matrix. By leveraging recent advances in low-rank matrix approximation techniques, we show that our algorithms achieve computational speed-ups over existing methods. Finally, we complement our theory with some numerical experiments.
Uniform Consistency in Stochastic Block Model with Continuous Community Label

\cite{bickel2009nonparametric} developed a general framework to establish consistency of community detection in stochastic block model (SBM). In most applications of this framework, the community label is discrete. For example, in \citep{bickel2009nonparametric,zhao2012consistency} the degree corrected SBM is assumed to have a discrete degree parameter. In this paper, we generalize the method of \cite{bickel2009nonparametric} to give consistency analysis of maximum likelihood estimator (MLE) in SBM with continuous community label. We show that there is a standard procedure to transform the $||\cdot||_2$ error bound to the uniform error bound. We demonstrate the application of our general results by proving the uniform consistency (strong consistency) of the MLE in the exponential network model with interaction effect. Unfortunately, in the continuous parameter case, the condition ensuring uniform consistency we obtained is much stronger than that in the discrete parameter case, namely $n\mu_n^5/(\log n)^{8}\rightarrow\infty$ versus $n\mu_n/\log n\rightarrow\infty$. Where $n\mu_n$ represents the average degree of the network. But continuous is the limit of discrete. So it is not surprising as we show that by discretizing the community label space into sufficiently small (but not too small) pieces and applying the MLE on the discretized community label space, uniform consistency holds under almost the same condition as in discrete community label space. Such a phenomenon is surprising since the discretization does not depend on the data or the model. This reminds us of the thresholding method.
A Statistical Model with Qualitative Input

A statistical estimation model with qualitative input provides a mechanism to fuse human intuition in a form of qualitative information into a quantitative statistical model. We investigate statistical properties and devise a numerical computation method for a model subclass with a uniform correlation structure. We show that, within this subclass, qualitative information can be as objective as quantitative information. We also show that the correlation between each pair of variables controls the accuracy of the statistical estimate. An application to portfolio selection is discussed. The correlation, although compromising the accuracy of the statistical estimation, affects the performance of the portfolio in a minimal way.
Semi-parametric Bayesian change-point model based on the Dirichlet process

In this work we introduce a semi-parametric Bayesian change-point model, defining its time dynamic as a latent Markov process based on the Dirichlet process. We treat the number of change point as a random variable and we estimate it during model fitting. Posterior inference is carried out using a Markov chain Monte Carlo algorithm based on a marginalized version of the proposed model. The model is illustrated using simulated examples and two real datasets, namely the coal- mining disasters, that is a widely used dataset for illustrative purpose, and a dataset of indoor radon recordings. With the simulated examples we show that the model is able to recover the parameters and number of change points, and we compare our results with the ones of the-state- of-the-art models, showing a clear improvement in terms of change points identification. The results obtained on the coal-mining disasters and radon data are coherent with previous literature.
The Sparse Latent Position Model for nonnegative weighted networks

This paper introduces a new methodology to analyse bipartite and unipartite networks with nonnegative edge values. The proposed approach combines and adapts a number of ideas from the literature on latent variable network models. The resulting framework is a new type of latent position model which exhibits great flexibility, and is able to capture important features that are generally exhibited by observed networks, such as sparsity and heavy tailed degree distributions. A crucial advantage of the proposed method is that the number of latent dimensions is automatically deduced from the data in one single algorithmic framework. In addition, the model attaches a weight to each of the latent dimensions, hence providing a measure of their relative importance. A fast variational Bayesian algorithm is proposed to estimate the parameters of the model. Finally, applications of the proposed methodology are illustrated on both artificial and real datasets, and comparisons with other existing procedures are provided.
Partial least squares discriminant analysis: A dimensionality reduction method to classify hyperspectral data

The recent development of more sophisticated spectroscopic methods allows acqui- sition of high dimensional datasets from which valuable information may be extracted using multivariate statistical analyses, such as dimensionality reduction and automatic classification (supervised and unsupervised). In this work, a supervised classification through a partial least squares discriminant analysis (PLS-DA) is performed on the hy- perspectral data. The obtained results are compared with those obtained by the most commonly used classification approaches.
Forecasting Based on Surveillance Data

Forecasting the future course of epidemics has always been one of the main goals of epidemic modelling. This chapter reviews statistical methods to quantify the accuracy of epidemic forecasts. We distinguish point and probabilistic forecasts and describe different methods to evaluate and compare the predictive performance across models. Two case studies demonstrate how to apply the different techniques to uni- and multivariate forecasts. We focus on forecasting count time series from routine public health surveillance: weekly counts of influenza-like illness in Switzerland, and age-stratified counts of norovirus gastroenteritis in Berlin, Germany. Data and code for all analyses are available in a supplementary R package.
MultiBUGS: A Parallel Implementation of the BUGS Modeling Framework for Faster Bayesian Inference

MultiBUGS is a new version of the general-purpose Bayesian modelling software BUGS that implements a generic algorithm for parallelising Markov chain Monte Carlo (MCMC) algorithms to speed up posterior inference of Bayesian models. The algorithm parallelises evaluation of the product-form likelihoods formed when a parameter has many children in the directed acyclic graph (DAG) representation; and parallelises sampling of conditionally-independent sets of parameters. A heuristic algorithm is used to decide which approach to use for each parameter and to apportion computation across computational cores. This enables MultiBUGS to automatically parallelise the broad range of statistical models that can be fitted using BUGS-language software, making the dramatic speed-ups of modern multi-core computing accessible to applied statisticians, without requiring any experience of parallel programming. We demonstrate the use of MultiBUGS on simulated data designed to mimic a hierarchical e-health linked-data study of methadone prescriptions including 425,112 observations and 20,426 random effects. Posterior inference for the e-health model takes several hours in existing software, but MultiBUGS can perform inference in only 28 minutes using 48 computational cores.
Pragmatic Hypotheses in the Evolution of Science

This paper introduces pragmatic hypotheses and relates this concept to the spiral of scientific evolution. Previous works determined a characterization of logically consistent statistical hypothesis tests and showed that the modal operators obtained from this test can be represented in the hexagon of oppositions. However, despite the importance of precise hypothesis in science, they cannot be accepted by logically consistent tests. Here, we show that this dilemma can be overcome by the use of pragmatic versions of precise hypotheses. These pragmatic versions allow a level of imprecision in the hypothesis that is small relative to other experimental conditions. The introduction of pragmatic hypotheses allows the evolution of scientific theories based on statistical hypothesis testing to be interpreted using the narratological structure of hexagonal spirals, as defined by Pierre Gallais.
Nonparametric Bayesian label prediction on a large graph using truncated Laplacian regularization

Abstract This article describes an implementation of a nonparametric Bayesian approach to solving binary classification problems on graphs. We consider a hierarchical Bayesian approach with a prior that is constructed by truncating a series expansion of the soft label function using the graph Laplacian eigenfunctions as basisfunctions. We compare our truncated prior to the untruncated Laplacian based prior in simulated and real data examples to illustrate the improved scalability in terms of the size of the underlying graph.
A distributed regression analysis application based on SAS software Part II: Cox proportional hazards regression

Previous work has demonstrated the feasibility and value of conducting distributed regression analysis (DRA), a privacy-protecting analytic method that performs multivariable-adjusted regression analysis with only summary-level information from participating sites. To our knowledge, there are no DRA applications in SAS, the statistical software used by several large national distributed data networks (DDNs), including the Sentinel System and PCORnet. SAS/IML is available to perform the required matrix computations for DRA in the SAS system. However, not all data partners in these large DDNs have access to SAS/IML, which is licensed separately. In this second article of a two-paper series, we describe a DRA application developed using Base SAS and SAS/STAT modules for distributed Cox proportional hazards regression within horizontally partitioned DDNs and its successful tests.
Global Consensus Monte Carlo

Abstract To conduct Bayesian inference with large datasets, it is often convenient or necessary to distribute the data across multiple machines. We consider a likelihood function expressed as a product of terms, each associated with a subset of the data. Inspired by global variable consensus optimization, we introduce an instrumental hierarchical model associating auxiliary statistical parameters with each term, which are conditionally independent given the top-level parameters. One of these top-level parameters controls the unconditional strength of association between the auxiliary parameters. This model leads to a distributed MCMC algorithm on an extended state space yielding approximations of posterior expectations. A trade-off between computational tractability and fidelity to the original model can be controlled by changing the association strength in the instrumental model. We further propose the use of an SMC sampler with a sequence of association strengths, allowing both the automatic determination of appropriate strengths and for a bias correction technique to be applied. In contrast to similar distributed Monte Carlo algorithms, this approach requires few distributional assumptions. The performance of the algorithms is illustrated with a number of simulated examples. Supplementary materials for this article are available online.
Staging of T1 esophageal adenocarcinoma with volumetric laser endomicroscopy: a feasibility study

Abstract Background and study aims Precise staging in T1 esophageal adenocarcinoma (EAC) is critical in determining candidacy for curative endoscopic resection. High-frequency endoscopic ultrasound (EUS) has demonstrated suboptimal accuracy in T1 EAC staging due to insufficient spatial resolution. Volumetric laser endomicroscopy (VLE) allows for high-resolution wide-field visualization of the esophageal microstructure. We aimed to investigate the role of VLE in staging T1 EAC. Patients and methods Patients undergoing endoscopic mucosal resection (EMR) were prospectively enrolled and only T1 EAC cases were included. EMR specimens were imaged using second-generation VLE immediately after resection. VLE images were analyzed for signal intensity by depth and signal attenuation (dB/mm) in both cross-sectional and en-face orientation. A decision tree model was constructed to combine measured VLE parameters and delineate diagnostic thresholds. Results Thirty EMR scans were obtained – 15 T1a specimens from 9 patients and 15 T1b specimens from 11 patients. T1b specimen VLE scans exhibited higher signal intensity (P < 0.0001) and higher signal attenuation compared to T1a specimens (P = 0.03). A combination of signal attenuation and signal intensity at 150 µm depth yielded optimal diagnostic thresholds and an area under the curve (AUC) of 0.77. VLE signal attenuation was significantly associated with grade of differentiation, irrespective of EAC stage. Conclusions VLE signal intensity and signal attenuation are quantitatively distinct in T1a and T1b EAC and associated with grade of differentiation. This is the first study examining the role of VLE for staging of T1 EAC and demonstrates promising diagnostic performance. With further in vivo validation, VLE may serve a role in staging superficial EAC.
Experimental Design Issues in Big Data. The Question of Bias

Data can be collected in scientific studies via a controlled experiment or passive observation. Big data is often collected in a passive way, e.g. from social media. In studies of causation great efforts are made to guard against bias and hidden confounders or feedback which can destroy the identification of causation by corrupting or omitting counterfactuals (controls). Various solutions of these problems are discussed, including randomization.
Adaptive Low-Nonnegative-Rank Approximation for State Aggregation of Markov Chains

This paper develops a low-nonnegative-rank approximation method to identify the state aggregation structure of a finite-state Markov chain under an assumption that the state space can be mapped into a handful of meta-states. The number of meta-states is characterized by the nonnegative rank of the Markov transition matrix. Motivated by the success of the nuclear norm relaxation in low rank minimization problems, we propose an atomic regularizer as a convex surrogate for the nonnegative rank and formulate a convex optimization problem. Because the atomic regularizer itself is not computationally tractable, we instead solve a sequence of problems involving a nonnegative factorization of the Markov transition matrices by using the proximal alternating linearized minimization method. Two methods for adjusting the rank of factorization are developed so that local minima are escaped. One is to append an additional column to the factorized matrices, which can be interpreted as an approximation of a negative subgradient step. The other is to reduce redundant dimensions by means of linear combinations. Overall, the proposed algorithm very likely converges to the global solution. The efficiency and statistical properties of our approach are illustrated on synthetic data. We also apply our state aggregation algorithm on a Manhattan transportation data set and make extensive comparisons with an existing method.
An\'alisis estad\'istico ex post del conteo r\'apido institucional de la elecci\'on de gobernador del Estado de M\'exico en 2017

Se realiza un análisis estad́ıstico de las estimaciones del conteo rápido institucional desde la perspectiva ideal de los resultados de los cómputos distritales de la elección de gobernador del Estado de México del año 2017, particularmente aspectos como la precisión de las estimaciones, el nivel de confianza de los intervalos, el posible sesgo respecto al cómputo distrital y las conclusiones que se derivaron y reportaron, con el objetivo de determinar el grado de cumplimiento de los objetivos de este ejercicio estad́ıstico de carácter informativo.
Approximate Bayesian Computation and Model Assessment for Repulsive Spatial Point Processes

ABSTRACT In many applications involving spatial point patterns, we find evidence of inhibition or repulsion. The most commonly used class of models for such settings are the Gibbs point processes. A recent alternative, at least to the statistical community, is the determinantal point process. Here, we examine model fitting and inference for both of these classes of processes in a Bayesian framework. While usual MCMC model fitting can be available, the algorithms are complex and are not always well behaved. We propose using approximate Bayesian computation (ABC) for such fitting. This approach becomes attractive because, though likelihoods are very challenging to work with for these processes, generation of realizations given parameter values is relatively straightforward. As a result, the ABC fitting approach is well-suited for these models. In addition, such simulation makes them well-suited for posterior predictive inference as well as for model assessment. We provide details for all of the above along with some simulation investigation and an illustrative analysis of a point pattern of tree data exhibiting repulsion. R code and datasets are included in the supplementary material.
Stable Approximation Schemes for Optimal Filters

We explore a general truncation scheme for the approximation of (possibly unstable) optimal filters. In particular, let $\mS = (\pi_0,\kappa_t,g_t)$ be a state space model defined by a prior distribution $\pi_0$, Markov kernels $\{\kappa_t\}_{t\ge 1}$ and potential functions $\{g_t\}_{t \ge 1}$, and let $\sfc = \{C_t\}_{t\ge 1}$ be a sequence of compact subsets of the state space. In the first part of the manuscript, we describe a systematic procedure to construct a system $\mS^\sfc=(\pi_0,\kappa_t^\sfc,g_t^\sfc)$, where each potential $g_t^\sfc$ is truncated to have null value outside the set $C_t$, such that the optimal filters generated by $S$ and $S^\sfc$ can be made arbitrarily close, with approximation errors independent of time $t$. Then, in a second part, we investigate the stability of the approximately-optimal filters. Specifically, given a system $\mS$ with a prescribed prior $\pi_0$, we seek sufficient conditions to guarantee that the truncated system $\mS^\sfc$ (with {\em the same} prior $\pi_0$) generates a sequence of optimal filters which are stable and, at the same time, can attain arbitrarily small approximation errors. Besides the design of approximate filters, the methods and results obtained in this paper can be applied to determine whether a prescribed system $\mS$ yields a sequence of stable filters and to investigate topological properties of classes of optimal filters. As an example of the latter, we explicitly construct a metric space $(\mfS,D_q)$, where $\mfS$ is a class of state space systems and $D_q$ is a proper metric on $\mfS$, which contains a dense subset $\mfS_0 \subset \mfS$ such that every element $\mS_0 \in \mfS_0$ is a state space system yielding a stable sequence of optimal filters.