Miracle-dz/UniFAR-FTUs huggingface dataset:
Classification of Functional Data with k-Nearest-Neighbor Ensembles by Fitting Constrained Multinomial Logit Models

During the last decades, many methods for the analysis of functional data including classification methods have been developed. Nonetheless, there are issues that have not been adressed satisfactorily by currently available methods, as, for example, feature selection combined with variable selection when using multiple functional covariates. In this paper, a functional ensemble is combined with a penalized and constrained multinomial logit model. It is shown that this synthesis yields a powerful classification tool for functional data (possibly mixed with non-functional predictors), which also provides automatic variable selection. The choice of an appropriate, sparsity-inducing penalty allows to estimate most model coefficients to exactly zero, and permits class-specific coefficients in multiclass problems, such that feature selection is obtained. An additional constraint within the multinomial logit model ensures that the model coefficients can be considered as weights. Thus, the estimation results become interpretable with respect to the discriminative importance of the selected features, which is rated by a feature importance measure. In two application examples, data of a cell chip used for water quality monitoring experiments and phoneme data used for speech recognition, the interpretability as well as the selection results are examined. The classification performance is compared to various other classification approaches which are in common use.
Testing for stationarity of functional time series in the frequency domain

Interest in functional time series has spiked in the recent past with papers covering both methodology and applications being published at a much increased pace. This article contributes to the research in this area by proposing a new stationarity test for functional time series based on frequency domain methods. The proposed test statistics is based on joint dimension reduction via functional principal components analysis across the spectral density operators at all Fourier frequencies, explicitly allowing for frequency-dependent levels of truncation to adapt to the dynamics of the underlying functional time series. The properties of the test are derived both under the null hypothesis of stationary functional time series and under the smooth alternative of locally stationary functional time series. The methodology is theoretically justified through asymptotic results. Evidence from simulation studies and an application to annual temperature curves suggests that the test works well in finite samples.
Inference for Stochastically Contaminated Variable Length Markov Chains

In this paper, we present a methodology to estimate the parameters of stochastically contaminated models under two contamination regimes. In both regimes, we assume that the original process is a variable length Markov chain that is contaminated by a random noise. In the first regime we consider that the random noise is added to the original source and in the second regime, the random noise is multiplied by the original source. Given a contaminated sample of these models, the original process is hidden. Then we propose a two steps estimator for the parameters of these models, that is, the probability transitions and the noise parameter, and prove its consistency. The first step is an adaptation of the Baum-Welch algorithm for Hidden Markov Models. This step provides an estimate of a complete order $k$ Markov chain, where $k$ is bigger than the order of the variable length Markov chain if it has finite order and is a constant depending on the sample size if the hidden process has infinite order. In the second estimation step, we propose a bootstrap Bayesian Information Criterion, given a sample of the Markov chain estimated in the first step, to obtain the variable length time dependence structure associated with the hidden process. We present a simulation study showing that our methodology is able to accurately recover the parameters of the models for a reasonable interval of random noises.
Change‐Point Detection in Autoregressive Models with no Moment Assumptions

In this paper we consider the problem of detecting a change in the parameters of an autoregressive process where the moments of the innovation process do not necessarily exist. An empirical likelihood ratio test for the existence of a change point is proposed and its asymptotic properties are studied. In contrast to other works on change‐point tests using empirical likelihood, we do not assume knowledge of the location of the change point. In particular, we prove that the maximizer of the empirical likelihood is a consistent estimator for the parameters of the autoregressive model in the case of no change point and derive the limiting distribution of the corresponding test statistic under the null hypothesis. We also establish consistency of the new test. A nice feature of the method is the fact that the resulting test is asymptotically distribution‐free and does not require an estimate of the long‐run variance. The asymptotic properties of the test are investigated by means of a small simulation study, which demonstrates good finite‐sample properties of the proposed method.
Monte Carlo Structured SVI for Two-Level Non-Conjugate Models

The stochastic variational inference (SVI) paradigm, which combines variational inference, natural gradients, and stochastic updates, was recently proposed for large-scale data analysis in conjugate Bayesian models and demonstrated to be effective in several problems. This paper studies a family of Bayesian latent variable models with two levels of hidden variables but without any conjugacy requirements, making several contributions in this context. The first is observing that SVI, with an improved structured variational approximation, is applicable under more general conditions than previously thought with the only requirement being that the approximating variational distribution be in the same family as the prior. The resulting approach, Monte Carlo Structured SVI (MC-SSVI), significantly extends the scope of SVI, enabling large-scale learning in non-conjugate models. For models with latent Gaussian variables we propose a hybrid algorithm, using both standard and natural gradients, which is shown to improve stability and convergence. Applications in mixed effects models, sparse Gaussian processes, probabilistic matrix factorization and correlated topic models demonstrate the generality of the approach and the advantages of the proposed algorithms.
Combining Functional Data Registration and Factor Analysis

ABSTRACT We extend the definition of functional data registration to encompass a larger class of registration models. In contrast to traditional registration models, we allow for registered functions that have more than one primary direction of variation. The proposed Bayesian hierarchical model simultaneously registers the observed functions and estimates the two primary factors that characterize variation in the registered functions. Each registered function is assumed to be predominantly composed of a linear combination of these two primary factors, and the function-specific weights for each observation are estimated within the registration model. We show how these estimated weights can easily be used to classify functions after registration using both simulated data and a juggling dataset. Supplementary materials for this article are available online.
Sequential Monte Carlo as approximate sampling: bounds, adaptive resampling via $\infty$-ESS, and an application to particle Gibbs

Sequential Monte Carlo (SMC) algorithms were originally designed for estimating intractable conditional expectations within state-space models, but are now routinely used to generate approximate samples in the context of general-purpose Bayesian inference. In particular, SMC algorithms are often used as subroutines within larger Monte Carlo schemes, and in this context, the demands placed on SMC are different: control of mean-squared error is insufficient---one needs to control the divergence from the target distribution directly. Towards this goal, we introduce the conditional adaptive resampling particle filter, building on the work of Gordon, Salmond, and Smith (1993), Andrieu, Doucet, and Holenstein (2010), and Whiteley, Lee, and Heine (2016). By controlling a novel notion of effective sample size, the $\infty$-ESS, we establish the efficiency of the resulting SMC sampling algorithm, providing an adaptive resampling extension of the work of Andrieu, Lee, and Vihola (2013). We apply our results to arrive at new divergence bounds for SMC samplers with adaptive resampling as well as an adaptive resampling version of the Particle Gibbs algorithm with the same geometric-ergodicity guarantees as its nonadaptive counterpart.
Two Stage Non-penalized Corrected Least Squares for High Dimensional Linear Models with Measurement error or Missing Covariates

This paper provides an alternative to penalized estimators for estimation and vari- able selection in high dimensional linear regression models with measurement error or missing covariates. We propose estimation via bias corrected least squares after model selection. We show that by separating model selection and estimation, it is possible to achieve an improved rate of convergence of the L2 estimation error compared to the rate sqrt{s log p/n} achieved by simultaneous estimation and variable selection methods such as L1 penalized corrected least squares. If the correct model is selected with high probability then the L2 rate of convergence for the proposed method is indeed the oracle rate of sqrt{s/n}. Here s, p are the number of non zero parameters and the model dimension, respectively, and n is the sample size. Under very general model selection criteria, the proposed method is computationally simpler and statistically at least as efficient as the L1 penalized corrected least squares method, performs model selection without the availability of the bias correction matrix, and is able to provide estimates with only a small sub-block of the bias correction covariance matrix of order s x s in comparison to the p x p correction matrix required for computation of the L1 penalized version. Furthermore we show that the model selection requirements are met by a correlation screening type method and the L1 penalized corrected least squares method. Also, the proposed methodology when applied to the estimation of precision matrices with missing observations, is seen to perform at least as well as existing L1 penalty based methods. All results are supported empirically by a simulation study.
Generalized Bouncy Particle Sampler

As a special example of piecewise deterministic Markov process, bouncy particle sampler is a rejection-free, irreversible Markov chain Monte Carlo algorithm and can draw samples from target distribution efficiently. We generalize bouncy particle sampler in terms of its transition dynamics. In BPS, the transition dynamic at event time is deterministic, but in GBPS, it is random. With the help of this randomness, GBPS can overcome the reducibility problem in BPS without refreshement.
On the Bernstein-Von Mises Theorem for High Dimensional Nonlinear Bayesian Inverse Problems

We prove a Bernstein-von Mises theorem for a general class of high dimensional nonlinear Bayesian inverse problems in the vanishing noise limit. We propose a sufficient condition on the growth rate of the number of unknown parameters under which the posterior distribution is asymptotically normal. This growth condition is expressed explicitly in terms of the model dimension, the degree of ill-posedness of the inverse problem and the noise parameter. The theoretical results are applied to a Bayesian estimation of the medium parameter in an elliptic problem.
Noise-Tolerant Interactive Learning from Pairwise Comparisons

We study the problem of interactively learning a binary classifier using noisy labeling and pairwise comparison oracles, where the comparison oracle answers which one in the given two instances is more likely to be positive. Learning from such oracles has multiple applications where obtaining direct labels is harder but pairwise comparisons are easier, and the algorithm can leverage both types of oracles. In this paper, we attempt to characterize how the access to an easier comparison oracle helps in improving the label and total query complexity. We show that the comparison oracle reduces the learning problem to that of learning a threshold function. We then present an algorithm that interactively queries the label and comparison oracles and we characterize its query complexity under Tsybakov and adversarial noise conditions for the comparison and labeling oracles. Our lower bounds show that our label and total query complexity is almost optimal.
Recurrent switching linear dynamical systems

Many natural systems, such as neurons firing in the brain or basketball teams traversing a court, give rise to time series data with complex, nonlinear dynamics. We can gain insight into these systems by decomposing the data into segments that are each explained by simpler dynamic units. Building on switching linear dynamical systems (SLDS), we present a new model class that not only discovers these dynamical units, but also explains how their switching behavior depends on observations or continuous latent states. These "recurrent" switching linear dynamical systems provide further insight by discovering the conditions under which each unit is deployed, something that traditional SLDS models fail to do. We leverage recent algorithmic advances in approximate inference to make Bayesian inference in these models easy, fast, and scalable.
Data-adaptive statistics for multiple hypothesis testing in high-dimensional settings

Current statistical inference problems in areas like astronomy, genomics, and marketing routinely involve the simultaneous testing of thousands -- even millions -- of null hypotheses. For high-dimensional multivariate distributions, these hypotheses may concern a wide range of parameters, with complex and unknown dependence structures among variables. In analyzing such hypothesis testing procedures, gains in efficiency and power can be achieved by performing variable reduction on the set of hypotheses prior to testing. We present in this paper an approach using data-adaptive multiple testing that serves exactly this purpose. This approach applies data mining techniques to screen the full set of covariates on equally sized partitions of the whole sample via cross-validation. This generalized screening procedure is used to create average ranks for covariates, which are then used to generate a reduced (sub)set of hypotheses, from which we compute test statistics that are subsequently subjected to standard multiple testing corrections. The principal advantage of this methodology lies in its providing valid statistical inference without the \textit{a priori} specifying which hypotheses will be tested. Here, we present the theoretical details of this approach, confirm its validity via a simulation study, and exemplify its use by applying it to the analysis of data on microRNA differential expression.
A critical analysis of resampling strategies for the regularized particle filter

We analyze the performance of different resampling strategies for the regularized particle filter regarding parameter estimation. We show in particular, building on analytical insight obtained in the linear Gaussian case, that resampling systematically can prevent the filtered density from converging towards the true posterior distribution. We discuss several means to overcome this limitation, including kernel bandwidth modulation, and provide evidence that the resulting particle filter clearly outperforms traditional bootstrap particle filters. Our results are supported by numerical simulations on a linear textbook example, the logistic map and a non-linear plant growth model.
Projection-based filtering for stochastic reaction networks

This study concerns online inference (i.e., filtering) on the state of reaction networks, conditioned on noisy and partial measurements. The difficulty in deriving the equation that the conditional probability distribution of the state satisfies stems from the fact that the master equation, which governs the evolution of the reaction networks, is analytically intractable. The linear noise approximation (LNA) technique, which is widely used in the analysis of reaction networks, has recently been applied to develop approximate inference. Here, we apply the projection method to derive approximate filters, and compare them to a filter based on the LNA numerically in their filtering performance. We also contrast the projection method with moment-closure techniques in terms of approximating the evolution of stochastic reaction networks.
Survivor-Complier Effects in the Presence of Selection on Treatment, With Application to a Study of Prompt ICU Admission

ABSTRACT Pretreatment selection or censoring (“selection on treatment”) can occur when two treatment levels are compared ignoring the third option of neither treatment, in “censoring by death” settings where treatment is only defined for those who survive long enough to receive it, or in general in studies where the treatment is only defined for a subset of the population. Unfortunately, the standard instrumental variable (IV) estimand is not defined in the presence of such selection, so we consider estimating a new survivor-complier causal effect. Although this effect is generally not identified under standard IV assumptions, it is possible to construct sharp bounds. We derive these bounds and give a corresponding data-driven sensitivity analysis, along with nonparametric yet efficient estimation methods. Importantly, our approach allows for high-dimensional confounding adjustment, and valid inference even after employing machine learning. Incorporating covariates can tighten bounds dramatically, especially when they are strong predictors of the selection process. We apply the methods in a UK cohort study of critical care patients to examine the mortality effects of prompt admission to the intensive care unit, using ICU bed availability as an instrument. Supplementary materials for this article are available online.
Testing degree corrections in stochastic block models

We study sharp detection thresholds for degree corrections in Stochastic Block Models in the context of a goodness of fit problem, and explore the effect of the unknown community assignment (a high dimensional nuisance parameter) and the graph density on testing for degree corrections. When degree corrections are relatively dense, a simple test based on the total number of edges is asymptotically optimal. For sparse degree corrections, the results undergo several changes in behavior depending on density of the underlying Stochastic Block Model. For graphs which are not extremely sparse, optimal tests are based on Higher Criticism or Maximum Degree type tests based on a linear combination of within and across (estimated) community degrees. In the special case of balanced communities, a simple degree based Higher Criticism Test (Mukherjee, Mukherjee, Sen 2016) is optimal in case the graph is not completely dense, while the more complicated linear combination based procedure is required in the completely dense setting. The ``necessity" of the two step procedure is demonstrated for the case of balanced communities by the failure of the ordinary Maximum Degree Test in achieving sharp constants. Finally for extremely sparse graphs the optimal rates change, and a version of the maximum degree test with a different rejection region is shown to be optimal.
Robust sparse covariance estimation by thresholding Tyler’s M-estimator

Estimating a high-dimensional sparse covariance matrix from a limited number of samples is a fundamental problem in contemporary data analysis. Most proposals to date, however, are not robust to outliers or heavy tails. Towards bridging this gap, in this work we consider estimating a sparse shape matrix from $n$ samples following a possibly heavy tailed elliptical distribution. We propose estimators based on thresholding either Tyler's M-estimator or its regularized variant. We derive bounds on the difference in spectral norm between our estimators and the shape matrix in the joint limit as the dimension $p$ and sample size $n$ tend to infinity with $p/n\to\gamma>0$. These bounds are minimax rate-optimal. Results on simulated data support our theoretical analysis.
ICA based on the data asymmetry

Independent Component Analysis (ICA) - one of the basic tools in data analysis - aims to find a coordinate system in which the components of the data are independent. Most of existing methods are based on the minimization of the function of fourth-order moment (kurtosis). Skewness (third-order moment) has received much less attention. In this paper we present a competitive approach to ICA based on the Split Gaussian distribution, which is well adapted to asymmetric data. Consequently, we obtain a method which works better than the classical approaches, especially in the case when the underlying density is not symmetric, which is a typical situation in the color distribution in images.
Profiled deviance for the multivariate linear mixed-effects model fitting

This paper focuses on the multivariate linear mixed-effects model, including all the correlations between the random effects when the marginal residual terms are assumed uncorrelated and homoscedastic with possibly different standard deviations. The random effects covariance matrix is Cholesky factorized to directly estimate the variance components of these random effects. This strategy enables a consistent estimate of the random effects covariance matrix which, generally, has a poor estimate when it is grossly (or directly) estimated, using the estimating methods such as the EM algorithm. By using simulated data sets, we compare the estimates based on the present method with the EM algorithm-based estimates. We provide an illustration by using the real-life data concerning the study of the child's immune against malaria in Benin (West Africa).
The Convergence of Markov Chain Monte Carlo Methods: From the Metropolis Method to Hamiltonian Monte Carlo

From its inception in the 1950s to the modern frontiers of applied statistics, Markov chain Monte Carlo has been one of the most ubiquitous and successful methods in statistical computing. The development of the method in that time has been fueled by not only increasingly difficult problems but also novel techniques adopted from physics. Here, the history of Markov chain Monte Carlo is reviewed from its inception with the Metropolis method to the contemporary state‐of‐the‐art in Hamiltonian Monte Carlo, focusing on the evolving interplay between the statistical and physical perspectives of the method.