Miracle-dz/UniFAR-FTUs huggingface dataset:
Feature vector regularization in machine learning

Problems in machine learning (ML) can involve noisy input data, and ML classification methods have reached limiting accuracies when based on standard ML data sets consisting of feature vectors and their classes. Greater accuracy will require incorporation of prior structural information on data into learning. We study methods to regularize feature vectors (unsupervised regularization methods), analogous to supervised regularization for estimating functions in ML. We study regularization (denoising) of ML feature vectors using Tikhonov and other regularization methods for functions on ${\bf R}^n$. A feature vector ${\bf x}=(x_1,\ldots,x_n)=\{x_q\}_{q=1}^n$ is viewed as a function of its index $q$, and smoothed using prior information on its structure. This can involve a penalty functional on feature vectors analogous to those in statistical learning, or use of proximity (e.g. graph) structure on the set of indices. Such feature vector regularization inherits a property from function denoising on ${\bf R}^n$, in that accuracy is non-monotonic in the denoising (regularization) parameter $\alpha$. Under some assumptions about the noise level and the data structure, we show that the best reconstruction accuracy also occurs at a finite positive $\alpha$ in index spaces with graph structures. We adapt two standard function denoising methods used on ${\bf R}^n$, local averaging and kernel regression. In general the index space can be any discrete set with a notion of proximity, e.g. a metric space, a subset of ${\bf R}^n$, or a graph/network, with feature vectors as functions with some notion of continuity. We show this improves feature vector recovery, and thus the subsequent classification or regression done on them. We give an example in gene expression analysis for cancer classification with the genome as an index space and network structure based protein-protein interactions.
Non-linear dimensionality reduction: Riemannian metric estimation and the problem of geometric discovery

In recent years, manifold learning has become increasingly popular as a tool for performing non-linear dimensionality reduction. This has led to the development of numerous algorithms of varying degrees of complexity that aim to recover man ifold geometry using either local or global features of the data. Building on the Laplacian Eigenmap and Diffusionmaps framework, we propose a new paradigm that offers a guarantee, under reasonable assumptions, that any manifo ld learning algorithm will preserve the geometry of a data set. Our approach is based on augmenting the output of embedding algorithms with geometric informatio n embodied in the Riemannian metric of the manifold. We provide an algorithm for estimating the Riemannian metric from data and demonstrate possible application s of our approach in a variety of examples.
A likelihood ratio test for monotone baseline hazard functions in the Cox model

We consider a likelihood ratio method for testing whether a monotone baseline hazard function in the Cox model has a particular value at a fixed point. The characterization of the estimators involved is provided both in the nondecreasing and the nonincreasing setting. These characterizations facilitate the derivation of the asymptotic distribution of the likelihood ratio test, which is identical in the nondecreasing and in the nonincreasing case. The asymptotic distribution of the likelihood ratio test enables, via inversion, the construction of pointwise confidence intervals. Simulations show that these confidence intervals exhibit comparable coverage probabilities with the confidence intervals based on the asymptotic distribution of the nonparametric maximum likelihood estimator of a monotone baseline hazard function.
A Review of Kernel Density Estimation with Applications to Econometrics

Nonparametric density estimation is of great importance when econometricians want to model the probabilistic or stochastic structure of a data set. This comprehensive review summarizes the most important theoretical aspects of kernel density estimation and provides an extensive description of classical and modern data analytic methods to compute the smoothing parameter. Throughout the text, several references can be found to the most up-to-date and cut point research approaches in this area, while econometric data sets are analyzed as examples. Lastly, we present SIZer, a new approach introduced by Chaudhuri and Marron (2000), whose objective is to analyze the visible features representing important underlying structures for different bandwidths.
When uniform weak convergence fails: empirical processes for dependence functions via epi- and hypographs

For copulas whose partial derivatives are not continuous everywhere on the interior of the unit cube, the empirical copula process does not converge weakly with respect to the supremum distance. This makes it hard to verify asymptotic properties of inference procedures for such copulas. To resolve the issue, a new metric for locally bounded functions is introduced and the corresponding weak convergence theory is developed. Convergence with respect to the new metric is related to epi- and hypoconvergence and is weaker than uniform convergence. Still, for continuous limits, it is equivalent to locally uniform convergence, whereas under mild side conditions, it implies Lp convergence. Even in cases where uniform convergence fails, weak convergence with respect to the new metric is established for empirical copula and tail dependence processes. No additional assumptions are needed for tail dependence functions, and for copulas, the assumptions reduce to existence and continuity of the partial derivatives almost everywhere on the unit cube. The results are applied to obtain asymptotic properties of minimum distance estimators, goodness-of-fit tests and resampling procedures.
Tracy-Widom Distribution for the Largest Eigenvalue of Real Sample Covariance Matrices with General Population

We consider sample covariance matrices of the form $\mathcal{Q}=(\Sigma^{1/2}X)(\Sigma^{1/2} X)^*$, where the sample $X$ is an $M\times N$ random matrix whose entries are real independent random variables with variance $1/N$ and where $\Sigma$ is an $M\times M$ positive-definite deterministic matrix. We analyze the asymptotic fluctuations of the largest rescaled eigenvalue of $\mathcal{Q}$ when both $M$ and $N$ tend to infinity with $N/M\to d\in(0,\infty)$. For a large class of populations $\Sigma$ in the sub-critical regime, we show that the distribution of the largest rescaled eigenvalue of $\mathcal{Q}$ is given by the type-1 Tracy-Widom distribution under the additional assumptions that (1) either the entries of $X$ are i.i. d. Gaussians or (2) that $\Sigma$ is diagonal and that the entries of $X$ have a subexponential decay.
Joint asymptotics for semi-nonparametric regression models with partially linear structure

We consider a joint asymptotic framework for studying semi-nonparametric regression models where (finite-dimensional) Euclidean parameters and (infinite-dimensional) functional parameters are both of interest. The class of models in consideration share a partially linear structure and are estimated in two general contexts: (i) quasi-likelihood and (ii) true likelihood. We first show that the Euclidean estimator and (pointwise) functional estimator, which are re-scaled at different rates, jointly converge to a zero-mean Gaussian vector. This weak convergence result reveals a surprising joint asymptotics phenomenon: these two estimators are asymptotically independent. A major goal of this paper is to gain first-hand insights into the above phenomenon. Moreover, a likelihood ratio testing is proposed for a set of joint local hypotheses, where a new version of the Wilks phenomenon [Ann. Math. Stat. 9 (1938) 60-62; Ann. Statist. 1 (2001) 153-193] is unveiled. A novel technical tool, called a joint Bahadur representation, is developed for studying these joint asymptotics results.
Optimal Kernel Combination for Test of Independence against Local Alternatives

Testing the independence between two random variables $x$ and $y$ is an important problem in statistics and machine learning, where the kernel-based tests of independence is focused to address the study of dependence recently. The advantage of the kernel framework rests on its flexibility in choice of kernel. The Hilbert-Schmidt Independence Criterion (HSIC) was shown to be equivalent to a class of tests, where the tests are based on different distance-induced kernel pairs. In this work, we propose to select the optimal kernel pair by considering local alternatives, and evaluate the efficiency using the quadratic time estimator of HSIC. The local alternative offers the advantage that the measure of efficiency do not depend on a particular alternative, and only requires the knowledge of the asymptotic null distribution of the test. We show in our experiments that the proposed strategy results in higher power than other existing kernel selection approaches.
Hypotheses tests in boundary regression models

We consider a nonparametric regression model with one-sided errors and regression function in a general H\"older class. Our aim is inference on the error distribution. To this end we estimate the regression function via minimization of the local integral of a polynomial approximation. We show uniform rates of convergence for the simple regression estimator as well as for a smooth version. Those rates continue to hold in mean regression models with symmetric and bounded error distribution. Here one obtains faster rates of convergence compared to optimal rates in mean regression when the error distribution is irregular in the sense that sufficient mass is concentrated near the endpoints. The results are applied to prove asymptotic $\sqrt{n}$-equivalence of a residual-based empirical distribution function to the empirical distribution function of unobserved errors in the case of irregular error distributions. This result is remarkably different from corresponding results in mean regression with regular errors. It can readily be applied to develop goodness-of-fit tests for the error distribution. We present some examples and investigate the small sample performance in a simulation study.
A new representation for multivariate tail probabilities

Existing theory for multivariate extreme values focuses upon characterizations of the distributional tails when all components of a random vector, standardized to identical margins, grow at the same rate. In this paper, we consider the effect of allowing the components to grow at different rates, and characterize the link between these marginal growth rates and the multivariate tail probability decay rate. Our approach leads to a whole class of univariate regular variation conditions, in place of the single but multivariate regular variation conditions that underpin the current theories. These conditions are indexed by a homogeneous function and an angular dependence function, which, for asymptotically independent random vectors, mirror the role played by the exponent measure and Pickands' dependence function in classical multivariate extremes. We additionally offer an inferential approach to joint survivor probability estimation. The key feature of our methodology is that extreme set probabilities can be estimated by extrapolating upon rays emanating from the origin when the margins of the variables are exponential. This offers an appreciable improvement over existing techniques where extrapolation in exponential margins is upon lines parallel to the diagonal.
Ensembling classification models based on phalanxes of variables with applications in drug discovery

Statistical detection of a rare class of objects in a two-class classification problem can pose several challenges. Because the class of interest is rare in the training data, there is relatively little information in the known class response labels for model building. At the same time the available explanatory variables are often moderately high dimensional. In the four assays of our drug-discovery application, compounds are active or not against a specific biological target, such as lung cancer tumor cells, and active compounds are rare. Several sets of chemical descriptor variables from computational chemistry are available to classify the active versus inactive class; each can have up to thousands of variables characterizing molecular structure of the compounds. The statistical challenge is to make use of the richness of the explanatory variables in the presence of scant response information. Our algorithm divides the explanatory variables into subsets adaptively and passes each subset to a base classifier. The various base classifiers are then ensembled to produce one model to rank new objects by their estimated probabilities of belonging to the rare class of interest. The essence of the algorithm is to choose the subsets such that variables in the same group work well together; we call such groups phalanxes.
High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification

We provide a unified analysis of the predictive risk of ridge regression and regularized discriminant analysis in a dense random effects model. We work in a high-dimensional asymptotic regime where $p, n \to \infty$ and $p/n \to \gamma \in (0, \, \infty)$, and allow for arbitrary covariance among the features. For both methods, we provide an explicit and efficiently computable expression for the limiting predictive risk, which depends only on the spectrum of the feature-covariance matrix, the signal strength, and the aspect ratio $\gamma$. Especially in the case of regularized discriminant analysis, we find that predictive accuracy has a nuanced dependence on the eigenvalue distribution of the covariance matrix, suggesting that analyses based on the operator norm of the covariance matrix may not be sharp. Our results also uncover several qualitative insights about both methods: for example, with ridge regression, there is an exact inverse relation between the limiting predictive risk and the limiting estimation risk given a fixed signal strength. Our analysis builds on recent advances in random matrix theory.
Modern Statistical Methods in Oceanography: A Hierarchical Perspective

Processes in ocean physics, air-sea interaction and ocean biogeochemistry span enormous ranges in spatial and temporal scales, that is, from molecular to planetary and from seconds to millennia. Identifying and implementing sustainable human practices depend critically on our understandings of key aspects of ocean physics and ecology within these scale ranges. The set of all ocean data is distorted such that three- and four-dimensional (i.e., time-dependent) in situ data are very sparse, while observations of surface and upper ocean properties from space-borne platforms have become abundant in the past few decades. Precisions in observations of all types vary as well. In the face of these challenges, the interface between Statistics and Oceanography has proven to be a fruitful area for research and the development of useful models. With the recognition of the key importance of identifying, quantifying and managing uncertainty in data and models of ocean processes, a hierarchical perspective has become increasingly productive. As examples, we review a heterogeneous mix of studies from our own work demonstrating Bayesian hierarchical model applications in ocean physics, air-sea interaction, ocean forecasting and ocean ecosystem models. This review is by no means exhaustive and we have endeavored to identify hierarchical modeling work reported by others across the broad range of ocean-related topics reported in the statistical literature. We conclude by noting relevant ocean-statistics problems on the immediate research horizon, and some technical challenges they pose, for example, in terms of nonlinearity, dimensionality and computing.
Generalized Gaussian Process Regression Model for Non-Gaussian Functional Data

In this article, we propose a generalized Gaussian process concurrent regression model for functional data, where the functional response variable has a binomial, Poisson, or other non-Gaussian distribution from an exponential family, while the covariates are mixed functional and scalar variables. The proposed model offers a nonparametric generalized concurrent regression method for functional data with multidimensional covariates, and provides a natural framework on modeling common mean structure and covariance structure simultaneously for repeatedly observed functional data. The mean structure provides overall information about the observations, while the covariance structure can be used to catch up the characteristic of each individual batch. The prior specification of covariance kernel enables us to accommodate a wide class of nonlinear models. The definition of the model, the inference, and the implementation as well as its asymptotic properties are discussed. Several numerical examples with different non-Gaussian response variables are presented. Some technical details and more numerical examples as well as an extension of the model are provided as supplementary materials.
A Generic Sample Splitting Approach for Refined Community Recovery in Stochastic Block Models

We propose and analyze a generic method for community recovery in stochastic block models and degree corrected block models. This approach can exactly recover the hidden communities with high probability when the expected node degrees are of order $\log n$ or higher. Starting from a roughly correct community partition given by some conventional community recovery algorithm, this method refines the partition in a cross clustering step. Our results simplify and extend some of the previous work on exact community recovery, discovering the key role played by sample splitting. The proposed method is simple and can be implemented with many practical community recovery algorithms.
Adaptive Estimation in Two-way Sparse Reduced-rank Regression

This paper studies the problem of estimating a large coefficient matrix in a multiple response linear regression model when the coefficient matrix could be both of low rank and sparse in the sense that most nonzero entries concentrate on a few rows and columns. We are especially interested in the high dimensional settings where the number of predictors and/or response variables can be much larger than the number of observations. We propose a new estimation scheme, which achieves competitive numerical performance and at the same time allows fast computation. Moreover, we show that (a slight variant of) the proposed estimator achieves near optimal non-asymptotic minimax rates of estimation under a collection of squared Schatten norm losses simultaneously by providing both the error bounds for the estimator and minimax lower bounds. The effectiveness of the proposed algorithm is also demonstrated on an \textit{in vivo} calcium imaging dataset.
A Statistical Approach to Crime Linkage

ABSTRACT The object of this article is to develop a statistical approach to criminal linkage analysis that discovers and groups crime events that share a common offender and prioritizes suspects for further investigation. Bayes factors are used to describe the strength of evidence that two crimes are linked. Using concepts from agglomerative hierarchical clustering, the Bayes factors for crime pairs are combined to provide similarity measures for comparing two crime series. This facilitates crime series clustering, crime series identification, and suspect prioritization. The ability of our models to make correct linkages and predictions is demonstrated under a variety of real-world scenarios with a large number of solved and unsolved breaking and entering crimes. For example, a naive Bayes model for pairwise case linkage can identify 82% of actual linkages with a 5% false positive rate. For crime series identification, 74%–89% of the additional crimes in a crime series can be identified from a ranked list of 50 incidents.
Fast Tree Inference With Weighted Fusion Penalties

ABSTRACT Given a dataset with many features observed in a large number of conditions, it is desirable to fuse and aggregate conditions that are similar to ease the interpretation and extract the main characteristics of the data. This article presents a multidimensional fusion penalty framework to address this question when the number of conditions are large. If the fusion penalty is encoded by an ℓq-norm, we prove for uniform weights that the path of solutions is a tree that is suitable for interpretability. For the ℓ1 and ℓ∞-norms, the path is piecewise linear and we derive a homotopy algorithm to recover exactly the whole tree structure. For weighted ℓ1-fusion penalties, we demonstrate that distance-decreasing weights lead to balanced tree structures. For a subclass of these weights that we call “exponentially adaptive,” we derive an homotopy algorithm and we prove an asymptotic oracle property. This guarantees that we recover the underlying structure of the data efficiently both from a statistical and a computational point of view. We provide a fast implementation of the homotopy algorithm for the single feature case, as well as an efficient embedded cross-validation procedure that takes advantage of the tree structure of the path of solutions. Our proposal outperforms its competing procedures on simulations both in terms of timings and prediction accuracy. As an example we consider phenotypic data: given one or several traits, we reconstruct a balanced tree structure and assess its agreement with the known taxonomy. Supplementary materials for this article are available online.
Regression shrinkage and grouping of highly correlated predictors with HORSES

Identifying homogeneous subgroups of variables can be challenging in high dimensional data analysis with highly correlated predictors. We propose a new method called Hexagonal Operator for Regression with Shrinkage and Equality Selection, HORSES for short, that simultaneously selects positively correlated variables and identifies them as predictive clusters. This is achieved via a constrained least-squares problem with regularization that consists of a linear combination of an L_1 penalty for the coefficients and another L_1 penalty for pairwise differences of the coefficients. This specification of the penalty function encourages grouping of positively correlated predictors combined with a sparsity solution. We construct an efficient algorithm to implement the HORSES procedure. We show via simulation that the proposed method outperforms other variable selection methods in terms of prediction error and parsimony. The technique is demonstrated on two data sets, a small data set from analysis of soil in Appalachia, and a high dimensional data set from a near infrared (NIR) spectroscopy study, showing the flexibility of the methodology.
A frequentist approach to computer model calibration

The paper considers the computer model calibration problem and provides a general frequentist solution. Under the framework proposed, the data model is semiparametric with a non‐parametric discrepancy function which accounts for any discrepancy between physical reality and the computer model. In an attempt to solve a fundamentally important (but often ignored) identifiability issue between the computer model parameters and the discrepancy function, the paper proposes a new and identifiable parameterization of the calibration problem. It also develops a two‐step procedure for estimating all the relevant quantities under the new parameterization. This estimation procedure is shown to enjoy excellent rates of convergence and can be straightforwardly implemented with existing software. For uncertainty quantification, bootstrapping is adopted to construct confidence regions for the quantities of interest. The practical performance of the methodology is illustrated through simulation examples and an application to a computational fluid dynamics model.