Uncertainty measurement in high-dimensional statistics
Creators: name: Shen, Yiwei
Publisher: name: No Publisher Supplied
Year: 2024

description: We construct confidence regions under various high-dimensional regression models, including 2-layer neural network estimates, Huber’s robust regression, and shape-constrained regression, highlighting isotonic regression as an example.In Chapter 2, we study two-layers Neural Networks (NN), where the first layer contains random weights, and the second layer is trained using Ridge regularization. We establish asymptotic distribution results for this 2-layers NN model in the regime where the ratios p n and d n have finite limits, n is the sample size, p is the ambient dimension and d is the width of the first layer. We show that a weighted average of the derivatives of the trained NN at the observed data is asymptotically normal, in a setting with Lipschitz activation functions in a linear regression response with Gaussian features under possibly non-linear perturbations. We then leverage this asymptotic normality result to construct confidence intervals (CIs) for single components of the unknown regression vector. It shows that the double-descent phenomenon occurs in terms of the length of the CIs, with the length increasing and then decreasing as d n ↗ +∞ for certain fixed values of p n.In Chapter 3, we develop asymptotic normality results for individual coordinates of robust M-estimators with convex penalty in high-dimensions, where the dimension p is at most of the same order as the sample size n, i.e, p/n ≤ γ for some fixed constant γ &gt; 0. The asymptotic normality requires a bias correction and holds for most coordinates of the M-estimator for a large class of loss functions including the Huber loss and its smoothed versions regularized with a strongly convex penalty. The asymptotic variance that characterizes the width of the resulting confidence intervals is estimated with data-driven quantities. This estimate of the variance adapts automatically to low (p/n → 0) or high (p/n ≤ γ) dimensions and does not involve the proximal operators seen in previous works on asymptotic normality of M-estimators. For the Huber loss, the estimated variance has a simple expression involving an effective degrees-of-freedom as well as an effective sample size. The case of the Huber loss with Elastic-Net penalty is studied in details and a simulation study confirms the theoretical findings.In Chapter 4, we construct optimal confidence balls for the shape restricted Gaussian sequence models, where the mean vector μ to estimate is assumed to be constrained in a closed convex subset K in Rn. Interested in the confidence balls centered at the MLE restricted in K and the optimal data-driven radius among all possible choices, we show that if the data-driven radius is taken to be the square root of the divergence of the MLE up to a constant, the confidence balls are honest but optimal in several regimes. One criterion of the optimality lies in the case when the noise level goes to 0, as is called the “low sigma limit” case. Provided that K is a closed convex subcone of Rn with statistical dimension δ(K) → +∞, the confidence balls are adaptive over conical sub-models of K and are optimal if the signal-to-noise level is extremely high or low. We study the univariate isotonic regression model as an example.