Title: Robust Estimation Under Heterogeneous Corruption Rates
Venue: NeurIPS 2025
Accepted: yes
Keywords: robust statistics

robust estimation

robust regression

minimax estimation
Abstract: We study the problem of robust estimation under heterogeneous corruption rates, where each sample may be independently corrupted with a known but non-identical probability. 
  This setting arises naturally in distributed and federated learning, crowdsourcing, and sensor networks, yet existing robust estimators typically assume uniform or worst-case corruption, ignoring structural heterogeneity.
  For mean estimation for multivariate bounded distributions and univariate gaussian distributions, we give tight minimax rates for all heterogeneous corruption patterns.
  For multivariate gaussian mean estimation and linear regression, we establish the minimax rate for squared error up to a factor of $\sqrt{d}$, where $d$ is the dimension.
    Roughly, our findings suggest that samples beyond a certain corruption threshold may be discarded by the optimal estimators -- this threshold is determined by the empirical distribution of the corruption rates given.


Peer reviews:
comments: SUMMARY:
The authors consider the problem of robust estimation under the Heterogeneous corruption model, where the corruption across samples is independently but not identically distributed. They derive upper and lower bounds on the minimax rates (and minimax PAC rates) for several canonical estimation problem such as mean estimation for Gaussian distributions and linear regression in this heterogeneously robust corruption model.

STRENGTHS AND WEAKNESSES:
Strengths: I find this particular model of heterogeneous robust corruption to be interesting and well supported by practical scenarios. The results are clearly presented and intuitions are given for the algorithms attaining the upper bounds and the lower bounds techniques. The difficulty of this scenario w.r.t. classical robust estimation is captured in the introduction, and the approaches of the authors are adjusted to retrieve non-trivial bounds.
Weaknesses: While I do think the results are intersting, they seem to be quite loose. I particularly refer to the results of Gaussian mean estimation and Linear Regression, which suffer from a O(\sqrt{d}) deficit between the upper and lower bounds, which is only tight in the non interesting constant dimension regime.

QUESTIONS:
I would like to see more discussion on the O(\sqrtt{d}) gap in the regime where the dimension in not constant. Which of the bounds is conjectured to be tight by the authors and why?

LIMITATIONS:
The authors adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for robust estimation under heterogeneous corruption, where each data point can be corrupted with a known, but potentially distinct, probability $\\lambda_i$. This model is a generalization of the standard Huber contamination model, which assumes a uniform corruption rate.  The authors provide a theoretical analysis for several fundamental estimation problems within this framework. For bounded mean estimation, they establish tight minimax optimal rates. For multivariate Gaussian mean estimation and linear regression, they establish minimax rates up to a factor of $\\sqrt{d}$, where $d$ is the dimension. The paper proposes two main types of estimators: a simple and intuitive thresholding-based method, and a more refined per-sample reweighting scheme. The results suggest that the optimal estimators may effectively discard samples whose corruption rates exceed a certain threshold, which is determined by the empirical distribution of the corruption rates themselves.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n\n1. **Originality and Significance:** The formulation of the robust estimation problem with heterogeneous corruption rates is novel and significant. It addresses a practical limitation of existing models that assume homogeneous corruption, making the framework relevant for applications in distributed settings like federated learning, crowdsourcing, and sensor networks. \n\n\n\n2. **Theoretical Depth and Rigor:** The paper provides a comprehensive theoretical analysis, deriving minimax rates for several key problems. The proofs for the upper and lower bounds are rigorous. For the bounded mean estimation problem, the derived rate is shown to be tight, which is a strong result. The use of both Le Cam's and Assouad's methods for the lower bounds demonstrates a sophisticated handling of the technical challenges. \n\n\n3. **Proposed Estimators:** The paper proposes both a simple, computationally efficient thresholding estimator and a more complex reweighting scheme (Algorithm 1). The thresholding approach is intuitive and shown to be minimax optimal (up to constants), providing a clear and actionable strategy. The introduction of weighted versions of Tukey depth and regression depth is a natural and elegant extension of classical robust methods to the heterogeneous setting. \n\n\n\n\n\n### Weaknesses\n\n\n1. **Gap in Bounds for High Dimensions:** The most significant limitation is the  $\\sqrt{d}$ gap between the upper and lower bounds for multivariate Gaussian mean estimation and linear regression.  While the authors are transparent about this gap, it means the characterization of the minimax rate in high dimensions is incomplete. It remains unclear whether this is an artifact of the analysis or a true phenomenon. \n\n\n2. **Assumption of Known Corruption Rates:** The entire framework relies on the assumption that the corruption rates $\\lambda_i$ are known exactly.  The authors mention that their bounds generalize to approximate knowledge of $\\lambda_i$, but this is not formally detailed. In many practical scenarios, these rates may need to be estimated, which would introduce an additional layer of error not accounted for in the current analysis.\n\n\n3. **Limited Practical Demonstration:** The experimental evaluation is presented as \"preliminary\". While the results in Figure 2 do support the theoretical claims by showing improvement over baselines, they are limited to low-dimensional synthetic data. Furthermore, the practical advantage of the more complex reweighting scheme over the simple thresholding method is described as \"marginal\" for the bounded case, which might underwhelm practitioners.\n\nQUESTIONS:\n1. The paper's main theoretical limitation is the $\\sqrt{d}$ gap in the high-dimensional settings (Theorem 4 and 5). The authors suggest this might be an \"artifact of the analysis\".  Could you provide more intuition on why the current analysis techniques (specifically, the uniform convergence bounds for the weighted Tukey median in Appendix C.1) might be losing this factor? Do you have a conjecture on whether the upper or lower bound is closer to the true rate, and what new analytical tools might be needed to close this gap?\n\n2. The framework assumes that the corruption rates $\\lambda_i$ are known precisely. How would the proposed rates change if one only had access to estimates $\\hat{\\lambda}_i$ such that, for instance, $|\\hat{\\lambda}_i - \\lambda_i| \\leq \\varepsilon$ for all $i$? Would the error from estimating $\\lambda_i$ dominate the statistical error rates derived in the paper, or would it add a lower-order term? A brief formal discussion on this point would strengthen the paper's practical relevance.\n\n3. The paper proposes both a simple thresholding estimator and a more complex reweighting estimator (Algorithm 1). For bounded mean estimation, the thresholding estimator is sufficient to achieve the minimax rate. The experiments in Figure 2a show only \"marginal improvement\" from the reweighting scheme.  Could the authors provide more insight into the scenarios where the additional complexity of the optimal reweighting scheme is justified? Is there a specific structure in the distribution of $\\lambda_i$ values where reweighting provides a significant practical advantage over simply choosing an optimal threshold?\n\n4. In the proof of the upper bound for the weighted Tukey median (Proposition 1, Appendix C.1), the result relies on Proposition 4, which provides a weighted generalization bound. The bound on the Rademacher complexity in Lemma 2 has a dependency on $\\|w\\|_2$.  This $\\|w\\|_2$ term eventually leads to the $\\frac{d}{N(t)}$ term for the thresholding estimator. Is this dependence on $\\|w\\|_2$ fundamental, or is it possible that a different analysis technique could yield a bound that depends on the weights w in a more nuanced way, potentially helping to close the  $\\sqrt{d}$ gap?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes and studies the model of heterogeneous corruption where the $i$-th sample in the dataset has a different (assumed to be known) corruption level $\\lambda_i$ and the $i$-th point is generated by (i) determining with probability $\\lambda_i$ whether the sample is inlier and (ii) in case of inlier the sample is drawn from the inlier distribution, otherwise it is drawn from a different distribution (given the previous samples).\n\nThe paper establishes the minimax rates for several estimation problems under this model: Mean estimation for multivariate bounded distributions, mean estimation for multivariate Gaussians with identity covariance and multivariate linear regression. For the last two there is a gap of $\\sqrt{d}$ between the upper and lower bounds.\n\nThe upper bounds all involve the function $f(\\lambda,k)=\\min_{t \\in [0,1]}(k/\\{\\lambda_i \\leq t\\}+t^2)$ which is related to the best effective corruption rate. The upper bounds can be achieved by a simple algorithm that chooses a corruption level $t$ and runs an estimator on the subset of the data data corresponding to $\\lambda_i \\leq t$ (where $t$ is chosen to minimize the error). The authors also show alternative methods based on reweighting the points (for the bounded distributions case they show a method that optimizes the weights of the points based on explicit calculations, and for the other problems they show variants of the Tukey median and Tukey regression depth). The lower bounds are built on Le Cam's two point method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Estimation with heterogeneous data is well motivated. The paper gives the first non-trivial results for basic estimation problems that remained unstudied.\n- The paper provides experiments showing the advantage of the proposed algorithms.\n\nWeaknesses:\n\n- The bounds on the minimax squared $\\ell_2$ error are tight only for the bounded support distributions and univariate Gaussians and has a $\\sqrt{d}$ gap otherwise. This is still better than naive coordinate-wise methods but means that the precise bound is still unknown.\n- Some of the narrative about the upper bound method was somewhat unclear to me even after the explanation of page 2. If the simple algorithm of running the homogeneous contamination estimator for the best level achieves the claimed error upper bound is there a theoretical advantage of the other methods presented or they are shown as alternative methods?\n\nQUESTIONS:\nThe third bullet in the definition of the contamination model is a bit stronger than Huber's model with heterogeneous corruption levels, since outliers are drawn adaptively. Is this adaptivity necessary to assume for the lower bounds?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper presents a generalization of the Huber contamination model to heterogenous known per-sample corruption rates and presents estimators and proofs of optimality (up to a constant factor) for mean estimation in two settings (bounded distributions and Gaussians) as well as linear regression with Gaussian covariates.

STRENGTHS AND WEAKNESSES:
### Strengths

Robust mean estimation is an important problem; the author's heterogenous contamination model is applicable to practical settings such as federated learning. The optimal estimators presented are valuable contributions; the proofs of optimality presented are clear and helpful.

### Weaknesses

As noted by the authors, the estimators presented are only tractable on low-dimensional data, and require that corruption rates for each sample are known, which may not always be realistic in practice.

The paper has a large number of minor typos and grammatical errors:
- "are not independent can dependent" on line 197
- $||\mu_{P_1} - \mu_{P_2}||^2_2$ should be $||\mu_{P_0} - \mu_{P_1}||^2_2$ on line 215
- "This conditions" should be "This condition" on line 228
- "get" -> "get a" on line 230
- $\hat{\mu}_S(\boldsymbol{Z}, w(t))$ is generally used instead of the $\hat{\mu}_S(\boldsymbol{Z}, t)$ from Equation 20 that appears to be intended

QUESTIONS:
How frequently is knowing $\lambda$ (or an accurate approximation) realistic in settings such as federated learning?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04