Title: Online robust locally differentially private learning for nonparametric regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online learning

nonparametric regression

local differential privacy

Huber loss
Abstract: The growing prevalence of streaming data and increasing concerns over data privacy pose significant challenges for traditional nonparametric regression methods, which are often ill-suited for real-time, privacy-aware learning. In this paper, we tackle these issues
by first proposing a novel one-pass online functional stochastic gradient descent algorithm that leverages the Huber loss (H-FSGD), to improve robustness against outliers and heavy-tailed errors in dynamic environments. To further accommodate privacy constraints, we introduce a locally differentially private extension, Private H-FSGD (PH-FSGD), designed to real-time, privacy-preserving estimation. Theoretically, we conduct a comprehensive non-asymptotic convergence analysis of the proposed estimators, establishing finite-sample guarantees and identifying optimal step size schedules that achieve optimal convergence rates. In particular, we provide practical insights into the impact of key hyperparameters, such as step size and privacy budget, on convergence behavior. Extensive experiments validate our theoretical findings, demonstrating that our methods achieve strong robustness and privacy protection without sacrificing efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of nonparametric regression in a streaming data setting, where data arrives sequentially and privacy is a major concern. The authors propose a novel framework and two new algorithms to tackle this challenge. The first algorithm, H-FSGD (Huber Functional Stochastic Gradient Descent), is a one-pass online method that uses the Huber loss to achieve robustness against heavy-tailed errors and outliers. The second algorithm, PH-FSGD, extends H-FSGD to provide Local Differential Privacy (LDP) by injecting calibrated Gaussian process noise at each iteration, thus protecting individual data points at the source without a trusted curator. The paper provides a comprehensive non-asymptotic convergence analysis for both estimators, establishing optimal convergence rates and exploring the trade-offs involving step size and privacy budget. The theoretical findings are supported by experiments on synthetic data, demonstrating the robustness and privacy-preserving capabilities of the proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Significance and Originality:** The paper tackles an important problem at the intersection of streaming data analysis, robust statistics, and privacy. To the best of the authors' knowledge and my own, developing a scalable, statistically sound method for online, robust, and locally private nonparametric regression is an open problem. The proposed framework, which combines functional SGD, Huber loss for robustness, and LDP for privacy, is a novel contribution to this area.\n\n2. **Theoretical Quality:** The paper provides a rigorous and comprehensive non-asymptotic convergence analysis. The authors establish finite-sample guarantees for their estimators in a general setting that does not assume the true function lies within the chosen RKHS. The analysis of convergence rates under various smoothness assumptions and step-size schemes is thorough and provides valuable theoretical insights into the behavior of the algorithms. The proofs provided in the appendix appear to be detailed and correct.\n\n3. **Clarity:** The paper is well-written. The authors clearly motivate the problem, lay out their contributions, and present their algorithms in a way that is easy to follow. The inclusion of a comparison table and flowchart is also helpful for understanding the overall structure of their contribution.\n\n### Weaknesses\n\n1. **Experimental Scope:** My main reservation about this paper is the limited scope of the experiments. While the synthetic data simulations do a good job of supporting the theoretical claims about efficiency and privacy, they stop short of showing how the method would perform on a real-world streaming dataset. Given that the work is motivated by practical applications like sensor networks and autonomous vehicles, the absence of a real-data experiment feels like a missed opportunity to demonstrate the framework's utility.\n\n2. **Characterization of Robustness:** The paper's notion of robustness is tied to handling heavy-tailed errors via the Huber loss. This is a perfectly reasonable approach, but it would be interesting to see a discussion of how this translates to robustness in an adversarial sense, for instance, in a Huber contamination model where a fraction of the data might be arbitrarily corrupted. The trade-off between the degree of robustness and the privacy guarantee is a subtle but important aspect that isn't fully explored. An example of this in the local privacy model is \"On the Privacy-Robustness-Utility Trilemma in Distributed Learning\", ICML '23.\n\n3. **Incremental Technical Novelty:** While the synthesis of ideas here is original and valuable, the core components (Functional SGD, Huber loss, and the Gaussian mechanism for privacy) are all well-established tools in their respective fields. The contribution is therefore less about creating a new technical tool and more about demonstrating how existing tools can be thoughtfully combined and analyzed to solve a new, complex problem. This is a valid and useful contribution, but it's worth noting.\n\nQUESTIONS:\nI have a few questions for the authors that I believe could help clarify some points and strengthen the paper:\n\n1. **Real-world Performance:** I understand that setting up real-world streaming experiments can be challenging. However, could you perhaps comment on how you expect your method to perform on a benchmark streaming dataset? Even a discussion of the potential challenges (e.g., concept drift, non-i.i.d. data) and how your framework might adapt would be very insightful.\n\n2. **Robustness in Contamination Models:** Your use of Huber loss is well-motivated for heavy-tailed noise. I'm curious if you have any intuition on how the method would perform under a stronger threat model, like Huber's contamination model, where an adversary could corrupt a small fraction of the data points. Does your analysis provide any clues about the breakdown point of the estimator?\n\n3. **The Robustness-Privacy-Utility Trilemma:** There's a known tension in the local model between robustness, privacy, and statistical utility (see e.g. \"On the Privacy-Robustness-Utility Trilemma in Distributed Learning\", ICML '23). It seems your framework has all the knobs to explore this: the Huber parameter τ, the privacy budget (ϵ,δ), and the step size γ. Could you elaborate on the interplay between these? For example, how should one co-tune τ and ϵ to achieve a good trade-off, especially when outliers are present?\n\n4. A quick note on notation: In line 185, you define $w_\\tau$ in terms of a variable $u$. It's clear from the line above that $u=y−f(x)$, but for maximum clarity, you might consider making that explicit in the definition itself.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper addresses nonparametric regression on streaming data by introducing two algorithms:

H-FSGD: A functional stochastic gradient descent method that employs the Huber loss to mitigate heavy-tailed noise and outliers by truncating large residuals.

PH-FSGD: A locally differentially private variant of H-FSGD that injects carefully calibrated Gaussian process noise at each iteration, ensuring each data point is protected at the source without requiring a trusted curator.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The manuscript is well written and clearly structured.

2. Incorporating the Huber loss is a sensible choice: it projects estimates into a finite-dimensional RKHS subspace, handles heavy-tailed noise and outliers effectively, and yields a uniformly bounded gradient—ideal for private algorithm design.

3. Both algorithms operate in a single pass, making them well suited for streaming data scenarios.

Weaknesses:

1. The notation is dense and could overwhelm readers; a summary table contrasting the theoretical guarantees for the non-private and private settings would improve clarity.

QUESTIONS:
1. In Corollary 1 (Line 264), under a constant step size, the bound appears independent of the privacy parameters. Could the authors clarify this?

2. In Line 144, how is the closure of H measured or defined?

LIMITATIONS:
See above.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of online nonparametric regression and propose an algorithm that is both outlier-robust and locally differentially private. The key novelty lies in the use of the Huber loss to enhance robustness against outliers, combined with the addition of Gaussian process noise to ensure local differential privacy. The paper provides non-asymptotic theoretical guarantees for the proposed method and validates the results with synthetic numerical experiments.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses a compelling problem that is likely to interest the machine learning community, particularly those focused on streaming data, robustness, and privacy. Building on functional SGD methods for the squared loss [Liu et al., 2023], the authors introduce the Huber loss to enhance robustness and incorporate Gaussian process noise to achieve local differential privacy.\n\nIn my opinion, the paper is difficult to read due to heavy notation and a lack of a clear formal problem setup. A more structured presentation would help place the contributions in a better context and improve overall readability.\n\n1. Could the authors provide formal definitions of outlier robustness and local differential privacy (LDP) in the context of this work? While the theorems address properties related to LDP, the main text does not clearly define it. Including formal definitions would help clarify the contributions and make the theoretical results more accessible. \n2. The paper does not appear to provide any formal guarantees regarding outlier robustness. Interestingly, the choice of $\\tau$ also influences the privacy guarantees through the term $8 \\tau^2 B^2 \\log (2/\\delta)/\\varepsilon^2$. Suppose one does not have any concern about outliers, can we keep $\\tau$ to $\\infty$ and essentially work with the squared loss? How does this affect the privacy guarantees?\n3. Similarly, if privacy is not a concern, then one does not see any direct dependency on $\\tau$ in Corollary 3. How does the choice of $\\tau$ affect the guarantees in Corollary 3?  \n4. In my opinion, Lemma 1 should be included in the main text, as it appears to play a central role in supporting the main results.\n5. Related to point 1, could the authors clarify how Proposition 1, which (as far as I understand) concerns central differential privacy, is used in the proof of Theorem 1, which provides guarantees for local differential privacy?\n6. The step size depends on hyperparameters $(B, M, \\kappa)$, which may not be readily available in practice. Could the authors clarify how the step size should be selected to ensure convergence when these parameters are unknown?\n\nQUESTIONS:\nPlease see strengths and weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe title says it all: this paper proposes an online learning method for solving nonparametric regression problems under the constraint of local differential privacy (LDP). The algorithm is stochastic gradient descent (SGD) on the reproducing kernel Hilbert space (RKHS) $\\mathcal{H}$. The main contributions are:\n\n  * finite sample analysis (theoretical)\n  * optimizing convergence rates over the hyperparameters (theoretical)\n  * experiments (empirical)\n\nThere are two key aspects of the problem which motivates their solution concept:\n\n1. They do not assume the true regressor $f^* \\in \\mathcal{H}$ (i.e. the problem is misspecified).\n2. They want the method to have good empirical performance when the noise is heavy-tailed.\n\nThe main idea is to use the Huber loss to get some robustness against the heavier noise. This is a standard approach (as they point out themselves). The benefit of this loss is that the gradient becomes bounded, avoiding the clipping needed for DP-SGD for quadratic losses like in ridge regression.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** I am not able to evaluate the novelty of the theory w.r.t. the literature on nonparametric regression and online learning. \n\n* The problem is straightforward to state, mathematically.\n* The analysis, although quite complex, is interesting.\n\n**Weaknesses:**\n\n* It's not clear why this combination of constraints/properties would be needed in any particular application. In particular, the privacy piece seems almost incidental.\n* The experimental evaluation is quite cursory and doesn't explore the parameter space adequately.\n\nQUESTIONS:\n**Main Question 1: Does the convergence analysis really take into account heavy-tailed noise?**\n\n* In Section 2, the error is zero-mean but has finite bounded variance $\\sigma^2$. While $\\sigma^2$ could be large, assuming it is finite means that whole classes of two-tailed distributions are out, such as $\\alpha$-stable distributions for $\\alpha \\ne 2$ or $t$-distributions with $\\nu \\le 2$.\n\n* In the experiments in the submitted manuscript (the supplement has code only as far as I could see) the authors use Gaussian noise with only one value of the variance ($0.25$) and the Student-$t$ distribution with $\\nu = 3$ so the variance is  $\\frac{3}{3-1} = 1.5$. If I am misunderstanding, I am happy to be corrected! However, these are hardly the \"extensive experiments\" promised in the abstract. Obviously, space is limited, but I would have expected a better exploration of how robust the method is to actual heavy tailed noise.\n\n* Considering this I think the claims that this method is itself robust is misleading: the loss function is chosen for its robustness properties, but neither the analysis nor the experiments properly take into account real heavy tails.\n\n**Main Question 2: How is this algorithm supposed to be run?** This is a bit of a disconnect between the idea of the algorithm and its implementation. Since empirically everything is computed on the grid of $m$ points, everything turns into an $m$-dimensional vector gradient problem. Of course, the underlying structure comes from the RKHS assumption.\n\n* Figure 1 suggests that for each individual, they either need privacy or do not need privacy so each gradient can be computed with or without noise. However, the experiments seem to only do private or nonprivate. Is the setting under consideration really reflected in Figure 1?\n\n* The privacy piece is \"turnkey\": just add Gaussian noise at each iteration according to the gradient sensitivity. This is the same SGD algorithm which has been studied for 10+ years now. So, the novelty is in the analysis of the functional gradient. The grid determines the covariance of the noise, but how should we choose the grid?\n\n* Because the analysis/theory is done in the RKHS, the grid size doesn't appear in the finite sample bounds. Commenting on the computational complexity in terms of the sample size alone also erases the impact of the grid: is there a way to incorporate it?\n\n**Main Question 3: Is this really about privacy at all?** In the end, this is just a noisy gradient algorithm for nonparametric regression. The privacy \"story\" is an OK motivation, but the experiments are on synthetic data and have no bearing on problems involving actual privacy considerations.\n\n* Under LDP there is no need to do privacy accounting so much of the \"privacy apparatus\" that has been developed over the years does not apply here.\n\n* If minibatching were allowed then perhaps we would see some more relevance for privacy. Is there a way to incorporate minibatching into this framework?\n\n* Several applications are mentioned in the introduction, but is there a vignette that the authors could provide in which someone has a privacy constraint, is worried about heavy-tailed noise, and has to use a nonparametric regression model (i.e. that's the right model for this kind of data)?\n\n**Other comments:**\n\n* There is some notation clash in using $z$ for the pair $(x,y)$ and a grid point (which is only in the $\\mathcal{X}$ space, right?)\n\n* Shouldn't the kernel function be an input to the algorithm as well?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}