Title: Robust Regression of General ReLUs with Queries
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning Theory

Active Learning

ReLU Regression

Agnostic Learning
Abstract: We study the task of 
agnostically learning general 
(as opposed to homogeneous) ReLUs 
under the Gaussian distribution with respect 
to the squared loss. In the passive learning setting, 
recent work gave a computationally efficient algorithm 
that uses $poly(d,1/\epsilon)$ labeled examples 
and outputs a hypothesis with error $O(opt)+\epsilon$, 
where $opt$ is the squared loss of the best fit ReLU.
Here we focus on 
the interactive setting, where the learner 
has some form of query access to the labels of unlabeled 
examples. 
Our main result is the first computationally 
efficient learner  
that uses 
$d polylog(1/\epsilon)+\tilde{O}(\min\{1/p, 1/\epsilon\})$ 
black-box label queries, 
where $p$ is the bias of the target function, and achieves error $O(opt)+\epsilon$.
We complement our algorithmic result by showing 
that its query complexity 
bound is qualitatively near-optimal, 
even ignoring computational constraints. 
Finally, we establish that query access 
is essentially necessary 
to improve on the label complexity of passive learning. Specifically, for pool-based active learning, 
any active learner 
requires $\tilde{\Omega}(d/\epsilon)$ labels, 
unless it draws a super-polynomial
 number of unlabeled examples.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers active learning, where an algorithm can query for the label corresponding to any data instance. This is a stronger oracle model than pool-based active learning where the unlabeled dataset is fixed. \n\nThe main contribution is a computationally-efficient algorithm that achieves a label complexity that nearly matches the lower bound. The authors show that this label complexity is lower than what is required for pool-based active learning, establishing the benefit of the stronger oracle model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* This paper identifies a clear gap in the literature and closes this gap. I especially value how the authors demonstrate improved label complexity that comes with the stronger oracle than pool-based active learning. \n\n**Weaknesses**\n* The paper is very dense. The introduction is a summary of the main results and a comparison with prior work, and there is no separate related works section. \n\n* The techniques used in this paper are specific to a Gaussian marginal distribution. The proofs rely heavily on the fact that they are Gaussian, e.g., relating to the Ornstein–Uhlenbeck semigroup. The proofs are very technical, with limited insights into how the arguments used in the paper extend to different marginals. It would be nice if the authors could describe some intuition on what the Gaussian marginals entail - is it expected that the problem becomes harder or easier for other marginal distributions?\n\n**Some Comments**\nThis paper solves a technically involved and important problem, and so it is understandable that the analysis is not too easy to follow. However, I find the Appendix to be a better description of the ideas underlying the algorithm and proof. I believe the paper presentation can be improved by modifying Section 3. For example, the authors could first clearly write out the main steps and then describe each step in detail in different subsections.\n\nQUESTIONS:\n* Could the authors describe the intuitions underlying the proofs? It seems that each step of the algorithm could be explained/motivated separately, and how they contribute to the overall complexity can be described in isolation. \n\n* What are the main bottlenecks in generalizing to more general marginal distributions? What is it about the Gaussian distribution that makes the analysis possible? Is there a natural extension to log-concave distributions, exponential families or some other classes of marginal distributions? While the analysis' being focused on Gaussian marginals is understandable, this would help better understand the intuition behind the improved label complexity that arises in the considered setting.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of agnostically learning ReLUs (with arbitrary biases) under the Gaussian distribution. They work in the label query model, where the learner can query an oracle for the (possibly noisy) value $y(x)$ at a datapoint $x$. The aim is to find a ReLU hypothesis $\\hat{h}$, such that the squared loss of $\\hat{h}$ is at most $O(OPT) +\\epsilon$ where $OPT$ is the squared loss of the best ReLU hypothesis. They solve this problem by designing a projected gradient descent (with warm start) based algorithm that uses $M= d\\text{poly}\\log (1/\\epsilon)+\\min (1/p,1/\\epsilon)$ queries, and runs in time $\\text{poly}(M)$ This was previously only known for the case where the ground truth ReLU was $\\max (w^* \\cdot x- t)$ with threshold $t\\leq 0$. In this work, they handle arbitrary thresholds. Here $p= Pr_{X\\sim \\mathcal{N}(0,I)}[w^*\\cdot x \\geq t]$. This is in contrast to the setting of learning with samples, where the best algorithms draw $\\text{poly}( d/\\epsilon)$ samples. Here, the authors leverage the query access model to obtain poly log sample complexity in the error parameter.\n\nThey support their upper bound with an information theoretic lower bound on the query complexity. They also show that any pool based active learner (algorithm first selects a pool of unlabelled samples and is allowed to query labels only for examples in this set) that doesnt draw a large number of unlabelled samples needs more query complexity than the learner equipped with label queries. This shows a separation in the power of the two model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper's main algorithm (other than the initialization/ warm start) is projected gradient descent. This is a practically successful algorithm and this paper adds to the theoretical evidence supporting the empirical success of gradient descent. \n- The analysis is highly non-trivial and cleverly employs various techniques from the analysis of Gaussian space and gradient descent.\n- Achieving polylog query complexity requires a lot of fine grained work, for instance, even hypothesis selection is non-trivial (see Lemma 2.11)\n- The question of learning ReLU activations is a fundamental question in learning theory, and this paper makes progress on it.\n\n## Weaknesses\n- The analysis only works in Gaussian space and uses many sophisticated tricks from Gaussian integration. \n- The technical proofs in the appendix are very heavy in equations. I suggest adding supporting text whenever possible. \n- Section 2 contains many structural lemmas (Lemma 2.2-2.5) . Are all of these novel to this paper, or have similar analyses appeared before? For instance Lemma 2.2 seems to be a natural statement linking the loss function to the angle between the vectors  with a natural Gaussian integration based proof.\n- The use of the term \"bias\" for $p$ seems a bit confusing (as it usually refers to the threshold of the relu, or even the expectation of the function). Is this terminology standard?\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nThis is a theoretical paper. I don't see any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses problem of agnostically learning a general ReLU function under the Gaussian distribution with respect to the squared loss. The authors focus on the interactive learning setting, where the learner has black-box query access to labels. The main contribution is a computationally efficient algorithm that achieves an error of O(opt) + ε with a query complexity of d*polylog(1/ε) + Õ(min{1/p, 1/ε}), where p is the bias of the optimal ReLU.  The authors complement their algorithm with a nearly-matching information-theoretic lower bound, demonstrating the near-optimality of their approach. Furthermore, they establish a strong separation result, proving that the weaker pool-based active learning model cannot achieve similar query improvements, thus justifying their use of a stronger membership query oracle.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The paper presents near optimal sample complexity in the interactive learning setting, along with an almost matching lower bound.\n- The paper also shows the importance of the interactive setup showing a lower bound on pool-based active learning algorithm.\n- The authors start off with the simpler of setup of learning a spherical GLM and generalize to the ReLU setup later.\n\nWeaknesses:\n- The bias p, defined on line 101, is used extensively in the introduction  to showcase the main results. It would be better to define it earlier in the introduction. Also, the term ‘bias’ is used confusingly to denote both p and t in the paper.\n- Despite the simpler setup, the overall presentation is a bit too dense and it is challenging to verify the correctness of each of the claims. \n- The oracle model with the interactive querying can be a pretty strong. Would a slightly stronger pool based membership query work? For example, a local pool based membership oracle which allows the learner to query in a neighborhood of the points sampled from the input distribution Dx?\n\nQUESTIONS:\n- How can the results be generalized to general sub-gaussian distributions over the input space Dx?\n- What are the challenges in obtaining an optimality guarantee on $opt +\\epsilon$ rather than O(opt) + \\epsilon?\n- Optimality Gap (Theorems 1.1 & 1.2): There is a small gap between the upper bound's Õ(1/p) and the lower bound's Ω(1/p^(1-o_d(1))) complexity. Do you believe this gap is an artifact of the analysis, or could there be a fundamental reason for the p^(-o_d(1)) factor?\n- What is the dependence on $\\delta$ on the sample complexity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper studies the problem of agnostically learning general (non-homogeneous) ReLU functions under the Gaussian distribution with respect to squared loss. While recent work has shown that, in the passive learning setting, one can learn with error O(opt) + ε using a polynomial number of labeled examples, this paper focuses on the interactive setting, where the learner can query labels of selected unlabeled examples.

The main contribution is a computationally efficient algorithm that achieves error O(opt) + ε using roughly d·polylog(1/ε) + O(min{1/p, 1/ε}) black-box label queries, where p is the bias of the target function.

STRENGTHS AND WEAKNESSES:
The main strength of the paper is to show a sample complexity upper bound (Theorem 3.1) and an almost matching information theoretic lower bound (theorem 4.1). They also provide a separation between pool-based active learning and query learning in this setting.

QUESTIONS:
Can this technique be generalized for learning more than one neuron?

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05