Title: Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networks
Authors: Sitan Chen
Aravind Gollakota
Adam Klivans
Raghu Meka
Venue: NeurIPS 2022
Keywords: PAC learning
neural networks
SQ lower bounds

Abstract: We give superpolynomial statistical query (SQ) lower bounds for learning two-hidden-layer ReLU networks with respect to Gaussian inputs in the standard (noise-free) model. No general SQ lower bounds were known for learning ReLU networks of any depth in this setting: previous SQ lower bounds held only for adversarial noise models (agnostic learning) (Kothari and Klivans 2014, Goel et al. 2020a, Diakonikolas et al. 2020a) or restricted models such as correlational SQ (Goel et al. 2020b, Diakonikolas et al. 2020b). Prior work hinted at the impossibility of our result: Vempala and Wilmes (2019) showed that general SQ lower bounds cannot apply to any real-valued family of functions that satisfies a simple non-degeneracy condition. To circumvent their result, we refine a lifting procedure due to Daniely and Vardi (2021) that reduces Boolean PAC learning problems to Gaussian ones. We show how to extend their technique to other learning models and, in many well-studied cases, obtain a more efficient reduction. As such, we also prove new cryptographic hardness results for PAC learning two-hidden-layer ReLU networks, as well as new lower bounds for learning constant-depth ReLU networks from membership queries.

TL;DR: We show full statistical query lower bounds, as well as cryptographic lower bounds, for learning two-hidden-layer ReLU networks.

Review discussion:
**Strengths**
- A solid theoretical work that establishes new hardness results for the now ubiquitous neural networks.
**Weaknesses**
- The paper does not explicitly indicate the limitations of the results.
For example, lines 67-68 say that "Theorem 1.1 rules out almost all known approaches for provably learning neural networks", but the most well-known approach to learn neural networks---SGD---is not ruled out by Thm 1.1.
No questions
See weaknesses above.
No
Thanks for the positive feedback! There is indeed a distinction between SQ and SGD, though note that by recent results of Abbe, Sandon, and coauthors, SGD with a suitable architecture + exotic initialization is essentially “P-complete,” so the only way to rule out SGD would be to show a computational lower bound by reducing from a complexity-theoretic conjecture. Fortunately, this is precisely what our lower bound in Theorem 1.3, based on LWR, achieves. We thus view our results as strong evidence that no efficient algorithm, not even SGD, can learn two hidden layer networks. We will tweak our language so this is clearer.
I am satisfied with the author's response.
The paper is well-written and the review of the related work is thorough. A great deal of effort has been put into making sure that the paper is clear and accessible for a wide audience, particularly in the technical overview. This paper is a bit outside my expertise, but the results and techniques used are interesting and could be of independent interest. I believe this paper is relevant to the learning theory community.
### Suggestions
A conclusion should be included in the main body were the paper to be accepted. If there is room, a table comparing different lower bounds setting and contrasting the current paper with related work would be a great addition.
Limitations: yes
Impact: N/A
No
Thanks for the very positive review! We will definitely try to add a conclusion and a table comparing the existing lower bounds in the final version of our paper if space allows.
# Pros
1. This is good result. The authors provide lower bounds under several assumptions/models.
2. This work is very well-written. Checked almost all the proofs and the claims are sound.
# Cons
Not really a con just a comment. The lower bounds are for 2-hidden layer networks where there are results for 1-hidden layer networks for the CSQ model. [GGJ+20],[DKKZ20] Basically, the trade-off is stronger model and 2-hidden layer instead of 1. In general, I would expect stronger lower bounds for 2-hidden layer network.
Overall, I recommend for acceptance.
Some presentation suggestions:
I would suggest the authors to add after section 1.2 lines 197 some more intuition about the construction and the idea that they use to decrease the size of the network.
Some fullstops and commas in equations are missing, i.e., eq. 2.
everything is good.
No
Thanks for the positive feedback and for closely engaging with our work! We will make sure to incorporate the presentation suggestions. The comment on the tradeoff between expressivity of CSQ/SQ vs. complexity of the concept class is indeed a valid one, though establishing SQ lower bounds is significantly more challenging: whereas there are non-CSQ algorithms that circumvent the CSQ lower bounds for 1-hidden-layer [Chen-Klivans-Meka ‘20], there are no known efficient algorithms that circumvent our SQ lower bounds.

Meta-review:
role: AC
data: metareview: This work provides lower bounds in the noise free setting for learning two hidden layer networks in the Gaussian space. Overall it is a fundamental result well within the scope of Neurips, continuing a solid line of work and I cannot see any reason for rejection.

The authors have engaged with the reviewers, and have committed to make minor revisions and clarifications which I am sure they will do.
role: PC

Decision: Accept