On the Cryptographic Hardness of Learning Single Periodic Neurons

1 Introduction . The empirical success of Deep Learning has given an impetus to provide theoretical foundations explaining when and why it is possible to efficiently learn from high-dimensional data with neural networks . Currently , there are large gaps between positive and negative results for learning , even for the simplest neural network architectures [ 61 , 29 , 14 , 27 ] . These gaps offer a large ground for debate , discussing the extent up to which improved learning algorithms can be designed , or whether a fundamental computational barrier has been reached . One particular challenge in closing these gaps is establishing negative results for improper learning in the distribution-specific setting , in which the learner can exploit the peculiarities of a known input distribution , and is not limited to outputting hypotheses from the target function class . Over the last few years , authors have successfully developed distribution-specific hardness results in the context of learning neural networks , offering different flavors . On one hand , there have been several results ⇤Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . proving the failure of a restricted class of algorithms , such as gradient-based algorithms [ 54 , 52 ] , or more generally Statistical Query ( SQ ) algorithms [ 33 , 20 , 57 , 29 , 17 ] . Notably , such results apply to the simplest cases , such as learning one-hidden-layer neural networks over the standard Gaussian input distribution [ 29 , 17 ] . On the other hand , a different line of work has shown the hardness of learning two-hidden-layer neural networks for any polynomial-time algorithm by leveraging cryptographic assumptions , such as the existence of local pseudorandom generators ( PRGs ) with polynomial stretch [ 15 ] . Despite such significant advances , important open questions remain , such as whether the simpler case of learning one hidden-layer neural network over standard Gaussian input remains hard for algorithms not captured by the SQ framework . To make this question more precise , are non-SQ polynomial-time algorithms , which may inspect individual samples – such as stochastic gradient descent ( SGD ) [ 1 ] – able to learn one-hidden layer neural networks over Gaussian input ? Understanding the answer to this question is a partial motivation of the present work . A key technique for constructing hard-to-learn functions is leveraging “ high-frequency ” oscillations in high-dimensions . The simplest instance of such functions is given by pure cosines of the form f ( x ) = cos ( 2⇡ hw , xi ) , where we refer to w 2 Sd 1 as its hidden direction , and as its frequency . Such functions have already been investigated by previous works [ 57 , 54 , 52 ] in the context of lower bounds for learning neural networks . For these hard constructions , the frequency is taken to scale polynomially with the dimension d. Note that as the univariate function cos ( 2⇡ t ) is O ( ) -Lipschitz , the function f is well-approximated by one-hidden-layer ReLU network of poly ( ) -width on any compact set ( see e.g. , Appendix G ) . Hence , understanding the hardness of learning such functions is an unavoidable step towards understanding the hardness of learning one-hidden-layer ReLU networks . In this work , we pursue this line of inquiry , focusing on weakly learning the cosine neuron class over the standard Gaussian input distribution in the presence of noise . Our main result is a proof , via a reduction from a fundamental problem in lattice-based cryptography called the Shortest Vector Problem ( SVP ) , that such learning task is hard for any polynomial-time algorithm , based on the widely-believed cryptographic assumption that ( approximate ) SVP is computationally intractable against quantum algorithms ( See e.g. , [ 49 , 43 , 19 , 3 ] and references therein ) . Our result therefore extends the hardness of learning such functions from a restricted family of algorithms , such as gradientbased algorithms or SQ , to all polynomial-time algorithms by leveraging cryptographic assumptions . Note , however , that SQ lower bounds are unconditional because they are of an information-theoretic nature . Therefore , our result , which is conditional on a computational hardness assumption , albeit a well-founded one in the cryptographic community , and SQ lower bounds are not directly comparable . The problem of learning cosine neurons with noise can be studied in the broader context of inferring hidden structures in noisy high-dimensional data , as a particular instance of the family of Generalized Linear Models ( GLM ) [ 45 , 44 ] . Multiple inference settings , including , for example , the wellknown planted clique model [ 30 , 5 ] , but also GLMs such as sparse regression [ 23 ] exhibit so-called computational-to-statistical gaps . These gaps refer to intervals of signal-to-noise ratio ( SNR ) values where inference of the hidden structure is possible by exponential-time estimators but appears out of reach for any polynomial-time estimator . Following this line of work , we define the SNR of our cosine neuron learning problem to be the inverse of the noise level , and analyze its hardness landscape . As it turns out , weakly learning the cosine neuron class provides a rich landscape , yielding a computationalto-statistical gap based on a worst-case hardness guarantee . We note that this is in contrast with the “ usual ” study of such gaps where such worst-case hardness guarantees are elusive and they are mostly based on the refutation of restricted computational classes , such as Sum-of-Squares [ 10 ] , low-degree polynomials [ 37 ] , Belief Propagation [ 9 ] , or local search methods [ 24 ] . Finally , we establish an upper bound for the computational threshold , thanks to a polynomial-time algorithm based on the Lenstra-Lenstra-Lovász ( LLL ) lattice basis reduction algorithm ( see details in Section 3.3 ) . Our proposed algorithm is shown to be highly versatile , in the sense that it can be directly used to solve two seemingly very different GLMs : the CLWE detection problem from cryptography and the phase retrieval problem from high-dimensional statistics . Remarkably , this method bypasses the SQ and gradient-based hardness established by previous works [ 54 , 57 ] . Our use of the LLL algorithm to bypass a previously considered “ computationally-hard ” region adjoins similar efforts to solve linear regression with discrete coefficients [ 60 , 26 ] , [ 37 , Sec . 4.2.1 ] , as well as the correspondence retrieval problem [ 6 ] , which includes phase retrieval as a special case . We show in Section 3.3 and Appendix F how our algorithms obtain optimal sample complexity for recovery in both these problems in the noiseless setting . An interesting observation is that in the latter case , the resulting algorithm , and also the very similar LLL-based algorithm by [ 6 ] , improves upon AMP-based algorithms [ 11 ] in terms of sample complexity , often thought to be optimal among all polynomial-time algorithms [ 41 ] . While our LLL algorithm can be seen as an appropriate modification of [ 6 ] , our analysis employs different tools , leading to improved guarantees . More precisely , our analysis easily extends to distributions that are both log-concave and sub-Gaussian , as opposed to solely Gaussian in [ 6 ] . In addition , our algorithm incorporates an explicit rounding step for LLL , which allows us to determine its precise noise-tolerance ( see details in Appendix F ) . 1.1 Related work . Hardness of learning from cryptographic assumptions . Among several previous works [ 34 , 35 ] which leverage cryptographic assumptions to establish hardness of improper learning , most relevant to our results is the seminal work of Klivans and Sherstov [ 36 ] whose hardness results are also based on SVP . To elaborate , they show that learning intersections of halfspaces , which can be seen as neural networks with the threshold activation , is hard based on the worst-case hardness of GapSVP , a decision version ( approximate ) SVP . Our work differs , though , in several important aspects from theirs . First , and perhaps most importantly , our result holds over the well-behaved Gaussian input distribution over Rd , whereas their hardness utilizes a non-uniform distribution over the Boolean hypercube { 0 , 1 } d. Second , at a technical level and in agreement with our continuous input domain and their discrete input domain , we take a different reduction route from SVP . Their link to SVP is the Learning with Errors ( LWE ) Problem [ 49 ] , whereas our link in the reduction is the recently developed Continuous Learning with Errors ( CLWE ) Problem [ 13 ] . On another front , very recently , [ 15 ] presented an abundance of novel hardness results in the context of improper learning by assuming the mere existence of Local Pseudorandom Generators ( LPRGs ) with polynomial stretch . While the LPRG and SVP assumptions are not directly comparable , we emphasize that we rely on the worstcase hardness of GapSVP , whereas LPRG assumes average-case hardness . A worst-case hardness assumption is arguably weaker as it requires only one instance to be hard , whereas an average-case hardness assumption requires instances to be hard on average . Lower bounds against restricted class of algorithms and upper bounds . As mentioned previously , a widely adapted method for proving hardness of learning is through SQ lower bounds [ 33 , 12 , 58 , 20 ] . Among previous work , most closely related to our work is [ 57 ] and [ 54 ] , who consider learning linear-periodic function classes which contain cosine neurons . By constructing a different class of hard one-hidden-layer networks , stronger SQ lower bounds over the Gaussian distribution , in terms of both query complexity and noise rate , have been established [ 29 , 17 ] . Yet , for technical reasons , the SQ model can not rule out algorithms such as stochastic gradient descent ( SGD ) , since these algorithms can in principle inspect each sample individually . In fact , [ 1 ] carry this advantage of SGD to the extreme and show that SGD is poly-time universal . [ 7 ] establishes sharp bounds using SGD for weakly learning a single planted neuron , and reveals a fundamental dependency between the regularity of their dimension-independent activation function , which they name the “ information exponent ” , and the sample complexity . The regularity of the activation function has been leveraged in several works to yield positive learning results [ 31 , 61 , 27 , 56 , 4 , 28 , 21 , 16 ] . Finally , statistical-to-computational gaps using the family of Approximate Message Passing ( AMP ) algorithms [ 18 , 47 ] for the algorithmic frontier have been established in various high-dimensional inference settings , including proper learning of certain single-hidden layer neural networks [ 8 ] , spiked matrix-tensor recovery [ 50 ] and also GLMs [ 11 ] . The LLL algorithm and statistical inference problems . For our algorithmic results , we employ the LLL algorithm . Specifically , our techniques are originally based on the breakthrough use of the LLL algorithm to solve a class of average-case subset sum problems in polynomial-time , as established first by Lagarias and Odlyzko [ 38 ] and later via a greatly simplified argument by Frieze [ 22 ] . While the power of LLL algorithm is very well established in the theoretical computer science [ 53 , 39 ] , integer programming [ 32 ] , and computational number theory communities ( see [ 55 ] for a survey ) , to the best of our knowledge , it has found only a handful of applications in the theory of statistical inference . Nevertheless , a few years ago , a strengthening of the original LLL-based arguments by Lagarias , Odlyzko and Frieze has been used to prove that linear regression with rational-valued hidden vector and continuous features can be solved in polynomial-time given access only to one sample [ 60 ] . This problem has been previously considered “ computationally-hard ” [ 23 ] and is proven to be impossible for the LASSO estimator [ 59 , 25 ] , greedy local-search methods [ 23 ] and the AMP algorithm [ 48 ] . In a subsequent work to [ 60 ] , the suggested techniques have been generalized to the linear regression and phase retrieval settings under the more relaxed assumptions of discrete ( and therefore potentially irrational ) -valued hidden vector [ 26 ] . Our work is based on insights from [ 60 , 26 ] , but is importantly generalizing the use of the LLL algorithm ( a ) for the recovery of an arbitrary unit continuous-valued hidden vector and ( b ) for multiple GLMs such as the cosine neuron , the phase retrieval problem , and the CLWE problem . However , for noiseless phase retrieval , we note that the optimal sample complexity of d+ 1 has previously been achieved by [ 6 ] using an LLL-based algorithm very similar to ours .

Summary: This paper considers a learning problem of the following form: given noisy labeled examples $(x, z)$ where $x \sim N(0, I)$ and $z = \cos(2\pi \gamma \langle w, x \rangle) + \xi$ (here $|\xi| \leq \beta$ is adversarial, additive, bounded noise), learn the hidden parameter $w$ (or at least weakly learn the mapping in terms of square loss). Thus $z$ is the (noisy) output of a single neuron with a periodic activation under Gaussian inputs. The main results are: - In the "moderate noise" $\beta = 1/\text{poly}(d)$ regime, a reduction showing that a learner for this problem could solve continuous LWE in a regime believed to be hard. - In the "small noise" $\beta = \exp(-d)$ regime, a novel efficient learning algorithm using LLL-based integer relation detection.  As one consequence, the hardness profile of this problem with respect to the noise rate $\beta$ exhibits interesting computational-to-statistical phase traditions / gaps.