On the Cryptographic Hardness of Learning Single Periodic Neurons

1 Introduction . The empirical success of Deep Learning has given an impetus to provide theoretical foundations explaining when and why it is possible to efficiently learn from high-dimensional data with neural networks . Currently , there are large gaps between positive and negative results for learning , even for the simplest neural network architectures [ 61 , 29 , 14 , 27 ] . These gaps offer a large ground for debate , discussing the extent up to which improved learning algorithms can be designed , or whether a fundamental computational barrier has been reached . One particular challenge in closing these gaps is establishing negative results for improper learning in the distribution-specific setting , in which the learner can exploit the peculiarities of a known input distribution , and is not limited to outputting hypotheses from the target function class . Over the last few years , authors have successfully developed distribution-specific hardness results in the context of learning neural networks , offering different flavors . On one hand , there have been several results ⇤Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . proving the failure of a restricted class of algorithms , such as gradient-based algorithms [ 54 , 52 ] , or more generally Statistical Query ( SQ ) algorithms [ 33 , 20 , 57 , 29 , 17 ] . Notably , such results apply to the simplest cases , such as learning one-hidden-layer neural networks over the standard Gaussian input distribution [ 29 , 17 ] . On the other hand , a different line of work has shown the hardness of learning two-hidden-layer neural networks for any polynomial-time algorithm by leveraging cryptographic assumptions , such as the existence of local pseudorandom generators ( PRGs ) with polynomial stretch [ 15 ] . Despite such significant advances , important open questions remain , such as whether the simpler case of learning one hidden-layer neural network over standard Gaussian input remains hard for algorithms not captured by the SQ framework . To make this question more precise , are non-SQ polynomial-time algorithms , which may inspect individual samples – such as stochastic gradient descent ( SGD ) [ 1 ] – able to learn one-hidden layer neural networks over Gaussian input ? Understanding the answer to this question is a partial motivation of the present work . A key technique for constructing hard-to-learn functions is leveraging “ high-frequency ” oscillations in high-dimensions . The simplest instance of such functions is given by pure cosines of the form f ( x ) = cos ( 2⇡ hw , xi ) , where we refer to w 2 Sd 1 as its hidden direction , and as its frequency . Such functions have already been investigated by previous works [ 57 , 54 , 52 ] in the context of lower bounds for learning neural networks . For these hard constructions , the frequency is taken to scale polynomially with the dimension d. Note that as the univariate function cos ( 2⇡ t ) is O ( ) -Lipschitz , the function f is well-approximated by one-hidden-layer ReLU network of poly ( ) -width on any compact set ( see e.g. , Appendix G ) . Hence , understanding the hardness of learning such functions is an unavoidable step towards understanding the hardness of learning one-hidden-layer ReLU networks . In this work , we pursue this line of inquiry , focusing on weakly learning the cosine neuron class over the standard Gaussian input distribution in the presence of noise . Our main result is a proof , via a reduction from a fundamental problem in lattice-based cryptography called the Shortest Vector Problem ( SVP ) , that such learning task is hard for any polynomial-time algorithm , based on the widely-believed cryptographic assumption that ( approximate ) SVP is computationally intractable against quantum algorithms ( See e.g. , [ 49 , 43 , 19 , 3 ] and references therein ) . Our result therefore extends the hardness of learning such functions from a restricted family of algorithms , such as gradientbased algorithms or SQ , to all polynomial-time algorithms by leveraging cryptographic assumptions . Note , however , that SQ lower bounds are unconditional because they are of an information-theoretic nature . Therefore , our result , which is conditional on a computational hardness assumption , albeit a well-founded one in the cryptographic community , and SQ lower bounds are not directly comparable . The problem of learning cosine neurons with noise can be studied in the broader context of inferring hidden structures in noisy high-dimensional data , as a particular instance of the family of Generalized Linear Models ( GLM ) [ 45 , 44 ] . Multiple inference settings , including , for example , the wellknown planted clique model [ 30 , 5 ] , but also GLMs such as sparse regression [ 23 ] exhibit so-called computational-to-statistical gaps . These gaps refer to intervals of signal-to-noise ratio ( SNR ) values where inference of the hidden structure is possible by exponential-time estimators but appears out of reach for any polynomial-time estimator . Following this line of work , we define the SNR of our cosine neuron learning problem to be the inverse of the noise level , and analyze its hardness landscape . As it turns out , weakly learning the cosine neuron class provides a rich landscape , yielding a computationalto-statistical gap based on a worst-case hardness guarantee . We note that this is in contrast with the “ usual ” study of such gaps where such worst-case hardness guarantees are elusive and they are mostly based on the refutation of restricted computational classes , such as Sum-of-Squares [ 10 ] , low-degree polynomials [ 37 ] , Belief Propagation [ 9 ] , or local search methods [ 24 ] . Finally , we establish an upper bound for the computational threshold , thanks to a polynomial-time algorithm based on the Lenstra-Lenstra-Lovász ( LLL ) lattice basis reduction algorithm ( see details in Section 3.3 ) . Our proposed algorithm is shown to be highly versatile , in the sense that it can be directly used to solve two seemingly very different GLMs : the CLWE detection problem from cryptography and the phase retrieval problem from high-dimensional statistics . Remarkably , this method bypasses the SQ and gradient-based hardness established by previous works [ 54 , 57 ] . Our use of the LLL algorithm to bypass a previously considered “ computationally-hard ” region adjoins similar efforts to solve linear regression with discrete coefficients [ 60 , 26 ] , [ 37 , Sec . 4.2.1 ] , as well as the correspondence retrieval problem [ 6 ] , which includes phase retrieval as a special case . We show in Section 3.3 and Appendix F how our algorithms obtain optimal sample complexity for recovery in both these problems in the noiseless setting . An interesting observation is that in the latter case , the resulting algorithm , and also the very similar LLL-based algorithm by [ 6 ] , improves upon AMP-based algorithms [ 11 ] in terms of sample complexity , often thought to be optimal among all polynomial-time algorithms [ 41 ] . While our LLL algorithm can be seen as an appropriate modification of [ 6 ] , our analysis employs different tools , leading to improved guarantees . More precisely , our analysis easily extends to distributions that are both log-concave and sub-Gaussian , as opposed to solely Gaussian in [ 6 ] . In addition , our algorithm incorporates an explicit rounding step for LLL , which allows us to determine its precise noise-tolerance ( see details in Appendix F ) . 1.1 Related work . Hardness of learning from cryptographic assumptions . Among several previous works [ 34 , 35 ] which leverage cryptographic assumptions to establish hardness of improper learning , most relevant to our results is the seminal work of Klivans and Sherstov [ 36 ] whose hardness results are also based on SVP . To elaborate , they show that learning intersections of halfspaces , which can be seen as neural networks with the threshold activation , is hard based on the worst-case hardness of GapSVP , a decision version ( approximate ) SVP . Our work differs , though , in several important aspects from theirs . First , and perhaps most importantly , our result holds over the well-behaved Gaussian input distribution over Rd , whereas their hardness utilizes a non-uniform distribution over the Boolean hypercube { 0 , 1 } d. Second , at a technical level and in agreement with our continuous input domain and their discrete input domain , we take a different reduction route from SVP . Their link to SVP is the Learning with Errors ( LWE ) Problem [ 49 ] , whereas our link in the reduction is the recently developed Continuous Learning with Errors ( CLWE ) Problem [ 13 ] . On another front , very recently , [ 15 ] presented an abundance of novel hardness results in the context of improper learning by assuming the mere existence of Local Pseudorandom Generators ( LPRGs ) with polynomial stretch . While the LPRG and SVP assumptions are not directly comparable , we emphasize that we rely on the worstcase hardness of GapSVP , whereas LPRG assumes average-case hardness . A worst-case hardness assumption is arguably weaker as it requires only one instance to be hard , whereas an average-case hardness assumption requires instances to be hard on average . Lower bounds against restricted class of algorithms and upper bounds . As mentioned previously , a widely adapted method for proving hardness of learning is through SQ lower bounds [ 33 , 12 , 58 , 20 ] . Among previous work , most closely related to our work is [ 57 ] and [ 54 ] , who consider learning linear-periodic function classes which contain cosine neurons . By constructing a different class of hard one-hidden-layer networks , stronger SQ lower bounds over the Gaussian distribution , in terms of both query complexity and noise rate , have been established [ 29 , 17 ] . Yet , for technical reasons , the SQ model can not rule out algorithms such as stochastic gradient descent ( SGD ) , since these algorithms can in principle inspect each sample individually . In fact , [ 1 ] carry this advantage of SGD to the extreme and show that SGD is poly-time universal . [ 7 ] establishes sharp bounds using SGD for weakly learning a single planted neuron , and reveals a fundamental dependency between the regularity of their dimension-independent activation function , which they name the “ information exponent ” , and the sample complexity . The regularity of the activation function has been leveraged in several works to yield positive learning results [ 31 , 61 , 27 , 56 , 4 , 28 , 21 , 16 ] . Finally , statistical-to-computational gaps using the family of Approximate Message Passing ( AMP ) algorithms [ 18 , 47 ] for the algorithmic frontier have been established in various high-dimensional inference settings , including proper learning of certain single-hidden layer neural networks [ 8 ] , spiked matrix-tensor recovery [ 50 ] and also GLMs [ 11 ] . The LLL algorithm and statistical inference problems . For our algorithmic results , we employ the LLL algorithm . Specifically , our techniques are originally based on the breakthrough use of the LLL algorithm to solve a class of average-case subset sum problems in polynomial-time , as established first by Lagarias and Odlyzko [ 38 ] and later via a greatly simplified argument by Frieze [ 22 ] . While the power of LLL algorithm is very well established in the theoretical computer science [ 53 , 39 ] , integer programming [ 32 ] , and computational number theory communities ( see [ 55 ] for a survey ) , to the best of our knowledge , it has found only a handful of applications in the theory of statistical inference . Nevertheless , a few years ago , a strengthening of the original LLL-based arguments by Lagarias , Odlyzko and Frieze has been used to prove that linear regression with rational-valued hidden vector and continuous features can be solved in polynomial-time given access only to one sample [ 60 ] . This problem has been previously considered “ computationally-hard ” [ 23 ] and is proven to be impossible for the LASSO estimator [ 59 , 25 ] , greedy local-search methods [ 23 ] and the AMP algorithm [ 48 ] . In a subsequent work to [ 60 ] , the suggested techniques have been generalized to the linear regression and phase retrieval settings under the more relaxed assumptions of discrete ( and therefore potentially irrational ) -valued hidden vector [ 26 ] . Our work is based on insights from [ 60 , 26 ] , but is importantly generalizing the use of the LLL algorithm ( a ) for the recovery of an arbitrary unit continuous-valued hidden vector and ( b ) for multiple GLMs such as the cosine neuron , the phase retrieval problem , and the CLWE problem . However , for noiseless phase retrieval , we note that the optimal sample complexity of d+ 1 has previously been achieved by [ 6 ] using an LLL-based algorithm very similar to ours .

Summary: The first line of the abstract nicely summarises the work and I cannot summarise this any better.  "We show a simple reduction which demonstrates the cryptographic hardness of learning a single periodic neuron over isotropic Gaussian distributions in the presence of noise".  The goal is to learn w using samples (x, cos(2 \pi \gamma <w, x>)+noise) where x~N(0, I_d). The noise considered is adversarial. Learning under large noise is not possible information theoretically. The authors give a discussion around their main result that includes an LLL based learning algorithm for the case when the noise is small. The main cryptographic reduction is for medium noise case and is through the CLWE (Continuous Learning with Errors) problem.