Statistically Meaningful Approximation: a Theoretical Analysis for Approximating Turing Machines with Transformers

1 INTRODUCTION . Dating back to the seminal works on universal approximation ( Cybenko , 1989 ; Hornik et al. , 1989 ; Park & Sandberg , 1991 ; Leshno et al. , 1993 ) , a common way to theoretically study neural nets has been through their expressivity , which measures the ability of neural nets to approximate well-behaved functions . This perspective has shaped how researchers perceive different types of deep learning architectures : a basic way to theoretically justify new architectures is to study their approximation capabilities . This has led to a number of analyses studying universal approximation capabilities for various widely-used architectures , such as recurrent neural nets ( RNNs ) ( Schäfer & Zimmermann , 2007 ) , graph neural nets ( Scarselli et al. , 2008 ) , convolutional networks ( Bao et al. , 2014 ; Zhou , 2020 ; Yarotsky , 2021 ) , residual networks ( Lin & Jegelka , 2018 ) , transformers ( Yun et al. , 2019 ) , and neural ODEs ( Teshima et al. , 2020 ; Zhang et al. , 2020 ) . However , approximation theoretic results often misalign with more meaningful end-to-end guarantees , because models constructed in the literature often exhibit unrealistic properties . For example , a common technique in the universal approximation literature is to rely strongly on infinite-precision weights and activations , or exponentially many parameters to encode the desired function values ( Hornik et al. , 1989 ; Cybenko , 1989 ; Leshno et al. , 1993 ; Lin & Jegelka , 2018 ; Yun et al. , 2019 ; Sannai et al. , 2019 ) . This issue even arises outside of universal approximation , e.g. , various papers demonstrate the ability of RNNs and transformers to simulate various computational models such as Turing machines and automata , but require strong reliance on arbitrary precision ( Siegelmann & Sontag , 1995 ; Pérez et al. , 2019 ; Korsky & Berwick , 2019 ; Bhattamishra et al. , 2020 ) . Infinite precision can inflate the expressivity of an architecture in a unrealistic and misleading way : for example , finite width RNNs with infinite precision can simulate Turing machines , but finite-precision , finite-width RNNs can not , as implied by streaming lower bounds ( Alon et al. , 1999 ) . As another example , Park et al . ( 2020 ) exploit infinite precision in the parameters to show that a neural net with parameter count sublinear in n can memorize n arbitrary input-label pairs . However , a simple counting argument reveals that this result can not be proven using finite precision networks – there are 2n input-labeling pairs , but only 2opnq finite precision networks with opnq parameters . More broadly , the ideal theoretical perspective should consider not only whether target functions can be expressed , but also whether the constructed networks are plausibly learnable . Learnability is important because empirical settings do not operate in the infinite data , unbounded computation regime – they require fitting the target function with access to limited number of samples from an empirical distribution . The question of studying learnability can be decomposed into studying optimization and generalization . Unfortunately , a rigorous analysis of optimization is unresolved even for simple two-layer nets ( Mei et al. , 2018 ) . Generalization is more tractable , so we propose to study expressivity and generalization together . Towards the goal of studying more meaningful notions of approximation , this work proposes the notion of statistically meaningful ( SM ) approximation . This definition requires not only the existence of an approximating network , but also that it has good statistical learnability . Consider a setting where the aim is to fit the target functionG using the approximating family F and a finite sample of training data . SM approximation requires existence of a loss whose empirical risk minimizer in F leads to a model with low approximation error in fittingG . We define the sample complexity of the approximation as the number of training samples needed to guarantee at most approximation error and study SM approximation with low sample complexity bounds . SM approximation essentially eliminates all statistical concerns for learnability ( optimization-related concerns can remain ) . We present two case studies on SM approximation . First , we demonstrate that overparameterized feedforward neural nets can SM approximate boolean circuits with a low sample complexity that depends only on the intrinsic circuit size . Though it is simple to construct neural nets to approximate boolean circuits , bounding the sample complexity of the approximation is challenging . For example , standard norm-based generalization bounds for the naive construction scale exponentially in depth ( Bartlett et al. , 2017 ) . Furthermore , VC dimension-based bounds would scale polynomially in the number of parameters in the network ( Harvey et al. , 2017 ) , which is problematic because for practical optimization concerns , neural nets are typically overparameterized in terms of width ( Zhang et al. , 2016 ) . In contrast , our sample complexity bound for SM approximation depends only on the intrinsic circuit size , up to logarithmic factors . Our second case study is on SM approximating Turing machines with transformers . We consider a class of Turing machines with bounded computation time T and construct encoder-decoder-based transformers ( Vaswani et al. , 2017 ) which SM approximate these Turing machines . The sample complexity of the approximation depends on a polynomial in logT and the sizes of the state space and the alphabet of the Turing machine . Though constructions for approximating Turing machines from prior work ( Siegelmann & Sontag , 1995 ; Pérez et al. , 2019 ; Bhattamishra et al. , 2020 ) have not been formally studied from a sample complexity perspective , existing bounds would depend at least linearly on T . Furthermore , our construction only uses loglogT precision , compared to at least logT in prior works , allowing us to achieve the exponential improvement in the sample complexity . Proving sample complexity guarantees for our statistically meaningful approximation results is nontrivial and requires additional insights , for both the constructions and the generalization analyses . To obtain our sample complexity bounds , we leverage a recent approach to bound generalization in terms of data-dependent notions of Lipschitzness ( Wei & Ma , 2019b ) . We develop theoretical tools to convert a broad class of neural nets , with possibly large Lipschitzness , into ones with small Lipschitzness on the training data , by introducing a number of new layers that is linear in depth . Our result applies to neural nets where each entry in the hidden representations on the training data takes values from a finite set ( e.g. , binary entries ) , and may be of independent interest . In summary , our contributions are : 1 ) we propose a new notion of statistically meaningful approximation , intended to provide more meaningful approximation guarantees by requiring that the approximating family have good statistical learnability ; 2 ) we prove that feedforward neural nets can meaningfully approximate boolean circuits with sample complexity that depends polynomially on the width and depth of the circuit ; and 3 ) we show that transformers can meaningfully approximate Turing machines with sample complexity logarithmic in the computation time . 1.1 RELATED WORKS . Classifical approximation theory for neural networks has a long history . Hornik et al . ( 1989 ) ; Cybenko ( 1989 ) , and Leshno et al . ( 1993 ) show that neural nets with one hidden layer are universal approximators but require the hidden layer size to grow exponentially in input dimension . Barron ( 1993 ) uses the Fourier transform to write target functions as infinite-width networks and subsamples neurons to obtain widths which depend only on target function properties . Lee et al . ( 2017 ) ; Ji et al . ( 2020 ) prove recent related developments in this direction of universal approximation . Many works study benefits of deep networks over shallow ones ( Bengio & Delalleau , 2011 ; Arora et al. , 2016 ; Telgarsky , 2016 ; Eldan & Shamir , 2016 ; Daniely , 2017 ; Chatziafratis et al. , 2020 ; 2019 ) . Bengio & Delalleau ( 2011 ) show separation for exact representation , whereas Telgarsky ( 2016 ) shows separation for approximate representations with univariate inputs . Eldan & Shamir ( 2016 ) demonstrate high-dimensional functions that can be approximated by two-layer polynomial-sized neural networks , but can not be approximated by one-layer neural nets with subexponential hidden units . Via reduction to certain complexity theo- retic questions , Vardi & Shamir ( 2020 ) show that proving constant depth separations may be hard . Malach et al . ( 2021 ) analyze the relationship between optimization and approximability , showing in various settings that deeper networks can not be optimized if shallow networks can not approximate them . This demonstrates that depth separation results ( Telgarsky , 2016 ) from approximation theory can be misleading in the sense that gradient descent anyways can not optimize the deep networks used to construct the approximation . Another area of study is on the ability of deep networks to memorize training data ( Zhang et al. , 2016 ; Yun et al. , 2018 ; Park et al. , 2020 ; Vershynin , 2020 ) . Yun et al . ( 2018 ) show that Θpnq parameters are sufficient to memorize Θpnq training points for ReLU nets with at least 3 layers , and Park et al . ( 2020 ) reduce the parameter requirement to sublinear in n. Similar results have been proven for residual architectures ( Hardt & Ma , 2016 ) and convolutional nets ( Nguyen & Hein , 2018 ) . Bartlett et al . ( 2019 ) analyze the VC-dimension of neural nets , leading to upper and lower bounds on the parameter count needed to fit training data . Other works study expressivity via connections to tensor approximation and sum-product networks ( Cohen & Shashua , 2016 ; Cohen et al. , 2016 ) . There is a long line of work on studying the ability of neural nets to recognize and represent formal languages . The seminal work of Siegelmann & Sontag ( 1995 ) shows that RNNs are Turing complete but leverages infinite precision in the hidden activations . Chen et al . ( 2018 ) extend this result to ReLU activations and study implications in language modeling . Many variants of transformers are shown to be Turing-complete , but these constructions also rely on arbitrary precision ( Pérez et al. , 2019 ; Bhattamishra et al. , 2020 ) . A number of recent works have also proven results for generating or recognizing formal languages with finite-precision neural nets ( Weiss et al. , 2018 ; Korsky & Berwick , 2019 ; Hewitt et al. , 2020 ) , but these results do not consider Turing machines or analyze statistical properties of their constructions . Bounding the sample complexity of SM approximation requires additional complications in both the construction and statistical analysis . 1.2 NOTATION . Let f˝g denote the composition of functions f and g. For a family of functions G , let f˝Gfitf˝g : gPGu denote the family of compositions between f and functions in G. For a set S and function f : SÑY , let fpSq denote the set tfpsq : sPSuĎY . We use 1d to denote the all-one ’ s vector in d dimensions , with the subscripted omitted if clear . For iPrds , we let 1dpiq denote the one-hot embedding in d-dimensions , which is 1 at index i and 0 everywhere else . We use the notation rOp¨q to hide poly-logarithmic factors in the argument . The notationÀ , Á indicates the existence of a constant factor such that the inequality holds . — denotes that theÁ andÀ relations simultaneously hold . We use polyp¨q to indicate the existence of a polynomial in the argument which makes the equation true . For a setA ( e.g. , the set of alphabet symbols for a Turing machine ) let A˚ denote the set of all sequences of elements of A , where sequence length can vary . Let P denote a distribution over a space of inputs X . Let ξ1 , ... , ξn be n i.i.d . Rademacher variables sampled from t´1 , ` 1u . The expected n-sample Rademacher complexity of F on P is as follows : Radn , P pFqfi E pxiqni “ 1 i.i.d „ P “ Eξ1 , ... , ξn “ supFPF 1 n řn i “ 1ξiFpxiq ‰‰ , where pxiqni “ 1 denotes n i.i.d . samples from P .

Summary: This paper studies the approximation of boolean circuits and Turing machines using neural networks. To this end, the authors propose a new notion of approximation criteria -- statistically meaningful (SM) approximation -- that specifies that the function class is both learnable and has a small bias in the sense of classical approximation theory. For both the boolean circuits and Turing machines, the authors explicitly construct two classes of neural networks, based on feedforward network and transformers respectively, that achieve SM approximation.