Statistically Meaningful Approximation: a Theoretical Analysis for Approximating Turing Machines with Transformers

1 INTRODUCTION . Dating back to the seminal works on universal approximation ( Cybenko , 1989 ; Hornik et al. , 1989 ; Park & Sandberg , 1991 ; Leshno et al. , 1993 ) , a common way to theoretically study neural nets has been through their expressivity , which measures the ability of neural nets to approximate well-behaved functions . This perspective has shaped how researchers perceive different types of deep learning architectures : a basic way to theoretically justify new architectures is to study their approximation capabilities . This has led to a number of analyses studying universal approximation capabilities for various widely-used architectures , such as recurrent neural nets ( RNNs ) ( Schäfer & Zimmermann , 2007 ) , graph neural nets ( Scarselli et al. , 2008 ) , convolutional networks ( Bao et al. , 2014 ; Zhou , 2020 ; Yarotsky , 2021 ) , residual networks ( Lin & Jegelka , 2018 ) , transformers ( Yun et al. , 2019 ) , and neural ODEs ( Teshima et al. , 2020 ; Zhang et al. , 2020 ) . However , approximation theoretic results often misalign with more meaningful end-to-end guarantees , because models constructed in the literature often exhibit unrealistic properties . For example , a common technique in the universal approximation literature is to rely strongly on infinite-precision weights and activations , or exponentially many parameters to encode the desired function values ( Hornik et al. , 1989 ; Cybenko , 1989 ; Leshno et al. , 1993 ; Lin & Jegelka , 2018 ; Yun et al. , 2019 ; Sannai et al. , 2019 ) . This issue even arises outside of universal approximation , e.g. , various papers demonstrate the ability of RNNs and transformers to simulate various computational models such as Turing machines and automata , but require strong reliance on arbitrary precision ( Siegelmann & Sontag , 1995 ; Pérez et al. , 2019 ; Korsky & Berwick , 2019 ; Bhattamishra et al. , 2020 ) . Infinite precision can inflate the expressivity of an architecture in a unrealistic and misleading way : for example , finite width RNNs with infinite precision can simulate Turing machines , but finite-precision , finite-width RNNs can not , as implied by streaming lower bounds ( Alon et al. , 1999 ) . As another example , Park et al . ( 2020 ) exploit infinite precision in the parameters to show that a neural net with parameter count sublinear in n can memorize n arbitrary input-label pairs . However , a simple counting argument reveals that this result can not be proven using finite precision networks – there are 2n input-labeling pairs , but only 2opnq finite precision networks with opnq parameters . More broadly , the ideal theoretical perspective should consider not only whether target functions can be expressed , but also whether the constructed networks are plausibly learnable . Learnability is important because empirical settings do not operate in the infinite data , unbounded computation regime – they require fitting the target function with access to limited number of samples from an empirical distribution . The question of studying learnability can be decomposed into studying optimization and generalization . Unfortunately , a rigorous analysis of optimization is unresolved even for simple two-layer nets ( Mei et al. , 2018 ) . Generalization is more tractable , so we propose to study expressivity and generalization together . Towards the goal of studying more meaningful notions of approximation , this work proposes the notion of statistically meaningful ( SM ) approximation . This definition requires not only the existence of an approximating network , but also that it has good statistical learnability . Consider a setting where the aim is to fit the target functionG using the approximating family F and a finite sample of training data . SM approximation requires existence of a loss whose empirical risk minimizer in F leads to a model with low approximation error in fittingG . We define the sample complexity of the approximation as the number of training samples needed to guarantee at most approximation error and study SM approximation with low sample complexity bounds . SM approximation essentially eliminates all statistical concerns for learnability ( optimization-related concerns can remain ) . We present two case studies on SM approximation . First , we demonstrate that overparameterized feedforward neural nets can SM approximate boolean circuits with a low sample complexity that depends only on the intrinsic circuit size . Though it is simple to construct neural nets to approximate boolean circuits , bounding the sample complexity of the approximation is challenging . For example , standard norm-based generalization bounds for the naive construction scale exponentially in depth ( Bartlett et al. , 2017 ) . Furthermore , VC dimension-based bounds would scale polynomially in the number of parameters in the network ( Harvey et al. , 2017 ) , which is problematic because for practical optimization concerns , neural nets are typically overparameterized in terms of width ( Zhang et al. , 2016 ) . In contrast , our sample complexity bound for SM approximation depends only on the intrinsic circuit size , up to logarithmic factors . Our second case study is on SM approximating Turing machines with transformers . We consider a class of Turing machines with bounded computation time T and construct encoder-decoder-based transformers ( Vaswani et al. , 2017 ) which SM approximate these Turing machines . The sample complexity of the approximation depends on a polynomial in logT and the sizes of the state space and the alphabet of the Turing machine . Though constructions for approximating Turing machines from prior work ( Siegelmann & Sontag , 1995 ; Pérez et al. , 2019 ; Bhattamishra et al. , 2020 ) have not been formally studied from a sample complexity perspective , existing bounds would depend at least linearly on T . Furthermore , our construction only uses loglogT precision , compared to at least logT in prior works , allowing us to achieve the exponential improvement in the sample complexity . Proving sample complexity guarantees for our statistically meaningful approximation results is nontrivial and requires additional insights , for both the constructions and the generalization analyses . To obtain our sample complexity bounds , we leverage a recent approach to bound generalization in terms of data-dependent notions of Lipschitzness ( Wei & Ma , 2019b ) . We develop theoretical tools to convert a broad class of neural nets , with possibly large Lipschitzness , into ones with small Lipschitzness on the training data , by introducing a number of new layers that is linear in depth . Our result applies to neural nets where each entry in the hidden representations on the training data takes values from a finite set ( e.g. , binary entries ) , and may be of independent interest . In summary , our contributions are : 1 ) we propose a new notion of statistically meaningful approximation , intended to provide more meaningful approximation guarantees by requiring that the approximating family have good statistical learnability ; 2 ) we prove that feedforward neural nets can meaningfully approximate boolean circuits with sample complexity that depends polynomially on the width and depth of the circuit ; and 3 ) we show that transformers can meaningfully approximate Turing machines with sample complexity logarithmic in the computation time . 1.1 RELATED WORKS . Classifical approximation theory for neural networks has a long history . Hornik et al . ( 1989 ) ; Cybenko ( 1989 ) , and Leshno et al . ( 1993 ) show that neural nets with one hidden layer are universal approximators but require the hidden layer size to grow exponentially in input dimension . Barron ( 1993 ) uses the Fourier transform to write target functions as infinite-width networks and subsamples neurons to obtain widths which depend only on target function properties . Lee et al . ( 2017 ) ; Ji et al . ( 2020 ) prove recent related developments in this direction of universal approximation . Many works study benefits of deep networks over shallow ones ( Bengio & Delalleau , 2011 ; Arora et al. , 2016 ; Telgarsky , 2016 ; Eldan & Shamir , 2016 ; Daniely , 2017 ; Chatziafratis et al. , 2020 ; 2019 ) . Bengio & Delalleau ( 2011 ) show separation for exact representation , whereas Telgarsky ( 2016 ) shows separation for approximate representations with univariate inputs . Eldan & Shamir ( 2016 ) demonstrate high-dimensional functions that can be approximated by two-layer polynomial-sized neural networks , but can not be approximated by one-layer neural nets with subexponential hidden units . Via reduction to certain complexity theo- retic questions , Vardi & Shamir ( 2020 ) show that proving constant depth separations may be hard . Malach et al . ( 2021 ) analyze the relationship between optimization and approximability , showing in various settings that deeper networks can not be optimized if shallow networks can not approximate them . This demonstrates that depth separation results ( Telgarsky , 2016 ) from approximation theory can be misleading in the sense that gradient descent anyways can not optimize the deep networks used to construct the approximation . Another area of study is on the ability of deep networks to memorize training data ( Zhang et al. , 2016 ; Yun et al. , 2018 ; Park et al. , 2020 ; Vershynin , 2020 ) . Yun et al . ( 2018 ) show that Θpnq parameters are sufficient to memorize Θpnq training points for ReLU nets with at least 3 layers , and Park et al . ( 2020 ) reduce the parameter requirement to sublinear in n. Similar results have been proven for residual architectures ( Hardt & Ma , 2016 ) and convolutional nets ( Nguyen & Hein , 2018 ) . Bartlett et al . ( 2019 ) analyze the VC-dimension of neural nets , leading to upper and lower bounds on the parameter count needed to fit training data . Other works study expressivity via connections to tensor approximation and sum-product networks ( Cohen & Shashua , 2016 ; Cohen et al. , 2016 ) . There is a long line of work on studying the ability of neural nets to recognize and represent formal languages . The seminal work of Siegelmann & Sontag ( 1995 ) shows that RNNs are Turing complete but leverages infinite precision in the hidden activations . Chen et al . ( 2018 ) extend this result to ReLU activations and study implications in language modeling . Many variants of transformers are shown to be Turing-complete , but these constructions also rely on arbitrary precision ( Pérez et al. , 2019 ; Bhattamishra et al. , 2020 ) . A number of recent works have also proven results for generating or recognizing formal languages with finite-precision neural nets ( Weiss et al. , 2018 ; Korsky & Berwick , 2019 ; Hewitt et al. , 2020 ) , but these results do not consider Turing machines or analyze statistical properties of their constructions . Bounding the sample complexity of SM approximation requires additional complications in both the construction and statistical analysis . 1.2 NOTATION . Let f˝g denote the composition of functions f and g. For a family of functions G , let f˝Gfitf˝g : gPGu denote the family of compositions between f and functions in G. For a set S and function f : SÑY , let fpSq denote the set tfpsq : sPSuĎY . We use 1d to denote the all-one ’ s vector in d dimensions , with the subscripted omitted if clear . For iPrds , we let 1dpiq denote the one-hot embedding in d-dimensions , which is 1 at index i and 0 everywhere else . We use the notation rOp¨q to hide poly-logarithmic factors in the argument . The notationÀ , Á indicates the existence of a constant factor such that the inequality holds . — denotes that theÁ andÀ relations simultaneously hold . We use polyp¨q to indicate the existence of a polynomial in the argument which makes the equation true . For a setA ( e.g. , the set of alphabet symbols for a Turing machine ) let A˚ denote the set of all sequences of elements of A , where sequence length can vary . Let P denote a distribution over a space of inputs X . Let ξ1 , ... , ξn be n i.i.d . Rademacher variables sampled from t´1 , ` 1u . The expected n-sample Rademacher complexity of F on P is as follows : Radn , P pFqfi E pxiqni “ 1 i.i.d „ P “ Eξ1 , ... , ξn “ supFPF 1 n řn i “ 1ξiFpxiq ‰‰ , where pxiqni “ 1 denotes n i.i.d . samples from P .

Summary: The paper proposes a novel concept called statistically meaningful approximation. Roughly speaking, SM approximation means that performing empirical risk minimization over some properly chosen surrogate loss would result in a hypothesis with small population risk. Then the paper develops necessary tools and proves two results: (1). fully-connected neural networks can SM approximate boolean circuits with a sample size that is poly in the number of gates and $\log$ in the width and depth of the circuit; (2). transformers can SM approximate Turing machines with a sample size that is poly in the number of states, alphabet size, and the log of the computation steps.