Geometry of Program Synthesis

1 INTRODUCTION . The idea of program synthesis dates back to the birth of modern computation itself ( Turing , 1948 ) and is recognised as one of the most important open problems in computer science ( Gulwani et al. , 2017 ) . However , there appear to be serious obstacles to synthesising programs by gradient descent at scale ( Neelakantan et al. , 2016 ; Kaiser & Sutskever , 2016 ; Bunel et al. , 2016 ; Gaunt et al. , 2016 ; Evans & Grefenstette , 2018 ; Chen et al. , 2018 ) and these problems suggest that it would be appropriate to make a fundamental study of the geometry of loss surfaces in program synthesis , since this geometry determines the learning process . To that end , in this paper we explain a new point of view on program synthesis using the singular learning theory of Watanabe ( 2009 ) and the smooth relaxation of Turing machines from Clift & Murfet ( 2018 ) . In broad strokes this new geometric point of view on program synthesis says : • Programs to be synthesised are singularities of analytic functions . If U ⊆ Rd is open and K : U −→ R is analytic , then x ∈ U is a critical point of K if ∇K ( x ) = 0 and a singularity of the function K if it is a critical point where K ( x ) = 0 . • The Kolmogorov complexity of a program is related to a geometric invariant of the associated singularity called the Real Log Canonical Threshold ( RLCT ) . This invariant controls both the generalisation error and the learning process , and is therefore an appropriate measure of “ complexity ” in continuous program synthesis . See Section 3 . • The geometry has concrete practical implications . For example , a MCMC-based approach to program synthesis will find , with high probability , a solution that is of low complexity ( if it finds a solution at all ) . We sketch a novel point of view on the problem of “ bad local minima ” ( Gaunt et al. , 2016 ) based on these ideas . See Section 4 . We demonstrate all of these principles in experiments with toy examples of synthesis problems . Program synthesis as inference . We use Turing machines , but mutatis mutandis everything applies to other programming languages . Let T be a Turing machine with tape alphabet Σ and set of states Q and assume that on any input x ∈ Σ∗ the machine eventually halts with output T ( x ) ∈ Σ∗ . Then to the machine T we may associate the set { ( x , T ( x ) ) } x∈Σ∗ ⊆ Σ∗ × Σ∗ . Program synthesis is the study of the inverse problem : given a subset of Σ∗ × Σ∗ we would like to determine ( if possible ) a Turing machine which computes the given outputs on the given inputs . If we presume given a probability distribution q ( x ) on Σ∗ then we can formulate this as a problem of statistical inference : given a probability distribution q ( x , y ) on Σ∗ × Σ∗ determine the most likely machine producing the observed distribution q ( x , y ) = q ( y|x ) q ( x ) . If we fix a universal Turing machine U then Turing machines can be parametrised by codes w ∈ W code with U ( x , w ) = T ( x ) for all x ∈ Σ∗ . We let p ( y|x , w ) denote the probability of U ( x , w ) = y ( which is either zero or one ) so that solutions to the synthesis problem are in bijection with the zeros of the Kullback-Leibler divergence between the true distribution and the model K ( w ) = ∫ ∫ q ( y|x ) q ( x ) log q ( y|x ) p ( y|x , w ) dxdy . ( 1 ) So far this is just a trivial rephrasing of the combinatorial optimisation problem of finding a Turing machine T with T ( x ) = y for all ( x , y ) with q ( x , y ) > 0 . Smooth relaxation . One approach is to seek a smooth relaxation of the synthesis problem consisting of an analytic manifold W ⊇ W code and an extension of K to an analytic function K : W −→ R so that we can search for the zeros of K using gradient descent . Perhaps the most natural way to construct such a smooth relaxation is to takeW to be a space of probability distributions overW code and prescribe a model p ( y|x , w ) for propagating uncertainty about codes to uncertainty about outputs ( Gaunt et al. , 2016 ; Evans & Grefenstette , 2018 ) . The particular model we choose is based on the semantics of linear logic ( Clift & Murfet , 2018 ) . Supposing that such a smooth relaxation has been chosen together with a prior ϕ ( w ) over W , smooth program synthesis becomes the study of the statistical learning theory of the triple ( p , q , ϕ ) . There are perhaps two primary reasons to consider the smooth relaxation . Firstly , one might hope that stochastic gradient descent or techniques like Markov chain Monte Carlo will be effective means of solving the original combinatorial optimisation problem . This is not a new idea ( Gulwani et al. , 2017 , §6 ) but so far its effectiveness for large programs has not been proven . Independently , one might hope to find powerful new mathematical ideas that apply to the relaxed problem and shed light on the nature of program synthesis . This is the purpose of the present paper . Singular learning theory . We denote by W0 = { w ∈W |K ( w ) = 0 } so that W0 ∩W code ⊆W0 ⊆W ( 2 ) whereW0∩W code is the discrete set of solutions to the original synthesis problem . We refer to these as the classical solutions . As the vanishing locus of an analytic function , W0 is an analytic space over R ( Hironaka , 1964 , §0.1 ) , ( Griffith & Harris , 1978 ) and it is interesting to study the geometry of this space near the classical solutions . Since K is a Kullback-Leibler divergence it is non-negative and so it not only vanishes onW0 but∇K also vanishes , hence every point ofW0 is a singular point . Beyond this the geometry ofW0 depends on the particular model p ( y|x , w ) that has been chosen , but some aspects are universal : the nature of program synthesis means that typically W0 is an extended object ( i.e . it contains points other than the classical solutions ) and the Hessian matrix of second order partial derivatives of K at a classical solution is not invertible - that is , the classical solutions are degenerate critical points of K. This means that singularity theory is the appropriate branch of mathematics for studying the geometry of W0 near a classical solution . It also means that the Fisher information matrix I ( w ) ij = ∫ ∫ ∂ ∂wi [ log p ( y|x , w ) ] ∂ ∂wj [ log p ( y|x , w ) ] q ( y|x ) q ( x ) dxdy , is degenerate at a classical solution , so that the appropriate branch of statistical learning theory is singular learning theory ( Watanabe , 2007 ; 2009 ) . For an introduction to singular learning theory in the context of deep learning see ( Murfet et al. , 2020 ) . Broadly speaking the contribution of this paper is to realise program synthesis within the framework of singular learning theory , at both a theoretical and an experimental level . In more detail the contents of the paper are : • We define a staged pseudo-UTM ( Appendix E ) which is well-suited to experiments with the ideas discussed above . Propagating uncertainty about the code through this UTM using the ideas of ( Clift & Murfet , 2018 ) defines a triple ( p , q , ϕ ) associated to a synthesis problem . This formally embeds program synthesis within singular learning theory . • We realise this embedding in code by providing an implementation in PyTorch of this propagation of uncertainty through a UTM . Using the No-U-Turn variant of MCMC ( Hoffman & Gelman , 2014 ) we can approximate the Bayesian posterior of any program synthesis problem ( of course in practice we are limited by computational constraints in doing so ) . • We explain how the real log canonical threshold ( a geometric invariant ) is related to Kolmogorov complexity ( Section 3 ) . • We give a simple example ( Appendix C ) in whichW0 contains the set of classical solutions as a proper subset and every point of W0 is a degenerate critical point of K. • For two simple synthesis problems detectA and parityCheck we demonstrate all of the above , using MCMC to approximate the Bayesian posterior and theorems from Watanabe ( 2013 ) to estimate the RLCT ( Section 5 ) . We discuss how W0 is an extended object and how the RLCT relates to the local dimension of W0 near a classical solution . RELATED WORK The idea of synthesising Turing machines can be traced back to the work of Solomonoff on inductive inference ( Solomonoff , 1964 ) . A more explicit form of the problem was given in Biermann ( 1972 ) who proposed an algorithmic method . Machine learning based approaches appear in Schmidhuber ( 1997 ) and Hutter ( 2004 ) , which pay particular attention to model complexity , and Gaunt et al . ( 2016 ) and Freer et al . ( 2014 ) , the latter using the notion of “ universal probabilistic Turing machine ” ( De Leeuw et al. , 1956 ) . A different probabilistic extension of a universal Turing machine was introduced in Clift & Murfet ( 2018 ) via linear logic . Studies of the singular geometry of learning models go back to Amari et al . ( 2003 ) and notably , the extensive work of Watanabe ( 2007 ; 2009 ) . 2 TURING MACHINE SYNTHESIS AS SINGULAR LEARNING . All known approaches to program synthesis can be formulated in terms of a singular learning problem . Singular learning theory is the extension of statistical learning theory to account for the fact that the set of learned parameters W0 has the structure of an analytic space as opposed to an analytic manifold ( Watanabe , 2007 ; 2009 ) . It is organised around triples ( p , q , ϕ ) consisting of a class of models { p ( y|x , w ) : w ∈W } , a true distribution q ( y|x ) and a prior ϕ on W . In our approach we fix a Universal Turing Machine ( UTM ) , denoted U , with a description tape ( which specifies the code of the Turing machine to be executed ) , a work tape ( simulating the tape of that Turing machine during its operation ) and a state tape ( simulating the state of that Turing machine ) . The general statistical learning problem that can be formulated using U is the following : given some initial string x on the work tape , predict the state of the simulated machine and the contents of the work tape after some specified number of steps ( Clift & Murfet , 2018 , §7.1 ) . For simplicity , in this paper we consider models that only predict the final state ; the necessary modifications in the general case are routine . We also assume that W parametrises Turing machines whose tape alphabet Σ and set of states Q have been encoded by individual symbols in the tape alphabet of U . Hence U is actually what we call a pseudo-UTM ( see Appendix E ) . Again , treating the general case is routine and for the present purposes only introduces uninteresting complexity . Let Σ denote the tape alphabet of the simulated machine , Q the set of states and let L , S , R stand for left , stay and right , the possible motions of the Turing machine head . We assume that |Q| > 1 since otherwise the synthesis problem is trivial . The set of ordinary codes W code for a Turing machine sits inside a compact space of probability distributions W over codes W code : = ∏ σ , q Σ×Q× { L , S , R } ⊆ ∏ σ , q ∆Σ×∆Q×∆ { L , S , R } = : W ( 3 ) where ∆X denotes the set of probability distributions over a set X , see ( 8 ) , and the product is over pairs ( σ , q ) ∈ Σ×Q.1 For example the point { ( σ′ , q′ , d ) } σ , q ∈ W code encodes the machine which when it reads σ under the head in state q writes σ′ , transitions into state q′ and moves in direction d. Given w ∈ W code let stept ( x , w ) ∈ Q denote the contents of the state tape of U after t timesteps ( of the simulated machine ) when the work tape is initialised with x and the description tape with w. 1The space W of parameters is clearly semi-analytic , that is , it is cut out of Rd for some d by the vanishing f1 ( x ) = · · · = fr ( x ) = 0 of finitely many analytic functions on open subsets of Rd together with finitely many inequalities g1 ( x ) ≥ 0 , . . . , gs ( x ) ≥ 0 where the gj ( x ) are analytic . In fact W is semi-algebraic , since the fi and gj may all be chosen to be polynomial functions . Under review as a conference paper at ICLR 2021 There is a principled extension of this operation of U to a smooth function ∆ stept : Σ∗ ×W −→ ∆Q ( 4 ) which propagates uncertainty about the symbols on the description tape to uncertainty about the final state and we refer to this extension as the smooth relaxation of U . The details are given in Appendix F but at an informal level the idea behind the relaxation is easy to understand : to sample from ∆ stept ( x , w ) we run U to simulate t timesteps in such a way that whenever the UTM needs to “ look at ” an entry on the description tape we sample from the corresponding distribution specified by w.2 The significance of the particular smooth relaxation that we use is that its derivatives have a logical interpretation ( Clift & Murfet , 2018 , §7.1 ) . The class of models that we consider is p ( y|x , w ) = ∆ stept ( x , w ) ( 5 ) where t is fixed for simplicity in this paper . More generally we could also view x as consisting of a sequence and a timeout , as is done in ( Clift & Murfet , 2018 , §7.1 ) . The construction of this model is summarised in Figure 1. w or k < latexit sha1_base64= '' TJWz1OU75YFDdX7wl0/SZiBZv6w= '' > AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyGiB4DXjxGMA9IljA7mSRD5rHMzCphyS948aCIV3/Im3/jbLIHTSxoKKq66e6KYs6M9f1vr7CxubW9U9wt7e0fHB6Vj0/aRiWa0BZRXOluhA3lTNKWZZbTbqwpFhGnnWh6m/mdR6oNU/LBzmIaCjyWbMQItpn0pPR0UK74VX8BtE6CnFQgR3NQ/uoPFUkElZZwbEwv8GMbplhbRjidl/qJoTEmUzymPUclFtSE6eLWObpwyhCNlHYlLVqovydSLIyZich1CmwnZtXLxP+8XmJHN2HKZJxYKsly0SjhyCqUPY6GTFNi+cwRTDRztyIywRoT6+IpuRCC1ZfXSbtWDerVq/tapVHP4yjCGZzDJQRwDQ24gya0gMAEnuEV3jzhvXjv3seyteDlM6fwB97nD0s7jl0= < /latexit > st at e < latexit sha1_base64= '' SC7T0UNtxJZeOTT+l5BLbIY5Owg= '' > AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lKRY8FLx4rmFZoQ9lsN+3SzSbsToQS+hu8eFDEqz/Im//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzSMUmmGfdZIhP9GFLDpVDcR4GSP6aa0ziUvBtObud+94lrIxL1gNOUBzEdKREJRtFKvkGKfFCtuXV3AbJOvILUoEB7UP3qDxOWxVwhk9SYnuemGORUo2CSzyr9zPCUsgkd8Z6lisbcBPni2Bm5sMqQRIm2pZAs1N8TOY2Nmcah7Ywpjs2qNxf/83oZRjdBLlSaIVdsuSjKJMGEzD8nQ6E5Qzm1hDIt7K2EjammDG0+FRuCt/ryOuk06l6zfnXfqLWaRRxlOINzuAQPrqEFd9AGHxgIeIZXeHOU8+K8Ox/L1pJTzJzCHzifPwBVjsU= < /latexit > co d e < latexit sha1_base64= '' om3Rg/XzVobOPXTCvLtEyjMGzbs= '' > AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyGiB4DXjxGMImQLGF2dpIMmccyMyuEJb/gxYMiXv0hb/6Ns8keNLGgoajqprsrSjgz1ve/vdLG5tb2Tnm3srd/cHhUPT7pGpVqQjtEcaUfI2woZ5J2LLOcPiaaYhFx2oumt7nfe6LaMCUf7CyhocBjyUaMYJtLRMV0WK35dX8BtE6CgtSgQHtY/RrEiqSCSks4NqYf+IkNM6wtI5zOK4PU0ASTKR7TvqMSC2rCbHHrHF04JUYjpV1Jixbq74kMC2NmInKdAtuJWfVy8T+vn9rRTZgxmaSWSrJcNEo5sgrlj6OYaUosnzmCiWbuVkQmWGNiXTwVF0Kw+vI66TbqQbN+dd+otZpFHGU4g3O4hACuoQV30IYOEJjAM7zCmye8F+/d+1i2lrxi5hT+wPv8AQ5RjjU= < /latexit > w < latexit sha1_base64= '' R7WKE1UfW8UlgiOq9cSzJJOgiHo= '' > AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHYJRo8kXjxCIo8ENmR2aGBkdnYzM6shG77AiweN8eonefNvHGAPClbSSaWqO91dQSy4Nq777eQ2Nre2d/K7hb39g8Oj4vFJS0eJYthkkYhUJ6AaBZfYNNwI7MQKaRgIbAeT27nffkSleSTvzTRGP6QjyYecUWOlxlO/WHLL7gJknXgZKUGGer/41RtELAlRGiao1l3PjY2fUmU4Ezgr9BKNMWUTOsKupZKGqP10ceiMXFhlQIaRsiUNWai/J1Iaaj0NA9sZUjPWq95c/M/rJmZ446dcxolByZaLhokgJiLzr8mAK2RGTC2hTHF7K2FjqigzNpuCDcFbfXmdtCplr1q+alRKtWoWRx7O4BwuwYNrqMEd1KEJDBCe4RXenAfnxXl3PpatOSebOYU/cD5/AOJ3jPM= < /latexit > x < latexit sha1_base64= '' tfWAW1SjmyNhrA0capfB+UlJ15k= '' > AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHYJRo8kXjxCIo8ENmR2aGBkdnYzM2skG77AiweN8eonefNvHGAPClbSSaWqO91dQSy4Nq777eQ2Nre2d/K7hb39g8Oj4vFJS0eJYthkkYhUJ6AaBZfYNNwI7MQKaRgIbAeT27nffkSleSTvzTRGP6QjyYecUWOlxlO/WHLL7gJknXgZKUGGer/41RtELAlRGiao1l3PjY2fUmU4Ezgr9BKNMWUTOsKupZKGqP10ceiMXFhlQIaRsiUNWai/J1Iaaj0NA9sZUjPWq95c/M/rJmZ446dcxolByZaLhokgJiLzr8mAK2RGTC2hTHF7K2FjqigzNpuCDcFbfXmdtCplr1q+alRKtWoWRx7O4BwuwYNrqMEd1KEJDBCe4RXenAfnxXl3PpatOSebOYU/cD5/AOP7jPQ= < /latexit > y < latexit sha1_base64= '' GM5ZmXW1PJCOVOCvnUffSh1cn3U= '' > AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4Kkmp6LHgxWML9gPaUDbbSbt2swm7GyGU/gIvHhTx6k/y5r9x2+agrQ8GHu/NMDMvSATXxnW/nY3Nre2d3cJecf/g8Oi4dHLa1nGqGLZYLGLVDahGwSW2DDcCu4lCGgUCO8Hkbu53nlBpHssHkyXoR3QkecgZNVZqZoNS2a24C5B14uWkDDkag9JXfxizNEJpmKBa9zw3Mf6UKsOZwFmxn2pMKJvQEfYslTRC7U8Xh87IpVWGJIyVLWnIQv09MaWR1lkU2M6ImrFe9ebif14vNeGtP+UySQ1KtlwUpoKYmMy/JkOukBmRWUKZ4vZWwsZUUWZsNkUbgrf68jppVyterXLdrJbrtTyOApzDBVyBBzdQh3toQAsYIDzDK7w5j86L8+58LFs3nHzmDP7A+fwB5X+M9Q== < /latexit > step < latexit sha1_base64= '' NI9QQ7XStFY2oO5uCpCo3hbaGCA= '' > AAACBXicbVA9SwNBEN3zM8avqKUWi4lgFe5CRMuAFpYRzAckIextJsmSvdtjd04IRxob/4qNhSK2/gc7/417SQpNfDDweG+GmXl+JIVB1/12VlbX1jc2M1vZ7Z3dvf3cwWHdqFhzqHEllW76zIAUIdRQoIRmpIEFvoSGP7pO/cYDaCNUeI/jCDoBG4SiLzhDK3VzJ4X2DUhktK0i0AyVDlkAiUGIJoVuLu8W3SnoMvHmJE/mqHZzX+2e4nEAIXLJjGl5boSdhGkUXMIk244NRIyP2ABalqarTCeZfjGhZ1bp0b7StkKkU/X3RMICY8aBbzsDhkOz6KXif14rxv5VJxFhFCOEfLaoH0uKiqaR0J7QwFGOLWFcC3sr5UOmGUcbXNaG4C2+vEzqpaJXLl7clfKV8jyODDkmp+SceOSSVMgtqZIa4eSRPJNX8uY8OS/Ou/Mxa11x5jNH5A+czx92LpiG < /latexit > init < latexit sha1_base64= '' eVVGLMwSrwOU7JIqXWoswyQdy7Y= '' > AAAB/nicbVBNSwMxEM36WevXqnjyEmwFT2W3VPRY8OKxgv2AdinZNG1Ds8mSzAplKfhXvHhQxKu/w5v/xmy7B219MPB4b4aZeWEsuAHP+3bW1jc2t7YLO8Xdvf2DQ/fouGVUoilrUiWU7oTEMMElawIHwTqxZiQKBWuHk9vMbz8ybbiSDzCNWRCRkeRDTglYqe+elnsqZpqA0pJELOWSw6zcd0texZsDrxI/JyWUo9F3v3oDRZOISaCCGNP1vRiClGjgVLBZsZcYFhM6ISPWtTRbZYJ0fv4MX1hlgIdK25KA5+rviZRExkyj0HZGBMZm2cvE/7xuAsObwL4UJ8AkXSwaJgKDwlkWeMA1oyCmlhCqub0V0zHRhIJNrGhD8JdfXiWtasWvVa7uq6V6LY+jgM7QObpEPrpGdXSHGqiJKErRM3pFb86T8+K8Ox+L1jUnnzlBf+B8/gCKDZXS < /latexit > w or k < latexit sha1_base64= '' TJWz1OU75YFDdX7wl0/SZiBZv6w= '' > AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyGiB4DXjxGMA9IljA7mSRD5rHMzCphyS948aCIV3/Im3/jbLIHTSxoKKq66e6KYs6M9f1vr7CxubW9U9wt7e0fHB6Vj0/aRiWa0BZRXOluhA3lTNKWZZbTbqwpFhGnnWh6m/mdR6oNU/LBzmIaCjyWbMQItpn0pPR0UK74VX8BtE6CnFQgR3NQ/uoPFUkElZZwbEwv8GMbplhbRjidl/qJoTEmUzymPUclFtSE6eLWObpwyhCNlHYlLVqovydSLIyZich1CmwnZtXLxP+8XmJHN2HKZJxYKsly0SjhyCqUPY6GTFNi+cwRTDRztyIywRoT6+IpuRCC1ZfXSbtWDerVq/tapVHP4yjCGZzDJQRwDQ24gya0gMAEnuEV3jzhvXjv3seyteDlM6fwB97nD0s7jl0= < /latexit > st at e < latexit sha1_base64= '' SC7T0UNtxJZeOTT+l5BLbIY5Owg= '' > AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lKRY8FLx4rmFZoQ9lsN+3SzSbsToQS+hu8eFDEqz/Im//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzSMUmmGfdZIhP9GFLDpVDcR4GSP6aa0ziUvBtObud+94lrIxL1gNOUBzEdKREJRtFKvkGKfFCtuXV3AbJOvILUoEB7UP3qDxOWxVwhk9SYnuemGORUo2CSzyr9zPCUsgkd8Z6lisbcBPni2Bm5sMqQRIm2pZAs1N8TOY2Nmcah7Ywpjs2qNxf/83oZRjdBLlSaIVdsuSjKJMGEzD8nQ6E5Qzm1hDIt7K2EjammDG0+FRuCt/ryOuk06l6zfnXfqLWaRRxlOINzuAQPrqEFd9AGHxgIeIZXeHOU8+K8Ox/L1pJTzJzCHzifPwBVjsU= < /latexit > co d e < latexit sha1_base64= '' om3Rg/XzVobOPXTCvLtEyjMGzbs= '' > AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyGiB4DXjxGMImQLGF2dpIMmccyMyuEJb/gxYMiXv0hb/6Ns8keNLGgoajqprsrSjgz1ve/vdLG5tb2Tnm3srd/cHhUPT7pGpVqQjtEcaUfI2woZ5J2LLOcPiaaYhFx2oumt7nfe6LaMCUf7CyhocBjyUaMYJtLRMV0WK35dX8BtE6CgtSgQHtY/RrEiqSCSks4NqYf+IkNM6wtI5zOK4PU0ASTKR7TvqMSC2rCbHHrHF04JUYjpV1Jixbq74kMC2NmInKdAtuJWfVy8T+vn9rRTZgxmaSWSrJcNEo5sgrlj6OYaUosnzmCiWbuVkQmWGNiXTwVF0Kw+vI66TbqQbN+dd+otZpFHGU4g3O4hACuoQV30IYOEJjAM7zCmye8F+/d+1i2lrxi5hT+wPv8AQ5RjjU= < /latexit > w or k < latexit sha1_base64= '' TJWz1OU75YFDdX7wl0/SZiBZv6w= '' > AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyGiB4DXjxGMA9IljA7mSRD5rHMzCphyS948aCIV3/Im3/jbLIHTSxoKKq66e6KYs6M9f1vr7CxubW9U9wt7e0fHB6Vj0/aRiWa0BZRXOluhA3lTNKWZZbTbqwpFhGnnWh6m/mdR6oNU/LBzmIaCjyWbMQItpn0pPR0UK74VX8BtE6CnFQgR3NQ/uoPFUkElZZwbEwv8GMbplhbRjidl/qJoTEmUzymPUclFtSE6eLWObpwyhCNlHYlLVqovydSLIyZich1CmwnZtXLxP+8XmJHN2HKZJxYKsly0SjhyCqUPY6GTFNi+cwRTDRztyIywRoT6+IpuRCC1ZfXSbtWDerVq/tapVHP4yjCGZzDJQRwDQ24gya0gMAEnuEV3jzhvXjv3seyteDlM6fwB97nD0s7jl0= < /latexit > st at e < latexit sha1_base64= '' SC7T0UNtxJZeOTT+l5BLbIY5Owg= '' > AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lKRY8FLx4rmFZoQ9lsN+3SzSbsToQS+hu8eFDEqz/Im//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzSMUmmGfdZIhP9GFLDpVDcR4GSP6aa0ziUvBtObud+94lrIxL1gNOUBzEdKREJRtFKvkGKfFCtuXV3AbJOvILUoEB7UP3qDxOWxVwhk9SYnuemGORUo2CSzyr9zPCUsgkd8Z6lisbcBPni2Bm5sMqQRIm2pZAs1N8TOY2Nmcah7Ywpjs2qNxf/83oZRjdBLlSaIVdsuSjKJMGEzD8nQ6E5Qzm1hDIt7K2EjammDG0+FRuCt/ryOuk06l6zfnXfqLWaRRxlOINzuAQPrqEFd9AGHxgIeIZXeHOU8+K8Ox/L1pJTzJzCHzifPwBVjsU= < /latexit > co d e < latexit sha1_base64= '' om3Rg/XzVobOPXTCvLtEyjMGzbs= '' > AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyGiB4DXjxGMImQLGF2dpIMmccyMyuEJb/gxYMiXv0hb/6Ns8keNLGgoajqprsrSjgz1ve/vdLG5tb2Tnm3srd/cHhUPT7pGpVqQjtEcaUfI2woZ5J2LLOcPiaaYhFx2oumt7nfe6LaMCUf7CyhocBjyUaMYJtLRMV0WK35dX8BtE6CgtSgQHtY/RrEiqSCSks4NqYf+IkNM6wtI5zOK4PU0ASTKR7TvqMSC2rCbHHrHF04JUYjpV1Jixbq74kMC2NmInKdAtuJWfVy8T+vn9rRTZgxmaSWSrJcNEo5sgrlj6OYaUosnzmCiWbuVkQmWGNiXTwVF0Kw+vI66TbqQbN+dd+otZpFHGU4g3O4hACuoQV30IYOEJjAM7zCmye8F+/d+1i2lrxi5hT+wPv8AQ5RjjU= < /latexit > · · · < latexit sha1_base64= '' 0bCcXyz6HriuD2v71FSD+L8Alno= '' > AAAB73icbVBNSwMxEJ2tX7V+VT16CbaCp7JbKnosePFYwX5Au5RsNtuGZpM1yQpl6Z/w4kERr/4db/4b03YP2vpg4PHeDDPzgoQzbVz32ylsbG5t7xR3S3v7B4dH5eOTjpapIrRNJJeqF2BNORO0bZjhtJcoiuOA024wuZ373SeqNJPiwUwT6sd4JFjECDZW6lUHJJRGV4fliltzF0DrxMtJBXK0huWvQShJGlNhCMda9z03MX6GlWGE01lpkGqaYDLBI9q3VOCYaj9b3DtDF1YJUSSVLWHQQv09keFY62kc2M4Ym7Fe9ebif14/NdGNnzGRpIYKslwUpRwZiebPo5ApSgyfWoKJYvZWRMZYYWJsRCUbgrf68jrp1Gteo3Z1X680G3kcRTiDc7gED66hCXfQgjYQ4PAMr/DmPDovzrvzsWwtOPnMKfyB8/kDZpuPgw== < /latexit > step < latexit sha1_base64= '' NI9QQ7XStFY2oO5uCpCo3hbaGCA= '' > AAACBXicbVA9SwNBEN3zM8avqKUWi4lgFe5CRMuAFpYRzAckIextJsmSvdtjd04IRxob/4qNhSK2/gc7/417SQpNfDDweG+GmXl+JIVB1/12VlbX1jc2M1vZ7Z3dvf3cwWHdqFhzqHEllW76zIAUIdRQoIRmpIEFvoSGP7pO/cYDaCNUeI/jCDoBG4SiLzhDK3VzJ4X2DUhktK0i0AyVDlkAiUGIJoVuLu8W3SnoMvHmJE/mqHZzX+2e4nEAIXLJjGl5boSdhGkUXMIk244NRIyP2ABalqarTCeZfjGhZ1bp0b7StkKkU/X3RMICY8aBbzsDhkOz6KXif14rxv5VJxFhFCOEfLaoH0uKiqaR0J7QwFGOLWFcC3sr5UOmGUcbXNaG4C2+vEzqpaJXLl7clfKV8jyODDkmp+SceOSSVMgtqZIa4eSRPJNX8uY8OS/Ou/Mxa11x5jNH5A+czx92LpiG < /latexit > step < latexit sha1_base64= '' NI9QQ7XStFY2oO5uCpCo3hbaGCA= '' > AAACBXicbVA9SwNBEN3zM8avqKUWi4lgFe5CRMuAFpYRzAckIextJsmSvdtjd04IRxob/4qNhSK2/gc7/417SQpNfDDweG+GmXl+JIVB1/12VlbX1jc2M1vZ7Z3dvf3cwWHdqFhzqHEllW76zIAUIdRQoIRmpIEFvoSGP7pO/cYDaCNUeI/jCDoBG4SiLzhDK3VzJ4X2DUhktK0i0AyVDlkAiUGIJoVuLu8W3SnoMvHmJE/mqHZzX+2e4nEAIXLJjGl5boSdhGkUXMIk244NRIyP2ABalqarTCeZfjGhZ1bp0b7StkKkU/X3RMICY8aBbzsDhkOz6KXif14rxv5VJxFhFCOEfLaoH0uKiqaR0J7QwFGOLWFcC3sr5UOmGUcbXNaG4C2+vEzqpaJXLl7clfKV8jyODDkmp+SceOSSVMgtqZIa4eSRPJNX8uY8OS/Ou/Mxa11x5jNH5A+czx92LpiG < /latexit > Figure 1 : The state of U is represented by the state of the work tape , state tape and description ( code ) tape . The work tape is initialised with a sequence x ∈ Σ∗ , the code tape with w ∈ W and the state tape with some standard initial state , the smooth relaxation ∆ step of the pseudo-UTM is run for t steps and the final probability distribution over states is y . Definition 2.1 ( Synthesis problem ) . A synthesis problem for U consists of a probability distribution q ( x , y ) over Σ∗ × Q . We say that the synthesis problem is deterministic if there is f : Σ∗ −→ Q such that q ( y = f ( x ) |x ) = 1 for all x ∈ Σ∗ . Definition 2.2 . The triple ( p , q , ϕ ) associated to a synthesis problem is the model p of ( 5 ) together with the true distribution q and uniform prior ϕ on the parameter space W . The Kullback-Leibler function K ( w ) of the synthesis problem is defined by ( 1 ) and a solution to the synthesis problem is a point of W0 . A classical solution is a point of W0 ∩W code . As ∆ stept is a polynomial function , K is analytic and so W0 is a semi-analytic space ( it is cut out of the semi-analytic space W by the vanishing of K ) . If the synthesis problem is deterministic and q ( x ) is uniform on some finite subset of Σ∗ then W0 is semi-algebraic ( it is cut out of W by polynomial equations ) and all solutions lie at the boundary of the parameter spaceW ( Appendix D ) . However in general W0 is only semi-analytic and intersects the interior of W ( Example C.2 ) . We assume that q ( y|x ) is realisable that is , there exists w0 ∈W with q ( y|x ) = p ( y|x , w0 ) . A triple ( p , q , ϕ ) is regular if the model is identifiable , ie . for all inputs x ∈ Rn , the map sending w to the conditional probability distribution p ( y|x , w ) is one-to-one , and the Fisher information matrix is non-degenerate . Otherwise , the learning machine is strictly singular ( Watanabe , 2009 , §1.2.1 ) . Triples arising from synthesis problems are typically singular : in Example 2.5 below we show an explicit example where multiple parameters w determine the same model , and in Example C.2 we give an example where the Hessian ofK is degenerate everywhere on W0 ( Watanabe , 2009 , §1.1.3 ) . 2Noting that this sampling procedure is repeated every time the UTM looks at a given entry . Remark 2.3 . Non-deterministic synthesis problems arise naturally in various contexts , for example in the fitting of algorithms to the behaviour of deep reinforcement learning agents . Suppose an agent is acting in an environment with starting states encoded by x ∈ Σ∗ and possible episode end states by y ∈ Q . Even if the optimal policy is known to determine a computable function Σ∗ −→ Q the statistics of the observed behaviour after finite training time will only provide a function Σ∗ −→ ∆Q and if we wish to fit algorithms to behaviour it makes sense to deal with this uncertainty directly . Definition 2.4 . Let ( p , q , ϕ ) be the triple associated to a synthesis problem . The Real Log Canonical Threshold ( RLCT ) λ of the synthesis problem is defined so that −λ is the largest pole of the meromorphic extension ( Atiyah , 1970 ) of the zeta function ζ ( z ) = ∫ K ( w ) zϕ ( w ) dw . The more singular the analytic space W0 of solutions is , the smaller the RLCT . One way to think of the RLCT is as a count of the effective number of parameters near W0 ( Murfet et al. , 2020 , §4 ) . In Section 3 we relate the RLCT to Kolmogorov complexity and in Section 5 we estimate the RLCT of the synthesis problem detectA given below , using the method explained in Appendix A . Example 2.5 ( detectA ) . The deterministic synthesis problem detectA has Σ = { , A , B } , Q = { reject , accept } and q ( y|x ) is determined by the function taking in a string x of A ’ s and B ’ s and returning the state accept if the string contains an A and state reject otherwise . The conditional true distribution q ( y|x ) is realisable because this function is computed by a Turing machine . Two solutions are shown in Figure 2 . On the left is a parameter wl ∈ W0 \ W code and on the right is wr ∈ W0 ∩W code . Varying the distributions in wl that have nonzero entropy we obtain a submanifold V ⊆ W0 containing wl of dimension 14 . This leads by ( Watanabe , 2009 , Remark 7.3 ) to a bound on the RLCT of λ ≤ 12 ( 30 − 14 ) = 8 which is consistent with the experimental results in Table 1 . This highlights that solutions need not lie at vertices of the probability simplex , and W0 may contain a high-dimensional submanifold around a given classical solution .

Summary: The authors apply algebraic geometry to program synthesis, by identifying programs with points of analytic varieties. They construct a smooth relaxation to the synthesis problem by considering the space of probability distributions over codes for universal turning machines (which is a smooth/continuous manifold), and then translate this probability into corresponding probabilities of generating a correct program. This allows them to extend the KL divergence on the discrete distribution of codes to a smooth function, whose zeros correspond to the desired programs. They use MCMC to find (approximate) these zeros.