Differentiable Synthesis of Program Architectures

1 Introduction . Program synthesis has recently emerged as an effective approach to address tasks in several fields where deep learning is applied traditionally . A synthesized program in a domain-specific language ( DSL ) provides a powerful abstraction for summarizing discovered knowledge from data and offers greater interpretability and transferability across tasks than a deep neural network model , while achieving competitive task performance [ 1–4 ] . A differentiable program encourages interpretability by using structured symbolic primitives to compose a set of differentiable modules with trainable parameters in its architecture . These parameters can be efficiently learned with respect to a differentiable loss function over the program ’ s outputs . However , synthesizing a reasonable program architecture remains challenging because the architecture search space is discrete and combinatorial . Various enumeration strategies have been developed to explore the program architecture space , including greedy enumeration [ 1 , 2 ] , evolutionary search [ 5 ] , and Monte Carlo sampling [ 6 ] . To prioritize highly likely top-down search directions in the combinatorial architecture space , NEAR [ 7 ] uses neural networks to approximate missing expressions in a partial program whose F1 score serves as an admissible heuristic to effective graph search algorithms such as A∗ [ 8 ] . However , since the discrete program architecture search space is intractably large , enumeration-based search strategies are inefficient in general . We propose to encode program architecture search as learning the probability distribution over all possible program architecture derivations induced by a context-free DSL grammar . This problem bears similarities with searching the structure of graphical models [ 9 ] and neural architecture search . Specifically , to support differentiable search , DARTS [ 10 ] uses a composition of softmaxes over all possible candidate operations between a fixed set of neural network nodes to relax the discrete search space of neural architectures . However , applying this method to program synthesis is challenging 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . because the program architecture search space is much richer [ 7 ] . Firstly , different sets of operations take different input and output types and may only be available at different points of a program . Secondly , there is no fixed bound on the number of expressions in a program architecture . To address the aforementioned challenges , we learn the probability distribution of program architectures in a continuous relaxation of the search space of DSL grammar rules . We conduct program architecture search in a program derivation graph , in which nodes encode architectures with missing expressions , and paths encode top-down program derivations . For each partial architecture f on a graph node , we relax the categorical choice of production rules for expanding a missing expression in f to a softmax over all possible production rules with trainable weights . A program derivation graph essentially expresses all possible program derivations under a context-free grammar up to a certain depth bound ( on the height of program abstract syntax trees ) , which can be iteratively increased during search to balance accuracy and architecture complexity . We encode a program derivation graph itself as a differentiable program whose output is weighted by the outputs of all the programs involved . We seek to optimize program architecture weights with respect to an accuracy loss function defined over the encoded program ’ s output . The learned weights allow our synthesis algorithm to efficiently prune away search directions to unlikely program derivations to discover optimal programs . Compared with enumeration-based synthesis strategies , differentiable program synthesis in the relaxed architecture space is easier and more efficient with gradient-based optimization . One major challenge of differentiable program architecture synthesis is that a program derivation graph involves an exponential number of programs and a huge set of trainable variables including architecture weights and program parameters . To curb the large program derivation search space , we introduce node sharing in program derivation graphs and iterative graph unfolding . Node sharing allows two partial architectures to share the same child nodes if the missing expressions in the two architectures can be expanded using the same grammar rules . Iterative graph unfolding allows the synthesis algorithm to construct a program derivation graph on the fly focusing on higher-quality program derivations than all the rest . These optimization strategies significantly reduce the program architecture search space , scaling differentiable program synthesis to real-world classification tasks . We evaluate our synthesis algorithm in the context of learning classifiers in sequence classification applications . We demonstrate that our algorithm substantially outperforms state-of-the-art methods for differentiable program synthesis , and can learn programmatic classifiers that are highly interpretable and are comparable to neural network models in terms of accuracy and F1-scores . As a summary , this paper makes three contributions . Firstly , we encode program synthesis as learning the probability distribution of program architectures in a continuous relaxation of the discrete space defined by programming language grammar rules , enabling differentiable program architecture search . Secondly , we instantiate differentiable program architecture synthesis with effective optimization strategies including node sharing and iterative graph unfolding , scaling it to real-world classification tasks . Lastly , we present state-of-the-art results in learning programmatic classifiers for four sequence classification applications . 2 Problem Formulation . A program in a domain-specific language ( DSL ) is a pair ( α , θ ) , where α is a discrete program architecture and θ is a vector of real-valued parameters of the program . Given a specification over the intended input-output behavior of an unknown program , program synthesis aims to discover the program ’ s architecture α and optimize the program parameters θ . In this paper , we focus on learning programmatic classifiers for sequence classification tasks [ 11 ] . We note that the proposed synthesis technique is applicable to learning any differentiable programs . Program Architecture Synthesis . A program architecture α is typically synthesized based on a context-free grammar [ 12 ] . Such a grammar consists of a set of production rules αk → { σj } Jj=0 over terminal symbols Σ and nonterminal symbols Y where αk ∈ Y and σj ∈ Σ ∪ Y . As an example , consider the context-free grammar of a DSL for sequence classification depicted in the standard Backus-Naur form [ 13 ] in Fig . 1 , adapted from [ 7 ] . A terminal in this grammar is a symbol that can appear in a program ’ s code , e.g . x and the map function symbol , while a nonterminal stands for a missing expression ( or subexpression ) , e.g . α2 and α3 . Any program in the DSL operates over a real vector or a sequence of real vector x . It may use constants c , arithmetic operations Add and Multiply , and an If-Then-Else branching construct ITE . To avoid discontinuities for α : := x | c | Add α1 α2 |Multiply α1 α2 | ITE α1 ≥ 0 α2 α3 | FS , θ ( x ) | map ( fun x1.α1 ) x | mapprefix ( fun x1.α1 ) x | fold ( fun x1.α1 ) c x | SlideWindowAvg ( fun x1.α1 ) x Figure 1 : Context-free DSL Grammar for Sequence Classification ( adapted from [ 7 ] ) . … … 0.2 0.34 0.46 0.5 0.17 0.33 0 0 0 𝐴𝑑𝑑 : 𝛼 ! `` , $ 𝐼𝑇𝐸 : 𝛼 ! `` , '' 𝑀𝑎𝑝 : ∅ 𝐹𝑜𝑙𝑑 𝐴𝑣𝑔 𝑥 𝑾 𝑾 Depth 0 Depth 1 Depth 2 Depth 3 Depth 0 Depth 1 … … … … Depth 2 𝑾 0.45 0.2 0.35𝐴𝑑𝑑 : 𝛼 '' `` , $ 𝐴𝑑𝑑 𝑀𝑢𝑙 𝑥 𝑾 0 0 1 2 3 4 5 𝐴𝑑𝑑 𝛼 '' `` , $ 𝛼 ! `` , $ 𝐼𝑇𝐸 𝛼 '' `` , '' 𝛼 ! `` , '' 𝛼 % `` , '' 𝑀𝑎𝑝 𝛼 '' `` , ! 𝐴𝑑𝑑 𝛼 '' ! , $ 𝛼 ! ! , $ 𝑀𝑢𝑙 𝛼 '' ! , '' 𝛼 ! ! , '' 𝑥 ∅ 𝐹𝑜𝑙𝑑 𝛼 '' % , $ 𝛼 '' % , '' 𝐴𝑣𝑔 𝑥 ∅ 𝐼𝑇𝐸 𝛼 '' & , '' 𝛼 ! & , '' 𝛼 % & , '' 𝑀𝑢𝑙 𝛼 '' & , ! 𝛼 ! & , ! 𝐴𝑑𝑑 𝛼 '' & , % 𝛼 ! & , % 𝐼𝑇𝐸 𝛼 '' ' , '' 𝛼 ! ' , '' 𝛼 % ' , '' 𝑀𝑢𝑙 𝛼 '' ' , ! 𝛼 ! ' , ! 𝑥 ∅ 𝑆𝑡𝑎𝑟𝑡 𝛼 '' $ , $ 𝑆𝑡𝑎𝑟𝑡 𝛼 '' $ , $ 1 𝐴𝑑𝑑 𝛼 '' `` , $ 𝛼 ! `` , $ 𝐼𝑇𝐸 𝛼 '' `` , '' 𝛼 ! `` , '' 𝛼 % `` , '' 𝑀𝑎𝑝 𝛼 '' `` , ! 2 𝐴𝑑𝑑 𝛼 '' ! , $ 𝛼 ! ! , $ 𝑀𝑢𝑙 𝛼 '' ! , '' 𝛼 ! ! , '' 𝑥 ∅ 3 𝐹𝑜𝑙𝑑 𝛼 '' % , $ 𝛼 '' % , '' 𝐴𝑣𝑔 𝑥 ∅ Figure 2 : Program Derivation Graph of the grammar in Fig . 1. differentiability , we interpret it in terms of a smooth approximation : JITE ( α1 ≥ 0 , α2 , α3 ) K ( x ) = σ ( Jα1K ( x ) ) · Jα2K ( x ) + ( 1 − σ ( Jα1K ( x ) ) · Jα3K ( x ) where σ is the sigmoid function . A program may invoke a customized library of differentiable , parameterized functions . In our context , these functions are in the shape FS , θ ( x ) that extract a vector consisting of a predefined subset S of the dimensions of an input x and pass the extracted vector through a linear function with trainable parameters θ . A program may also use a set of higher-order combinators to recurse over sequences including the standard map and fold combinators . The higher-order combinators takes as input an anonymous function fun x.e ( x ) that evaluates an expression e ( x ) over the input x . For a sequence x , the mapprefix higher-order combinator returns a sequence f ( x [ 1 : 1 ] ) , f ( x [ 1 : 2 ] ) , . . . , f ( x [ 1 : n ] ) , where x [ 1 : i ] is the i-th prefix of x . The SlideWindowAvg function computes the average of a sequence over a moving window of frames . We define the complexity of a program architecture α . Let each grammar rule r have a non-negative real-valued cost c ( r ) . The structural cost c ( α ) is the sum of the costs of the multi-set of rules used to create α . Intuitively , program architectures with lower structural cost are more interpretable . In the context of this paper , program synthesis aims to learn a simple program ( in terms of structure cost ) that satisfies some specifications over program input-output behaviors . In this paper , we set c ( r ) = 1 for any production rule r. Program Synthesis Specifications . In a sequence classification task , a set of feature sequences { ik } Kk=1 are taken as input and we expect to classify ik into a certain category ok. Each ik is a sequence of observations . Each observation captures features extracted at a frame as a 1-dimensional real-valued vector . We aim to synthesize a program P ( · ; α , θ ) as a classifier with high accuracy and low architecture cost . Our program synthesis goal is formalized as follows : arg min θ , α Eik , ok∼D [ ` ( P ( ik ; α , θ ) , ok ) ] + c ( α ) ( 1 ) where D ( ik , ok ) is an unknown distribution over input sequences ik and labels ok . The first term of Equation ( 1 ) defines some prediction error loss ` of a program P ( · ) for a classification task over P ’ s predicted labels and the ground truth labels . The second term enforces program synthesis to learn an architecturally simple programmatic classifier .

Summary: This work proposes a method for differentiable synthesis of program architectures. A program derivation graph is encoded as a differentiable program whose output is a weighted output of all the programs in the graph. The method learns a probability distribution over all possible program derivations induced by a context-free grammar. The grammar rule selection weights and the program parameters are co-optimized iteratively with respect to a loss function over program outputs. The approach is evaluated on four sequence classification tasks. The discovered architectures have better F1 scores as compared to the state-of-the-art program synthesis methods (A*-NEAR and IDS-BB-NEAR).