Differentiable Synthesis of Program Architectures

1 Introduction . Program synthesis has recently emerged as an effective approach to address tasks in several fields where deep learning is applied traditionally . A synthesized program in a domain-specific language ( DSL ) provides a powerful abstraction for summarizing discovered knowledge from data and offers greater interpretability and transferability across tasks than a deep neural network model , while achieving competitive task performance [ 1–4 ] . A differentiable program encourages interpretability by using structured symbolic primitives to compose a set of differentiable modules with trainable parameters in its architecture . These parameters can be efficiently learned with respect to a differentiable loss function over the program ’ s outputs . However , synthesizing a reasonable program architecture remains challenging because the architecture search space is discrete and combinatorial . Various enumeration strategies have been developed to explore the program architecture space , including greedy enumeration [ 1 , 2 ] , evolutionary search [ 5 ] , and Monte Carlo sampling [ 6 ] . To prioritize highly likely top-down search directions in the combinatorial architecture space , NEAR [ 7 ] uses neural networks to approximate missing expressions in a partial program whose F1 score serves as an admissible heuristic to effective graph search algorithms such as A∗ [ 8 ] . However , since the discrete program architecture search space is intractably large , enumeration-based search strategies are inefficient in general . We propose to encode program architecture search as learning the probability distribution over all possible program architecture derivations induced by a context-free DSL grammar . This problem bears similarities with searching the structure of graphical models [ 9 ] and neural architecture search . Specifically , to support differentiable search , DARTS [ 10 ] uses a composition of softmaxes over all possible candidate operations between a fixed set of neural network nodes to relax the discrete search space of neural architectures . However , applying this method to program synthesis is challenging 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . because the program architecture search space is much richer [ 7 ] . Firstly , different sets of operations take different input and output types and may only be available at different points of a program . Secondly , there is no fixed bound on the number of expressions in a program architecture . To address the aforementioned challenges , we learn the probability distribution of program architectures in a continuous relaxation of the search space of DSL grammar rules . We conduct program architecture search in a program derivation graph , in which nodes encode architectures with missing expressions , and paths encode top-down program derivations . For each partial architecture f on a graph node , we relax the categorical choice of production rules for expanding a missing expression in f to a softmax over all possible production rules with trainable weights . A program derivation graph essentially expresses all possible program derivations under a context-free grammar up to a certain depth bound ( on the height of program abstract syntax trees ) , which can be iteratively increased during search to balance accuracy and architecture complexity . We encode a program derivation graph itself as a differentiable program whose output is weighted by the outputs of all the programs involved . We seek to optimize program architecture weights with respect to an accuracy loss function defined over the encoded program ’ s output . The learned weights allow our synthesis algorithm to efficiently prune away search directions to unlikely program derivations to discover optimal programs . Compared with enumeration-based synthesis strategies , differentiable program synthesis in the relaxed architecture space is easier and more efficient with gradient-based optimization . One major challenge of differentiable program architecture synthesis is that a program derivation graph involves an exponential number of programs and a huge set of trainable variables including architecture weights and program parameters . To curb the large program derivation search space , we introduce node sharing in program derivation graphs and iterative graph unfolding . Node sharing allows two partial architectures to share the same child nodes if the missing expressions in the two architectures can be expanded using the same grammar rules . Iterative graph unfolding allows the synthesis algorithm to construct a program derivation graph on the fly focusing on higher-quality program derivations than all the rest . These optimization strategies significantly reduce the program architecture search space , scaling differentiable program synthesis to real-world classification tasks . We evaluate our synthesis algorithm in the context of learning classifiers in sequence classification applications . We demonstrate that our algorithm substantially outperforms state-of-the-art methods for differentiable program synthesis , and can learn programmatic classifiers that are highly interpretable and are comparable to neural network models in terms of accuracy and F1-scores . As a summary , this paper makes three contributions . Firstly , we encode program synthesis as learning the probability distribution of program architectures in a continuous relaxation of the discrete space defined by programming language grammar rules , enabling differentiable program architecture search . Secondly , we instantiate differentiable program architecture synthesis with effective optimization strategies including node sharing and iterative graph unfolding , scaling it to real-world classification tasks . Lastly , we present state-of-the-art results in learning programmatic classifiers for four sequence classification applications . 2 Problem Formulation . A program in a domain-specific language ( DSL ) is a pair ( α , θ ) , where α is a discrete program architecture and θ is a vector of real-valued parameters of the program . Given a specification over the intended input-output behavior of an unknown program , program synthesis aims to discover the program ’ s architecture α and optimize the program parameters θ . In this paper , we focus on learning programmatic classifiers for sequence classification tasks [ 11 ] . We note that the proposed synthesis technique is applicable to learning any differentiable programs . Program Architecture Synthesis . A program architecture α is typically synthesized based on a context-free grammar [ 12 ] . Such a grammar consists of a set of production rules αk → { σj } Jj=0 over terminal symbols Σ and nonterminal symbols Y where αk ∈ Y and σj ∈ Σ ∪ Y . As an example , consider the context-free grammar of a DSL for sequence classification depicted in the standard Backus-Naur form [ 13 ] in Fig . 1 , adapted from [ 7 ] . A terminal in this grammar is a symbol that can appear in a program ’ s code , e.g . x and the map function symbol , while a nonterminal stands for a missing expression ( or subexpression ) , e.g . α2 and α3 . Any program in the DSL operates over a real vector or a sequence of real vector x . It may use constants c , arithmetic operations Add and Multiply , and an If-Then-Else branching construct ITE . To avoid discontinuities for α : := x | c | Add α1 α2 |Multiply α1 α2 | ITE α1 ≥ 0 α2 α3 | FS , θ ( x ) | map ( fun x1.α1 ) x | mapprefix ( fun x1.α1 ) x | fold ( fun x1.α1 ) c x | SlideWindowAvg ( fun x1.α1 ) x Figure 1 : Context-free DSL Grammar for Sequence Classification ( adapted from [ 7 ] ) . … … 0.2 0.34 0.46 0.5 0.17 0.33 0 0 0 𝐴𝑑𝑑 : 𝛼 ! `` , $ 𝐼𝑇𝐸 : 𝛼 ! `` , '' 𝑀𝑎𝑝 : ∅ 𝐹𝑜𝑙𝑑 𝐴𝑣𝑔 𝑥 𝑾 𝑾 Depth 0 Depth 1 Depth 2 Depth 3 Depth 0 Depth 1 … … … … Depth 2 𝑾 0.45 0.2 0.35𝐴𝑑𝑑 : 𝛼 '' `` , $ 𝐴𝑑𝑑 𝑀𝑢𝑙 𝑥 𝑾 0 0 1 2 3 4 5 𝐴𝑑𝑑 𝛼 '' `` , $ 𝛼 ! `` , $ 𝐼𝑇𝐸 𝛼 '' `` , '' 𝛼 ! `` , '' 𝛼 % `` , '' 𝑀𝑎𝑝 𝛼 '' `` , ! 𝐴𝑑𝑑 𝛼 '' ! , $ 𝛼 ! ! , $ 𝑀𝑢𝑙 𝛼 '' ! , '' 𝛼 ! ! , '' 𝑥 ∅ 𝐹𝑜𝑙𝑑 𝛼 '' % , $ 𝛼 '' % , '' 𝐴𝑣𝑔 𝑥 ∅ 𝐼𝑇𝐸 𝛼 '' & , '' 𝛼 ! & , '' 𝛼 % & , '' 𝑀𝑢𝑙 𝛼 '' & , ! 𝛼 ! & , ! 𝐴𝑑𝑑 𝛼 '' & , % 𝛼 ! & , % 𝐼𝑇𝐸 𝛼 '' ' , '' 𝛼 ! ' , '' 𝛼 % ' , '' 𝑀𝑢𝑙 𝛼 '' ' , ! 𝛼 ! ' , ! 𝑥 ∅ 𝑆𝑡𝑎𝑟𝑡 𝛼 '' $ , $ 𝑆𝑡𝑎𝑟𝑡 𝛼 '' $ , $ 1 𝐴𝑑𝑑 𝛼 '' `` , $ 𝛼 ! `` , $ 𝐼𝑇𝐸 𝛼 '' `` , '' 𝛼 ! `` , '' 𝛼 % `` , '' 𝑀𝑎𝑝 𝛼 '' `` , ! 2 𝐴𝑑𝑑 𝛼 '' ! , $ 𝛼 ! ! , $ 𝑀𝑢𝑙 𝛼 '' ! , '' 𝛼 ! ! , '' 𝑥 ∅ 3 𝐹𝑜𝑙𝑑 𝛼 '' % , $ 𝛼 '' % , '' 𝐴𝑣𝑔 𝑥 ∅ Figure 2 : Program Derivation Graph of the grammar in Fig . 1. differentiability , we interpret it in terms of a smooth approximation : JITE ( α1 ≥ 0 , α2 , α3 ) K ( x ) = σ ( Jα1K ( x ) ) · Jα2K ( x ) + ( 1 − σ ( Jα1K ( x ) ) · Jα3K ( x ) where σ is the sigmoid function . A program may invoke a customized library of differentiable , parameterized functions . In our context , these functions are in the shape FS , θ ( x ) that extract a vector consisting of a predefined subset S of the dimensions of an input x and pass the extracted vector through a linear function with trainable parameters θ . A program may also use a set of higher-order combinators to recurse over sequences including the standard map and fold combinators . The higher-order combinators takes as input an anonymous function fun x.e ( x ) that evaluates an expression e ( x ) over the input x . For a sequence x , the mapprefix higher-order combinator returns a sequence f ( x [ 1 : 1 ] ) , f ( x [ 1 : 2 ] ) , . . . , f ( x [ 1 : n ] ) , where x [ 1 : i ] is the i-th prefix of x . The SlideWindowAvg function computes the average of a sequence over a moving window of frames . We define the complexity of a program architecture α . Let each grammar rule r have a non-negative real-valued cost c ( r ) . The structural cost c ( α ) is the sum of the costs of the multi-set of rules used to create α . Intuitively , program architectures with lower structural cost are more interpretable . In the context of this paper , program synthesis aims to learn a simple program ( in terms of structure cost ) that satisfies some specifications over program input-output behaviors . In this paper , we set c ( r ) = 1 for any production rule r. Program Synthesis Specifications . In a sequence classification task , a set of feature sequences { ik } Kk=1 are taken as input and we expect to classify ik into a certain category ok. Each ik is a sequence of observations . Each observation captures features extracted at a frame as a 1-dimensional real-valued vector . We aim to synthesize a program P ( · ; α , θ ) as a classifier with high accuracy and low architecture cost . Our program synthesis goal is formalized as follows : arg min θ , α Eik , ok∼D [ ` ( P ( ik ; α , θ ) , ok ) ] + c ( α ) ( 1 ) where D ( ik , ok ) is an unknown distribution over input sequences ik and labels ok . The first term of Equation ( 1 ) defines some prediction error loss ` of a program P ( · ) for a classification task over P ’ s predicted labels and the ground truth labels . The second term enforces program synthesis to learn an architecturally simple programmatic classifier .

Summary: This paper introduces dPads, an approach to learning interpretable programs to perform sequence classification tasks, as an alternative to using RNNs which are less interpretable and possibly less robust. The programs are generated using a context-free grammar (adapted from prior work) of differentiable operations applied to sequences. The ultimate objective is to find a program that obtains high classification accuracy on a dataset, while having low architectural cost. This is done by using a continuous relaxation of the discrete choices in the grammar, using a softmax over all possible grammar production rules instead of a categorical choice, with learnable weights. Further optimizations include “node sharing” (where two nodes in the program derivation graph are shared if they represent the same nonterminal and thus can be expanded the same way) and “iterative graph unfolding” which is somewhat like a beam search over the program derivation process, narrowing down the earlier choices using a guess for their quality, resulting in less combinatorial explosion (higher efficiency) at the cost of a fully complete search. As a final step, an A* search is used to choose a single complete program with high accuracy and low cost. Experiments on 4 sequence classification datasets show that dPads performs better than prior work NEAR, but worse than plain RNNs (which are not interpretable programs).