Generating Symbolic Reasoning Problems with Transformer GANs

1 INTRODUCTION . Deep learning is increasingly applied to more untraditional domains that involve complex symbolic reasoning . Examples include the application of deep neural network architectures to SAT ( Selsam et al. , 2019 ; Selsam & Bjørner , 2019 ; Ozolins et al. , 2021 ) , SMT ( Balunovic et al. , 2018 ) , temporal specifications in verification ( Hahn et al. , 2021 ; Schmitt et al. , 2021 ) , symbolic mathematics ( Lample & Charton , 2020 ) , or theorem proving ( Loos et al. , 2017 ; Bansal et al. , 2019 ; Huang et al. , 2019 ; Urban & Jakubuv , 2020 ) . The acquisition of training data for symbolic reasoning domains , however , is a challenge . Existing instances , such as benchmarks in competitions ( Biere & Claessen , 2010 ; Froleyks et al. , 2021 ; Jacobs et al. , 2017 ) are typically hand-crafted , for example , in a “ bring your own benchmarks ” setting ( Balyo et al. , 2017 ) . Since the instances are too few to be trained on , training data is , thus , typically generated synthetically . For example by random sampling ( Selsam et al. , 2019 ; Lample & Charton , 2020 ) , or by randomly re-combining parts of existing instances ( Schmitt et al. , 2021 ) . Although these data generation methods already lead to good results , training on randomly generated data carries the risk of training on meaningless data or the risk of introducing unwanted biases . In this paper , we study the generation of symbolic reasoning problems with Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and show that they can be used to construct large amounts of meaningful training data from a significantly smaller data source . GANs , however , can not immediately be applied : Symbolic reasoning problems reside typically in a discontinuous domain and , additionally , training data is typically sequential and of variable length . We show that training directly in the one-hot encoding space is possible when adding Gaussian noise to each position . We , furthermore , use a Transformer ( Vaswani et al. , 2017 ) encoder to cope with the sequential form of the data and the variable length of the problem instances . We provide experiments to show the usefulness of a GAN approach for the generation of reasoning problems . The experiments are based around two symbolic reasoning domains where recent studies on the applicability of deep learning relied on large amounts of artificially generated data : symbolic mathematics and linear-time temporal logic ( LTL ) specifications in verification . We report our experimental results in three sections . We first provide details on how to achieve a stable training of a standard GAN and a Wasserstein GAN ( Arjovsky et al. , 2017 ) both equipped with Transformer encoders . We analyze the particularities of their training behavior , such as the effects of adding different amounts of noise to the one hot embeddings . Secondly , we show for an LTL satisfiability classifier that the generated data can be used as a substitute for real training data , and , especially , that training data can be generated from a real dataset that is too small to be trained on directly . In particular , we show that out of 10K real training instances , a dataset consisting of 400K instances can be generated , on which a classifier can successfully be trained on . Lastly , we show that generating symbolic reasoning problems in a GAN setting has a specialty : We can alter the target distribution by adding a classifier uncertainty part to the generator objective . By doing this , we show that we can obtain a dataset that is even harder to solve than the original dataset which has been used to generate the data from . The remainder of this paper is structured as follows . In Section 2 , we give a short introduction to the problem domains considered in this paper and describe how the origin training data has been constructed . In Section 3 , we present our Transformer GAN architecture ( s ) , before providing experimental results in Section 4 . We give an overview over related work in Section 5 before concluding in Section 6 . 2 PROBLEM DOMAIN AND BASE DATASETS . In this section , we introduce the two problem domains on which we base our experiments on : satisfiability of temporal specifications for formal verification and function integration and ordinary differential equations ( ODEs ) for symbolic mathematics . We furthermore give an overview over the data generation processes of these base datasets . 2.1 HARDWARE SPECIFICATIONS IN LINEAR-TIME TEMPORAL LOGIC ( LTL ) . Linear-time Temporal Logic ( LTL ) ( Pnueli , 1977 ) is the basis for industrial hardware specification languages like the IEEE standard PSL ( IEEE-Commission et al. , 2005 ) . It is an extension of propositional logic with temporal modalities , such as the Next-operator ( ) and the Until-operator ( U ) . There also exist derived operators , such as “ eventually ” ϕ ( ≡ true U ϕ ) and “ globally ” ϕ ( ≡ ¬ ¬ϕ ) . For example , mutual exclusion can be expressed as the following specification : ( ¬ ( accessp0 ∧ accessp1 ) , stating that processes p0 and p1 should have no access to a shared resource at the same time . The base problem of any logic is its satisfiability problem . It is the problem to decide whether there exists a solution to a given formula . The satisfiability problem of LTL is a hard problem , in fact , it is PSPACE-hard Sistla & Clarke ( 1982 ) . The full syntax , semantics and additional information on the satisfiability problem can be found in Appendix A . So far , the construction of datasets for LTL formulas has been done in two ways ( Hahn et al. , 2021 ) : Either by obtaining LTL formulas from a fully random generation process , which likely results in unrealistic formulas , or by sampling conjunctions of LTL specification patterns ( Dwyer et al. , 1999 ) . To obtain a healthy amount of unsatisfiable and satisfiable instances in this artificial generation process , we slightly refined the pattern-based generation method with two operations . Details can be found in Appendix B . Since the formula length correlates to unsatisfiability , we filter for equal proportions of classes per formula length . We restrict the tree size of the formulas to 50 . We call this dataset LTLbase . 2.2 SYMBOLIC MATHEMATICS . Lample & Charton ( 2020 ) showed that Transformer models perform surprisingly well on symbolic mathematics . More precisely , they applied the models to function integration and ordinary differential equations ( ODEs ) . We consider the function integration problem and use the forward generated dataset ( https : //github.com/facebookresearch/SymbolicMathematics ) . Random functions with up to n operators are generated and their integrals are calculated with computer algebra systems . Functions that the system can not integrate are discarded . Mathematical expressions are generated randomly . The dataset is cleaned , with equation simplification , coefficients simplification , and filtering out invalid expressions ( Lample & Charton , 2020 ) . We restrict the tree size to 50 . 3 ARCHITECTURE . The Transformer GAN architecture for generating symbolic reasoning problems ( TGAN-SR ) is depicted in Figure 1 . It consists of two Transformer encoders as discriminator/critic and generator , respectively . The inner layers of the encoders are largely identical to standard transformers ( Vaswani et al. , 2017 ) , but their input and output processing is adjusted to the GAN setting . We use an embedding dimension of demb = 128 , nh = 8 attention heads , and a feed-forward network dimension of dFF = 1024 for both encoders as default . The generator ’ s input is a real scalar random value with uniform distribution [ 0 , 1 ] for each position in the sequence . It is mapped to demb by an affine transformation before being processed by the first layer . The position-wise padding mask is copied from the real data during training , so the lengths of real and generated formulas at the same position in a batch are always identical . During inference , the lengths can either be sampled randomly or copied from an existing dataset similar to training . Either way , the generator encoder ’ s padding mask is predetermined so it has to adequately populate the unmasked positions . With V being the vocabulary , and |V | being the size of the vocabulary , an affine transformation to dimensionality |V | and a softmax is applied after the last layer . The generator ’ s output lies , thus , in the same space as one-hot encoded tokens . We use nlG = 6 layers for our default model ’ s generator . A GAN discriminator and WGAN critic are virtually identical in terms of their architecture . The only difference is that a critic outputs a real scalar value where a discriminator is limited to the range [ 0 , 1 ] , which we achieve by applying an additional logistic sigmoid in the end . To honor their differences regarding the training scheme , we use both terms when referring to exchangeable properties and make no further distinctions between them . For input processing , their |V |-dimensional ( per position ) input is mapped to demb by an affine transformation . After the last layer , the final embeddings are aggregated over the sequence by averaging and a linear projection to a scalar value ( the prediction logit ) is applied . Our default model uses nlD = 4 layers . We achieved best results with slightly more generator than discriminator/critic layers . A full hyperparameter study can be found in AppendixC.2 . Working in the |V |-sized one-hot domain poses harsh constraints on the generator ’ s output . Contrary to continuous domains were GANs are usually employed , each component of a real one-hot vector is , by definition , either 0 or 1 . If the generator were to identify this distribution and use it as criterion to tell real and generated instances apart , this would pose a serious difficulty for training . We therefore sample a |V |-sized vector of Gaussian noise N ( 0 , σ2real ) for each position ( see Figure 1 ) . We add it to the real samples ’ one-hot encoding and re-normalize it to sum 1 before handing them to the discriminator/critic . By default , we use a value of σreal = 0.1 for all models to get comparable results . We study the effect of different amounts of noise more closely in Section 4.1.2 . 4 EXPERIMENTS . In this section , we report our experimental findings . We structure our results in three sections . We first report on the performance of the TGAN-SR architecture in constructing syntactically correct instances of temporal specifications and mathematical expressions . Secondly , we show , exemplary for LTL formulas , that the newly generated dataset can be used as a substitute for the origin dataset . Lastly , we show , by altering the target distribution , that the network can generate a dataset that is harder to solve for a classifier . We trained the models on an NVIDIA DGX A100 system for around 8 hours . We begin each subsection with a short preamble on the training setting . 4.1 PRODUCING SYNTACTICALLY CORRECT SYMBOLIC REASONING PROBLEMS . The goal of the experiments in this section is to asses the generator ’ s capability in creating valid symbolic reasoning problems as objectively as possible . If not stated otherwise , in plots and tables , we report results from our default model averaged across three runs and with an exponential smoothing ( α = 0.95 ) applied . For temporal specifications , we use LTLbase as training set and for symbolic math the dataset described in section 2.2 .

Summary: This paper aims to generate training data for symbolic reasoning by training GANs based on Transformers. Specifically, the paper explores two methods --- standard GAN and Wasserstein GAN --- which has a Transformer as an encoder and a decoder, respectively. Training these models on symbolic reasoning data that is randomly generated --- LTL and Symbolic mathematics --- is found to generate high quality formulas.