Generating Symbolic Reasoning Problems with Transformer GANs

1 INTRODUCTION . Deep learning is increasingly applied to more untraditional domains that involve complex symbolic reasoning . Examples include the application of deep neural network architectures to SAT ( Selsam et al. , 2019 ; Selsam & Bjørner , 2019 ; Ozolins et al. , 2021 ) , SMT ( Balunovic et al. , 2018 ) , temporal specifications in verification ( Hahn et al. , 2021 ; Schmitt et al. , 2021 ) , symbolic mathematics ( Lample & Charton , 2020 ) , or theorem proving ( Loos et al. , 2017 ; Bansal et al. , 2019 ; Huang et al. , 2019 ; Urban & Jakubuv , 2020 ) . The acquisition of training data for symbolic reasoning domains , however , is a challenge . Existing instances , such as benchmarks in competitions ( Biere & Claessen , 2010 ; Froleyks et al. , 2021 ; Jacobs et al. , 2017 ) are typically hand-crafted , for example , in a “ bring your own benchmarks ” setting ( Balyo et al. , 2017 ) . Since the instances are too few to be trained on , training data is , thus , typically generated synthetically . For example by random sampling ( Selsam et al. , 2019 ; Lample & Charton , 2020 ) , or by randomly re-combining parts of existing instances ( Schmitt et al. , 2021 ) . Although these data generation methods already lead to good results , training on randomly generated data carries the risk of training on meaningless data or the risk of introducing unwanted biases . In this paper , we study the generation of symbolic reasoning problems with Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and show that they can be used to construct large amounts of meaningful training data from a significantly smaller data source . GANs , however , can not immediately be applied : Symbolic reasoning problems reside typically in a discontinuous domain and , additionally , training data is typically sequential and of variable length . We show that training directly in the one-hot encoding space is possible when adding Gaussian noise to each position . We , furthermore , use a Transformer ( Vaswani et al. , 2017 ) encoder to cope with the sequential form of the data and the variable length of the problem instances . We provide experiments to show the usefulness of a GAN approach for the generation of reasoning problems . The experiments are based around two symbolic reasoning domains where recent studies on the applicability of deep learning relied on large amounts of artificially generated data : symbolic mathematics and linear-time temporal logic ( LTL ) specifications in verification . We report our experimental results in three sections . We first provide details on how to achieve a stable training of a standard GAN and a Wasserstein GAN ( Arjovsky et al. , 2017 ) both equipped with Transformer encoders . We analyze the particularities of their training behavior , such as the effects of adding different amounts of noise to the one hot embeddings . Secondly , we show for an LTL satisfiability classifier that the generated data can be used as a substitute for real training data , and , especially , that training data can be generated from a real dataset that is too small to be trained on directly . In particular , we show that out of 10K real training instances , a dataset consisting of 400K instances can be generated , on which a classifier can successfully be trained on . Lastly , we show that generating symbolic reasoning problems in a GAN setting has a specialty : We can alter the target distribution by adding a classifier uncertainty part to the generator objective . By doing this , we show that we can obtain a dataset that is even harder to solve than the original dataset which has been used to generate the data from . The remainder of this paper is structured as follows . In Section 2 , we give a short introduction to the problem domains considered in this paper and describe how the origin training data has been constructed . In Section 3 , we present our Transformer GAN architecture ( s ) , before providing experimental results in Section 4 . We give an overview over related work in Section 5 before concluding in Section 6 . 2 PROBLEM DOMAIN AND BASE DATASETS . In this section , we introduce the two problem domains on which we base our experiments on : satisfiability of temporal specifications for formal verification and function integration and ordinary differential equations ( ODEs ) for symbolic mathematics . We furthermore give an overview over the data generation processes of these base datasets . 2.1 HARDWARE SPECIFICATIONS IN LINEAR-TIME TEMPORAL LOGIC ( LTL ) . Linear-time Temporal Logic ( LTL ) ( Pnueli , 1977 ) is the basis for industrial hardware specification languages like the IEEE standard PSL ( IEEE-Commission et al. , 2005 ) . It is an extension of propositional logic with temporal modalities , such as the Next-operator ( ) and the Until-operator ( U ) . There also exist derived operators , such as “ eventually ” ϕ ( ≡ true U ϕ ) and “ globally ” ϕ ( ≡ ¬ ¬ϕ ) . For example , mutual exclusion can be expressed as the following specification : ( ¬ ( accessp0 ∧ accessp1 ) , stating that processes p0 and p1 should have no access to a shared resource at the same time . The base problem of any logic is its satisfiability problem . It is the problem to decide whether there exists a solution to a given formula . The satisfiability problem of LTL is a hard problem , in fact , it is PSPACE-hard Sistla & Clarke ( 1982 ) . The full syntax , semantics and additional information on the satisfiability problem can be found in Appendix A . So far , the construction of datasets for LTL formulas has been done in two ways ( Hahn et al. , 2021 ) : Either by obtaining LTL formulas from a fully random generation process , which likely results in unrealistic formulas , or by sampling conjunctions of LTL specification patterns ( Dwyer et al. , 1999 ) . To obtain a healthy amount of unsatisfiable and satisfiable instances in this artificial generation process , we slightly refined the pattern-based generation method with two operations . Details can be found in Appendix B . Since the formula length correlates to unsatisfiability , we filter for equal proportions of classes per formula length . We restrict the tree size of the formulas to 50 . We call this dataset LTLbase . 2.2 SYMBOLIC MATHEMATICS . Lample & Charton ( 2020 ) showed that Transformer models perform surprisingly well on symbolic mathematics . More precisely , they applied the models to function integration and ordinary differential equations ( ODEs ) . We consider the function integration problem and use the forward generated dataset ( https : //github.com/facebookresearch/SymbolicMathematics ) . Random functions with up to n operators are generated and their integrals are calculated with computer algebra systems . Functions that the system can not integrate are discarded . Mathematical expressions are generated randomly . The dataset is cleaned , with equation simplification , coefficients simplification , and filtering out invalid expressions ( Lample & Charton , 2020 ) . We restrict the tree size to 50 . 3 ARCHITECTURE . The Transformer GAN architecture for generating symbolic reasoning problems ( TGAN-SR ) is depicted in Figure 1 . It consists of two Transformer encoders as discriminator/critic and generator , respectively . The inner layers of the encoders are largely identical to standard transformers ( Vaswani et al. , 2017 ) , but their input and output processing is adjusted to the GAN setting . We use an embedding dimension of demb = 128 , nh = 8 attention heads , and a feed-forward network dimension of dFF = 1024 for both encoders as default . The generator ’ s input is a real scalar random value with uniform distribution [ 0 , 1 ] for each position in the sequence . It is mapped to demb by an affine transformation before being processed by the first layer . The position-wise padding mask is copied from the real data during training , so the lengths of real and generated formulas at the same position in a batch are always identical . During inference , the lengths can either be sampled randomly or copied from an existing dataset similar to training . Either way , the generator encoder ’ s padding mask is predetermined so it has to adequately populate the unmasked positions . With V being the vocabulary , and |V | being the size of the vocabulary , an affine transformation to dimensionality |V | and a softmax is applied after the last layer . The generator ’ s output lies , thus , in the same space as one-hot encoded tokens . We use nlG = 6 layers for our default model ’ s generator . A GAN discriminator and WGAN critic are virtually identical in terms of their architecture . The only difference is that a critic outputs a real scalar value where a discriminator is limited to the range [ 0 , 1 ] , which we achieve by applying an additional logistic sigmoid in the end . To honor their differences regarding the training scheme , we use both terms when referring to exchangeable properties and make no further distinctions between them . For input processing , their |V |-dimensional ( per position ) input is mapped to demb by an affine transformation . After the last layer , the final embeddings are aggregated over the sequence by averaging and a linear projection to a scalar value ( the prediction logit ) is applied . Our default model uses nlD = 4 layers . We achieved best results with slightly more generator than discriminator/critic layers . A full hyperparameter study can be found in AppendixC.2 . Working in the |V |-sized one-hot domain poses harsh constraints on the generator ’ s output . Contrary to continuous domains were GANs are usually employed , each component of a real one-hot vector is , by definition , either 0 or 1 . If the generator were to identify this distribution and use it as criterion to tell real and generated instances apart , this would pose a serious difficulty for training . We therefore sample a |V |-sized vector of Gaussian noise N ( 0 , σ2real ) for each position ( see Figure 1 ) . We add it to the real samples ’ one-hot encoding and re-normalize it to sum 1 before handing them to the discriminator/critic . By default , we use a value of σreal = 0.1 for all models to get comparable results . We study the effect of different amounts of noise more closely in Section 4.1.2 . 4 EXPERIMENTS . In this section , we report our experimental findings . We structure our results in three sections . We first report on the performance of the TGAN-SR architecture in constructing syntactically correct instances of temporal specifications and mathematical expressions . Secondly , we show , exemplary for LTL formulas , that the newly generated dataset can be used as a substitute for the origin dataset . Lastly , we show , by altering the target distribution , that the network can generate a dataset that is harder to solve for a classifier . We trained the models on an NVIDIA DGX A100 system for around 8 hours . We begin each subsection with a short preamble on the training setting . 4.1 PRODUCING SYNTACTICALLY CORRECT SYMBOLIC REASONING PROBLEMS . The goal of the experiments in this section is to asses the generator ’ s capability in creating valid symbolic reasoning problems as objectively as possible . If not stated otherwise , in plots and tables , we report results from our default model averaged across three runs and with an exponential smoothing ( α = 0.95 ) applied . For temporal specifications , we use LTLbase as training set and for symbolic math the dataset described in section 2.2 .

Summary: The authors apply (W)GANs with transformer encoders for data-augmentation in two symbolic domains: LTL and function integration. There are many interesting findings, three of which stand out. First, in both domains, the network learns to generate syntactically-correct examples roughly 30% of the time. Second, for LTL, a GAN trained on a dataset of size 10k can produce a much larger dataset, such that training on the new dataset is almost as good as training on the original distribution (when evaluating on the original distribution). Third, by also rewarding the generator for confusing a classifier, they can generate new problems that are harder to classify than those in the original distribution.