Learning Heuristics for Quantified Boolean Formulas through Reinforcement Learning

1 INTRODUCTION . One of the most intriguing questions for artificial intelligence is : can ( deep ) learning be effectively used for symbolic reasoning ? The benefits of combining deductive reasoning with inductive learning for automated reasoning and in formal methods for system design have been noted ( e.g. , see Seshia ( 2015 ) ) . There is a whole spectrum of approaches to combine them : One extreme is to use learning for predicting which of a small pool of algorithms ( or heuristics ) performs best , and run only that one to solve the given problem ( e.g . SATzilla ( Xu et al. , 2008 ) ) . This approach is clearly limited by the availability of handwritten algorithms and heuristics ( i.e . it can only solve problems for which we have written at least one algorithm that can solve it ) . The other extreme is to analyze formulas solely with deep learning approaches ( Allamanis et al. , 2016 ; Evans et al. , 2018 ; Selsam et al. , 2018 ; Amizadeh et al. , 2019 ) . However , this approach shows poor scalability compared to the state-of-the-art in the respective domains depsite the recent breakthroughs in deep learning . Instead of relying entirely on deep learning or on the availability of good handwritten algorithms , we explore the middle ground . We ask the question how to tightly combine deep learning with formal reasoning algorithms with the goal to improve the state-of-the-art , i.e . to solve formulas that could not be solved previously . Existing formal reasoning tools work in a mechanical way : they only apply a small number of carefully crafted operations and use heuristics to resolve the degrees of freedom in how to apply them . We address the problem of automatically learning better heuristics for a given set of formulas . We focus on the branching heuristic in modern backtracking search algorithms , as they are known to have a high impact on the performance of the algorithm . We cast the problem to learn better branching heuristics for backtracking search algorithms as a reinforcement learning problem : Initially , the reinforcement learning environment randomly picks a formula from a given set of formulas , and then runs the backtracking search algorithm on that formula . The actions that are controlled by the learning agent are the branching decisions , i.e . pick a variable and assign it a value - everything else is handled by the solver . Challenges This reinforcement learning problem comes with several unique challenges : Representation : While learning algorithms for images and board-games usually rely on the grid-like structure of the input and employ neural networks that match that structure ( e.g . convolutional neural networks ) . For formulas , however , there is no standard representation for learning algorithms . It may seem reasonable to treat Boolean formulas as text and learn embeddings for formulas through techniques such as word2vec ( Mikolov et al. , 2013 ) , LSTMs , or tree RNNs . However , formulas in formal reasoning tools typically consist of thousands of variables , which is much larger than the text-fragments typically analyzed with neural networks . Further , unlike words in natural language , individual variables in Boolean formulas are completely devoid of meaning . The meaning of variable x in one formula is basically independent from variable x in a second formula . Hence , learning embeddings for variables and sharing them between formulas would be futile . Unbounded action space : An action consists of a choice of variable and value . While values will be Boolean , the number of variables depends on the size of the input formula . Therefore , we have an unbounded number of actions , which are further different for every formula . Length of episodes : As we are dealing with a highly complex search problem , solver runs ( = learning episodes ) can be very long—in fact , for many of the formulas we have never observed a terminating run—and we observed a huge variance in the length of runs . Performance : Our aim is to solve more formulas in less time . The use of neural networks incurs a huge runtime overhead for each decision ( the solver takes ≥10x fewer decisions per second ) . So the decisions taken by the neural networks need to be dramatically better than the handcoded heuristic decisions to outweigh their runtime cost . Correctness : Reinforcement learning algorithms have shown to often find and exploit subtle implementation errors in the environment , instead of solving the intended problem . While testing and manual inspection of the results is a feasible approach for board games and Atari games , it is neither possible nor sufficient in large-scale formal reasoning - a solver run is simply too large to inspect manually and even tiny mistake can invalidate the result . In order to ensure correctness , we need an environment with the ability to produce formal proofs , and check the proofs by an independent tool . Quantified Boolean Formulas In this paper we focus on 2QBF , that is quantified Boolean formulas of the form ∀X.∃Y.ϕ , where X and Y are sets of Boolean variables and ϕ is in conjunctive normal form . 2QBFs are complex enough to serve as an interesting proxy for complex mathematical reasoning tasks . Challenging applications such as program synthesis and the synthesis of controllers and ranking functions have been encoded into 2QBFs Solar-Lezama et al . ( 2006 ) ; Faymonville et al . ( 2017 ) ; Cook et al . ( 2013 ) . However , the problem definition and also the syntactical structure of 2QBF is simple compared to more general settings , such as first-order or even higher-order logics . This makes algorithms for 2QBF a good target for the study of neural architectures . While our approach in principle works with most algorithms for QBF , we decided to demonstrate its use in Incremental Determinization ( Rabe & Seshia , 2016 ) . We modified CADET , an opensource implementation of Incremental Determinization that performed competitively in recent QBF competitions ( Pulina , 2016 ) and turned it into a reinforcement learning environment . The advantage of CADET in the context of reinforcement learning is its ability to produce proofs ( which most other solvers do not ) , which ensures that the reinforcement learning can not simply learn to exploit bugs in the environment . Graph Neural Networks We consider each constraint and each variable of a given formula as a node in a graph . Whenever a variable occurs in a constraint , we draw an edge between their nodes . We then use a Graph Neural Network ( GNN ) ( Scarselli et al. , 2009 ) to predict the quality of each variable as a decision variable , and pick our next action accordingly . GNNs allow us to compute an embedding for every variable , based on the occurrences of that variable in the given formula , instead of learning an embedding that is shared across all formulas . Based on this embedding , we then use a policy network to predict the quality of each variable ( or literal ) , and choose the next action accordingly . GNNs also allow us to scale to arbitrarily large formulas with a small and constant number of parameters . Contributions This paper presents the successful integration of GNNs in a modern automated reasoning algorithm in a reinforcement learning setup.1 Our approach balances the performance penalty incured by the use of neural networks with the impact that improved heuristic decisions have on the overall reasoning capabilities . The branching heuristic that we learn significantly improves CADET ’ s reasoning capabilities on the test set of the benchmark , i.e . it solves more formulas within the same resource constraints . This is a huge step towards replacing VSIDS , the dominant branching heuristic in CDCL-based solvers for the last 20 years Moskewicz et al . ( 2001b ) ; Eén & Sörensson ( 2003 ) ; Biere et al . ( 2009 ) ; Lonsing & Biere ( 2010 ) ; Rabe & Seshia ( 2016 ) . We also study the generalization properties of our approach : We show that training a heuristic on small and easy formulas helps us to solve much larger and harder formulas ; generalization to formulas from different benchmarks is still limited though . Further , we provide an open-source learning environment for reasoning in quantified Boolean formulas . The environment includes the ability to verify its own runs , and thereby ensures that the reinforcement learning agent does not only learn to exploit implementation errors of the environment . Structure : After a primer on Boolean logics in Section 2 we define the problem in Section 3 , and describe the network architecture in Section 4 . We describe our experiments in Section 5 , discuss related work in Section 6 and present our conclusions in Section 7 . 2 BOOLEAN LOGICS AND SEARCH ALGORITHMS . We start with describing propositional ( i.e . quantifier-free ) Boolean logic . Propositional Boolean logic allows us to use the constants 0 ( false ) and 1 ( true ) , variables , and the standard Boolean operators like ∧ ( “ and ” ) , ∨ ( “ or ” ) , and ¬ ( “ not ” ) . A literal of variable v is either the variable itself or its negation ¬v . By l̄ we denote the logical negation of literal l. We call a disjunction of literals a clause and say that a formula is in conjunctive normal form ( CNF ) , if it is a conjunction of clauses . For example , ( x ∨ y ) ∧ ( ¬x ∨ y ) is in CNF . It is well known that any Boolean formula can be transformed into CNF . It is less well known that this increases the size only linearly , if we allow the transformation to introduce additional variables ( Tseitin , 1968 ) . We thus assume that all formulas in this work are given in CNF . DPLL and CDCL . The satisfiability problem of propositional Boolean logics ( SAT ) is to find a satisfying assignment for a given Boolean formula or to determine that there is no such assignment . SAT is the prototypical NP-complete problem and many other problems in NP can be easily reduced to it . The first backtracking search algorithms for SAT are attributed to Davis , Putnam , Logemann , and Loveland ( DPLL ) ( Davis & Putnam , 1960 ; Davis et al. , 1962 ) . Backtracking search algorithms gradually extend a partial assignment until it becomes a satisfying assignment , or until a conflict is reached . A conflict is reached when the current partial assignment violates one of the clauses and hence can not be completed to a satisfying assignment . In case of a conflict , the search has to backtrack and continue in a different part of the search tree . Conflict-driven clause learning ( CDCL ) is a significant improvement over DPLL due to MarquesSilva and Sakallah ( Marques-Silva & Sakallah , 1997 ) . CDCL combines backtracking search with clause learning . While DPLL simply backtracks out of conflicts , CDCL “ analyzes ” the conflict by performing a couple of resolution steps . Resolution is an operation that takes two existing clauses ( l1 ∨ · · · ∨ ln ) and ( l′1 ∨ · · · ∨ l′n ) that contain a pair of complementary literals l1 = ¬l′1 , and derives the clause ( l2 ∨ · · · ∨ ln ∨ l′2 ∨ · · · ∨ l′n ) . Conflict analysis adds new clauses over time , which cuts off large parts of the search space and thereby speeds up the search process . Since the introduction of CDCL in 1997 , countless refinements of CDCL have been explored and clever data structures improved its efficiency significantly ( Moskewicz et al. , 2001a ; Eén & Sörensson , 2003 ; Goldberg & Novikov , 2007 ) . Today , the top-performing SAT solvers , such as Lingeling ( Biere , 2010 ) , Crypominisat ( Soos , 2014 ) , Glucose ( Audemard & Simon , 2014 ) , and MapleSAT ( Liang et al. , 2016 ) , all rely on CDCL and they solve formulas with millions of variables for industrial applications such as bounded model checking ( Biere et al. , 2003 ) . 1An earlier version of this work was published on https : //arxiv.org/abs/1807.08058 Quantified Boolean Formulas . QBF extends propositional Boolean logic by quantifiers , which are statements of the form “ for all x ” ( ∀x ) and “ there is an x ” ( ∃x ) . The formula ∀x . ϕ is true if , and only if , ϕ is true if x is replaced by 0 ( false ) and also if x is replaced by 1 ( true ) . The semantics of ∃ arises from ∃x . ϕ = ¬∀x . ¬ϕ . We say that a QBF is in prenex normal form if all quantifiers are in the beginning of the formula . W.l.o.g. , we will only consider QBF that are in prenex normal form and whose propositional part is in CNF . Further , we assume that for every variable in the formula there is exactly one quantifier in the prefix . An example QBF in prenex CNF is ∀x . ∃y . ( x ∨ y ) ∧ ( ¬x ∨ y ) . We focus on 2QBF , a subset of QBF that admits only one quantifier alternation . W.l.o.g . we can assume that the quantifier prefix of formulas in 2QBF consists of a sequence of universal quantifiers ∀x1 . . . ∀xn , followed by a sequence of existential quantifiers ∃y1 . . . ∃ym . While 2QBF is less powerful than QBF , we can encode many interesting applications from verification and synthesis , e.g . program synthesis ( Solar-Lezama et al. , 2006 ; Alur et al. , 2013 ) . Solvers for ( 2 ) QBF typically address the decision problem to determine the truth of a given formula ( TQBF ) . After the success of CDCL for SAT , CDCL-like algorithms have been explored for QBF as well ( Giunchiglia et al. , 2001 ; Lonsing & Biere , 2010 ; Rabe & Seshia , 2016 ; Rabe et al. , 2018 ) . We focus on CADET , a solver that implements Incremental Determinization a generalized CDCL backtracking search algorithm ( Rabe & Seshia , 2016 ; Rabe et al. , 2018 ) . Instead of considering only Booleans as values , the Incremental Determinization algorithm assigns and propagates on the level of Skolem functions . For the purpose of this work , however , we do not have to dive into the details of Incremental Determinization and can consider it simply as some backtracking algorithm . Correctness . Writing performant code is an error-prone task , and correctness is critical for many applications of formal reasoning . Some automated reasoning tools hence have the ability to produce proofs , which can be checked independently . CADET is one of the few QBF solvers that can produce proofs without runtime overhead . We believe that the ability to verify results of solvers is particularly crucial for learning applications , as it allows us to ensure that the reinforcement learning algorithm does not simply exploit implementation error ( bugs ) in the environment .

Summary: This paper investigates the problem of predicting the truth of quantified boolean formulae using deep reinforcement learning. In this setting, the problem is formulated as a reinforcement learning task, in which the learner is interacting with a solver (CADET), and its goal is to find a sequence of actions (each associated with a choice of a variable and a value) in order to reach a terminal state as fast as possible. The neural architecture includes a GNN encoder for the input formula, a policy neural net for iteratively assessing the quality of literals, and a final softmax layer for choosing the final literal. Experiments, performed on various 2QBF instances, address several questions such as the ability to compete with existing heuristics (VSIDS) in CADET and to generalize predictions on long episodes or different formulae.