Learning Heuristics for Quantified Boolean Formulas through Reinforcement Learning

1 INTRODUCTION . One of the most intriguing questions for artificial intelligence is : can ( deep ) learning be effectively used for symbolic reasoning ? The benefits of combining deductive reasoning with inductive learning for automated reasoning and in formal methods for system design have been noted ( e.g. , see Seshia ( 2015 ) ) . There is a whole spectrum of approaches to combine them : One extreme is to use learning for predicting which of a small pool of algorithms ( or heuristics ) performs best , and run only that one to solve the given problem ( e.g . SATzilla ( Xu et al. , 2008 ) ) . This approach is clearly limited by the availability of handwritten algorithms and heuristics ( i.e . it can only solve problems for which we have written at least one algorithm that can solve it ) . The other extreme is to analyze formulas solely with deep learning approaches ( Allamanis et al. , 2016 ; Evans et al. , 2018 ; Selsam et al. , 2018 ; Amizadeh et al. , 2019 ) . However , this approach shows poor scalability compared to the state-of-the-art in the respective domains depsite the recent breakthroughs in deep learning . Instead of relying entirely on deep learning or on the availability of good handwritten algorithms , we explore the middle ground . We ask the question how to tightly combine deep learning with formal reasoning algorithms with the goal to improve the state-of-the-art , i.e . to solve formulas that could not be solved previously . Existing formal reasoning tools work in a mechanical way : they only apply a small number of carefully crafted operations and use heuristics to resolve the degrees of freedom in how to apply them . We address the problem of automatically learning better heuristics for a given set of formulas . We focus on the branching heuristic in modern backtracking search algorithms , as they are known to have a high impact on the performance of the algorithm . We cast the problem to learn better branching heuristics for backtracking search algorithms as a reinforcement learning problem : Initially , the reinforcement learning environment randomly picks a formula from a given set of formulas , and then runs the backtracking search algorithm on that formula . The actions that are controlled by the learning agent are the branching decisions , i.e . pick a variable and assign it a value - everything else is handled by the solver . Challenges This reinforcement learning problem comes with several unique challenges : Representation : While learning algorithms for images and board-games usually rely on the grid-like structure of the input and employ neural networks that match that structure ( e.g . convolutional neural networks ) . For formulas , however , there is no standard representation for learning algorithms . It may seem reasonable to treat Boolean formulas as text and learn embeddings for formulas through techniques such as word2vec ( Mikolov et al. , 2013 ) , LSTMs , or tree RNNs . However , formulas in formal reasoning tools typically consist of thousands of variables , which is much larger than the text-fragments typically analyzed with neural networks . Further , unlike words in natural language , individual variables in Boolean formulas are completely devoid of meaning . The meaning of variable x in one formula is basically independent from variable x in a second formula . Hence , learning embeddings for variables and sharing them between formulas would be futile . Unbounded action space : An action consists of a choice of variable and value . While values will be Boolean , the number of variables depends on the size of the input formula . Therefore , we have an unbounded number of actions , which are further different for every formula . Length of episodes : As we are dealing with a highly complex search problem , solver runs ( = learning episodes ) can be very long—in fact , for many of the formulas we have never observed a terminating run—and we observed a huge variance in the length of runs . Performance : Our aim is to solve more formulas in less time . The use of neural networks incurs a huge runtime overhead for each decision ( the solver takes ≥10x fewer decisions per second ) . So the decisions taken by the neural networks need to be dramatically better than the handcoded heuristic decisions to outweigh their runtime cost . Correctness : Reinforcement learning algorithms have shown to often find and exploit subtle implementation errors in the environment , instead of solving the intended problem . While testing and manual inspection of the results is a feasible approach for board games and Atari games , it is neither possible nor sufficient in large-scale formal reasoning - a solver run is simply too large to inspect manually and even tiny mistake can invalidate the result . In order to ensure correctness , we need an environment with the ability to produce formal proofs , and check the proofs by an independent tool . Quantified Boolean Formulas In this paper we focus on 2QBF , that is quantified Boolean formulas of the form ∀X.∃Y.ϕ , where X and Y are sets of Boolean variables and ϕ is in conjunctive normal form . 2QBFs are complex enough to serve as an interesting proxy for complex mathematical reasoning tasks . Challenging applications such as program synthesis and the synthesis of controllers and ranking functions have been encoded into 2QBFs Solar-Lezama et al . ( 2006 ) ; Faymonville et al . ( 2017 ) ; Cook et al . ( 2013 ) . However , the problem definition and also the syntactical structure of 2QBF is simple compared to more general settings , such as first-order or even higher-order logics . This makes algorithms for 2QBF a good target for the study of neural architectures . While our approach in principle works with most algorithms for QBF , we decided to demonstrate its use in Incremental Determinization ( Rabe & Seshia , 2016 ) . We modified CADET , an opensource implementation of Incremental Determinization that performed competitively in recent QBF competitions ( Pulina , 2016 ) and turned it into a reinforcement learning environment . The advantage of CADET in the context of reinforcement learning is its ability to produce proofs ( which most other solvers do not ) , which ensures that the reinforcement learning can not simply learn to exploit bugs in the environment . Graph Neural Networks We consider each constraint and each variable of a given formula as a node in a graph . Whenever a variable occurs in a constraint , we draw an edge between their nodes . We then use a Graph Neural Network ( GNN ) ( Scarselli et al. , 2009 ) to predict the quality of each variable as a decision variable , and pick our next action accordingly . GNNs allow us to compute an embedding for every variable , based on the occurrences of that variable in the given formula , instead of learning an embedding that is shared across all formulas . Based on this embedding , we then use a policy network to predict the quality of each variable ( or literal ) , and choose the next action accordingly . GNNs also allow us to scale to arbitrarily large formulas with a small and constant number of parameters . Contributions This paper presents the successful integration of GNNs in a modern automated reasoning algorithm in a reinforcement learning setup.1 Our approach balances the performance penalty incured by the use of neural networks with the impact that improved heuristic decisions have on the overall reasoning capabilities . The branching heuristic that we learn significantly improves CADET ’ s reasoning capabilities on the test set of the benchmark , i.e . it solves more formulas within the same resource constraints . This is a huge step towards replacing VSIDS , the dominant branching heuristic in CDCL-based solvers for the last 20 years Moskewicz et al . ( 2001b ) ; Eén & Sörensson ( 2003 ) ; Biere et al . ( 2009 ) ; Lonsing & Biere ( 2010 ) ; Rabe & Seshia ( 2016 ) . We also study the generalization properties of our approach : We show that training a heuristic on small and easy formulas helps us to solve much larger and harder formulas ; generalization to formulas from different benchmarks is still limited though . Further , we provide an open-source learning environment for reasoning in quantified Boolean formulas . The environment includes the ability to verify its own runs , and thereby ensures that the reinforcement learning agent does not only learn to exploit implementation errors of the environment . Structure : After a primer on Boolean logics in Section 2 we define the problem in Section 3 , and describe the network architecture in Section 4 . We describe our experiments in Section 5 , discuss related work in Section 6 and present our conclusions in Section 7 . 2 BOOLEAN LOGICS AND SEARCH ALGORITHMS . We start with describing propositional ( i.e . quantifier-free ) Boolean logic . Propositional Boolean logic allows us to use the constants 0 ( false ) and 1 ( true ) , variables , and the standard Boolean operators like ∧ ( “ and ” ) , ∨ ( “ or ” ) , and ¬ ( “ not ” ) . A literal of variable v is either the variable itself or its negation ¬v . By l̄ we denote the logical negation of literal l. We call a disjunction of literals a clause and say that a formula is in conjunctive normal form ( CNF ) , if it is a conjunction of clauses . For example , ( x ∨ y ) ∧ ( ¬x ∨ y ) is in CNF . It is well known that any Boolean formula can be transformed into CNF . It is less well known that this increases the size only linearly , if we allow the transformation to introduce additional variables ( Tseitin , 1968 ) . We thus assume that all formulas in this work are given in CNF . DPLL and CDCL . The satisfiability problem of propositional Boolean logics ( SAT ) is to find a satisfying assignment for a given Boolean formula or to determine that there is no such assignment . SAT is the prototypical NP-complete problem and many other problems in NP can be easily reduced to it . The first backtracking search algorithms for SAT are attributed to Davis , Putnam , Logemann , and Loveland ( DPLL ) ( Davis & Putnam , 1960 ; Davis et al. , 1962 ) . Backtracking search algorithms gradually extend a partial assignment until it becomes a satisfying assignment , or until a conflict is reached . A conflict is reached when the current partial assignment violates one of the clauses and hence can not be completed to a satisfying assignment . In case of a conflict , the search has to backtrack and continue in a different part of the search tree . Conflict-driven clause learning ( CDCL ) is a significant improvement over DPLL due to MarquesSilva and Sakallah ( Marques-Silva & Sakallah , 1997 ) . CDCL combines backtracking search with clause learning . While DPLL simply backtracks out of conflicts , CDCL “ analyzes ” the conflict by performing a couple of resolution steps . Resolution is an operation that takes two existing clauses ( l1 ∨ · · · ∨ ln ) and ( l′1 ∨ · · · ∨ l′n ) that contain a pair of complementary literals l1 = ¬l′1 , and derives the clause ( l2 ∨ · · · ∨ ln ∨ l′2 ∨ · · · ∨ l′n ) . Conflict analysis adds new clauses over time , which cuts off large parts of the search space and thereby speeds up the search process . Since the introduction of CDCL in 1997 , countless refinements of CDCL have been explored and clever data structures improved its efficiency significantly ( Moskewicz et al. , 2001a ; Eén & Sörensson , 2003 ; Goldberg & Novikov , 2007 ) . Today , the top-performing SAT solvers , such as Lingeling ( Biere , 2010 ) , Crypominisat ( Soos , 2014 ) , Glucose ( Audemard & Simon , 2014 ) , and MapleSAT ( Liang et al. , 2016 ) , all rely on CDCL and they solve formulas with millions of variables for industrial applications such as bounded model checking ( Biere et al. , 2003 ) . 1An earlier version of this work was published on https : //arxiv.org/abs/1807.08058 Quantified Boolean Formulas . QBF extends propositional Boolean logic by quantifiers , which are statements of the form “ for all x ” ( ∀x ) and “ there is an x ” ( ∃x ) . The formula ∀x . ϕ is true if , and only if , ϕ is true if x is replaced by 0 ( false ) and also if x is replaced by 1 ( true ) . The semantics of ∃ arises from ∃x . ϕ = ¬∀x . ¬ϕ . We say that a QBF is in prenex normal form if all quantifiers are in the beginning of the formula . W.l.o.g. , we will only consider QBF that are in prenex normal form and whose propositional part is in CNF . Further , we assume that for every variable in the formula there is exactly one quantifier in the prefix . An example QBF in prenex CNF is ∀x . ∃y . ( x ∨ y ) ∧ ( ¬x ∨ y ) . We focus on 2QBF , a subset of QBF that admits only one quantifier alternation . W.l.o.g . we can assume that the quantifier prefix of formulas in 2QBF consists of a sequence of universal quantifiers ∀x1 . . . ∀xn , followed by a sequence of existential quantifiers ∃y1 . . . ∃ym . While 2QBF is less powerful than QBF , we can encode many interesting applications from verification and synthesis , e.g . program synthesis ( Solar-Lezama et al. , 2006 ; Alur et al. , 2013 ) . Solvers for ( 2 ) QBF typically address the decision problem to determine the truth of a given formula ( TQBF ) . After the success of CDCL for SAT , CDCL-like algorithms have been explored for QBF as well ( Giunchiglia et al. , 2001 ; Lonsing & Biere , 2010 ; Rabe & Seshia , 2016 ; Rabe et al. , 2018 ) . We focus on CADET , a solver that implements Incremental Determinization a generalized CDCL backtracking search algorithm ( Rabe & Seshia , 2016 ; Rabe et al. , 2018 ) . Instead of considering only Booleans as values , the Incremental Determinization algorithm assigns and propagates on the level of Skolem functions . For the purpose of this work , however , we do not have to dive into the details of Incremental Determinization and can consider it simply as some backtracking algorithm . Correctness . Writing performant code is an error-prone task , and correctness is critical for many applications of formal reasoning . Some automated reasoning tools hence have the ability to produce proofs , which can be checked independently . CADET is one of the few QBF solvers that can produce proofs without runtime overhead . We believe that the ability to verify results of solvers is particularly crucial for learning applications , as it allows us to ensure that the reinforcement learning algorithm does not simply exploit implementation error ( bugs ) in the environment .

Summary: This will be an uncharacteristically short review. The work poses an interesting idea: why not mix heuristics and learning. It reads as if the paper was written a while ago and the intro was not updated, since there is a lot of related work using the same concept. Please cite existing work in the introduction, it reflects negatively on the paper.