Improving SAT Solver Heuristics with Graph Networks and Reinforcement Learning

1 INTRODUCTION . Boolean satisfiability ( SAT ) is an important problem for both industry and academia impacting various fields , including circuit design , computer security , artificial intelligence , automatic theorem proving , and combinatorial optimization . As a result , modern SAT solvers are well-crafted , sophisticated , reliable pieces of software that can scale to problems with hundreds of thousands of variables ( Ohrimenko et al. , 2009 ) . SAT is known to be NP-complete ( Karp , 1972 ) , and most state-of-the-art open-source and commercial solvers rely on multiple heuristics to speed up the exhaustive search , which is otherwise intractable . These heuristics are usually meticulously crafted using expert domain knowledge and are often iterated on using trial and error . In this paper , we investigate how we can use machine learning to improve upon an existing branching heuristic without leveraging domain expertise . We present Graph-Q-SAT ( GQSAT ) , a branching heuristic in a Conflict Driven Clause Learning ( Marques-Silva & Sakallah , 1999 ; Bayardo Jr & Schrag , 1997 , CDCL ) SAT solver trained with value-based reinforcement learning ( RL ) , based on DQN ( Mnih et al. , 2015 ) . GQSAT uses a graph representation of SAT problems similar to Selsam et al . ( 2018 ) which provides permutation and variable relabeling invariance . It uses a Graph Neural Network ( Gori et al. , 2005 ; Battaglia et al. , 2018 , GNN ) as a function approximator to provide generalization as well as support for a dynamic state-action space . GQSAT uses a simple state representation and a binary reward that requires no feature engineering or problem domain knowledge . GQSAT modifies only part of the CDCL based solver , keeping it complete , i.e . always leading to a correct solution . We demonstrate that GQSAT outperforms Variable State Independent Decaying Sum ( Moskewicz et al. , 2001 , VSIDS ) , most frequently used CDCL branching heuristic , reducing the number of iterations required to solve SAT problems by 2-3X . GQSAT is trained to examine the structure of the particular problem instance to make better decisions at the beginning of the search whereas the VSIDS heuristic suffers from bad decision during the warm-up period . We show that our method generalizes to problems five times larger than those it was trained on . We also show that our method generalizes across problem types from SAT to unSAT . We also show , to a lesser extent , it generalizes to SAT problems from different domains , such as graph colouring . Finally , we show that some of these improvements are achieved even when training is limited to single SAT problem demonstrating data efficiency of our method . We believe GQSAT is a stepping stone to a new generation of SAT solvers leveraging data to build better heuristics learned from past experience . 2 BACKGROUND . 2.1 SAT PROBLEM . A SAT problem involves finding variable assignments such that a propositional logic formula is satisfied or showing that such an assignment does not exist . A propositional formula is a Boolean expression , including Boolen variables , ANDs , ORs and negations . ’ x ’ or ’ NOT x ’ make up a literal . It is convenient to represent Boolean formulas in conjunctive normal form ( CNF ) , i.e. , conjunctions ( AND ) of clauses , where a clause is a disjunction ( OR ) of literals . An example of a CNF is ( x1 ∨ ¬x2 ) ∧ ( x2 ∨ ¬x3 ) , where ∧ , ∨ , ¬ are AND , OR , and negation respectively . This CNF has two clauses : ( x1 ∨ ¬x2 ) and ( x2 ∨ ¬x3 ) . In this work , we use SAT to both denote the Boolean Satisfiability problem and a satisfiable instance , which should be clear from the context . We use unSAT to denote unsatisfiable instances . There are many types of SAT solvers . In this work , we focus on CDCL solvers , MiniSat ( Sorensson & Een , 2005 ) in particular , because it is an open source , minimal , but powerful implementation . A CDCL solver repeats the following steps : every iteration it picks a literal , assigns a variable a binary value . This is called a decision . After deciding , the solver simplifies the formula building an implication graph , and checks whether a conflict emerged . Given a conflict , it can infer ( learn ) new clauses and backtrack to the variable assignments where the newly learned clause becomes unit ( consisting of a single literal ) forcing a variable assignment which avoids the previous conflict . Sometimes , CDCL solver undoes all the variable assignments keeping the learned clauses to escape futile regions of the search space . This is called a restart . We focus on the branching heuristic because it is one of the most heavily used during the solution procedure . The branching heuristic is responsible for picking the variable and assigning some value to it . VSIDS ( Moskewicz et al. , 2001 ) is one of the most used CDCL branching heuristics . It is a counter-based heuristic which keeps a scalar value for each literal or variable ( MiniSat uses the latter ) . These values are increased every time a variable gets involved in a conflict . The algorithm behaves greedily with respect to these values called activities . Activities are usually initialized with zeroes ( Liang et al. , 2015 ) . 2.2 REINFORCEMENT LEARNING . We formulate the RL problem as a Markov decision process ( MDP ) . An MDP is a tuple 〈S , A , R , T , ρ , γ〉with a set of states S , a set of actionsA , a reward functionR = R ( s , a , s′ ) and the transition function T = p ( s , a , s′ ) , where p ( s , a , s′ ) is a probability distribution , s , s′ ∈ S , a ∈ A. ρ is the probability distribution over initial states . γ ∈ [ 0 , 1 ) is the discount factor responsible for trading off the preferences between the current immediate reward and the future reward . In the case of episodic tasks , the state space is split into the set of non-terminal states and the terminal state S+ . To solve an MDP means to find an optimal policy , a mapping which outputs an action or distribution over actions given the state , such that we maximize the expected discounted cumulative return R = E [ ∑∞ t=0 γ trt ] , where rt = R ( st , at , st+1 ) is the reward for the transition from st to st+1 . In Section 3 we apply deep Q-networks ( Mnih et al. , 2015 , DQN ) , a value-based RL algorithm that approximates an optimal Q-function , an action-value function that estimates the sum of future rewards after taking an action a in state s and following the optimal policy π thereafter : Q∗ ( s , a ) = Eπ , T , ρ [ R ( s , a , s′ ) + γmaxa′ Q∗ ( s′ , a′ ) ] . A mean squared temporal difference ( TD ) error is used to make an update step : L ( θ ) = ( Qθ ( s , a ) − r − γmaxa′ Qθ̄ ( s′ , a′ ) ) 2 . Qθ̄ is called a target network ( Mnih et al. , 2015 ) . It is used to stabilise DQN by splitting the decision and evaluation operations . Its weights are copied from the main network Qθ after each k minibatch updates . 2.3 GRAPH NEURAL NETWORKS . We use Graph Neural Networks ( Gori et al. , 2005 , GNN ) to approximate ourQ-function due to their input size , structure , and permutation invariance . We use the formalism of Battaglia et al . ( 2018 ) which unifies most existing GNN approaches . Under this formalism , GNN is a set of functions that take a labeled graph as input and output a graph with modified labels but the same topology . Here , a graph is a directed graph 〈V , E , U〉 , where V is the set of vertices , E is the set of edges with e = ( s , r ) ∈ E , s , r ∈ V , and U is a global attribute . The global attribute contains information , relevant to the whole graph . We call vertices , edges and the global attribute entities . Each entity has its features vectors . A GNN changes this features as a result of its operations . A graph network can be seen as a set of six functions : three update functions and three aggregation functions . The information propagates between vertices along graph edges . Update functions compute new entity labels . Aggregation functions exist to ensure the GNN ’ s ability to process graphs of arbitrary topologies , compressing multiple entities features into vectors of fixed size . GNN blocks can be combined such that the output of one becomes input of the other . For example , the EncodeProcess-Decode architecture ( Battaglia et al. , 2018 ) processes the graph in a recurrent way , enabling information propagation between remote vertices . 3 GQSAT . As noted in Section 2.2 , we use the MDP formalism for our purposes . Each state of our MDP consists of unassigned variables and unsatisfied clauses containing these variables . The initial state distribution is a distribution over all possible SAT problems . Our problem has an episodic nature with a clear terminal state : when a satisfying assignment is found or the algorithm has exhausted all the possible options proving unSAT . The action set includes two actions for each unassigned variable : assigning it to true or false . We modify the MiniSat-based environment of Wang & Rompf ( 2018 ) which is responsible for the transition function . It takes the actions , modifies its implication graph internally and returns a new state , containing newly learned clauses and without the variables removed after the propagation . Strictly speaking , this state is not fully observable . In the case of a conflict , the solver undoes the assignments for variables that are not in the agent ’ s observation . However , in practice , this should not inhibit the goal of quickly pruning the search tree : the information in the state is enough to pick a variable that leads to more propagations in the remaining formula . We use a simple reward function : the agent gets a negative reinforcement p for each non-terminal transition and 0 for reaching the terminal state . This reward encourages an agent to finish an episode as quickly as possible and does not require elaborate reward shaping to start using GQSAT . 3.1 STATE REPRESENTATION . We represent a SAT problem as a graph similar to Selsam et al . ( 2018 ) . We make it more compact , using vertices to denote variables instead of literals . We use nodes to encode clauses as well . Our state representation is simple and does not require scrupulous feature engineering . An edge ( xi , ci ) means that a clause ci contains literal xi . If a literal contains a negation , a corresponding edge has a [ 1 , 0 ] label and [ 0 , 1 ] otherwise . GNN process directed graphs so we create two directed edges with the same labels : from a variable to a clause and vice-versa . Vertex features are two dimensional one-hot vectors , denoting either a variable or a clause . We do not provide any other information to the model . The global attribute input is empty and is only used for message passing . Figure 1a gives an example of the state for ( x1 ∨ x2 ) ∧ ( ¬x2 ∨ x3 ) .

Summary: The paper proposes learning a branching heuristic to be used inside the SAT solver MiniSat using reinforcement learning. The state is represented as a graph representation of the Boolean formula as in previous works, and the policy is parameterized as a graph neural network. At each step of an episode the policy selects a variable to branch on and assigns a value to it. The episode terminates once the solver finds a satisfying assignment or proves unsatisfiability. The reward function encourages the policy to reach terminal state in as few steps as possible. The policy is trained using DQN. Results on randomly generated SAT instances show that the learned policy is able to solve problems with fewer steps than VSIDS, the branching heuristic commonly used by state-of-the-art solvers.