Transformers satisfy

1 INTRODUCTION . The Constraint Satisfaction Problems ( CSP ) is of central importance in several aspects of computer science , including theoretical computer science , complexity theory , algorithmics , cryptography , and artificial intelligence . CSP aims at finding a consistent assignment of values to variables such that all constraints , which are typically defined over a finite domain , are satisfied . In particular , there is an assortment of problems arising from artificial intelligence and circuit design that can be reduced to CSP subtypes , including map coloring , vertex cover , independent set , dominating set , and clique detection . Solving a CSP on a finite domain is often an NP-complete problem with respect to the domain size . The conventional CSP-solvers rely on handcrafted heuristics that guide the search for satisfying assignments . These algorithms are focused on solving CSP via backtracking or local search . Hence , the resulted model is bounded by the greedy strategy , which is generally sub-optimal . With the advent of Graph Neural Networks ( Scarselli et al . ( 2009 ) ) , the geometric deep learning ( Bronstein et al . ( 2017 ) ) for Non-Euclidean data has become one of the most emerging fields of machine learning . In particular , it brought deep learning solutions to one of the most dominant combinatorial optimization problems , the Constraint Satisfaction Problem ( CSP ) ( Khalil et al . ( 2017 ) ) . Works including NeuroSAT ( Selsam et al . ( 2018 ) ) and Circuit-SAT ( Amizadeh et al . ( 2018 ) ) commenced the study of neural methods targeted at CSP . Later works , such as Yolcu & Póczos ( 2019 ) and You et al . ( 2019 ) , attempted to solve CSP through different deep learning approaches . However , most pioneering works , such as neural approaches utilizing RNN or Reinforcement Learning , are still restricted to sequential algorithms , while clauses are parallelizable even though they are strongly correlated through shared variables . In this work , we propose a hybrid model of the Transformer architecture ( Vaswani et al . ( 2017 ) ) and the Graph Neural Network for solving combinatorial problems , especially CSP . Our main contributions in this work are : ( a ) We derived meta-paths adopted from Sun et al . ( 2011 ) to formulate the message passing mechanism between homogeneous nodes ( i.e. , variable to variable , or clause to clause ) , which enable us to perform self-attention and let message pass through either variables sharing the same clauses , or clauses that include the same variables . We apply the cross-attention mechanism to optimize message exchanges between heterogeneous nodes ( i.e. , clause to variable , or variable to clause ) . ( b ) With the combination of homogeneous attention and heterogeneous atten- tion mechanisms on bipartite graph structure , we then combine Transformer with Neuro-Symbolic methods to resolve combinatorial optimization on graphs . ( c ) We proposed Heterogeneous Graph Transformer ( HGT ) , a general framework for graphs with heterogeneous nodes . In this work , we trained the HGT framework to approximate the solutions of CSP ( but not limited to CSP ) . Our model is able to achieve competitive accuracy , parallelism , and generality on CSP problems with arbitrary sizes 2 RELATED WORK . Recently , the Machine Learning community has seen an increasing interest in applications and optimizations related to constraint satisfaction problem solving . Various frameworks utilising diverse methodologies have been proposed , offering new insights into developing CSP solvers and classifiers . For example , Bello et al . ( 2016 ) adopts Reinforcement Learning in their Neural Combinatorial Optimization , with an approach based on policy gradients . On the other hand , works such as Evans et al . ( 2018 ) and Arabshahi et al . ( 2018 ) have demonstrated the effectiveness of recursive neural networks in modeling symbolic expressions . Meanwhile , Prates et al . ( 2019 ) proposed an embeddingbased message-passing algorithm for solving Traveling Salesman Problem ( TSP ) , a highly relevant CSP problem . NeuroSAT ( Selsam et al . ( 2018 ) ) is a graph neural network model that aims at solving the Boolean Satisfiability Problem ( SAT ) without leveraging the greedy search paradigm . It approaches SAT as a binary classification problem during training and finds an SAT assignment from the latent representations during inference . NeuroSAT is able to search for solutions to problems of various difficulties despite training for relatively small number of iterations . As an extension to this line of work , Selsam & Bjørner ( 2019 ) proposes a neural network that facilitates variable branching decision making within high-performance SAT solvers on real problems . PDP Amizadeh et al . ( 2019 ) is a generic neural framework for learning CSP solvers based on the idea of Propagation , Decimation , and Prediction . PDP provides a completely unsupervised training mechanism for solving SAT via energy minimization , and can be seen as learning optimal message passing strategy on probabilistic graphical models . G2SAT ( You et al . ( 2019 ) ) is a deep generative framework that learns to generate SAT formulas from a given set of input formulas while preserving the graph statistics . Even though G2SAT lacks the ability to derive solutions , it provides synthetic formulas for hyperparameter optimization . RLSAT ( Yolcu & Póczos ( 2019 ) ) learns SAT solvers through deep reinforcement learning and iterative refinement . It incorporates a graph neural network into a Stochastic Local Search ( SLS ) algorithm to act as the variable selection heuristic during training . However , since SLS begins with randomly initialized parameters , and a non-zero terminal reward is given only when a satisfying assignment is found , RLSAT requires a curriculum learning process for performance improvement . As a result , RLSAT becomes inefficient when its learning process starts with large complex graphs , in which satisfying assignments are hard to obtain . In our HGT , each possible state of assignment corresponds to a likelihood , which can be minimized to train the model . In particular , HGT is able to achieve optimal performance in efficiency and accuracy regardless of input graph sizes . 3 BACKGROUND . 3.1 CONSTRAINT SATISFACTION PROBLEMS . Constraint Satisfaction Problems ( CSP ) ( Kumar ( 1992 ) ) is a fundamental problem in logic study that constitutes the cornerstone of combinatorial optimization . It provides feasible models to real world applications and is intensively involved in the design of artificial intelligence . An instance of CSP problem , CSP ( V , U ) , is constituted of two main components : a set of N variables V = { vi ∈ D : i ∈ 1 ... N } , defined over a discrete domain D ; and a set of M constraint functions or factors , U = { uj ( qj ) : j ∈ 1 ... M } , where qj is a subset of V subject to uj . For each uj ∈ U , uj : D |qj | → { 0 , 1 } outputs 1 if the input qj satisfies constraint uj , and 0 otherwise . A CSP problem can be formulated in Conjunctive Normal Form ( CNF ) ( Pfahringer ( 2010 ) ) with the goal of finding an assignment of variables that satisfies all constraints . For a given assignment to V , the measure of a CSP problem , φ : { 0 , 1 } N → { 0 , 1 } , is a function that returns 1 if all the clauses in the problem are satisfied , and 0 otherwise . Such a measure function is in CNF if 1 . For all uj ∈ U , uj is a disjunction of literals . Here literals refer to vi or ¬vi , for vi ∈ qj ; 2. φ ( V ) = ∧uj , j = { 1 , ... , M } . 3.2 GRAPH NEURAL NETWORKS AND GRAPH ATTENTION NETWORKS . Graph Neural Network ( GNN ) ( Scarselli et al . ( 2009 ) ) provides a basic architecture to operate neural network on graphs . Suppose V is a set of nodes , where each vertex vi ∈ V carries a node feature hi ∈ RdV , and E ⊂ V × V is a set of undirected edges , where each edge ei , j ∈ E that connects vi to vj ∈ V carries an edge feature hi , j ∈ RdE . For time step t ∈ { 1 , ... , T } , a GNN on the graph G ( V , E ) passes messages between direct neighbors , and iteratively updates the feature vector of each vi ∈ V following : htvi = Q ht−1vi , ⋃ vj∈Ni M t ( ht−1vi , h t−1 vj , hi , j ) ( 1 ) where Ni = { vj ∈ V : ∃ei , j ∈ E } , and M , Q are the message function and the update function to be learned , respectively . 3.3 TRANSFORMER AND THE RELATION WITH GNN . Transformer ( Vaswani et al . ( 2017 ) ) is the state-of-the-art approach for sequential data and transduction problems that relies on self-attention mechanism . When looking into the architecture of Transformer , the computation of similarity between queries and keys highly resembles finding a correlation between two types of nodes in fully connected bipartite graphs , while the self-attention mechanism among queries or keys ( or tokens in NLP terminology ) can be seen as calculating the weighted contribution in a half-complete connected graph , where an output token does not depend on future words . With these resemblance , we want to extend the Transformer architecture to heterogeneous graph structures , such as bipartite graphs . 4 METHODOLOGY . 4.1 FACTOR GRAPHS AND META-PATHS . The value of each constraint is binary , which makes it possible to express the measure of a CNF as : φ ( V , U ) = M∏ j=1 uj ( qj ) ( 2 ) The expression can be properly presented as an undirected bipartite factor graph . We construct such a factor graph G ( ( V , U ) , E ) by defining the set of variables V = { v1 , . . . , vn } , the set of clauses U = { u1 , . . . , um } , and edges E by : ei , j ∈ E iff variable vi is involved in constraint uj either in positive or negative relation . In Yolcu & Póczos ( 2019 ) , each edge is assigned with a type depending on the polarity of the variable it connects to . The positive occurrence of a variable vi in a clause ( or factor ) uj is represented with the positive sign ( + ) , whereas its negative occurrence ¬vi in uj is represented with the negative sign ( − ) . Hence , a pair of n×m bi-adjacency matrix A = ( A+ , A− ) is used to store two types of edges such that A+ ( i , j ) = 1⇔ vi ∈ uj and A− ( i , j ) = 1⇔ ¬vi ∈ uj . Here vi ∈ uj implies that vi instead of its negation ¬vi is directly involved in uj . Each edge ei , j ∈ E is then assigned a value equals to 1 for edges in A+ and −1 for edges in A− . With the factor graph representation , graph neural network can be applied as a CSP-solver ( Selsam et al . ( 2018 ) ; Yolcu & Póczos ( 2019 ) ) . However , due to the fact that factor graph is bipartite in which variables are only connected to clauses , clauses are only connected to variables . Furthermore , traditional GNN only passes messages to local neighbors , and is thus inefficient at passing messages between variables or between clauses . Consequently , variable nodes update their state solely based on their current states and the states of the clauses they affect , vice versa . In this work , we propose to pass message through length-2 meta-paths in addition to existing edges , which enables : 1 ) variables to incorporate the information from variables that share the same clauses during state updates ; 2 ) clauses to incorporate the information from clauses that share some variables when update their states . In a CSP factor graph , we define that a meta-path mi , j = ( vi , vk , vj ) between nodes vi and vj exists if there exists some vk ∈ V s.t . ∃ei , k ∈ E and ∃ek , j ∈ E . Since self-attention mechanism is not symmetric , our meta-path is directed . As a result , we get four types of meta-paths in total ( i.e. , { ( + , + ) , ( + , − ) , ( − , + ) , ( − , − ) } ) , as illustrated in Figure 1 ( c ) . The adjacent matrix of such a meta-path can be easily computed by matrix multiplication of A+ and A− or their transposes . Take A ( + , + ) , A ( + , − ) as examples , the adjacency matrix A ( + , + ) = A+AT+ stores all ( + , + ) meta-paths , and A ( + , − ) = A+AT− stores all ( + , − ) meta-paths . A diagonal entry A ( + , + ) [ i , i ] indicates the number of positive edges that vi has , and an off-diagonal entry A ( + , + ) [ i , j ] indicates the existence of ( + , + ) meta-path from vi to vj .

Summary: This paper presents Heterogeneous Graph Transformer (HGT), a new architecture that combines useful properties of GNNs and Transformers to design HGT for combinatorial reasoning problems, particularly Boolean Satisfiability (SAT). The idea of combining these two powerful models is appealing. The motivation behind considering both homogenous attention (between two literals, or between two clauses) and heterogenous attention (between a literal and a clause) seems new and interesting too. However, there are several missing pieces in the paper in the present form, as outlined below.