Transformers satisfy

1 INTRODUCTION . The Constraint Satisfaction Problems ( CSP ) is of central importance in several aspects of computer science , including theoretical computer science , complexity theory , algorithmics , cryptography , and artificial intelligence . CSP aims at finding a consistent assignment of values to variables such that all constraints , which are typically defined over a finite domain , are satisfied . In particular , there is an assortment of problems arising from artificial intelligence and circuit design that can be reduced to CSP subtypes , including map coloring , vertex cover , independent set , dominating set , and clique detection . Solving a CSP on a finite domain is often an NP-complete problem with respect to the domain size . The conventional CSP-solvers rely on handcrafted heuristics that guide the search for satisfying assignments . These algorithms are focused on solving CSP via backtracking or local search . Hence , the resulted model is bounded by the greedy strategy , which is generally sub-optimal . With the advent of Graph Neural Networks ( Scarselli et al . ( 2009 ) ) , the geometric deep learning ( Bronstein et al . ( 2017 ) ) for Non-Euclidean data has become one of the most emerging fields of machine learning . In particular , it brought deep learning solutions to one of the most dominant combinatorial optimization problems , the Constraint Satisfaction Problem ( CSP ) ( Khalil et al . ( 2017 ) ) . Works including NeuroSAT ( Selsam et al . ( 2018 ) ) and Circuit-SAT ( Amizadeh et al . ( 2018 ) ) commenced the study of neural methods targeted at CSP . Later works , such as Yolcu & Póczos ( 2019 ) and You et al . ( 2019 ) , attempted to solve CSP through different deep learning approaches . However , most pioneering works , such as neural approaches utilizing RNN or Reinforcement Learning , are still restricted to sequential algorithms , while clauses are parallelizable even though they are strongly correlated through shared variables . In this work , we propose a hybrid model of the Transformer architecture ( Vaswani et al . ( 2017 ) ) and the Graph Neural Network for solving combinatorial problems , especially CSP . Our main contributions in this work are : ( a ) We derived meta-paths adopted from Sun et al . ( 2011 ) to formulate the message passing mechanism between homogeneous nodes ( i.e. , variable to variable , or clause to clause ) , which enable us to perform self-attention and let message pass through either variables sharing the same clauses , or clauses that include the same variables . We apply the cross-attention mechanism to optimize message exchanges between heterogeneous nodes ( i.e. , clause to variable , or variable to clause ) . ( b ) With the combination of homogeneous attention and heterogeneous atten- tion mechanisms on bipartite graph structure , we then combine Transformer with Neuro-Symbolic methods to resolve combinatorial optimization on graphs . ( c ) We proposed Heterogeneous Graph Transformer ( HGT ) , a general framework for graphs with heterogeneous nodes . In this work , we trained the HGT framework to approximate the solutions of CSP ( but not limited to CSP ) . Our model is able to achieve competitive accuracy , parallelism , and generality on CSP problems with arbitrary sizes 2 RELATED WORK . Recently , the Machine Learning community has seen an increasing interest in applications and optimizations related to constraint satisfaction problem solving . Various frameworks utilising diverse methodologies have been proposed , offering new insights into developing CSP solvers and classifiers . For example , Bello et al . ( 2016 ) adopts Reinforcement Learning in their Neural Combinatorial Optimization , with an approach based on policy gradients . On the other hand , works such as Evans et al . ( 2018 ) and Arabshahi et al . ( 2018 ) have demonstrated the effectiveness of recursive neural networks in modeling symbolic expressions . Meanwhile , Prates et al . ( 2019 ) proposed an embeddingbased message-passing algorithm for solving Traveling Salesman Problem ( TSP ) , a highly relevant CSP problem . NeuroSAT ( Selsam et al . ( 2018 ) ) is a graph neural network model that aims at solving the Boolean Satisfiability Problem ( SAT ) without leveraging the greedy search paradigm . It approaches SAT as a binary classification problem during training and finds an SAT assignment from the latent representations during inference . NeuroSAT is able to search for solutions to problems of various difficulties despite training for relatively small number of iterations . As an extension to this line of work , Selsam & Bjørner ( 2019 ) proposes a neural network that facilitates variable branching decision making within high-performance SAT solvers on real problems . PDP Amizadeh et al . ( 2019 ) is a generic neural framework for learning CSP solvers based on the idea of Propagation , Decimation , and Prediction . PDP provides a completely unsupervised training mechanism for solving SAT via energy minimization , and can be seen as learning optimal message passing strategy on probabilistic graphical models . G2SAT ( You et al . ( 2019 ) ) is a deep generative framework that learns to generate SAT formulas from a given set of input formulas while preserving the graph statistics . Even though G2SAT lacks the ability to derive solutions , it provides synthetic formulas for hyperparameter optimization . RLSAT ( Yolcu & Póczos ( 2019 ) ) learns SAT solvers through deep reinforcement learning and iterative refinement . It incorporates a graph neural network into a Stochastic Local Search ( SLS ) algorithm to act as the variable selection heuristic during training . However , since SLS begins with randomly initialized parameters , and a non-zero terminal reward is given only when a satisfying assignment is found , RLSAT requires a curriculum learning process for performance improvement . As a result , RLSAT becomes inefficient when its learning process starts with large complex graphs , in which satisfying assignments are hard to obtain . In our HGT , each possible state of assignment corresponds to a likelihood , which can be minimized to train the model . In particular , HGT is able to achieve optimal performance in efficiency and accuracy regardless of input graph sizes . 3 BACKGROUND . 3.1 CONSTRAINT SATISFACTION PROBLEMS . Constraint Satisfaction Problems ( CSP ) ( Kumar ( 1992 ) ) is a fundamental problem in logic study that constitutes the cornerstone of combinatorial optimization . It provides feasible models to real world applications and is intensively involved in the design of artificial intelligence . An instance of CSP problem , CSP ( V , U ) , is constituted of two main components : a set of N variables V = { vi ∈ D : i ∈ 1 ... N } , defined over a discrete domain D ; and a set of M constraint functions or factors , U = { uj ( qj ) : j ∈ 1 ... M } , where qj is a subset of V subject to uj . For each uj ∈ U , uj : D |qj | → { 0 , 1 } outputs 1 if the input qj satisfies constraint uj , and 0 otherwise . A CSP problem can be formulated in Conjunctive Normal Form ( CNF ) ( Pfahringer ( 2010 ) ) with the goal of finding an assignment of variables that satisfies all constraints . For a given assignment to V , the measure of a CSP problem , φ : { 0 , 1 } N → { 0 , 1 } , is a function that returns 1 if all the clauses in the problem are satisfied , and 0 otherwise . Such a measure function is in CNF if 1 . For all uj ∈ U , uj is a disjunction of literals . Here literals refer to vi or ¬vi , for vi ∈ qj ; 2. φ ( V ) = ∧uj , j = { 1 , ... , M } . 3.2 GRAPH NEURAL NETWORKS AND GRAPH ATTENTION NETWORKS . Graph Neural Network ( GNN ) ( Scarselli et al . ( 2009 ) ) provides a basic architecture to operate neural network on graphs . Suppose V is a set of nodes , where each vertex vi ∈ V carries a node feature hi ∈ RdV , and E ⊂ V × V is a set of undirected edges , where each edge ei , j ∈ E that connects vi to vj ∈ V carries an edge feature hi , j ∈ RdE . For time step t ∈ { 1 , ... , T } , a GNN on the graph G ( V , E ) passes messages between direct neighbors , and iteratively updates the feature vector of each vi ∈ V following : htvi = Q ht−1vi , ⋃ vj∈Ni M t ( ht−1vi , h t−1 vj , hi , j ) ( 1 ) where Ni = { vj ∈ V : ∃ei , j ∈ E } , and M , Q are the message function and the update function to be learned , respectively . 3.3 TRANSFORMER AND THE RELATION WITH GNN . Transformer ( Vaswani et al . ( 2017 ) ) is the state-of-the-art approach for sequential data and transduction problems that relies on self-attention mechanism . When looking into the architecture of Transformer , the computation of similarity between queries and keys highly resembles finding a correlation between two types of nodes in fully connected bipartite graphs , while the self-attention mechanism among queries or keys ( or tokens in NLP terminology ) can be seen as calculating the weighted contribution in a half-complete connected graph , where an output token does not depend on future words . With these resemblance , we want to extend the Transformer architecture to heterogeneous graph structures , such as bipartite graphs . 4 METHODOLOGY . 4.1 FACTOR GRAPHS AND META-PATHS . The value of each constraint is binary , which makes it possible to express the measure of a CNF as : φ ( V , U ) = M∏ j=1 uj ( qj ) ( 2 ) The expression can be properly presented as an undirected bipartite factor graph . We construct such a factor graph G ( ( V , U ) , E ) by defining the set of variables V = { v1 , . . . , vn } , the set of clauses U = { u1 , . . . , um } , and edges E by : ei , j ∈ E iff variable vi is involved in constraint uj either in positive or negative relation . In Yolcu & Póczos ( 2019 ) , each edge is assigned with a type depending on the polarity of the variable it connects to . The positive occurrence of a variable vi in a clause ( or factor ) uj is represented with the positive sign ( + ) , whereas its negative occurrence ¬vi in uj is represented with the negative sign ( − ) . Hence , a pair of n×m bi-adjacency matrix A = ( A+ , A− ) is used to store two types of edges such that A+ ( i , j ) = 1⇔ vi ∈ uj and A− ( i , j ) = 1⇔ ¬vi ∈ uj . Here vi ∈ uj implies that vi instead of its negation ¬vi is directly involved in uj . Each edge ei , j ∈ E is then assigned a value equals to 1 for edges in A+ and −1 for edges in A− . With the factor graph representation , graph neural network can be applied as a CSP-solver ( Selsam et al . ( 2018 ) ; Yolcu & Póczos ( 2019 ) ) . However , due to the fact that factor graph is bipartite in which variables are only connected to clauses , clauses are only connected to variables . Furthermore , traditional GNN only passes messages to local neighbors , and is thus inefficient at passing messages between variables or between clauses . Consequently , variable nodes update their state solely based on their current states and the states of the clauses they affect , vice versa . In this work , we propose to pass message through length-2 meta-paths in addition to existing edges , which enables : 1 ) variables to incorporate the information from variables that share the same clauses during state updates ; 2 ) clauses to incorporate the information from clauses that share some variables when update their states . In a CSP factor graph , we define that a meta-path mi , j = ( vi , vk , vj ) between nodes vi and vj exists if there exists some vk ∈ V s.t . ∃ei , k ∈ E and ∃ek , j ∈ E . Since self-attention mechanism is not symmetric , our meta-path is directed . As a result , we get four types of meta-paths in total ( i.e. , { ( + , + ) , ( + , − ) , ( − , + ) , ( − , − ) } ) , as illustrated in Figure 1 ( c ) . The adjacent matrix of such a meta-path can be easily computed by matrix multiplication of A+ and A− or their transposes . Take A ( + , + ) , A ( + , − ) as examples , the adjacency matrix A ( + , + ) = A+AT+ stores all ( + , + ) meta-paths , and A ( + , − ) = A+AT− stores all ( + , − ) meta-paths . A diagonal entry A ( + , + ) [ i , i ] indicates the number of positive edges that vi has , and an off-diagonal entry A ( + , + ) [ i , j ] indicates the existence of ( + , + ) meta-path from vi to vj .

Summary: The paper presents a model for inferring the solution of a constraint satisfaction problem over boolean variables expressed in *Conjunctive Normal Form (CNF)*. The proposed model builds on existing works which represent the factor graph of the CNF as a bipartite graph, and use a graph neural network for the learning task. It re-defines the graph to facilitate the message passing among the nodes which are in the same part of the graph (i.e. CNF variables/clauses). It also introduces a cross-attention mechanism between the nodes belonging to different parts.