SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shiqiang Zhang*
Imperial College London London, UK
Juan S. Campos
Christian Feldmann
David Walz
Frederik Sandfort
Miriam Mathea
Calvin Tsay
Ruth Misener
Optimization over trained machine learning models has applications including: verification, minimizing neural acquisition functions, and integrating a trained surrogate into a larger decision-making problem. This paper formulates and solves optimization problems constrained by trained graph neural networks (GNNs). To circumvent the symmetry issue caused by graph isomorphism, we propose two types of symmetry-breaking constraints: one indexing a node 0 and one indexing the remaining nodes by lexicographically ordering their neighbor sets. To guarantee that adding these constraints will not remove all symmetric solutions, we construct a graph indexing algorithm and prove that the resulting graph indexing satisfies the proposed symmetry-breaking constraints. For the classical GNN architectures considered in this paper, optimizing over a GNN with a fixed graph is equivalent to optimizing over a dense neural network. Thus, we study the case where the input graph is not fixed, implying that each edge is a decision variable, and develop two mixed-integer optimization formulations. To test our symmetry-breaking strategies and optimization formulations, we consider an application in molecular design.
Graph neural networks (GNNs) [1-3] are designed to operate on graph-structured data. By passing messages between nodes via edges (or vice versa), GNNs can efficiently capture and aggregate local information within graphs. GNN architectures including spectral approaches [4-9] and spatial approaches [10-18], are proposed based on various motivations. Due to their ability to learn non-Euclidean data structure, GNNs have recently been applied to many graph-specific tasks, demonstrating incredible performance. In drug discovery, for example, GNNs can predict molecular properties given the graph representation of molecules [16, 19-23]. Other applications using GNNs include graph clustering[24, 25], text classification [26, 27], and social recommendation [28, 29].
Although GNNs are powerful tools for these "forward" prediction tasks, few works discuss the "backward" (or inverse) problem defined on trained GNNs. Specifically, many applications motivate the inverse problem of using machine learning (ML) to predict the inputs corresponding to some output specification(s). For example, computer-aided molecular design (CAMD) [30-32] aims to
*Corresponding author: s.zhang21@imperial.ac.uk
Figure 1: Illustration of forward and backward problems defined on trained GNNs.
design the optimal molecule(s) for a given target based on a trained ML model [33-36]. Several GNN-based approaches have been proposed for CAMD [37-40], however, these works still use GNNs as "forward" functions to make predictions over graph-domain inputs. Therefore, both the graph structures of inputs and the inner structures of the trained GNNs are ignored. Figure 1 conceptually depicts the difference between forward and backward/inverse problems.
Mathematical optimization over trained ML models is an area of increasing interest. In this paradigm, a trained ML model is translated into an optimization formulation, thereby enabling decision-making problems over model predictions. For continuous, differentiable models, these problems can be solved via gradient-based methods [41-44]. Mixed-integer programming (MIP) has also been proposed, mainly to support non-smooth models such as ReLU neural networks [45-50], their ensembles [51], and tree ensembles [52-54]. Optimization over trained ReLU neural networks has been especially prominent [55], finding applications such as verification [56-64], reinforcement learning [65-67], compression [68], and black-box optimization [69]. There is no reason to exclude GNNs from this rich field. Wu et al. [70] managed to verify a GNN-based job scheduler, where MIP is used to obtain tighter bounds in a forward-backward abstraction refinement framework. Mc Donald [71] built MIP formulations for GCN [6] and GraphSAGE [13] and tested their performance on a CAMD problem, which is the first work to directly apply MIP on GNNs to the best of our knowledge.
Optimization over trained GNNs involves two significant challenges. One is the symmetry issue arising from the permutation invariance of GNNs (i.e., isomorphism graphs share the same GNN output(s)). For forward problems, this invariance is preferred since it admits any graph representation for inputs. From optimization perspective, however, symmetric solutions can significantly enlarge the feasible domain and impede a branch-and-bound (B&B) search tree. One way to break symmetry in integer programs is by adding constraints to remove symmetric solutions [72-78]. The involvement of graphs not only adds complexity to the problem in terms of symmetry, but also in terms of implementation. Due to the complexity and variety of GNN architectures, a general framework is needed. This framework should be compatible with symmetry-breaking techniques.
This paper first defines optimization problems on trained GNNs. To handle the innate symmetry, we propose two sets of constraints to remove most isomorphism graphs (i.e., different indexing for any abstract graph). To guarantee that these constraints do not exclude all possible symmetries (i.e., any abstract graph should have at least one feasible indexing), we design an algorithm to index any undirected graph and prove that the resulting indexing is feasible under the proposed constraints. To encode GNNs, we build a general framework for MIP on GNNs based on the open-source Optimization and Machine Learning Toolkit (OMLT) [79]. Many classic GNN architectures are supported after proper transformations. Two formulations are provided and compared on a CAMD problem. Numerical results show the outstanding improvement of symmetry-breaking.
Paper structure: Section 2 introduces the problem definition, proposes the symmetry-breaking constraints, and gives theoretical guarantees. Section 3 discusses the application, implementation details, and numerical results. Section 4 concludes and discusses future work.
Prediction (Forward): What are the properties for a given molecule?
2.1 Definition of optimization over trained GNNs
We consider optimization over a trained GNN on a given dataset  $D = \{(X^i, A^i), y^i\}_{i=1}^M$ . Each sample in the dataset consists of an input graph (with features  $X^i$  and adjacency matrix  $A^i$ ) and an output property  $y^i$ . The target of training is to approximate properties, that is:
$$
G N N (X ^ {i}, A ^ {i}) \approx y ^ {i}, \forall 1 \leq i \leq M
$$
When optimizing over a trained GNN, the goal is to find the input with optimal property:
$$
(X ^ {*}, A ^ {*}) = \underset {(X, A)} {\arg \min } G N N (X, A)
$$
$$
s. t. f _ {j} (X, A) \leq 0, j \in \mathcal {J} \tag {OPT}
$$
$$
g _ {k} (X, A) = 0, k \in \mathcal {K}
$$
where  $f_{j}, g_{k}$  are problem-specific constraints and  $\mathcal{J}, \mathcal{K}$  are index sets. Note that optimality of (OPT) is defined on the trained GNN instead of true properties. To simplify our presentation, we will focus on undirected graphs with node features, noting that directed graphs can be naturally transformed into undirected graphs. Likewise, using edge features does not influence our analyses. We discuss practical impacts of such modifications later.
2.2 Symmetry handling
For any input  $(X,A)$ , assume that there are  $N$  nodes, giving a  $N\times N$  adjacency matrix  $A$  and a feature matrix  $X = (X_0,X_1,\ldots ,X_{N - 1})^T\in \mathbb{R}^{N\times F}$ , where  $X_{i}\in \mathbb{R}^{F}$  contains the features for  $i$ -th node. For any permutation  $\gamma$  over set  $[N]\coloneqq \{0,1,2\dots ,N - 1\}$ , let:
$$
A _ {u, v} ^ {\prime} = A _ {\gamma (u), \gamma (v)}, X _ {v} ^ {\prime} = X _ {\gamma (v)}, \forall u, v \in [ N ]
$$
Then the permuted input  $(X', A')$  has isomorphic graph structure, and therefore the same output, due to the permutation invariance of GNNs (i.e.,  $GNN(X, A) = GNN(X', A')$ ). In other words, the symmetries result from different indexing for the same graph. In general, there exist  $N!$  ways to index  $N$  nodes, each of which corresponds to one solution of (OPT). Mc Donald [71] proposed a set of constraints to force molecules connected, which can also be used to break symmetry in our general setting. Mathematically, they can be written as:
$$
\forall v \in [ N ] \backslash \{0 \}, \exists u <   v, s. t. A _ {u, v} = 1 \tag {S1}
$$
Constraints (S1) require each node (except for node 0) to be linked with a node with smaller index. Even though constraints (S1) help break symmetry, there can still exist many symmetric solutions. To resolve, or at least relieve, this issue, we need to construct more symmetry-breaking constraints.
Breaking symmetry at the feature level. We can first design some constraints on the features to define a starting point for the graph (i.e., assigning index 0 to a chosen node). Note that multiple nodes may share the same features. Specifically, we define an arbitrary function  $h: \mathbb{R}^F \to \mathbb{R}$ , to assign a hierarchy to each node and force that node 0 has the minimal function value:
$$
h \left(X _ {0}\right) \leq h \left(X _ {v}\right), \forall v \in [ N ] \backslash \{0 \} \tag {S2}
$$
The choice of  $h$  can be application-specific, for example, as in Section 3.1. The key is to define  $h$  such that not too many nodes share the same minimal function value.
Breaking symmetry at the graph level. It is unlikely that constraints on features will break much of the symmetry. After adding constraints (S1) and (S2), any neighbor of node 0 could be indexed 1, then any neighbor of node 0 or 1 could be indexed 2, and so on. We want to limit the number of possible indexing. A natural idea is to account for neighbors of each node: the neighbor set of a node with smaller index should also have smaller lexicographical order.
Before further discussion, we need to define the lexicographical order. Let  $S(M,L)$  be the set of all non-decreasing sequences with  $L$  integer elements in  $[0,M]$  (i.e.,  $S(M,L) = \{(a_1,a_2,\dots ,a_L)\mid 0\leq a_1\leq a_2\leq \dots \leq a_L\leq M\}$ ). For any  $a\in S(M,L)$ , denote its lexicographical order by  $LO(a)$ . Then for any  $a,b\in S(M,L)$ , we have:
$$
L O (a) <   L O (b) \Leftrightarrow \exists l \in \{1, 2, \dots , L \}, \text {s . t .} \left\{ \begin{array}{l} a _ {i} = b _ {i}, \quad 1 \leq i <   l \\ a _ {l} <   b _ {l} \end{array} \right. \tag {LO}
$$
For any multiset  $A$  with no more than  $L$  integer elements in  $[0, M - 1]$ , we first sort all elements in  $A$  in a non-decreasing order, then add elements with value  $M$  until the length equals to  $L$ . In that way, we build an injective mapping from  $A$  to a sequence  $a \in S(M, L)$ . Define the lexicographical order of  $A$  by  $LO(a)$ . For simplicity, we use  $LO(A)$  to denote the lexicographical order of  $A$ .
Algorithm 1 Indexing algorithm
Input:  $G = (V,E)$  with node set  $V = \{v_{0},v_{1},\ldots ,v_{N - 1}\}$ $(N\coloneqq |V|)$  . Denote the neighbor set of node  $v$  as  $\mathcal{N}(v),\forall v\in V$ $\mathcal{I}(v_0)\gets 0$ $\triangleright$  Assume that  $v_{0}$  is indexed with O   
 $s\gets 1$ $\triangleright$  Index for next node   
 $V_{1}^{1}\leftarrow \{v_{0}\}$ $\triangleright$  Initialize set of indexed nodes   
while  $s <   N$  do   
 $V_{2}^{s}\gets V\backslash V_{1}^{s}$ $\triangleright$  Set of unindexed nodes   
 $\mathcal{N}^s (v)\gets \{\mathcal{I}(u)\mid u\in \mathcal{N}(v)\cap V_1^s\} ,\forall v\in V_2^s$ $\triangleright$  Obtain all indexed neighbors   
rank(s(v)  $\leftarrow |\{LO(\mathcal{N}^s (u)) < LO(\mathcal{N}^s (v))|\forall u\in V_2^s\} |,\forall v\in V_2^s$  Assign a rank to each unindexed node   
 $\mathcal{I}^s (v)\gets \left\{\begin{array}{ll}\mathcal{I}(v), & \forall v\in V_1^s\\ \text{rank}^s (v) + s, & \forall v\in V_2^s \end{array}\right.$  Assign temporary indexes   
 $\mathcal{N}_t^s (v)\gets \{\mathcal{I}^s (u)\mid u\in \mathcal{N}(v)\} ,\forall v\in V_2^s$ $\triangleright$  Define temporary neighbor sets based on  $\mathcal{I}^s$ $v^{s}\gets \arg \min_{v\in V_{2}^{s}}LO(\mathcal{N}_{t}^{s}(v))$  Neighbors of  $v^{s}$  has minimal order If multiple nodes share the same minimal order, arbitrarily choose one   
 $\mathcal{I}(v^{s}) = s$  Index s to node  $v^{s}$ $V_{1}^{s + 1}\gets V_{1}^{s}\cup \{v^{s}\}$  Add  $v^{s}$  to set of indexed nodes   
 $s\gets s + 1$  Next index is  $s + 1$    
end while   
Output:  $\mathcal{I}(v),v\in V$  Result indexing
Remark: The definition of  $LO(\cdot)$  and Section 2.3 reuse  $A$  to denote a multiset. This will not cause ambiguity since multisets are only introduced in the theoretical part, where the adjacency matrix is not involved. For brevity, we use capital letters  $A, B, \ldots$  to denote multisets and lowercase letters  $a, b, \ldots$  to denote the corresponding sequences in  $S(M, L)$ .
Since we only need lexicographical orders for all neighbor sets of the nodes of the graph, let  $M = N$ ,  $L = N - 1$ . With the definition of  $LO(\cdot)$ , we can represent the aforementioned idea by:
$$
L O (\mathcal {N} (v) \backslash \{v + 1 \}) \leq L O (\mathcal {N} (v + 1) \backslash \{v \}), \forall v \in [ N - 1 ] \backslash \{0 \} \tag {S3}
$$
where  $\mathcal{N}(v)$  is the neighbor set of node  $v$ . Note that the possible edge between nodes  $v$  and  $v + 1$  is ignored in (S3) to include the cases that they are linked and share all neighbors with indexes less than  $v$ . In this case,  $LO(\mathcal{N}(v + 1))$  is necessarily smaller than  $LO(\mathcal{N}(v))$  since  $v \in \mathcal{N}(v + 1)$ .
Constraints (S3) exclude many ways to index the rest of the nodes, for example they reduce the possible ways to index the graph in Figure 2 from 120 to 4 ways. But, we still need to ensure that there exists at least one feasible indexing for any graph after applying constraints (S3). Algorithm 1 constructs an indexing and Section 2.3 provides a theoretical guarantee of its feasibility.
2.3 Theoretical guarantee
This section proves that Algorithm 1 always provides feasible indexing. We first give some properties that are used to derive an important lemma.
Property 1. For any  $s = 1,2,\ldots ,N - 1,\mathcal{T}^s (v)\left\{ \begin{array}{ll} <   s, & \forall v\in V_1^s\\ \geq s, & \forall v\in V_2^s. \end{array} \right.$
Figure 2: Consider a graph having 6 nodes with different features. Without any constraints, there are  $6! = 720$  ways to index it. Utilizing (S1) to force connectivity results in 636 ways. Using (S2) to choose  $v_{0}$  to be indexed 0, then there are  $5! = 120$  ways. Finally, applying (S3) to order the rest of the nodes, there are only 4 ways, 2 of which can be derived from Algorithm 1. For details on using Algorithm 1 to index this graph, see Appendix A.2.
Property 2. For any  $s_1, s_2 = 1, 2, \ldots, N - 1$ ,
$$
s _ {1} \leq s _ {2} \Rightarrow \mathcal {N} ^ {s _ {1}} (v) = \mathcal {N} ^ {s _ {2}} (v) \cap [ s _ {1} ], \forall v \in V _ {2} ^ {s _ {2}}
$$
Property 3. Given any multisets  $A, B$  with no more than  $L$  integer elements in  $[0, M - 1]$ , we have:
$$
L O (A) \leq L O (B) \Rightarrow L O (A \cap [ m ]) \leq L O (B \cap [ m ]), \forall m = 1, 2, \dots , M
$$
Using these properties, we can prove Lemma 1, which shows that if the final index of node  $u$  is smaller than  $v$ , then at each iteration, the temporary index assigned to  $u$  is not greater than  $v$ .
Lemma 1. For any two nodes  $u$  and  $v$ ,
$$
\mathcal {I} (u) <   \mathcal {I} (v) \Rightarrow \mathcal {I} ^ {s} (u) \leq \mathcal {I} ^ {s} (v), \forall s = 1, 2, \dots , N - 1
$$
Now we can prove Theorem 1.
Theorem 1. Given any undirected graph  $G = (V, E)$  with one node indexed 0. The indexing yielded from Algorithm 1 satisfies (S3).
Proof. Assume that constraints (S3) are not satisfied and the minimal index that violates (S3) is  $s$ . Denote nodes with index  $s$  and  $s + 1$  by  $u, v$  respectively (i.e.,  $\mathcal{I}(u) = s, \mathcal{I}(v) = s + 1$ ).
Let  $\mathcal{N}(u)\backslash \{v\} \coloneqq \{u_1,u_2,\ldots ,u_m\}$  be all neighbors of  $u$  except for  $v$ , where:
$$
\mathcal {I} (u _ {i}) <   \mathcal {I} (u _ {i + 1}), \forall i = 1, 2, \dots , m - 1
$$
Similarly, let  $\mathcal{N}(v) \backslash \{u\} \coloneqq \{v_1, v_2, \ldots, v_n\}$  be all neighbors of  $v$  except for  $u$ , where:
$$
\mathcal {I} (v _ {j}) <   \mathcal {I} (v _ {j + 1}), \forall j = 1, 2, \dots , n - 1
$$
Denote the sequences in  $S(N, N - 1)$  corresponding to sets  $\{\mathcal{I}(u_i) \mid 1 \leq i \leq m\}$  and  $\{\mathcal{I}(v_j) \mid 1 \leq j \leq n\}$  by  $a = (a_1, a_2, \ldots, a_{N-1}), b = (b_1, b_2, \ldots, b_{N-1})$ . By definition of  $LO(\cdot)$ :
$$
a _ {i} = \left\{ \begin{array}{l l} \mathcal {I} (u _ {i}), & 1 \leq i \leq m \\ N, & m <   i <   N \end{array} \right., b _ {j} = \left\{ \begin{array}{l l} \mathcal {I} (v _ {j}), & 1 \leq j \leq n \\ N, & n <   j <   N \end{array} \right.
$$
Since nodes  $u$  and  $v$  violate constraints (S3), there exists a position  $1 \leq k \leq N - 1$  satisfying:
$$
\left\{ \begin{array}{l} a _ {i} = b _ {i}, \quad \forall 1 \leq i <   k \\ a _ {k} > b _ {k} \end{array} \right.
$$
from where we know that nodes  $u$  and  $v$  share the first  $k - 1$  neighbors (i.e.  $u_{i} = v_{i}$ ,  $\forall 1 \leq i < k$ ). From  $b_{k} < a_{k} \leq N$  we know that node  $v$  definitely has its  $k$ -th neighbor node  $v_{k}$ . Also, note that  $v_{k}$  is not a neighbor of node  $u$ . Otherwise, we have  $u_{k} = v_{k}$  and then  $a_{k} = \mathcal{I}(u_{k}) = \mathcal{I}(v_{k}) = b_{k}$ .
Case 1: If  $a_{k} = N$ , that is, node  $u$  has  $k - 1$  neighbors.
In this case, node  $v$  has all neighbors of node  $u$  as well as node  $v_{k}$ . Therefore, we have:
$$
L O \left(\mathcal {N} _ {t} ^ {s} (u)\right) > L O \left(\mathcal {N} _ {t} ^ {s} (v)\right)
$$
which violates the fact that node  $u$  is chosen to be indexed  $s$  at  $s$ -th iteration of Algorithm 1.
Case 2: If  $a_{k} < N$ , that is, node  $u$  has nodes  $u_{k}$  as its  $k$ -th neighbor.
Since  $\mathcal{I}(u_k) = a_k > b_k = \mathcal{I}(v_k)$ , we can apply Lemma 1 on node  $u_k$  and  $v_k$  at  $(s + 1)$ -th iteration, to obtain
$$
\mathcal {I} ^ {s + 1} \left(u _ {k}\right) \geq \mathcal {I} ^ {s + 1} \left(v _ {k}\right) \quad (\geq)
$$
Similarly, if we apply Lemma 1 to all the neighbors of node  $u$  and node  $v$  at  $s$ -th iteration, we have:
$$
\mathcal {I} ^ {s} (u _ {i}) \leq \mathcal {I} ^ {s} (u _ {i + 1}), \forall i = 1, 2, \dots , m - 1
$$
$$
\mathcal {I} ^ {s} (v _ {j}) \leq \mathcal {I} ^ {s} (v _ {j + 1}), \forall j = 1, 2, \dots , n - 1
$$
Given that  $a_{k} = \mathcal{I}(u_{k})$  is the  $k$ -th smallest number in  $a$ , we conclude that  $\mathcal{I}^s(u_k)$  is equal to the  $k$ -th smallest number in  $\mathcal{N}_t^s(u)$ . Likewise,  $\mathcal{I}^s(v_k)$  equals to the  $k$ -th smallest number in  $\mathcal{N}_t^s(v)$ . Meanwhile,  $\mathcal{I}^s(u_i) = \mathcal{I}^s(v_i)$  since  $u_i = v_i$ ,  $\forall 1 \leq i < k$ . After comparing the lexicographical orders between of  $\mathcal{N}_t^s(u)$  and  $\mathcal{N}_t^s(v)$  (with the same  $k - 1$  smallest elements,  $\mathcal{I}^s(u_k)$  and  $\mathcal{I}^s(v_k)$  as the  $k$ -th smallest element, respectively), node  $u$  is chosen. Therefore, we have:
$$
\mathcal {I} ^ {s} (u _ {k}) \leq \mathcal {I} ^ {s} (v _ {k})
$$
from which we know that:
$$
L O \left(\mathcal {N} ^ {s} \left(u _ {k}\right)\right) \leq L O \left(\mathcal {N} ^ {s} \left(v _ {k}\right)\right)
$$
At  $(s + 1)$ -th iteration, node  $u_{k}$  has one more indexed neighbor (i.e., node  $u$  with index  $s$ ), while node  $v_{k}$  has no new indexed neighbor. Thus we have:
$$
L O (\mathcal {N} ^ {s + 1} (u _ {k})) = L O (\mathcal {N} ^ {s} (u _ {k}) \cup \{s \}) <   L O (\mathcal {N} ^ {s} (u _ {k})) \leq L O (\mathcal {N} ^ {s} (v _ {k})) = L O (\mathcal {N} ^ {s + 1} (v _ {k}))
$$
which yields:
$$
\mathcal {I} ^ {s + 1} \left(u _ {k}\right) <   \mathcal {I} ^ {s + 1} \left(v _ {k}\right) \tag {< }
$$
The contradiction between  $(\geq)$  and  $(<)$  completes this proof.
Given any undirected graph with node features, after using (S2) to choose node 0, Theorem 1 guarantees that there exists at least one indexing satisfying (S3). However, when applying (S1) to force a connected graph, we need to show the compatibility between (S1) and (S3), as shown in Lemma 2. Appendix A provides proofs of properties and lemmas.
Lemma 2. For any undirected, connected graph  $G = (V, E)$ , if one indexing of  $G$  sastifts (S3), then it satisfies (S1).
Note that (S1) - (S3) are not limited to optimizing over trained GNNs. In fact, they can be employed in generic graph-based search problems, as long as there exists a symmetry issue caused by graph isomorphism. GNNs can be generalized to all permutation invariant functions defined over graphs. Although the next sections apply MIP formulations to demonstrate the symmetry-breaking constraints, we could alternatively use a constraint programming [80] or satisfiability [81] paradigm. For example, we could have encoded the Elgabou and Frisch [82] constraints in Reluplex [83].
2.4 Connection & Difference to the symmetry-breaking literature
Typically, MIP solvers detect symmetry using graph automorphism, for example SCIP [84] uses BLISS [85, 86], and both Gurobi [87] and SCIP break symmetry using orbital fixing [88] and orbital pruning [89]. When a MIP solver detects symmetry, the only graph available is the graph formed by the variables and constraints in the MIP.
Our symmetries come from alternative indexing of abstract nodes. Each indexing results in an isomorphic graph. In MIP, however, each indexing corresponds to an element of a much larger symmetry group defined on all variables, including node features  $(O(NF))$ , adjacency matrix  $(O(N^2))$ , model (e.g., a GNN), and problem-specific features. For instance, in the first row of Table 6, the input graph has  $N = 4$  nodes, but the MIP has  $616 + 411$  variables. We only need to consider a permutation group with  $N! = 24$  elements. However, because the MIP solver does not have access to the input graph structure, it needs to consider all possible automorphic graphs with  $616 + 411$  nodes. By adding constraints (S1) - (S3), there is no need to consider the symmetry group of all variables to find a much smaller subgroup corresponding to the permutation group defined on abstract nodes.
The closest setting to ours is distributing  $m$  different jobs to  $n$  identical machines and then minimizing the total cost. Binary variable  $A_{i,j}$  denotes if job  $i$  is assigned to machine  $j$ . The requirement is that each job can only be assigned to one machine (but each machine can be assigned to multiple jobs). Symmetries come from all permutations of machines. This setting appears in noise dosage problems [90, 91], packing and partitioning orbitopes [92, 93], and scheduling problems [94]. However, requiring that the sum of each row in  $A_{i,j}$  equals to 1 simplifies the problem. By forcing decreasing lexicographical orders for all columns, the symmetry issue is handled well. Constraints (S3) can be regarded as a non-trivial generalization of these constraints from a bipartite graph to an arbitrary undirected graph: following Algorithm 1 will produce the same indexing documented in [90-94].
2.5 Mixed-integer formulations for optimizing over trained GNNs
As mentioned before, there are many variants of GNNs [4-18] with different theoretical and practical motivations. Although it is possible to build a MIP for a specific GNN from the scratch (e.g., [71]), a general definition that supports multiple architectures is preferable.
2.5.1 Definition of GNNs
We define a GNN with  $L$  layers as follows:
$$
G N N: \underbrace {\mathbb {R} ^ {d _ {0}} \otimes \cdots \otimes \mathbb {R} ^ {d _ {0}}} _ {| V | \text {t i m e s}} \to \underbrace {\mathbb {R} ^ {d _ {L}} \otimes \cdots \otimes \mathbb {R} ^ {d _ {L}}} _ {| V | \text {t i m e s}}
$$
where  $V$  is the set of nodes of the input graph.
Let  $\pmb{x}_v^{(0)} \in \mathbb{R}^{d_0}$  be the input features for node  $v$ . Then, the  $l$ -th layer  $(l = 1,2,\dots ,L)$  is defined by: