SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PLANE: Representation Learning over Planar Graphs
Radoslav Dimitrov*†
contact@radoslav11.com
Ralph Abboud*
ralph@ralphabb.ai
Zeyang Zhao*†
zeyzang.zhao@cs.ox.ac.uk
Graph neural networks iteratively compute representations of nodes of an input graph through a series of transformations in such a way that the learned graph function is isomorphism invariant on graphs, which makes the learned representations graph invariants. On the other hand, it is well-known that graph invariants learned by these class of models are incomplete: there are pairs of non-isomorphic graphs which cannot be distinguished by standard graph neural networks. This is unsurprising given the computational difficulty of graph isomorphism testing on general graphs, but the situation begs to differ for special graph classes, for which efficient graph isomorphism testing algorithms are known, such as planar graphs. The goal of this work is to design architectures for efficiently learning complete invariants of planar graphs. Inspired by the classical planar graph isomorphism algorithm of Hopcroft and Tarjan, we propose PLANE as a framework for planar representation learning. PLANE includes architectures which can learn complete invariants over planar graphs while remaining practically scalable. We validate the strong performance of PLANE architectures on various planar graph benchmarks.
Graphs are used for representing relational data in a wide range of domains, including physical [53], chemical [17, 36], and biological [21, 68] systems, which led to increasing interest in machine learning (ML) over graphs. Graph neural networks (GNNs) [23, 51] have become prominent for graph ML for a wide range of tasks, owing to their capacity to explicitly encode desirable relational inductive biases [6]. GNNs iteratively compute representations of nodes of an input graph through a series of transformations in such a way that the learned graph-level function represents a graph invariant: a property of graphs which is preserved under all isomorphic transformations.
Learning functions on graphs is challenging for various reasons, particularly since the learning problem contains the infamous graph isomorphism problem, for which the best known algorithm, given in a breakthrough result by Babai [4], runs in quasi-polynomial time. A large class of GNNs can thus only learn incomplete graph invariants for scalability purposes. In fact, standard GNNs are known to be at most as expressive as the 1-dimensional Weisfeiler-Leman algorithm (1-WL)[59] in terms of distinguishing power [45, 63]. There are simple non-isomorphic pairs of graphs, such as the pair shown in Figure 1, which cannot be distinguished by 1-WL and by a large class of GNNs. This
*This work is largely conducted while these authors were still affiliated with the University of Oxford.
limitation motivated a large body of work aiming to explain and overcome the expressiveness barrier of these architectures [1, 5, 7, 9, 11, 15, 37, 42, 43, 45, 50].
The expressiveness limitations of standard GNNs already apply on planar graphs, since, e.g., the graphs  $G_{1}$  and  $G_{2}$  from Figure 1 are planar. There are, however, efficient and complete graph isomorphism testing algorithms for planar graphs [28, 29, 41], which motivates an aligned design of dedicated architectures with better properties over planar graphs. Building on this idea, we propose architectures for efficiently learning complete invariants over planar graphs.
Figure 1: Two graphs indistinguishable by 1-WL.
Figure 2: Acid anhydride as a planar graph.
Planar structures appear, e.g., in road networks [62], circuit design [8], and most importantly, in biochemistry [54]. Molecules are commonly encoded as planar graphs with node and edge types, as shown in Figure 2, which enables the application of many graph ML architectures on biochemistry tasks. For example, all molecule datasets in OGBG [33] consist of planar graphs. Biochemistry has been a very important application domain for both classical graph isomorphism testing $^{3}$  and for graph ML. By designing architectures for learning complete invariants over planar graphs, our work unlocks the full potential of both these domains.
- Building on the classical literature, we introduce PLANE as a framework for learning isomorphism-complete invariant functions over planar graphs and derive the BASEPLANE architecture (Section 5). We prove that BASEPLANE is a scalable, complete learning algorithm on planar graphs: BASEPLANE can distinguish any pair of non-isomorphic planar graphs (Section 6).
- We conduct an empirical analysis for BASEPLANE evaluating its expressive power on three tasks (Section 7.1), and validating its performance on real-world graph classification (Section 7.2) and regression benchmarks (Section 7.3 and 7.4). The empirical results support the presented theory, and also yield multiple state-of-the-art results for BASEPLANE on molecular datasets.
The proofs and additional experimental details are delegated to the appendix of this paper.
2 A primer on graphs, invariants, and graph neural networks
Graphs. Consider simple, undirected graphs  $G = (V, E, \zeta)$ , where  $V$  is a set of nodes,  $E \subseteq V \times V$  is a set of edges, and  $\zeta : V \to \mathbb{C}$  is a (coloring) function. If the range of this map is  $\mathbb{C} = \mathbb{R}^d$ , we refer to it as a  $d$ -dimensional feature map. A graph is connected if there is a path between any two nodes and disconnected otherwise. A graph is biconnected if it cannot become disconnected by removing any single node. A graph is triconnected if the graph cannot become disconnected by removing any two nodes. A graph is planar if it can be drawn on a plane such that no two edges intersect.
Components. A biconnected component of a graph  $G$  is a maximal biconnected subgraph. Any connected graph  $G$  can be efficiently decomposed into a tree of biconnected components called the Block-Cut tree of  $G$  [30], which we denote as BLOCKCUT( $G$ ). The blocks are attached to each other at shared nodes called cut nodes. A triconnected component of a graph  $G$  is a maximal triconnected subgraph. Triconnected components of a graph  $G$  can also be compiled (very efficiently [27]) into a tree, known as the SPQR tree [16], which we denote as SPQR( $G$ ). Given a graph  $G$ , we denote by  $\sigma^G$  the set of all SPQR components of  $G$  (i.e., nodes of SPQR( $G$ )), and by  $\pi^G$  the set of all biconnected components of  $G$ . Moreover, we denote by  $\sigma_u^G$  the set of all SPQR components of  $G$  where  $u$  appears as a node, and by  $\pi_u^G$  the set of all biconnected components of  $G$  where  $u$  appears as a node.
Labeled trees. We sometimes refer to rooted, undirected, labeled trees  $\Gamma = (V,E,\zeta)$ , where the canonical root node is given as one of tree's centroids: a node with the property that none of its branches contains more than half of the other nodes. We denote by  $\mathrm{ROOT}(\Gamma)$  the canonical root of  $\Gamma$ , and define the depth  $d_u$  of a node  $u$  in the tree as the node's minimal distance from the canonical root. The children of a node  $u$  is the set  $\chi(u) = \{v|(u,v) \in E, d_v = d_u + 1\}$ . The descendants of a node
<sup>3</sup>Graph isomorphism first appears in the chemical documentation literature [49], as the problem of matching a molecular graph against a database of such graphs; see, e.g., Grohe and Schweitzer [26] for a recent survey.
$u$  is given by set of all nodes reachable from  $u$  through a path of length  $k \geq 0$  such that the node at position  $j + 1$  has one more depth than the node at position  $j$ , for every  $0 \leq j \leq k$ . Given a rooted tree  $\Gamma$  and a node  $u$ , the subtree of  $\Gamma$  rooted at node  $u$  is the tree induced by the descendants of  $u$ , which we denote by  $\Gamma_u$ . For technical convenience, we allow the induced subtree  $\Gamma_u$  of a node  $u$  even if  $u$  does not appear in the tree  $\Gamma$ , in which case  $\Gamma_u$  is the empty tree.
Node and graph invariants. An isomorphism from a graph  $G = (V, E, \zeta)$  to a graph  $G' = (V', E', \zeta')$  is a bijection  $f: V \to V'$  such that  $\zeta(u) = \zeta'(f(u))$  for all  $u \in V$ , and  $(u, v) \in E$  if and only if  $(f(u), f(v)) \in E'$ , for all  $u, v \in V$ . A node invariant is a function  $\xi$  that associates with each graph  $G = (V, E, \zeta)$  a function  $\xi(G)$  defined on  $V$  such that for all graphs  $G$  and  $G'$ , all isomorphisms  $f$  from  $G$  to  $G'$ , and all nodes  $u \in V$ , it holds that  $\xi(G)(u) = \xi(G')(f(u))$ . A graph invariant is a function  $\xi$  defined on graphs such that  $\xi(G) = \xi(G')$  for all isomorphic graphs  $G$  and  $G'$ . We can derive a graph invariant  $\xi$  from a node invariant  $\xi'$  by mapping each graph  $G$  to the multiset  $\{\{\xi'(G)(u) | u \in V\}\}$ . We say that a graph invariant  $\xi$  distinguishes two graphs  $G$  and  $G'$  if  $\xi(G) \neq \xi(G')$ . If a graph invariant  $\xi$  distinguishes  $G$  and  $G'$  then there is no isomorphism from  $G$  to  $G'$ . If the converse also holds, then  $\xi$  is a complete graph invariant. We can speak of (in)completeness of invariants on special classes of graphs, e.g., 1-WL computes an incomplete invariant on general graphs, but it is well-known to compute a complete invariant on trees [24].
Message passing neural networks. A vast majority of GNNs are instances of message passing neural networks (MPNNs) [22]. Given an input graph  $G = (V, E, \zeta)$ , an MPNN sets, for each node  $u \in V$ , an initial node representation  $\zeta(u) = h_u^{(0)}$ , and iteratively computes representations  $h_u^{(\ell)}$  for a fixed number of layers  $0 \leq \ell \leq L$  as:
$$
\left. \boldsymbol {h} _ {u} ^ {(\ell + 1)} := \phi \left(\boldsymbol {h} _ {u} ^ {(\ell)}, \psi \left(\boldsymbol {h} _ {u} ^ {(\ell)}, \{\{\boldsymbol {h} _ {v} ^ {(\ell)} | v \in N _ {u} \}\}\right)\right), \right.
$$
where  $\phi$  and  $\psi$  are respectively update and aggregation functions, and  $N_{u}$  is the neighborhood of  $u$ . Node representations can be pooled to obtain graph-level embeddings by, e.g., summing all node embeddings. We denote by  $z^{(L)}$  the resulting graph-level embeddings. In this case, an MPNN can be viewed as an encoder that maps each graph  $G$  to a representation  $z_{G}^{(L)}$ , computing a graph invariant.
The expressive power of MPNNs is upper bounded by 1-WL [45, 63] in terms of distinguishing graphs, and by the logic  $C^2$  in terms of capturing functions over graphs [5], motivating a body of work to improve on these bounds. One notable direction has been to enrich node features with unique node identifiers [42, 65], random discrete colors [15], or random noisy dimensions [1, 50]. Another line of work proposes higher-order architectures [37, 43-45] based on higher-order tensors [44], or a higher-order form of message passing [45], which typically align with a  $k$ -dimensional WL test $^4$ , for some  $k > 1$ . Higher-order architectures are not scalable, and most existing models are upper bounded by 2-WL (or, oblivious 3-WL). Another body of work is based on sub-graph sampling [7, 9, 11, 56], with pre-set sub-graphs used within model computations. These approaches can yield substantial expressiveness improvements, but they rely on manual sub-graph selection, and require running expensive pre-computations. Finally, MPNNs have been extended to incorporate other graph kernels, i.e., shortest paths [2, 64], random walks [46, 47] and nested color refinement [67].
The bottleneck limiting the expressiveness of MPNNs is the implicit need to perform graph isomorphism checking, which is challenging in the general case. However, there are classes of graphs, such as planar graphs, with efficient and complete isomorphism algorithms, thus eliminating this bottleneck. For planar graphs, the first complete algorithm for isomorphism testing was presented by Hopcroft and Tarjan [29], and was followed up by a series of algorithms [10, 19, 28, 52]. Kukluk et al. [41] presented an algorithm, which we refer to as KHC, that is more suitable for practical applications, and that we align with in our work. As a result of this alignment, our approach is the first efficient and complete learning algorithm on planar graphs. Observe that 3-WL (or, oblivious 4-WL) is also known to be complete on planar graphs [38], but this algorithm is far from being scalable [35] and as a result there is no neural, learnable version implemented. By contrast, our architecture learns representations of efficiently computed components and uses these to obtain refined representations. Our approach extends the inductive biases of MPNNs based on structures, such as biconnected components, which are recently noted to be beneficial in the literature [66].
<sup>4</sup>We write  $k$ -WL to refer to the folklore (more standard) version of the algorithm following Grohe [25].
4 A practical planar isomorphism algorithm
The idea behind the KHC algorithm is to compute a canonical code for planar graphs, allowing us to reduce the problem of isomorphism testing to checking whether the codes of the respective graphs are equal. Importantly, we do not use the codes generated by KHC in our model, and view codes as an abstraction through which alignment with KHC is later proven. Formally, we can define a code as a string over the alphabet  $\Sigma \cup \mathbb{N}$ : for each graph, the KHC algorithm computes codes for various components resulting from decompositions, and gradually builds a code representation for the graph. For readability, we allow reserved symbols  $''$ ,  $''$  and  $''$ , in the generated codes. We present an overview of KHC and refer to Kukluk et al. [41] for details. We can assume that the planar graphs are connected, as the algorithm can be extended to disconnected graphs by independently computing the codes for each of the components, and then concatenating them in their lexicographical order.
Generating a code for the graph. Given a connected planar graph  $G = (V, E, \zeta)$ , KHC decomposes  $G$  into a Block-Cut tree  $\delta = \mathrm{BLOCKCUT}(G)$ . Every node  $u \in \delta$  is either a cut node of  $G$ , or a virtual node associated with a biconnected component of  $G$ . KHC iteratively removes the leaf nodes of  $\delta$  as follows: if  $u$  is a leaf node associated with a biconnected component  $B$ , KHC uses a subprocedure BICODE to compute the canonical code  $\mathrm{CODE}(\delta_u) = \mathrm{BICODE}(B)$  and removes  $u$  from the tree; otherwise, if the leaf node  $u$  is a cut node, KHC overrides the initial code  $\mathrm{CODE}((\{u\}, \{\})$ ), using an aggregate code of the removed biconnected components in which  $u$  occurs as a node. This procedure continues until there is a single node left, and the code for this remaining node is taken as the code of the entire connected graph. This process yields a complete graph invariant. Conceptually, it is more convenient for our purposes to reformulate this procedure as follows: we first canonically root  $\delta$ , and then code the subtrees in a bottom-up procedure. Specifically, we iteratively generate codes for subtrees and the final code for  $G$  is then simply the code of  $\delta_{\mathrm{ROOT}(\delta)}$ .
Generating a code for biconnected components. KHC relies on a subprocedure BICODE to compute a code, given a biconnected planar graph  $B$ . Specifically, it uses the SPQR tree  $\gamma = \mathrm{SPQR}(B)$  which uniquely decomposes a biconnected component into a tree with virtual nodes of one of four types:  $S$  for cycle graphs,  $P$  for two-node dipole graphs,  $Q$  for a graph that has a single edge, and finally  $R$  for a triconnected component that is not a dipole or a cycle. We first generate codes for the induced sub-graphs based on these virtual nodes. Then the SPQR tree is canonically rooted, and similarly to the procedure on the Block-Cut tree, we iteratively build codes for the subtrees of  $\gamma$  in a bottom-up fashion. Due to the simpler structure of the SPQR tree, instead of making overrides, the recursive code generation is done by prepending a number  $\theta(C, C')$  for a parent SPQR tree node  $C$  and each  $C' \in \chi(C)$ . This number is generated based on the way  $C$  and  $C'$  are connected in  $B$ . Generating codes for the virtual  $P$  and  $Q$  nodes is trivial. For  $S$  nodes, we use the lexicographically smallest ordering of the cycle, and concatenate the individual node codes. However, for  $R$  nodes we require a more complex procedure and this is done using Weinberg's algorithm as a subroutine.
Generating a code for triconnected components. Whitney [60] has shown that triconnected graphs have only two planar embeddings, and Weinberg introduced an algorithm that computes a canonical code for triconnected planar graphs [58] which we call TRICODE. This code is used as one of the building blocks of the KHC algorithm and can be extended to labeled triconnected planar graphs, which is essential for our purposes. Weinberg's algorithm [58] generates a canonical order for a triconnected component  $T$ , by traversing all the edges in both directions via a walk. Let  $\omega$  be the sequence of visited nodes in this particular walk and write  $\omega[i]$  to denote  $i$ -th node in it. This walk is then used to generate a sequence  $\kappa$  of same length, that corresponds to the order in which we first visit the nodes: for each node  $u = \omega[i]$  that occurs in the walk, we set  $\kappa[i] = 1 + |\{\kappa[j] \mid j < i\}|$  if  $\omega[i]$  is the first occurrence of  $u$ , or  $\kappa[i] = \kappa[\min\{j \mid \omega[i] = \omega[j]\}]$  otherwise. For example, the walk  $\omega = \langle v_1, v_3, v_2, v_3, v_1 \rangle$  yields  $\kappa = \langle 1, 2, 3, 2, 1 \rangle$ . Given such a walk of length  $k$  and a corresponding sequence  $\kappa$ , we compute the following canonical code: TRicode( $T = (\kappa[1], \text{CODE}((\{\omega[1]\}, \{\}))), \ldots, (\kappa[k], \text{CODE}((\{\omega[k]\}, \{\}))$ ).
5 PLANE: Representation learning over planar graphs
KHC generates a unique code for every planar graph in a hierarchical manner based on the decompositions. Our framework aligns with this algorithm: given an input graph  $G = (V, E, \zeta)$ , PLANE first computes the BLOCKCUT and SPQR trees (A - C), and then learns representations for the nodes, the components, and the whole graph (D - G), as illustrated in Figure 3.
A. Input Graph
B. Block-Cut Tree
C. SPQR Trees
Figure 3: Given an input graph (A) PlanE computes the BLOCKCUT (B) and SPQR (C) trees and assigns initial embeddings to SPQR components using TRIENC (D). Then, BIENC iteratively traverses each SPQR tree bottom-up (numbered), updates SPQR embeddings, and computes biconnected component embeddings (E). CUTENC then analogously traverses the BLOCKCUT tree to compute a graph embedding (F). Finally, the node representation is updated (G).
Target Node
$\odot$  Neighbor Node
Cut Node
G. Update
F. CutEnc
E. BiEnc
D. TriEnc
Biconnected Component
SPQR Component
Formally, PLANE sets the initial node representations  $\pmb{h}_u^{(0)} = \zeta(u)$  for each node  $u \in V$ , and computes, for every layer  $1 \leq \ell \leq L$ , the representations  $\widehat{\pmb{h}}_C^{(\ell)}$  of SPQR components of  $\sigma^G$ , the representations  $\widetilde{\pmb{h}}_B^{(\ell)}$  of biconnected components  $B \in \pi^G$ , the representations  $\bar{\pmb{h}}_{\delta_u}^{(\ell)}$  of subtrees in the Block-Cut tree  $\delta = \text{BLOCKCUT}(G)$  for each node  $u \in V$ , and the representations  $\pmb{h}_u^{(\ell)}$  of nodes as:
$$
\widehat {\boldsymbol {h}} _ {C} ^ {(\ell)} = \operatorname {T R I E N C} \left(C, \left\{\left(\boldsymbol {h} _ {u} ^ {(\ell - 1)}, u\right) \mid u \in C \right\}\right)
$$
$$
\widetilde {\boldsymbol {h}} _ {B} ^ {(\ell)} = \operatorname {B I E N C} \left(B, \left\{\left(\widehat {\boldsymbol {h}} _ {C} ^ {(\ell)}, C\right) \mid C \in \sigma^ {B} \right\}\right)
$$
$$
\bar {h} _ {\delta_ {u}} ^ {(\ell)} = \operatorname {C U T E N C} \left(\delta_ {u}, \left\{\left(h _ {v} ^ {(\ell - 1)}, v\right) \mid v \in V \right\}, \left\{\left(\tilde {h} _ {B} ^ {(\ell)}, B\right) \mid B \in \pi^ {G} \right\}\right)
$$
$$
\boldsymbol {h} _ {u} ^ {(\ell)} = \phi \big (\boldsymbol {h} _ {u} ^ {(\ell - 1)}, \{\{\boldsymbol {h} _ {v} ^ {(\ell - 1)} | v \in N _ {u} \}, \{\{\boldsymbol {h} _ {v} ^ {(\ell - 1)} | v \in V \}, \{\{\widehat {\boldsymbol {h}} _ {T} ^ {(\ell)} | T \in \sigma_ {u} ^ {G} \}, \{\{\widetilde {\boldsymbol {h}} _ {B} ^ {(\ell)} | B \in \pi_ {u} ^ {G} \}, \overline {{\boldsymbol {h}}} _ {\delta_ {u}} ^ {(\ell)} \}
$$
where TRIENC, BIENC, and CUTENC are invariant encoders and  $\phi$  is an UPDATE function. The encoders TRIENC and BIENC are analogous to TRICODE and BICODE of the KHC algorithm. In PLANE, we further simplify the final graph code generation, by learning embeddings for the cut nodes, which is implemented by the CUTENC encoder. For graph-level tasks, we apply a pooling function on final node embeddings to obtain a graph-level embedding  $z_{G}^{(L)}$ .
There are many choices for deriving PLANE architectures, but we propose a simple model, BASE-PLANE, to clearly identify the virtue of the model architecture which aligns with KHC, as follows:
TRIENC. Given an SPQR component  $C$  and the node representations  $\pmb{h}_u^{(\ell -1)}$  of each node  $u$ , TRIENC encodes  $C$  based on the walk  $\omega$  given by Weinberg's algorithm, and its corresponding sequence  $\kappa$  as:
$$
\widehat {\boldsymbol {h}} _ {C} ^ {(\ell)} = \operatorname {M L P} \left(\sum_ {i = 1} ^ {| \omega |} \operatorname {M L P} \left(\boldsymbol {h} _ {\omega [ i ]} ^ {(\ell - 1)} \| \boldsymbol {p} _ {\kappa [ i ]} \| \boldsymbol {p} _ {i}\right)\right),
$$
where  $\pmb{p}_x\in \mathbb{R}^d$  is the positional embedding [57]. This is a simple sequence model with a positional encoding on the walk, and a second one based on the generated sequence  $\kappa$ . Edge features can also be
concatenated while respecting the edge order given by the walk. The nodes of  $\mathrm{SPQR}(G)$  are one of the types  $S, P, Q, R$ , where for  $S, P, Q$  types, we have a trivial ordering for the induced components, and Weinberg's algorithm also gives an ordering for  $R$  nodes that correspond to triconnected components.
BIENC. Given a biconnected component  $B$  and the representations  $\pmb{h}_C^{(\ell)}$  of each component induced by a node  $C$  in  $\gamma = \mathrm{SPQR}(B)$ , BIENC uses the SPQR tree and the integers  $\theta(C, C')$  corresponding to how we connect  $C$  and  $C' \in \chi(C)$ . BIENC then computes a representation for each subtree  $\gamma_C$  induced by a node  $C$  in a bottom up fashion as:
$$
\widetilde {\boldsymbol {h}} _ {\gamma_ {C}} ^ {(\ell)} = \operatorname {M L P} \left(\widehat {\boldsymbol {h}} _ {C} ^ {(\ell)} + \sum_ {C ^ {\prime} \in \chi (C)} \operatorname {M L P} \left(\widetilde {\boldsymbol {h}} _ {\gamma_ {C ^ {\prime}}} ^ {(\ell)} \| \boldsymbol {p} _ {\theta (C, C ^ {\prime})}\right)\right).
$$
This encoder operates in a bottom up fashion to ensure that a subtree representation of the children of  $C$  exists before it encodes the subtree  $\gamma_{C}$ . The representation of the canonical root node in  $\gamma$  is used as the representation of the biconnected component  $B$  by setting:  $\tilde{\pmb{h}}_B^{(\ell)} = \tilde{\pmb{h}}_{\gamma_{\mathrm{ROOT}}(\gamma)}^{(\ell)}$ .
CUTENC. Given a subtree  $\delta_u$  of a Block-Cut tree  $\delta$ , the representations  $\widetilde{\pmb{h}}_B^{(\ell)}$  of each biconnected component  $B$ , and the node representations  $\pmb{h}_u^{(\ell-1)}$  of each node, CUTENC sets  $\bar{\pmb{h}}_{\delta_u}^{(\ell)} = \pmb{0}^{d(\ell)}$  if  $u$  is not a cut node; otherwise, it computes the subtree representations as:
$$
\bar {\boldsymbol {h}} _ {\delta_ {u}} ^ {(\ell)} = \operatorname {M L P} \left(\boldsymbol {h} _ {u} ^ {(\ell - 1)} + \sum_ {B \in \chi (u)} \operatorname {M L P} \left(\widetilde {\boldsymbol {h}} _ {B} ^ {(\ell)} + \sum_ {v \in \chi (B)} \bar {\boldsymbol {h}} _ {\delta_ {v}} ^ {(\ell)}\right)\right).
$$