SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Generate an Erdos-Renyi graph  $G_{\mathrm{init}}$  with 5 nodes and a random probability  $p$ . We randomly select one of the nodes as the source node  $u$ .
2.  $G_{1}$  is constructed by disjoint union  $G_{\mathrm{init}}$  with two triangles, one with edges relation of  $r_1$ , and the other with  $r_2$ . The target query is  $r_3(u,v)$  for all  $v$  in a triangle with an edge relation of  $r_1$ .
3. Similarly,  $G_{2}$  is constructed by disjoint union another copy of  $G_{\mathrm{init}}$  with two squares, one with edges relation of  $r_1$ , and the other with  $r_2$ . The target query is  $r_3(u,v)$  for all  $v$  in a square with an edge relation of  $r_2$ .
One example of such a pair can be shown in Figure 6. We generate 100 graph pairs and assign 70 pairs as the training set and the remaining 30 pairs as the testing set (140 training graphs and 60 testing graphs). In total, there are 490 training triplets and 210 testing triplets.
Objective. For all node  $v \notin G_{\mathrm{init}}$ , we want a higher score for all the links  $r_0(u, v)$  if either  $v$  is in a triangle consists of  $r_1$  relation or  $v$  is in a square consists of  $r_2$  relation, and a lower score otherwise. For negative sampling, we choose counterpart triplets for each graph, that is, we take  $r_0(u, v)$  for all  $v$  in a triangle with an edge relation of  $r_2$  in  $G_1$  and in a square with an edge relation of  $r_1$  in  $G_2$ .
Model architectures. We have considered two model architectures, namely C-MPNNs with  $\mathrm{INIT} = \mathrm{INIT}^2$ ,  $\mathrm{AGG} = \mathrm{sum}$ ,  $\mathrm{Msg} = \mathrm{Msg}_r^2$ , and  $f(t) = t$  without sum global readout:
$$
\begin{array}{l} \boldsymbol {h} _ {v | u, q} ^ {(0)} = \mathbb {1} _ {u = v} * \mathbf {z} _ {q}, \\ \boldsymbol {h} _ {v | u, q} ^ {(t + 1)} = \sigma \Big (\mathbf {W} _ {0} ^ {(t)} \big (\mathbf {h} _ {v | u, q} ^ {(t)} + \sum_ {r \in R} \sum_ {w \in \mathcal {N} _ {r} (v)} \mathrm {M S G} _ {r} ^ {2} (\mathbf {h} _ {w | u, q} ^ {(t)}, \mathbf {z} _ {q}) \big) \Big), \\ \end{array}
$$
Figure 6: Construction of graph pair in TRI-SQR for global readout
and the same C-MPNN architecture with sum global readout:
$$
\boldsymbol {h} _ {v | u, q} ^ {(0)} = \mathbb {1} _ {u = v} * \mathbf {z} _ {q},
$$
$$
\boldsymbol {h} _ {v | u, q} ^ {(t + 1)} = \sigma \Big (\mathbf {W} _ {0} ^ {(t)} \big (\mathbf {h} _ {v | u, q} ^ {(t)} + \sum_ {r \in R} \sum_ {w \in \mathcal {N} _ {r} (v)} \mathrm {M S G} _ {r} ^ {2} (\mathbf {h} _ {w | u, q} ^ {(t)}, \mathbf {z} _ {q}) \big) + \mathbf {W} _ {1} ^ {(t)} \sum_ {w \in V} \boldsymbol {h} _ {w | u, q} ^ {(t)} \Big),
$$
Design. We claim that C-MPNNs with sum readout can correctly predict all testing triplets, whereas C-MPNNs without sum readout will fail to learn this pattern and achieve  $50\%$  as random guessing. Theoretically, the TRI-SQR dataset is designed in such a way that any R-MPNN model assigns identical node representations to nodes in triangles and squares with the same relation type. Consequently, any C-MPNN model without global readout will be unable to determine which graph between  $G_{1}$  and  $G_{2}$  in the graph pair is being predicted, making it challenging to learn the conditional representation  $\pmb{h}_{u|v,q}$ . However, we anticipate that a C-MPNN model with sum readout can differentiate between  $G_{1}$  and  $G_{2}$  in each pair, as it can access global information like the total number of nodes in the graph. This allows it to accurately identify the graph being predicted in the graph pair, even when the representations of the triangle and square with the same relation are identical. As a result, it can learn the target rules and achieve  $100\%$  accuracy.
Experiment details. We configure each model variant with four layers and 32 hidden dimensions per layer. We set a learning rate of 1e-4 for both models and train them for 500 epochs. Empirically, we find that C-MPNN with global sum readout achieves  $100\%$  accuracy, while C-MPNN without global sum readout reaches  $50\%$  accuracy as random guesses, which is consistent with our theoretical expectations.