SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From Equation 3 and the assumption that the COMBINE' function is injective it follows that
$$
\left(h _ {v} ^ {l + 1, 1}, \dots , h _ {v} ^ {l + 1, K}\right) = \left(h _ {w} ^ {l + 1, 1}, \dots , h _ {w} ^ {l + 1, K}\right).
$$
Thus, for every  $j \in [[K]]$  it holds that  $h_v^{l+1,j} = h_w^{l+1,j}$ . This implies Equation 4.
Lemma F.7. sKP-GNN can be strongly simulated.
Proof. The proof reuses large parts the proof that KP-GNNs can be weakly simulated (Theorem 5.3). As sKP-GNN only differs from KP-GNN in the definition of  $h_v^{t+1}$  it follows from the proof that  $h_v^{l+1,k}$  is AMP for every  $k \in [[K]]$ . Then,  $h_v^{t+1}$  is AMP that applies the function COMBINE $^t$  (S4) to a tuple (S3) built from the AMPs  $h_v^{l+1,1}, \ldots, h_v^{l+1,K}$ . From Theorem 3.5 it follows that sKP-GNN can be strongly simulated.
Proof. (Theorem F.4) The theorem follows from Lemma F.6 and Lemma F.7.  $\square$
G Intuition for CWN and CWN
Figure 5: Left: graph. Center: regular cell complex built from the graph through a graph-to-structure encoding [Bodnar et al., 2021a]. Vertices correspond to 0-dimensional cells, edges to 1-dimensional cells (yellow) and induced cycles to 2-dimensional cells (blue). Right: a graph created by structure-to-graph encoding the regular cell complex to a graph. Vertices corresponds to cells as indicated by color.
CW Networks [Bodnar et al., 2021a, CWN] perform message passing on topological structures called regular cell complexes. A regular cell complex is built out of individual cells of some dimension.
Similar to message passing on graphs, CWN computes a representation (or color) for each cell by aggregating from the neighboring cells. To apply CWNs to graphs, a regular cell complex has to be constructed from a graph (see Figure 5), we call this process graph-to-structure encoding. For this vertices are transformed into 0-dimensional cells and edges into 1-dimensional cells. In this process additional structures such as (induced) cycles or cliques can also be lifted to cells to improve the expressiveness and connectivity in the structure. In the experiments we use a lifting map that lifts induced cycles of length at most 6 to cells. Figure 5 shows how a regular cell complex is transformed to a graph and shows how our structure-to-graph encoding  $\overline{\mathrm{CWN}}$  transforms the complex back to a graph. Note that in our implementation of  $\overline{\mathrm{CWN}}$  we use undirected instead of directed for the sake of simplicity. When transforming a cell to a vertex, we set the vertex features to the most common feature of the vertices making up the cell for each feature. For example, a cell that corresponds to a cycle  $v$ ,  $w$ ,  $p$  is transformed to a vertex whose features are the most common of the features of  $v$ ,  $w$ ,  $p$ . Similarly, the features of edges connecting cells are the most common of the edge features between the vertices making up the cells.
H More Details about the Experiments
We provide further details about our experiments. In Section H.1 we elaborate on our setup and in Section H.2 we present more details on the results from the main paper. In Section H.3 we present an additional evaluation of transformation and training speed. The code for our experiments can be found at https://github.com/ocatias/GNN-Simulation.
H.1 Setup
All models are implemented in Pytorch Geometric [Fey and Lenssen, 2019]. We use WandB [Biewald, 2020] to track our experiments. Unfortunately, CWN requires an old version of PyTorch [Paszke et al., 2019] meaning that we have to train it on older GPUs. CWN training and all speed evaluations are performed on an an NVIDIA GeForce GTX 1080 GPU. All other experiments are performed on NVIDIA GeForce RTX 3080 GPUs.
Type Pooling. In our transformations  $\overline{\mathrm{DSS}}$  and  $\overline{\mathrm{CWN}}$  vertices get assigned additional features that encode the type of a vertex. In  $\overline{\mathrm{CWN}}$ , these features encode whether the cell was built from a vertex, edge or cycle. In  $\overline{\mathrm{DSS}}$ , these features encode whether a vertex is part of the original graph or in a subgraph. Inspired by a pooling operation by Bodnar et al. [2021a] we propose type pooling. The idea is that vertices with different types contain different information and have varying importance. Thus, instead of pooling all vertices in a graph, we instead pool all vertices of a type. To get a single output for the entire graph, we then apply a multilayer perceptron with a ReLU activation function to each pooled vector and then sum the results. Let POOL be a pooling operation such as  $\sum$ . Let  $T$  be the set of types and let  $H_{t}$  with  $t \in T$  be the multiset of all representations of vertices of type  $t$ , and  $\mathrm{MLP}_{t}$  be a multilayer perceptron with activation function. Then, type pooling computes
$$
\text {T y p e - P o o l} = \sum_ {t \in T} \mathrm {M L P} _ {t} \left(\mathrm {P O O L} \left(H _ {t}\right)\right).
$$
We use type pooling for all combinations of an MPNN plus  $\overline{\mathrm{DS}}$ ,  $\overline{\mathrm{DSS}}$  and  $\overline{\mathrm{CWN}}$ . Note that for  $\overline{\mathrm{DS}}$  all nodes have just one type.
Hyperparameter Tuning. Table 3 contains all hyperparameter grids for the real life tasks. During preliminary experiments we found that it is crucial to tune the pooling operation and number of layers to achieve strong results. For DS(S) we do not tune the pooling function as the code provided by Bevilacqua et al. [2021] does not directly support this. Since CWN is very slow, we use a smaller hyperparameter grid, containing a superset of the hyperparameters used by Bodnar et al. [2021a].
Setup Real-Life Datasets. For all real-life datasets we tune the hyperparameters on the validation set and evaluate the best hyperparameters on the test set 10 times. We report the average and standard deviation of the metric that is typically used on the given dataset. For all OGB datasets, we train with a batch size of 32 for 100 epochs with a fixed learning rate to allow for fast training of many models. For ZINC we train as described by Bevilacqua et al. [2021], Bodnar et al. [2021a] and Dwivedi et al.
Table 3: Hyperparameter grid used for the real life datasets. The learning rate is only tuned on OGB tasks. On ZINC all models used the same initial learning rate that decays over time.
MODEL → ↓ PARAMETERS	GIN, GCN	DSS, DS	CIN ON OGB	CIN ON ZINC
EMBEDDING DIMENSION	64, 256	64, 256	64, 256	64, 256
NUMBER OF LAYERS	1, ..., 5	1, ..., 5	1,2,3	3,4,5
LEARNING RATE	10-3, 10-4	10-3, 10-4	10-3, 10-4	10-3
POOLING OPERATION	SUM, MEAN	MEAN	SUM, MEAN	SUM, MEAN
DROPOUT RATE	0.5	0.5	0.5	0.5
[2023] i.e., for up to 500 epochs with a batch size of 128 with an initial learning rate of  $10^{-3}$  that reduces by a factor of 0.5 every time the validation result has not improved for 20 epochs. If the learning rate dips below  $10^{-5}$  the training stops.
Setup CSL. For the synthetic CSL dataset, we only evaluate GIN plus the graph transformations and compare to the results reported by Bevilacqua et al. [2021] and Bodnar et al. [2021a]. We perform 10-fold cross validation on this dataset. For all models with graph transformations we replicate the hyperparameters used by the corresponding non-standard message passing algorithms described in the original papers. For GIN, we tune hyperparameters as described for real-life datasets.
H.2 Results on Real-Life Datasets
Tables 4 and 5 show the prediction quality of our models on all real-life datasets.
Table 4: Bold results are better than the corresponding message passing baseline, red results are graph transformation based methods that are better than the corresponding non-standard message passing variation.  $\downarrow$  means lower is better.  $\uparrow$  means higher is better.
DATA SET → ↓MODEL	MOLHIV ROC-AUC ↑	MOLTOX21 ROC-AUC ↑	MOLESOL RSMSE ↓	ZINC MAE ↓
GIN	77.7 ± 1.0	76.1 ± 0.5	0.946 ± 0.086	0.306 ± 0.035
GCN	76.6 ± 1.0	74.9 ± 0.6	0.928 ± 0.049	0.456 ± 0.021
MLP	72.2 ± 0.9	71.2 ± 0.7	1.128 ± 0.021	1.44 ± 0.001
DS	75.4 ± 1.4	75.4 ± 0.6	0.986 ± 0.051	0.161 ± 0.008
DS + GIN	77.2 ± 1.2	77.0 ± 0.6	0.924 ± 0.03	0.143 ± 0.008
DS + GCN	75.4 ± 0.8	76.6 ± 0.6	0.872 ± 0.037	0.285 ± 0.044
DS + MLP	69.8 ± 6.1	70.8 ± 0.8	1.292 ± 0.115	1.281 ± 0.002
DSS	76.8 ± 1.3	77.0 ± 1.0	0.864 ± 0.024	0.105 ± 0.005
DSS + GIN	74.4 ± 1.3	78.1 ± 0.7	0.853 ± 0.015	0.149 ± 0.008
DSS + GCN	74.8 ± 2.1	76.3 ± 0.7	0.868 ± 0.022	0.221 ± 0.028
DSS + MLP	71.9 ± 0.8	69.7 ± 0.5	1.272 ± 0.025	1.283 ± 0.006
CWN	78.6 ± 1.5	75.2 ± 0.9	1.117 ± 0.051	0.13 ± 0.003
CWN + GIN	78.4 ± 1.5	74.4 ± 0.7	0.958 ± 0.021	0.137 ± 0.007
CWN + GCN	78.1 ± 1.0	74.2 ± 0.9	0.933 ± 0.036	0.155 ± 0.013
CWN + MLP	71.8 ± 0.8	71.0 ± 0.5	1.217 ± 0.032	1.045 ± 0.006
H.3 Speed Evaluation
Setup. We evaluate the preprocessing and training speed on ZINC for GIN and our graph transformations against DS(S) and CWN. We select the hyperparameters for all GNNs such that they all have roughly  $8 \cdot 10^{5}$  trainable parameters. To measure the transformation speed, we begin the time measurement after the graphs have been loaded into memory and stop before we store the final objects. For our graph transformations, this measures how long it takes to run the graph transformations. For CWN, this measures how long it takes to create the cell complexes from the graphs. Additionally, we measure how long it takes to train the model for 100 epochs on ZINC evaluating it after every epoch. We average all time measurements over 10 trials.
Table 5: Bold results are better than the corresponding message passing baseline, red results are graph transformation based methods that are better than the corresponding non-standard message passing variation.  $\downarrow$  means lower is better.  $\uparrow$  means higher is better.
DATA SET → ↓MODEL	MOL-BACE ROC-AUC ↑	MOL-CLINTOX ROC-AUC ↑	MOL-BBBP ROC-AUC↑	MOL-SIDER ROC-AUC ↑	MOL-TOXCAST ROC-AUC ↑	MOL-LIPO RMSE ↓
GIN	75.3 ± 4.9	87.1 ± 1.5	66.8 ± 2.3	56.8 ± 1.0	64.2 ± 0.7	0.787 ± 0.026
GCN	76.9 ± 3.8	88.2 ± 1.2	66.6 ± 2.1	57.5 ± 0.9	63.2 ± 0.5	0.793 ± 0.028
MLP	41.6 ± 11.2	57.1 ± 3.9	63.7 ± 1.8	58.9 ± 1.4	62.8 ± 0.3	1.048 ± 0.009
DS	79.2 ± 1.9	85.7 ± 2.9	67.4 ± 1.3	60.6 ± 1.1	66.3 ± 0.5	0.736 ± 0.022
DS + GIN	75.3 ± 3.5	81.4 ± 4.2	66.2 ± 1.8	59.3 ± 0.9	66.7 ± 0.5	0.716 ± 0.014
DS + GCN	77.7 ± 2.8	83.8 ± 2.4	66.0 ± 1.1	61.0 ± 0.9	66.1 ± 0.4	0.754 ± 0.018
DS + MLP	71.8 ± 5.6	55.4 ± 5.0	63.6 ± 2.1	52.9 ± 1.6	58.5 ± 5.5	1.048 ± 0.011
DSS	74.5 ± 7.4	84.6 ± 2.6	67.5 ± 1.5	57.5 ± 1.5	66.4 ± 0.8	0.652 ± 0.009
DSS + GIN	77.2 ± 3.0	86.4 ± 1.2	64.3 ± 2.5	59.0 ± 1.1	66.8 ± 0.4	0.76 ± 0.018
DSS + GCN	74.8 ± 3.9	87.3 ± 1.6	67.3 ± 2.0	62.0 ± 1.0	66.1 ± 0.6	0.767 ± 0.027
DSS + MLP	72.8 ± 4.3	51.9 ± 7.7	63.5 ± 2.0	51.6 ± 2.0	62.0 ± 0.8	1.054 ± 0.01
CWN	75.0 ± 2.7	84.7 ± 2.4	69.5 ± 1.6	57.6 ± 1.6	64.7 ± 0.9	0.755 ± 0.018
CWN + GIN	75.2 ± 3.3	85.4 ± 2.4	68.1 ± 2.4	59.1 ± 1.0	64.5 ± 1.1	0.749 ± 0.013
CWN + GCN	72.2 ± 3.8	88.0 ± 1.6	61.5 ± 3.0	62.4 ± 1.1	64.4 ± 0.6	0.787 ± 0.016
CWN + MLP	38.1 ± 0.6	55.7 ± 4.8	62.7 ± 1.6	59.9 ± 1.1	63.1 ± 0.5	1.055 ± 0.01
Table 6: Speed comparison of different GNNs. TRANSFORM is the time to transform the graphs to the required structure to perform message passing over. TRAIN is the time to train a model to train for 100 epochs. COMBINED is the combined time of training and transformation. Bold indicates the best result for a given non-standard message passing algorithm and corresponding graph transformation.
MODEL	TRANSFORM
SEC. ↓	TRAIN.
SEC. ↓	COMBINED
SEC. ↓
GIN	NA	170 ± 1	170 ± 1
CWN	37 ± 1	3259 ± 16	3296 ± 17
CWN + GIN	67 ± 1	286 ± 6	353 ± 7
DSS	95 ± 4	955 ± 5	1050 ± 9
DSS + GIN	183 ± 8	1466 ± 7	1649 ± 15
DS	95 ± 4	1235 ± 5	1330 ± 9
DS + GIN	171 ± 8	1060 ± 5	1231 ± 13
Results. Table 6 shows the results of the speed evaluation. We can see that the graph transformation are roughly twice as slow as the alternative transformation for the non-standard message passing. However, this runtime is still significantly smaller than the time it takes to train the model a single time. Thus, the runtime of graph transformations is insignificant as the transformation only needs to be performed a single time whereas the model might need to be trained dozens of times to find good hyperparameters. With respect to the training time,  $\overline{\mathrm{DSS}} +\mathrm{GIN}$  is roughly  $54\%$  slower than DSS and  $\overline{\mathrm{DS}} +\mathrm{GIN}$  is roughly  $17\%$  faster than DS. Interestingly,  $\overline{\mathrm{CWN}} +\mathrm{GIN}$  is 11 times faster than CWN. While it might seem surprising that our graph transformation based variant of CWN is 11x faster than the original implementation, the reasons for that might be: (1) a bug, for example in how the original code of CWN interacts with our environment; (2) CWN requires a large amount of code built on top of PyTorch Geometric, whereas  $\overline{\mathrm{CWN}} +\mathrm{GIN}$  just performs standard message passing; (3) the implementation of CWN by Bodnar et al. [2021a] is not compatible with newer variants of PyTorch (Geometric) meaning that  $\overline{\mathrm{CWN}} +\mathrm{GIN}$  can make use of speed increases due to improvements in PyToch (Geometric). Reasons (2) and (3) highlight the strength of graph transformations in that they are mostly platform and implementation independent as the transformed graphs can be stored in file formats that are independent from the used deep learning framework.