SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since 1-WL showed  $\chi_{\mathrm{CSL}(n,2)}^{1} = \chi_{\mathrm{CSL}(n,r)}^{1}$  and augmented graphs showed  $\chi_{\mathrm{CSL}(n,2)^1}^1 = \chi_{\mathrm{CSL}(n,r)^1}^1$  two graphs are indistinguishable.
Iteration 2. Again, we focus on four nodes connected to  $v_{0}$ . Here, we describe color refinement for nodes  $v_{1}, v_{n-1}, v_{2}, v_{n-2}$ , since the following is enough to prove two graph colors are distinct.
- For  $v \in \{v_1, v_{n-1}\}, \chi_{\mathrm{CSL}(n,2)^1}^2(v) = \mathrm{hash}(c_2, \{\{c_1, c_2, c_2, c_3\}\}) = c_4$ .
- For  $v \in \{v_2, v_{n-2}\}$ ,  $\chi_{\mathrm{CSL}(n,2)^1}^2(u) = \mathrm{hash}(c_2, \{c_1, c_2, c_3, c_3\}) = c_5$ .
- For  $v \in \{v_1, v_{n-1}\}$ ,  $\chi_{\mathrm{CSL}(n,r)^1}^2(v) = \mathrm{hash}(c_2, \{\{c_1, c_3, c_3, c_3\}\}) = c_6$ .
Since  $c_{6} \notin \chi_{\mathrm{CSL}(n,2)^{1}}^{2}$ , two graphs are distinguishable, i.e.,  $\chi_{\mathrm{CSL}(n,2)^{1}}^{t} \neq \chi_{\mathrm{CSL}(n,r)^{1}}^{t}$ .
Then  $\chi_{\mathrm{CSL}(n,2)^1}^T \neq \chi_{\mathrm{CSL}(n,r)^1}^T$  is satisfied as proved in Lemma 1, thereby two graphs  $\mathrm{CSL}(n,2)$  and  $\mathrm{CSL}(n,r)$  are distinguishable by the AGG-WL.
In Figure 9, we provide an example with  $\mathrm{CSL}(8,2)$  and  $\mathrm{CSL}(8,3)$ , indistinguishable by the 1-WL but distinguishable the AGG-WL.
B.3.2 Correspondence between DPM-SNC and AGG-WL
Now, we explain the connection between the DPM-SNC and the AGG-WL test. At a high-level, we show how DPM-SNC simulates the color refinement process of the AGG-WL. Our main idea stems from the marginalization in DPM-SNC, which can define the probability of the graph color  $\chi$  with the marginalization over augmented graphs, i.e.,  $p_{\theta}(\chi | G) = \int p_{\theta}(\chi | G, z) p(z | G) dz$ . Specifically, DPM-SNC considers sample space of graph colors, where each member  $\chi$  is obtained from  $p_{\theta}(\chi | G, z)$  with an augmented graph  $G^{m}$  represented by  $(G, z^{m})$ . By construction, DPM-SNC considers graph colors refinements for multiple augmented graphs, similar to how AGG-WL works.
Lemma 5. DPM-SNC using a 1-WL-GNN is as powerful as the AGG-WL test in distinguishing non-isomorphic graphs.
The node labels  $y$  is replaced to the graph color  $\chi$ , and applying a latent variable is interpreted as applying an augmentation to the given graph.
Proof. We show that if two graphs  $G, H$  are distinguishable by AGG-WL test, it is also distinguishable by the DPM-SNC. Assume that the two graphs  $G, H$  are distinguishable by the AGG-WL. Then, the following condition is satisfied.
$$
\chi_ {G} ^ {\mathrm {A G G}} = \{\{\chi_ {G ^ {m}} ^ {T}: m \in [ | \mathcal {Z} ^ {G} | ] \} \} \neq \chi_ {H} ^ {\mathrm {A G G}} = \{\{\chi_ {H ^ {m}} ^ {T}: m \in [ | \mathcal {Z} ^ {H} | ] \} \},
$$
where the AGG-WL produces distinct sets of refined graph colors for  $G, H$ . Next, we discuss how the DPM-SNC can distinguish  $G, H$ . To this end, we first assume a GNN which is as powerful as the 1-WL test [20], denoted as 1-WL-GNN. Specifically, we denote  $\chi_G^{\mathrm{GNN}}$  as a graph color refined by the 1-WL-GNN, where  $\chi_{G^m}^T \neq \chi_{H^n}^T$  implies  $\chi_{G^m}^{\mathrm{GNN}} \neq \chi_{H^n}^{\mathrm{GNN}}$  for any augmented graph pair  $G^m, H^n$ . Then, under the  $\chi_G^{\mathrm{AGG}} \neq \chi_H^{\mathrm{AGG}}$ , the following condition is also satisfied.
$$
\{\{\chi_ {G ^ {m}} ^ {\mathrm {G N N}}: m \in [ | \mathcal {Z} ^ {G} | ] \} \} \neq \{\{\chi_ {H ^ {m}} ^ {\mathrm {G N N}}: m \in [ | \mathcal {Z} ^ {H} | ] \} \}.
$$
Here, we assume that DPM-SNC utilizes this 1-WL-GNN to output a graph color conditioned on an augmented graph, i.e.,  $p_{\theta}(\chi | G, \mathbf{z})$ . Then, we can connect  $p_{\theta}(\chi | G, \mathbf{z}^m)$  with  $\chi_{G^m}^{\mathrm{GNN}}$  as follows:
$$
p _ {\boldsymbol {\theta}} (\chi | G, \boldsymbol {z} ^ {m}) = \mathbb {I} [ \chi = \chi_ {G ^ {m}} ^ {\mathrm {G N N}} ].
$$
where  $\mathbb{I}[a = b]$  is an indicator function whose value is 1 if  $a = b$  and 0 otherwise. Next, we also assume that  $p(z|G)$  is a uniform distribution over  $\mathcal{Z}^G$ . Then one can show that:
$$
p _ {\theta} (\chi | G) = \frac {1}{| \mathcal {Z} ^ {G} |} \sum_ {m \in [ | \mathcal {Z} ^ {G} | ]} \mathbb {I} [ \chi = \chi_ {G ^ {m}} ^ {\mathrm {G N N}} ].
$$
Then it follows that  $p_{\theta}(\chi |G)\neq p_{\theta}(\chi |H)$  since  $\{\chi_{G^m}^{\mathrm{GNN}}:m\in [|\mathcal{Z}^G ||]\} \neq \{\chi_{H^m}^{\mathrm{GNN}}:m\in [|\mathcal{Z}^H ||]\}$ . Therefore,  $G,H$  are distinguishable by DPM-SNC.
Our proof is valid when considering the multiple outputs. However, our practical implementation of DPM-SNC does not use multiple outputs at inference time since the DPM only considers the multiple random variables in the training objective. To this end, in Appendix F, we also investigate the inference scheme which aggregates multiple outputs to make final predictions.
$^{6}$ Our inference method is described in Appendix C.1
In practice, we also use a Gaussian diffusion for defining  $p(z|G)$ , which still allows the GNN to consider infinite augmentations.
C Implementation
In this section, we provide more details on how we implement the DPM-SNC for experiments.
C.1 Transductive settings
Here, we provide the detailed implementation of DPM-SNC for transductive node classification.
Model architecture. We parameterize the residual function  $\epsilon_{\theta}(\pmb{y}^{(t)}, G, t)$  of reverse diffusion step using a  $L$ -layer message-passing GNN as follows:
$$
\begin{array}{l} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {y} ^ {(t)}, G, t) = g (\boldsymbol {h} ^ {(L)}), \\ h _ {i} ^ {(\ell)} = (\mathrm {C O M B I N E} ^ {(\ell)} (h _ {i} ^ {(\ell - 1)}, a _ {i} ^ {(\ell)}) + f (t)) \| y _ {i} ^ {(t)}), \\ a _ {i} ^ {(\ell)} = \operatorname {A G G R E G A T E} ^ {(\ell)} \left(\left\{h _ {j} ^ {(\ell - 1)} \right| (i, j) \in \mathcal {E} \right\}), \\ \end{array}
$$
where  $g(h^{(L)})$  is a multi-layer perceptron that estimates the residual using the final node representation. AGGREGATE(\cdot) and COMBINE(\cdot) functions are identical to the backbone GNN, and  $\cdot \| \cdot$  indicates the concatenation. Here,  $h_i^0$  is  $x_i\| y_i^{(t)}$ . The  $f(\cdot)$  is a sinusoidal positional embedding function [62]. We fix the dimension of sinusoidal positional embedding to 128.
Buffer construction. Following the temperature annealing approach of Qu et al. [7] in sampling pseudo-labels for optimization, we also control the temperature of randomness in obtaining  $\mathbf{y}_U$  from  $p_{\theta}(\mathbf{y}_U|G,\mathbf{y}_L)$  for buffer construction. To be specific, we use the variance multiplied by the temperature  $\tau \in [0,1]$ , instead of the original variance in the reverse diffusion step, e.g., setting  $\tau$  to zero makes the deterministic sampling.
Inference. To make final predictions  $\mathbf{y}_U$  from  $p_{\theta}(\mathbf{y}_U|G,\mathbf{y}_L)$  for evaluation, we eliminate the randomness in inference time, i.e., set temperature  $\tau$  to zero. If the target is one-hot relaxation of discrete labels, we also discretize the final prediction by choosing a dimension with maximum value.
C.2 Inductive setting
Here, we provide the details of DPM-SNC for inductive node classification and graph algorithmic reasoning. For the inductive node classification, we use the same model architecture as in the transductive setting and use the deterministic inference strategy. For the graph algorithmic, we modify DPM-SNC to perform edge-wise prediction.
Graph algorithmic reasoning. Since the targets of the graph algorithmic reasoning task are defined on the edge-level, we apply a diffusion process to the edge labels. We then recover the edge-level noisy labels through the reverse process.
The denoising model architecture in the reverse process has a similar architecture to the model architecture of IREM [26]. Specifically, the noisy edge target  $\pmb{y}^{(t)}$  is updated as follows. First, the noisy edge labels  $\pmb{y}^{(t)}$  and edge features are concatenated and passed through to the GNN layer, which aggregates them to obtain the node representation. Next, we apply element-wise addition of the time embedding vector to the node representation. Then, we concatenate a pair of node representations and noisy targets for the given edges and then apply a two-layer MLP to update edge labels.
In contrast to the node classification, we maintain randomness at inference time, i.e., we use the stochastic reverse process for obtaining edge labels. This approach is consistent with the IREM, which also includes randomness at inference time.
We also investigate various stochastic inference strategies in Appendix F
D Data statistics
D.1 Synthetic data
We generate  $1000 \times 2$  non-attributed cyclic grid and  $100 \times 2$  non-attributed cyclic grid for scattered and localized training nodes scenarios, respectively. Then, we split  $30\%$ ,  $30\%$ , and  $40\%$  of the entire nodes into training, validation, and test nodes.
- Scattered training nodes: We randomly sample nodes in the graph to split them into training, validation, and test nodes.
- Localized training nodes: We select the nodes in the region within the  $30 \times 2$  grid as training nodes. Then, we randomly sample the remaining nodes in the graph to split them into validation and test nodes.
For illustrative purposes, we also describe both scenarios in Figure 10 with smaller graphs.
(a) Scattered-training
(b) Scattered-test/validation
Figure 10: Illustration of two scenarios. The non-gray nodes represent nodes in each split.
(c) Localized-training
(d) test/validation
D.2 Transductive node classification datasets
Table 6: The data statistics of transductive node classification datasets.
Dataset	# nodes	# edges	# features	# classes	# (training/validation/test) nodes
Pubmed [22]	19717	44338	500	3	(60/500/1000)
Cora [22]	2708	5429	1433	7	(140/500/1000)
CiteSeer [22]	3327	4732	3703	6	(120/500/1000)
Photo [23]	7487	119043	745	8	(160/240/7087)
Computers [23]	13381	34493	767	10	(200/300/12881)
Roman [24]	22662	32927	300	18	(11331/5665/5666)
Ratings [24]	24492	93050	300	5	(12246/6123/6123)
Here, we consider a graph with partially labeled nodes. We describe the data statistics in Table 6.
D.3 Inductive node classification datasets
Table 7: The data statistics of inductive node classification datasets.
Dataset	# features	# classes	(training/validation/test) data
# graphs	Avg. # nodes	Avg. # edges
Pubmed [9]	500	3	(60/500/1000)	(6.0/5.4/5.6)	(6.7/5.8/6.7)
Cora [9]	1433	7	(140/500/1000)	(5.6/4.9/4.7)	(7.0/5.8/5.3)
CiteSeer [9]	3703	6	(120/500/1000)	(4.0/3.8/3.8)	(4.3/4.0, 3/8)
PPI [25]	500	3	(20/2/2)	(2245.3/3257.0/2762.0)	(61318.4/99460.0/80988.0)
Here, we consider datasets consists of a set of graphs. We describe the data statistics in Table 7.
${}^{9}$  Additionally,we also consider training with an additional  $2 \times  {20}$  cyclic grid for visualization in Figure 1.
D.4 Graph algorithmic reasoning datasets
Following Du et al. [26], we generate training graphs in each training step. Here, the training graphs are composed of graphs of varying sizes, ranging from two to ten nodes. The node features are initialized to zero, and the labels are defined on the edges, e.g., the shortest distance between two nodes. Then, we evaluate performance on graphs with ten nodes. Furthermore, we also use graphs with 15 nodes to evaluate generalization capabilities.
E Experiments setup
In this section, we describe the detailed experimental setup. For all experiments, we use a single GPU of NVIDIA GeForce RTX 3090. The hyper-parameters for each experiment are described in the following subsections.
E.1 Synthetic dataset
In this experiment, we implement each method with a one-layer GCN with 16 hidden dimensions. We search the learning rate within  $\{1\mathrm{e} - 3,5\mathrm{e} - 3,1\mathrm{e} - 2\}$  for all methods. Other hyper-parameters of each method follow their default settings. For DPM-SNC, we fix the diffusion step to 100. We also set the size of the buffer to 50 and insert five samples into the buffer for every 30 training step. We use a pre-trained mean-field GNN until the buffer is updated 10 times.
E.2 Transductive node classification
Table 8: The hyper-parameter search ranges for the homophilic graph. For hyper-parameters without a specific method in parentheses, it applies to all methods in the respective category.
Method	Hyper-parameters	Search range
All methods	learning rate	{1e-3, 5e-3, 1e-2}
weight decay	{1e-3, 5e-3, 1e-2}
GNN-based methods (LPA, GMNN, G3NN, CLGNN, DPM-SNC)	number of layers	{2,4}
hidden dimension	{64,128}
weight of constraints for structured-prediction (LPA, G3NN)	{0.1,1.0,10.0}
pseudo-labels sampling temperature (GMNN, CLGNN, DPM-SNC)	{0.1,0.3,1.0}
Non-GNN methods (LP, PTA)	number of label propagation	{10,100}
hidden dimension (PTA)	{64,128}
damping factor	{0.1,0.3,0.5}
Homophilic graph. We describe the hyper-parameter search ranges in Table 8. Additionally, we apply dropout with  $p = 0.5$  except for LP. Other hyper-parameters of each method follow their default settings. For DPM-SNC, we fix the diffusion step to 80. We also set the size of the buffer to 50 and insert five samples into the buffer for every 100 training step. We use a pre-trained mean-field GNN until the buffer is updated 20 times.
Heterophilic graph. Here, we describe the hyperparameter settings for DPM-SNC as we use the numbers reported by Plantov et al. [24] for baselines. We describe the hyper-parameter search ranges in Table 9. Additionally, we apply dropout with  $p = 0.5$ , and we fix the diffusion step to 80. We also set the size of the buffer to 50 and insert five samples into the buffer for every 100 training step. We use a pre-trained meanfield GNN until the buffer is updated 100 times.
Table 9: The hyper-parameter search ranges of DPM-SNC for the heterophilic graph.
Hyper-parameters	Search range
learning rate	{3e-5, 1e-4, 3e-4}
weight decay	{0, 1e-5, 1e-4}
number of layers	{2, 4}
hidden dimension	{256, 512}
sampling temperature	{0.1, 0.3, 1.0}
E.3 Inductive node classification
Table 10: The hyper-parameter search ranges of all methods for the inductive node classificaiton.
Hyper-parameters	Search range
learning rate	{1e-3, 5e-3, 1e-2} for small-scale graphs and {3e-5, 1e-4, 3e-4} for huge-scale graphs
weight decay	{1e-3, 5e-3, 1e-2} for small-scale graphs and {0, 1e-5, 1e-4} for huge-scale graphs
number of layers	{2, 4}
hidden dimension	{64, 128} for small-scale graphs and {512, 1024} for huge-scale graphs
We describe the hyper-parameter search ranges in Table 10. For the small-scale graph datasets, i.e., Pubmed, Cora, and Citeseer, we apply dropout with  $p = 0.5$ . For the huge-scale graph datasets, i.e., PPI, we include the linear skip connection between each GNN layer. Other hyper-parameters of each method follow their default settings. For DPM-SNC, we fix the diffusion step to 80.
E.4 Algorithmic reasoning
Here, we describe the hyper-parameter settings for DPM-SNC as we use the numbers reported by Du et al. [26] for baselines. We search the learning rate and weight decay within  $\{1\mathrm{e} - 4,3\mathrm{e} - 4,1\mathrm{e} - 3\}$  and  $\{0,1\mathrm{e} - 5,1\mathrm{e} - 4\}$ , respectively. The hyper-parameters of the model are the same as the model implementation of IREM, using a three-layer GINEConv [63] with a 128 hidden dimension. We fix the diffusion step to 80.
F Additional experiments
Ablations on various inference schemes. We also study various inference strategies for our DPM-SNC. We first investigate how temperature control affects label inference in real-world node classification tasks. In Figure 11(a), we plot the changes in accuracy for various temperatures  $\tau$ . One can see that reducing the randomness of DPM-SNC gives a better prediction in real-world node classification tasks.
We also consider sampling various numbers of predictions for node-wise aggregation to improve performance. In Figure 11(b), even the number of samples is increased to  $2^{10}$ , the deterministic and the node-wise aggregation inference schemes show similar accuracy, and there are only minor performance improvements on CiteSeer. In practice, we use deterministic inference in the node classification tasks since the node-wise aggregation requires a relatively long time.
(b)
Figure 11: (a) Accuracy with varying temperature. (b) Accuracy with the varying number of samples. The dashed line represents the accuracy of the deterministic inference scheme.