SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\mathrm {d} w}{\mathrm {d} t} = - \nabla_ {w} \mathcal {L} \tag {16}
$$
which implies:
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left\| W ^ {l} [ i,: ] \right\| ^ {2} = 2 \langle W ^ {l} [ i,: ], \frac {\mathrm {d}}{\mathrm {d} t} W ^ {l} [ i,: ] \rangle = - 2 \langle W ^ {l} [ i,: ], \nabla_ {W ^ {l} [ i,: ]} \mathcal {L} \rangle \tag {17}
$$
substituting in Eq.(4) similarly for gradient flow of  $\mathbf{W}^{l + 1}[:,i]$  and  $a^l [i]$ , as done in Eq.(17) yields Theorem 2.2:
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} \left(\left\| W ^ {l} [ i,: ] \right\| ^ {2} - \left\| a ^ {l} [ i ] \right\| ^ {2}\right) = \frac {\mathrm {d}}{\mathrm {d} t} \left(\left\| W ^ {l + 1} [ : i ] \right\| ^ {2}\right). \\ - 2 \langle W ^ {l} [ i,: ], \nabla_ {W ^ {l} [ i,: ]} \mathcal {L} \rangle - (- 2) \langle a ^ {l} [ i ], \nabla_ {a ^ {l} [ i ]} \mathcal {L} \rangle = - 2 \langle W ^ {l + 1} [:,: i ], \nabla_ {W ^ {l + 1} [:,: i ]} \mathcal {L} \rangle . \\ \end{array}
$$
Therefore, Eq.(4) and consequently Eq.(5) in Corollary 2.3 hold.
Summing over  $i \in [n_l]$  on both sides of Eq.(4) yields Corollary 2.4.
Theorem 6.3 (Structure of gradients for GAT without weight-sharing). Let a GAT network as defined by Def. 2.1 consisting of  $L$  layers be given. The feature transformation parameters  $\mathbf{W}_s^l$  and  $\mathbf{W}_t^l$  of the source and target nodes of an edge and attention parameters  $\mathbf{a}^l$  of a GAT layer  $l$  are defined according to Eq.(1) and (2).
Then the gradients for layer  $l \in [L - 1]$  in the network are governed by the following law:
$$
\left\langle W _ {s} ^ {l} [ i,: ], \nabla_ {W _ {s} ^ {l} [ i,: ]} \mathcal {L} \right\rangle + \left\langle W _ {t} ^ {l} [ i,: ], \nabla_ {W _ {t} ^ {l} [ i,: ]} \mathcal {L} \right\rangle - \left\langle a ^ {l} [ i ], \nabla_ {a ^ {l} [ i ]} \mathcal {L} \right\rangle = \left\langle W _ {s} ^ {l + 1} [:,: i ], \nabla_ {W _ {s} ^ {l + 1} [:,: i ]} \mathcal {L} \right\rangle \tag {18}
$$
Proof. The proof is analogous to the derivation of Theorem 2.2. We follow the same principles and define the disjoint subsets  $\theta_{1}$  and  $\theta_{2}$  of the parameter set  $\theta$ , associated with a neuron  $i$  in layer  $l$  accordingly, as follows:
$$
\theta_ {1} = \{w | w \in W _ {s} ^ {l} [ i,: ] \} \cup \{w | w \in W _ {t} ^ {l} [ i,: ] \}
$$
$$
\theta_ {2} = \{w | w \in W _ {s} ^ {l + 1} [:, i ] \} \cup \{a ^ {l} [ i ] \}
$$
Then, the invariance of node representations follows similarly to the proof of Theorem 2.2.
The only components of the network that potentially change under rescaling are  $h_u^l [i]$ ,  $h_v^{l + 1}[j]$ , and  $\alpha_{uv}^{l}$ . We denote the scaled network components with a tilde resulting in  $\tilde{h}_u^l [i]$ ,  $\tilde{h}_v^{l + 1}[k]$ , and  $\tilde{\alpha}_{uv}^{l}$ . As we show, parameters of upper layers remain unaffected, as  $\tilde{h}_v^{l + 1}[k]$  coincides with its original non-scaled variant  $\tilde{h}_v^{l + 1}[k] = h_v^{l + 1}[k]$ .
Let us start with the attention coefficients. Note that  $\tilde{a}^l [i] = \lambda^{-1}a^l [i]$ ,  $\tilde{W}_s^l [i,j] = \lambda W_s^l [i,j]$  and  $\tilde{W}_t^l [i,j] = \lambda W_t^l [i,j]$ . This implies that
$$
\tilde {\alpha} _ {u v} ^ {l} = \frac {\exp \left(e _ {u v} ^ {l}\right)}{\sum_ {u ^ {\prime} \in \mathcal {N} (v)} \exp \left(e _ {u v} ^ {l}\right)} = \alpha_ {u v} ^ {l}, \text {b e c a u s e} \tag {19}
$$
$$
\tilde {e} _ {u v} ^ {l} = \left(a ^ {l}\right) ^ {\top} \cdot \phi_ {2} \left(W _ {s} ^ {l} h _ {u} ^ {l - 1} + W _ {t} ^ {l} h _ {v} ^ {l - 1}\right) = e _ {u v} ^ {l}, \tag {20}
$$
$$
\begin{array}{l} \tilde {e} _ {u v} ^ {l} = \lambda^ {- 1} a ^ {l} [ i ] \phi_ {2} \left(\sum_ {j} ^ {n _ {l - 1}} \lambda W _ {s} ^ {l} [ i, j ] h _ {u} ^ {l - 1} [ j ] + \lambda W _ {t} ^ {l} [ i, j ] h _ {v} ^ {l - 1} [ j ] \right. (21) \\ + \sum_ {i ^ {\prime} \neq i} ^ {n _ {l}} a ^ {l} [ i ] \phi_ {2} \left(\sum_ {j} ^ {n _ {l - 1}} W _ {s} ^ {l} [ i, j ] h _ {u} ^ {l - 1} [ j ] + W _ {t} ^ {l} [ i, j ] h _ {v} ^ {l - 1} [ j ]\right) (22) \\ = \lambda^ {- 1} \lambda a ^ {l} [ i ] \phi_ {2} \left(\sum_ {j} ^ {n _ {l - 1}} W _ {s} ^ {l} [ i, j ] h _ {u} ^ {l - 1} [ j ] + W _ {t} ^ {l} [ i, j ] h _ {v} ^ {l - 1} [ j ] \right. (23) \\ + \sum_ {i ^ {\prime} \neq i} ^ {n _ {l}} a ^ {l} [ i ] \phi_ {2} \left(\sum_ {j} ^ {n _ {l - 1}} W _ {s} ^ {l} [ i, j ] h _ {u} ^ {l - 1} [ j ] + W _ {t} ^ {l} [ i, j ] h _ {v} ^ {l - 1} [ j ]\right) (24) \\ = e _ {u v} ^ {l}. (25) \\ \end{array}
$$
$$
\begin{array}{l} \tilde {h _ {u}} ^ {l} [ i ] = \phi_ {1} \left(\sum_ {z \in \mathcal {N} (u)} \alpha_ {z u} ^ {l} \sum_ {j} ^ {n _ {l - 1}} \lambda W _ {s} ^ {l} [ i, j ] h _ {z} ^ {l - 1} [ j ]\right) \\ = \lambda \phi_ {1} \left(\sum_ {z \in \mathcal {N} (u)} \alpha_ {z u} ^ {l} \sum_ {j} ^ {n _ {l - 1}} W _ {s} ^ {l} [ i, j ] h _ {z} ^ {l - 1} [ j ]\right) \\ = \lambda h _ {u} ^ {l} [ i ] \\ \end{array}
$$
$$
\begin{array}{l} \tilde {h _ {v}} ^ {l + 1} [ k ] = \phi_ {1} \left(\sum_ {u \in \mathcal {N} (v)} \alpha_ {u v} ^ {l + 1} \sum_ {i} ^ {n _ {l}} \lambda^ {- 1} W _ {s} ^ {l + 1} [ k, i ] \tilde {h _ {u}} ^ {l} [ i ]\right) \\ = \phi_ {1} \left(\sum_ {u \in \mathcal {N} (v)} \alpha_ {u v} ^ {l + 1} \sum_ {i} ^ {n _ {l}} \lambda^ {- 1} W _ {s} ^ {l + 1} [ k, i ] \lambda h _ {u} ^ {l} [ i ]\right) \\ = \phi_ {1} \left(\sum_ {u \in \mathcal {N} (v)} \alpha_ {u v} ^ {l + 1} \sum_ {i} ^ {n _ {l}} W _ {s} ^ {l + 1} [ k, i ] h _ {u} ^ {l} [ i ]\right) \\ = h _ {v} ^ {l + 1} [ k ] \\ \end{array}
$$
Thus, the application of Lemma 6.2 derives Theorem 6.3.
Theorem 6.4 (Structure of gradients for GAT with multi-headed attention). Given the feature transformation parameters  $W_{k}^{l}$  and attention parameters  $a_{k}^{l}$  of an attention head  $k \in [K]$  in a GAT layer of a  $L$  layer network. Then the gradients of layer  $l \in [L - 1]$  respect the law:
$$
\sum_ {k} ^ {K} \left\langle W _ {k} ^ {l} [ i, : ], \nabla_ {W _ {k} ^ {l} [ i, : ]} \mathcal {L} \right\rangle - \sum_ {k} ^ {K} \left\langle a _ {k} ^ {l} [ i ], \nabla_ {a _ {k} ^ {l} [ i ]} \mathcal {L} \right\rangle = \sum_ {k} ^ {K} \left\langle W _ {k} ^ {l + 1} [:, i ], \nabla_ {W _ {k} ^ {l + 1} [:, i ]} \mathcal {L} \right\rangle . \tag {26}
$$
Proof. Each attention head is independently defined by Eq.1 and Eq.2, and thus Theorem 2.2 holds for each head, separately. The aggregation of multiple heads in a layer is done over node representations of each head in either one of two ways:
Concatenation:  $h_v^l = \|_k^K \phi(\sum_{u \in \mathcal{N}(v)} \alpha_{kuv}^l \cdot W_k^l h_u^{l-1})$ , or
Average:  $h_v^l = \frac{1}{K} \sum_k^K \phi \left( \sum_{u \in \mathcal{N}(v)} \alpha_{kuv}^l \cdot W_k^l h_u^{l-1} \right)$
Thus, the rescale invariance and consequent structure of gradients of the parameters in each head are not altered and Eq.(6.2) holds by summation of the conservation law over all heads.
For the most general case of a GAT layer without weight-sharing and with multi-headed attention, each term in Theorem 6.3 is summed over all heads. Further implications analogous to Corollary 2.3, and 2.4 can be derived for these variants by similar principles.
7 Training Dynamics
In the main paper, we have shared observations regarding the learning dynamics of GAT networks of varying depths with standard initialization and or balanced initialization schemes. Here, we report additional figures for more varied depths to complete the picture.
All figures in this section correspond to a  $L$  layer GAT network trained on Cora. For SGD, the learning rate is set to  $\gamma = 0.1$ ,  $\gamma = 0.05$  and  $\gamma = 0.01$  for  $L = 5$ ,  $L = 10$  and  $L = 20$ , respectively. For Adam,  $\gamma = 0.005$  and  $\gamma = 0.0001$  for  $L = 5$  and  $L = 10$ , respectively.
Relative change of a feature transformation parameter  $w$  is defined as  $\left| \left( w^{*} - w_{0} \right) / w^{*} \right|$  where  $w_{0}$  is the initialized value and  $w^{*}$  is the value for the model with maximum validation accuracy during training. Absolute change  $\left| a^{*} - a_{0} \right|$  is used for attention parameters  $a$  since attention parameters are initialized with 0 in the balanced initialization. We view the fraction of parameters that remain unchanged during training separately in Figure 8, and examine the layer-wise distribution of change in parameters in Figure 11 without considering the unchanged parameters.
(a)  $L = 5$
(b)  $L = 10$
Figure 8: Layer-wise fraction of feature transformation parameters  $W^l$  and attention parameters  $a^l$  with zero relative and absolute change, respectively, trained with SGD. For  $L = 2$ , no parameters with zero change existed. A small number of feature weights do not change in a 5 layer unbalanced network initialized with Xavier, but this fraction becomes significantly large when depth is increased to  $L = 10$ . Note that  $W^1$  contains a much larger number of parameters compared to the intermediate layers (specifically in this case, though it is generally common). At  $L = 20$ , nearly all  $w \in W^l$  with an unbalanced initialization (Xav. and Xav+ZeroAtt) struggle to change during training, whereas the balanced Xavier and LL-orthogonal initialization are able to drive change in most  $w \in W^l$  and all  $a \in a^l$  parameters, allowing the network to train.
(c)  $L = 20$
To analyze how gradients vary in the network during training, we define the relative gradient norm of feature transformation and attention parameters for a layer  $l$  as  $\| \nabla_{W^l}W^l\| _F / \| W^l\| _F$  and  $\| \nabla_{a^l a^l}\| /\| a^l\|$ , respectively. Figures 3b and 4b, and Figure 12 depict relative gradient norms for training under SGD and Adam respectively.
8 Additional Results
The results of the main paper have focused on the vanilla GAT having ReLU activation between consecutive layers, a single attention head, and shared weights for feature transformation of the source and target nodes, optimized with vanilla gradient descent (or Adam) without any regularization. Here, we present additional results for training with architectural variations, comparing a balanced initialization with a normalization scheme focused on GATs, and discussing the impact of an orthogonal initialization and the applicability of the derived conservation law to other message-passing GNNs (MPGNNs).
Training variations To understand the impact of different architectural variations, common regularization strategies such as dropout and weight decay, and different activation functions, we conducted additional experiments. For a 10-layer network with width 64, Table 3 and 4 report our results for SGD and Adam, respectively. The values of hyperparameters were taken from [48].
Table 3: Mean test accuracy(%) ± 95% CI over five runs of GAT trained on Cora using SGD
Variation	L	Xav	BalX	BalO
None (Vanilla GAT)	5	73.00 ± 3.02	76.96 ± 2.21	79.48 ± 0.43
10	25.48 ± 18.13	77.72 ± 1.49	79.46 ± 1.34
attention heads = 8	5	73.56 ± 2.71	77.44 ± 1.54	79.58 ± 0.53
10	25.50 ± 18.18	77.02 ± 2.76	79.06 ± 0.73
activation = ELU	5	75.68 ± 1.80	79.20 ± 1.07	79.64 ± 0.36
10	73.02 ± 2.27	78.64 ± 1.72	47.76 ± 7.39
dropout = 0.6	5	42.14 ± 15.97	79.18 ± 1.17	81.00 ± 0.62
10	24.90 ± 9.50	30.94 ± 1.04	44.40 ± 1.84
weight decay = 0.0005	5	67.26 ± 6.30	77.36 ± 1.74	79.56 ± 0.48
10	18.78 ± 11.96	76.56 ± 2.91	79.40 ± 1.15
weight sharing = False	5	70.04 ± 3.45	77.46 ± 1.19	79.66 ± 0.72
10	19.54 ± 15.64	76.16 ± 2.28	80.06 ± 1.12
Table 4: Mean test accuracy(%) ± 95% CI over five runs of GAT trained on Cora using Adam
Variation	L	Xav	Balx	BalO
None (Vanilla GAT)	5	76.18 ± 1.61	79.38 ± 2.24	80.20 ± 0.57
10	70.86 ± 3.99	75.72 ± 2.35	79.62 ± 1.27
attention heads = 8	5	75.62 ± 1.74	78.54 ± 1.06	79.56 ± 0.85
10	70.94 ± 2.76	75.48 ± 2.48	79.74 ± 1.10
activation = ELU	5	76.56 ± 1.72	78.72 ± 1.02	79.56 ± 0.64
10	75.30 ± 1.42	78.48 ± 1.79	76.52 ± 0.97
dropout = 0.6	5	76.74 ± 1.44	78.42 ± 1.28	79.92 ± 0.48
10	32.48 ± 6.99	31.76 ± 1.33	76.34 ± 0.95
weight decay = 0.0005	5	75.10 ± 2.05	78.52 ± 1.41	79.80 ± 0.63
10	28.74 ± 12.04	74.68 ± 3.06	79.70 ± 1.14
weight sharing = False	5	76.90 ± 0.92	78.90 ± 1.07	80.62 ± 1.36
10	74.46 ± 1.90	75.50 ± 3.49	79.78 ± 0.70
We find that the balanced initialization performs similar in most variations, outperforming the unbalanced standard initialization by a substantial margin. Dropout generally does not seem to be helpful to the deeper network ( $L = 10$ ), regardless of the initialization. From Table 3 and 4, it is evident that although techniques like dropout and weight decay may aid optimization, they are alone not enough to enable the trainability of deeper network GATs and thus are complementary to balancedness.
Note that ELU is not a positively homogeneous activation function, which is an assumption in our theory. In practice, however, it does not impact the Xavier-balanced initialization  $(\mathrm{Bal}_X)$ . However, the Looks-Linear orthogonal structure is specific to ReLUs. Therefore, the orthogonality and balancedness of the  $\mathrm{Bal}_O$  initialization are negatively impacted by ELU, although the Adam optimizer seems to compensate for it to some extent.
In addition to increasing the trainability of deeper networks, the balanced initialization matches the state-of-the-art performance of Xavier initialized 2 layer GAT, given the architecture and optimization hyperparameters as reported in [48]. We used an existing GAT implementation and training script<sup>2</sup> of
$^{2}$ https://github.com/Diego999/pyGAT
Cora that follows the original GAT(v1)[48] paper and added our balanced initialization to the code. As evident from Table 5, the balanced initialization matches SOTA performance of GAT(v1) on Cora  $(83 - 84\%)$  (on a version of the dataset with duplicate edges). GAT(v2) achieved  $78.5\%$  accuracy on Pubmed but the corresponding hyperparameter values were not reported. Hence, we transferred them from [48]. This way, we are able to match the state-of-the-art performance of GAT(v2) on Pubmed, as shown in Table 5.
Table 5: GAT performance on Cora and Pubmed: mean test accuracy(%) ± 95% CI over five runs.
Xav	BalX	BalO
Cora	84.50 ± 0.52	84.58 ± 0.65	84.55 ± 0.47
Pubmed	78.38 ± 0.77	78.52 ± 0.54	78.56 ± 0.20
Comparison with Lipschitz Normalization A feasible comparison can be carried out with [14] that proposes a Lipschitz normalization technique aimed at improving the performance of deeper GATs in particular. We use their provided code to reproduce their experiments on Cora, Citeseer, and Pubmed for 2,5,10,20 and 40-layer GATs with Lipschitz normalization and compare them with LLortho+Bal initialization in Table 6. Note that Lipschitz normalization has been shown to outperform other previous normalization techniques for GNNs such as pair-norm [64] and layer-norm [5].
Table 6: Comparing a balanced LL-orthogonal initialization to Lipschitz normalization applied with a standard (imbalanced) Xavier initialization: the balanced initialization results in a much higher accuracy as the depth of the network increases.
Dataset	Cora	Citeseer	Pubmed
Layers	Lip. Norm.	BalO Init.	Lip. Norm.	BalO Init.	Lip. Norm.	BalO Init.
2	82.1	79.5	65.4	67.7	74.8	76.0
5	77.1	80.2	63.0	67.7	73.7	75.7
10	78.0	79.6	43.6	67.4	52.8	76.9
20	72.2	77.3	18.2	66.3	23.3	77.3
40	12.9	75.9	18.1	63.2	36.6	77.5
Impact of orthogonal initialization In the main paper, we have advocated and presented results for using a balanced LL-orthogonal initialization. Here, we discuss two special cases of orthogonal initialization (and their balanced versions): identity matrices that have also been used for GNNs in [18, 16], and matrices with looks-linear structure using an identity submatrix (LLidentity) since a looks-linear structure would be more effective for ReLUs [10].
In line with [18], identity and LLidentity are used to initialize the hidden layers while Xavier is used to initialize the first and last layers. A network initialized with identity is balanced by adjusting the weights of the first and last layer to have norm 1 (as identity matrices have row-wise and column-wise norm). A network with LLidentity initialization is balanced to have norm 2 in the first and last layer, similar to LLortho initialization. We compare the performance using these four base initializations (one standard Xavier, and three orthogonal cases) and their balanced counterparts in Fig. 9.
We observe that balancing an (LL-)orthogonal initialization results in an improvement in the generalization ability of the network in most cases and speeds up training, particularly for deeper networks. However, note that an (LL-)orthogonal initialization itself also has a positive effect on trainability in particular of deep networks. Contributing to this fact is the mostly-balanced nature of an (LL-)orthogonal initialization i.e. given hidden layers of equal dimensions the network is balanced in all layers except the first and last layers (assuming zero attention parameters), which allows the model to train, as opposed to the more severely imbalanced standard Xavier initialization. This further enforces the key takeaway of our work that norm imbalance at initialization hampers the trainability of GATs. In addition, the LLortho+Bal initialization also speeds up training over the LLortho initialization even in cases in which the generalization performance of the model is at par for both initializations.
Figure 9: GAT network with varying initialization trained using SGD.
Note that identity initializations have also been explored in the context of standard feed-forward neural networks. While they tend to work in practice, the lack of induced feature diversity can be problematic from a theoretical point of view (see e.g. for a counter-example [7]). However, potentially due to the reasons regarding feature diversity and ReLU activation discussed above, the balanced looks-linear random orthogonal initialization (LLortho+Bal.) outperforms initialization with identity matrices (see Fig. 9). In most cases, the balanced versions outperform the imbalanced version of the base initialization and the performance gap becomes exceedingly large as network depth increases.
Applicability to other MPGNNs As GAT is a generalization of the GCN, all theorems are also applicable to GCNs (where the attention parameters are simply 0). We provide additional experiments in Table 7, comparing the performance of GCN models of varying depths with imbalanced and balanced initializations trained on Cora using the SGD optimizer. We use the same hyperparameters as reported for GAT. As expected, the trainability benefits of a balanced initialization are also evident in deeper GCN networks.
Table 7: GCN with varying initialization trained using SGD.
Dataset	Depth	Xavier	Xavier+Bal	LLortho	LLortho+Bal
Cora	2	77.8 ± 0.9	80.5 ± 0.5	78.0 ± 0.3	80.9 ± 0.4
5	73.2 ± 3.4	78.3 ± 0.8	80.3 ± 0.8	79.6 ± 1.0
10	24.1 ± 4.5	77.6 ± 2.0	80.0 ± 1.1	80.0 ± 0.9
20	14.4 ± 11.2	62.8 ± 3.6	78.7 ± 0.5	78.8 ± 1.5
40	13.4 ± 0.9	65.9 ± 7.3	28.3 ± 16.2	77.1 ± 0.9
64	9.8 ± 5.4	33.0 ± 13.4	27.3 ± 12.7	76.7 ± 1.3
80	12.4 ± 19.3	33.8 ± 12.9	38.9 ± 21.3	77.1 ± 1.3
Citeseer	2	66.6 ± 20.0	71.3 ± 1.8	66.0 ± 3.2	72.3 ± 0.9
5	60.9 ± 12.3	66.9 ± 15.0	69.0 ± 6.4	70.1 ± 1.8
10	23.8 ± 36.8	66.0 ± 5.9	70.6 ± 0.9	69.8 ± 10.9
20	16.4 ± 18.2	47.9 ± 10.0	67.0 ± 8.6	69.7 ± 4.5
40	13.9 ± 56.8	37.3 ± 92.8	44.8 ± 6.8	64.7 ± 13.6
64	13.8 ± 41.4	29.5 ± 15.0	37.3 ± 79.6	66.3 ± 0.5
80	12.4 ± 42.7	25.8 ± 3.6	30.1 ± 21.8	64.1 ± 3.2
The underlying principle of a balanced network initialization holds in general. However, adapting the balanced initialization to different methods entails modification of the conservation law derived for GATs to correspond to the specific architecture of the other method. For example, the proposed balanced initialization can be applied directly to a more recent variant of GAT,  $\omega$ GAT [18], for which the same conservation law holds and also benefits from a balanced initialization. We conduct
additional experiments to verify this. The results in Fig. 10 follow a similar pattern as for GATs: a balanced orthogonal initialization with looks linear structure (LLortho+Bal) of  $\omega$ GAT performs the best, particularly by a wide margin at much higher depths (64 and 80 layers).
Figure 10:  $\omega$ GAT network with varying trained on Cora.
In this work, we focus our exposition on GATs and take the first step in modeling the training dynamics of attention-based models for graph learning. An intriguing direction for future work is to derive modifications in the conservation law for other attention-based models utilizing the dot-product self-attention mechanism such as SuperGAT [28] and other architectures based on Transformers[47], that are widely used in Large Language Models (LLMs) and have recently also been adapted for graph learning [58, 60].
9 Datasets Summary
We used nine benchmark datasets for the transductive node classification task in our experiments, as summarized in Table 8.
For the Platenoid datasets (Cora, CiteSeer, and Pubmed) [56], we use the graphs provided by Pytorch Geometric (PyG), in which each link is saved as an undirected edge in both directions. However, the number of edges is not exactly double (for example, 5429 edges are reported in the raw Cora dataset, but the PyG dataset has  $10556 < (2 \times 5429)$ ) edges as duplicate edges have been removed in preprocessing. We also remove isolated nodes in the CiteSeer dataset.
The WebKB (Cornell, Texas, and Wisconsin), Wikipedia (Squirrel and Chameleon) and Actor datasets [39], are used from the replication package provided by [40], where duplicate edges are removed.
All results in this paper are according to the dataset statistics reported in Table 8.
Table 8: Summary of datasets used in experiments.
Dataset	Cora	Cite.	Pubmed	Cham.	Squi.	Actor	Corn.	Texas	Wisc.
# Nodes	2708	3327	19717	2277	5201	7600	183	183	251
# Edges	10556	9104	88648	31371	198353	26659	277	279	450
# Features	1433	3703	500	2325	2089	932	1703	1703	1703
# Classes	7	6	3	5	5	5	5	5	5
# Train	140	120	60	1092	2496	3648	87	87	120
# Validation	500	500	500	729	1664	2432	59	59	80
# Test	1000	1000	1000	456	1041	1520	37	37	51
(a)  $L = 10$ : The balanced initialization allows larger changes in a higher number of parameters in  $W^l$  across all layers  $l$  with the highest margin in  $l = 1$ . The change distribution for the parameters in  $a^l$  is missing for  $l = 5$  and  $l \in [7, 10]$  because these parameters remain unchanged (see Fig.8).
(b)  $L = 20$ : The same pattern as in  $L = 10$  is seen. However, the difference between the trainability of models with unbalanced and balanced initialization becomes even more prominent. No attention parameters change at all with standard Xavier initialization (and hence their change distribution is not visible in the plot).
Figure 11: Distributions of the relative and absolute change in the values of feature transformation parameters  $W^l$  and attention parameters  $a^l$ , respectively, when trained using SGD with unbalanced (Xav. and Xav+ZeroAtt) and balanced (Xav+Bal and LLortho+Bal) initialization. The black markers in each standard box-whisker plot denote the mean. In general, larger changes occur in attention parameters at later layers closer to the output of the network, whereas feature parameters change more in the earlier layers at the input of the network. We observe this also from the perspective of relative gradients, as higher gradients cause higher parameter changes.
(a)  $L = 5$ : test accuracy (left to right) is  $71.6\%$ ,  $76.2\%$ ,  $77.3\%$ , and  $80.5\%$ .