SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b)  $L = 10$ : test accuracy (left to right) is  $63.2\%$ ,  $68.8\%$ ,  $71.8\%$ , and  $78.0\%$ .
Figure 12: Layer-wise relative gradient norms of GAT parameters trained with Adam, which in itself improves trainability over SGD, especially for  $L = 10$  (see Fig.4b) even with standard initialization. However, with a balanced initialization, the training is faster and the model also generalizes better, as evident from the significant margin in test accuracy. A general noticeable trend, particularly with balanced initialization, is that the attention parameters have higher gradients at the end of the network (and thus change more in the later layers (see Fig. 11)) whereas the feature parameters have high gradients at the start of the network. Also, it can be seen that the feature parameters begin to change a few epochs before the attention parameters change noticeably, while the attention parameters continue to change for a few more epochs after the gradients of feature parameters have dropped. We speculate a representational benefit could drive this behaviour, i.e. with depth, the learned representation of neighbors becomes increasingly informative for a node and thus leads to a higher activity of the attention mechanism.