SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nimrah Mustafa*
nimrah.mustafa@cispa.de
Aleksandar Bojchevski†
a.bojchevski@uni-koeln.de
Rebekka Burkholz*
burkholz@cispa.de
$^{*}$ CISPA Helmholtz Center for Information Security, 66123 Saarbrücken, Germany
†University of Cologne, 50923 Köln, Germany
While the expressive power and computational capabilities of graph neural networks (GNNs) have been theoretically studied, their optimization and learning dynamics, in general, remain largely unexplored. Our study undertakes the Graph Attention Network (GAT), a popular GNN architecture in which a node's neighborhood aggregation is weighted by parameterized attention coefficients. We derive a conservation law of GAT gradient flow dynamics, which explains why a high portion of parameters in GATs with standard initialization struggle to change during training. This effect is amplified in deeper GATs, which perform significantly worse than their shallow counterparts. To alleviate this problem, we devise an initialization scheme that balances the GAT network. Our approach i) allows more effective propagation of gradients and in turn enables trainability of deeper networks, and ii) attains a considerable speedup in training and convergence time in comparison to the standard initialization. Our main theorem serves as a stepping stone to studying the learning dynamics of positive homogeneous models with attention mechanisms.
A rapidly growing class of model architectures for graph representation learning are Graph Neural Networks (GNNs) [20] which have achieved strong empirical performance across various applications such as social network analysis [8], drug discovery [27], recommendation systems [61], and traffic forecasting [50]. This has driven research focused on constructing and assessing specific architectural designs [59] tailored to various tasks.
On the theoretical front, mostly the expressive power [51, 26] and computational capabilities [45] of GNNs have been studied. However, there is limited understanding of the underlying learning mechanics. We undertake a classical variant of GNNs, the Graph Attention Network (GAT) [48, 9]. It overcomes the limitation of standard neighborhood representation averaging in GCNs [29, 22] by employing a self-attention mechanism [47] on nodes. Attention performs a weighted aggregation over a node's neighbors to learn their relative importance. This increases the expressiveness of the neural network. GAT is a popular model that continues to serve as strong baseline.
However, a major limitation of message-passing GNNs in general, and therefore GATs as well, is severe performance degradation when the depth of the network is even slightly increased. SOTA results are reportedly achieved by models of 2 or 3 layers. This problem has largely been attributed to over-smoothing [35], a phenomenon in which node representations become indistinguishable when multiple layers are stacked. To relieve GNNs of over-smoothing, practical techniques inspired by classical deep neural networks tweak the training process, e.g. by normalization [11, 64, 65, 66] or regularization [38, 42, 55, 67], and impose architectural changes such as skip connections (dense
and residual) [34, 53] or offer other engineering solutions [33] or their combinations [12]. Other approaches to overcome over-smoothing include learning CNN-like spatial operators from random paths in the graph instead of the point-wise graph Laplacian operator [17] and learning adaptive receptive fields (different 'effective' neighborhoods for different nodes) [63, 36, 52]. Another problem associated with loss of performance in deeper networks is over-squashing [1], but we do not discuss it since non-attentive models are more susceptible to it, which is not our focus.
Alternative causes for the challenged trainability of deeper GNNs beyond over-smoothing is hampered signal propagation during training (i.e. backpropagation) [25, 62, 13]. To alleviate this problem for GCNs, a dynamic addition of skip connections to vanilla-GCNs, which is guided by gradient flow, and a topology-aware isometric initialization are proposed in [25]. A combination of orthogonal initialization and regularization facilitates gradient flow in [21]. To the best of our knowledge, these are the only works that discuss the trainability issue of deep GNNs from the perspective of signal propagation, which studies a randomly initialized network at initialization. Here, we offer an insight into a mechanism that allows these approaches to improve the trainability of deeper networks that is related to the entire gradient flow dynamics. We focus in particular on GAT architectures and the specific challenges that are introduced by attention.
Our work translates the concept of neuronwise balancedness from traditional deep neural networks to GNNs, where a deeper understanding of gradient dynamics has been developed and norm balancedness is a critical assumption that induces successful training conditions. One reason is that the degree of initial balancedness is conserved throughout the training dynamics as described by gradient flow (i.e. a model of gradient descent with infinitesimally small learning rate or step size). Concretely, for fully connected feed-forward networks and deep convolutional neural networks with continuous homogenous activation functions such as ReLUs, the difference between the squared  $l2$ -norms of incoming and outgoing parameters to a neuron stays approximately constant during training [15, 32]. Realistic optimization elements such as finite learning rates, batch stochasticity, momentum, and weight decay break the symmetries induced by these laws. Yet, gradient flow equations can be extended to take these practicalities into account [31].
Inspired by these advances, we derive a conservation law for GATs with positive homogeneous activation functions such as ReLUs. As GATs are a generalization of GCNs, most aspects of our insights can also be transferred to other architectures. Our consideration of the attention mechanism would also be novel in the context of classic feed-forward architectures and could have intriguing implications for transformers that are of independent interest. In this work, however, we focus on GATs and derive a relationship between model parameters and their gradients, which induces a conservation law under gradient flow. Based on this insight, we identify a reason for the lack of trainability in deeper GATs and GCNs, which can be alleviated with a balanced parameter initialization. Experiments on multiple benchmark datasets demonstrate that our proposal is effective in mitigating the highlighted trainability issues, as it leads to considerable training speed-ups and enables significant parameter changes across all layers. This establishes a causal relationship between trainability and parameter balancedness also empirically, as our theory predicts. The main theorem presented in this work serves as a stepping stone to studying the learning dynamics of positive homogeneous models with attention mechanisms such as transformers [47] and in particular, vision transformers which require depth more than NLP transformers[49]. Our contributions are as follows.
- We derive a conservation law of gradient flow dynamics for GATs, including its variations such as shared feature weights and multiple attention heads.
- This law offers an explanation for the lack of trainability of deeper GATs with standard initialization.
- To demonstrate empirically that our theory has established a causal link between balancedness and trainability, we propose a balanced initialization scheme that improves the trainability of deeper networks and attains considerable speedup in training and convergence time, as our theory predicts.
2 Theory: Conservation laws in GATs
Preliminaries For a graph  $G = (V, E)$  with node set  $V$  and edge set  $E \subseteq V \times V$ , where the neighborhood of a node  $v$  is given as  $\mathcal{N}(v) = \{u | (u, v) \in E\}$ , a GNN layer computes each node's representation by aggregating over its neighbors' representation. In GATs, this aggregation is weighted by parameterized attention coefficients  $\alpha_{uv}$ , which indicate the importance of node  $u$  for  $v$ .
Table 1: Attributes of init. schemes
Init.	Feat.	Attn.	Bal.?
Xav	Xavier	Xavier	No
XavZ	Xavier	Zero	No
BalX	Xavier	Zero	Yes
BalO	LLortho	Zero	Yes
Figure 1: Params. to balance for neuron  $i \in [n_l]$
Given input representations  $h_v^{l-1}$  for all nodes  $v \in V$ , a GAT layer  $l$  transforms those to:
$$
h _ {v} ^ {l} = \phi \left(\sum_ {u \in \mathcal {N} (v)} \alpha_ {u v} ^ {l} \cdot W _ {s} ^ {l} h _ {u} ^ {l - 1}\right), \quad \text {w h e r e} \tag {1}
$$
$$
\alpha_ {u v} ^ {l} = \frac {\exp \left(\left(a ^ {l}\right) ^ {\top} \cdot \text {L e a k y R e L U} \left(W _ {s} ^ {l} h _ {u} ^ {l - 1} + W _ {t} ^ {l} h _ {v} ^ {l - 1}\right)\right)}{\sum_ {u ^ {\prime} \in \mathcal {N} (v)} \exp \left(\left(a ^ {l}\right) ^ {\top} \cdot \text {L e a k y R e L U} \left(W _ {s} ^ {l} h _ {u ^ {\prime}} ^ {l - 1} + W _ {t} ^ {l} h _ {v} ^ {l - 1}\right)\right)}. \tag {2}
$$
We consider  $\phi$  to be a positively homogeneous activation functions (i.e  $\phi(x) = x\phi'(x)$  and consequently,  $\phi(ax) = a\phi(x)$  for positive scalars  $a$ ), such as a ReLU  $\phi(x) = \max\{x, 0\}$  or LeakyReLU  $\phi(x) = \max\{x, 0\} + -\alpha \max\{-x, 0\}$ . The feature transformation weights  $W_s$  and  $W_t$  for source and target nodes, respectively, may also be shared such that  $W = W_s = W_t$ .
Definition 2.1. Given training data  $\{(x_m, y_m)\}_{m=1}^M \subset \mathbb{R}^d \times \mathbb{R}^p$  for  $M \leq V$ , let  $f: \mathbb{R}^d \to \mathbb{R}^p$  be the function represented by a network constructed by stacking  $L$  GAT layers as defined in Eq. (1) and (2) with  $W = W_s = W_t$  and  $h_m^0 = g(x_m)$ . Each layer  $l \in [L]$  of size  $n_l$  has associated parameters: a feature weight matrix  $W^l \in \mathbb{R}^{n_l \times n_{l-1}}$  and an attention vector  $a^l \in \mathbb{R}^{n_l}$ , where  $n_0 = d$  and  $n_L = p$ . Given a differentiable loss function  $\ell: \mathbb{R}^d \times \mathbb{R}^p \to \mathbb{R}$ , the loss  $\mathcal{L} = 1 / M \sum_{i=1}^M \ell(f(x_m), y_m)$  is used to update model parameters  $w \in \{W^l, a^l\}_{l=1}^L$  with learning rate  $\gamma$  by gradient descent, i.e.,  $w^{t+1} = w^t - \gamma \nabla_w \mathcal{L}$ , where  $\nabla_w \mathcal{L} = [\partial \mathcal{L} / \partial w_1, \dots, \partial \mathcal{L} / \partial w_{|w|}]$  and  $w^0$  is set by the initialization scheme. For an infinitesimal  $\gamma \to 0$ , the dynamics of gradient descent behave similarly to gradient flow defined by  $\frac{\mathrm{d}w}{\mathrm{d}t} = -\nabla_w \mathcal{L}$ , where  $t$  is the continuous time index.
We use  $W[i, \cdot]$ ,  $W(:, i]$ , and  $a[i]$  to denote the  $i^{th}$  row of  $W$ , column of  $W$ , and entry of  $a$ , respectively. Note that  $W^l[i, \cdot]$  is the vector of weights incoming to neuron  $i \in [n_l]$  and  $W^{l+1}[., i]$  is the vector of weights outgoing from the same neuron. For the purpose of discussion, let  $i \in [d_l]$ ,  $j \in [d_{l-1}]$ , and  $k \in [d_{l+1}]$ , as depicted in Fig. 1.  $\langle \cdot, \cdot \rangle$  denotes the scalar product.
Conservation Laws We focus the following exposition on weight-sharing GATs, as this variant has the practical benefit that it requires fewer parameters. Yet, similar laws also hold for the non-weight-sharing case, as we state and discuss in the supplement. For brevity, we also defer the discussion of laws for the multi-headed attention mechanism to the supplement.
Theorem 2.2 (Structure of gradients). Given the feature weight and attention parameters  $W^l$  and  $a^l$  of a layer  $l$  in a GAT network as described in Def. 2.1, the structure of the gradients for layer  $l \in [L - 1]$  in the network is conserved according to the following law:
$$
\left\langle W ^ {l} [ i,: ], \nabla_ {W ^ {l} [ i,: ]} \mathcal {L} \right\rangle - \left\langle a ^ {l} [ i ], \nabla_ {a ^ {l} [ i ]} \mathcal {L} \right\rangle = \left\langle W ^ {l + 1} [:,: i ], \nabla_ {W ^ {l + 1} [:,: i ]} \mathcal {L} \right\rangle . \tag {3}
$$
Corollary 2.3 (Norm conservation of weights incoming and outgoing at every neuron). Given the gradient structure of Theorem 2.2 and positive homogeneity of the activation  $\phi$  in Eq. (1), gradient flow in the network adheres to the following invariance for  $l \in [L - 1]$ :
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left(\left\| W ^ {l} [ i,: ] \right\| ^ {2} - \left\| a ^ {l} [ i ] \right\| ^ {2}\right) = \frac {\mathrm {d}}{\mathrm {d} t} \left(\left\| W ^ {l + 1} [ : i ] \right\| ^ {2}\right), \tag {4}
$$
This definition of GAT is termed GATv2 in [9]. Hereafter, we refer to GATv2 as GAT.
(a)  $\delta$  with GD
(b)  $\delta$  with Adam
(c)  $c$  with Xavier
(d)  $c$  with  $\mathrm{Bal}_O$
Figure 2: Visualizing conservation laws: (a),(b) show  $\delta \approx 0$  (can not empirically be exactly zero due to finite  $\gamma = 0.1$ ) from Eq. (7) for hidden layers when  $L = 3$ . (c),(d) show the value of  $c$  from Eq. (5), which is determined by the initialization and should be 0 when the network is balanced, for  $L = 10$  trained with GD. With Xav., the network is unbalanced and hence  $c \neq 0$  for all neurons. With  $\mathrm{Bal}_O$ ,  $c = 0$  exactly at initialization for all neurons by design but during training, it varies slightly and  $c \approx 0$  as the network becomes slightly unbalanced due to the finite  $\gamma = 0.1$ . As both  $\delta$  and  $c$  approximately meet their required value, we conclude that the derived law holds for practical purposes as well.
It then directly follows by integration over time that the l2-norms of feature and attention weights incoming to and outgoing from a neuron are preserved such that
$$
\left\| W ^ {l} [ i,: ] \right\| ^ {2} - \left\| a ^ {l} [ i ] \right\| ^ {2} - \left\| W ^ {l + 1} [ \colon i ] \right\| ^ {2} = c, \tag {5}
$$
where  $c = \frac{\mathrm{d}}{\mathrm{d}t}\left(\left\| W^l [i,:]\right\|^2 -\left\| a^l [i]\right\|^2\right)$  at initialization (i.e.  $t = 0$ ).
We denote the 'degree of balancedness' by  $c$  and call an initialization balanced if  $c = 0$ .
Corollary 2.4 (Norm conservation across layers). Given Eq. (4), the invariance of gradient flow at the layer level for  $l \in [L - 1]$  by summing over  $i \in [n_l]$  is:
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left(\left\| W ^ {l} \right\| _ {F} ^ {2} - \left\| a ^ {l} \right\| ^ {2}\right) = \frac {\mathrm {d}}{\mathrm {d} t} \left(\left\| W ^ {l + 1} \right\| _ {F} ^ {2}\right). \tag {6}
$$
Remark 2.5. Similar conservation laws also hold for the original less expressive GAT version [48] as well as for the vanilla GCN [29] yielded by fixing  $a^l = \mathbf{0}$ .
Insights We first verify our theory and then discuss its implications in the context of four different initializations for GATs, which are summarized in Table 1. Xavier (Xav.) initialization [19] is the standard default for GATs. We describe the Looks-Linear-Orthogonal (LLortho) initialization later.
In order to empirically validate Theorem 2.2, we rewrite Equation 3 to define  $\delta$  as:
$$
\delta = \left\langle W ^ {l} [ i,: ], \nabla_ {W ^ {l} [ i,: ]} \mathcal {L} \right\rangle - \left\langle a ^ {l}, \nabla_ {a ^ {l}} \mathcal {L} \right\rangle - \left\langle W ^ {l + 1} [:,: i ], \nabla_ {W ^ {l + 1} [:,: i ]} \mathcal {L} \right\rangle = 0. \tag {7}
$$
We observe how the value of  $\delta$  varies during training in Fig. 2 for both GD and Adam optimizers with  $\gamma = 0.1$ . Although the theory assumes infinitesimal learning rates due to gradient flow, it still holds sufficiently ( $\delta \approx 0$ ) in practice for normally used learning rates as we validate in Fig. 2. Furthermore, although the theory assumes vanilla gradient descent optimization, it still holds for the Adam optimizer (Fig. 2b). This is so because the derived law on the level of scalar products between gradients and parameters (see Theorem 2.2) still holds, as it states a relationship between gradients and parameters that is independent of the learning rate or precise gradient updates.
Having verified Theorem 2.2, we use it to deduce an explanation for a major trainability issue of GATs with Xavier initialization that is amplified with increased network depth.
The main reason is that the last layer has a comparatively small average weight norm, as  $\mathbb{E}\left\| W^L [:,i]\right\| ^2 = 2n_L / (n_L + n_{L - 1}) < < 1$ , where the number of outputs is smaller than the layer width  $n_L < < n_{L - 1}$ . In contrast,  $\mathbb{E}\left\| W^{L - 1}[i,:]\right\| ^2 = 2n_{L - 1} / (n_{L - 1} + n_{L - 2}) = 1$  and  $\mathbb{E}a^{L - 1}[i]^2 = 2 / (1 + n_{L - 1})$ . In consequence, the right-hand side of our derived conservation law
$$
\sum_ {j = 1} ^ {n _ {L - 2}} \left(W _ {i j} ^ {L - 1}\right) ^ {2} \frac {\nabla_ {W _ {i j} ^ {L - 1}} \mathcal {L}}{W _ {i j} ^ {L - 1}} - \left(a _ {i} ^ {L - 1}\right) ^ {2} \frac {\nabla_ {a _ {i} ^ {L - 1}} \mathcal {L}}{a _ {i} ^ {L - 1}} = \sum_ {k = 1} ^ {n _ {L}} \left(W _ {k i} ^ {L}\right) ^ {2} \frac {\nabla_ {W _ {k i} ^ {L}} \mathcal {L}}{W _ {k i} ^ {L}} \tag {8}
$$
(a) Frac. of sig. params with relative change  $>0.05$
(b) Rel. grad. norms of feat. wghts. & training curve
(a) Frac. of sig. params with relative change  $>0.5$
Figure 4: For  $L = 10$ , unbalanced initialization is unable to train the model as relative gradients are extremely small, and the parameters (and loss) do not change. However, the balanced initialization is able to propagate gradients, change parameters, and thus attain training loss  $\rightarrow 0$ . Xav. and  $\mathrm{Bal}_O$  achieve  $39.3\%$  and  $80.2\%$  test accuracy, respectively. Note that the balanced initialization in (b) is also able to train faster than the 5 layer network with Xavier initialization in Fig. 3b.
Figure 3: For  $L = 5$ , a high fraction of parameters change, and gradients propagate to earlier layers for both the unbalanced and balanced initialization Xav. and  $\mathrm{Bal}_O$  initialization achieve 75.5% and 79.9% test accuracy, respectively. Note that the balanced initialization in (b) is able to train faster.
(b) Rel. grad. norms of feat. wghts. & training curve
is relatively small, as  $\sum_{k=1}^{n_L} (W_{ki}^L)^2 < 1$ , on average, while the weights on the left-hand side are orders of magnitude larger. This implies that the relative gradients on the left-hand side are likely relatively small to meet the equality unless all the signs of change are set in such a way that they satisfy the equality, which is highly unlikely during training. Therefore, relative gradients in layer  $L-1$  and accordingly also the previous layers of the same dimension can only change in minor ways.
The amplification of this trainability issue with depth can also be explained by the recursive substitution of Theorem 2.2 in Eq. (8) that results in a telescoping series yielding:
$$
\sum_ {j = 1} ^ {n _ {1}} \sum_ {m = 1} ^ {n _ {0}} W _ {j m} ^ {(1) ^ {2}} \frac {\nabla_ {W _ {j m} ^ {(1)}} \mathcal {L}}{W _ {j m} ^ {(1)}} - \sum_ {l = 1} ^ {L - 1} \sum_ {o = 1} ^ {n _ {l}} a _ {o} ^ {(l) ^ {2}} \frac {\nabla_ {a _ {o} ^ {(l)}} \mathcal {L}}{a _ {o} ^ {(l)}} = \sum_ {i = 1} ^ {n _ {L - 1}} \sum_ {k = 1} ^ {n _ {L}} W _ {k i} ^ {(L) ^ {2}} \frac {\nabla W _ {k i} ^ {(L)} \mathcal {L}}{W _ {k i} ^ {(L)}} \tag {9}
$$
Generally,  $2n_{1} < n_{0}$  and thus  $\mathbb{E}\left\| W^{1}[j:]\right\|^{2} = 2n_{1} / (n_{1} + n_{0}) < 1$ . Since the weights in the first layer and the gradients propagated to the first layer are both small, gradients of attention parameters of the intermediate hidden layers must also be very small in order to satisfy Eq. (9). Evidently, the problem aggravates with depth where the same value must now be distributed over the parameters and gradients of more layers. Note that the norms of the first and last layers are usually not arbitrary but rather determined by the input and output scale of the network.
Analogously, we can see how a balanced initialization would mitigate the problem. Equal weight norms  $\left\| W^{L - 1}[i,:]\right\|^2$  and  $\left\| W^L [.;,i]\right\|^2$  in Eq. (8) (as the attention parameters are set to 0 during balancing, see Procedure 2.6) would allow larger relative gradients in layer  $L - 1$ , as compared to the imbalanced case, that can enhance gradient flow in the network to earlier layers. In other words, gradients on both sides of the equation have equal room to drive parameter change.
Figure 5: Relative gradient norms of feature (left axis, solid) and of attention (right axis, stylized) parameters for  $l \in [1, 5, 10]$  and  $L = 10$ , sampled every 25 epochs. Test accuracy is at the top. Both attention and feature gradients at the first, middle, and last layer of the network with both balanced initializations are much larger than with unbalanced initialization (note axis scales).
To visualize the trainability issue, we study the relative change  $\left| \left( w^{*} - w_{0} \right) / w^{*} \right|$ . of trained network parameters  $(w^{*})$  w.r.t. their initial value. In order to observe meaningful relative change, we only consider parameters with a significant contribution to the model output  $(w^{*} \geq 10^{-4})$ . We also plot the relative gradient norms of the feature weights across all layers to visualize their propagation. We display these values for  $L = 5$  and  $L = 10$  in Fig. 3 and 4, respectively, and observe a stark contrast, similar to the trends in relative gradient norms of selected layers of a 10 layer GAT in Fig. 5. In all experiments, GAT was trained with the specified initialization on Cora using GD with  $\gamma = 0.1$  for 5000 epochs.
Interestingly, the attention parameters change most in the first layer and increasingly more towards the last layer, as seen in Fig. 3a and 4a. This is consistently observed in both the 5 and 10 layer networks if the initialization is balanced, but with unbalanced initialization, the 10 layer network is unable to produce the same effect. As our theory defines a coarse-level conservation law, such fine-grained training dynamics cannot be completely explained by it.
Since the attention parameters are set to 0 in the balanced initializations (see Procedure 2.6), as ablation, we also observe the effect of initializing attention and feature parameters with zero and Xavier, respectively. From Eq.(2),  $a^l = 0$  leads to  $\alpha_{uv} = 1 / |\mathcal{N}(v)|$ , implying that all neighbors  $u$  of the node  $v$  are equally important at initialization. Intuitively, this allows the network to learn the importance of neighbors without any initially introduced bias and thus avoids the 'chicken and egg' problem that arises in the initialization of attention over nodes[30]. Although this helps generalization in some cases (see Fig. 6), it alone does not help the trainability issue as seen in Fig. 4a. It may in fact worsen it (see Fig. 5), which is explained by Eq. 8. These observations underline the further need for parameter balancing.
Procedure 2.6 (Balancing). Based on Eq. (5) from the norm preservation law 2.3, we note that in order to achieve balancedness, (i.e. set  $c = 0$  in Eq.(5)), the randomly initialized parameters  $W^l$  and  $a^l$  must satisfy the following equality for  $l \in [L]$ :
$$
\left\| W ^ {l} [ i,: ] \right\| ^ {2} - \left\| a ^ {l} [ i ] \right\| ^ {2} = \left\| W ^ {l + 1} [: i ] \right\| ^ {2}
$$
This can be achieved by scaling the randomly initialized weights as follows:
1. Set  $a^l = \mathbf{0}$  for  $l\in [L]$
2. Set  $W^{1}[i,:] = \frac{W^{1}[i,:]}{\|W^{1}[i,:]\|}\sqrt{\beta_{i}}$ , for  $i \in [n_1]$  where  $\beta_{i}$  is a hyperparameter
3. Set  $W^{l + 1}[:,i] = \frac{W^{l + 1}[.;i]}{\|W^{l + 1}[.;i]\|}\left\| W^l [i,:]\right\|$  for  $i\in [n_l]$  and  $l\in [L - 1]$
In step 2,  $\beta_{i}$  determines the initial squared norm of the incoming weights to neuron  $i$  in the first layer. It can be any constant thus also randomly sampled from a normal or uniform distribution. We explain why we set  $\beta_{i} = 2$  for  $i\in [n_1]$  in the context of an orthogonal initialization.