## 3 WGN for RNNs
### Mathematical Explanation of the WGN  
In this section, taking the simple RNN as an example, we analyze the WGN method mathematically. According to the model training process, after normalization, the sparsity of the weight gradient matrix decreases, and the variance approaches 1. The distribution of the weight gradient matrix before and after normalization is shown in Figure 3.  
The formula for weight update is shown in Equation (6):  
\[w^{iter}=w^{iter-1}-\Delta w^{iter}=w^{0}-\sum_{i=1}^{iter}\eta\Delta w^{i}, iter=0,1,\dots,n \tag{6}\]  
\[\Delta w^{i}=\frac{\Delta w^{i}-E(\Delta w^{i})}{\sqrt{\text{Var}\big{(}\Delta w ^{i}\big{)}+\delta}} \tag{7}\]where \(w\) denotes the weight of the model, \(w^{iter}\) represents the weights at the iter-th iteration of training, \(w^{0}\) denotes the model's initialized weights, \(\eta\) stands for the learning rate, and \(\Delta w^{i}\)denotes the normalized gradient. Taking the mean on both sides of Equation (6) and considering \(E(\eta\Delta w^{i})=0\), we obtain Equation (8). After normalization, the mean of the weights remains in a good initial state. \[E\Big{(}w^{iter}\Big{)}=E\Big{(}w^{0}\Big{)} \tag{8}\]  
Solving for the variance of both sides of the Equation (6) yields Equation (9):  
\[\text{Var}\Big{(}w^{iter}\Big{)}=\text{Var}\Big{(}w^{0}\Big{)}+\eta^{2}\,\text{ Var}\left(\sum_{i=1}^{iter}\Delta w^{i}\right)-2\,\text{cov}\left(w^{0},\eta \sum_{i=1}^{iter}\Delta w^{i}\right) \tag{9}\]  
Expanding \(\eta^{2}\,\text{Var}(\sum_{i=1}^{iter}\Delta w^{i})\) yields Equation (10):  
\[\eta^{2}\,\text{Var}\left(\sum_{i=1}^{iter}\Delta w^{i}\right)=\eta^{2}\Bigg{(} \text{Var}\Big{(}\Delta w^{1}\Big{)}+\text{Var}\Bigg{(}\sum_{i=2}^{iter} \Delta w^{i}\Bigg{)}+2\,\text{cov}\left(\Delta w^{1},\sum_{i=2}^{iter}\Delta w ^{i}\right)\Bigg{)} \tag{10}\]  
Since \(\text{Var}(\Delta w^{i})\) is constant at \(1\) after normalization, and so on, the simplification leads to Equation (11). \[\eta^{2}\,\text{Var}\Bigg{(}\sum_{i=1}^{iter}\text{Var}\Big{(}\Delta w^{i} \Big{)}\Bigg{)}=\eta^{2}\Bigg{(}iter+2\sum_{i=1}^{i}\sum_{j=i+1}^{iter}\,\text {cov}\Big{(}\Delta w^{i},\Delta w^{j}\Big{)}\Bigg{)} \tag{11}\]  
Figure 3: The top two plots depict the distribution and heatmap of the unnormalized gradients, while the bottom two plots depict the distribution and heatmap of the normalized gradients, where the red dashed line indicates the mean and the green dashed line indicates positive or negative standard deviation, respectively. By observing the distribution and heatmap of the normalized and unnormalized gradients, it can be observed that the mean of the gradients after normalization is closer to \(0\), and their variance is closer to \(1\). The sparsity of the gradient matrix decreases after normalization. Expanding \(\text{cov}(w^{0},\eta\sum_{i=1}^{iter}\Delta w^{i})\) yields Equation (12). \[\text{cov}\left(w^{0},\eta\sum_{i=1}^{iter}\Delta w^{i}\right)=E\left(\left(w^{0 }-E\Big{(}w^{0}\Big{)}\right)\cdot\left(\left(\sum_{i=1}^{iter}\eta\Delta w^{i} \right)-E\left(\sum_{i=1}^{iter}\eta\Delta w^{i}\right)\right)\right) \tag{12}\]  
Since the values of \(E(\sum_{i=1}^{iter}\eta\Delta w^{i})\) and \(E(\eta\Delta w^{i})\) are 0, the further expansion of \(E(w^{0}\cdot\Delta w^{i})\) leads to Equation (13). \[\text{cov}\left(w^{0},\sum_{i=1}^{iter}\Delta w^{i}\right)=\eta\sum_{i=1}^{iter }\text{cov}\left(w^{0}\cdot\Delta w^{i}\right) \tag{13}\]  
Thus, Equation (9) can be summarized as Equation (14). \[w^{iter}=w^{iter-1}-\Delta w^{iter}=w^{0}-\sum_{i=1}^{iter}\eta\Delta w^{i}, iter=0,1,\dots,n \tag{6}\]0  
By summing up the last two terms of Equation (14), the sum is found to be close to 0 when choosing a suitable value of \(\eta\).