# On the Redundancy in the Rank of Neural Network Parameters and Its Controllability
## 4 Proposed Method
### Training Process  
Applying the proposed method to a neural network is simple and straightforward. The detailed training process is shown in Algorithm 1. Given a network with \(N\) weight matrices \(\mathbf{W}_{1},\mathbf{W}_{2},\cdots,\mathbf{W}_{N}\), each matrix should be replaced with the initial factorized form \(\mathbf{P}_{i}\mathbf{W}^{\prime}_{i}\mathbf{Q}_{i}\), where \(\mathbf{W}^{\prime}_{i}\in\mathbb{R}^{m_{i}^{\prime}\times n_{i}^{\prime}}\) is initialized with the values of \(\mathbf{W}_{i}\), and \(\mathbf{P}_{i}\in\mathbb{R}^{m_{i}^{\prime}\times m_{i}^{\prime}}\), \(\mathbf{Q}_{i}\in\mathbb{R}^{n_{i}^{\prime}\times n_{i}}\) are initialized with an identity matrix. Then the total RaRe loss for this network is calculated as follows:  
\[\mathcal{L}_{RaRe}=\frac{1}{2c}\sum_{i=1}^{N}m_{i}^{\prime}n_{i}^{\prime}[RaRe (\mathbf{W}^{\prime}_{i})+RaRe(\mathbf{W}^{\prime\top}_{i})],\ c=\sum_{i=1}^{N }m_{i}^{\prime}n_{i}^{\prime}. \tag{16}\]  
```
Input : neural network with parameters \(\theta\), set of 2-dimensional matrices \(\mathcal{W}\subseteq\theta\) Output : trained neural network parameters \(\theta\)
1 foreach\(\mathbf{W}_{k}\in\mathcal{W}\)do//W\({}_{k}\)is a\(m\times n\)matrix. \(\mathbf{P}_{k}\leftarrow\mathbf{I}_{m}\), \(\mathbf{Q}_{k}\leftarrow\mathbf{I}_{n}\), \(\mathbf{W}^{\prime}_{k}\leftarrow\mathbf{W}_{k}\)
2 end foreachwhilenot convergeddo
3 Update \(\theta\) towards minimizing \(\mathcal{L}\)
4 foreach\(\mathbf{W}_{k}\in\mathcal{W}\)do\(i,j\leftarrow\underset{i,j}{\arg\max}\|\mathcal{D}_{cos}(i,j,\mathbf{W}^{ \prime}_{k})\|\)
5 if\(1-|\mathcal{D}_{cos}(i,j,\mathbf{W}^{\prime}_{k})|<\tau_{cos}\)and\(\mathcal{D}_{euc}(i,j,\mathbf{W}^{\prime}_{k})<\tau_{euc}\)then
6 Update \(\mathbf{W}^{\prime}_{k},\mathbf{Q}_{k}\) via (8), (9), and (11)
7 end if
8 Update \(\mathbf{W}^{\prime}_{k},\mathbf{P}_{k}\) by performing line 7-10 on the rows of \(\mathbf{W}^{\prime}_{k}\)
9\(\mathbf{W}_{k}\leftarrow\mathbf{P}_{k}\mathbf{W}^{\prime}_{k}\mathbf{Q}_{k}\)
10 end foreach
11 end foreach
12 return\(\theta\)
```  
**Algorithm 1**Training a neural network with Dynamic Low-Rank Approximation.  
The weighting term \(m_{i}^{\prime}n_{i}^{\prime}\) is introduced to encourage more reduction in larger parameters. As mentioned in the earlier section, the L2 decay loss is calculated as follows:  
\[\mathcal{L}_{L2}=\frac{1}{N}\sum_{i=1}^{N}\bigl{\|}\mathbf{W}^{\prime}_{i} \bigr{\|}_{2}^{2}. \tag{17}\]  
These two loss functions are optimized in conjunction with the task specific loss \(\mathcal{L}_{task}\):  
\[\mathcal{L}=\mathcal{L}_{task}+\lambda_{1}\mathcal{L}_{RaRe}+\lambda_{2} \mathcal{L}_{L2}. \tag{18}\]  
After each parameter update, the pair of vectors that satisfies (13) is merged using the method described by (8), (9), (11).  
Here, we assume that \(\mathcal{L}\) is optimized using a gradient-based method, which requires \(\mathcal{L}_{task}\) to be differentiable w.r.t. the input of the neural network in use. However, this requirement puts minimal restrictions on the type of neural network that can be used, since the use of differentiable neural architecture and loss function (e.g., cross-entropy loss) is the de facto standard in current practice. This setting also mitigates the convergence issue that the nonconvexity of \(\mathcal{L}_{RaRe}\) might carry. It has been suggested that gradient descent is reasonably effective at optimizing nonconvex functions [45; 46], and in some cases the benefits of using nonconvex functions outweigh the absence of convergence guarantee [47]. The empirical results of this study also suggest that \(\mathcal{L}_{RaRe}\), despite its nonconvexity, improves the stability of the training process (Section 5.2).