# On the Redundancy in the Rank of Neural Network Parameters and Its Controllability
## 4 Proposed Method
### Dynamic Low-Rank Approximation  
When a pair of linearly dependent vectors in a matrix is identified, the size of this matrix can be reduced by removing one of the vectors and representing it using the other one. Given the network layer described by (1) and the columns \(i,j(i<j)\) as the linearly dependent pair in \(\mathbf{W}\), we can create an equivalent function as follows:  
\[\mathbf{x}^{\prime}=\Bigl{[}\cdots,x_{i-1},x_{i+1},\cdots,\frac{(r+1)}{s}x_{j },\cdots\Bigr{]} \tag{7}\]  
\[r=\Sigma\mathbf{w}_{i}/\Sigma\mathbf{w}_{j},\quad s=\sqrt{r^{2}+1} \tag{8}\]  
\[\mathbf{W}^{\prime}=\Bigl{[}\cdots,\mathbf{w}_{i-1}^{\top},\mathbf{w}_{i+1}^{ \top},\cdots,s\mathbf{w}_{j}^{\top},\cdots\Bigr{]} \tag{9}\]  
\[\mathbf{\hat{y}}=\mathbf{W}^{\prime}\mathbf{x}^{\prime\top}+\mathbf{b}. \tag{10}\]  
This can be easily proved by setting \(\mathbf{w}_{i}=r\mathbf{w}_{j}\). Since the transformation from \(\mathbf{x}\) to \(\mathbf{x}^{\prime}\) can be implemented with a matrix multiplication, the problem becomes finding the matrices \(\mathbf{W}^{\prime}\in\mathbb{R}^{m\times n^{\prime}}\), \(\mathbf{Q}\in\mathbb{R}^{n^{\prime}\times n}\) for the original matrix \(\mathbf{W}\in\mathbb{R}^{m\times n}\), such that \(\mathbf{W}^{\prime}\mathbf{Q}=\mathbf{W}\) where \(n^{\prime}\leq n\). This can be accomplished by initializing \(\mathbf{Q}_{0}\in\mathbb{R}^{n\times n}\) with an identity matrix, and setting the \(j\)th row as the weighted sum of the \(i\)th and \(j\)th row as follows:  
\[\mathbf{Q}=\bigl{[}\cdots,\mathbf{q}_{i-1},\mathbf{q}_{i+1},\cdots,\bigl{(} \mathbf{q}_{j}+\mathbf{q}_{j}\times r\bigr{)}/s,\cdots\bigr{]}. \tag{11}\]Then, (10) can be rewritten as:  
\[\hat{\mathbf{y}}=\mathbf{W}^{\prime}\mathbf{Q}\mathbf{x}^{\top}+\mathbf{b}. \tag{12}\]  
The reduction process in (8), (9), (11), and (12) can be repeatedly applied whenever a linearly dependent pair of columns is identified. Similarly, it is possible to fuse pairs of linearly dependent rows. Once linearly dependent rows has been identified after minimizing \(RaRe(\mathbf{W}^{\top})\), these rows can be fused by introducing a matrix \(\mathbf{P}\in\mathbb{R}^{m\times m^{\prime}}\) that satisfies \(\mathbf{PW}^{\prime}\mathbf{Q}=\mathbf{W}\) where \(m^{\prime}\leq m\). The process of finding \(\mathbf{P}\) and \(\mathbf{W}^{\prime}\) is almost identical to (9) and (11), except the column operations become row operations and vice versa. When reducing both rows and columns, the vector fusing process along one axis can alter the pairwise angular distances along the other axis. The \(r\) and \(s\) terms in (9) and (11) are introduced to prevent this issue. When fusing two vectors with this method, one must choose the vector that is merged to the other one. Since \(s\) is always greater than 1, the remaining vector's magnitude increases. If we choose to keep the vector with greater magnitude, it is more likely to be kept after successive merges and become extremely large. This led to unstable training, so we choose to keep the shorter vector. Even with this precaution, the average magnitude of the vectors tends to increase after multiple merges. Therefore, we utilize L2 weight decay to counter this effect. Another factor to consider when shrinking the weight parameter is the choice of axis to reduce. Since the upper bound of a \(m\times n\) matrix's rank is \(\min(m,n)\), the longer axis is likely to have more redundancy. This hypothesis was supported by preliminary experiments, and we design this method to reduce along the longer axis only, keeping its shape close to a square. Theoretically, this is not an approximation method, since it produces an equivalent function given that the merged pair of vectors are parallel or opposite to each other. However, in practice we find it neither necessary nor efficient to impose such a strict condition. Thus, we approximate the original function by allowing a slight angle between the two vectors. More formally, we fuse \(\mathbf{w}_{i}\) and \(\mathbf{w}_{j}\) if \(1-|D_{cos}(i,j)|<\tau_{cos}\).