# On the Redundancy in the Rank of Neural Network Parameters and Its Controllability
## 3 Redundancy in the Rank of Parameters  
A neural network layer can be seen as a mapping from one (input) feature space to another (output) feature space. This can be accomplished by combining several transformations. The most typical setting is to map the input feature space to an intermediate space via an affine or linear transformation and map this space to the output feature space with a nonlinearity. This not only applies to fully-connected networks and LSTMs but also to CNNs since in practice convolution operation is reduced to a batch matrix multiplication for efficiency [37]. We will later discuss how this setting can be extended to other neural architectures. Let \(\mathbf{x}=[x_{1},x_{2},\cdots,x_{n}]\in\mathbb{R}^{n}\) and \(\mathbf{y}=[y_{1},y_{2},\cdots,y_{m}]\in\mathbb{R}^{m}\) be some elements in the input and output feature space of a fully-connected layer, respectively, and \(\mathbf{W}=\left[\mathbf{w}_{1}^{\top},\mathbf{w}_{2}^{\top},\cdots,\mathbf{w }_{n}^{\top}\right]\in\mathbb{R}^{m\times n}\) and \(\mathbf{b}=[b_{1},b_{2},\cdots,b_{m}]\in\mathbb{R}^{m}\) be the trainable parameters of it. Then, the mapping from \(\mathbf{x}\) to \(\mathbf{y}\) using this layer is computed by  
\[\mathbf{\hat{y}}=\sum_{k=1}^{n}x_{k}\mathbf{w}_{k}+\mathbf{b} \tag{1}\]  
\[\mathbf{y}=f(\mathbf{\hat{y}}), \tag{2}\]  
where \(f(\cdot)\) is an element-wise nonlinearity such as the rectifier or sigmoid. Here, \(\mathbf{\hat{y}}\) in (1) can be interpreted as the coordinates of a point in the Cartesian coordinate system, given \(\mathbf{x}\) as the affine coordinates over the affine frame with origin \(\mathbf{b}\) and basis \(\mathbf{W}\) (assuming \(\mathbf{W}\) has full rank). Thus, the intermediate feature space is a linear subspace of \(\mathbb{R}^{m}\) where its dimension is the rank of \(\mathbf{\hat{W}}=\left[\mathbf{W},\mathbf{b}^{\top}\right]\) (i.e., the column space of \(\mathbf{\hat{W}}\)). The key idea is that if \(\mathbf{\hat{W}}\) is rank-deficient, the same mapping can be represented with a smaller \(MLIS_{strict}(\mathbf{\hat{W}})\), where \(MLIS_{strict}(\mathbf{A})\) denotes the maximal linearly independent subset of matrix \(\mathbf{A}\). However, for fully-connected layers in modern neural architectures, \(n\) and \(m\) are often in the order of hundreds or thousands [7; 38]. In such a high dimension, a set of vectors is almost always linearly independent, especially when they are randomly initialized and optimized towards a certain objective. We argue that even in such a case,\(\hat{\mathbf{W}}\) can be _almost_ linearly dependent, where some vectors in \(\hat{\mathbf{W}}\) can be defined as a linear combination of the other vectors with a small error. More formally, let \(\mathbf{a}=[a_{1},a_{2},\cdots,a_{n+1}]\in\mathbb{R}^{n+1}\) be a vector that satisfies  
\[\hat{\mathbf{W}}\mathbf{a}^{\top}=\mathbf{0}^{\top},\quad\mathbf{a}\
eq \mathbf{0}, \tag{3}\]  
where \(\mathbf{0}\) denotes the zero vector. \(\hat{\mathbf{W}}\) is said to be linearly dependent if such \(\mathbf{a}\) exists. We propose to introduce a slack variable \(\boldsymbol{\xi}\in\mathbb{R}^{m}\) such that  
\[\hat{\mathbf{W}}\mathbf{a}^{\top}+\boldsymbol{\xi}^{\top}=\mathbf{0}^{\top}, \quad\mathbf{a}\
eq\mathbf{0}. \tag{4}\]  
Then, we can find the maximal linearly pseudoindependent subset of a matrix under the condition given by (4). We denote this subset by \(MLIS_{soft}(\mathbf{A})\). The amount of difference that occurs when representing the column space of \(\hat{\mathbf{W}}\) using \(MLIS_{soft}(\hat{\mathbf{W}})\) can be controlled by introducing a condition on the slack variable \(\boldsymbol{\xi}\), such as \(\|\boldsymbol{\xi}\|_{2}<\tau\). If the rank of \(MLIS_{soft}(\hat{\mathbf{W}})\) is smaller than \(min(n+1,m)\), which is the upper bound of the rank of \(\hat{\mathbf{W}}\), we can get a lossy mapping from the input feature space to the output feature space with less free parameters. In such case, it can be said that there is redundancy in the rank of \(\hat{\mathbf{W}}\).