# An Insightful Overview of the Wiener Filter for System Identification
## 3 Useful Definitions  
In this section, many definitions that will be useful in the rest of this paper are presented. Let  
\[\mathbf{h}=\left[ \begin{array}{cccc}h_{1}&h_{2}&\cdots&h_{L}\end{array} \right]^ {T}\]  
be a real-value vector of length \(L\). The \(\ell_{2}\) (or Euclidean) norm can be defined as  
\[\left\|\mathbf{h}\right\|_{2}=\sqrt{\mathbf{h}^{T}\mathbf{h}}=\sqrt{\sum_{l=1 }^{L}h_{l}^{2}}. \tag{6}\]  
The \(\ell_{1}\) norm is given by  
\[\left\|\mathbf{h}\right\|_{1}=\sum_{l=1}^{L}|h_{l}|. \tag{7}\]  
Another very useful vector norm is the \(\ell_{0}\) norm:  
\[\left\|\mathbf{h}\right\|_{0}=\text{card}\{h_{l}|h_{l}\
eq 0\}, \tag{8}\]  
where \(\text{card}\{\cdot\}\) stands for cardinality. In other words, the \(\ell_{0}\) norm of \(\mathbf{h}\) is simply the number of nonzero elements of \(\mathbf{h}\). Let \(\mathbf{H}\) be a rectangular matrix of size \(L_{1}\times L_{2}\) containing real values, where without loss of generality, it is always assumed that \(L_{1}\geq L_{2}\). The Frobenius norm of \(\mathbf{H}\) is defined as  
\[\left\|\mathbf{H}\right\|_{\text{F}}=\sqrt{\text{tr}(\mathbf{H}^{T}\mathbf{H} )}, \tag{9}\]  
where \(\text{tr}(\cdot)\) denotes the trace of a square matrix. The singular value decomposition (SVD) technique [20] is a very powerful tool that can decompose any matrix \(\mathbf{H}\) as  
\[\mathbf{H}=\mathbf{U}_{1}\mathbf{\Sigma}\mathbf{U}_{2}^{T}=\sum_{l=1}^{L_{2} }\sigma_{l}\mathbf{u}_{1,l}\mathbf{u}_{2,l}^{T}, \tag{10}\]  
where \(\mathbf{U}_{1}=\left[ \begin{array}{cccc}\mathbf{u}_{1,1}&\mathbf{u}_{1,2}& \cdots&\mathbf{u}_{1,L_{1}}\end{array} \right]\) and \(\mathbf{U}_{2}=\left[ \begin{array}{cccc}\mathbf{u}_{2,1}&\mathbf{u}_{2,2}& \cdots&\mathbf{u}_{2,L_{2}}\end{array} \right]\) are two orthogonal matrices of sizes \(L_{1}\times L_{1}\) and \(L_{2}\times L_{2}\), respectively, and \(\mathbf{\Sigma}\) is an \(L_{1}\times L_{2}\) rectangular diagonal matrix having on its main diagonal nonnegative real numbers. The columns of \(\mathbf{U}_{1}\) (resp. \(\mathbf{U}_{2}\)) are known as the left-singular (resp. right-singular) vectors of \(\mathbf{H}\), whereas the diagonal entries \(\sigma_{l}\), \(l=1,2,\ldots,L_{2}\) of \(\mathbf{\Sigma}\) are called the singular values of \(\mathbf{H}\) with \(\sigma_{1}\geq\sigma_{2}\geq\cdots\geq\sigma_{L_{2}}\geq 0\). In the case when the rank of \(\mathbf{H}\) is equal to \(P\leq L_{2}\), i.e., \(\text{rank}(\mathbf{H})=P\), then this matrix may be decomposed as  
\[\mathbf{H}=\sum_{p=1}^{p}\sigma_{p}\mathbf{u}_{1,p}\mathbf{u}_{2,p}^{T}. \tag{11}\]  
From (10), it is deduced that the nuclear norm of \(\mathbf{H}\) is  
\[\left\|\mathbf{H}\right\|_{*}=\sum_{l=1}^{L_{2}}\sigma_{l}=\text{tr}\Big{(} \sqrt{\mathbf{H}^{T}\mathbf{H}}\Big{)}. \tag{12}\]Let \(L=L_{1}L_{2}\). \(\mathbf{h}\) can be decomposed as  
\[\mathbf{h}=\Big{[} \begin{array}{cccc}\mathbf{s}_{1}^{T}&\mathbf{s}_{2}^{T}& \cdots&\mathbf{s}_{L_{2}}^{T}\end{array} \Big{]}^{T}, \tag{13}\]  
where \(\mathbf{s}_{l}\), \(l=1,2,\ldots,L_{2}\) are \(L_{2}\) short vectors of length \(L_{1}\) each. As a result, the vector \(\mathbf{h}\) of length \(L_{1}L_{2}\) can be transformed into a rectangular matrix of size \(L_{1}\times L_{2}\):  
\[\mathbf{H}=\big{[} \begin{array}{cccc}\mathbf{s}_{1}&\mathbf{s}_{2}&\cdots& \mathbf{s}_{L_{2}}\end{array} \big{]}. \tag{14}\]  
Therefore, \(\mathbf{h}=\text{vec}(\mathbf{H})\), where by \(\text{vec}(\cdot)\) it is denoted the vectorization operation, consisting of converting a matrix into a vector. Furthermore, \(\mathbf{H}=\text{vec}(\mathbf{h})\), where \(\text{vec}(\cdot)\) is the inverse of the vectorization operation. Let \(\mathbf{h}\) be the system impulse response of length \(L\). A good measure of the sparseness of \(\mathbf{h}\) is [21; 22]  
\[\xi_{12}(\mathbf{h})=\frac{L}{L-\sqrt{L}}\Bigg{(}1-\frac{\|\mathbf{h}\|_{1}}{ \sqrt{L}\|\mathbf{h}\|_{2}}\Bigg{)}. \tag{15}\]  
The closer \(\xi_{12}(\mathbf{h})\) is to 1, the sparser is \(\mathbf{h}\).