# An Insightful Overview of the Wiener Filter for System Identification
## 4 Wiener Filter with Different Kinds of Regularization  
The Wiener filter can definitely be improved with an appropriate regularization, which should depend on the properties of the system that needs to be identified. In this section, the most useful regularization techniques are exposed. The most well-known regularization approach is the so-called Tikhonov regularization, which is based on the \(\ell_{2}\) norm. The Wiener problem is now  
\[\min_{\mathbf{\widehat{h}}}\Bigg{[}\mathcal{J}\Big{(}\mathbf{\widehat{h}} \Big{)}+\delta\Big{\|}\mathbf{\widehat{h}}\Big{\|}_{2}^{2}\Bigg{]}, \tag{18}\]  
where \(\delta\geq 0\) is the regularization parameter, which controls the tradeoff between the fidelity of the original data and large values of the coefficients of the filter. The optimization problem in (18) has obviously a closed-form solution, which is  
\[\mathbf{\widehat{h}}_{\text{W},\delta}=\big{(}\mathbf{R}+\delta\mathbf{I}_{L }\big{)}^{-1}\mathbf{p}, \tag{19}\]  
where \(\mathbf{I}_{L}\) is the \(L\times L\) identity matrix. The filter \(\mathbf{\widehat{h}}_{\text{W},\delta}\) is much more reliable than \(\mathbf{\widehat{h}}_{\text{W}}\), especially when \(\mathbf{R}\) is ill conditioned, and \(\text{SNR}(\mathbf{h})\) is low. If the system is sparse, the following optimization problem is more appropriate [24]:  
\[\min_{\hat{\mathbf{h}}}\Bigl{[}\mathcal{J}\left(\widehat{\mathbf{h}}\right)+2 \delta\Bigl{\|}\widehat{\mathbf{h}}\Bigr{\|}_{0}\Bigr{]}. \tag{20}\]  
This problem finds the sparsest solution, but unfortunately, it is NP-hard [24]. Clearly, a large value of \(\delta\) leads to a sparse filter. Using the fact that the convex envelop of \(\Bigl{\|}\widehat{\mathbf{h}}\Bigr{\|}_{0}\) is \(\Bigl{\|}\widehat{\mathbf{h}}\Bigr{\|}_{1},\Bigl{\|}\widehat{\mathbf{h}} \Bigr{\|}_{0}\) can be replaced by \(\Bigl{\|}\widehat{\mathbf{h}}\Bigr{\|}_{1}\) in (20) to obtain  
\[\min_{\hat{\mathbf{h}}}\Bigl{[}\mathcal{J}\left(\widehat{\mathbf{h}}\right)+ 2\delta\Bigl{\|}\widehat{\mathbf{h}}\Bigr{\|}_{1}\Bigr{]}, \tag{21}\]  
which is now a convex problem [25] and is roughly equivalent to (20). The optimization problem in (21) does not have a closed-form solution, but many iterative/adaptive algorithms exist in the literature that converge to the optimal solution, including proportionate-type adaptive filters [22]. The main advantage of (21) is that even if \(\mathbf{R}\) is very ill conditioned, or its estimate is performed with a limited number of observations, if the system is sparse, then a better solution than the conventional Wiener filter can be expected. Taking the gradient of \(\mathcal{J}\left(\widehat{\mathbf{h}}\right)+2\delta\Bigl{\|}\widehat{ \mathbf{h}}\Bigr{\|}_{1}\) with respect to \(\widehat{\mathbf{h}}\) and equating the result to zero, it can be found that  
\[-\mathbf{p}+\mathbf{R}\widehat{\mathbf{h}}+\delta\mathbf{D}^{-1}\left( \widehat{\mathbf{h}}\right)\widehat{\mathbf{h}}=\mathbf{0}, \tag{22}\]  
where  
\[\mathbf{D}\Bigl{(}\widehat{\mathbf{h}}\Bigr{)}=\text{diag}\Bigl{(}\bigl{|} \widehat{h}_{1}\bigr{|},\bigl{|}\widehat{h}_{2}\bigr{|},\ldots,\bigl{|} \widehat{h}_{L}\bigr{|}\Bigr{)} \tag{23}\]  
is a diagonal matrix and \(\widehat{h}_{l},\ l=1,2,\ldots,L\) are the elements of \(\widehat{\mathbf{h}}\). As a result, (22) can be rewritten as  
\[\widehat{\mathbf{h}}=\Bigl{[}\mathbf{D}\Bigl{(}\widehat{\mathbf{h}}\Bigr{)} \mathbf{R}+\delta\mathbf{I}_{L}\Bigr{]}^{-1}\mathbf{D}\Bigl{(}\widehat{ \mathbf{h}}\Bigr{)}\mathbf{p}. \tag{24}\]  
From the previous expression, it is possible to derive a simple iterative algorithm. At iteration 0, one may take  
\[\widehat{\mathbf{h}}^{(0)}=\widehat{\mathbf{h}}_{\text{W},\delta}. \tag{25}\]  
Then, at iteration \(i\), it can be written  
\[\widehat{\mathbf{h}}^{(i)}=\Bigl{[}\mathbf{D}\Bigl{(}\widehat{\mathbf{h}}^{(i -1)}\Bigr{)}\mathbf{R}+\delta_{1}\mathbf{I}_{L}\Bigr{]}^{-1}\mathbf{D}\Bigl{(} \widehat{\mathbf{h}}^{(i-1)}\Bigr{)}\mathbf{p}, \tag{26}\]  
where \(\delta_{1}>0\) is a regularization parameter that is not necessarily equal to \(\delta\).