# Kernel Identification of Non-Linear Systems with General Structure
## 3 Non-Parametric Regression  
### General Overview  
Let us introduce the \(s\)-dimensional input regressor  
\[\mathbf{u}_{k}^{\left(s\right)}=\left(u_{k},u_{k-1},...,u_{k-\left(s-1\right) }\right)^{T}, \tag{13}\]  
and the regression function  
\[R_{s}\left(\mathbf{x}^{\left(s\right)}\right)=E\left(y_{k}|\mathbf{u}_{k}^{ \left(s\right)}=\mathbf{x}^{\left(s\right)}\right), \tag{14}\]  
with the argument vector  
\[\mathbf{x}^{\left(s\right)}=\left(x_{0},x_{1},...,x_{s-1}\right)^{T}. \tag{15}\]  
In particular, for \(s=1\) we get  
\[R_{1}\left(x_{0}\right)=E\left(y_{k}|u_{k}=x_{0}\right), \tag{16}\]  
and for \(s=2\)  
\[R_{2}(x_{0},x_{1})=E\left(y_{k}|u_{k}=x_{0}\cap u_{k-1}=x_{1}\right). \tag{17}\]  
The non-parametric kernel estimate ([20; 21; 22; 23; 33]) of \(R_{s}\left(\mathbf{x}^{\left(s\right)}\right)\) has the following form  
Figure 2: Wiener System. \[\widehat{R}_{s}\left(\mathbf{x}^{(s)}\right)=\frac{\sum_{k=1}^{N}y_{k}K\left( \frac{\left\|\mathbf{u}_{k}^{(s)}-\mathbf{x}^{(s)}\right\|}{h}\right)}{\sum_{k= 1}^{N}K\left(\frac{\left\|\mathbf{u}_{k}^{(s)}-\mathbf{x}^{(s)}\right\|}{h} \right)}, \tag{18}\]  
where \(\left\|\cdot\right\|\) denotes Euclidean norm, \(K\left(\right)\) plays the role of kernel function, e.g.,  
\[K\left(v\right)=\left\{ \begin{array}{l}1,\,\text{as}\,\left|v\right|\leq 1\\ 0,\,\text{as}\,\left|v\right|>1\end{array} \right., \tag{19}\]  
and \(h\) is a bandwidth parameter, responsible for the balance between bias and variance of the estimate. The class of possible kernels can obviously be generalized. Nevertheless, previous experiences shows that the kind of kernel function used for estimation is of secondary importance, whereas behavior of \(h(N)\) with respect to \(N\) is fundamental. We limited the presentation to Parzen (window) kernel for clarity of exposition. It fulfills all general assumptions made for kernels, i.e., it is even, non-negative and square integrable. The system (1) is thus approximated by the model  
\[\widehat{R}_{s}\left(u_{k},u_{k-1},...,u_{k-(s-1)}\right), \tag{20}\]  
and \(s\) can be interpreted as its complexity. Obviously, both \(h=h(N)\) and \(s=s(N)\) need to be set depending on the number of measurements \(N\). Observing that  
\[\mathcal{F}\left(\{u_{k-i}\}_{i=0}^{\infty}\right)=R_{\infty}\left(\mathbf{u} _{k}^{(\infty)}\right), \tag{21}\]  
the mean squared error of the model \(\widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right)\) can be expressed as follows  
\[MSE\left(\widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right)\right)=E\left( \widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right)-\mathcal{F}\left(\{u_{k-i} \}_{i=0}^{\infty}\right)\right)^{2}=E\left(\widehat{R}_{s}\left(\mathbf{u}_{ k}^{(s)}\right)-R_{\infty}\left(\mathbf{u}_{k}^{(\infty)}\right)\right)^{2},\]  
and introducing the true finite-dimensional regression function \(R_{s}\left(\mathbf{u}_{k}^{(s)}\right)\) we get  
\[MSE\left(\widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right)\right)=E\left\{ \left(\widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right)-R_{s}\left(\mathbf{u} _{k}^{(s)}\right)\right)+\left(R_{s}\left(\mathbf{u}_{k}^{(s)}\right)-R_{ \infty}\left(\mathbf{u}_{k}^{(\infty)}\right)\right)\right\}^{2}=\] \[E\left\{\widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right)-R_{s} \left(\mathbf{u}_{k}^{(s)}\right)\right\}^{2}+E\left\{R_{s}\left(\mathbf{u}_{ k}^{(s)}\right)-R_{\infty}\left(\mathbf{u}_{k}^{(\infty)}\right)\right\}^{2}+\] \[+2E\left\{\left(\widehat{R}_{s}\left(\mathbf{u}_{k}^{(s)}\right) -R_{s}\left(\mathbf{u}_{k}^{(s)}\right)\right)\left(R_{s}\left(\mathbf{u}_{ k}^{(s)}\right)-R_{\infty}\left(\mathbf{u}_{k}^{(\infty)}\right)\right)\right\}.\]  
Since both \(E\left\{R_{s}\left(\mathbf{u}_{k}^{(s)}\right)-R_{\infty}\left(\mathbf{u}_{ k}^{(\infty)}\right)\right\}^{2}\to 0\) as \(N\rightarrow\infty\) and \(\left(R_{s}\left(\mathbf{u}_{k}^{(s)}\right)-R_{\infty}\left(\mathbf{u}_{k}^{( \infty)}\right)\right)\to 0\) as \(N\rightarrow\infty\), these components can be set arbitrarily small by using appropriate scale \(s\).