## 5 Convergence of the Learning Procedure  
In this section, we prove the convergence of the learning procedure in a simplified version similar to that described in Section 2.2, but without the discretization with respect to \(r\). Then, we shall discuss additional conditions for the convergence of its discretized version. Let \(X(r),X_{1}(r),\ldots,X_{n}\) be independent, identically distributed random variables with parameter \(r\), with p.d.f. \(f(.,\,r)\in L^{2}([R_{1},\,R_{2}])\), where--for simplicity--we assume that \(n\) is the same for each \(r\). Then, \(f\) has the representation  
\[f(x,\,r)=\sum_{k=1}^{\infty}a_{k}(r)\,v_{k}(x), \tag{19}\]  
which is convergent in the \(L^{2}([S_{1},\,S_{2}])\) norm, where  
\[a_{k}(r)=\int_{S_{1}}^{S_{2}}v_{k}(x)\,f(x,\,r)\,dx=E_{r}(v_{k}(X(r))). \tag{20}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Mean** & **Standard Deviation** \\ \hline [PERSON] & 0.00007 & \(3.7\times 10^{-6}\) \\ Kullback–Leibler & 0.00014 & \(5.9\times 10^{-6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Result of distance calculation between synthetic data and reconstruction. Figure 1: A result for the synthetic problem. Then, we estimate \(a_{k}(r)\)'s as follows:  
\[\hat{a}_{k}(r)=\frac{1}{n}\sum_{i=1}^{n}v_{k}(X_{i}(r)). \tag{21}\]  
**Lemma 1**. : _For every \(r\in[R_{1},\ R_{2}]\)\(\hat{a}_{k}(r)\) is the unbiased estimator of \(a_{k}(r)\)._  
Proof. : Indeed,  
\[E_{r}(\hat{a}_{k}(r))=\frac{1}{n}\sum_{i=1}^{n}\,E_{r}[v_{k}(X_{i}(r))]=a_{k}( r). \tag{22}\]  
As an estimator of \(f(.,\ r)\), we take the truncated version of (19):  
\[\tilde{f}_{n}(x,r)=\sum_{k=1}^{K(n)}\hat{a}_{k}(r)\,v_{k}(x), \tag{23}\]  
where the truncation point \(K\) depends on \(n\). It may also depend on \(r\), but, for the asymptotic analysis, we take this simpler version. The standard way of assessing the distance between \(f\) and its estimator \(\tilde{f}_{n}\) is the mean integrated squared error (\(MISE\)). Notice, however, that, in our case, the MISE additionally depends on \(r\), which is further denoted as \(MISE(r)\). Thus, in order to have a global error, we consider the integrated \(MISE(r)\) that is defined as follows:  
\[IMISE_{n}\,=\,\int_{R_{1}}^{R_{2}}MISE_{n}(r)\,dr. \tag{24}\]  
The \(MISE(r)\) is defined as follows:  
\[MISE_{n}(r)=E_{r}\int_{S_{1}}^{S_{2}}[f(x,\,r)-\tilde{f}_{n}(x,\,r)]^{2}dx, \tag{25}\]  
where the expectation w.r.t. \(f(.,\ r)\) concerns all \(X_{i}(r)\)'s that are present in (23). Using the orthonormality of \(v_{k}\)'s, we obtain:  
\[MISE_{n}(r)=E_{r}\int_{S_{1}}^{S_{2}}[\sum_{k=1}^{K(n)}(a_{k}(r)-\hat{a}_{k}( r))^{2}\,v_{k}(x)-\sum_{k=K(n)+1}^{\infty}a_{k}(r)\,v_{k}(x)]^{2}dx \tag{26}\]  
Continuing (26), we obtain for each \(r\in[R_{1},\ R_{2}]\):  
\[MISE_{n}(r)=\underbrace{E_{r}\left[\sum_{k=1}^{K(n)}(a_{k}(r)-\hat{a}_{k}(r)) ^{2}\right]}_{Var(r,\,K(n))}+\underbrace{\sum_{k=K(n)+1}^{\infty}a_{k}(r)^{2} }_{Bias^{2}(r,\,K(n))}. \tag{27}\]  
It is known that that for squared integrable \(f\) we have: \(\sum_{k=1}^{\infty}a_{k}^{2}(r)<\infty\). Thus, if \(K(n)\to\infty\) when \(n\to\infty\), then for every \(r\in[R_{1},\ R_{2}]\) we obtain:  
\[Bias^{2}(r,\,K(n))=\sum_{k=K(n)+1}^{\infty}a_{k}^{2}(r)\to 0\,\,\,\,\text{when}\,\,\,n \to\infty. \tag{28}\]  
This result is not sufficient to prove the convergence of \(IMISE_{n}\) to zero as \(n\to\infty\). To this end, we need a stronger result, namely an upper bound on \(Bias^{2}(r\,K(n))\), which is independent of \(r\). **Proof. ** By Lemmas 2 and 3, we have uniform (in \(r\)) bounds for the variance and for the squared bias, respectively. Thus,  
Figure 1:0  
Hence,  
Figure 1:1  
which finishes the proof by applying (36). Observe that for \(v_{k}\)'s being the trigonometric system we have \(p=0\) and the r.h.s. of (38) is, roughly, of the form: \(c_{1}\,K(n)/n+c_{2}/K(n)\), \(c1>0\), \(c_{2}>0\), which is minimized by \(K(n)=c_{3}\,\sqrt{n}\) for a certain constant \(c_{3}>0\). This implies that \(IMISE_{n}=O(n^{-1/2})\). **6.