# Time Series Prediction Based on Adaptive Weight Online Sequential Extreme Learning Machine
## 2 Preliminaries
### Online Sequential Extreme Learning Machine  
In many practical instances, the sequential training samples \(\aleph=\{(\mathbf{x}_{i},\mathbf{t}_{i})|\mathbf{x}_{i}\in\Re^{n},\mathbf{t}_ {i}\in\Re^{m},i=1,2,\ldots\}\) are produced chunk by chunk, and the chunk size may be fixed or various. Assume that the \(j\)th data chunk has \(N_{j}\) samples, then the chunk at time k is able to be represented as \(\aleph_{k}=\{(\mathbf{x}_{i},\mathbf{t}_{i})\}^{\sum\limits_{j=0}^{k}N_{j}}_{ j=(\sum\limits_{j=0}^{k-1}N_{j})+1}\). The initialization of the learning process is carried out according to a small data chunk \(\aleph_{0}=\{(\mathbf{x}_{i},\mathbf{t}_{i})\}_{i=1}^{N_{0}}\), where \(N_{0}\) is the samples number of data chunk \(\aleph_{0}\), and \(N_{0}\) ought to be equal to or greater than \(L\). With the hidden-layer parameters \((\boldsymbol{\omega}_{i},b_{i}),i=1,2,\ldots,L\) assigned into random values, the initial \(\mathbf{H}_{0}\) can be computed as the following equation:  
\[\mathbf{H}_{0}:=\mathbf{H}\big{(}\boldsymbol{\omega}_{1},\ldots,\boldsymbol{ \omega}_{L},b_{1},\ldots,b_{L},\mathbf{x}_{1},\ldots,\mathbf{x}_{N_{0}}\big{)} \tag{9}\]  
and then the initial \(\beta_{0}\) is able to be obtained according to ELM as follows:  
\[\beta_{0}=\mathbf{P}_{0}\mathbf{H}_{0}^{T}\mathbf{T}_{0} \tag{10}\]  
where \(\mathbf{P}_{0}=\left(\frac{I}{C}+\mathbf{H}_{0}^{T}\mathbf{H}_{0}\right)^{-1}\) and \(\mathbf{T}_{0}=\left[\mathbf{t}_{1},\ldots,\mathbf{t}_{N_{0}}\right]^{T}\). The partial hidden-layer output matrixes \(\mathbf{h}_{k+1}\) and the partial output-layer matrixes \(\mathbf{t}_{k+1}\) corresponding to data chunk at time \(k+1\) are respectively defined as:  
\[\mathbf{h}_{k+1}:=\mathbf{H}\bigg{(}\boldsymbol{\omega}_{1},\ldots,\boldsymbol {\omega}_{L},b_{1},\ldots,b_{L},\mathbf{x}_{(\sum\
olimits_{j=0}^{k}N_{j})+ 1},\ldots,\mathbf{x}_{\sum\
olimits_{j=0}^{k+1}N_{j}}\bigg{)} \tag{11}\]  
\[\mathbf{t}_{k+1}:=\left[\mathbf{t}_{(\sum\
olimits_{j=0}^{k}N_{j})+1},\ldots, \mathbf{t}_{\sum\
olimits_{j=0}^{k+1}N_{j}}\right]^{T} \tag{12}\]  
Then \(\mathbf{H}_{k}\) and \(\mathbf{T}_{k}\) can be respectively expressed as:  
\[\mathbf{H}_{k}=\mathbf{H}\bigg{(}\boldsymbol{\omega}_{1},\ldots,\boldsymbol{ \omega}_{L},b_{1},\ldots,b_{L},\mathbf{x}_{1},\ldots,\mathbf{x}_{\sum \
olimits_{j=0}^{k}N_{j}}\bigg{)},\mathbf{T}_{k}=\left[ \begin{array}{c} \mathbf{t}_{1}^{T}\\ \vdots\\ \mathbf{t}_{\sum\
olimits_{j=0}^{k}N_{j}}^{T}\end{array} \right] \tag{13}\]  
and we have  
\[\mathbf{H}_{k+1}=\left[ \begin{array}{c}\mathbf{H}_{k}\\ \mathbf{h}_{k+1}\end{array} \right]\text{, }\mathbf{T}_{k+1}=\left[ \begin{array}{c} \mathbf{T}_{k}\\ \mathbf{t}_{k+1}^{T}\end{array} \right] \tag{14}\]  
The least squares solution of \(\mathbf{H}_{k+1}\beta=\mathbf{T}_{k+1}\) should be the output weight at time \(k+1\), \(\beta_{k+1}\) and it is able to be computed in an iterative way as follows:  
\[\beta_{k+1}=\beta_{k}+\mathbf{P}_{k+1}\mathbf{h}_{k+1}^{T}\Big{(}\mathbf{t}_ {k+1}^{T}-\mathbf{h}_{k+1}\beta_{k}\Big{)} \tag{15}\]  
\[\mathbf{P}_{k+1}=\mathbf{P}_{k}-\mathbf{P}_{k}\mathbf{h}_{k+1}^{T}\Big{(}I+ \mathbf{h}_{k+1}\mathbf{P}_{k}\mathbf{h}_{k+1}^{T}\Big{)}^{-1}\mathbf{h}_{k+1} \mathbf{P}_{k} \tag{16}\]OS-ELM is composed of an initialization phase and sequential learning phase and need not retain all the historic data. In the initialization phase, \(\mathbf{H}_{0}\), \(\mathbf{\beta}_{0}\), \(\mathbf{P}_{0}\), and \(\mathbf{T}_{0}\) are initialized for the use in the sequential learning phase. The samples number of the initialization chunk should be equal to or greater than the hidden nodes number. In the sequential learning phase, the sequential date chunk is commenced on iteratively. Once the training process on the latest coming data chunk is completed, the historic data can be discarded and not used any more. From the derivation of OS-ELM, it is easy to find that OS-ELM and ELM have similar generalization performances.