# FnmOS-ELM: A Flexible Neural Network Mixed Online Sequential Elm
## 3 Proposal and Implementation of FnmOS-ELM  
### Proposal of Fnmms-ELM  
The ELM randomly selects the weights of the input layer and the biases of the hidden layer and uses the Moore-Penrose generalized inverse [31] to calculate the output weights. On this basis,OS-ELM uses online learning to update the output weights with one-by-one or chunk-by-chunk data samples. We propose the fnnmOS-ELM based on the OS-ELM.  
For \(N_{0}\) samples \((X,T_{0})\), \(X\) is the input vector and \(T_{0}\) is the target vector, which is given by  
\[\mathrm{T}_{0}=\left[\begin{array}{c}\mathrm{t}_{1}^{\mathrm{T}}\\ \vdots\\ \mathrm{t}_{N_{0}}^{\mathrm{T}}\end{array}\right]_{\mathrm{N}_{0}\times \mathrm{L}}, \tag{1}\]  
where \(L\) is the label dimension of a single data point. Typically, \(L\) usually represents a category in a classification problem.  
Let \(\mathrm{Net}_{n}\) be a pre-trained network, where \(n\) indexes over the network layers. Here, we consider \(Y_{i}\) to be the \(i\)-th layer output of the neural network, where \(i=0,1,\ldots,n\). The data used for online learning is a batch or a sample. In this paper, we use a batch. \(D\) is the batch size, and the output \(Y_{i}\) is divided into \(Z\) batches, i.e., \(Y_{iz}\) For the first batch, \(Z=0\), and  
\[Y_{i0}=\left[\begin{array}{ccc}y_{00}&\ldots&y_{0M}\\ \vdots&\ddots&\vdots\\ y_{D0}&\ldots&y_{DM}\end{array}\right]_{D\times M}, \tag{2}\]  
where \(M\) is the output dimension of a sample in the \(i\)-th neural network. With random weights \(w_{i}\) and biases \(b_{i}\),  
\[W_{i}=\left[\begin{array}{ccc}w_{00}&\cdots&w_{0S}\\ \vdots&\ddots&\vdots\\ w_{M0}&\cdots&w_{MS}\end{array}\right]_{M\times S}, \tag{3}\]  
\[b_{i}=\left[\begin{array}{c}b_{i0}\\ \vdots\\ b_{iD}\end{array}\right]_{D\times 1}, \tag{4}\]  
where \(S\) is the number of fnnmOS-ELM hidden layer nodes. The output matrix of the hidden layer is \(H_{0}\), which can be written as  
\[H_{0}=g(Y_{i0}\cdot W_{i}+b_{i}), \tag{5}\]  
where \(g()\) is the activation function. We formulate \(H_{0}\) as  
\[H_{0}=\left[\begin{array}{ccc}h_{00}&\cdots&h_{0S}\\ \vdots&\ddots&\vdots\\ h_{D0}&\cdots&h_{DS}\end{array}\right]_{D\times S}. \tag{6}\]  
Then, the output matrix \(\beta^{(0)}\) is  
\[\beta^{(0)}=\left[\begin{array}{ccc}\beta_{00}&\cdots&\beta_{0L}\\ \vdots&\ddots&\vdots\\ \beta_{S0}&\cdots&\beta_{SL}\end{array}\right]_{S\times L}, \tag{7}\]  
When \(\left\|H_{0}\beta-T_{0}\right\|\) is a minimum, \(\beta^{(0)}\) can be written as  
\[\beta^{(0)}=K_{0}^{-1}H_{0}^{T}T_{0}, \tag{8}\]where \(K_{0}=H_{0}^{T}H_{0}\). When the next \(Y_{i1}\) batch arrives (\(Z=1\)), the problem is reformed as minimizing  
\[\left\|\left[\begin{array}{c}H_{0}\\ H_{1}\end{array}\right]\beta^{(1)}-\left[\begin{array}{c}T_{0}\\ T_{1}\end{array}\right]\right\|, \tag{9}\]  
According to the results of [PERSON] et al. [15], \(\beta^{(1)}\) can be expressed as \(\beta^{(0)}\). Thus, the recursive formula of online learning can be written as  
\[\begin{array}{c}P_{k+1}=P_{k}-P_{k}H_{k+1}^{T}\big{(}I+H_{k+1}P_{k}H_{k+1}^ {T}\big{)}^{-1}H_{k+1}P_{k}\\ \beta^{(k+1)}=\beta^{(k)}+P_{k+1}H_{k+1}^{T}\big{(}T_{k+1}-H_{k+1}\beta^{(k)} \big{)}\end{array}\,. \tag{10}\]  
The recursive formula of fnnmOS-ELM online learning is the same as the OS-ELM recursion formula. The difference is that the input data of fnnmOS-ELM is no longer original data, but the output of a layer of the CNNs and RNNs can be transformed by simple dimension or block processing and be send to the fnnmOS-ELM. The purpose of this method is to extract some features from the original data by traditional networks, and to achieve fast training and inference by OS-ELM, so our method combines the advantages of both networks and OS-ELM.