# A Gradient-Based Particle-Bat Algorithm for Stochastic Configuration Network
## 2 Preliminaries  
### Stochastic Configuration Network  
SCN has been a powerful data modeling tool in recent years. The model is incrementally generated by stochastic configuration algorithms, which performs well in classification and regression problems and has the ability of universal approximation. The structure of SCN is shown in Figure 1, where \(x\) denotes the input data, \(s\) denotes the activation function, and \(f\) denotes the output of network.  
For the dataset \(X=[x^{1},x^{2},...,x^{N}],x^{i}\in R^{d},i=1,2,...,N\), and the corresponding output datasets \(T=[t^{1},t^{2},...,t^{N}]\), \(t^{i}\in R^{m}\). If \(L-1\) nodes have been configured on the network, the outputs of L-th node in the hidden layer is calculated by Formula (1)  
\[s_{L}=h(w_{L}^{T}x_{u}+b_{L}),(L=1,2,...;u=1,2,...,d) \tag{1}\]  
where \(h\) stands for the activation function, \(w_{L}\) and \(b_{L}\) is the weight and bias of the first layer. The weights of the second layer is calculated by the least squares algorithm, as shown in Formula (2).  
\[\beta_{L,q}=\frac{\langle e_{L-1,q},s_{L}\rangle}{\|s_{L}\|^{2}} \tag{2}\]  
\[e_{L-1,q}=t_{q}-f_{q},\ (q=1,2,...,m) \tag{3}\]  
Figure 1: The structure of SCN with a hidden layer.  
where \(e_{L-1,q}\) stands for the \(q\)-th error of the network output when there are already \(L-1\) nodes in the hidden layer. \(f_{q}\) stands for the \(q\)-th output of SCN, which is calculated by Formula (4), and the error of SCN with \(L\)-nodes is shown in Formula (5).  
\[f_{q}=\sum_{i=1}^{L}\beta_{iq}s_{i},(q=1,2,...m) \tag{4}\]  
\[e_{L}=[e_{L,1},...,e_{L,m}]=t-f \tag{5}\]  
\[f=[f_{1},...,f_{m}] \tag{6}\]  
\(w_{L}\) and \(b_{L}\) need satisfy the supervision mechanism (7).  
\[\sum_{q=1}^{m}\langle e_{L-1,q^{\prime}}s_{L}\rangle^{2}\geq bound_{s}^{2} \delta_{L} \tag{7}\]  
\[\delta_{L}=(1-r-\mu_{L})\|e_{L-1}\|^{2} \tag{8}\]  
where \(bound_{s}\) stands for the upper bound of the activation function, \(\mu_{L}\) is a sequence that goes to zero, and \(r\) is a regularization parameter close to 1. Hidden nodes are added in this sequence until the model reaches the tolerance, that is, when the error between the model outputs and target outputs reaches the preset threshold, the structure, weight, and threshold parameters of the SCN are determined, then the learning process stops. It has been proved mathematically that the model has universal approximation capability, namely \(\lim\limits_{L\rightarrow+\infty}\|e_{L}\|=0\). Principles and Formulas (1)-(8) of SCN are from Ref. [8].  
The specific search algorithm for the input weights and biases of SCN is as follows.  
Step 1. Set the sequence of interval length for searching the parameters,  
\(\lambda=\lambda_{min}:\Delta\lambda:\lambda_{max}\);  
Step 2. Generate a set of random values in the interval \([-\lambda_{min},\lambda_{min}]\), then judge whether it satisfies the inequality constraint or not;  
Step 3. If there are parameters satisfying the inequality constraints, the parameters that could maximize \(e_{L}\) (Formula (9)) are selected to configure the new hidden node. Otherwise, expand the search interval to \([-(\lambda_{min}+\Delta\lambda):(\lambda_{min}+\Delta\lambda)]\), and repeat the above steps until the tolerance is reached [8].  
\[e_{L}=\sum_{q=1}^{m}(\frac{(e_{L-1,q}(X)^{T}s_{L}(X))^{2}}{s_{L}(X)^{T}s_{L}( X)}-(1-r-\mu_{L})e_{L-1,q}(X)^{T}e_{L-1,q}(X)) \tag{9}\]  
\[s_{L}(X)=[s_{L}(w_{L}^{T}x_{1}+b_{L}),s_{L}(w_{L}^{T}x_{2}+b_{L}),...,s_{L}(w_ {L}^{T}x_{N}+b_{L})] \tag{10}\]