# Statistical Analysis on Random Matrices of Echo State Network in PEMFC System's Lifetime Prediction
## 2 Mathematical Backgrounds  
### Echo State Network  
Similar to the structure of RNN, there are three parts in the ESN: input layer, reservoir, and output layer. The different parts are connected by the weight matrix elements of input weight matrix \(\mathbf{W}_{in}\)\(\epsilon\)\(\mathbb{R}^{N\times K}\), internal weight matrix \(\mathbf{W}\)\(\in\)\(\mathbb{R}^{N\times N}\), feedback weight matrix \(\mathbf{W}_{fb}\)\(\in\)\(\mathbb{R}^{N\times L}\), and output weight matrix \(\mathbf{W}_{out}\)\(\in\)\(\mathbb{R}^{L\times(N+K)}\). \(K\) is the dimension of the input signal, \(N\) is the number of neurons in the reservoir, and \(L\) is the dimension of the output signal. The spectral radius \(\rho\) is the maximum eigenvalue of \(\mathbf{W}\). The \(\mathbf{W}_{fb}\) is set to be \(\mathbf{0}\) in this task. At step \(n\), the neuron update in the reservoir is determined by  
\[\mathbf{\tilde{x}}(n)=f(\mathbf{W}_{in}\cdot\mathbf{u}(n)+\mathbf{W}\cdot \mathbf{x}(n)) \tag{1}\]  
where \(\mathbf{\tilde{x}}(n)\) is the neuron update, \(\mathbf{u}(n)\) is the input signal at step \(n\), \(\mathbf{x}(n)\) is the neuron state at step \(n\), and \(f\) is the activation function. Then, the neuron state at step \((n+1)\) can be expressed as  
\[\mathbf{x}(n+1)=(1-\alpha)\cdot\mathbf{x}(n)+\alpha\cdot\mathbf{\tilde{x}}(n) \tag{2}\]  
where \(\alpha\) is the leaking rate of the reservoir. The ESN output is represented by  
\[\mathbf{y}(n+1)=\mathbf{W}_{out}\cdot(\mathbf{x}(n+1);\mathbf{u}(n)) \tag{3}\]  
where \(\mathbf{y}(n+1)\) is the output signal. In the training part, the \(\mathbf{W}_{out}\) is calculated to minimize the root mean square error (RMSE) between the predicted value \(\mathbf{y}^{pre}(n)\) and target value \(\mathbf{y}^{tar}(n)\)  
\[\mathbf{W}_{out}=\text{argmin}\sqrt{\frac{1}{m}\sum_{n=1}^{m}\left(\mathbf{y} ^{pre}(n)-\mathbf{y}^{tar}(n)\right)^{2}} \tag{4}\]  
where \(m\) is the total data points in the training part. When the ridge regression is utilized, the \(\mathbf{W}_{out}\) can be calculated by linear regression  
\[\mathbf{W}_{out}=\mathbf{Y}^{tar}\mathbf{X}^{\mathsf{T}}\Big{(}\mathbf{X} \mathbf{X}^{\mathsf{T}}+\gamma\mathbf{I}\Big{)}^{-1} \tag{5}\]where **X** represents the neuron state and inputs (\(\textbf{x}(n+1)\);**u**(\(n\))), **Y**\({}^{tar}\) is the target value in the training part, \(\gamma\) is the regularization coefficient, and **I** is the identity matrix. In this task, the reservoir neuron number \(N\) is 400, the bounds of leaking rate \(\alpha\), spectral radius \(\rho\), and the regularization coefficient \(\gamma\) are (0, 1), (0, 2), and (1\(\times\)10\({}^{-3}\), 9 \(\times\)10\({}^{-3}\)), respectively. The implementation time of ESN without hyperparameter optimization takes about 20 s in Matlab 2018, Intel-i7 CPU (2.6 GHz), and 16 G RAM environment.  
The **W**\({}_{in}\) and **W** are generated randomly and sparsely. For simplicity, they are usually generated in the same distribution form. Based on the practical guide to applying ESN [28], the uniform distribution and Gaussian distribution are two popular distribution shapes. The uniform distribution is first performed for its continuity of values and boundedness. The distribution interval is [\(-\)0.5, 0.5], and the variance \(D(x)\) is calculated as 1/12. When the Gaussian distribution has the same variance as uniform distribution, the weight matrix follows the normal distribution \(N\) (0, 1/12) [29]. The probability density functions of uniform distribution and Gaussian distribution are shown in Figure 1.