## 2 Deep Echo State Network
### Neuronal Similarity-Based Iterative Pruning Merging Algorithm to Deep Echo State Network  
Due to the simple topology of SCRN, only one neuron (\(x_{m+1}\)) receives input from the pruned neuron (\(x_{m}\)), and perturbation elimination in SCRN is easy to perform. However, the topology of DeepESN is much more complicated, and more extensive perturbation will be raised by pruning one neuron in DeepESN because, in a highly coupled reservoir, perturbation at any neuron will be diffused to every neuron of the same reservoir. Furthermore, in hierarchically stacked reservoirs, perturbation in the lower layer will be transmitted to every higher layer above, and it is very difficult to eliminate all these perturbations. The merging method is designed to solve this difficulty, and this coincides with [PERSON]'s idea [PERSON] (2018), where two neurons are merged by averaging their input weights. Consider the following ideal scenario: merging two identical twin neurons in the same reservoir will derive a new neuron that is identical to two antecedent twins; this new burned neuron could act as an equivalent substitution for two antecedent twins. Consequently, a neuron is pruned without leading any perturbation through superimposing the output weights of the merged twins as well. This ideal perturbation-free merging has the prerequisite that two identical neurons have to be found in the same reservoir. The more similar the two merged neurons are, the weaker perturbation that will be raised by merging. Neuronal similarity could be assessed by some quantitative relations of the collected training state matrix. Distance and correlation are commonly used to quantify similarity, and four similarity estimation criteria are given in this paper, including the inverse of the Euclidean distance (\(ED\)), [PERSON]'s correlation coefficient (\(PC\)), [PERSON]'s correlation coefficient (\(SC\)) and [PERSON]'s correlation coefficient (\(KC\)) as follows:  
Noting the total number of neurons in all reservoirs \(M=\sum_{i=0}^{L}N_{x}^{l}\), recall the train state matrix  
\[\tilde{\mathbf{X}}_{train}= \begin{bmatrix}\mathbf{x}^{1}(1)&\mathbf{x}^{1}(2 )&\ldots&\mathbf{x}^{1}(L_{train})\\ \mathbf{x}^{2}(1)&\mathbf{x}^{2}(2)&\ldots&\mathbf{x}^{1}(L_{train})\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{x}^{L}(1)&\mathbf{x}^{L}(2)&\ldots&\mathbf{x}^{L}(L_{train})\end{bmatrix}  = \begin{bmatrix}\mathbf{n}_{1}\\ \mathbf{n}_{2}\\ \vdots\\ \mathbf{n}_{M}\end{bmatrix} , \tag{16}\]  
where \(L_{train}\) is the number of training samples, and \(\mathbf{n}_{i}\) is the historical state of the i-th neuron state during training. The similarity of \(\mathbf{n}_{i}\) and \(\mathbf{n}_{j}\) is derived by:  
Figure 3: Weight coefficient diagram of connections of neurons in the reservoir of SCRN before (**a**) and after (**b**) pruning a neuron by SIPA. \[ED(i,j) =\frac{1}{\parallel\mathbf{n}_{i}-\mathbf{n}_{j}\parallel_{2}}, \tag{17}\] \[PC(i,j) =\frac{\sigma_{\mathbf{n}_{i}\mathbf{n}_{j}}}{\sqrt{\sigma_{ \mathbf{n}_{i}\mathbf{n}_{i}}\sigma_{\mathbf{n}_{j}\mathbf{n}_{j}}}},\] (18) \[SC(i,j) =1-\frac{6\sum_{j=1}^{L_{train}}d_{T}^{2}}{L_{train}\Big{(}L_{ train}^{2}-1\Big{)}},\] (19) \[KC(i,j) =\frac{c-d}{\frac{1}{2}L_{train}\big{(}L_{train}-1\big{)}}, \tag{20}\]  
where \(\sigma_{\mathbf{n}_{i}\mathbf{n}_{j}}\) represents the cross-correlation of \(\mathbf{n}_{i}\) and \(\mathbf{n}_{j}\), and \(\sigma_{\mathbf{n}_{i}\mathbf{n}_{j}}\) and \(\sigma_{\mathbf{n}_{j}\mathbf{n}_{j}}\) represent the autocorrelation of \(\mathbf{n}_{i}\) and \(\mathbf{n}_{j}\). \(d_{t}\) is the rank difference of \(\mathbf{n}_{i}(t)\) and \(\mathbf{n}_{j}(t)\), \(c\) is the number of concordant pairs, and \(d\) is the number of discordant pairs in \(\mathbf{n}_{i}\) and \(\mathbf{n}_{j}\). NS-IPMA based on different similarity estimation criteria are named correspondingly; for instance, ES-IPMA means NS-IPMA based on the inverse of Euclidean distance criterion, etc. The main process of NS-IPMA is illustrated in Figure 4, and the NS-IPMA method is carried out in the following steps:  
1.