## 1 Introduction  
Recurrent neural networks (RNNs) represent a consolidated computational abstraction for learning with variable-length time series data [1]. As a simplified paradigm of RNN, the echo state network [2; 3] (ESN) provides a prominent reduction in the computational cost compared to other paradigms of RNNs (e.g., [4] (LSTM), [5] (GRU)), for which the hidden layer of ESN is constructed by a randomly generated reservoir instead of independent neurons and the output weights of ESN are trained by a simple linear regression rather than a backpropagation algorithm. Thus, ESN has a successful application in various time series prediction problems (e.g., [6; 7; 8; 9]). Deep neural networks [10] (DNNs) have the potential to learn data representations at various levels of abstraction and are being increasingly stressed in the machine learning community. Recently, a layer-stacked ESN model named deep echo state network (DeepESN) was established and investigated, theoretically and experimentally, by [PERSON] et al: the inherent characterization of the system dynamics developed at the different layers of DeepESN is experimentally analyzed in [11] and theoretically explained in [12]; A theoretical foundation for the study of DeepESN from a dynamical system point of is introduced in [13]; further details on the analysis and advancements of DeepESN could be found in [14]. As an interactional model of RNN and DNN, investigations of DeepESN are of significant importance in both areas. On the one hand, DeepESN expands our knowledge on how the information with memory attracted by RNN is extracted by hierarchical neural networks; on the other hand, DeepESN helps us better understand how the abstract intrinsic representations of time series extracted by DNN are recalledin reservoirs. Furthermore, DeepESN has richer nonlinear representation capacity, less computational complexity, and better predictive performance than single-layer ESN [15]. Optimizing the structure of neural networks remains a common task in artificial neural networks, and the same question of how many neurons should be used in each layer must be stressed in all types of neural networks. If the neurons are too few, the architecture does not satisfy the error demand by learning from the data, whereas if there are too many neurons, learning leads to the well-known overfitting problem [16]. As far as we know, minimal research has been carried out on optimizing the architecture of DeepESN. For research that has already been carried out on DeepESN, the same number is commonly assigned in each layer, which is acceptable but not optimal. In this paper, our aim is to solve the problem of choosing the optimized size of DeepESN, especially the number of neurons in different layers. In 2014, a sensitive iterative pruning algorithm (SIPA) was proposed by [PERSON] and [PERSON] [17] to optimize the simple cycle reservoir network (SCRN). The algorithm was used to prune the least sensitive neurons one by one according to the sensitivity analysis, and the results showed that the SIPA method can optimize the structure and improve the generalization performance of the SCRN; meanwhile, pruning out redundant neurons could contribute to reducing the calculation and improving the computing efficiency of the network. Inspired by these advantages of SIPA, we wanted to apply a similar iterative pruning approach to DeepESN. However, SCRN is a kind of minimal-complexity ESN with a simple cycle topology, and the topology of DeepESN is much more complex than SCRN. Pruning a neuron in the network will raise perturbations on adjacent neurons, resulting in unstable network performance; in SIPA, the perturbations could be eliminated by adjusting the input weights into the perturbed neurons to minimize the distance of its input signal before and after pruning.