## 1 Introduction  
As a mimic of the human brain, neural-like architectures have been proposed to learn, memorize, steer and game in a complex world [1]; one of the most famous and widely used structures is the feed-forward neural network [2; 3; 4]. In recent years, convolutional neural networks (CNNs) [5] have led to impressive results due to their ability to automatically learn complex feature representations using convolutional layers. CNNs exploit translational invariance within their structures by extracting features through receptive fields and learning with weight sharing, becoming the state-of-the-art approach in various image recognition and computer vision tasks. Another improvement and a more powerful architecture is the recurrent neural network (RNN), which holds at least one cyclic pathway of feedback connections (Figure 1a). RNN mathematically implements dynamical systems and can in theory approximate (universal approximation property) an arbitrary nonlinear dynamical system with arbitrary precision by constantly adjusting the weights of the connections [1]. However, since relational weight-update algorithms are mostly based on gradient-descend, they unavoidably lead to suboptimal solutions [6]. Despite this disadvantage, the difficult of training RNNs also includes slowly convergence, high complexity [6; 7] and the most well-known problem: the exploding and vanishing gradients [8]. Recently, a new scheme known as reservoir computing (RC), which greatly helped the practical application of RNNs and exceeded traditional fully trained RNNs in many fields [1], has been proposedto solve these problems. The echo state network (ESN, Figure 1b) [9] is the most popular example of RC. When an input sequence is fed into the reservoir, a complex non-linear state compute is aroused and then processed by a simple readout procedure to produce the output. By generating and training RNNs without changing the transient dynamics of the recurrent network, only updating weights of connections from the reservoir to the output, RC avoids the conundrums of gradient-based algorithms such as slow and difficult progress. These advantages allow ESN to be applied in numerous domains, including pattern classification [10; 11], time-series prediction [12; 13; 14], intrusion detection [15], etc. With the rapid development of wireless transmission technology, sensor technology, network communication technology and the speedy growth of smart mobile devices, large amounts of data have been accumulated in almost every aspect of our lives. Moreover, the volume of data is growing rapidly with increasingly complex structures and forms [16]. The Big Data era which is characterized by _volume, velocity and variety_ has arrived [17], and it leads to the training samples' explosive growth in the machine learning (ML) field, including RNN/ESN in this paper. In most existing frameworks, the learning and training of RNN/ESN are processed in a centralized way [6; 8; 12; 13; 14; 18; 19]. However, in the era of Big Data, centralized data storage becomes technologically unsuitable [20]; as a result, solutions relying on centralized training in ML could face challenges such as single point of failure, communication bottlenecks, training consistency, etc. Meanwhile, in multiple real-world applications, large-scale training data is not possible to store in a centralized location, but large amounts of it are dispersed throughout a cluster of interconnected nodes (e.g., wireless sensor networks, peer-to-peer networks) [21; 22; 23]. Thus, a decentralized training algorithm for RNN/ESN, at any rate, would be an invaluable tool. In this paper, we propose a distributed training algorithm and platform for ESN. The optimization algorithm in our platform is the well-known particle swarm optimization (PSO) technique [24], which is good at solving optimization problems in dynamic environments. Further, a parallel PSO (P-PSO) algorithm is presented for distributed ESN.