## 4 Experimental Analysis
### Analysis
#### 4.4.1 Training Time  
We recorded the training time of the four datasets in different lengths and nodes. For convenience, the logarithmic scale was used in the total training time presentation. In the training process (see Figure 3), the iterations process is much more critical, so the time spent on this stage (stage 2) is illustrated strikingly. Figure 6 shows the training time in detail.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Algorithm** & **Configuration** \\ \hline \multirow{5}{*}{**PSO**} & Initiate range of particle: [\(-\)0.5 0.5] \\  & Lower bounds: \(-\)1 \\  & Upper bounds: 1 \\  & Generations: 500 \\  & PopulationSize: 80 \\ \hline \multirow{5}{*}{**ADMM**} & Iterations of ADMM: 1000 \\  & Regularization parameter of ADMM (rho): 0.01 \\ \cline{1-1}  & Relative tolerance value: 10\({}^{-4}\) \\ \cline{1-1}  & Absolute tolerance value: 10\({}^{-4}\) \\ \hline \multirow{2}{*}{**CNN**} & Network structuer: input—6 convolution (kernel 5)—2 sub \\  & sampling—12 convolution (kernel 5)—2 sub sampling—output \\ \hline \hline \end{tabular}
\end{table}
Table 3: Software and hardware configuration.  
Figure 6: _Cont._Several points can be concluded from Figure 6:  
1. For a fixed node number, observed by the length of the dataset, we will find the training time becomes a bit shorter at first (1 \(\times\) 10\({}^{4}\)-1 \(\times\) 10\({}^{5}\)), and then increases rapidly (>3 \(\times\) 10\({}^{5}\)) along with the larger size of the dataset.
2. Observed by the number of nodes, when the dataset is big (1 \(\times\) 10\({}^{7}\)), the training time decreases when the nodes become larger, and the training time is more stable; however, a relatively small dataset (1 \(\times\) 10\({}^{4}\)) shows the opposite trend. Figure 7 shows some details about this issue.
3. More time will be spent on reading data, especially when dataset is huge (>3 \(\times\) 10\({}^{6}\)).
4. In Stage 2 (iteration stage), in small datasets, with the increase of nodes, the time increases slowly. In large datasets, more nodes may save a lot of time. Nevertheless, the effect is gradually fading.  
The relations of the training time between small and large datasets are interesting, but the reason for it is trivial. In a small dataset, i.e., 1 \(\times\) 10\({}^{4}\), with 16 nodes, each node could get 625 records on average. However, it can only get 208 records when 48 nodes are tested. Even discarded elements are not taken into consideration, as the samples are still too few to train. This leads to a relatively longer time convergence. When the dataset is a bit larger, say 1 \(\times\) 10\({}^{5}\), every node can get 2000-6000 records, and sufficient samples are provided. So the training time becomes shorter. When the dataset continues to be large, the calculation burden becomes gradually heavy, and more time would be spent, especially in the iteration stage. At this moment, more nodes may help significantly.  
Table 4 shows the comparative results of the training time between different algorithms on four datasets, with a typical length of data size of (1 \(\times\) 10\({}^{5}\)). With all four datasets, distributed  
Figure 6: Training time of four datasets from top to bottom. (**a1–a4**) Training time of each dataset in different lengths and nodes, which is stacked by three stages. Logarithmic scale is used for convenience. (**b1–b4**) Time spent on iterations (stage 2).  
training algorithms showed greater advantages than the centralized ones, shortening the training time efficiently. Especially in former three datasets, the P-PSO-based algorithm could save more than 90% of the training time. In dataset 4, since the dimension of the input data increases significantly, the huge amount of computation makes the training time of the distributed algorithm increase. Nevertheless, the proposed P-PSO-based ESN algorithm is the most efficient algorithm. Additionally, it should be noted that this result is based on a cluster (six nodes) and not on a single machine.