## 3 Algorithm and Platform
### Spark-Based Platform  
It is necessary to take a few words for Map-Reduce before introducing Spark. Map-Reduce is the basic open-source, scalable and fault-tolerant performance model for distributed processing. The core of its performance functionality is conducted through two main functions: (1) Map, which is responsible for loading the data into a powerful system and defining a set of keys, uses a key-value pair (\(k1\), \(v1\)) as the input and generates intermediate key-value pairs [(\(k2\), \(v2\))]; and (2) Reduce, which collects the built set of key-based data from the map function and distributes or reduces the whole implementation through a multi-core HPC platform, merges all the key-value pairs (\(k2\), [\(v2\)]) with the same key into output key-value pairs [(\(k3\), \(v3\))]. Map-Reduce has many drawbacks, especially in disk I/O processing and re-loading of data at each Map-Reduce processing function. Thus, it is very difficult to develop an appropriate iterative-computational algorithm for reliable and efficient predictive modeling within the Map-Reduce system. Distinct from the Map-Reduce model on the Hadoop platform, Spark is an in-memory computing solution to reduce disk I/O and also an efficient and stable solver of large-scale datasets with both iterative least squares and maximum likelihood optimizations. The intermediate results generated in the training process of Spark are stored in the memory system on the Spark framework as RDD (resilient distributed datasets) objects, which is the most important highlight in Spark. In the RDD programming paradigm, each RDD object supports two types of operations, _transformation_ and _action_, concretely. Within the whole computation progress, _transformations_ just modify RDDs, and only _actions_ trigger task submission and then calculate the result. _Transformation_ operations include several operations, such as _map_(), _flatMap_(), _filter_(), _mapPartitions_(), _union_(), and _join_(). Then, a new RDD object is produced from each _transformation_ operation. _Action_ operations include operations such as _reduce_(), _collect_(), _count_(), _saveAsHadoopFile_(), and _countBykey_(), which compute a result and call back to the driver program or save it to an external storage system [(26)]. Although the in-memory computations of Spark reduce the disk I/O, developing an efficient and stable machine learning algorithm on Spark is difficult and not straightforward. Therefore, the motivation in this research is to parallelize the PSO-based ESN algorithm on the Spark framework with the RDD programming model. In this work, the training process of the parallelized PSO-based ESN algorithm includes three stages, which are shown in Figure 3. Stage 1 is for data preprocessing and variable initialization, stage 2 trains the ESN model with several iterations, and then stage 3 calculates errors and saves the results. The \(L\) training datasets are trained in a parallel process, and both \(\mathbf{W}_{i}^{o}\) and \(\mathbf{W}_{r}^{o}\) are calculated at the same time. In stage 1, the training data are loaded from HDFS to the memory and stored as a RDD object, namely \(RDD_{Original\_data\_set}\), which is defined to save the original data. Then, the _sortByKey_() function is performed to make sure the original data is ordered by time series and evenly distributed to multiple nodes to avoid data skew. In every slave node \(k\), data is split into two parts for training and testing, called \(RDD_{traink}\) and \(RDD_{testk}\). Typically, 70% of data is used for training and the rest for testing. Next, we assign matrices \(\mathbf{W}_{i}^{r}\), \(\mathbf{W}_{r}^{r}\), \(\mathbf{W}_{o}^{r}\) and broadcast them to every node, followed by initializing \(\mathbf{x}_{i}(0)\), \(\mathbf{v}_{i}(0)\), \(\mathbf{P}_{i}\), \(\mathbf{P}_{g}\) and \(\mathbf{P}_{u}\). In stage 2, with \(RDD_{traini}\) and \(RDD_{testi}\) for every slave node \(k\), both \(\mathbf{W}_{i}^{o}\) and \(\mathbf{W}_{r}^{o}\) are calculated in a parallel process.