## 5 Experimental Evaluation  
Using the PySpark based Map-reduce programming model, the parallel MIR algorithm for head imaging big data is implemented on Eddie. Eddie (see www.ecdf.ed.ac.uk) is the University of Edinburgh's research computing cluster. It has 7000 CPU cores. Each server (compute node) has up to 3 TB of RAM. It runs Open Grid schedule and Scientific Linux 7.x. An overview of Eddie architecture is shown in Figure 4.  
The login node has limited memory and lets users submit job scripts or start an interactive session. All these nodes are connected with a 10 Gbps Ethernet network. The Open Grid Scheduler is responsible for scheduling the jobs. All nodes have access to shared memory called Eddie distribute  
Figure 4: Overview of Eddie architecture. Login nodes lets users start an interactive session or submit batch jobs. The submitted jobs run on worker nodes connected through 10 Gbps Ethernet network depicted by double lines. file system. The user either submits batch jobs from login nodes to the cluster or starts an interactive session by specifying the number of cores required and memory per core. After setting up the environment for PySpark and Python, PySpark was launched with a configuration of having one master and three workers node. Each node has the same configuration as shown in Table 2.  
In order to run the job on the localhost, Spark magic is used. Spark magic is a set of tools used to connect to remote Spark clusters through Livy, a Spark REST server, in Jupyter notebooks, as shown in Figure 5.  
The input data contains 06 antenna and 201 points each making 1206 input points in total for a single output image. The efficiency of the MIR and PMIR algorithm is evaluated based on single-image reconstruction in a homogeneous environment. Performance Analysis of MIR and PMIR  
The efficiency is evaluated using a serial approach and a parallelized approach based on the PySpark Map-reduce programming model. Both approaches are tested on Eddie and GCP. For the performance evaluation on Eddie, an interactive session was started with 04 nodes, each having 04 cores and 16 gigabytes of memory. The cluster was configured based on master-slave architecture, where one node acts as a master and others as workers. The master node reads the input data and location of antennas stored on the Eddie distributed file system and creates RDDs. These RDDs are broadcasted to the worker nodes. The worker nodes calculate subsets of pixel values based on the algorithm discussed previously and return the results to the master node using the Map-reduce programming model. The master node converts the matrix values into an image using simple Python functions and saves the results back to Eddie storage. An example image is shown in Figure 6. \begin{table}
\begin{tabular}{c c} \hline \hline
**Item** & **Value** \\ \hline CPU & Intel® Xeon® Processor E5-2630 v3 (2.4 GHz) \\ Memory & 16 G \\ Operating system & Scientific Linux 7 \\ PySpark version & 2.4.1 \\ JDK version & 1.8.0 \\ Hadoop version & 3.1.1 \\ Python version & 3.4.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental setup (each node). Figure 5: Spark magic provides tools for managing Spark sessions on an external cluster through Livy. Using Spark magic with Jupyter notebook, users can use Spark from their own Jupyter notebook, running on localhost. The serial MIR algorithm implemented in Python programming language was first run three times on the cluster and the average wall time was noted. Then the parallelized version of the same algorithm (keeping the input data and antenna location same) was run with the same configurations and the average wall time was noted. The result is shown in Figure 7 and Table 3. It can be noted that the average wall time decreased from 8.14 s to 285 ms, which means that using PySpark is 28.56 times faster than sequential Python implementation. Figure 6: An example of the output image reconstructed from 1206 backscattered points collected using six antennas.