## 3 Image Reconstruction and the MIR Algorithm
### Apache Spark Framework  
Recently, there has been renewed interest in using big data frameworks, such as Apache Hadoop and Spark, for large scale data/image processing. Apache Hadoop based on the Map-reduce programming model provides a relatively structured and easy approach to process big-data on commodity clusters. However, for computations intensive applications, Apache Spark shows better performance due to its in-memory processing. Apache Spark is an open-source framework originally developed by Berkeley's AMP Lab based on the Map-reduce programming paradigm [19]. The Map-reduce is a programming paradigm that allows users to break larger tasks into smaller tasks, run it in parallel, and integrate the output of each task into the final output. The Hadoop Distributed File System (HDFS) provides this capability. It is a distributed file system designed to run on commodity hardware. HDFS let users distribute data to multiple nodes across a cluster and take advantage of the parallel processing of Map-reduce. A typical Map-reduce program contains three classes as shown in Figure 2. The driver class initializes a Spark Context (SC) object, which coordinates the independent set of processes running on a cluster. The SC can be connected to many kinds of cluster managers, such as standalone, YARN [48], or Mesos [49]. The purpose of these cluster managers is to allocate resources to multiple applications. Once created, it gets executors on different nodes, which are processes that run computations and store data for the application. Finally, it sends the application code (Python files in this work) to the executors. Spark programs can be run on Hadoop YARN, Kubernetes, Apache Mesos, and standalonecluster mode, as well as in the cloud. It can access data from various sources including HDFS, Apache Cassandra, HBase, and many others. Apache Spark is different than Hadoop, as it has a distributed memory model and allows to store intermediate results in cashed memory in order to improve the performance. [PERSON] and his team's paper [50] states that, while Spark was still currently a working prototype, the performance results were very encouraging. Even at that time, Spark could outperform ML workloads by a factor of 10. They crash a node to demonstrate that Spark could continue to function with fewer nodes and compared results with the Hadoop implementation. With Hadoop, each iteration would take over 2 min. In the case of Spark, the first iteration took almost 3 min because the data was cached for further iterations, but only took six seconds to complete each of the remaining iterations. PySpark is a Python library that give access to Spark using Python programming language. It used Py4J library to handle communications with Spark. Advantages include easy integration with other languages, such as Java, Scala, and R. It uses the resilient distributed data set (RDD) to speed up the executions by performing in-memory computations. The RDD is the architectural foundation of Spark, which are read-only sets of data, spread over a cluster of different machines, being operated in parallel. Records are split into logical partitions and distributed to all machines in the cluster, allowing RDD to provide high-level programming interface by hiding the data partitions. These logical Partitions are the primary units of parallelism. Spark supports two types of operations to work on these partitions: transformations, such as \"map\", which creates new RDDs; and actions, such as \"reduce\", which returns a value to the driver program after performing an operation on the data set. The transformation operation creates new RDDs and, consequently, creates dependency between the old and the new RDDs. In short, Apache Spark is an efficient and general-purpose engine for big data processing. It was initially developed specifically for data science but has evolved to support yet more use  
Figure 2: Overview of Spark framework. The master node contains driver program, which drives the application by creating Spark context object.