# A SqueezeSAR Spatially Adaptive Filtering Algorithm Based on Hadoop Distributed Cluster Environment
## 2 Spatial Adaptive Filtering Algorithm in Hadoop Environment
### Spark-Space Adaptive Filtering Algorithm  
In the field of data computing, frequent I/O operations can cause significant losses in the data-processing performance [44]. Owing to the framework limitation of MapReduce, a MapReduce task can only contain one Map and Reduce call, and after the computation is completed, MapReduce writes the results back to the disk (distributed file system) for the next computation, which involves a large amount of I/O and requires a considerable amount of time [45]. Many studies have shown that the MapReduce-based algorithms suffer a significant performance loss when frequent I/O operations and network communications are performed [46; 47; 48]. Spark solves these problems in several ways. It is mainly based on in-memory computing, which enhances multi-iterative batch processing and ensures a high fault tolerance and scalability, while improving the real-time processing of data in big data environments. As Spark supports in-memory computation and acyclic data flow through a directed acyclic graph (DAG) execution engine, the number of shuffles can be reduced, in most cases, compared with MapReduce. In particular, in the case of computation that does not involve data communication between nodes, Spark can complete the entire algorithm in memory at once, greatly reducing the disk I/O operations. As InSAR is more complex in terms of the data structure and algorithm, one Map and Reduce call may not be able to  
Figure 3: Hadoop Distributed File System (HDFS) framework and processing schematic. complete the entire filtering algorithm, particularly considering that the filtering algorithm needs to load N(N\(-\)1)/2 sets of interferograms for filtering, and using MapReduce may not be very suitable. A Resilient Distributed Dataset (RDD) is an abstraction of distributed memory that provides a highly constrained shared-memory model in which an RDD is only a collection of read-only record partitions. For developers, an RDD can be considered as a Spark object that runs in the memory itself. Under this concept, reading files, data computation, the resulting set of files, different partitions, dependencies between data, etc., can all be seen as RDDs. Spark provides rich RDDs that make it possible to perform complex algorithmic tasks in interferogram filtering in a single Spark program. As Spark does not have a distributed storage system or resource scheduling manager, it needs to be stored and managed by other distributed frameworks, and the parallelization solution we use is on a YARN cluster built under the Hadoop framework. In fact, most Spark-based companies and organizations are currently using the Hadoop framework to develop on the YARN cluster. The workflow of Spark on a YARN cluster is shown in Figure 4; we summarize its main principles as follows:  
1. The user submits the application in the YARN cluster client and launches the driver process;
2. The driver process sends a request to the ResourceManager (RS) to launch the ApplicationMaster (AM);
3. RS receives the request and randomly selects a NodeManager (NM) to launch the AM;
4. After the AM is launched, a multi-process executor is launched via the NM for the RS requesting container resources;
5. Finally, the executor is reverse-registered to the driver, who sends a task to the executor for data processing. It is worth noting that in an actual production environment, YARN automatically schedules and manages these resources, and the user only needs to set the relevant hardware parameters (memory, cores, etc.) without considering the complex resource scheduling issues. The spatial adaptive filtering algorithm must extract the phase information for each pixel of the generated N(N\(-\)1)/2 interferograms according to the predefined search window size, and each pixel must read the corresponding phase information in the interferogram. The frequent use of memory to transfer large data to computational nodes can significantly degrade the performance and even lead to memory overflow when the data volume is large.