# A SqueezeSAR Spatially Adaptive Filtering Algorithm Based on Hadoop Distributed Cluster Environment
## 2 Spatial Adaptive Filtering Algorithm in Hadoop Environment
### Spark-Space Adaptive Filtering Algorithm
In this case, Spark provides a broadcast RDD to effectively solve this problem by providing a large copy of the input data for each node through serialization. Figure 5 shows how the data are transferred after the broadcast RDD is used. When the data are broadcasted, they are distributed to the executors of each node, and each executor has a complete copy of the broadcasted data. This reasonably solves the problem of the memory overflow  
Figure 4: Principle of Spark on YARN operation. and performance loss caused by oversized shared datasets. In addition, in a distributed computing environment, if the image data are chunked and distributed to each node for parallel processing, the inherent connectivity of the image data itself and the characteristics of the graph computation exhibiting strong coupling need to be taken into account [49]. In fact, in the spatial adaptive filtering algorithm, the interferogram phase information data that need to take into account the above characteristics can be broadcast by the broadcast RDD so that a complete copy of the data is available at each node, thus eliminating the need to consider the loss in accuracy due to these characteristics. The proposed Spark parallelization algorithm mainly splits and computes the row file corresponding to each pixel point. As there is no direct correlation between each pixel point, the concept of RDD can be used to slice the data and distribute the data to each executor by using the memory interaction to start a task for independent computation. Finally, the data are merged by the RDD to complete the filtering process. In Algorithm 2, we present the pseudocode for parallelized processing using Spark. The RDD operations primarily used include textFile, broadcast, map, and collect. Please refer to the definition and explanation given in Algorithm 2 for the specific calculation and parameters. The principle of Spark-based spatially adaptive filtering is shown in Figure 6. In this process, we mainly used the various operators provided by Spark for data distribution, computation, and subsumption processing. First, we read the phase information of the N(N\(-\)1)/2 interferograms and the regular parameter information of the remaining interferograms through the master node. As the interferogram phase information requires the entire file to be transferred to each executor and, thus, occupies a large amount of memory, we need to serialize the data into the executors using the broadcast algorithm to reduce the communication between the data and relieve the pressure of excessive memory. This ensures that the executors can read the phase data as read-only variables without consuming a large amount of performance. Subsequently, we created an RDD by reading the SHP data files pre-stored in the HDFS through the textFile operator, which mainly include the rows and columns corresponding to each DS candidate point that meets the threshold requirement (25) in the interferogram and the SHP data identified in the search window. We then parallelized the process by loading these data into the memory and splitting and distributing them into multiple generated tasks. It should be noted that the remaining regular parameters are simply distributed based on the memory interactions, owing to the small amount of data, which has a minimal impact on the performance. In addition, Spark provides a map algorithm that performs spatially adaptive filtering of the data loaded into the memory in different tasks, in which each task performs interferogram filtering using the spatially adaptive filtering algorithm imported by the map operator  
Figure 5: The principle of distributing data to executors via Broadcast. separately to return a new distributed dataset. Finally, the resulting data, after spatial adaptive filtering, are collected by the master node through the collect operator and are written to the local node to complete the interferogram phase filtering. ```
Input: Three-dimensional complex array of interferogram phase values p[s,i,j].