# A SqueezeSAR Spatially Adaptive Filtering Algorithm Based on Hadoop Distributed Cluster Environment
## 3 Experimental Results and Analysis
### Comparative Experiments and Analysis of Results
#### 3.3.2 Performance Evaluation  
The parallelization strategy used in this study is mainly a YARN cluster consisting of three nodes deployed in a Hadoop environment, and the entire cluster environment consists of one master node and two worker nodes, each equipped with an 8-core CPU (Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70 GHz) and 64 GB of RAM. Among them, the master node acts as a data node and computation node; therefore, 2 TB of hard disk space is allocated for storing the intermediate and resultant data generated by the data processing chain of the SqueeSAR algorithm, while the worker nodes only act as computation nodes and therefore do not require more hard disk space, and the software configuration of the entire cluster is shown in Table 1. The proposed parallelization strategy filters the data with single and 2-core processing units in each executor. In this configuration environment, owing to YARN's load-balancing technology, the entire cluster does not suffer from memory overflow. Figure 11: Comparison of the two processing methods: (**a**) filtering with conventional data processing, (**b**) filtering with spark parallel algorithm. Figure 10: A certain differential interferogram before filtering using Doris to generate the study area. To evaluate the performance of the spatial adaptive filtering in a distributed cluster environment, we used the concept of speedup as a metric. The metric was estimated using a common dataset and a varying amount of executor processing data to evaluate the improvement as available computational resources. Speedup is defined as follows:  
\[\text{Sp}= \frac{T_{seq}}{T_{par}(\text{p})} \tag{3}\]  
where \(T_{seq}\) represents the execution time of the regular algorithm, \(T_{par}(\text{p})\) represents the execution time of the algorithm by the p executors processing data in parallel, and Sp represents the concept of speedup. In addition, the selection of the resource-allocation strategies for the different data streams is of great significance to the load and task division of a cluster. There are three main resource allocation strategies for YARN clusters: the FIFO Scheduler, Fair Scheduler, and Capacity Scheduler. Considering that the filtering algorithm only requires the spatial averaging of the data within the search window, we use the FIFO Scheduler for resource scheduling, and this strategy can effectively utilize the idle computing resources. In Spark, the dataset must be partitioned for optimal task selection, and we use different numbers of partitions for the performance evaluation. For the YARN clusters, a reasonable number of partitions can ensure that the time of the data transfer and the task start-up overhead are maintained in a relatively balanced state with the task processing time. When the number of tasks is large, the start-up time dominates, thus leading to a longer overall data processing time. When the number of tasks is small, the time of the data transfer dominates and, in this case, the data processing time will also be extended accordingly. To maximize the performance, we used different numbers of tasks to evaluate the performance of the entire YARN cluster in terms of the data processing of the filtering algorithm. Table 2 shows the time comparison of the impact of the different numbers of tasks on the parallel processing performance of the entire dataset when we started eight executors (each executor was assigned two core processing units). As can be seen in Table 2, the use of different tasks has a significant impact on the performance, with the data processing time being reduced from 99 s to 86 s with 240 and 480 tasks, respectively, resulting in an overall performance improvement of 13.1%. When the number of tasks increases from 480 to 640, the overall data processing time does not decrease, but rather increases, which confirms our suspicion that the task startup time dominates in this process.