# Efficient Parallel Processing of Big Data on Supercomputers for Industrial IoT Environments
## 5 Results, Discussion, and Implications  
This section recapitulates the results obtained in our extensive evaluation and reports our experience with the configuration of our testbed and of the Spark platform. As we previously commented, the execution of typical big data processing workloads on our testbed yielded consistently good performance across the evaluated scenarios. The detailed analysis presented in Section 4 highlights how the HPC-like ARM-based testbed influences the runtime behavior of Spark. As expected, the most critical performance bottleneck lies in the data shuffling phase, during which intermediate data must be exchanged among Spark workers. Our results demonstrate that the configuration of the input dataset significantly affects this phase. Specifically, datasets with fewer rows and more columns result in better performance compared to those with many rows and few columns. This is because Spark typically partitions data by rows; reducing the number of partitions (i.e., rows) lowers the volume of data transferred across the network during shuffle-intensive stages. The experiment in Section 4.2 further examines the impact of enabling Scalable Vector Extension (SVE) instructions on performance. We observed that vectorization is particularly effective when the degree of application-level parallelism is high, confirming the relevance of these features in compute-bound workloads. However, these optimizations must be supported by the software stack and properly exposed through the runtime environment. Our findings suggest that, in addition to selecting hardware configurations carefully, users must also pay close attention to software compatibility and workload characteristics. The choice of the JVM version, Spark release, and associated frameworks must be aligned with the hardware features to fully exploit their capabilities. Moreover, Spark parameters, such as the number of partitions, executor resources, and shuffle settings, should be tuned according to the size and shape of the input data. These aspects remain critical for ARM architectures as well, and failure to adapt the platform configuration accordingly can lead to suboptimal results, despite the underlying hardware's capabilities. Despite the representativeness of our testbed for industrial HPC environments and the breadth of the conducted evaluation, our study has some limitations. In particular, the use of synthetic benchmarks rather than actual industrial datasets may not capture all the complexities of real-world workloads. Incorporating domain-specific applications and data from operational environments would provide a more comprehensive assessment and strengthen the applicability of our findings to production settings. Overall, we conclude that the results we obtained show promising performance and follow a trend that emphasizes the scalability of the Spark platform, even on the HPC platform we used, despite it being based on ARM processors. This demonstrates that the new generation of ARM chips we used is a promising template for the incumbent development of the new European processors, which is the goal of the EPI project. We want to emphasize the relevance of the adoption of vectorial instructions to achieve these results, and we recommend their adoption in the future Rhea-GPP processor. In Section 3, we stressed that SVE instructions can be effective only if there is adequate support in the software stack between the processor and the running application. In this paper, we were able to identify OpenJDK 16 as the main tool offering such support, as it provides a specific runtime flag to optimize the machine code for SVE on ARM processors. However, we found that the degree of maturity of such a solution still requires improvement in terms of integration with the existing platforms and tools belonging to the big data processing ecosystem.