# Efficient Parallel Processing of Big Data on Supercomputers for Industrial IoT Environments
###### Abstract  
The integration of distributed big data analytics into modern industrial environments has become increasingly critical, particularly with the rise of data-intensive applications and the need for real-time processing at the edge. While High-Performance Computing (HPC) systems offer robust petabyte-scale capabilities for efficient big data analytics, the performance of big data frameworks, especially on ARM-based HPC systems, remains underexplored. This paper presents an extensive experimental study on deploying Apache Spark 3.0.2, the de facto standard in-memory processing system, on an ARM-based HPC system. This study conducts a comprehensive performance evaluation of Apache Spark through representative big data workloads, including K-means clustering, to assess the effects of latency variations, such as those induced by network delays, memory bottlenecks, or computational overheads, on application performance in industrial IoT and edge computing environments. Our findings contribute to an understanding of how big data frameworks like Apache Spark can be effectively deployed and optimized on ARM-based HPC systems, particularly when leveraging vectorized instruction sets such as SVE, contributing to the broader goal of enhancing the integration of cloud-edge computing paradigms in modern industrial environments. We also discuss potential improvements and strategies for leveraging ARM-based architectures to support scalable, efficient, and real-time data processing in Industry 4.0 and beyond. HPC; MapReduce; Apache Spark; Industrial Internet of Things; Scalable Vector Extensions (SVE) 2025202514  
has become indispensable. To meet time-based quality-of-service (QoS) requirements, such as reduced latency and high throughput, big data workflows are increasingly being deployed in distributed environments, including clusters, clouds, and edge computing systems. These environments must handle the dual challenges of processing massive data volumes while maintaining the low-latency demands of industrial applications [3]. Over the past decade, several big data processing paradigms have emerged to address these challenges. Among them, the MapReduce paradigm and its initial implementation, Apache Hadoop [4] have been widely adopted for various real-world applications. However, the limitations of disk-based processing have led to the rise of in-memory systems like Apache Spark [5], which offer significantly faster data processing by keeping data in memory. These systems are typically deployed in large clusters of bare-metal devices or virtual machines, operating in a parallel computing fashion. Meanwhile, High-Performance Computing (HPC) systems, known for their fast computation power and high-performance capabilities, are gaining traction for big data processing in industrial settings. HPC systems, with their thousands of nodes and multiple processing cores, are particularly appealing for handling the massive data loads generated in modern factories. Despite their potential, HPC systems and big data processing frameworks have traditionally been designed with different architectures. HPC systems follow a compute-centric paradigm, where data is separated from computing resources, while big data systems like Apache Spark [5] employ a data-centric approach, co-locating data with computing resources to enable data parallelization. This architectural difference poses a significant challenge when integrating big data processing frameworks into HPC environments, as the frequent data transfers between parallel file systems and computation nodes can introduce latency, impacting QoS requirements [6]. Recent research has explored the performance of deploying big data processing frameworks, such as Apache Spark [5], on HPC hardware [7; 8; 6]. However, these studies have not addressed the performance of such frameworks on ARM-based HPC architectures, particularly those utilizing vector instructions like Scalable Vector Extensions (SVEs).