# Efficient Parallel Processing of Big Data on Supercomputers for Industrial IoT Environments
## 6 The Related Literature  
Much research has been conducted to measure big data analytics frameworks performance on HPC systems. The authors of [20] designed a scalable computing resources system for big data storage, processing, and visualization, specifically for remote sensing (RS) applications. It emphasizes the use of Apache Spark, deployed on ARM-based High-Performance Computing (HPC) systems, to enhance the performance of data-intensive tasks. The framework utilizes Kubernetes (K8s) for container orchestration and HDFS for efficient data storage, enabling high-speed read throughput that is crucial for handling large RS datasets. In the realm of distributed computing, serverless architectures like Lithops present a compelling approach to simplify application deployment and scaling by abstracting server management from developers. The Function-as-a-Service (FaaS) model, central to serverless computing, allows for the automatic scaling of discrete functions in response to demand, with Lithops enabling the execution of Python code across thousands of cloud cores without altering local scripts. While Lithops has primarily been utilized in cloud environments for big data processing, its potential in High-Performance Computing (HPC) systems remains largely unexplored until recently. A novel architecture has now been proposed to deploy Lithops on HPC systems, such as the MareNostrum 5 supercomputer, leveraging its computational power and the FaaS model to achieve high performance and scalability while reducing CPU wastage. The integration of serverless computing with HPC systems reflects an emerging trend that could influence how big data frameworks are deployed and optimized, particularly in ARM-based environments targeting industrial and edge computing scenarios [21]. Ref. [22] evaluated MapReduce applications on scale-up and scale-out clusters and proposed a hybrid scale-up/out Hadoop architecture based on the results. Other studies have focused on improving communication between compute nodes (horizontal data movement), and researchers have utilized RDMA-based mechanisms [23; 24; 25]. It is only during the concluding part of the transfer, when data is held in memory, that these optimized RDMA techniques offer their greatest benefit value. Even though data residence is not assured, requests tend to be serviced using a client-server programming model. Performance can be optimized through multiple strategies, such as using a bounded thread pool SEDA-based approach to avoid overtaxing computing resources or having one server thread per connection, given a sufficient number of cores is available [26]. On the other hand, for the optimization of vertical data movement, one of the main reasons for the emergence of Spark, researchers have investigated file consolidation optimization and optimizations intended to keep objects in memory when possible [27]. Our research aims to measure the suitability of the Rhea-GPP processor for efficient big data workflows that match the performance of existing HPC systems. We aimed to evaluate the software stack necessary for running big data workloads, particularly those based on Apache Spark, and their complexity. Additionally, we aimed to examine big data processing performance on ARM processors in an HPC environment, particularly the degree of vectorial instruction supports in the software stack and the performance of the software stack given the inclusion of JVM support for SVE instructions. Lastly, we compared HPC environments to standard data center computations with regards to data storage and movement. Several research efforts have been dedicated to measuring big data analytics frameworks performance on HPC systems, as highlighted previously. However, with the rise of Industry 4.0 and the increasing integration of cloud and edge computing in the industrial IoT (IIoT), it is becoming essential to consider the latest advancements in these research areas.