## 3 Target Data Processing and Computing
### Three Data Processing Paradigms
#### 3.1.1 Batch Processing  
Batch processing involves the intensive computation of high-volume, repetitive data processing tasks. Nowadays, data are generated from several sources, and there is a huge demand for the storage, processing, and querying of big data. The use cases for batch processing include machine learning, deep learning, big data analytics, and high-throughput data processing. Batch processing, which is designed for the handling of large volumes of data in situations where real-time processing is unnecessary, takes on a strategic role when deployed on collaborative local edge/fog devices. This deployment approach, represented by applications such as machine learning training, minimizes the need for extensive data transmission to centralized cloud servers. By harnessing the collective power of local devices, this strategy enhances data security and privacy while optimizing resource utilization. The MapReduce model and its implementations, such as Apache Hadoop [48] and Apache Spark [27], are the de facto solutions for big data batch processing. This approach to distributed computing provides a robust and scalable framework for the handling of large datasets by dividing complex processing tasks into manageable subtasks across a cluster of nodes. Apache Hadoop, an open-source software framework, pioneered the practical implementation of the MapReduce model, offering a distributed file system (Hadoop Distributed File System) and MapReduce programming. Similarly, Apache Spark emerged as a powerful alternative, introducing in-memory processing capabilities that significantly accelerate data processing workflows. These solutions offer platforms for the development, storage, and deployment of big data applications from large-scale analytics to machine learning applications. Commercial cloud providers such as Amazon EMR,Google Cloud Dataproc, and Azure HDInsight offer services for batch processing. These de facto big data solutions can also be hosted on private clusters or a private cloud. Big data batch processing platforms are organized with the master-workers distributed computing architecture. It consists of one or more master nodes that are responsible for distributing tasks to the workers and managing the workers. The workers perform the actual data processing tasks assigned to them. Generally, the data are partitioned and distributed among the nodes, resulting in the parallel processing of data by the nodes.  
Among the many big data platforms available for batch processing, we selected Apache Spark, which is an open-source, distributed computing system that has gained widespread popularity for the efficient processing of large-scale datasets. Its technological underpinnings are grounded in a resilient distributed dataset (RDD) abstraction, which allows data to be distributed across a cluster of machines while ensuring fault tolerance and parallel processing [27; 52]. In the context of edge and fog computing, where resource constraints and network variability pose significant challenges, Apache Spark offers several features to overcome these obstacles. One notable feature is its ability to optimize data locality, minimizing data transfer across the network [53]. Furthermore, Spark's flexibility in terms of deployment in various types of infrastructure, including edge devices and fog nodes, makes it adaptable to diverse computing environments [46].