# A Scalable Framework for Sensor Data Ingestion and Real-Time Processing in Cloud Manufacturing
## Appendix A
```
0: Imports and Initial Setup Import necessary libraries: - Spark SQL libraries - System monitoring and time libraries Define data schema for incoming temperature and humidity data ```
**Algorithm A6** Spark Streaming Code for Real-Time Monitoring
```
Initialization Start Spark Session for real-time streaming and configure it to read from Kafka Stream Kafka data by: - Connecting to Kafka server - Subscribing to the specified topic - Loading data stream Parse and structure data by: - Converting data to string - Parsing JSON data based on schema Define Termination and Metrics Variables Set termination parameters for detecting a halt in new data Initialize cumulative metrics dictionary to track performance and data averages over time Processing Function: Process Each Micro-Batch function PROCESS_BATCH(batch_df,batch_id) Record start time of processing Capture CPU and memory usage before batch processing Count records in the batch if no new data detected after multiple batches then Increment empty batch counter if counter exceeds limit then Terminate the streaming job endif else Reset empty batch counter endif Calculate data averages by: - Computing average temperature and humidity values for each sensor - Storing these averages in a temporary structure for this batch Calculate performance metrics by: - Computing processing time, throughput, and latency - Recording CPU and memory usage after processing Update cumulative metrics with batch statistics (latency, temperature, and humidity averages) Log performance data by writing batch ID, record count, processing time, and system metrics to a log file endfunction Streaming Job Execution Configure streaming query to process each batch with process_batch by: - Using foreachBatch to apply the function to each micro-batch - Setting outputMode to update with each new batch Start and await termination of the streaming query with a timeout
```
**Algorithm A5**_Cont._  
Algorithm A6 reports the pseudocode for the Hadoop batch processing system. ```
0: Batch processing results and system metrics Imports and Initial Setup Import necessary libraries, including Spark SQL and system monitoring libraries Define data schema for temperature and humidity Initialize Spark Session Start a Spark session with Kafka configuration - Configure Kafka settings by: - Setting bootstrap_servers - Subscribing to the specified topic - Setting offset to earliest to process all data ```
Data Loading and Batch Limiting Load data from Kafka in batch mode Limit data to first 10,000 records for batch simulation Introduce artificial delay for simulated batch processing System Metrics Before Processing Record CPU and memory usage before processing Start timer to measure batch processing time Data Processing and Aggregation Convert batch data to string format Parse JSON and extract relevant fields according to schema Calculate average values for each temperature and humidity sensor Calculate Performance Metrics Record total processing time Count records processed and calculate throughput System Metrics After Processing Record CPU and memory usage after processing Log Output to File Format log output with batch processing results by: - Total records processed, processing time, throughput - CPU and memory usage before and after processing - Calculated averages for sensor data Write log output to file Terminate Spark Session Stop the Spark session
```  
**Algorithm A6**_Cont._  
In Algorithm A7, the data processing for temperature and humidity is implemented for the monolithic app and the performance of the monolithic system is further evaluated through a simulation in Algorithm A8. For the microservices architecture, the ingestion service is described in Algorithm A9, followed by the processing service in Algorithm A10 and storage service in Algorithm A11. Deployment of these services is handled by Kubernetes, as described in Algorithms A12-A14. Performance testing using Locust is outlined in Algorithm A15.