# A Scalable Framework for Sensor Data Ingestion and Real-Time Processing in Cloud Manufacturing
## 5 Results
### Data Processing and Analytic Systems  
After the evaluation of the two data ingestion systems, a detailed assessment was conducted on the performance of two data processing systems: Spark Streaming and Hadoop Batch Processing. The Kafka producer script was executed with a maximum message count of 10,000, leveraging its favorable throughput and processing time results compared to RabbitMQ under similar input conditions (see Table 1). The comparative performance metrics for Spark Streaming and Hadoop Batch Processing are summarized in Table 2, which serves as a critical reference for analyzing their suitability within the CM framework. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Messages** & **System Component** & **Time (s)** & **Throughput (msg/s)** \\ \hline \multirow{4}{*}{100} & Kafka Producer & 0.16 & 644.87 \\  & Kafka Consumer & 2.21 & 45.28 \\  & RabbitMQ Producer & 0.16 & 644.83 \\  & RabbitMQ Consumer & 0.06 & 1614.62 \\ \hline \multirow{4}{*}{1000} & Kafka Producer & 1.40 & 714.21 \\  & Kafka Consumer & 0.47 & 2137.99 \\  & RabbitMQ Producer & 1.49 & 671.55 \\  & RabbitMQ Consumer & 0.46 & 2161.49 \\ \hline \multirow{4}{*}{10,000} & Kafka Producer & 14.02 & 713.25 \\  & Kafka Consumer & 5.27 & 1895.80 \\ \cline{1-1}  & RabbitMQ Producer & 15.97 & 626.34 \\ \cline{1-1}  & RabbitMQ Consumer & 4.61 & 2169.57 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance comparison of Apache Kafka and RabbitMQ across varying message loads. An analysis of the metrics presented in Table 2 demonstrates their pivotal role in substantiating the advantages of the proposed framework, as follows:  
* Total Records Processed: While Hadoop Batch Processing handles a larger volume of records (10,000 compared to 2588 for Spark Streaming), the discrepancy arises from the batch-oriented nature of Hadoop versus the real-time, continuous nature of Spark Streaming. This distinction reflects the suitability of Spark Streaming for low-latency CM applications, in which immediate insights are prioritized over total volume. * Total Processing Time (s): Spark Streaming outperforms Hadoop Batch Processing with a shorter processing time (4.236 s vs. 5.976 s). This metric underscores the efficiency of Spark Streaming's in-memory computation model, which reduces the time required to derive actionable insights, and is an essential feature for dynamic and fast-paced manufacturing environments. * Throughput (records/s): While Hadoop exhibits higher throughput (1673.45 records/s compared to 612.25 records/s of Spark), this metric aligns with its design for batch \begin{table}
\begin{tabular}{c c c} \hline \hline
**Metric** & **Spark Streaming** & **Hadoop Batch** \\ \hline Batch ID 0 Record Count & 103 & - \\ Batch ID 1 Record Count & 2324 & - \\ Batch ID 2 Record Count & 161 & - \\ Total Records Processed & 2588 & 10,000 \\ Total Processing Time (s) & 4.236 & 5.976 \\ Throughput (records/s) & 612.25 (avg) & 1673.45 \\ Average Latency (s) & 1.390 & - \\ CPU Usage (Before) & 33.5\% & 28.6\% \\ CPU Usage (After) & 39.4\% (avg) & 33.1\% \\ Memory Usage (Before) & 64.1\% & 63.5\% \\ Memory Usage (After) & 64.0\% (avg) & 63.8\% \\ \hline Average Temperature (avg\_T1) & 21.68 & 21.67 \\ Average Relative Humidity (avg\_RH\_out) & 82.43 & 79.77 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance comparison of Spark Streaming and Hadoop batch processing systems. Figure 5: Bar chart of performance comparison of Apache Kafka and RabbitMQ across varying message loads. processing scenarios. The relatively low throughput of Spark is a trade-off for its capability to provide real-time processing and immediate feedback. * Average Latency (s): The 1.390 s average latency of Spark Streaming is a critical metric that highlights its superiority in delivering near-instantaneous data processing. Hadoop, which is inherently batch-oriented, does not support latency measurements in the same sense that its outputs are processed in bulk.