# A Scalable Framework for Sensor Data Ingestion and Real-Time Processing in Cloud Manufacturing
## 6 Discussion  
The proposed nine-layer framework was designed to address the challenges of scalability, resource orchestration, and data security in modern decentralized manufacturing settings. For this implementation, we use technologies that excel in handling the ingestion and processing of large-scale IoT sensor data with low latency, thereby ensuring real-time responsiveness and long-term operational reliability. This is especially critical for applications in CM and AM, where the ability to monitor and control processing parameters such as temperature and humidity is essential for ensuring the quality and consistency of 3D-printed parts produced through filament material extrusion. The experimental campaigns clearly demonstrated the performance gains achieved through the integration of Apache Kafka for data ingestion, Spark Streaming for processing, and a microservice architecture for overall framework scalability and resilience. Specifically, the performance evaluation of Kafka and RabbitMQ (Table 1) highlighted the superior capabilities of Kafka in high-throughput, low-latency data ingestion scenarios, which are critical for CM and AM applications. The log-based architecture of Kafka, combined with its durability and fault tolerance, makes it particularly well-suited for handling continuous streams of sensor data generated in AM workflows. For example, Kafka achieved a throughput of 714.21 messages per second for a workload of 1000 messages, outperforming the 671.55 messages per second of RabbitMQ, under the same conditions. This superior throughput, coupled with the consistent performance of Kafka under increased data volumes, ensures that the framework can handle the large-scale datasets required for real-time monitoring of processing parameters such as temperature and humidity in AM. Such capabilities enable immediate fault detection and corrective action, reduce the risk of defects in 3D-printed parts, and enhance the overall operational efficiency. Similarly, the adoption of Spark Streaming for real-time processing has demonstrated significant advantages over traditional batch processing systems such as Hadoop, as shown  
Figure 6: Bar chart of performance metrics for the Monolithic app vs. Microservice architecture. in Table 2. The micro-batch architecture of Spark allows for the near-instantaneous ingestion and analysis of IoT sensor data, achieving substantial reductions in processing time and latency. In our experiments, Spark processed equivalent workloads in 4.236 s compared with 5.976 s obtained with Hadoop, achieving a lower average latency of 1.390 s. This real-time processing capability is particularly advantageous for CM and AM scenarios, which require rapid responsiveness to environmental changes during the production of 3D-printed parts. By leveraging Spark's ability to process large volumes of streaming sensor data in real-time, the framework can quickly identify inefficiencies or faults in AM systems, thereby ensuring optimal parameter control and product quality. The use of microservice architecture further enhances the ability of the framework to support high-throughput, low-latency operations across its layers, as indicated by the comparative results in Tables 3 and 4. Microservices demonstrate superior scalability and fault tolerance under an increasing workload compared with monolithic systems. For instance, under a 10,000 user load, the microservice-based ingestion service processed 366,940 requests compared to the 102,972 requests of the monolithic system. Additionally, microservices maintained sub-10 ms response times for ingestion, processing, and storage tasks, while the monolithic system exhibited response times exceeding 9000 ms. This modularity ensures that failures are isolated into specific components, thereby maintaining the functionality of other parts of the framework during partial disruptions.