# A Scalable Framework for Sensor Data Ingestion and Real-Time Processing in Cloud Manufacturing
## 3 The Proposed Framework  
Following the recent advancements in CM framework design [26; 27], this section presents a novel nine-layer framework designed to enhance CM systems with a modular service-oriented architecture that promotes efficient, secure, and scalable operations. The framework depicted in Figure 2 is organized into nine layers: (1) data collection, (2) communication, (3) data ingestion, (4) data processing and analytics, (5) application, (6) storage, (7) elastic computing and orchestration, (8) security and access control, and (9) user interface and visualization [28]. The data collection layer gathers raw data from various sources, which are then transmitted through the communication layer. The data ingestion, processing, analytics, and application layers form the core of the system, wherein data are transformed into actionable insights. The storage layer ensures data persistence, while the elastic computing and orchestration layers manage resource allocation. The security and access control layers safeguard the entire system, whereas the user interface and visualization layers present processed information in an accessible format. In existing architectures, tools such as Apache Kafka, Spark Streaming, and microservices have been widely employed to address challenges in CM environments. For instance, frameworks leveraging RabbitMQ or Apache Kafka for data ingestion have demonstrated varying successes in managing high-throughput data streams. However, RabbitMQ often struggles with scalability in distributed systems, whereas the Kafka subscribe architecture offers exceptional scalability and fault tolerance. Similarly, Hadoop-based systems have historically dominated data processing, but have been increasingly replaced by Spark Streaming because of their low-latency, real-time processing capabilities. Microservices have gradually supplanted monolithic architectures, owing to their flexibility in independently scaling, updating, and managing discrete services [29; 30]. The proposed nine-layer framework extends the existing solutions by integrating and optimizing these tools to achieve a higher degree of responsiveness, scalability, and modularity. Notably, it introduces a seamless integration of these technologies, paired with innovations, such as enhanced elastic computing mechanisms, advanced orchestration  
Figure 2: Schematic representation of the proposed framework. The dashed red rectangle highlights the focus of this study. strategies, and comprehensive data privacy and access control policies. This enables the framework to satisfy the critical demands of modern CM systems, such as real-time adaptability, robust security, and user-friendly operations. Specifically, the framework was designed to process the sensor readings generated by devices operating in the AM field, such as 3D printers. These devices produce large datasets that capture critical parameters such as temperature and relative humidity during the material extrusion process when printing polymer filaments. The choice of this dataset highlights the applicability of the framework for optimizing the production workflow in AM. The data ingestion layer utilizes Apache Kafka, an open-source streaming platform that supports high-throughput real-time data streaming with exceptional scalability and fault tolerance. The publish-subscribe architecture of Kafka enables efficient queuing and seamless data flow to downstream services, making it ideal for CM environments in which IoT devices continuously generate high volumes of data. The experimental findings in Section 5 demonstrate that Kafka significantly outperforms RabbitMQ in handling high-velocity data streams, thereby achieving lower latency and improved scalability within distributed CM systems [31]. In the data processing and analytics layer, Spark Streaming provides real-time analytics capabilities and supports the integration of Machine Learning models for adaptive data-driven decision making.