# Industry 4.0 towards Forestry 4.0: Fire Detection Use Case +
## 5 Methodology
### Data Stream Processing Pipeline
#### 5.2.1 IoT Streaming Data Pipeline over Apache Flink  
A typical data pipeline scenario could be as collecting generated data from IoT devices by Kafka once the events are observed. Then, the Kafka produced data are ingested into Flink by Flink consumers. The Flink executes the deployed continuous join query over the ingested data by allowing the running windows to trigger its output based on its timestamps. The processed output stream will then be sent again to Kafka to be consumed within the data application pipeline. In doing so, the workflow for IoT data stream pipeline, shown in Figure 8, consisting of three phases which are; (1) data collection (2) data processing using Apache Flink) stream data output. Each stage can be described as follows:  
In the data collection phase, the input data stream is gathered from IoT devices (i.e., data source generators) using Apache Kafka as a scalable message queuing system. Apache Kafka is responsible for sending input streams to Flink and receiving output stream from Flink. Worthwhile, other distributed queuing management technologies such as RabbitMQ, Amazon Kinesis, Microsoft Event Hubs, and Google Pub/Sub. We chose Kafka as it is the state-of-the-art, distributed large-scale real-time data applications. It has high delivered and ordering guarantees of the data stream, making Kafka reliable in data drift situations.  
For data processing over Apache Flink, this phase contains two components; stream processing engine used Apache Flink ([[https://flink.apache.org/](https://flink.apache.org/)]([https://flink.apache.org/](https://flink.apache.org/))), and stream query deployer to deploy a stream query, i.e., join stream query over Apache Flink. The choice of using Apache Flink to implement this phase can be attributed to three characteristics of its approach. First, it integrates with the best existing ideas in the industry, such as the open-source adopter of the Dataflow/Beam model and compatible with Kafka to guarantee the reliability of the data pipeline with different stream rates. Secondly, it has a powerful consistency via snapshots, savepoints, and streaming SQL. Third, it is a highly-scalable for the world's most demanding stream processing applications. It can scale to thousands of cores and terabytes of application state and deliver high throughput and low latency for latency-critical applications. Fourthly, it is continuously improving for streaming processing across the industry around the globe such as Alibaba, eBay, Lyft, Uber... etc. Fifth, it is automatic scaling with the fluctuated data stream rate.  
In the output stream to data sink phase, the output stream is sent back again to Kafka to be consumed by other data sinks. These data sinks could be any data consumer within the real-world application in the industrial setting such as web service, real-time dashboard, storing in big data storage (e.g., HDFS, MongoDB), and sending to alarm systems. Moreover, the output stream could be passed to the next data analytic phase, such as machine learning and/or deep learning.  
Figure 8: The workflow of data stream pipeline over Apache Flink consisting of three phases (1) data collection (2) data processing and (3) data output.