# Fog Computing for Smart Cities' Big Data Management and Analytics: A Review
## 7 Toward a Fog-Based Real-Time Big Data Pipeline
### Implementation Scenario  
As we have discussed earlier, the fog node's data processing components would typically include a data ingestion platform, a scalable messaging system, a data stream processing engine, and sufficient storage capacity. Apache Nifi [93], which automates the transfer of data between disparate systems, would allow the ingestion of data from different sources. It provides real-time control that makes it easy to manage data movement between all sources and destinations. It is data source agnostic and supports disparate and distributed sources of diverse formats, schemas, and protocols. Over the last decade, many publish-subscribe messaging systems have been developed. ActiveMQ [94], RabbitMQ [95], and Kafka are the most popular ones to date. They support several protocols including Advanced Message Queuing Protocol (AMQP), Message Queuing Telemetry Transport (MQTT) to exchange data with edge gateways. Apache Kafka's popularity is growing due to its advanced features for handling multiple data streams efficiently and scalably. Several real-time applications, such as social media analytics, telemetry from connected devices, network monitoring, clickstream analysis, and financial analytics and alerts, have successfully used Kafka. Many data processing engines like Apache Storm, Apache Flink, Apache Spark, Google Cloud Dataflow [96], and AWS IoT [97] can integrate well with Apache Kafka. Figure 8 illustrates the software components required to implement data stream processing in a fog node effectively. They include Apache Nifi for data ingestion, a Kafka cluster with multiple topics, a Storm topology for data processing, and a NoSQL database like Cassandra [98] or MongoDB [99] and query tools Apache Drill [100], and visualization and monitoring dashboards. Apache Kafka is typically deployed and executed on a cluster of one or several brokers (also called servers). It can immutably store messages received from multiple data sources, called producers, in topics or queues. The topics are organized into multiple partitions replicated to all clusters' brokers to guarantee fault tolerance. Each partition stores incoming events or messages, with an index and a timestamp. Consumers, which can be different applications, can then query the messages stored in Kafka partitions. Apache Storm is an open-source real-time data processing engine designed to accept tons of high-speed incoming data, possibly from multiple sources, for analysis and for publishing real-time updates to a user interface or application without storing actual data. Apache Storm relies on Apache Zookeeper, which is an open-source platform for cluster state management. A Storm application for real-time data processing is typically designed and implemented as a directed acyclic graph workflow with spouts and bolts forming the graph's vertices. A storm workflow is called a topology. Topology edges, called streams, carry data from one vertex of the workflow to another. Thus, a data stream retrieved by a spout from a data source is routed to various bolts where the data are filtered, aggregated, analyzed, and then sent to a user interface such as a dashboard or other application. Apache Storm is highly  
scalable and offers job processing guarantees. It can process more than a million tuples per second per node. Furthermore, it integrates very well with Apache Kafka to read and write data from Kafka topics using KafkaSpout and KafkaBolt. These software tools are not specific to fog computing solutions, but they are deployed by most public cloud providers such as Amazon AWS, Google Cloud, and Microsoft Azure to support large-scale data processing. They have also been used or suggested in many fog-based solutions [101; 102; 103]. We are at the early phases of implementing a fog node prototype to carry out data analytics at the edge using ThingsBoard IoT Gateway [104], an open-source IoT platform for the rapid ingestion of IoT data, a Kafka cluster and Apache Storm for data analytics.