# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
###### Abstract
In the project, we exploit the recent trend of breaking down the computation elements of the system into three layers in a hierarchy [12; 13]: the _edge_ layer is closest to the sensing and embedded computation devices, but it consists of devices with modest capabilities in terms of computation, connectivity, and storage; the _fog_ layer has more powerful capabilities and we treat it similar to a small private cloud, which is, however, under the control of the end user; and, finally, the cloud layer is hosted on public remote data centers, which have virtually infinite capacity but incur a high latency and usage costs. The H2020 MARVEL project covers all aspects, from the development of new sensors, e.g., directional microphone arrays, to the efficient training of Artificial Intelligence (AI)/Machine Learning (ML) models on devices with limited capabilities and the ethics of data collection and analysis. However, in this paper, we focus only on the DMP, which is the core of the project's software architecture and the main contribution of this paper. The  
Figure 1: Transition of smart city data management and distribution systems from vertical silos to horizontal (cloud-based) platforms and positioning of the H2020 MARVEL project’s proposition for multi-layer Edge to Fog to Cloud (E2F2C) environments. DMP attempts to solve the problem of big data management in the cases of Smart City IoT applications by addressing the following aspects:  
* Maximise the adaptability of the overall framework according to smart city end-user needs/use cases. The DMP adopts loosely coupled interfaces (e.g., MQTT and Kafka message brokers) and standardised data models (Smart Data Models initiative) that allow the interoperability of diverse AI components, which are selected to be applied on a case by case basis. * Allow the extensibility of the framework to be able to accommodate future needs by adopting new sensing technologies and analytics mechanisms. The standardised RTSP protocol allows the reception of AV streams from most commercial IP cameras and microphones. The loosely coupled interfaces and standardised data models allow the ingestion of data from any other potential future source (e.g., new AI component). * Remain fully scalable to be able to accommodate any data/source volume requirement. This is achieved by employing fully scalable data ingestion and processing mechanisms at the edge and fog layer (DatAna and StreamHandler), which can be deployed on virtually any additional infrastructure edge and fog node that become available. At the same time, the big data aggregation mechanism at the cloud layer (DFB) is based on fully scalable industry-standard technologies (Kafka, Elastic Search) and its operation is further optimised in cases of significant loads by the HDD. The DMP offers a full-featured, adaptable design solution for transferring, fusing, processing, and providing persistence to inference results generated by various AI components as well as raw audiovisual data. The DMP consists of the following components:  
* **DataAna**, for the processing and transmission of structured data produced by AI components to all the layers in the inference pipeline;
* **DFB** (Data Fusion Bus), which is in charge of managing heterogeneous data across multiple components in the cloud;
* **StreamHandler**, which processes, stores, and delivers real-time AV data at the fog layer towards the processing servers and the Data corpus;
* **HDD** (Hierarchical Data Distribution), which can optimize the data management based on the available resources and current workload. Those core components work in close connection to the **Data corpus**, which is the repository of the data collected from the sensors, mainly consisting of microphones and cameras, for visualisation and augmentation, and the **SmartViz**, which provides the Human Machine Interface (HMI) for visualisation and analysis.