# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
###### Abstract  
In terms of the calibre and variety of services offered to end users, smart city management is undergoing a dramatic transformation. The parties involved in delivering pervasive applications can now solve key issues in the big data value chain, including data gathering, analysis, and processing, storage, curation, and real-world data visualisation. This trend is being driven by Industry 4.0, which calls for the servitisation of data and products across all industries, including the field of smart cities, where people, sensors, and technology work closely together. In order to implement reactive services such as situational awareness, video surveillance, and geo-localisation while constantly preserving the safety and privacy of affected persons, the data generated by omnipresent devices needs to be processed fast. This paper proposes a modular architecture to (i) leverage cutting-edge technologies for data acquisition, management, and distribution (such as Apache Kafka and Apache NiFi); (ii) develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in the context of smart cities processing multi-modal, real-time, and heterogeneous data flows; and (iii) address the key challenges in tasks involving complex data flows and offer general guidelines to solve them. In order to create an effective system for the monitoring and servitisation of smart city assets with a scalable platform that proves its usefulness in numerous smart city use cases with various needs, we deduced some guidelines from an experimental setting performed in collaboration with leading industrial technical departments. Ultimately, when deployed in production, the proposed data platform will contribute toward the goal of revealing valuable and hidden societal knowledge in the context of smart cities. s +
Footnote †: This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ([[https://creativecommons.org/licenses/py/4.0/](https://creativecommons.org/licenses/py/4.0/)]([https://creativecommons.org/licenses/py/4.0/](https://creativecommons.org/licenses/py/4.0/))). the platforms offered Application Programming Interfaces (APIs) for the consolidation of data across multiple services in the cloud, e.g., for integrated user dashboards or big data analysis of historical data [7]. Indeed, many studies have focused on supporting the semantic interoperability of data only _after_ they have been safely stored in a common repository (currently referred to as \"data lake\") [8; 9]. However, such a compartmentalised structure had limitations, especially in terms of redundant deployed resources and inefficient management. Therefore, the community has moved towards a horizontal approach, where a common platform is able to communicate with all kinds of devices, which are most often sensors in smart city applications [10]. This evolution is illustrated in Figure 1, which also shows the high-level architecture of the Data Management Platform (DMP)-defined H2020 MARVEL project, which aspires to define a comprehensive solution for multi-modal real-time analytics applications. The H2020 MARVEL project aims to support efficient and informed decision-making for smart city administration and public authorities by adopting an IoT approach for analysing and fusing multimodal data from sensors installed at a city level. Such applications derive from the analysis of the requirements and expectations in several use cases of practical interest and the high impact on the qualities of life of citizens; they will be validated in three small-scale field trials across Europe, i.e., in Malta, Serbia, and Italy [11].