# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
## 2 Background
### Related Work  
The topic of the efficient allocation of resources in E2F2C has been extensively studied in the scientific literature (e.g., [15]). The vast majority of the works propose highly simplified mathematical models, which are then solved to maximise a given objective function, but they do not linger on the practical implementation of their solution using industry-grade tools. On the other hand, in the H2020 MARVEL project, our goal was to realise a solution based on widely-used, reliable, and open source software. Furthermore, some works have focused on the specific aspect of efficient data distribution. In [16], the authors have proposed Peer-to-Peer (P2P) as a means to distribute data in a robust and decentralised manner among agents at the edge. Such an approach only partially covers the needs of the H2020 MARVEL project, since, due to its use case requirements, the H2020 MARVEL project AV real-time streams cannot be stored efficiently in a P2P overlay. An in-network storage management is put forward in [17], to place both raw and structured data on distributed resources using the Google File System, which is, however, proprietary. Another perspective is adopted in [18], which builds on top of the distributed Apache Cassandra ([[https://cassandra.apache.org/](https://cassandra.apache.org/)]([https://cassandra.apache.org/](https://cassandra.apache.org/)), accessed on 16 January 2023) database by adding geolocalisation tags that help distribute the load across edge/fog nodes; we have not considered this feature since, at least in the project pilots, the edge/fog nodes are all deployed in close proximity. The solutions mentioned can be gradually incorporated in the DMP at a later stage of the project, which is still ongoing with an expected completion date at the end of 2023. More specifically, Apache Kafka has also been widely used in the streaming applications domain. In [19], the authors replicate Apache Kafka logs for various distributed data-driven systems of LinkedIn, including source-of-truth data storage and stream processing. In [20], the authors design a distributed cluster processing model based on Apache Kafka data queues to optimise the inbound efficiency of seismic waveform data. In [21], the authors extend Apache Kafka by building an in-memory distributed complex event recognition engine built on top of Apache Kafka streams. In [22], the authors design a simulation platform enabling evaluations of future mobility scenarios, based on an Apache Kafka architecture. In [23], the authors break the streaming pipeline into two distinct phases and evaluate the percentile latencies for two different networks, namely 40 GbE and InfiniBand EDR (100 Gbps), to determine if a typical streaming application is network intensive enough to benefit from a faster interconnect. Moreover, they explore whether the volume of an input data stream has any effect on the latency characteristics of the streaming pipeline and, if so, how it compares for different stages in the streaming pipeline and different network interconnections. In [24], the authors propose a distributed framework for the application of stream processing on heterogeneous environmental data, which addresses  
Figure 4: Schematic of a typical multi-modal, i.e., audio and video, real-time analytics application in H2020 MARVEL project. The mapping of components to the edge/fog/cloud layers is only indicative: in real applications, this may depend on physical, environmental, and administrative constraints. the challenges of data heterogeneity from heterogeneous systems and offers real-time processing of huge environmental datasets through a publish/subscribe method via a unified data pipeline with the application of Apache Kafka for real-time analytics. In [25], the authors find that filtering on large datasets is best performed in a common upstream point instead of being pushed to, and repeated, in downstream components.