# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
## 2 Background  
In this section, we provide an overview of the aspects of the H2020 MARVEL project that are relevant to the design and implementation of the DMP, which is the main subject of this work.  
A layered view of the physical deployment of the project's pilots is illustrated in Figure 3. The lowest layer is the _edge_, which contains the sensors and embedded devices to perform on-site operations. We have two types of sensors: microphone arrays, producing audio streams, and cameras producing AV streams. All the sensors are expected to operate continuously during the service lifetime. The embedded devices, i.e., Raspberry Pis and NVIDIA Jetson boards, perform operations directly at the edge, which include the anonymisation of the AV streams and basic inference operations. The intermediate layer is called the _fog_ (we note that, in the scientific literature and in the market press, the terms \"edge\" and \"fog\" do not have universally accepted meanings. Sometimes they are even used interchangeably), which includes more powerful computation resources  
Figure 2: The flowchart of our methodology.  
that are shared by multiple edge sites, e.g., workstations and rack-mountable servers with Graphics Processing Units (GPUs), which are suitable for not only managing the different data streams but also performing more advanced inference tasks, as well as training AI models. Finally, the _cloud_ is hosted in an infrastructure provided by a project partner and it is common for all the pilots. The cloud provides the long-term storage of data and all the services for HMI, i.e., visualisation and real-time (on-demand) analysis.  
In Figure 3, we distinguish between raw data vs. structured data:  
* The **raw data** are the AV streams generated by the sensors, irrespective of whether they have been anonymised or not. Their formats and characteristics are heterogeneous because they depend on the physical devices installed (e.g., may use different codec or sample AV at different rates). In any case, the throughput is generally high, especially for video, which requires carefully provisioned bandwidth, long-term storage, and configuration of the data distribution services.
* The **structured data**, instead, are data generated by the analytics applications, i.e., the inference components. In the project, there are different data models for each AI component, but all of them have been based on the fully Smart-Data-Model (SDM)-compliant ([[https://smartdatamodels.org/](https://smartdatamodels.org/)]([https://smartdatamodels.org/](https://smartdatamodels.org/)), accessed on 16 January 2023) data models that DatAna are producing. Furthermore, their throughput is generally much lower than that of raw data.  
A schematic of a typical multi-modal application in the H2020 MARVEL project is illustrated in Figure 4. We have used \"thick\" arrows to represent the raw data flows, while \"thin\" arrows refer to structured data flows; this reflects, in a visual manner, their different bandwidth requirements. The diagram is intended to provide the reader with a sketch of the components that are involved in the deployment of a service, which can differ in practice depending on the specific use case and pilot. The services we support in the project include surveillance applications (e.g., detecting anomalies in public spaces), emotion recognition (with aerial images taken by drones), and road safety (e.g., monitoring junctions with mixed cars and bike traffic).  
Figure 3: E2F2C view of the H2020 MARVEL project’s pilots.