# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
## 7 Limitations  
The DMP has been designed and realised within an international collaborative R&D project, which has allowed us to freely experiment with cutting-edge technologies following a revolutionary, rather than evolutionary, approach in a heterogeneous cross-disciplinary environment. While this setting is ideal to inspire and foster new ideas compared to a more traditional compartmentalised scenario typical of (especially large) industries, with hindsight we can identify some possible weaknesses of our framework when deployed in a large-scale production system related to: universality, security, reliability, and scalability. We discuss each item separately in the following. Universality. First, we have designed the DMP with the system and technical requirements of the project's use cases in mind. The latter have been carefully selected to cover a broad selection of applications currently of high impact in smart cities, with the help of end users involved in the project. However, it is evident that the landscape of Information and Communication Technologies (ICT) for smart cities already today is extremely fragmented and heterogeneous, due to the high variability of size/density factors and geographical, historical, and cultural differences, even within the European Union only. Therefore, we could not have aspired to design a one-size-fits-all solution. Instead, we have defined a modular structure, with the goal of supporting a swift replacement of individual components that might be needed to address a specific unforeseen need in a future deployment. Nonetheless, whether the DMP is the ideal solution for a given analytics application in a specific smart city deployment remains to be evaluated on a case-by-case manner. Security. As it is known by whomever is familiar with security in ICT, a system is as weak as its weakest component. Therefore, we have identified the HMI and, in particular, the access to the data corpus, as the most critical component security-wise. Accordingly, the state-of-the-art solutions for encryption and user authentication have been put in place, as described in Section 4. However, emerging technologies invariably carry new threats: this is true, in general, for smart city technologies [35] and, in particular, for edge computing environments [36]. The design of the DMP adopts a neutral perspective towards such issues, since we assume that the best practices are being followed at every step of the deployment of the components involved, both internally and externally. Such an approach may not be sufficient for all use cases depending on the threat models and local regulations, hence it may have to be revisited before going to production. Reliability. In practice, most smart city services run for a very long time (years) after their initial deployment. Therefore, it is very likely that during their lifetime they will experience all kinds of difficulties, from hardware failures to network disconnections, from power outages to system-wide upgrades. Most individual components of the DMP natively support redundancy and are very resilient to sudden unexpected behaviours because they are based on mature and industry-grade software. However, guaranteeing availability was not one of the objectives of the H2020 MARVEL project, which aimed at achieving a prototype-level technology readiness. Long-term evolutions of the DMP based on serverless computing can help increasing the reliability of the framework as a whole; we elaborate more on this in Section 8 below. Scalability. A common trait of most smart city developments is that it happens incrementally over time: except for those rare occasions where a new city is founded, the city councils and other local administrators usually allocate a (variable) amount of budget to be spent over a time span of many years for the installation of new services. Often, it is difficult to predict at Day 0 the final amount of resources that will be employed, thus scalability by design is important to allow for frictionless and organic growth.