# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
## 4 Data Corpus-as-a-Service
### Privacy Assessments for Data Corpus  
In general, the datasets that are stored in the Data Corpus are meant to become public. Therefore, those data have been anonymised by the rest of the H2020 MARVEL project platform components (e.g., VideoAnony). Nevertheless, there can exist datasets that are in progress and/or the data owner (pilot) has not yet realisd/authorised their public use. Three dissemination levels are considered. The public datasets can be viewed by everyone (including external users). The private datasets can be viewed only by the H2020 MARVEL project partners. The in-progress datasets may not have been anonymised yet and are visible only among the consortium members. The data owners (pilots) can set the visibility scope as they are processing their datasets. Next, upon user registration and login in, the platform, the authentication/authorisation policy for a user type is implemented by the above-mentioned security mechanisms. Apart from those built-in controls, the Assurance Platform [33] has been deployed in the H2020 MARVEL project backend. This platform can additionally perform a security/risk analysis for the Corpus components, as well as penetrate testing elements. Moreover, the Monitoring Assessment Profiles can be developed in order to continuously monitor the real-time operation of the system in terms of security and/or privacy. For each profile, a set of Event Captors gather information from the system, such as event logs. The main capturing mechanisms are implemented with Elastic Beats, while customised events captors are also supported. These sensing mechanisms are sending information back to the platform and its Monitor subcomponent, which analyses the events with a rule-based logic (implemented in the Drools rule engine). Machine Learning (ML) evaluations can also be supported for cases that require more complex analysis. Henceforth, security and/or privacy assessments can be performed. Two assessment examples are considered so far:  
* Availability of service: Event captors are periodically checking wherever a service is up or down. In the later case, the monitoring module can calculate metrics, such as the average downtime time, the mean time to repair/restore, or the mean time to respond to an incident. Thus, the Assurance Platform can assess the availability of the internal Hadoop/HBase services as well as the web-based Graphical User Interface (GUI) that is offered to the external users.
* User access and Data Ownership: Event captors can collect the user-related actions from logfiles. Therefore, the Assurance Platform can assess the applicability of privacy controls for accessing the data. For example, the platform can recognise wherever private or in-progress datasets have been viewed/downloaded by an unauthorised external user (an attacker that gained access to the system) and raise an alert to the system administrator. The platform can also measure the number of violations in the case of such an attack, which may also reflect the time that the administrator required to fix/mitigate the problem (i.e., block the malicious or compromised user account). Similarly, the platform can audit the processing of aspects for the datasets, i.e., wherever an unauthorised user account managed to ingest new data or update/delete existing datasets.  
The rulesets for the deployed assessment profiles form the basis for the Service Level Agreement (SLA) driven verification of the system. The aforementioned metrics (e.g., mean time to respond, number of violations, etc.) can be utilised with thresholds specifying the fair use of the system.