# Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka +
## 4 Data Corpus-as-a-Service
### Data Corpus Architecture  
Figure 5 represents a high-level architecture of the H2020 MARVEL project Data Corpus-as-a-Service including the previous subcomponents. The extreme-scale audio-visual recording repository where processed data are stored adapts the HBase distributed database, which is an open source non-relational distributed database modelled after Google's Bigtable that is being built and run on top of the Hadoop Distributed File System (HDFS).  
Figure 5: H2020 MARVEL project’s Data Corpus architecture.  
Based on Figure 5, the management application of the H2020 MARVEL project database is based on Zookeeeper, an open source project that provides services such as maintaining configuration information and naming and providing distributed synchronisation. The database that stores the respective metadata is based on an HBase distributed database in which a master server assigns regions to the region servers and handles load balancing of the regions across region servers by unloading the busy servers and shifting the regions to less occupied servers. In HBase, a table is both spread across a number of region servers as well as consisting of individual regions. As the tables are split, the splits become regions. The regions store a range of key-value pairs and each region server manages a configurable number of regions. The management application of the H2020 MARVEL project Corpus has been based on the Ambari project that is aimed at simplifying Hadoop management by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web user interface backed by its RESTful APIs.  
The storing of the audio/video files themselves is performed using the Hadoop Distributed File System (HDFS). It saves files on commodity machines, providing high-aggregate bandwidth across the Corpus cluster. The administration of this repository is supported by the HBase distributed database (an open source non-relational distributed database). The management application of this database is implemented by the Zookeeeper. It is an open source module that provides administrative services, such as maintaining the naming and configuration information and providing distributed synchronisation, etc. Thereupon, queries (accessing) are performed via the web interface of Ambari that offers an intuitive, easy-to-use Hadoop management solution with Web UI and RESTful APIs. The storing of the audio/video files themselves is performed by the Hadoop Distributed File System (HDFS). It saves files on commodity machines, providing high aggregate bandwidth across the Corpus cluster. The administration of this repository is supported by the HBase distributed database; the management application of this database is implemented by the Zookeeper while the queries (accessing) of the HDFS are performed via the web interface of Ambari.  
The H2020 MARVEL project data corpus data storage is based on the Hadoop Distributed File System (HDFS) fault-tolerant data storage file system [27]. There are various tools and solutions available to secure the HDFS environment and each of them has different features and effectiveness under various contexts. The security tools and solutions can be divided into four categories [28; 29], which are (i) encryption, (ii) authentication, (iii) authorization, and (iv) auditing.