# Big Data in Smart City: Management Challenges
## 3 Materials and Methods
In the process of data reduction, there are components of HDFS (Hadoop distributed file systems) that break objects into smaller sub-elements, which are located on clusters of network systems. By creating blocks, distribution is performed through the cluster, and in that way, the source data are reduced, and appropriate mapping is performed in the system. The basic abstract phenomenon defined in the Apache Spark environment is the existence of a resilient distributed dataset (RDD), which is a collection of elements that are partitioned through cluster nodes and mapped into this structure, which is unique and provides the possibility of parallelization. In the GAMINESS environment, RDD is created from the Scala collection of input data mapped to the Apache Spark structure. The Scala has full support for functional programming and a very strong system of static types. This allows programs written in the Scala to be very concise and thus smaller in size than other general purpose programming languages [53]. The system enables the persistence of RDD in the system memory, which provides the possibility of reusing the mentioned RDD through parallel operations. The use of such reduced system files results in scalability and shortening of data processing time, which is one of the biggest contributions in structuring smart city data in this model. With addition of the geospatial component in this management system, smart cities can include different geospatial data that are in different structures, e.g., data collected from geotechnical sensors. In Section 4.3, an example of using the temperature sensors is presented. Measured values are mapped into elements which are part of the geosensor in the distributed environment based on the Apache Spark framework. The system provides reduction and scalability of the three-dimensional space to a one-dimensional array while preserving the spatial location of points with corresponding attributes regardless of their position. This provides the possibility of conection and management of a large number of different sensors with a huge input of data in a small space. Experimental Platform Based on Apache Spark Framework in the Process of Creating the Smart City Management System Model  
Apache Spark is built on the resilient distributed dataset (RDD) abstraction, which distributes datasets in the distributed memory of a cluster. RDDs are resilient to lost tasks or hosts due to their functional programming approach. Apache Spark provides a language-integrated Scala API enabling the expression of programs as data flows of transformations (e.g., map, filter) on RDDs. RDDs are composed of rows of objects and have additional schema information about the data types of each column (19). The collection of RDDs in Apache Spark is defined as DataFrames. RDDs can be explicitly cached by a programmer in memory or on a disk by workers. Fault tolerance is provided by recomputing the sequence of transformations for the missing partition(s). Apache Spark provides four high level libraries: streaming for micro batch processing, GraphX for graph computation, MLlib for machine learning, which was used by [PERSON] and [PERSON] [31] for lidar point cloud classification, and Spark SQL. Spark SQL, as Spark's structured data and relational processing module, supports a subset of SQL [54]. Spark SQL as Apache Spark library works with logical and physical relational operators. In this framework, in the context of variety advancement, there is a need to define new types which are going to exceed default as user defined types. User defined types can be used to define the DataFrame schema, and they can be used by Apache Spark to optimize query execution [27; 55]. For this reason, in this paper, the Apache Spark framework was selected to develop a model. The model is in accordance with [27]. In order to describe data with all attributes together with smart city elements, a conceptual model of GAMINESS was developed. In the conceptual model, CityGML standard was extended to implement IndoorGML standard and IoT standard.