# Smart Grid Data Management in a Heterogeneous Environment with a Hybrid Load Forecasting Model
## 5 Results and Discussion (Case Study 1)
### Case Study 1: Data Management
#### 5.1.1 Data Quality and Design Structure  
To overcome the above limitations, this study proposes to isolate data in the transitional phase instead of data integration itself, i.e., designing two independent databases constructed of a structured database for mechanical meters and an unstructured database for smart meters until the transition phase to smart meters is fully completed. As a result, the system will fully operate using the unstructured database in the future, while the structured database will be converted into historical data. Many procedures must be completed before these data can be ready for any following processes such as data visualization and data analysis. These procedures are further elaborated as follows:  
1. Design an independent structured database to host mechanical meter data and an independent unstructured database (NoSQL) to host future smart meter data. The unstructured database will have a dynamic scheme with horizontal scalability, as it can be a document, key-value, images, or wide column store, which can be modified at any stage. This scheme will be fully dependent on the future controls that are set by the ministry of electricity requirements, where both databases are connected \begin{table}
\begin{tabular}{c c} \hline
**Language** & **English** \\ \hline Privacy & Private \\ Source and Ownership & Iraqi Ministry of Electricity (MOELC). \\ Sampling & 5,189,000 subscribers \\ Sampling After Data Cleaning & 1,445,000 Active subscribers \\ Data Collection Period & January 2019 to September 2019 \\ Database Type & CSV and Microsoft SQL Database \\ Parameters & 9 \\ Disc Size & Around 3.5 GB \\ \hline \end{tabular}
\end{table} 
Table 2: Dataset Profile Information. through the API gateway of our system. The system can access both databases with an integrated interface through the PIAS's front-end. This proposed design gives the capability to host data from various sources such as mechanical and smart meters independently but integrated into the system's back-end through API and the front end of the system using GUI. 2. Data Quality: Data cleaning and data pre-processing will be applied before any data import process. These processes will be only applied to mechanical meter data in the offline form (i.e., after manual reading and direct data feeding or digital transformation through the mobile application platform designed specifically for this purpose) or any other historical data form. This process can also be used to migrate the historical data, while the real-time data from smart meters will be a direct injection into our unstructured database through an enterprise API. Moreover, this process can remove irrelevant and obsolete data in historical data from the mechanical meters. These might include inactive accounts, missing information, closed accounts, or empty accounts from the dataset. A powerful cleaning procedure with Microsoft SQL was used to gain clean data. For the case study, a meter must fulfill the following four requirements to be considered. After the cleaning process was completed, 1,445,000 active and clean record entries of subscribers (mechanical meters) were retrieved. The following steps have been followed to ensure the data quality: * Step 1: Check if the basic account information has been updated in the last 15 years. * Step 2: Check if the account opens or active flags. * Step 3: Check and remove special characters such as \(\langle\).\(\downarrow\)\(\downarrow\)\(\downarrow\)\(  
The designed structured database has two main tables and five lookup tables, as seen in Tables 3 and 4. It includes the reading information tables linked to the account information table by a unique identifier (GUID) with (many-to-one) relationships to enable multiple readings for the same account over time. Information such as reading value is a dependent variable in our model, while the maximum temperature, minimum temperature, average temperature, amount of rainfall, humidity, and the length of daytime are the independent variables.