# Long-Term Forecasting Potential of Photo-Voltaic Electricity Generation and Demand Using R
## 2 Materials and Methods
### Data Cleansing  
After the initial cleansing and averaging of the per second micro-grid data as per [9], additional cleansing steps were required. The JMA weather data similarly required cleansing. The cleansing of both datasets are described below. In all cases of missing data, the entire row of data was removed so as not to lead to invalid correlations.  
1. Removing missing values from the micro-grid data and replacing negative values with 0. For the micro-grid, this concerned the maintenance days in November (2015: November 14, 8:00-20:00, 2016: November 19, 8:00-19:00, 2017: November 18, 8:00-18:00). Two additional instances were found to have missing values (2015/11/13 12:00, 2017/09/25 21:00). For solar power generation, 269 negative values were recorded during nighttime in 2017, which were replaced with 0.
2. Removing JMA data with poor data quality. The JMA has excellent record keeping concerning data quality. Most values have the data quality value of \"8: There are no missing data on the statistical basis (normal value)\". The number of instances when the data quality was less than this are described in Table 2. Values with data quality values of 4 (10 instances) or 5 (2 instances) were kept. There were several instances where data was missing for multiple parameters at the same time.
3. Removing outliers (see SM1: R code for outlier removal). Note that the parameter (relative) humidity is in % unit, and sunlight is relative to one full hour (1), and therefore these parameters should not have outliers removed. The hourly electricity price is given based on predetermined conditions and cannot have outliers. Precipitation occurs intermittently and in irregular amounts, meaning it is prone to outliers. This leaves three weather parameters and the two micro-grid parameters to remove outliers for. For the parameters \"solar irradiation\" and \"total pv ac\" outliers were calculated without taking the values of 0 into account. Extreme outliers were removed by calculating three times the interquartile range (iqr) using the following code: extreme_threshold_upper = (iqr * 3)+ upperq (1) extreme_threshold_lower = lowerq - (iqr * 3) Boxplot results were used to show the difference before and after outlier removal. As an example, Figure 2 shows the data before and after removing outliers for the parameter wind speed, using data from 2015. Outliers were checked for overlap between parameters and consequently removed. Before outlier removal the standard deviation, mean, minimum and maximum valueswere calculated for the five parameters: wind speed, temperature, solar irradiation, total pv ac, and purchased electricity, for each year. Table 3 gives an overview of the found outliers.
4. Removing additional outliers occurring around maintenance days. After removal of the values in steps 1-3, we observed several remaining uncharacteristic data values occurring around the maintenance days. As example, from November 13 to 16 in 2015 the values for purchased electricity from the micro-grid were uncharacteristically low. The cause of these low values is most likely the shutdown of electricity dependent equipment in anticipation of and in reaction to the maintenance procedures, during which all electricity through the main power line is stopped. This led to the removal of 47 rows of data in 2015; 47 in 2016 and 49 in 2017. After this final removal step the standard deviation, mean, minimum and maximum values were calculated for the five parameters wind speed, temperature, solar irradiation, total pv ac, and purchased electricity, for each year.