# Generation of Data-Driven Expected Energy Models for Photovoltaic Systems
## 2 Methodology
### Preprocessing  
Data quality issues stemming from measurement errors and system anomalous conditions (reflecting local field failures, such as communication loss) could introduce signal variations in field data that would hinder model performance. Problematic data convolute the relationships between features, making it more difficult to measure the true parameter estimates; these potential irreducible errors are decreased through numerous data quality filters (Figure 2). Missing values (i.e., NaN or None values) were removed prior to applying data quality filters. An evaluation of these missing values revealed that a majority of them (\(\sim\)88%) occurred during nighttime hours (\(\sim\)7 p.m. to 8 a.m.), indicating that some sites captured night-time entries as null (Figure 2). After removing these missing values, \(\sim\)900 K data points remained, which were then subject to a series of data quality filtering steps. Data were filtered to ensure they are within nominal sensor ranges, using thresholds following [24] and the IEC 61724-1 standard [15]. Specifically, we retained data that met the following criteria:  
* 20 \(\frac{W}{m^{2}}\leq\) Irradiance (\(I\)) \(\leq\) 1500 \(\frac{W}{m^{2}}\);
* Energy (\(E\)) \(>\) 0 kWh;
* Ambient temperature (\(T_{amb}\)) \(\leq\) 50 \({}^{\circ}\)C and module temperature (\(T_{mod}\)) \(\leq\) 90 \({}^{\circ}\)C.  
Figure 1: Geographical coverage of sites within the PVROM database. A majority of the sites are located in California. Figure 2: Data preprocessing activities included both data quality- and anomaly-related filters. Data quality filters were conducted independently; only data points that passed all quality-based filters were subject to the anomaly-based filters. Wind speed was not consistently available from partners and thus was excluded from analysis. Although available temperature data were used in the preprocessing steps, they are not used as a predictor variable in the regression models, since they are not included in current standard models [15]. Flatlining values--determined by periods where consecutive data changed by less than a threshold--were flagged for removal using the pecos package [25], which follows the IEC 61724-3 standard [26]. Specifically, four consecutive hours with either \(\Delta E<0.01\%\) of the site's capacity or \(\Delta I<10\)\(\frac{W}{m^{2}}\) were filtered. Lastly, inverter clipping, which occurs when the DC energy surpasses an inverter's DC energy rating, was addressed by mathematically observing plateaus in the energy signal using the pvanalytics package [27]. Dropping energy measurements during inverter clipping, which manifest as a static value across high irradiance levels, would create a better linear fit. After data quality checks, 429 K data points across 150 sites remained (Figure 2). Data points that passed all quality checks were also assessed for system-level anomalies. These anomalies likely reflect abnormal operating conditions (i.e., local failures) and thus require removal to ensure the trained baseline energy models reflect nominal system performance. Anomalous entries were detected using a comparison of observed energy to irradiance and site capacities (Figure 3). The comparison of observed energy and irradiance filter focuses on removing data where the E-I ratio (\(\lambda\)) is outside its nominal distribution by 3 standard deviations \(\{\lambda:\lambda<\mu_{\lambda}-3\sigma_{\lambda}\cup\lambda>\mu_{\lambda}+3 \sigma_{\lambda}\}\), where \(\mu_{\lambda}\) and \(\sigma_{\lambda}\) are the mean and standard deviation of the E-I ratio, respectively [28]. This filter was implemented for each site separately to capture site-specific variations (including system capacity) and resulted in the removal of 70 K data points (Figure 2). The second system anomaly filter focused on removing sites with mismatches between observed energy and site capacity.