# Applications of Probabilistic Forecasting in Demand Response
## 2 Materials and Methods
### Evaluation of the Probabilistic Forecast  
Point predictions (deterministic approach) are easy to evaluate by measuring the deviations between the real demand and the forecast. However, assessing probabilistic predictions is a difficult task. For example, [30] identifies some relevant properties of a probabilistic forecast: accuracy, calibration (reliability), and sharpness. Accuracy refers to the same concept as in point prediction, thus the same measurements such as _MAE_ or _RMSE_ can be used, defined as follows:  
\[RMSE=\sqrt{\frac{1}{N}\sum_{t=h_{1}}^{t=h_{2}}(y_{t}-\hat{y}_{t})^{2}} \tag{1}\]  
\[MAE=\frac{1}{N}\sum_{t=h_{1}}^{t=h_{2}}|y_{t}-\hat{y}_{t}| \tag{2}\]  
where \(y_{t}\) is the real value, \(\hat{y}_{t}\) is the point forecast, \(h_{1}\) and \(h_{2}\) the boundaries hours of the test period, and \(N\) the overall number of hours in that period. It is said that a probabilistic prediction is calibrated if it seems to come from the same distribution as the observations. Two useful tools to check this property are the Prediction Interval Coverage Probability (_PICP_) and the Verification Rank Histogram (VRH). Recall that probabilistic predictions can be provided by means of prediction intervals. For a given coverage probability, \(\alpha\) in (0, 1), we look for an interval that will contain the future prediction with probability \(\alpha\). In this context, the predicted quantiles play an important role in obtaining those prediction intervals. For example, the prediction interval at time \"\(t\)\" with probability \(\alpha\) in (0, 1) is given by the following:  
\[PI_{t,\;\alpha}=\left(\hat{Q}_{t,\frac{1-\alpha}{2}},\hat{Q}_{t,\frac{1+\alpha }{2}}\right) \tag{3}\]  
where \(\hat{Q}_{t,\beta}\) denotes the predicted quantile \(\alpha\), time \"\(t\)\", with lower-tail probability \(\beta\). For the test dataset and given a fixed \(\alpha\) in (0, 1), we can count the number of observations (real demand of the test dataset) that lay inside the prediction intervals at level \(\alpha\), and this quantity is used to calculate the _PICP_, which is given by the following:  
\[PICP_{\alpha}=\frac{1}{N}\sum_{t=h_{1}}^{t=h_{2}}\mathbf{1}(y_{t}\in PI_{t,\; \alpha}) \tag{4}\]  
where \(\mathbf{1}_{(. )}\) denotes the indicator function. Note that \(PICP_{\alpha}\approx\alpha\) for calibrated models, so a linear equation of the form \(y=x\) must follow when plotting _PICPs_ against \(\alpha\). An alternative way to contrast whether probabilistic forecasts are calibrated is through the _VRH_. For each observation \(y_{t}\) in the test dataset, we compute its rank related to the predicted quantiles at that time \"\(t\)\". Under calibration, all the positions must appear with a similar frequency. Therefore, the corresponding histogram (VRH) should be approximately flat. It is important to highlight that a flat VRH is a necessary but not sufficient condition for calibration (see [31]). The property of sharpness is related to the width of the prediction intervals. The widths can be analyzed using boxplots; however, it is preferable to use objective indices. [PERSON] et al. [32] propose computing the Prediction Interval Normalized Width (_PINW_) and the Prediction Interval Normalized Average Width (_PINAW_), given by the following:  
\[PINW_{t,\;\alpha}=\frac{\hat{Q}_{t_{1}\frac{1+\alpha}{2}}-\hat{Q}_{t_{1}\frac{1 -\alpha}{2}}}{\frac{1}{N}\sum_{t=h_{1}}^{t=h_{2}}y_{t}} \tag{5}\]  
\[PINAW_{\alpha}=\frac{1}{N}\sum_{t=h_{1}}^{t=h_{2}}PINW_{t,\;\alpha} \tag{6}\]  
The lower the _PINAW_, the higher the sharpness. However, sharpness can be increased at the cost of losing calibration. Thus, the aim is to minimize the _PINAW_ while keeping calibration. The Continuous Rank Probability Score (_CRPS_) is a measure that can be used to evaluate jointly the three properties (accuracy, calibration, and sharpness) [33]. It is widely used in weather forecasting, and it reduces to _MAE_ for deterministic forecasts. The lower the _CRPS_, the better the probabilistic forecast.