# Differential Learning for Outliers: A Case Study of Water Demand Prediction
## 5 Differential Learning Prediction Models
### Positive and Negative Outlier Networks  
To estimate the water demand for outlier days as a result of restrictions or under/over-consumption a differential learning model is proposed. The new model includes the reduced feature model as the baseline, along with parallel network(s) for outlier range estimation. The general configuration of the proposed model is shown in Figure 8.  
The first differential learning model investigated consisted of the 12 reduced features monthly recurrent networks (baseline network) and one single outlier network. The additional outlier network is also a recurrent network and was trained by using only the data points with an error \(\geq\pm\)10%. The baseline network was trained by using the data points that produced an error \(\leq\pm\)10%. For each predicted data point in the differential network, the error is calculated as the minimum error of either the baseline network or the outlier network compared to the actual consumption level. With this configuration, the error improved for some of the over-consumption days. However, the error increased for under-consumption days. The average error for this model was 2.59%. The above result indicates that over-consumption should be treated differently from under-consumption. Therefore, a new configuration was developed. This configuration consisted of the same baseline network and two outlier networks, one for positive outliers (over-consumption) and one for negative outliers (under-consumption). Moreover, for this new configuration, the baseline network was trained by using all the data points, the positive outlier network was trained by using data points with error \(\geq\) 10% and the negative outlier network was trained by using data points with error \(\leq-\) 10%. This training approach is referred to as Training Method A in the remainder of the paper. The resulting model produced an average error of 2.36% for 2011-2017 as show in Table 3. This error is lower than that of both the original and the reduced feature recurrent network models shown in the last row of Table 1.  
Other training strategies were also tested. In the first variant, the baseline network was trained by using only the data points that produced an error less than \(\pm 10\%\) (Training Method B). That is, the positive and negative outlier data points were not used in the training of the baseline network. The training of the positive and negative outlier networks was maintained as in Method A. The resulting average error for this training approach over the testing period from 2011-2017 was 2.87% (Table 3). This error is higher than the previously obtained average error where the baseline network was trained with all the data points. A third training approach (Training Method C) was also investigated in order to help avoid overfitting in the outlier networks. In this method, the training of both the positive and the negative outlier networks is delayed 50,000 epochs after the start of the training of the baseline network. Training Method C has an improved average error compared to Training Method A (Table 3). Overfitting can occur in the outlier networks because of the limited number of outlier data points [23]. Table 4 shows the number of these outliers for each month over the period 2011-2017. Most of the outliers are negative (i.e., under-consumption). Moreover, the number of positive outliers (i.e., over-consumption) during the months of July-September are much higher than the rest of the year. A review of the number of outliers over the time period of this study shows that on average, the period from 2011-2015 had more outliers than the period 2016-2017. The proposed machine learning model is sensitive to variations in the distribution of the outliers. Understanding this variation is therefore important for the development of an accurate predictive model. For example, due to the distribution of the outliers in the water demand data set, the differential learning model improved accuracy prediction for the period 2011-2015 more than for the period 2016-2017.