## 6 Experimental Evaluation
### Results on Alerts Forecasting  
Once the notices were reconstructed, we grouped them by component. Each group was reordered in ascending chronological order. In this way, each group corresponded to a data structure that we generated that was used for the predictive models. For each component we have analysed, we split our process into two parts:  
* Extraction of the time series of faults and creation of training data and test data;
* Predictive phase. We have already discussed the first part in Section 5.4, where we have also mentioned the three models that we have used for the forecasting phase. We are targeting a regression problem, and the prediction we will obtain is represented by a number corresponding to an elapsed interval time. This is considered good and reliable from ITALTELECO if it is less or equal than three months from the real value. Therefore, in the rest of this section, this will be our reference threshold. To let the reader understand the process of our evaluation, let us consider the RT2 MP4-P component, which corresponds to a pump. For this component, there are 43 warnings relating to faults. These were transformed into a vector of dates ordered chronologically. This vector was then divided into two parts. * A total of 80% of the data (corresponding to 34 faults for the example above) became the training vector;
* The remaining 20% (corresponding to 8 faults) were used for the test phase. The training vector was fed to the three predictive models introduced in Section 5.4. Figure 8 shows the forecast of the eight elements in the test set performed by the linear  
Figure 6: Frequency distribution of the fault category with respect to the notices related to compressors. Figure 7: Frequency distribution of the fault category with respect to the notices related to pumps. regression. The gray points represent the training data, the red point is the forecasted element, and the blue element is its corresponding real value. The reader notices that the red point at step \(i\) becomes a new training element in the next iteration (the last gray point in the graph at step \(i+1\)). Moreover, the slope of the forecast line, shown in green, changes as we add new elements in the training data as the course of the line tends to follow the real data. The errors are computed as the absolute difference on the \(y\)-axis between the forecasted point (red) and the real one (blue). Related to Figure 8, Table 12 illustrates the value of the \(y\) coordinates of the eight points of the test set, the predicted value using the linear regression model, and the error. We applied the three regression models to the alerts of 181 system components, and the results are shown in Table 13. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**\#Step** & **y\_TEST** & **Predict Value LR** & **Error LR** \\ \hline step 0 & 6510 & 6413.62521 & 96.37 \\ \hline step 1 & 6578 & 6575.287302 & 2.71 \\ \hline step 2 & 6611 & 6726.968468 & 115.96 \\ \hline step 3 & 6744 & 6866.16074 & 122.16 \\ \hline step 4 & 7129 & 7004.561404 & 124.43 \\ \hline step 5 & 7136 & 7167.465385 & 31.46 \\ \hline step 6 & 7249 & 7315.310976 & 66.31 \\ \hline step 7 & 7375 & 7459.801394 & 84.80 \\ \hline _Average error_ & & & 95.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Results table of the RT2 MP4-P component performed by the linear regression Model. Figure 8: Step by step predictions performed by the linear regression model of the RT2 MP4-P component. The overall average error is very high: more than 647 days for the linear regression, more than 418 days for ARIMA and more than 476 days for the polynomial integration method. This phenomenon is due to the large amount of components having a very small number of notices. In fact, we analyzed 181 out of the overall number of 1,036 components because of the insufficient number of samples found in the \"Notice\" data to be applied to the models. In particular, we processed components with a number of notices higher than 3 for the linear regression and the polynomial integration method.