## 6 Performance Evaluation
### Methodology  
For experiment purposes we considered the top 100,000 Amazon products with the highest number of _prices_ entries. We used three different _test sizes_, 10%, 20% and 30% related to the prices entries of each product. _Test size_ reflects the number of predictions made. Thus, for a given product having a time series of 100 prices elements, if we have a _test set size_ of 10%, we will use the first 90 elements to train the model, and will perform the forecast on the last 10 elements left calculating the related _MAPE_. We calculated the prediction for the other two test sizes (20% and 30%) similarly. For each percentage, we have used the time series cross-validation function offered by scikit-learn Python library and detailed here ([[https://towardsdatascience.com/time-series-nested-cross-validation-76](https://towardsdatascience.com/time-series-nested-cross-validation-76) adba623 eb9]([https://towardsdatascience.com/time-series-nested-cross-validation-76](https://towardsdatascience.com/time-series-nested-cross-validation-76) adba623 eb9)) and in compliance with research work performed by authors in [55]. We have used several combinations of features as exogenous variables for products and in Table 1 we indicate how many times a given feature has been found within the 100,000 Amazon products of our analysis. \begin{table}
\begin{tabular}{c c c} \hline
**Extracted Features** & **Occurrences** & **Percentage** \\ \hline _Manufacturer_ & 36k & 36\% \\ _Sentiment_ & 55k & 55.2\% \\ _Manufacturer, Google Trends_ & 23k & 23\% \\ _Manufacturer, Sentiment_ & 33k & 33.1\% \\ _Manufacturer, Google Trends, Sentiment_ & 21k & 21.1\% \\ \hline \end{tabular}
\end{table} 
Table 1: Occurrences of extracted features within 100,000 collected products and their percentage. We recall that we used the manufacturer exclusively to extract the Google Trends time series of the products. Therefore the manufacturer has not been used as an exogenous variable for the ARIMA model whereas Google Trends has. Also, the reader might notice how the co-occurrences of multiple features (e.g., _(manufacturer, trend)_ or _(manufacturer, sentiment)_) is lower than that of single features. In Table 2 we include the MAPE values for each combination of exogenous variables we used for the three different test sets size. In most of the cases, decreasing values for the _test size_ corresponds to more accurate scores of the prediction. In particular, we can observe that (_price, date, Google Trends_) combination (that is, Google Trends represents the single exogenous variable) is the most promising setting, having the lowest _Average MAPE_. We have carried out a statistical test using the Diebold-Mariano test [56] ([[https://www.real-statistics.com/time-series-analysis/forecasting-accuracy/diebold-mariano-test/](https://www.real-statistics.com/time-series-analysis/forecasting-accuracy/diebold-mariano-test/)]([https://www.real-statistics.com/time-series-analysis/forecasting-accuracy/diebold-mariano-test/](https://www.real-statistics.com/time-series-analysis/forecasting-accuracy/diebold-mariano-test/))) which confirmed that the MAPE results we obtained using Google Trends and Sentiment as exogenous variables are better than those obtained using just Sentiment which is, in turn, better than the MAPEs obtained using Google Trends as exogenous variable. In support of that, in Table 3 we have highlighted the number of products out of 100,000 having the manufacturer information, how many manufacturers have Google Trends information and how many times the tuple (_price, date, Google Trends_) has been the best combination among the others for our three test dataset size. The second and third columns are related to the entire collection of 100,000 products and, therefore, they have the same number of the three datasets. The last column indicates how often \((price,date,trend)\) combination has been the best with respect to the others. For the test size of 10% Google Trends as exogenous variable has been found the best combination for 61% of the total products.