## 4 Experiment
### Execution and Results  
#### 4.4.1 Comparing Stand-Alone Forecasting Performance  
The first stage of the experiment is a straightforward comparison of the forecast accuracy of the RNN and heuristic approaches. The RNN sales- and returns-forecasting models have the following main parameters (see Table 1): a single hidden layer (the former with 225 units and the latter with 150 units), a dropout rate of 10%, a mean squared error (MSE) loss function, and 150 epochs of training. As noted earlier, the sales model uses LSTM and a returns model GRU. (Including dropouts is another effective means to avoid overfitting as the dropping out of units from the network with a certain probability (rate) leads to more robust overall learning since it trains the elements to be more self-reliant and discourages excessive reliance on specific inputs.)  
Figure 4: An illustration of how the anchor-and-adjust heuristic works.  
Figure 3: Schematic representation of the unrolling through time in the RNN architecture.  
After model fit, an evaluation of the model to assess overfitting (Figure 5) shows the converging training and validation loss curves, which indicates a robust fit. The canonical overfitting behavior is when training loss decreases while the validation loss increases--the divergence is predictive of poor generalizability.  
Carrying out a partial model test to verify intended rationality [56] by embedding the forecasting routine in the system dynamics model shows that the accuracy (measured using the mean squared error metric) of RNN is 87.4% better for sales (21.4 compared to 2.7) and 81% better for returns (5.8 compared to 1.1). Figure 6 below shows the system dynamics model for returns; the sales model follows the same structure. Table 1 in Appendix A provides a complete list of the parameters for the system dynamics model.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Parameter**} & \multirow{2}{*}{**Description**} & \multicolumn{2}{c}{**Value**} \\ \cline{3-4}  & & **Sales** & **Returns** \\ \hline \multirow{2}{*}{Historical periods} & Actual historical sales or returns horizon to & \multirow{2}{*}{36 months} & \multirow{2}{*}{1 month} \\  & use for forecasting. & & \\ Forecast periods & Forecast horizon. & & \\ \multirow{2}{*}{RNN layer} & There are three built-in layers in Keras: & \multirow{2}{*}{1 month} & \multirow{2}{*}{1 month} \\  & SimpleRNN, GRU, and LSTM (the latter & & \\  & two support longer time-series sequences; & & \\  & we describe the rationale in the text). & & \\ Optimizer & Gradient method used. & & Adam \\ Layers & Depth of the neural network. & 1 & 1 \\ Number of units & Artificial neurons per layer. & 225 & 150 \\ Dropout & Regularization parameter & & 10\% & 10\% \\ Epochs & Training iterations. & 150 & 150 \\ \hline \hline \end{tabular}
\end{table}
Table 1: RNN model parameters.  
Figure 5: Loss curves for the sales and returns models.  
The graphs below (Figures 7 and 8) compare the RNN and heuristic predictions against the actual sales and returns orders. At first glance, a more significant improvement in sales-forecasting accuracy with RNN might be surprising. However, this is because the sales time series shows seasonality, but the anchor-and-adjust heuristic does not account for seasonal factors, providing a satisfactory explanation.  
As an additional sanity check, a comparison of the accuracy of the heuristic to a simple exponential smoothing procedure using the Statsmodels library [75] (with a smoothing equivalent to a delay of three months, as described earlier) shows that the MSEs are roughly the same (Figure 9). It confirms the magnitude of the local improvement indicated earlier.  
Figure 6: The partial system dynamics model for predicting returns.  
Figure 7: Comparing heuristic and RNN sales predictions.  
Figure 8: Comparing heuristic and RNN returns predictions.  
Figure 9: An exponential smoothing approach to forecasting sales and returns.