## 4 Experiment
### Execution and Results
#### 4.4.2 Comparing Overall System Performance  
The second and most crucial stage involves evaluating the impact of the forecasting intervention (use of RNN) in its proper context by including relevant aspects of Acme's overall forecasting, order fulfillment, and extended production processes. A decision parameter in the whole-model simulation (Figure 10a) determines the source of predictions (for sales and returns)--heuristics (or judgmental) or RNN. If set, PySD substitutes a hook in the model with a function that makes online predictions that integrate with the rest of the model (Figure 10b). Otherwise, judgmental or heuristic predictions take effect (Figure 10c). Acme follows a make-to-stock strategy, implying that it fulfills orders from inventory. Here (order fulfillment sector in the figure below), the system dynamics model includes simple rules to satisfy orders as they come in and to ascertain backorders and lost sales if there is a shortage--in other words, when the forecast is inaccurate. If the delivery lead time exceeds the goal, delivery pressure (a function of backlog) builds up, resulting in lost sales if the delay exceeds the tolerance limit (Figure 10d). Production orders are simply the difference between forecasted sales and returns in the production sector. For simplicity, the experiment assumes a negligible production lead time (a few days) relative to the planning periodicity (of months). At the end of the month, the forecasts generate production orders, assumed to be available as inventory at the start of the following period. The decision rules discussed thus far are operational (in their evolutionary theory of economic change, [PERSON] and [PERSON] use the term \"operating characteristics\" to characterize such rules [43]). However, there is a routine in the capacity management sector that is at  
Figure 10: Whole simulation model. (**a**): Choice of forecasting procedure. (**b**): RNN forecast—a “hook” that is programmatically substituted with online predictions via PySD. (**c**): Judgmental forecast. (**d**): Delay tolerance. (**e**): Capacity adjustment under delivery pressure. (**f**): Capacity ramp-up and ramp-down delays. a higher level, typically considered tactical by supply chains. It is a routine that calibrates the available capacity and is a crucial determinant of overall performance. The routine is responsible for augmenting capacity under delivery pressure (Figure 10e). Augmentation increases the capacity at a rate defined by the ramp-up delay (Figure 10f) to a maximum capacity determined by the available discretionary capacity. On the other hand, capacity normalization happens as the pressure eases. The equation for delivery lead time overshoot captures the easing of delivery pressure. This is the difference between the goal delivery lead time and the delivery lead time outlook (the ratio between the backlog and the current clearance rate). As the overshoot moves close to zero--all other things being equal--the normalization rule down-regulates the available capacity until it reaches the standard available capacity. As all the rules in the whole-model simulation, except the source of predictions, are the same, verifying the effect of improved forecast accuracy on the overall performance is allowed. In an ideal scenario--perfect forecast accuracy--the lost sales are zero, and the average inventory equals half of the average production orders (since the planned production is available at the start of the period and the consumption of inventory by sales is assumed to proceed at a constant rate). Thus, lost sales and average inventory are the outcome metrics--closer to actual business performance--that provide a window into how well the process metric (forecast accuracy) translates to improved performance, seen holistically.