## Appendix C Description of Machine Learning Algorithm for Inventory Replenishment Decision
### Forecasting and Inventory Model Parameters  
We conduct extensive sensitivity analysis before selecting hyper-parameters for machine learning models as well as classical approaches for forecasting, and the overview of parameter settings is presented below in Table 11. As we found, the cV for all three products are different, and due to that fluctuation, the parameters need to be adjusted for the model's outcome. Definitely, we do not integrate any hyper-parameter optimization tools, but in practice the operator might fix it by conducting sensitivity analysis, and this is also not a key focus of this study. For all the neural-network-based models, where an activation function is required, ReLU activation has been utilized.  
```
1:Initialize replay memory \(D\) to capacity \(N\)
2:Initialize action-value function \(Q\) with random weights \(\theta\)
3:Initialize target action-value function \(\hat{Q}\) with weights \(\theta^{-}=\theta\)
4:for episode \(=1,M\)do
5: Initialize sequence \(s_{1}=x_{1}\) and preprocessed sequence \(\phi_{1}=\phi(s_{1})\)
6:for\(t=1,T\)do
7: With probability \(\epsilon\) select a random action \(a_{t}\)
8: otherwise select \(a_{t}=\operatorname*{argmax}_{a}Q(\phi(s_{t}),a;\theta)\)
9: Execute action \(a_{t}\) in emulator and observe reward \(r_{t}\) and image \(x_{t+1}\)
10: Set \(s_{t+1}=s_{t},a_{t},x_{t+1}\) and preprocess \(\phi_{t+1}=\phi(s_{t+1})\)
11: Store transition \((\phi_{t},a_{t},r_{t},\phi_{t+1})\) in \(D\)
12: Sample random minibatch of transitions \((\phi_{j},a_{j},r_{j},\phi_{j+1})\) from \(D\)
13: Set \(y_{j}=\begin{cases}r_{j}\&\text{for terminal}\phi_{j+1}\\ r_{j}+\gamma\operatorname*{max}_{a^{\prime}}\hat{Q}(\phi_{j+1},a^{\prime}; \theta^{-})\&\text{for non-terminal}\phi_{j+1}\end{cases}\)
14: Perform a gradient descent step on \((y_{j}-Q(\phi_{j},a_{j};\theta))^{2}\) according to \(\theta\)
15: Every \(C\) steps reset \(\hat{Q}=Q\)
16:endfor
17:endfor
```  
**Algorithm A2** DQN Algorithm.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**Forecasting Parameters**} \\ \hline
**Method** & **Item-1** & **Item-2** & **Item-3** \\ \hline
**SES** & \(\alpha\):0.2 & \(\alpha\):0.2 & \(\alpha\):0.2 \\ \hline
**ARIMA** & p,d,q(2,1,0) & p,d,q(1,0,1) & p,d,q(1,0,1) \\ \hline \multirow{3}{*}{**ANN**} & Epochs: 200 & Epochs: 100 & Epochs: 200 \\  & Neurons: 80, 60 & Neurons: 80, 60 & Neurons: 100, 60 \\  & Hidden layer(s): 2 & Hidden layer(s): 2 & Hidden layer(s): 2 \\ \hline \multirow{3}{*}{**LSTM**} & Epochs: 200 & Epochs: 200 & Epochs: 200 \\  & Neurons: 100, 60 & Neurons: 120, 80 & Neurons: 64, 32 \\  & Hidden layer(s): 2 & Hidden layer(s): 2 & Hidden layer(s): 2 \\ \hline \multirow{3}{*}{**SVR**} & Epochs: 200 & Epochs: 200 & Epochs: 100 \\  & Neurons: 120, 60 & Neurons: 80, 60 & Neurons: 80, 20 \\  & Hidden layer(s): 2 & Hidden layer(s): 2 & Hidden layer(s): 2 \\ \hline \multirow{3}{*}{**RF**} & Ntree: 961 & Ntree: 1500 & Ntree: 1449 \\  & Mtry: 1 & Mtry: 1 \\  & Max\_depth: default (None) & Max\_depth: default (None) & Max\_depth: default (None) \\ \hline \multirow{3}{*}{**W-ANN**} & Epochs: 200 & Epochs: 200 & Epochs: 100 \\  & Neurons: 80, 20 & Neurons: 24, 8 & Neurons: 64, 32 \\  & Hidden layer(s): 2 & Hidden layer(s): 2 & Hidden layer(s): 2 \\ \hline \multirow{3}{*}{**W-LSTM**} & Epochs: 100 & Epochs: 200 & Epochs: 200 \\  & Neurons: 80, 40 & Neurons: 60, 24 & Neurons: 80, 40 \\ \cline{1-1}  & Hidden layer(s): 2 & Hidden layer(s): 2 & Hidden layer(s): 2 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Forecasting models parameter settings.  
Similarly, we present the parameters used for implementing Q-learning and DQN policy, for determining optimal procurement schedule as shown in Table A3. The following parameter settings are also used by [63].