## 2 Methods
### Technical Details
#### 2.4.3 Methods for Inventory  
Over the years, many replenishment decisions have been proposed for defining the replenishment strategy under time-varying demand. Definitely, stochasticity is an issue, but in this study, we focus on replenishment decisions under time-varying demand. One of the simplest ways to decide a replenishment value is the implementation of Economic Order Quantity (EOQ), based on aggregated demand level for a fixed planning horizon. Very recently, data-driven approaches are also being tested. We made the following assumptions when we implemented various replenishment policies: (i) demand is considered to be time-varying, (ii) capacity of the supplier is unlimited, (iii) for each order, the sponsor company needs to incur a fixed ordering cost, (iv) the holding cost is considered as constant, the purchase cost, and sales price remain the constant during the evaluation period and (v) the lead time is constant for each product and remains the same throughout the evaluation period. EOQ: The economic order quantity (EOQ) model is a simple fixed order quantity model that minimizes the holding and setup cost, which are both associated with carrying and procuring inventory on an annual basis [48]. However, the disadvantages in the context of our study is that it fails to capture the time-varying nature. Q-Learning Algorithm: Q-Learning is a class of reinforcement learning, which gain growing popularity to semi-automatised inventory replenishment decision in recent years. The algorithm operates as an off-policy, which means the optimal policy is obtained independently of the agent's interactions with a given environment. In Q-learning algorithms,the decision-maker needs to set the following parameters: \(\alpha\), the learning rate and \(\gamma\), the discount factor to optimize the model's performance [49]. If we assume \(Q(S,A)\) represents the Q-value for a given action, A, and the state, S. The algorithm updates the Q-value using a learning rate \(\alpha\), the reward function R, which is obtained by taking an action, the discount factor \(\gamma\), and the maximum Q-value of the next state-action pairs. The reward function defined in the Q-learning and DQN is as follows:  
\[Total\ profit(TP)=<\text{sales revenue}>-<\text{holding cost}>-<\text{ purchase cost}>- \tag{1}\]  
\[<\text{ordering cost}>-<\text{ssalvage cost}>\]  
Note that if the product is non-deteriorating, we exclude the salvage cost. \[Q(S,A)\leftarrow(1-\alpha)Q(S,A)+\alpha[R+\gamma maxQ(S^{\prime},A)-Q(S,A)] \tag{2}\]  
where \(R\) represents reward, which is problem-specific. For instance, when we implement th algorithm, we use the reward function as presented in Appendix B. Note that we use the following cost parameter, as presented in Table 2, to make inventory planning. Based on the literature, the main advantages of Q-learning are the algorithm's ability to learn from past experience and that it can be adjusted to prioritize long-term or short-term gain. The disadvantage of Q-learning is the algorithm becomes inefficient when interacting with large scale environments and the algorithm is sensitive to hyper-parameter adjustment. Deep Q-Network (DQN): As we mentioned earlier, although Q-learning is extensively used in various decision-making contexts, we also introduce DQN, which is more robust in terms of computational efficiency and capabilities [50]. DQN incorporates neural networks by utilizing neural networks' ability to approximate the optimal action-value function. The neural networks perform this function by minimizing a loss value.