# Optimization of Inventory Management to Prevent Drug Shortages in the Hospital Supply Chain
## 4 Reinforcement Learning
### Deep Q-Learning  
As shown in Figure 3, the agent takes actions depending on the state, called the policy \(\pi\), which maps the state \(s_{i}(t)\) to the action \(a_{i}(t)\). Mathematically, we express this as \(\pi_{i}:s_{i}(t)\in\mathcal{S}\to a_{i}(t)\in\mathcal{A}\). The Q-learning mechanism is used to maximize the long-term expected accumulated discounted rewards [10]. Considering the \(Q_{i}(s_{i}(t),a_{i}(t))\) value of \(\pi_{i}\) for a state \(s_{i}(t)\) and action \(a_{i}(t)\) pair, this value is calculated by the expected accumulated discounted rewards. Therefore, the policy \(\pi_{i}\) is constructed by taking the action:  
\[a_{i}(t)=\arg\max_{a\in A}Q_{i}(s_{i}(t),a_{i}(t)),\forall i\in\mathcal{I}. \tag{14}\]  
Based on the Q-function from the Bellman equation [10], the optimal policy \(\pi_{i}\) with value \(Q_{i}\) can be obtained by:  
\[Q_{i}(s_{i}(t),a_{i}(t))=E\big{[}w_{i}(t+1)+\psi w_{i}(t+2)+\psi^{2}w_{i}(t+3) +\ldots|s_{i}(t),a_{i}(t)\big{]} \tag{15}\]  
so that the \(Q\)-value for the state, given a particular state, is the expected discounted cumulative reward. Therefore, in the MDP, we aim to determine an optimal policy expressed as \(\pi^{*}:\mathcal{S}_{i}\rightarrow\mathcal{A}_{i}\). Following the stationary distribution in the MDP, the Q-values will converge to the optimal value \(Q^{*}\) with the following equivalent calculation [10]:  
\[Q_{i}(s_{i}(t),a_{i}(t))=Q^{\prime}_{i}(s_{i}(t),a_{i}(t))+\psi[w_{i}(t+1)+ \delta\max_{s_{i}\in\mathcal{S}}Q^{\prime}_{i}(s_{i}(t),a_{i}(t))-Q^{\prime}_{ i}(s_{i}(t),a_{i}(t))], \tag{16}\]  
where \(Q^{\prime}_{i}(. )\) is the old value. To obtain the optimal \(Q\)-value, the algorithm is executed until the mean changed values of the \(Q\)-values is less than a threshold, called the training phase. The details of the training phase can be described as shown in Algorithm 1. At the beginning, a random \(Q\)-matrix is generated (Line 3). A loop is executed (from Lines  
Figure 3: Markov decision process. 5 to 9) to modify this matrix until convergence (i.e., where the change is less than the threshold \(\epsilon\)). A random state \(s_{i}(t)\) is selected to start the training (Line 4). Line 6 randomly selects a possible state \(a_{i}(t)\) to move to the next state \(s^{\prime}_{i}(t)\). Line 7 updates the \(Q\)-value of the state-action pair \((s_{i}(t),a_{i}(t))\) using (16). The algorithm continues until it meets the convergence condition. ```
1 Input: \(\mathcal{S}\) and \(\mathcal{A}\);
2 Output: \(Q\)-matrix;
3 Initialization: \(Q\)-matrix;
4 Select a random starting state \(s_{i}(t)\) that has some possible actions from \(\mathcal{A}\);
5 while\(||Q-Q^{\prime}||\leq\epsilon\)do
6 Select one of the possible actions \(a_{i}(t)\) that moves to the next state \(s^{\prime}_{i}(t)\);
7 Update the \(Q\)-value of the state-action pair \((s_{i}(t),a_{i}(t))\) according to (16), then set to \(Q^{\prime}\);
8 Set \(s_{i}(t)\gets s^{\prime}_{i}(t)\);
9 Go to Step 5;
10
11 end while
```  
**Algorithm 1**Reinforcement learning. According to the traditional \(Q\)-learning method, it is not difficult to obtain the convergence result with a small state-action space. However, the classic \(Q\)-learning model cannot be applied directly to our work because the state-action space is huge, given that the state in DR2O is comprised of a tuple of parameters that generate a huge combination. Furthermore, the level of drug refilling \(x_{i}\) is set flexibly in a specific range, and so, it also increases the size of the state-action space. In this case, there are two problems: (i) it is too difficult to build the transition probability for the MDP, and (ii) some states that are not visited and that are updated infrequently lead to a long convergence in the training phase to obtain the \(Q^{*}\)-value. To deal with this problem, a Deep Neural Network (DNN) is used to approximate the \(Q\)-function [9]. Given the input information of a state \(s_{i}(T)\), the DNN is trained to learn an optimal mapping \(s_{i}(t)\) to \(a_{i}(t)\).