# Optimization of Inventory Management to Prevent Drug Shortages in the Hospital Supply Chain
## 4 Reinforcement Learning
### Training and Testing Phases  
We present the training algorithm in Algorithm 2, which is called the Deep Reinforcement Learning approach for Drug inventory (DRLD). Similar to the standard \(Q\)-learning algorithm, an action of DRLD is selected based on the environment and the reward. Instead of using a full history as the \(Q\)-learning method to take an action for a current state, we limit the length of history defined by \(\gamma(.)\), which is related to the number of input nodes in the DNN. In the training phase, we use \(\epsilon\)-sampling to generate and collect the data to train the weight parameter of the DNN.  
The details of the algorithm are as follows. At the beginning, we initialize the current simulation environment with given sets of \(\mathcal{S}\) and \(\mathcal{A}\) (Line 1) and initialize for the learning sequence set \(\mathcal{D}\) and the DNN model (Lines 3-4). We select a random starting state \(s(0)\) from the current \(\mathcal{S}\) (Line 6) to execute \(M\) training times. An action \(a(t)\) is selected with an \(\epsilon-\)probability; otherwise, an action that maximizes \(Q(s(t),a(t))\) is selected (Line 8). According to the action \(a(t)\), the environment is set by a new state \(s(t+1)\) with a reward \(r(t)\) (Line 9). If the state \(S(t+1)\) does not exist in \(\mathcal{S}\), it will be added for training (Line 10). Furthermore, \(\mathcal{D}\) is appended by the transition \(s(t)\gets s(t+1)\) (Line 11). The training in the DNN is started by sampling a sequence \(s(j)\to s(j+1)\) in \(\mathcal{D}\) (Line 12). The training phase is designed to update the weight parameters in DNN with the given state \(s(j)\) and the output that is calculated by (18) (Lines 13-14). Note that the values of \(M\) and \(T\) are defined by our experience (e.g., \(M=7\) and \(T=30\)). These values are changeable and can be observed depending on each system. However, technically, while setting the values of these parameters higher will increase the accuracy, in fact, it will result in a long convergence since it adds more iterations.  
In the testing phase, the trained agent will select an action \(a(t)\) with the maximum \(Q\)-value given by the training phase. Based on the training \(Q\)-matrix, we can operate the drug refilling model, DR2O, as an online mechanism.  
```
1 Input: Start the environments \(\mathcal{S}\) and \(\mathcal{A}\);
2 Output: \(Q\)-matrix;
3 Initialization: \(Q\)-matrix, DNN model;
4 Set empty for the learning sequence \(\mathcal{D}\);
5 forj=1:Mdo
6 Select a random starting state \(s(0)\);
7 fort=0:T-1 do
8 Sample an action \(a(t)\) with probability \(\epsilon\); otherwise, select \(a(t)=\arg\max Q(s(t),a(t))\);
9 Generate a next state \(s(t+1)\) and reward \(r(t)\);
10 Add \(\mathcal{S}=\mathcal{S}\cup s(t)\);
11 Add the transition \(s(t)\to s(t+1)\) to \(\gamma(s(0)\to s(t+1))\), and save in \(\mathcal{D}\);
12 Sample a transition \(\gamma_{j}(s(j)\to s(j^{\prime}))\) from \(\mathcal{D}\);
13 Set the output \(y(k)\) by (18) to train the DNN;
14 Using a gradient descent to update weights of the DNN;
15
16 end for
17
18 end for
```  
**Algorithm 2**DRLD- training algorithm.