# Optimization of Inventory Management to Prevent Drug Shortages in the Hospital Supply Chain
## 4 Reinforcement Learning  
### Markov Decision Process Model  
Firstly, we present the Markov Decision Process (MDP) to formulate our problem. Based on this model, we then propose a deep Q-network algorithm to find a solution for DR2O. In general, the MDP model is comprised of three concepts: a state, an action corresponding to a state, and a reward for that action.  
**The system state \(\mathcal{S}\)**: We consider the system state at time instant \(t\), including the current volume of drugs, the user demand, and the costs of buying and storing medicine. In addition, due to specific requirements at time \(t\), some other information could be changed, such as the purchasing budget, the storage capacity for drug \(i\), and the minimum and maximum amount of drugs to refill. For the set \(\mathcal{I}\) of drugs \(i\), we could make an order to make a decision for the drugs that have higher priority and then consider the remaining budget for the rest. Therefore, we make a loop with \(|\mathcal{I}|\) iterations to make a decision for each drug \(i\) in turn. We denote the state \(s_{i}(t)=<p_{i}(t),r_{i}(t),\lambda_{i}(t),B_{i}(t),e_{i}(t),\underline{\zeta }_{i}(t),C(t),\underline{\rho}_{i}(t),\bar{\rho}_{i}(t)>\) of drug \(i\) at time \(t\).  
**Action set \(\mathcal{A}\)**: The action set in our model reflects the refilling decision of each drug \(i\). In particular, the action \(a(t)=\{a_{i}(t)\}_{\forall i\in\mathcal{I}}\), where \(a_{i}(t)\) is the decision of drug \(i\) at time \(t\) that responds to the state \(s_{i}(t)\).  
**Reward \(W\)**: As presented for the objective function, our model aims to minimize the refilling cost including the purchasing and storing costs. Therefore, we first model the reward function based on the purchasing cost as follows \(\frac{1}{L_{i}(x_{i}(t))}\), where \(L_{i}^{(1)}(x_{i}(t))=\alpha_{i}p_{i}(t)x_{i}(t)+R_{i}(v_{i}(t),r_{i}(t))+ \psi(v_{i}(t))\), and \(k\) is the considered timeslot. This means that the more money the system needs for its purchases, the less reward it has. The function attempts to navigate the system, so it selects actions that obtain higher rewards, which is equivalent to minimizing the purchasing cost.  
The next term involves the penalty for the shortage situation, as we aim to avoid this problem for all the drugs in storage. The penalty term is defined by \(\psi(v_{i}(t))\) as formulated above.  
Figure 2: An example of the offline Dynamic Refilling dRug Optimization (DR2O).  
We combine these terms in the penalty function by:  
\[w_{i}(t)=(1-\theta)\frac{1}{L_{i}(x_{i}(t))}+\theta\psi(v_{i}(t)), \tag{12}\]  
where \(\theta\in(0,1]\) is the weighted parameter that is designed to set the priority of each term.  
Considering duration \(T\), the state of each drug and its reward are stochastic and follow the MDP, where the state \(s_{i}(t)\) changes to \(s_{i}(t+1)\) with a transition probability, and the reward depends on the state and the selection action.  
To go from \(s_{i}(t)\) to \(s_{i}(t+1)\) with reward \(w_{i}(t)\), we consider the conditional transition probability, \(p(s_{i}(t+1),w_{i}(t)|s_{i}(t),a_{i}(t))\). It should be noted that the agent can only control its own actions and has no prior knowledge about the transition probability matrix \(P=p(s_{i}(t+1),w_{i}(t)|s_{i}(t),a_{i}(t))\), which is determined by the environment. The intuition of the MDP is presented in Figure 3. Therefore, the main objective of this reinforcement learning is to find a policy to maximize the expected cumulative reward. We have:  
\[R_{i}=E\big{[}\sum_{t=k}^{T}w_{i}(t)\big{]} \tag{13}\]