# Adaptive Supply Chain: Demand-Supply Synchronization Using Deep Reinforcement Learning
## 3 Methodology
### Supply Chain Environment  
SCE can be naturally formulated as MDP. MDP serves as the mathematical foundation and straightforward framework for goal-directed learning from interaction with a virtual environment. An RL agent interacts with an environment over multiple time steps (Figure 1). The RL agent and environment interact at each instance of a sequence of discrete time steps \(t\in T\). At each time step \(t\), the agent receives a representation of the environmental state \(S_{t}\in S\) and numerical reward \(R_{t}\in R\) in response to a performed action \(A_{t}\in A\). After that, the agent finds itself in a new state \(S_{t+1}\). The sequence of states, actions, and rewards gives rise to a trajectory of the following form \(S_{0}\), \(A_{0}\), \(R_{1}\), \(S_{1}\), \(A_{1}\), \(R_{2}\), \(S_{2}\), \(A_{2}\), \(R_{3}\),..., \(S_{n}\), where \(S_{n}\) stands for the terminal state [21]. At each time step, the agent conducts multiple trials with the stochastic environment (episodes). The agent observes \(S_{t}\) and \(R_{t}\) performing actions under a policy \(\pi\) that maps states to actions. In such settings, the agent's goal is to learn a policy that leads to the highest cumulative reward. In this regard, RL can be considered a stochastic optimization process [22]. The function \(p(. )\) describes the probability of an RL agent finding itself in a particular state \(S^{*}\) and obtaining the particular reward \(R^{*}\), when taking action \(A^{*}\) in state \(S_{t}\). This dynamics within the MDP framework can be described by the following equation:  
\[p(S^{*},\ R^{*}|S,\ A^{*})=Pr\{S_{t+1}=S^{*},\ R_{t+1}=R^{*}|S_{t}=S,\ A_{t}=A ^{*}\} \tag{1}\]  
Figure 1: Agent–environment relations in a Markov decision process. In SCE, the agent must decide how many goods to order for each stage \(m\) at each time step \(t\). The action \(A_{t}\) is an integer value corresponding to each reorder quantity at each stage along the supply chain. The state \(S_{t}\) constitutes a vector that includes the inventory level for each stage and the previously taken actions. Thus, the RL agent attempts to synchronize the supply chain so that the total revenue over the time horizon is maximized (see Equation (7)). SCE is based on the seminal work [23] and implemented within the OR-gym, an open-source collection of RL environments related to logistics and operations research [20]. SCE represents a single-product multi-echelon supply chain model. The model makes several assumptions. The tradable goods do not perish over time, and replenishment sizes are integers. Economic agents involved in the supply chains can be represented as stages \(M=\{0,1,\,\ldots,\,m_{ent}\}\). Stage 0 stands for a retailer that fulfills a customer's demand. Stage \(m_{end}\) stands for a raw material supplier. Stages from 1 to \(m_{end-1}\) represent intermediaries involved in the product lifecycle, for example, wholesalers and manufacturers (Figure 2). One unit from the previous stage is transformed into one unit in the following stage until the final product is obtained. Replenishment lead times between stages are constant, measured in days, and include production and transportation times. Both production and inventory capacities are limited for all stages except the last one (raw material supply is assumed to be infinite). During the simulation at each time step \(t\in T\), the following sequence of events take place:  
1. All the stages except the raw material pool place replenishment orders. 2. Replenishment orders are satisfied according to available production and inventory capacity. 3. Demand arises and is satisfied according to the inventory on hand at stage 0 (retailer). 4. Unsatisfied demand and replenishment orders are lost (backlogging is not allowed). 5. Holding costs are charged for each unit of excess inventory.