# Adaptive Supply Chain: Demand-Supply Synchronization Using Deep Reinforcement Learning
## 3 Methodology
### Proximal Policy Optimization Algorithm  
PPO is applied to train the RL agent. PPO is developed by the OpenAI team and distinguished by simple implementation, generality, and low sample complexity (the number of training samples that the RL agent needs to learn a target function with a sufficient degree of accuracy). On the other hand, the vast majority of machine learning algorithms require hyperparameters that must be finetuned external to the learning algorithm itself. In this regard, an extra advantage of PPO is that it can provide solid results with either default parameters or relatively little hyperparameter tuning [24].  
PPO is an actor-critic approach to DRL. It uses two deep artificial neural networks, one to generate actions at each time step (_actor_) and one to predict the corresponding rewards (_critic_). The actor learns a policy that produces a probability distribution of feasible actions. On the other hand, the critic learns to estimate the reward function for given states and actions. The difference between the reward predicted by the critic with parameters \(\theta\) and the actual reward received from the environment is reflected in the loss function \(Loss(\theta)\). PPO limits the update of the parameters by clipping the loss function Equation (8). This loss function constitutes the distinguishing feature of PPO and demonstrates more stable learning compared to other state-of-the-art policy gradient methods across various benchmarks.  
\[Loss(\theta)=\min\left[\ \frac{\pi_{k-1}}{\pi_{k}}\hat{A}_{t},\ cl\left(\frac{ \pi_{k-1}}{\pi_{k}},\ 1-\epsilon,\ 1+\epsilon\right)\hat{A}_{t}\right] \tag{8}\]  
where \(\pi_{k-1}\) and \(\pi_{k}\) denote the previous policy and the new policy. \(k\) stands for the number of updates to the policy since initialization. The function \(cl(.)\) imposes the constraint of form \(1-\epsilon\leq\pi_{k-1}/\pi_{k}\leq 1+\epsilon\). \(\epsilon\) is a tunable hyperparameter that prevents policy updates that are too radical. The advantage estimation of the state is the sum of the discountedprediction errors over \(T\) time steps \(\hat{A}_{t}=\sum\gamma^{T-t+1}\delta_{T}\), where \(\delta_{T}\) is the difference between the actual and the estimated rewards provided by the critic artificial neural network, also known as the temporal difference error. \(\gamma\) stands for the discount rate. Algorithm 1 demonstrates the implementation of PPO in Actor-Critic style.  
```
for iteration = 1, 2,... do for actor = 1, 2,..., \(N\)do Run policy \(\pi_{\mathit{old}}\) in environment for \(T\) timesteps  Compute advantage estimates \(\hat{A}_{1}\),..., \(\hat{A}_{T}\) endfor Optimize surrogate Loss with regard to \(\theta\) \(\theta_{\mathit{old}}\leftarrow\theta\) endfor
```  
**Algorithm 1** Actor-Critic Style within PPO  
The implemented PPO uses fixed-length trajectory segments. In each iteration, each of \(N\) actors collects \(T\) timesteps of data in parallel. Then, the surrogate loss function is obtained and can be further optimized with mini-batch stochastic gradient descent. Therefore, the algorithm aims to compute an update at each iteration that minimizes the loss function while ensuring the relatively small deviation from the previous policy. As a result, the algorithm reaches a balance between ease of implementation and tuning and sample complexity.