## 4 Approach to Perishable Inventory Management Problem
### Application of Proximal Policy Optimization  
Reinforcement learning (RL) is a type of machine learning in which an agent learns a policy of actions by interacting with its environment. The objective of RL is to maximize the cumulative expected reward. The learning process typically follows the steps below:  
1. The agent observes the current state \(s_{t}\) of the environment.
2. Based on the policy \(\pi(s)\), the agent selects an action \(a_{t}\).
3. The environment provides a reward \(r_{t}\) for the agent's action \(a_{t}\) and transitions to the next state \(s_{t+1}\).
4. The agent uses the reward \(r_{t}\) and state transition to update the policy \(\pi(s)\) or the value function \(V(s)\), improving its policy \(\pi(s)\) over time.  
DRL is a method that combines reinforcement learning with deep learning. Using neural networks to approximate policies and value functions enables DRL's handling of complex and large-scale problems. PPO is one of the DRL methods proposed by [PERSON] et al. [30]. In PPO, the policy and value functions are approximated using neural networks. The goal of PPO is to maximize the objective function expressed as shown in Equation (11).  
\[L_{t}^{\text{CLIP+VF+S}}(\theta)=\hat{E}_{t}\big{[}L_{t}^{\text{CLIP}}(\theta) -c_{1}L_{t}^{\text{VF}}(\theta)+c_{2}S[\pi_{\theta}](s_{t})\big{]} \tag{11}\]  
Here, \(c_{1}\) and \(c_{2}\) are coefficients, \(L_{t}^{\text{CLIP}}\) is the clipped surrogate objective function, \(L_{t}^{\text{VF}}\) represents the squared error loss \((V_{\theta}(s_{t})-V_{t}^{\text{targ}})^{2}\), and \(S\) denotes the exploration entropy.  
The surrogate objective function \(L_{\text{CLIP}}\) is defined as shown in Equation (12).  
\[L_{t}^{\text{CLIP}}(\theta)=\hat{E}_{t}\big{[}\min\big{(}\rho_{t}(\theta)\hat {A}^{t},\text{clip}(\rho_{t}(\theta),1-e,1+e)\hat{A}^{t}\big{)}\big{]} \tag{12}\]  
Here, \(e\) is a hyper-parameter, \(\hat{A}^{t}\) is the advantage function, and \(\rho_{t}(\theta)\) is the importance sampling ratio, which is defined as shown in Equation (13). Additionally, \(\text{clip}(x,a,b)\) is a function that clips \(x\) to the range \([a,b]\).  
\[\rho_{t}(\theta)=\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{\text{old} }}(a_{t}\mid s_{t})} \tag{13}\]  
Here, \(\theta_{\text{old}}\) is the policy parameter before the update.  
The algorithm for PPO is shown in Algorithm 1. The policy and value functions are updated iteratively by collecting trajectories, computing rewards, and estimating advantages. The policy is updated by maximizing the clipped surrogate objective function, and the value function is updated by regression on the mean-squared error.  
```
Input: Initial policy parameters \(L_{0}^{\text{CLIP}}(\theta)\), initial value function parameters \(L_{0}^{\text{VF}}(\theta)\) Output: Policy \(\pi(s)\) for\(k=0,1,2\dots\)do  Collect set of trajectories \(D_{k}=\{\tau_{i}\}\) by running policy \(\pi_{k}=\pi(\theta_{k})\) in the environment;  Compute rewards \(\hat{R}_{t}\);  Compute advantage estimates \(\hat{A}_{t}\), based on the current value function \(V_{L_{t}^{\text{VF}}(\theta)}\);  Update the policy by maximizing the PPO-Clip objective with Equation (12);  Fit value function by regression on mean-squared error \((V_{L_{k}^{\text{VF}}(\theta)}-R_{t})^{2}\);  end for
```  
**Algorithm 1**Proximal Policy Optimization