## 5 Numerical Experiments
### Analysis of Performance of DP, ar-DP, and PPO
#### 5.3.3 Evaluation of PPO Algorithm  
To comprehensively evaluate the PPO algorithm, we analyze the learning curves of PPO for Cases 1 to 7. The optimal results obtained by DP are also shown for comparison. Figure 1(a),b show the learning curves for Cases 1 to 4, and Figure 2(a)-c show the learning curves for Cases 5 to 7. The vertical axis represents the reward value, which corresponds to the total profit, and the horizontal axis represents the number of episodes. The blue line indicates the average value achieved by PPO, and the red dashed line represents the optimal value obtained by DP.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Case 1** & **Case 2** & **Case 3** & **Case 4** & **Case 5** & **Case 6** & **Case 7** \\ \hline DP (\%) & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ ar-DP (\%) & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ PPO (\%) & 0.95 & 0.02 & 0.04 & 17.95 & 1.60 & 2.27 & 8.39 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of the performance for DP and PPO.  
Figure 2: Learning curves for Cases 1 to 4.  
Figure 2a-d show the learning curves of PPO for Cases 1 to 4. The vertical axis of the figure represents the reward value, which corresponds to the total profit, while the horizontal axis represents the number of episodes. The blue line indicates the average value achieved by PPO, and the red dashed line represents the optimal value obtained by DP. Convergence was observed in all cases between 100 and 150 episodes. Cases 1 to 3 showed values close to the optimal, whereas Case 4 converged to a local solution.  
Figure 3a-c show the learning curves of PPO for Cases 5 to 7. In these cases, convergence was also observed between 100 and 150 episodes. It is noted that when the optimal solution is poorer, as in Case 7 compared to other cases, the optimality gap of PPO tends to increase slightly.  
Based on the results, the following observations can be made:  
* In cases such as Case 4, where price sensitivity is generally high, demand tends to concentrate on products at specific price points. This leads to a monotonic pattern in state transitions, limiting the information available for PPO to learn effectively and increasing the likelihood of converging to a local solution.
* In cases such as Case 7, where costs are high, even slight policy differences result in significant cost variations. This makes the optimality gap larger.  
To further evaluate the PPO algorithm, we investigated the computation time for problems with different dimensions. Figure 4 shows the computation time of PPO for problems with a lead time of 1 day and shelf lives of 3, 4, and 5 days, corresponding to 4-, 5-, and 6-dimensional problems.  
Figure 3: Learning curves for Cases 5 to 7.  
It was observed that the computation time increased as the dimension increased. The reason for this is that with an increase in the number of dimensions, the state space expands, leading to an increase in the computation time for simulation and learning. However, the computation time for PPO remained within a reasonable range, even for problems with a higher dimension. This indicates that PPO is suitable for solving large-scale problems.