## 4 Experiment
### Results and Analysis  
To evaluate the performance of our proposed SAC-rainbow (our proposed algorithm) approach against other state-of-the-art algorithms, we conducted experiments and compared the learning curves. Figure 5 illustrates the episode reward achieved by each algorithm over the course of training.  
As evidenced by the learning curves, SAC-rainbow consistently outperforms the other algorithms, demonstrating superior convergence speed and higher episode rewards. The SAC algorithm, which serves as the foundation for our proposed method, also exhibits strong performance compared to PPO [26], DDPG [27], and TD3 [28]. This can be attributed to SAC's entropy regularization and ability to efficiently explore the action space.  
While PPO initially shows promising results, its learning curve plateaus early on, indicating limitations in adapting to the complex supply chain environment. DDPG and TD3, despite being popular deep reinforcement learning algorithms, struggle to match  
Figure 5: The learning curves for comparison methods.  
the performance of SAC and SAC-rainbow. This highlights the challenges posed by the high-dimensional state and action spaces in supply chain optimization.  
The superior performance of SAC-rainbow can be attributed to several key factors. First, the integration of prioritized experience replay allows the algorithm to learn more efficiently by prioritizing valuable experiences. Second, the use of a dueling network architecture in the critic network enhances the estimation of state-action values, leading to more accurate Q-value predictions. Finally, the distributional perspective adopted by SAC-rainbow enables better capture of the underlying reward distribution, resulting in more stable and robust learning.  
To quantitatively assess the performance, we evaluated the algorithms based on the average episode reward achieved over the last 100 training episodes. As Table 1 indicates, SAC-rainbow obtained an average reward of 0.92, surpassing SAC (0.87), PPO (0.76), DDPG (0.71), and TD3 (0.74). These results demonstrate the significant advantages of our proposed approach for optimizing supply chain inventory management.  
Furthermore, SAC-rainbow converges faster than the other algorithms, requiring only 10.7 min of training time to reach its peak performance. In contrast, SAC, PPO, DDPG, and TD3 take 13.2, 13.6, 12.9, and 12.4 min, respectively. The efficient learning of SAC-rainbow can be attributed to its effective exploration strategy guided by entropy regularization.  
Then, we analyzed the learned policies' ability to handle demand uncertainty and adapt to dynamic market conditions. SAC-rainbow consistently maintained optimal inventory levels and minimized stockouts and overstocking costs, showcasing its robustness and adaptability.  
In summary, our experimental results validate the effectiveness of SAC-rainbow in solving the complex supply chain inventory optimization problem. The proposed approach outperforms existing state-of-the-art algorithms, exhibiting faster convergence, higher episode rewards, and robust performance under uncertainty. These findings highlight the potential of SAC-rainbow as a powerful tool for optimizing supply chain operations and decision making.