## 3 Methodology
### Soft Actor-Critic with Prioritized Experience Replay  
To learn adaptive strategies for supply chain inventory optimization, we propose the SAC algorithm with prioritized experience replay. SAC is a state-of-the-art reinforcement learning algorithm that combines the benefits of both value-based and policy-based methods. It introduces an entropy term in the objective function to encourage exploration and improve the robustness of the learned policies. The SAC algorithm consists of an actor network and a critic network. The actor network generates actions based on the current state, while the critic network estimates the Q-values of state-action pairs. The algorithm iteratively updates the actor and critic networks using the experience replay buffer, which stores past transitions (state, action, reward, and next state). To further enhance the learning efficiency and prioritize important experiences, we integrate prioritized experience replay (PER) into the SAC algorithm. PER assigns higher sampling probabilities to transitions with larger temporal-difference errors, allowing the agent to learn more effectively from informative experiences. Prioritized experience replay is a technique that enhances the efficiency and effectiveness of the learning process in reinforcement learning algorithms. Unlike standard  
Figure 1: The Markov Decision Process of a factory supply chain. experience replay, which uniformly samples transitions from the replay buffer, PER assigns higher sampling probabilities to transitions that are deemed more informative or valuable for learning. The key idea behind PER is to prioritize the transitions in the replay buffer based on their temporal-difference (TD) error. The TD error measures the difference between the predicted Q-value and the target Q-value for a given transition. Transitions with larger TD errors are considered more surprising or informative, as they indicate a significant discrepancy between the current estimate and the target value. In PER, each transition in the replay buffer is assigned a priority value that is proportional to its TD error, as shown in Figure 2. The probability of sampling a transition is determined by its priority value relative to the sum of all priorities in the buffer. This allows the algorithm to focus more on transitions that have a higher potential for improving the learned policy and value estimates. To implement PER, the following steps are typically followed:  
1. Compute the TD error for each transition when it is added to the replay buffer. 2. Assign a priority value to each transition based on its TD error, using a priority function (e.g., proportional or rank-based). 3. When sampling a batch of transitions from the replay buffer, use the priority values to determine the sampling probabilities. 4. Update the priorities of the sampled transitions based on their updated TD errors after the learning step. By incorporating PER into the SAC algorithm, the learning process can be accelerated, and the quality of the learned policies can be improved. PER helps the agent focus on the most informative experiences, allowing it to quickly identify and correct errors in its value estimates and policy. The SAC algorithm with PER proceeds as follows:  
1. Initialize the actor network with parameters \(\theta\), the critic networks with parameters \(\phi\)1 and \(\phi\)2, and the replay buffer D.
2. Set the target network parameters equal to the main network parameters: \(\phi\)tar,1 \(\leftarrow\)\(\phi\)1, \(\phi\)tar,2 \(\leftarrow\)\(\phi\)2. Figure 2: The framework of prioritized experience replay. 3. For each episode: 1. For each time step: 1. Observe the current state s and select an action a\(\
eg\pi\theta(\cdot\,\mathrm{s})\) using the actor network. 2. Execute the action a in the environment and observe the next state's one-step reward r, and store the transition (s, a, r, s') in the replay buffer D. 3. Sample a batch B of transitions from the replay buffer D based on their priority scores. 4.