## 3 System Scheme and Implementation
### Multi-Step Experience Replay Strategy  
The experience replay buffer stores previous experiences in a buffer so that agents can more efficiently utilize these experiences for learning when training neural networks. It can provide training stability in multi-agent scenarios. Based on experience replay, agents not only update the parameters of neural networks using the current interaction experiences but also conduct batch learning by randomly sampling a batch of past experiences from the buffer. This helps to avoid the model only learning the current specific contexts and improve sample efficiency, that is, using a limited number of interactions and sampling through reasonable strategies to interact with the environment, and a more comprehensive exploration of the environment is achieved to optimize strategies. The experience information randomly sampled during the experience replay can be represented by a tuple \((s_{t},a_{t},r_{t},s_{t+1})\), including the current state, action, reward, and next state. However, randomly sampling only one set of network information may disrupt the temporal correlation in the state sequence. In traffic scheduling tasks, the network state information at certain moments may be closely related to previous or subsequent network states. If only the information from the current state is randomly sampled, it may lose this temporal structure, thereby affecting the learning effect. In some cases, a continuous sequence of states may contain more information about the environment, and random sampling taking only one set of state information may not capture this continuity. Therefore, this paper sets up a multi-step experience replay strategy, which considers sampling a continuous sequence of network states rather than just single network state points. Retaining a certain number of continuous state experiences allows for making more use of temporal information and capturing more contextual information. The structure of the multi-step experience replay buffer is shown in Figure 3.  
Figure 3: Multi-step experience replay of MAFS. The MAFS method introduces temporal correlation information by extending the experience tuples stored in the experience replay buffer. Add the previous time-step state, action, and reward of the agent to the experience information, forming the new tuple as follows:  
\[(s_{t-1},a_{t-1},r_{t-1},s_{t},a_{t},r_{t},s_{t+1}), \tag{1}\]  
The optimized experience information is stored in the experience replay buffer and sampled randomly. In order to improve the computational performance, effective dimensionality reduction and consolidation of the data are conducted. In the distributed execution of the CTN-MADDPG algorithm, the input data of the Actor network is the network state information from the previous time step and the current time step. The Critic network not only inputs the current state information and routing strategy of all agents but also includes information from the previous state. In the centralized training process, the input information of the Target Actor network includes the network state information of the current moment and next moment so as to output the routing strategy of the next time step, which is then passed to the Target Critic network to update the network parameters. The input information in the Target Critic network includes the current and next state information of all the agents, as well as the routing strategies they have made. Temporal features are introduced in the training process of the CTN-MADDPG algorithm to optimize the updating process of the network parameters. By extending and optimizing the experience information stored and sampled from the experience replay buffer, multi-agent systems can more effectively utilize past experiences, thereby accelerating the convergence process of deep reinforcement learning.