## 3 ERA-MADDPG Intelligent Routing Solution  
### ERA-MADDPG Intelligent Routing Algorithm  
This paper improves the multi-agent deep deterministic policy gradient algorithm (DDPG) [17; 18] in DRL [19; 20] and combines it with the actor-critic framework [21]. The actor network and critic network in the actor-critic framework adopted are the two networks of the intelligent agent in the MADDPG algorithm. The critic network evaluates the actions calculated by the actor network to improve the performance of the actor network. The actor network calculates the actions to be taken based on the obtained state. In order to reduce the correlation in the training data and increase the stability of the training, a certain number of training experiences with better reward values are stored in the experience replay memory and randomly read by the critic network during updating. Compared with the DDPG algorithm, the MADDPG algorithm adopts distributed execution and centralized training, which can effectively improve training efficiency, enhance learning stability, promote collaboration, and improve the versatility and applicability of the algorithm. The actor network simply collects its own observed data during the training phase, while the critic network collects information about the actions and observations of other agents. The ERA-MADDPG algorithm architecture diagram is shown in Figure 3.  
MADDPG builds a centralized critic network for each agent, where the critic network builds a shared value function network that can obtain global information, including the global state and the actions of all agents. This enables each agent can consider the impact of other agents' behaviors on the global environment, thereby evaluating the value of its  
Figure 3: ERA-MADDPG algorithm architecture. own actions more accurately. The output of the centralized critic is the value function \(Q_{i}^{\pi}(x,a_{1},a_{2},\cdots,a_{N})\) corresponding to each agent. \(\pi_{i}\) represents the random policy of the agent, mapping states to action probabilities or deterministic actions. \(\mu_{i}\) is a deterministic policy that directly maps states to actions. For the intelligent routing task with \(N\) agents, the joint routing strategy is defined as shown in Equation (10). \[\pi=\{\pi_{1},\cdots\pi_{i},\cdots\pi_{N}\} \tag{10}\]  
where \(\pi=\{\pi_{1},\pi_{2},\cdots,\pi_{N}\}\), \(\pi\) represents the random strategy of agent \(i\). The action taken by the \(i-th\) agent under the observed parameter information is represented by the corresponding policy function \(\pi_{i}\). The information of the entire network environment is represented by \(x\), and the gradient of the stochastic \(\pi_{i}\) can be expressed by (11), where log denotes the natural logarithm, used to compute the log-likelihood term of the policy gradient for stabilizing gradient updates during training. \(J(\theta_{i})\) denotes the expected cumulative reward for agent \(i\) under policy \(\pi_{i}\) with parameters \(\theta_{i}\). \[\
abla_{\theta_{i}}J(\theta_{i})=\mathrm{E}_{s\sim\rho^{\pi},a_{i}\sim\pi_{i} }\big{[}\
abla_{\theta_{i}}\log\pi_{i}(a_{i}|s_{i})Q^{\pi}(x,a_{1},\cdots,a_{ N})\big{]} \tag{11}\]  
where \(Q_{i}^{\pi}(x,a_{1},a_{2},\cdots,a_{N})\) is the evaluation function of the \(i-th\) agent, the input is the obtained network information \(x\) and the action \(a_{i}\) taken by each agent, and the output is the \(Q\) value of the agent. \(B\) is the batch size (number of samples drawn from replay buffer, \(B=128\)), \(|D|=5000\) is the buffer capacity, and \(j\in[1,B]\) indexes the samples in the minibatch. The critic network is updated by minimizing the loss function in (12), which computes the Mean Squared Error (MSE) between its predicted \(Q\)-value and the target value \(y^{i}\). This loss function ensures that the critic's evaluation aligns with the agent's long-term reward objectives.