## 3 ERA-MADDPG Intelligent Routing Solution
### ERA-MADDPG Intelligent Routing Algorithm
\[L(\theta_{t})=\frac{1}{B}{\sum_{j}}\bigg{[}\Big{(}Q_{i}^{\mu}\Big{(}x^{j},a_{ 1}^{j},\cdots,a_{N}^{j}\Big{)}-y^{i}\Big{)}^{2}\bigg{]} \tag{12}\]  
The actor function is updated through gradient descent using (13). The gradient for the deterministic policy \(\mu_{i}\) is defined in (13), which omits the log-likelihood term as \(\mu_{i}\) directly outputs actions. \[\
abla_{\theta_{j}}J(\mu)=\frac{1}{|S|}{\sum_{j}}\bigg{[}\
abla_{\theta_{i}} \mu_{i}\Big{(}o_{i}^{j}\Big{)}Q_{i}^{\mu}\Big{(}x,a_{1}^{j},a_{2}^{j},\cdots, a_{N}^{j}\Big{)}\bigg{|}_{a_{i}^{j}=\mu_{i}(a_{i}^{j})}\bigg{]} \tag{13}\]  
where \(y\) is the objective function, which is calculated by the intelligent agent critic. The calculation process of \(y\) is shown in Equation (14). \[y=r_{i}+\gamma Q_{i}^{\prime}\Big{(}x^{i},a_{1}^{\prime},\cdots,a_{N}^{\prime }\Big{)}\bigg{|}_{a_{i}^{j}=\mu_{i}^{\prime}(x^{\prime})} \tag{14}\]  
In Equations (12)-(14), \(\mu^{\prime}\) is the action function of the target network, where \(\mu^{\prime}=\Big{\{}\mu_{\theta^{\prime}},\cdots,\mu_{\theta^{\prime}_{N}} \Big{\}}\). \(\gamma\) is the discount factor, the algorithm uses a target network to speed up the learning process of the evaluation function and action function. \(\theta\)\({}_{i}^{\prime}\) is the action function parameter corresponding to the \(i-th\) target network, and \(Q_{i}^{\prime}\) is the target network evaluation function of the \(i-th\) agent. Each element in the experience replays cache pool \(D\) consists of a four-tuple \((s_{t},a,r,s_{t+1})\), which is used to record multi-agent experience, including \(a=\{a_{1},a_{2},\cdots,a_{N}\}\)\(r=\{r_{1},r_{2},\cdots,r_{N}\}\). The action value function of each agent is updated through back propagation, and the updating process is shown in Equation (15). \[L(\theta_{i})=E_{x,a,r,x^{\prime}}\bigg{[}\Big{(}Q_{i}^{\mu}(x,a_{1},a_{2}\cdots a _{N})-y\Big{)}^{2}\bigg{]} \tag{15}\]  
Since the strategy of each agent in the environment is iteratively updated, it is easy for the strategy of a single agent to overfit to the strategies of other agents. In order to alleviate the over-fitting problem, for a single agent \(i\), the algorithm's strategy \(\mu_{i}\) is set of sub-strategies \(\mu_{i}^{k}\) (\(k\): sub-strategy index, \(k\in[1,k]\)). In each training cycle, a sub-policy \(\mu_{i}^{(k)}\) is sampled, and its interactions with the environment generate transitions stored in \(D_{i}^{(k)}\). This ensures each sub-strategy maintains a dedicated experience buffer for focused learning. During the learning process, the maximization goal is the expected return of all sub-strategies as shown in (16). \[J_{e}(\mu_{i})=\text{E}_{k\sim unif(1,k),s\sim p^{\mu},a\sim\mu_{i}^{(k)}}[r_{ i}(s,a)] \tag{16}\]  
To mitigate overfitting to static agent strategies, ERA-MADDPG decomposes each agent's policy \(\mu_{i}\) into a set of sub-policies \(\left\{\mu_{i}^{k}\right\}\). During training, a random sub-policy is selected for each episode, forcing the agent to learn robust strategies that generalize across diverse interaction scenarios. The update gradient of each sub-strategy \(\mu_{i}^{(k)}\) is computed using its dedicated experience buffer \(D_{i}^{(k)}\) as shown in (17). \(D_{i}^{(k)}\) is the experience subset corresponding to the sub-strategy \(\mu_{i}^{(k)}\), containing transitions \((s_{t},a,r,s_{t+1})\) generated by \(\mu_{i}^{(k)}\). The expectation \(\text{E}_{x,a\sim D_{i}^{(k)}}\) denotes averaging over samples drawn from \(D_{i}^{(k)}\). \[\
abla_{\theta_{i}^{(k)}}J_{e}(\mu_{i})=\frac{1}{K}\text{E}_{x,a\sim D_{i}^{(k )}}\bigg{[}\
abla_{\theta_{i}^{(k)}}\mu_{i}^{(k)}(a_{i}|o_{i})\
abla_{a_{i}}Q ^{\mu_{i}}(x,a_{1},\cdots,a_{N})\bigg{|}_{a_{i}=\mu_{i}^{(k)}(o_{i})}\bigg{]} \tag{17}\]  
In Equations (16) and (17), \(\mu_{i}^{k}\) is the set of sub-strategies, \(Q\) is the evaluation function of \(\mu_{i}\) under sub-strategy \(\mu_{i}^{k}\), and \(\theta=\{\theta_{1},\cdots,\theta_{N}\}\) represents the input parameters.