## 3 Active Guidance Controller Design for IRWs Based on MARL
### Improved MADDPG with PER  
The standard experience replay mechanism randomly resamples the stored experiences without considering sample quality. In our case, the quality of the stored data is influenced by variations in wheel-rail contact and system noise in IRWs under different control strategies. For example, during the early training stages, experiences restoring IRWs from large yaw angles or lateral displacements are more informative. These experiences, which could have been utilized more efficiently to improve the controller, have fewer chances of being chosen because of uniform random sampling. To address this issue, we introduce the PER mechanism, a method widely used in single-agent off-policy reinforcement learning [21]. PER assigns priorities to experience samples based on the absolute value of the TD error, enhancing the probability of selecting high-value experiences for training. This prioritization ensures a more frequent selection of samples poorly predicted under the current policy. In addition, we use a centralized replay buffer based on a SumTree structure to store experience and its TD-based prioritization. SumTree is a complete binary tree in which each leaf node contains an individual sample state transition tuple and its corresponding priority value. Each nonleaf node holds the cumulative sum of the priorities of its child nodes. When new experiences are added or the priorities of existing experiences are updated, only the specific path from the relevant leaf node to the root requires adjustment rather than continuously sorting samples according to priority in the experience buffer. An illustrative representation of the PER-MADDPG algorithm framework is shown in Figure 4.  
The TD error calculated by the critic network determines the deviation between the estimated and target \(Q\) values derived from samples. For a given experience tuple \(j\), agent \(i\) calculates the current \(Q\) value dependent on the \(s^{j}\) and the collective actions, while the target critic network determines the \(Q\) value for \(s^{j+1}\). The TD error can be formulated using Equation (13). \[\delta_{i}^{j}=r_{i}^{j}+\gamma Q_{i}^{\prime}\Big{(}s^{j+1},a_{1}^{j+1}, \ldots,a_{N}^{j+1}\Big{)}-Q_{i}(s^{j},a_{1}^{j},\ldots,a_{N}^{j}) \tag{13}\]  
where \(\delta_{i}^{j}\) denotes the TD error of experience sample \(j\) used by agent \(i\) and \(Q_{i}^{\prime}\Big{(}s^{j+1},a_{1}^{j+1},\ldots,a_{N}^{j+1}\Big{)}\) represents the \(Q\) value predicted by the target-critic network for the subsequent state \(s^{j+1}\). When employing the PER method, the priority \(p_{j}\) of each experience based on the TD error is defined as \(p_{j}=\left|\delta^{j}\right|+\epsilon\). Here, \(\epsilon\) is a tiny positive constant that ensures a nonzero probability of selection for all experiences. The sampling probability \(P_{j}\) for the \(j\)-th experience is calculated based on the priority \(p_{j}\) relative to the weighted sum of all experience priorities using the formula: \(P_{j}=\left(p_{j}\right)^{\alpha}/\sum\limits_{k\in D}\left(p_{k}\right)^{\alpha}\), with \(\alpha\in\left[0,\,1\right]\) balancing random and greedy sampling. Moreover, we apply importance sampling (IS) to rectify the introduced sampling bias via nonuniform sampling in the PER, and the IS weight \(w_{j}\) is calculated using the formula \(w_{j}=\left(\left|D\right|\cdot P_{j}\right)^{-\beta}\), where \(\left|D\right|\) is the capacity of the experience replay buffer and \(\beta\) is an adjustment parameter that controls the bias correction, with \(\beta\in\left[0,\,1\right]\). The IS weights should also be normalized to ensure that no sample weight dominates the update step \(w_{j}\gets w_{j}/\max_{k\in D}w_{k}\), where \(k\) is the index of the sample with the maximum weight and \(\max\,w_{k}\) is the highest weight among all the experience data. The training phase of the PER-MADDPG is shown in Algorithm 1.