## 4 Solution Based on Multi-Agent Reinforcement Learning
### The Prioritized Experience Replay Strategy  
The experiences are sampled from the replay buffer during the training process, and the sampled experiences determine the learning effect of the neural network. In order to improve the learning efficiency, this paper proposes to use the prioritized-experience-replay strategy in the multi-agent training process. The value of experience in the replay buffer can be measured by the absolute value of TD error, as shown in Equation (35). A larger positive value of \(\delta_{i,n}\) means that the action selection of the experience was relatively successful, and a large negative value means that there was a mistake in the action selection. These experiences can be used to quickly learn successful strategies and avoid major mistakes. Therefore, the sampling priority can be determined according to the absolute value of TD error to improve learning efficiency. \[\delta_{i,n}=r_{i,n}+\gamma Q_{i,n}^{\mu^{\prime}}-Q_{i,n}^{\mu} \tag{35}\]  
The probability of experience being sampled is positively related to its priority. For agent \(i\), when experience \(n\) is sampled and trained, its priority \(p_{i,n}\) is updated as shown in (36):  
\[p_{i,n}=|\delta_{i,n}| \tag{36}\]  
Figure 2: Framework of multi-agent centralized training. When the newly generated experience is stored, its priority is equal to the current maximum priority, so as to avoid the newly generated experience not obtaining sampling opportunities. In order to effectively update and sample experience, as shown in Figure 3, a binary tree structure of sum tree is used to store experience and its priority. Each node is the sum of its child nodes. The priority of each experience is taken as the leaf node, and the root node is the sum of all experience priorities. Sample \(N_{s}\) samples in the replay buffer. The value of root node is divided into \(N_{s}\) intervals equally, and one randomly samples in each interval. For a certain sampling value \(p_{s}\), if it is less than the value \(c_{II}\) of the left child node, the path on the left will be followed; otherwise, the path on the right will be followed. If \(p_{s}<c_{II}\), \(p_{s}\) is compared with the values of the two child nodes in the lower layer on the left. If \(p_{s}<c_{II}\), the left path will be followed. If \(p_{s}>c_{II}\), the right path will be followed and \(p_{s}\) is updated. The new \(p_{s}\) is equal to the difference between the original value and the value of the left child node. It will continue according to the above rules until a leaf node is reached. Finally, the experience of the selected leaf node is trained. For the leaf node with priority of \(p_{i,n}\), the probability of being selected is \(P(n)\):  
\[P(n)=\frac{p_{i,n}^{\beta_{1}}}{\sum_{k}p_{i,k}^{\beta_{1}}} \tag{37}\]  
where \(\beta_{1}\) is the priority factor, which changes the original priority in an exponential form. The value of \(\beta_{1}\) is taken in the interval [0,1]. It is worth noting that compared with random sampling, there is a deviation between the state distribution obtained by priority sampling and the actual distribution. In order to correct this deviation, the importance sampling (IS) weight is introduced here (\(\omega_{i,n}\)):  
\[\omega_{i,n}=(N_{e}P(n))^{-\beta_{2}}/\max_{k}\omega_{i,k} \tag{38}\]  
where \(\beta_{2}\) controls the correction degree of deviation. When the value is zero, it represents no deviation correction. When the value is one, it represents the complete correction of the deviation. At the end of the training, unbiased sampling of experience is very important, so \(\beta_{2}\) increases linearly from the initial value to one in steps of \(\Delta\beta\). At the beginning of training, more priority sampling is used to improve learning efficiency. Additionally, at the end of training, sampling is closer to unbiased sampling.