# Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL
## 5 The Multi-Agent Particle Environment Experiment  
The multi-agent particle environment was used as the experimental environment in this study, proposed by [PERSON] et al. (2017), and used for the verification of MADDPG algorithm. The multi-agent particle environment is a recognized multi-agent reinforcement learning environment ([PERSON] and [PERSON], 2016; [PERSON] and [PERSON], 2016; [PERSON] et al., 2017), including different tasks, covering competition, cooperation, and communication scenarios in multi-agent problems. The particle environment is composed of a two-dimensional continuous state space, in which agents move to fulfill specific tasks. The state information includes the agent's coordinates, direction, speed, etc. The shape of the agent is spherical, and actual collision between agents is simulated through rigid body collision. Figure 4 displays the cooperative communication task and the cooperative navigation task for the particle experiment.  
Cooperative communication task: This task involves two types of cooperative agents who play the roles of speaker and listener, respectively, and there are three landmarks with different colors in the environment. In each round of the game, the listener's objective is to reach the designated color landmark, and the reward obtained is inversely proportional to the distance from the correct landmark. The listener can obtain the relative position and color information of the landmarks, but they cannot identify the target landmark on their own. The observation information for the speaker is the color information of the target landmark, and they send messages at every moment to help the listener reach the designated landmark.  
Cooperative navigation task: This task involves \(N\) cooperative types of agents and \(N\) landmarks. The aim of the task is for the \(N\) agents to move to different \(N\) landmarks, respectively, while at the same time avoiding collisions with other agents. Each agent's observation information comprises its own and the landmarks' relative position information and the relative position information with other agents. The reward function is calculated based on the closeness of each agent to its landmark--the closer it is, the higher the reward. Additionally, each agent actually occupies a certain physical volume and will be punished when it collides with other agents.  
Under the multi-agent particle environment, the M2 ATD3 algorithm proposed in this paper was compared with the MASTD3 algorithm, the previous MATD3 algorithm, and the MADDPG algorithm. Below is a presentation of the hyperparameter settings for this study. To ensure the fairness of the experiment, general parameters were set uniformly, learning rate \(\alpha=0.01\), batch sample size \(N=1024\). All algorithms share the same neural network structure, i.e., two fully connected layers with a width of 64 serving as the hidden layer of the policy network and Q network. The activation function of the neural network used Rectified Linear Units (Relu), and all experiments employed Adam as the optimizer of the neural network.