## 6 Experiment
### Simulation Experiment and Result Analysis  
#### 6.3.1 Algorithm Comparison  
In order to verify the effect of Nash Q-learning algorithm on solving the model, we also introduce Minimax-Q algorithm [31] to solve the model. We also adopt the action policy of \(\varepsilon\)-_greedy_, and set \(\varepsilon\) as 0.1, and the other parameters are the same. The result obtained is shown in Figure 7. In addition, we also conducted experiments on the case where both the attacker and defender adopt a random strategy, that is, \(\epsilon=1\), and the result is shown in Figure 8.  
Figure 7: Training Results using Minimax-Q policy.  
Our analysis reveals that the Minimax-Q algorithm typically requires more steps to finalize an episode. The primary reason for this disparity is the Minimax-Q's singular focus on actions that maximize its Q-value in the agent's current state, without accounting for a balanced strategy. So, agents may get ensnared in local optimal policies and engage in repetitive, often futile, actions--leading to an extended number of steps to wrap up an episode. Insights from Figure 7 further underscore this point. The depicted step curve is noticeably erratic, lacking any discernible convergence even after 500 training episodes. This suggests that the agent struggles to pinpoint a equilibrium path but fails to stabilize in a convergent state. Contrastingly, Figure 8 shows that an agent employing a random strategy experiences a more consistent oscillation in the step curve, with the required steps to finish an episode typically capped at 200. Compared to Minimax-Q, this random strategy proves advantageous in that the agent does not get trapped in localized optima. Instead, the agent freely chooses actions from its set, leading to a reduced step count.  
Finally, we trained the Nash Q-learning algorithm with the same parameters for 500 episodes, and the results obtained are shown in Figure 9. The figure illuminates that during the initial phase of multi-agent system training, the number of steps to finalize an episode experienced significant volatility. This suggests that neither the attacker nor the defender had discerned their respective equilibrium paths, resulting in seemingly arbitrary movements within the maze. The pattern persisted until either the attacker reached the target or was intercepted by the defender. As training progressed, the variation in steps began to stabilize, although occasional spikes in step counts were evident. These anomalies can be attributed to certain episodes where random selections recurrently fell within the [0.1, 1) range, leading to a localized optimal strategy. The agent will continue to choose the local optimal move strategy and make it wander back and forth in the maze until it can randomly move to explore other paths.  
Figure 8: Training results using random policy.  
By the latter stages of training, the step count variations became minor and tended to oscillate around a consistent mean, suggestive of both players aligning with their equilibrium paths and the training converging to a stable state. Analyzing the accumulated data revealed that defenders primarily intercepted attackers at the three nodes illustrated in Figure 4b. Nonetheless, inherent randomness occasionally diverted the attacker and defender from their balanced strategies, sometimes culminating in the defender's inability to intercept or necessitating an extended number of steps for successful interception.  
In conclusion, Nash Q-learning algorithm has better performance than Minimax-Q and random strategy in solving maze model. Therefore, we chose the Nash Q-learning algorithm to solve the deception asset deployment strategy.