## 5 Game Model Solution
### Nash Q-Learning Based Multi-Agent Deception Deployment Solution Algorithm  
In this paper, Nash Q-learning algorithm is used to solve the multi-agent stochastic game model. The Nash Q-learning algorithm extends Q-learning from a single-agent to non-cooperative multi-agent environment, which needs to consider the joint action of multiple agents. For a system with \(n\) agents, the Q-function of each individual agent changes from \(Q(s,a)\) to \(Q(s,a^{1},\cdots,a^{n})\). The authors of [30] considered the extended concept of Q-function and adopted the concept of Nash equilibrium as a solution, defining Nash Q-value as the expected sum of discounted rewards of all agents following the specified Nash equilibrium strategy from the next period. This definition differs from the single-agent case, where future rewards are based only on the agent's own optimal policy. The Nash Q-function of agent \(i\) represents the sum of agent \(i\)'s current reward plus his future reward when all agents follow the joint Nash equilibrium strategy, and is expressed as follows.  
\[Q_{t}^{i}(s,a^{1},\cdots,a^{n})=r^{i}(s,a^{1},\cdots,a^{n})+\gamma\sum_{s^{ \prime}\in S}p(s^{\prime}|s,a^{1},\cdots,a^{n})v^{i}(s^{\prime},\pi_{*}^{1}, \cdots,\pi_{*}^{n}) \tag{3}\]  
where \(r^{i}(s,a^{1},\cdots,a^{n})\) represents the reward for one step obtained by the agent \(i\) by taking the joint strategy \((a^{1},\cdots,a^{n})\) in state \(s\), \(\gamma\) is the discount factor, \(p(s^{\prime}|s,a^{1},\cdots,a^{n})\) is the state transition probability, \((\pi_{*}^{1},\cdots,\pi_{*}^{n})\) is the joint Nash equilibrium strategy, and \(v^{i}(s^{\prime},\pi_{*}^{1},\cdots,\pi_{*}^{n})\) is the total discounted payoff of agent \(i\) starting from state \(s^{\prime}\) in a finite step when all agents follow the Nash equilibrium strategy.  
The agent initializes Q at step 0, which is usually set to \(Q_{0}^{i}(s,a^{1},\cdots,a^{n})=0\) for all \(s\in S,a^{1}\in A^{1},\ldots,a^{n}\in A^{n}\). At step \(t\), agent \(i\) observes the current state, executes its action, and then continues to observe its own reward, the actions and rewards of other agents, and the new state \(s^{\prime}\). Then, a Nash equilibrium strategy \(\pi^{1}(s^{\prime})\cdots\pi^{n}(s^{\prime})\) is calculated for stage game \((Q_{t}^{1}(s^{\prime}),\cdots,Q_{t}^{n}(s^{\prime}))\). Let \(\alpha_{t}\) denote the learning rate of agent at step \(t\), the learning rate defines how much weight of the new Q an old Q-value will learn from the new Q-value to itself. If the learning rate is 0, it means that the agent will not learn anything (old information is important), and if learning rate is 1, it means that the newly discovered information is the only information that matters. Then, agent \(i\) updates the value of Q according to Equation (4).  
\[Q_{t+1}^{i}(s,a^{1},\cdots,a^{n})=(1-\alpha_{t})Q_{t}^{i}(s,a^{1},\cdots,a^{ n})+\alpha_{t}[r_{t}^{i}+\gamma NashQ_{t}^{i}(s^{\prime})] \tag{4}\]  
where:  
\[NashQ_{t}^{i}(s^{\prime})=\pi^{1}(s^{\prime})\cdots\pi^{n}(s^{\prime})\cdot Q _{t}^{i}(s^{\prime}) \tag{5}\]  
According to the model we proposed in the previous section, the adaptation of the Nash Q-learning algorithm is as in Algorithm 1:We adapt the model based on the original Nash Q-learning algorithm, set the multi-agent reinforcement learning parameters, and train the model.