# Catch the Cyber Thief: A Multi-Dimensional Asymmetric Network Attack-Defense Game
## 4 Strategies
### Strategies of the Attack Agent  
In the \"Catch the Cyber Thief\" game, the attack agent cannot observe the state and action of the defense agent. Therefore, classic multi-agent reinforcement learning algorithms are no longer applicable. For example, the Minimax-Q algorithm is a multi-agent reinforcement learning algorithm used to solve two-player zero-sum games [14]. The algorithm considers the opponent's goal to minimize its own return and calculates its Q function based on the mini-max principle. The update formula of Q function is \(Q^{i}\gets Q^{i}+\alpha\cdot\left(R^{i}+\gamma V(s_{t+1})-Q^{i}\right)\), where \(V(s_{t+1})=\underset{\pi}{max}\underset{a_{-i}}{min}\sum\limits_{a^{i}\in A^{ i}}Q^{i}\pi_{i}(s,a_{i})\), \(i=1,2\) represents the player's index, and \(Q^{i}\) is a simplified form of the \(Q^{i}(s_{t},a_{t}^{i}-i)\).  
The biggest limitation of the Minimax-Q algorithm is that it needs to know the action space of both agents. Similar to it is the Nash-Q algorithm [15]. The Nash-Q algorithm extends a two-person zero-sum game to a multi-person general-sum game and updates its Q function to \(Q^{i}+\alpha\cdot\left(R^{i}+\gamma\text{Nash}Q_{i}(s^{\prime})-Q^{i}\right)\).  
In the Nash-Q algorithm, each state \(s\) is regarded as a stage game. The agent needs to observe the actions \(a_{i}\) and reward \(r_{i}\) of all other agents to find a global optimal point or saddle point in each state \(s\). However, due to the fact that the attack agent can only receive feedback from the environment, it cannot see any information about the opponent, and multi-agent algorithms such as MiniMax-Q and Nash-Q cannot optimize their own strategies.  
Therefore, an alternative approach is to use single-agent reinforcement learning algorithms to train the attack agent, such as Q-learning [16], SARSA(\(\lambda\)) [17], Double Q-learning [18], and DQN [19]. Then the attack agent can adjust some parameters to better respond to the opponent. In this case, the attack agent can be seen as an experienced \"thief\". Although it cannot see any information about the defense agent, it is able to evade detection by increasing the cleaning trace action, adjusting the probability of action selection, and other methods.  
Assume that the attack agent uses the Q-learning algorithm as its attack strategy. The characteristic of the Q-learning algorithm is that its behavior strategy is different from the learning strategy. Its behavior strategy can be selected through the \(\varepsilon\)-greedy method. The \(\varepsilon\)-greedy method selects actions according to the Q table; its purpose is to maintain a balance between exploration and exploitation. Its calculation formula is  
\[\pi(a|s)=\begin{cases}1-\varepsilon+\dfrac{\varepsilon}{|A(F)|},&\text{IF}\quad a =\arg\max_{a}Q^{\pi}(F|a);\\ \dfrac{\varepsilon}{|A(F)|},&\text{Otherwise}\end{cases} \tag{4}\]  
The learning strategy is through sampling \((s,a,R,s^{\prime})\) to approximate the Bellman optimal operator \(\mathbb{T}\), namely \(\underset{a\in\Lambda}{max}\mathbb{E}[R(s,a,s^{\prime})+\gamma V(s^{\prime})] \approx\underset{a\in\Lambda}{max}[R(s,a,s^{\prime})+\gamma V(s^{\prime})]\) to approach the optimal value function for learning control. However, under the definition of the \(CNT\), the attack agent needs to select an action from the footholds \(F\), so the Q function that needs to be updated here is \(Q(F_{t},a_{t})\). The Q-learning algorithm process of the attack agent under the CNT is shown in Algorithm 1.  
The SARSA(\(\lambda\)) and DQN can also be used in Algorithm 1 by replacing the Q value update strategy function.