# Catch the Cyber Thief: A Multi-Dimensional Asymmetric Network Attack-Defense Game
## 4 Strategies
### Strategies of the Defense Agent  
For the defense agent, the attack information is implicit in the received observations, so its strategies can be divided into four categories. (1) The first category includes heuristic strategies that do not use observations and rewards. Since the defender can directly access any host in the network, strategies can be designed according to its detection preference. We can divide the most commonly used three-layer intranet into the DMZ layer (outermost layer), the internal office layer (second layer), and the core layer (innermost layer). Three types of strategies can be designed:Priority searching hosts in the Demilitarized Zone (PDM), Priority searching hosts in the Core Layer (PDC), and Random Detection (RD). The probability that these strategies detect each host is given by the formula  
\[\text{PDM}(i)=\frac{1}{n},\quad\text{PDC(i)}=\frac{i}{\sum_{i=1}^{n}i},\quad \text{RD(i)}=\frac{n-i}{\sum_{i=1}^{n}i}, \tag{5}\]  
where \(i=\{1,2,\cdots,n\}\) is the index of hosts. Assigning values according to the way that one host is in the DMZ layer and \(n\) host is in the core layer. ```
1) Set \(\gamma\), \(\alpha\) and \(\epsilon\) to specified values; 2) Set the footholds \(F\), rewards \(R\) and the termination condition of the game. 2\(\pi(a|s)\) of the attack agent
3: Initialize the Q table
4:for each episode do
5: Set initial state \(s\)
6:for each step in a episode do
7: Select an action \(a_{t}\) from the Q table according to \(\epsilon\)-greedy strategy
8: The attack agent receives the reward \(r_{t}\) and observes \(o_{t+1}\), where \(o_{t}\) contains the privilege of host \(i\)
9:if Host \(i\) is not in current footholds \(F_{t}\)then
10:\(F_{t}= \begin{cases}F_{t}\text{-append(node)},&0\leq\text{len}(F_{t})<L;\\ F_{t}[0]=node,&\text{len}(F_{t})=L.\end{cases} \)
11:else
12: Keep \(F_{t}\) unchanged
13:endif
14:\(Q\gets Q+\alpha\cdot\left(r^{t}+\gamma\cdot\underset{a\in\mathbb{A}}{max} Q-Q\right)\), Where \(Q\) is a simplified form of \(Q(F_{t},a_{t})\). 15:if\(s_{t+1}=done\)then
16: Theepisode ends
17:else
18:\(s_{t}\gets s_{t+1}\)
19:endif
20:endfor
```  
**Algorithm 1** The Q-learning algorithm under the CNT  
(2) The second category includes learning strategies that only use rewards. The defense agent optimizes its strategy depending on whether its task is successful or not. A typical algorithm is the Upper Confidence Bounds (UCB1) algorithm [20]. UCB1 is a classic algorithm for the MAB problem that uses the optimism principle to select actions under uncertainty. The agent initially pulls each arm once, then selects the arm \(i(t)\) according to the following calculation formula. \[i(t)=\underset{a}{\arg\max}\left[Q_{t}(j)+c\sqrt{\frac{\ln t}{N_{t}(j)}}\right], \tag{6}\]  
where \(t\) is the number of episodes experienced so far, \(Q_{t}(j)\) is the average reward obtained from host \(j\), \(N_{t}(j)\) means that host \(j\) was selected before the number of times \(t\), and \(c\) is a constant used to control the degree of exploration. (3) The third category includes heuristic strategies that only use observations. Two strategies are designed here: the Maximum Observation detection strategy (MaxObs(\(O_{t}\))) and the Random Maximum Observation detection strategy (RMaxObs(\(O_{t}\))). MaxObs(\(O_{t}\)) means that the defense agent chooses the host with the maximum number of alarms. The calculation formula is \[\text{MaxObs}(O_{t})=O_{t}(i),\quad if\ O_{t}(i)\geq O_{t}(j), \tag{7}\]  
where \(i,j\in\{1,2,\cdots,n\}\) is the index of a host, and \(O_{t}(i)\) is the number of alarms of the host \(i\) at time \(t\). The \(\text{RMaxObs}(O_{t})\) adds randomness to the \(\text{MaxObs}(O_{t})\)).