# Network Defense Strategy Selection with Reinforcement Learning and Pareto Optimization
## 3 Game Model on Pareto Optima and Q-learning  
### Game Definition  
The network attacker and defender interaction can be modelled as an information-incomplete two-player zero-sum game. Using such a model, we will teach our software agent to determine the optimal action when confronted with an actual network attack [37]. The specific network interaction we wish to model is described as the following game:  
\[ \begin{array}{c}Game=(G,\ S,\ A,\ D,\ P,R,\gamma\ )\\ G=(\mathit{nodes},\ \mathit{edges})\\ S=\{s_{1},\ s_{2},\ \ldots,\ s_{S}\}\\ A=\{a_{1},a_{2},\ldots,a_{A}\ \}\\ D=\{d_{1},d_{2},\ldots,d_{D}\}\\ P:S\times A\times D\times S\rightarrow[0,1]\\ R:S\times A\times D\to RV\\ 0\leq\gamma\leq 1\end{array}  \tag{12}\]  
Here, \(G\) is the network, \(S\) is the set of all possible states the game can be in; \(A\) is the set of possible actions that an attacker can perform on a node; \(D\) is the set of possible actions that a defender can perform on a node; \(P\) is the state transition function; \(R\) is the reward function defined in Equation (3); and \(\gamma\) is a discount factor. \(RV\) refers to the solution space for the reward function. A network \(G\) consists of a set of nodes connected by edges. Each node represents a hardware device, which runs several services, can be infected by a virus, and contains data. The value/weight of the services is given by \(\beta\), the value of keeping the node virus free is \(\upsilon\), and the value of the data on the node is \(\omega\). These values can differ per node. An edge represents the connection between two nodes. If two nodes are running the same service, they will use a connecting edge to communicate. \[\mathit{nodes}\ \subseteq\ \{\beta,\upsilon,\omega\ |\ \beta\in\mathbb{N}^{|\mathit{ serrices}|},\upsilon\in\mathbb{N},\ \omega\in\mathbb{N}\} \tag{13}\]  
\[\mathit{edges}\ \subseteq\ \{(x,y)|x,y\ \in\mathit{nodes},\ x\
eq y\} \tag{14}\]  
A game state \(s^{i}\ \epsilon\ S\) is a combination of the network state and each player's resource points. The network state is the state of all the nodes, i.e., which services are running on the node, if it is infected by a virus, and whether or not the data has been stolen, and the state of all the edges identified by service availability. Resource points are points that players can spend to perform an action. The players can only perform actions on the nodes, so the network state is simply the state of the nodes. Let \(n\) be the state of the nodes. In the node state, \(b\) is a vector showing which services are running on the node (1 = running and 0 = down), \(v\) indicates whether or not the node has been virus free (1 = virus-free and 0 = infected) and \(w\) indicates whether or not the data is secure (1 = secure and 0 = data is stolen). \[n\ \in\ \{b,\ v,w\ |\ b\in\{0,1\}^{|\mathit{serrices}|},\ v\ \in\{0,1\},\ w\ \in\{0,1\}\}^{|\mathit{nodes}|} \tag{15}\]  
In this game, we introduce the concept of _resource points_. From existing research, we know it is common to assume that each action taken by a player has an associated cost. To keep track of costs, we start a game, by giving each player a fixed set of resource points. The resource points tell us if a player can afford to take a certain action, e.g., if the cost of an action exceeds the available resource points, the player cannot take that action. If a player runs out of resource points, the game ends. Let \(p\) be a vector containing the player resource points: \(p\in\mathbb{N}^{2}\). The final definition of a game state is \(<n,\ p>\epsilon\ S\). Figure 4 shows a game state with compromised services in \(n_{1},n_{2}\), and \(n_{3}\) and a virus installed in \(n_{2}\). We do not need to formally define the state transition function \(P\). The game will use a combined Pareto-Q-learning algorithm to determine the best policy for the defender and attacker. Hence, Q-values will be determined iteratively according to Equation (8), rather than by the explicitly defined function of Equation (6).