## 2 Research Background
### Decision-Making Models  
In a decision-making task, an agent perceives states (or observations) from the environment and takes an action. The environment transfers to a new state (or observation) and feeds back to the agent for a reward. This interaction process between the agent and the environment will keep going or end when it reaches the end state. In this process, the agent learns an optimal policy to maximize the accumulated reward. The decision-making model has a variety of formal abstractions, in which the most classic one is the Markov Decision Process (MDP) [15]. Beyond that, the most commonly used intelligent decision-making models for cyber defense decision-making tasks include (1) Markov Game which is extended from one agent to multiple agents [16]; (2) a Partially Observable Markov Decision Process (POMDP) [17; 18; 19] which is obtained from a fully observable state to a partially observable state.  
MDP: Only one agent exists in a decision-making task, and the environment satisfies the Markov property, which means that the change of state depends only on the current state and action, instead of the historical state and action. An MDP involves a quadruple  
Figure 2: The structure of this paper.  
\(\langle S,A,R,T\rangle\), where \(S\) indicates the status space, and \(A\) indicates the executable action space \(R:s,a,s^{\prime}\in S\times A\times S\to r=R(s,a,s^{\prime})\in\mathbb{R}\) is a bounded reward function, and \(T:S\times A\times S\to[0,1]\) is a probability function of state transition, which satisfies the \(\forall s,a,\)\(\sum_{s^{\prime}\in S}T(s,a,s^{\prime})=1\). With its interacting with the Environment, the Agent will learn and optimize a policy \(\pi:S\times A\to[0,1]\), which is made to maximize the cumulative rewards over the long term \(Return=\mathbb{E}_{\pi}\big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{t}\big{]}\), where \(\gamma\in(0,1]\) indicates the discount rate, \(t\) indicates the discrete time step, and \(r_{t}=R(s_{t},a_{t},s_{t+1})\) indicates the Rewards for \(t\) time, \(a_{t}\sim\pi(s_{t},a_{t})\).  
Markov Game: It is a quintuple consisting of agents:\(\langle N,S,\{A_{i}\}_{i=1}^{n},\{R_{i}\}_{i=1}^{n},T\rangle\). Among them, \(N\) represent the set of agents, \(n\) represent the quantity of agents, and \(A_{i}\) represents the action space of the agent \(i\). If \(A=A_{1}\times A_{2}\cdots\times A_{n}\) is the joint action space for all agents, then \(R_{i}:S\times A\times S\to\mathbb{R}\) is the reward function for the Agent \(i\). \(R_{i}(s,\vec{a},s^{\prime})\) indicates the reward for Agent \(i\) fed back by the Environment after all agents take a joint action \(\vec{a}\) on the state \(S\), and \(T(s,\vec{a},s^{\prime})\) represents the probability that the Environment moves from the state \(s\) to the subsequent state \(s^{\prime}\) through the joint action \(\vec{a}\).  
POMDP: In this process, an Agent cannot obtain the true state of the Environment, but can only observe a part of the state or the state after interference. This is called observation. POMDP is a six-tuple \(\langle S,A,R,T,\Omega,O\rangle\), where \(\langle S,A,R,T\rangle\) constitute a potential MDP, \(\Omega\) indicates the limited set of Observations available to an Agent, and the Observation function \(O:S\times A\times\Omega\to[0,1]\) indicates that the Observation \(o\) is obtained according to the Probability \(O(o|s^{\prime},a)\) on the subsequent state \(s^{\prime}\) after the action \(a\) is adopted on the state \(s\), and the reward \(R(s,a,s^{\prime})\) is also obtained.  
In addition, according to the time of the Agent's decision-making from discrete time step to continuous retention time, the Semi-Markov Decision Process (SMDP) [20; 21], Continuous-time Markov Decision Process (CTMDP) [22; 23; 24] and other decision-making models can also be applied.