## 5 Survey from the Defender's Perspective
### Problem Characteristics
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Type** & **Model** & **Ref** & **F** & **L** & **C** & **Policy** & **Description** \\ \hline \multirow{4}{*}{Design} & \multirow{4}{*}{MDP} & [61] & \(\times\) & \(\times\) & ✓ & Value Iteration &
\begin{tabular}{} \end{tabular} \\ \cline{2-7}  & & [62] & ✓ & \(\times\) & ✓ & Q-learning & Proposed a new adaptive control framework. \\ \cline{2-7}  & & [63] & \(\times\) & ✓ & ✓ & Q-learning & Proposed a spoofing resource deploy algorithm without strict constraints on attacker. \\ \cline{2-7}  & \multirow{2}{*}{Markov Game} & [64] & \(-\) & \(\times\) & ✓ & Q-learning & An algorithm that can obtain the optimal security configuration in multiple scenarios; \\ \hline \multirow{4}{*}{MDP} & [65] & ✓ & \(\times\) & ✓ & DDQN, A3C & (1) Design an autonomous defense scenarios and interfere with training process; (2) observation is less limited. \\ \cline{2-7}  & & [66] & ✓ & ✓ & \(\times\) & DQN & (1) The convergence is accelerated by expert artificial reward \\ \cline{2-7}  & & [67] & \(\times\) & \(\times\) & \(\times\) & Q-learning & (2) Limited reward setting. \\ \cline{2-7}  & & [68] & \(-\) & \(\times\) & ✓ & Value Iteration & (1) Introduces a botnet detector based on RL; (2) Action cost not taken into account. \\ \cline{2-7}  & & [69] & \(-\) & \(\times\) & ✓ & Reward structure & Multi-agent network with collaborative framework and group decision-making. \\ \cline{2-7}  & & [70] & \(\times\) & \(\times\) & ✓ & Sarsa & (1) Hierarchical collaborative team learning can extend to large scenarios; (2) Difficult to guarantee convergence. \\ \hline \multirow{4}{*}{Response} & \multirow{4}{*}{Markov Game} & [71] & \(\times\) & ✓ & ✓ & Q-learning & (1) Pareto optimization is used to improve Q-learning’s efficiency; (2) the attacker takes random actions. \\ \cline{2-7}  & & [72] & \(\times\) & ✓ & ✓ & Q-learning & (1) A defense policy selection simulator is constructed; (2) Random attack actions. \\ \cline{2-7}  & & [73] & ✓ & ✓ & \(\times\) & Q-learning & (1) Respond to attacks with limited information; (2) Rely on expertise to evaluate rewards. \\ \cline{2-7}  & & [74] & ✓ & ✓ & ✓ & MA-ARNE & Solving resource utilization and effective detection of APT with MARL. \\ \hline \multirow{4}{*}{POMDP} & [75] & \(\times\) & ✓ & ✓ & Q-learning & POMDP is modeled based on Bayes attack graph. \\ \cline{2-7}  & & [76] & \(\times\) & ✓ & ✓ & Q-learning & (1) The transfer probability is estimated using Thompson sampling; (2) Fixed attack transfer probability. \\ \cline{2-7}  & & [77] & ✓ & ✓ & \(\times\) & Value Iteration & (1) Adaptively adjust the recognition configuration according to the false alarm rate; (2) Cannot face adaptive attackers well. \\ \cline{2-7}  & & [78] & ✓ & ✓ & ✓ & Q-learning & (1) Multi-agent collaboration model based on signal and hierarchy; (2) Cannot guarantee the convergence. \\ \cline{2-7}  & & [79] & ✓ & ✓ & ✓ & Q-learning & (1) A decentralized threat perception model is proposed; (2) Value functions needs expert knowledge. \\ \hline \multirow{2}{*}{MDP} & [80,81] & \(-\) & \(\times\) & ✓ & Q-learning & The interaction in the honeypot is modeled as a semi-Markov process. \\ \hline \multirow{4}{*}{Recovery} & \multirow{4}{*}{MDP} & [82] & \(-\) & \(\times\) & \(\times\) & DDPG & (1) The continuous numerical method is used to obtain a better real-time recovery policy; (2) Low convergence rate. \\ \cline{2-7}  & & [83] & \(-\) & \(\times\) & \(\times\) & Q-learning & (1) Proposed an effective strategy based on a realistic power flow model with RL; (2) Low convergence rate. \\ \cline{2-7}  & & [84] & \(-\) & \(\times\) & \(\times\) & DDPG & Applied reinforcement learning to backup policy. \\ \cline{1-1} \cline{2-7}  & & & & & & & & \\ \hline \end{tabular} Note: F indicates “False Positive Rate”, O indicates “Local Information Known ”, C indicates “Constraint of Resources”.  
\end{table} 
Table 2: Survey from the defender’s perspective.