## 4 Evaluation
### Qualitative Study  
The proposed method was compared with the existing scenario generation methods. We established the following three considerations for comparison between scenario generation methodologies. The scenario generation methodologies compared in this section are studies related to the 'generation' of threat models such as Attack Sequence, Graph, Path, and Tree that we analyzed, and we selected studies that considered one or more of the following: realism of the scenario, formalization of the generation, and training effectiveness. First, the realism of the scenario for the exercise must be considered. Exercises are conducted to prepare for possible future cyberattacks in a real-world environment. Therefore, trainees can train more effectively when they train under realistic conditions. In HSEEP for Security Training and Evaluation, modeling and simulation are used for realism to replicate various attacks in a realistic manner. Therefore, the trainees can train more effectively when they train under realistic conditions. Second, the attack scenario generation process should be systematic: the exercise planner should generate attack scenarios systematically without missing steps, and the scenarios should be generated systematically using the same method for a fair evaluation. For example, [PERSON] et al. [35] present a number of rules to establish a systematic procedure for generating an attack graph. Third, ensure the effectiveness of exercises utilizing attack scenarios. NIST 800-61 [36] states that Phase 1 (Preparation) of the incident response methodology includes training the incident response team. The training should be effective in demonstrating the use of tools and familiarization with the response process for responding to cyber incidents. Evaluate existing research and whether it includes methods for quantitative evaluation to ensure training effectiveness. From the above considerations, we prepared seven evaluation items for the proposal. * Generation scenarios targeting ICS: Evaluating whether the method can model general scenarios for various ICS facilities. * Identifying assets: Evaluating whether critical assets of the facility are identified during scenario modeling. * The basis for attack techniques: Evaluating whether the attack techniques appearing in the scenario were presented using a known framework such as MITRE ATT&CK
* The basis for Vulnerability: Evaluating whether the vulnerability information presented in the scenario was delivered using a knowledge base such as CVE. * Scenario modeling for each asset: Evaluating whether the attack technique that is performed from the asset where the attack starts to the final asset is presented for each asset. * Quantification: Evaluating whether one can prioritize the identified scenarios  
Table 13 shows the results of comparing the methods for generating cyberattack scenarios. Utilize a straightforward notation system to evaluate the capabilities of the methodologies in relation to the listed criteria. An 'O' mark indicates that the methodology can effectively provide the function associated with the respective criterion, whereas an 'X' denotes that providing the function is challenging or not feasible within the context of that methodology. The seven criteria are a selection of the conditions required for effective scenario building and training in an ICS environment. [PERSON] et al. [13] generated an attack sequence targeting the ship control system. This study identified in-ship assets such as shore control center (SCC) and global positioning system (GPS), and attack techniques constituting attack scenarios are based on MITRE ATT&CK's techniques. The vulnerabilities are not used in the attack scenario, and multiple attacks are modeled for each component and link located in the attack vector. Also, no quantification of scenarios is carried out. [PERSON] et al. [16] suggested a framework dynamic generating an attack graph, which is not intended for ICS.