## 6 Accident Case Development and Implementation for the Accident Response--A Case Study
### Testing and Evaluation
#### Evaluation  
The evaluation was performed to test the ability of the proposed method to accurately retrieve a suitable case from the case database. Metrics like \"precision\" and \"recall\" are usually used in the evaluation of performance in CBR research [50]. \"Precision\" and \"recall\" were computed by utilizing Equations (8) and (9):  
\[\text{Precision }=\frac{\text{number of true positives}}{\text{number of true positives}}+\text{number of false positives}, \tag{8}\]  
\[\text{Recall}=\frac{\text{number of true positives}}{\text{number of true positives}+\text{number of false negatives}}, \tag{9}\]  
where \"true positive (TP)\" represents a retrieved case that provided a feasible solution for the query case, \"false positive (FP)\" means that a solution of the recommended case was not reliable for the query case, and \"false negative (FN)\" reflects that the stored cases with suitable solutions for the query case were not retrieved from the library. In this study, the \"precision\" metric is more important than \"recall.\" \"Precision\" is the portion of retrieved cases that were deemed by the system to provide suitable solutions for the query cases, while \"recall\" is the portion of all cases in the case library with feasible solutions, including those that were not retrieved [8]. In the context of a metro accident response, the accurate retrieval of cases and avoiding the provision of unfeasible cases (high precision) is more important than retrieving all the relevant cases (high recall). The retrieved erroneous cases may affect the decision-making in the emergent response of metro accidents during operations and further lead to serious consequences, such as casualties or heavy economic loss. Therefore, \"precision\" was selected  
Figure 10: Retrieval results of “cutout” in descending order based on their similarity.  
as the evaluation metric since \"precision\" and \"recall\" are often negatively correlated. Additionally, considering the circumstance that in the real working environment, managers expect to get the required information within a limited amount of time, the case with the highest similarity to the input query has the most value to the users. Thus, in the evaluation process, we verified the suitability of the solution included in the retrieved case with the highest similarity to each given query through expert reviews.  
A set of keywords (e.g., \"cutout,\" \"single VOBC failure,\" \"TMS failure,\" \"ATC equipment failure\") that are relevant to metro operational accidents were defined for making up 10 testing queries. Retrieved cases with a similarity of more than 0.7 were selected as the retrieved results in this study, which were then evaluated via expert reviews. Then, five experts in a metro company in Wuhan (filed engineers and metro dispatchers with an average of 10 years of experience) were invited to evaluate the performance of the retrieval results because they were familiar with metro accident responses. The experts would discuss the retrieved results for each testing query with each other to judge whether the retrieved cases belonged to \"TP\" or \"FP\" and verify the feasibility of the highest similarity case. Table 5 shows the result of the expert review on the 10 testing queries. The mean precision of the proposed method was 91%. Additionally, the recommended solutions in the most similar case were feasible to solve the problems in the query case, which could help the manager to undertake better decision-making in a limited amount of time in the accident. The accuracy of the highest similarity case verified the feasibility of the proposed method in practice.