# Thermal, Multispectral, and RGB Vision Systems Analysis for Victim Detection in SAR Robotics
## Appendix 1```
1:Input: Trained CNN models \(M_{1,2,3}\)
2:\(\text{I}_{\text{RGB}}\leftarrow\text{RGB image }[640\times 480]\)
3:\(\text{I}_{\text{thermal}}\leftarrow\text{Thermal image }[640\times 480]\)
4:\(\text{I}_{\text{multispectral}}\leftarrow\text{Multi-spectral image }[1500\times 1500]\)
5:ground truth labels \(GT\)
6:Output: Number of victims \(N_{\text{victims}}\)
7:Detection precision \(P_{\text{detection}}\)
8:IoU metric
9:Best detection metric
10:Position information
11:Initialize \(N_{\text{victims}}\gets 0\), \(N_{\text{correct}}\gets 0\), \(IoU_{\text{sum}}\gets 0\), \(Best_{\text{metric}}\gets 0\)
12:Inference on New Data:
13:For each image \(i_{\text{RGB-thermal-multisp}}\) in \(Inference-Data\):
14:Obtain prediction from \(M_{1,2,3}\)
15:Store prediction, associated probability, and predicted bounding box
16:If prediction is \"victim\" and corresponding \(GT_{i}\) is also \"victim\" and not a \"rescuer\":
17:\(N_{\text{correct}}\gets N_{\text{correct}}+1\)
18:Calculate IoU between predicted box and ground truth box
19:\(IoU_{\text{sum}}\gets IoU_{\text{sum}}+IoU\)
20:If probability is greater than current \(Best_{\text{metric}}\):
21:\(Best_{\text{metric}}\leftarrow\text{probability}\)
22:Store position information of the best detection
23:Categorization of Results:
24:Calculate \(P_{\text{detection}}\leftarrow\frac{N_{\text{correct}}}{\text{Total number of victims in GT}}\)
25: Calculate average IoU \(IoU\leftarrow\frac{IoU_{\text{sum}}}{N_{\text{correct}}}\)
26:Return:\(N_{\text{victims}}\), \(P_{\text{detection}}\), \(IoU\), \(Best_{\text{metric}}\), Position information
```

**Algorithm 2** CNN-based algorithm. Confusion matrices were employed to highlight the precision of class detection in order to evaluate the trained models. Figure 5 shows the three confusion matrices obtained by evaluating the trained models. The key findings reveal that the main diagonals exhibited high values, emphasizing the correct functionality of the models, with Thermal, Multispectral, and RGB models ranked in descending order of effectiveness. Figure 5: Cont. #### 3.3.3 Proposed Metrics for Method Analysis

A set of metrics is proposed, as described in Table 3, to assess the effectiveness of each vision system in victim detection. Following the state of the art and the authors' framework, these metrics encompass the primary evaluation criteria in the field of search and rescue for each system individually, addressing both functional conditions and the general performance of the proposed detection system. In a general mode, a direct evaluation of three types of sensors is proposed for the generic conditions present in diverse environments, incorporating normalized coefficients [0-100] by functionality efficacy, as delineated by Equation (3) (Thermal-RGB-Multi). Given the generality of the application framework, the time parameter was not considered in this instance, and the coefficients relating to victims focused generically on objects. \[\mathit{Score}_{\mathit{General}}(Thermal-RGB-Multi)=\alpha+\beta+\gamma++\epsilon+ \theta+\phi+\omega+\lambda+\tau \tag{3}\]

On the other hand, in a SAR approach, systems are evaluated concerning their correct functionality in each envisaged scenario. Similarly, specific parameters, such as processing time, are negatively penalized because time is a critical parameter in exploration. In contrast, others, such as robustness in changing light conditions, contribute more significantly to the total score. Likewise, critical situations, such as in identifying a concealed victim, are of particular interest, as is the ability to identify victims in poor lighting conditions, a recurrent situation in post-disaster environments. The expressions encapsulating the weighting relationship for the coefficients are summarized in Equation (4) for each type of image detection. The modified coefficients were established following the conducted experiments, assigning greater weight to those deemed more pertinent in Search and Rescue operations. Figure 5: Confusion matrix for CNN-trained models.