## 3 Results
The darker the region, the smaller the value on the map. The bright areas correspond to a value of 1 on the GradCAM map. Figure 10: CART explanation generated by the proposed GradCAM-based method for the Agent’s NN with \((64\times 64)\) input image signal, “Simple” convolutional feature embedder and \((\alpha=1,t=0.3)\) parameters of Algorithm 2. This particular tree explains the rules of left–right motion. The size of the tree is limited to maximal depth of 4. Instances of classes among certain features are presented in color-coded bars. The black arrow positioned under the horizontal axis indicates the splitting threshold. Figure 9: CART explanation generated by the proposed GradCAM-based method for [PERSON]’s NN with \((64\times 64)\) input image signal, “Simple” convolutional feature embedder and \((\alpha=1,t=0.3)\) parameters of Algorithm 2. This particular tree explains the rules of forward–backward motion. The size of the tree is limited to maximal depth of 4. Instances of classes among certain features are presented in color-coded bars. The black arrow positioned under the horizontal axis indicates the splitting threshold. Figure 4 in each row presents the triplet of the input image (view from the Agent's camera), its semantic (object class-based) segmentation and visualization of the Agent's position (represented by a white capsule shape) towards the blue platform (a bird-eye view of the scene). Figures 12-14 present the decision process (predictions) of the CART tree generated by the proposed GradCAM-based method for Agent's NN with \((64\times 64)\) input image signal, \"Simple\" convolutional feature embedder, \((\alpha=1,t=0.3)\) on images from Figure 4. The orange arrow under the horizontal axis indicates the actual feature value. These figures show an explanation of the decision-making process for a single-agent camera reading. They contain single tree paths from trees shown in Figures 9-11. In Tables 1 and 2, we present the results of statistical cross-validation analysis of our proposed method; best results are marked bold. We utilized the leave-one-out accuracy test of various configurations of the Agent's NN explanation method presented in Section 2.4. Rows represent various parameter settings, and columns represent the resolution of the input image. Results are averaged values of all correct classifications of three possible agent actions (motion, rotation, jumping) over all possible decisions, which means the following:  
\[ACC=\frac{P_{fb}+P_{lr}+P_{j}}{P_{fb}+N_{fb}+P_{lr}+N_{lr}+P_{j}+N_{j}} \tag{7}\]  
where \(P_{fb}\)\(P_{lr}\)\(P_{j}\) are positive decisions (true positive + true negative) and \(N_{fb}\)\(N_{lr}\)\(N_{j}\) are negative decisions (false positive + false negative) for forward-backward, left-right and jump decisions appropriately. Figure 11: CART explanation generated by the proposed GradCAM-based method for Agent’s NN with \(64\times 64\) input image signal, “Simple” convolutional feature embedder and \((\alpha=1,t=0.3)\) parameters of Algorithm 2. This particular tree explains the rules of jump motion. Instances of classes among certain features are presented in color-coded bars. The black arrow positioned under the horizontal axis indicates the splitting threshold. Figure 12: This Figure presents the decision process (predictions) of the CART tree generated by the proposed GradCAM-based method for Agent’s NN with \((64\times 64)\) input image signal, “Simple” convolutional features embedder, \((\alpha=1,t=0.3)\) on images from Figure 6:0 The orange arrow under the horizontal axis indicates the actual feature value. This Figure shows an explanation of the decision-making process for a single agent camera reading. It contains a single tree path from the tree shown in Figure 6:1 (**a**) Prediction for input image from Figure 4a. (**b**) Prediction for input image from Figure 4d.