## 3 Results  
The proposed method for explaining reinforcement learning-based machine learning agents trained with Proximal Policy Optimization was evaluated on a PC computer with Intel i7 2.3 GHz, 32 GB RAM, GeForce RTX 3050 GPU running Windows 11 OS. We used Unity engine 2021.3.18f1 and MI Agents 2.0.1 for machine learning. The framework for agent training was Python 3.9 with libraries Torch 2.2.2, Tensorboard 2.17.1, mlagents 0.30.0, mlagents-envs 0.30.0, gym 0.26.2. For model explanation, we used Python 3.10, tensorflow 2.8, keras 2.8, scikit-learn 1.5.2, onnx 1.14.1, onnx2 kears 0.0.24 ([[https://github.com/gmalivenko/onnx2](https://github.com/gmalivenko/onnx2) keras]([https://github.com/gmalivenko/onnx2](https://github.com/gmalivenko/onnx2) keras) [accessed on 9 October 2024]). MI Agents generate a network in Onnx format that has to be converted to the Keras NN model to enable numerical differentiation of the network graph. We performed all 12 experiments described in Section 2.6. A total of \(15\cdot 10^{5}\) episodes were enough for each evaluated NN model to obtain 100% efficiency in solving the problem of acquiring a Collectible object by an Agent. The plot with Cumulative Reward value during PPO training is presented in Figure 6. The vertical axis presents the number of training steps, while the vertical axis presents the cumulative reward of an agent. After training the agents so that each configuration had 100% efficiency, we generated recordings of 1000 consecutive images from that Agent's visual sensor and the corresponding semantic segmentations for those images. We used these images and segmentations to generate explanations according to our proposed method and validation with the leave-one-out test. In other words, each CART tree was trained on 999 images and validated on one, and the validation process was repeated 1000 times for each tree. A detailed description of the experiment setup can be found in Section 2.6. Figure 6: The plot of Cumulative Reward value during PPO training steps (episodes). In Figures 7 and 8, we present example GradCAM results for various Agent NNs differing with a resolution of the image sensor. All agents use a \"Simple\" convolutional features embedder. In (**a**), the semantic clustering of the input image (**h**) is presented. The first row presents the GradCAM color-coded map generated as the response of the NN for the input image (**h**). The second row shows the same GradCAM map but is imposed onto the input signal. The darker the region, the smaller the value on the map. The bright areas correspond to a value of one on the GradCAM map. In Figures 9-11, we present a CART explanation generated by the proposed GradCAM-based method for the Agent's NN with \((64\times 64)\) input image signal, \"Simple\" convolutional feature embedder, and \((\alpha=1,t=0.3)\) parameters of Algorithm 2. Those trees explain the rules of front-back, left-right, and jumping motion, respectively. Instances of classes among certain features are presented in color-coded bars. The black arrow positioned under the horizontal axis indicates the splitting threshold. Figure 8: Example GradCAM results for various Agent NNs differing with a resolution of the image sensor. All agents use a “Simple” convolutional features embedder. (**a**) Semantic clustering of the input image (**h**). The first row presents the GradCAM color-coded map generated as the response of NN for input image (**h**). The second row shows the same GradCAM map but is imposed onto the input signal. The darker the region, the smaller the value on the map. The bright areas correspond to a value of 1 on the GradCAM map. Figure 7: Example GradCAM results for various Agent NNs differing with a resolution of the image sensor. All agents use a “Simple” convolutional features embedder. (**a**) The semantic clustering of the input image (**h**). The first row presents the GradCAM color-coded map generated as the response of NN for input image (**h**). The second row shows the same GradCAM map but is imposed onto the input signal.