## 2 Materials and Methods
### Decision Trees, Proxy Geometry and Semantic Scene Segmentation  
An informative way to represent an algorithm is to describe it as a flowchart containing control instructions such as loops, conditional statements, etc. An alternative approach is to use a special case of the block diagram as what can be considered a decision tree. A representation of a deep model in the form of a decision tree can meet the requirements for an explanation for a black-box machine learning model because we can estimate its accuracy (how well an explanation predicts unseen data), fidelity (explanations ought to be close to the predictions of the explained) and comprehensibility (explanations should be understandable to humans) [53]. Using a decision tree to explain an algorithm that relies on visual data is more complex. Several factors influence this:  
* Input image to the model, even if it is not high-resolution, for example, 48 \(\times\) 48 pixels, contains 2304 features. This is far too many to build an explanation of the model. * Individual pixels are not a semantic interpretation of what is in the image. What is interpretable is their position relative to each other. The convolution layer, acting as a multi-cascade image filter, analyzes the textures and mutual position of graphic primitives in the image. Therefore, to make the vision model explainable as a decision tree, it is necessary to reduce the number of features and replace visual information, which is based on colored pixels, with semantic information, which is based on types (classes) of objects located in specific areas of the input image. The limitation of the number of features should be tuned so that the new set of features retains information about the spatial position of objects in the image and is easy to interpret for human cognitive interpretation. The labels of individual features should also be easy to identify. To meet all these requirements, we use a proxy geometry known for creating visual human-computer interfaces [55] based on dividing the space into up to \((3\times 3)\) parts whose names are based on geographic directions. Depending on the resolution of the \((n\times m)\) GradCAM map \(M^{c}\), there may be several combinations of the set of proxy geometries, which we show in Figure 3. The type of proxy geometry is calculated according to Algorithm 1 by executing it for the values \(n\) and \(m\), and the proxy region borders thus built are stored in variables \(\mathit{borders}_{n},\mathit{borders}_{m}\), respectively. The naming convention for the individual features that are in matrix \(F\) is as in Figure 3. This algorithm does not consider the size of the input image \(I\) but the resolution of \(M^{c}\). This is because if \(M^{c}\) has a relatively small resolution, after scaling it to the resolution of \(I\) and extracting features from it in the rest of the algorithm, interpolation of map values during resampling may result in the assignment of objects from different areas of \(M^{c}\) to one proxy geometry area. If the resolution of \(M^{c}\) is much smaller than the resolution of the image, this can result in an erroneous averaging of the excitation areas of \(M^{c}\) into several adjacent proxy geometry areas. The next important step is to extract the semantic information from the input image, in other words, segmenting the objects that make up the observed scene. This is an intuitive approach: to understand the operation of the decision-making process based on the image content, the user does not derive relevant information from the color values in the observed proxy geometry area but is rather interested in the types of objects that the algorithm observes. It is therefore necessary to provide a semantic segmentation of the observed scene to the explanatory algorithm. Another machine learning algorithm can be used for  
Figure 3: Proxy geometry we used in proposed method. this, for example, [56; 57; 58; 59], or if the algorithm is tested in a virtual environment, data can be obtained directly from it.