# Explaining Deep Learning-Based Driver Models
## 3 Application of XRAI in Driver Modeling
### Applied XAI Technique: XRAI  
In this subsection, we explain briefly the _XRAI_[60], the XAI technique used to explain the decisions that the models of the ADAS are taking. As we mentioned in the background section, it is a post-hoc explainability technique, which means that it allows one to interpret easily why a black-box model, such as a deep or a convolutional neural network, is making its decisions. It is also a visualization technique, because this method generates explanatory images that aid in comprehending the model. As one would expect from a post-hoc technique, _XRAI_ takes the already trained model and adds a new layer of explainability. Because _XRAI_ is also implemented using Python and Tensorflow, it is perfectly compatible with the models of the target ADAS.  
Introducing _XRAI_'s explainability layer over the model allows us to explain a specific input. If we provide this layer with an image of the driver and the prediction of the driver's mood or activity according to the models, it will point out which part of the image was the most important to determine the models' predictions. To do this, it essentially follows three steps: divide the image into regions, calculate the attributions of these regions, and select which regions are more relevant to the decision.  
To divide the input image in regions, _XRAI_ first divides the image in six different ways by using [PERSON]'s graph-based method [69] with different scales, so that there are six sets of regions of different sizes. Since these sections' boundaries tend to align with the edges in the image, the regions are dilated so that they include thin edges inside the segment.  
After that, the attribution of the regions (that is, their importance for the prediction of that particular input) is calculated. To do that, _XRAI_ uses backpropagation in a similar way to the _integrated gradients_ technique, except that it uses two baselines (black and white) instead of one (black). To summarize it, it computes the gradient of all the points between the _baselines_ and the input, and these gradients are accumulated to obtain the attribution.  
Finally, the regions' attributions are evaluated to determine which regions are the most important. This process is relatively simple: the attributions of each region are summed, and those sections that sum to more positive values are the most important. This means that the regions can be sorted by importance, which gives the user the opportunity to choose how many regions they want to see: the top 10% more important, the top 20%, etc.  
Once _XRAI_ has calculated the regions' importance, we can visualize those sections. One way is representing those regions as a heatmap, which will allow us to notice the importance of the whole image, and the other way is to use the segments as a mask and show only a percentage of the original image based on the importance of each part. Figure 5 shows an example of both visualization methods.  
Figure 4: 3D representation of the activity model’s architecture. Photograph taken from the 3 MDAD dataset.  
To be able to see all the important characteristics of the image, in the Results section, we show the original image along with various heatmaps and segments visible. There are four views of the image where only a small percentage of it, corresponding to the relevant regions, are visible, which are associated with the top 10%, top 15%, top 20%, and top 30% most important regions. This allows us to understand which segments of the image the models are taking into account to make their predictions.  
Below the top regions, there are heatmaps with a different color intensity, since details about the regions are more noticeable at one intensity or another depending on the image. Heatmaps show all regions colored on a scale from black to white, where the most important regions appear in a brighter color. The preprocessed image is shown next to both visualizations, in black and white, as a reference.  
Figure 6 shows an example of an XRAI visualization.