# Comparing Approaches for Explaining DNN-Based Facial Expression Classifications
## 4 Experimental Setting
### User Survey Construction
#### 4.4.2 Questions  
With the help of the user study, we want to quantify several qualities of the geometric features-based explanations. On top of that, we want to compare those explanations with the ones for the CNN on a human-level. All questions are answered via a Likert item from 1 (Strongly Disagree) to 5 (Strongly Agree).  
In [8], [PERSON] et al. propose several checklists to evaluate the goodness, satisfaction and trust of explanations generated for AI systems. Goodness refers to how good an explanation is, determined by factors such as clarity and precision. Satisfaction is defined as: \"the degree to which users feel that they understand the AI system or process being explained to them\" [8]. Several of their proposed questions are used in the first set of questions for the user study.  
The System Usability Scale (SUS) [48] is a widely used tool to measure the usability of a system. This scale can be adapted to refer to a system's explanatory power instead of referring to a system's usability. As is done in [49], where [PERSON] et al. propose the System Causability Scale (SCS), which extends SUS to measure the quality of explanations in terms of causability, we extend SUS to be used in our user study. Several questions in the first question set are based on the SUS. Ultimately, the first question set consists of the following:  
1. The output representations help me understand how the model works. (adapted from [8])
2. The output representations of how the model works are satisfying. (adapted from [8])
3. The output representations are sufficiently detailed. (adapted from [8])
4. The output representations let me know how confident the model is for individual predictions.  
The output representations let me know how trustworthy the model is. (adapted from [8])
6. I found the output representations unnecessarily complex. (adapted from [48])
7. I think I would need an expert to give me additional explanations. (adapted from [48])
8. The outputs of the model are very predictable. (adapted from [8])
9. The model can perform the task better than a novice human. (adapted from [8])
10. I am confident in the model. I believe it works well. (adapted from [8])  
The second set of questions is partly extended from the first set. Instead of referring to the explainability of a single model, these questions make a comparison between two models. Again, the questions deal with goodness, satisfaction and trust, but this time in terms of which explanation the user finds better on several aspects. Again, we took into account questions regarding the intelligibility, complexity, level of detail and trust in the explanations. In these questions, there is a consistent reference to \"model 1\" and \"model 2\". In all cases, model 1 refers to the CNN and model 2 refers to the DNN. The second question set consists of the following:  
1. The explanations for model 1 are more understandable than those for model 2.
2. I trust model 1 more than model 2.
3. I would prefer the explanations of model 1 over those for model 2.
4. The explanations for model 1 are more detailed than those for model 2.
5. The explanations for model 1 are clearer on the model's accuracy than those for model 2.
6. The explanations for model 1 reflect the model's confidence on each prediction better than those of model 2.
7. Model 1's explanations are more unnecessarily complex than those of model 2.
8. The explanations for model 1 were more precise than those for model 2.
9. I would follow model 1's advice over that of model 2.
10. The outputs of model 1 were more predictable than those of model 2.