SlowGuess/NeurIPS_2023_OCR huggingface dataset:
User Perception Analyses Below we present additional evaluations performed on user responses in our questionnaire. Specifically, we present participant's perceived performance Perf and participant's perceived explanation utility EU. Note, Perf is measured on a 5-point Likert scale to the following question "I believe my game performance improved after witnessing the expert play and seeing the expert's provided reasoning". The EU metric is measured on a 5-point Likert scale to the following question "I believe the expert's provided reasoning helped improve my understanding of the game", and is only measured for the explanation conditions. From both Figure 14 and Figure 15 we observe that participants across all study conditions had high agrees for Perf and EU, despite their being significant differences in their objective task performance. These qualitative results demonstrate the Dunning-Krunger Effect [13], in that users with low expertise in an area often overestimate their own performance or knowledge.
Introduction to Connect 4
In this study, you will be playing the game of Connect 4. The image below shows what the game environment looks like. Please use a desktop or laptop computer for this study.
The primary objective of the game is to achieve a four in a row with your colored tokens before the opponent. You will always be the first player to play.
To place a token in a particular column, specify the column number as depicted in the picture above. For example, the fourth column from the right is denoted as "C4".
Your Task
We will be examining how different forms of information help in improving your understanding of the game as well as your task performance. Your task will consist of the following:
watch 1 expert game to get acquainted to the game
- play 2 practice games to get acquainted to the game
- play 3 assessment games that will measure your ability to achieve 4-in-a-row before your opponent
3 interactive sessions watching an expert player playing the game
- play 3 more assessment games that will measure your ability to achieve 4-in-a-row before your opponent
- short survey related to your experience and background.
Continue Below to Enter TRIK ID.
How to Enter York ID
Introduction to Lunar Lander
In this study, you will be playing the game of lunar lander. The image below shows what the game environment looks like. The lander has 3 different engines: a main engine, and two side engines. Please use a desktop or laptop computer for this study.
__________
The primary objective of the game is to get the lander to land between the flag poles, as close to the center as possible, without crashing. Additional information is provided to enable the lander without any angle tilt, and conserve as much fuel as possible. The two types of engines (pole,桨) have the following characteristics:
Actions: to move the lander, you can play one of four actions at a time. Note, you will be specifying the action to play by clicking buttons on each page. The four possible actions are:
Fire Left Engine
Fire Right Engine
- Fire Main Engine
We will be examining how different forms of information help in improving your understanding of the game as well as your task performance. Your task will consist of the following:
watch 1 expert game to get acquainted to the game
- play 2 practice games to get acquainted to the game
- play 3 assessment games that will measure your ability to land the lander
- 3 Interplay exercises (optional) on expert probes relating the game
- 3 interactive sessions watching an expert player playing the game
- play 3 more assessment games that will measure user ability to land the land
- play a more assessment game that will measure your job
- short survey related to your experience and background
Continue Below to Enter TURK ID
Figure 16: Participants saw the following as introductions to the task for (a) Connect 4 and (b) Lunar Lander.
(b)
Figure 17: Participants saw the following as introductions to the game for (a) Connect 4 and (b) Lunar Lander.
E.4 Visuals of User Study
In this section, we provide visuals of the user interface for each user study.
Introduction Stage In both user studies, after receiving consent of participation, all participants were given an introduction to both the user study as well as the specific domain of the study. Figure 16a(a) and Figure 16b(b) show the introductions participants received for Connect 4, while Figure 17a(a) and Figure 17b(b) show the introductions participants received for Lunar Lander. Note the
Figure 18: Visualizations of the game interface participants utilized during the Practice, Pre-Test and Post-Test stages for both Connect 4 (a) and Lunar Lander (b).
(a)
(b)
Figure 19: Visualizations of the Explanation stage for both (a) Connect 4 and (b) Lunar Lander.
introduction to the task was first presented to the users, and then the introduction to the domain was presented.
Game-Playing Stages During the study, participants played various games during the Practice, Pre-Test and Post-Test stages. These games were in real-time; users clicked the action to play via a button, and saw the game interface update accordingly. Figure 18a(a) and Figure 18b(b) present visuals on how participants played games in each domain.
Explanation Stage During the Explanation stage, participants interacted with an expert player (expert RL agent) and stepped through the expert player's actions while being exposed to explanations from a given study condition. The participants were not told that the expert player was an RL agent to limit the confounding effect of user biases towards AI agents in study. Figure 19a(a) and Figure 19b(b) provide an example of the explanation stage for Connect 4 and Lunar Lander. Specifically, in Figure 19a(a), a  $\mathcal{E}_{C_i}$  is provided, and in Figure 19b(b), a  $\mathcal{E}_{C_i\mathrm{w / InF},\mathrm{TeG}}$  is provided.
Survey At the end of the user study, participants were asked to fill out a short survey including demographic questions, as well as additional Likert questions to gauge user experiences with computers, the domain, as well as their perceived performance and explanation utility. Figure 20 provides a visual of what the survey looked liked in the domain of Connect 4. The survey for Lunar Lander participants was identical, except making an reference to Lunar Lander as opposed to Connect 4.
F Github Repo
https://anonymous.4open.science/r/S2E/README.md
Wrap Up Survey
Great job, you are almost done! Please answer a few final questions to help us with our study.
Question 1. Please select your age range:
18-24
25-34
35-44
45-54
55 and older
$\bigcirc$  Prefer not to say
Question 2. Please select your gender:
Male
Female
Other
$\bigcirc$  Prefer not to say
Question 3. Please indicate your computer usage frequency
Question 4. Please indicate the frequency in which you play computer games
Question 5. Please indicate your level of exposure to Connect 4 prior to this study
Very Familiar
Moderately Familiar
Slightly Familiar
Not Familiar
Question 6. I believe my game performance improved after witnessing the expert play and seeing the expert's provided reasoning
Strongly Disagree
Question 7. I believe the expert's provided reasoning helped improve my understanding of the game
Continue to Finish
Figure 20: Visualization of the survey questions participants received at the end of the Connect 4 Study; questions were identical for the Lunar Lander study.