SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We further included full results for our Mixed Linear Effects Modeling statistical tests in Table 6. All but two effects are not significant at the  $p < 0.05$  level. The two significant effects are 1) Question 1 had a significant negative effect on accuracy rate, and 2) there was a significant positive interaction effect between visualizing VQ-VIB $_C$  prototypes and Question 1. Jointly, these two effects show that Question 1 was harder for users than the other questions, but that seeing VQ-VIB $_C$  prototypes to some extent mitigated this increased difficulty.
Table 6: Results of the Mixed Linear Effects Modeling of user responses, predicting accuracy as a function of model and visualization type and Question. Our key results is that there is a significant positive interaction effect between visualizing VQ-VIB $_C$  prototypes and performing better on Question 1 (bolded for emphasis). The intercept row corresponds to Question 2 with VQ-VIB $_C$ -MSE.
Coef.	Std.Err.	z	P|z|	[0.025	0.975]
Intercept	0.700	0.084	8.356	0.000	0.536	0.864
VQ-VIBC-Proto	0.050	0.118	0.421	0.674	-0.182	0.282
VQ-VIBN-MSE	0.100	0.118	0.845	0.398	-0.132	0.332
VQ-VIBN-Proto	0.200	0.119	1.692	0.091	-0.032	0.433
Question 1	-0.600	0.118	-5.085	0.000	-0.831	-0.369
Question 3	0.150	0.118	1.271	0.204	-0.081	0.381
VQ-VIBC-Proto:Question 1	0.400	0.167	2.397	0.017	0.073	0.727
VQ-VIBN-MSE:Question 1	-0.100	0.167	-0.599	0.549	-0.427	0.227
VQ-VIBN-Proto:Question 1	-0.200	0.167	-1.199	0.231	-0.527	0.127
VQ-VIBC-Proto:Question 3	-0.000	0.167	-0.000	1.000	-0.327	0.327
VQ-VIBN-MSE:Question 3	-0.100	0.167	-0.599	0.549	-0.427	0.227
VQ-VIBN-Proto:Question 3	-0.150	0.167	-0.899	0.369	-0.477	0.177
Group Var	0.001	0.024
F User Study
In the subsequent pages, we have included the exact pdf document of a survey shared with participants of the user study. This pdf was generated for a participant viewing VQ-VIB $_C$  prototypes; similar surveys were populated with data for other models (VQ-VIB $_N$ ) or visualization methods (MSE plots). Furthermore, the order of the three questions was randomized to avoid ordering effects.
Please do not take this study on a mobile phone, the text and images will not display correctly.
You are free to leave this study at any time.
We will (1) explain the survey format, (2) include two examples of how to complete the survey, and then (3) ask three questions. In total, the survey will take around five minutes.
After you are finished with the survey, you will be redirected to Prolific to recieve your payment.
Please do not take this study if you have done a similar one before.
Thank you for your participation!
Please enter your Prolific ID
A computer program is good at classifying clothes into these 10 categories
T-shirt
- Trouser
- Pullover
- Dress
Coat
- Sandal
- Shirt
Sneaker
- Bag
- Ankleboot
See below for example visualizations of these 10 categories
However, imagine that you don't care about these 10 specific categories.
Instead, you want to categorize them into more high-level groups.
For example, you may only want to sort your clothes into these more general 3 groups:
- Group 1: t-shirt, shirt, pullover, and dress
- Group 2: boot, sandal, ankleboot, and sneaker
- Group 3: bag and trouser
Your task is the following:
Step 1: Read a description of the categories that the computer program needs to sort items of clothing into. These categories will change for different questions, so make sure to re-read the categories.
Step 2: Look at visualizations of different computer programs.
Step 3: Based on the visualizations from Step 2, select which of the programs you think will be best able to accomplish the task described in Step 1. In general, try to select the visualization that 1) can represent the different categories we ask for and 2) has the fewest representations, while still having enough representations to distinguish between categories.
On the next page, we have included an example of how to follow these steps.
Example Question
Example Question 1:
Step 1:
The categories are:
1. Group 1: T-shirt, Shirt, Pullover
2. Group 2: Purse
Step 2:
Select which of the following visualizations best reflects the categories from Step 1.
Option 3
Option 2 is best for this task because it demonstrates visualizations of the two groupings that we care about here (distinguishing between purses and different types of shirts) while using few representations.
Options 1 has enough representations for the two groups we care about, but it has more representations than Option 2, so Option 2 is a better choice.
Option 3 is too simple and therefore cannot tell the two groupings apart.
Step 3 (Example 1):
Please select which option you think best represents the groupings listed:
Example Question 2:
1. Group 1: T-shirt
2. Group 2: Pullover and Shirt
3. Group 3: Purse
Option 1 is best for this task because it demonstrates visualizations of the three groupings that we care about here. Because we care about distinguishing between t-shirts and shirts (Group 1 vs. 2), we need to be able to tell t-shirts apart based on the visualizations.
Options 2 and 3 are bad choices because they cannot tell t-shirts (Group 1) apart from other pullovers or shirts (Group 2). Therefore, even though they use fewer representations
than Option 1, they are bad options.
Step 3 (Example 2):
Please only participate in this study if you are confident you understand the instructions correctly
Click next to begin the study
The computer program needs to sort the items into the following 3 categories:
1. Group 1: T-shirt, Trouser, Sandal
2. Group 2: Pullover, Dress, Sneaker
- 3. Group 3: Coat, Shirt, Bag, Ankle Boot
Please look at the options below:
Option 4
Step 3:
1. Group 1: T-shirt, Pullover, Coat, Shirt
2. Group 2: Trouser, Dress, Bag
- 3. Group 3: Sandal, Sneaker, Ankle boot
The computer program needs to sort the items into the following 10 categories:
1. T-shirt
2. Trouser
3. Pullover
4. Dress
5. Coat
6. Sandal
7.Shirt
8. Sneaker
9. Bag
10. Ankleboot
Block 9
Debrief Page
Thank you for your participation, this study was designed to evaluate people's ability to select the right level of abstraction for an AI to perform well at a particular task.
if you have any questions please contact ekenny@mit.edu
Click next to finish
Powered by Qualtrics