SlowGuess/NAACL_2021_OCR huggingface dataset:
As described in Section 6.1 of the main paper, EASE selects samples with difficult question types for VQA2.0. Figure 2 (main paper) reports the proportion of question types that present in each split defined by EASE: as conjectured, we see a higher proportion of the other question type (i.e., wh-) and number questions in the hardest split of both datasets compared to the others. Yes/No questions are poorly represented in the hardest split, which suggests they are less challenging for humans and the models.
Figure 4: Distribution of samples in the validation splits of VQA2.0, against number of unique answers. E.g., in  $33\%$  samples in VQA2.0, all annotators gave the same answer.
Figure 5: Distribution of samples in the validation splits of VizWiz against number of unique answers. E.g., in  $3\%$  samples in VizWiz, all annotators gave the same answer.
Figure 6 shows similar pattern in VizWiz where the percentage of Other question types is higher in TOP-HARD split selected by EASE. It is interesting to see that the number of Unanswerable questions are very low in TOP-HARD. This shows another
Figure 6: Number of samples per question-type in VizWiz-train for each of the three splits used in HF training regime. Here, Other belongs to reasoning questions (why, which, where), Number to counting questions, and Yes/No to polar questions.
Dataset/Split	BUTD	LXM	LXM-S
VQA	TH	34.73	42.2	34.89
BH	71.31	78.66	71.22
E	74.98	84.38	74.21
VW	TH	44.40	46.79	41.48
BH	59.25	64.02	53.31
E	52.25	64.86	40.54
Table 4: Accuracy by BUTD, LXM, and LXM-S on three validation splits of VQA2.0 (VQA) and VW. The splits are obtained via Entropy.
property of EASE in which it didn't consider the usual notion of associating Unanswerable questions with hard ones, while look at human agreement/disagreement to decide difficulty.
D Other methods to split evaluation data
As discussed in Section 3.3, we obtained TH, BH, E splits using Entropy and Random Selection. We use Eq. 4 to compute Entropy over the original answer distribution, and then subtract the score from 1.
$$
E \left(p _ {f}\right) = \frac {- 1}{\eta} \sum_ {k = 1} ^ {M} p _ {k} * \log \left(p _ {k}\right) \tag {4}
$$
We use the same criterion, as EASE, to divide our samples into TH, BH, and E. Table 4 shows that, contrary to EASE splits, in VizWiz, both BUTD and LXM-S achieve higher results in BH compared
Dataset/Split	BUTD	LXM	LXM-S
VQA	TH	63.42	71.56	63.10
BH	63.45	71.43	63.16
E	63.35	71.46	63.17
VW	TH	50.31	53.59	45.79
BH	49.88	53.31	45.97
E	50.26	53.77	46.18
Table 5: Accuracy by BUTD, LXM, and LXM-S on three random splits of validation data of VQA2.0 (VQA) and VW. The random splits are of same size as that of TH, BH, and E as mentioned in Section 3.3
Figure 7: Average EaSe scores per confidence scores provided by annotators for both splits of VizWiz. Open/close brackets indicate that values are not/ included.
to E, which indicates that Entropy is not as effective as our tool in measuring the difficulty of a sample.
In the Random Selection, we tested the model's performance on splits having similar sizes of EASE's TH, BH, and E split. Table 5 shows that the three splits have the almost same accuracy. In particular, the random splits don't show any pattern, unlike EASE in which TH always gets the lowest accuracy and E gets the highest (Table 2 of main paper). These results clearly show that EASE selects the smallest subset of training data which is both informative and hard.
E EASE vs. Confidence scores
As discussed in Section 6.2 of main paper, we test correlation of EaSe scores with already available human confidences. We map confidence scores yes, maybe, no to 1, 0.5, and 0, respectively, and compute the average confidence score for each sample.
We then compute relationship between confidence scores and EASE scores. Fig. 7 shows the analysis for VizWiz data, where higher confidence scores correspond to increasingly higher EASE values. This shows that EaSe correlates with human intuition of having difficulty to answer a question.