SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Compare Thrust to its various variants following the setting of the original work [63], where higher correlation denotes that the metric can better capture the hardness of tasks with respect to a given model (RoBERTa-large [32]). without direction denotes the variant to use scalar instead of vectors for Thrust. The best-performing entry is marked in bold.
Metric	Correlation
Thrust	0.45
without cluster size	0.23
without direction	0.19
without distance	0.06
cosine distance	0.08
one cluster per label class	0.32
ten clusters per label class	0.12
cluster size to inertia	0.03
Figure 8: Distribution of Thrust scores for all involved tasks with UnifiedQA-3b to create the instance representation. The distribution is normalized by Kernel Density Estimation. Low scores denote the cases where internal knowledge is not enough, and vice versa.
A.6 Full distribution of Thrust across tasks
Figure 8 demonstrates the distribution of Thrust scores for each of the involved tasks. Besides the findings in the main content that Thrust can help identify the knowledge necessity for various tasks through viewing the distribution, we can also observe that Thrust can lead to a diverse distribution of scores that may contain multiple peaks.
A.7 Performance of using external knowledge (in table)
Table 6 presents the performance in Figure 3 of the original submission in a table format. Similarly, we can observe that it is not trivial to use external knowledge, especially in the zero-shot settings, it is possible that models get worse performance with external knowledge, for example, for ARC-C, both T5-base and T5-large show worse performance with the extra knowledge injected. Also, we can observe that external knowledge is crucial for open-domain QA tasks. The gain can be huge, for example, for UnifiedQA-3b, the performance is improved from 18.6 to 80.0, in terms of QA-F1 on TriviaQA, with the external knowledge.
Figure 9: Performance of various models on MC classification tasks (accuracy) and open-domain QA tasks (QA-F1), denoted by (cls) and (qa), respectively. The x-axis represents the model names, which are shared across sub-figures. Use knowledge: yes or no denotes using full knowledge or not for all queries. UnifiedQA denotes T5 models with different sizes fine-tuned on the UnifiedQA dataset.
Table 6: Performance of various models on the MC classification tasks (accuracy) and open-domain QA tasks (QA-F1). Performances without/with knowledge external knowledge are presented before/after the vertical bar, respectively. UnifiedQA-X denotes T5 models with corresponding sizes fine-tuned on the UnifiedQA dataset.
Model	parameters	AGNews	e-SNLI	CIKQA	StrategyQA	BoolQ	ARC-E	ARC-C
Zero-shot
T5-base	220M	30.2	44.4	65.2	65.1	51.5	51.8	54.1	50.2	48.3	38.9	27.8	28.8	31.4	29.4
T5-large	770M	25.8	25.2	65.7	65.7	50.0	50.0	53.3	53.3	37.8	38.6	25.1	27.7	27.7	24.7
T5-3b	3B	27.9	39.1	57.6	61.5	52.6	50.5	44.5	48.9	56.6	45.3	25.8	26.0	26.4	28.4
GPT-J	6B	25.1	26.9	40.8	37.0	49.8	50.7	47.2	55.9	60.2	47.2	25.4	29.5	28.4	27.1
OPT-30b	30B	25.0	25.0	65.7	65.7	50.0	50.0	53.3	53.3	37.8	37.8	27.4	27.7	25.8	26.4
Transfer-learning
UnifiedQA-base	220M	46.6	35.7	38.5	70.2	56.0	59.6	48.5	57.2	60.4	80.8	50.2	61.6	44.8	45.2
UnifiedQA-large	770M	71.0	67.9	42.8	74.2	59.6	62.1	48.5	66.4	59.8	84.5	64.0	66.0	55.2	49.5
UnifiedQA-3b	3B	75.7	84.5	62.2	89.6	61.3	66.9	57.6	83.4	61.5	87.8	73.7	76.5	64.5	64.2
Model	parameters	Web Questions	Curated TREC	HotpotQA	NQ	TriviaQA
Zero-shot
T5-base	220M	6.7	8.7	2.5	3.8	6.0	9.1	1.9	6.0	8.9	13.1
T5-large	770M	5.7	7.4	1.9	3.0	5.1	6.6	1.6	2.7	8.3	9.0
T5-3b	3B	4.9	4.0	2.0	1.0	4.9	6.8	1.7	6.6	8.3	5.6
GPT-J	6B	4.3	6.3	7.4	2.7	5.9	5.1	10.9	6.9	1.7	2.1
OPT-30b	30B	18.3	6.3	16.0	2.4	11.4	2.4	5.3	2.1	16.3	6.9
Transfer-learning
UnifiedQA-base	220M	10.6	44.2	3.6	36.9	13.1	40.3	3.0	34.6	11.7	65.6
UnifiedQA-large	770M	12.9	46.5	8.2	36.6	14.2	42.7	3.8	36.3	13.7	74.6
UnifiedQA-3b	3B	11.5	48.1	9.9	41.8	17.0	47.0	4.4	37.6	18.6	80.0