SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 7: XNLI Pearson correlation between the performance on English and the performance on other languages using the same set of samples.
de	en	es	fr	ja	ko	zh
de	1.00	0.66	0.52	0.56	0.21	0.54	0.64
en	0.66	1.00	0.56	0.41	0.11	0.37	0.36
es	0.52	0.56	1.00	0.57	0.22	0.54	0.57
fr	0.56	0.41	0.57	1.00	0.03	0.59	0.55
ja	0.21	0.11	0.22	0.03	1.00	0.16	0.32
ko	0.54	0.37	0.54	0.59	0.16	1.00	0.54
zh	0.64	0.36	0.57	0.55	0.32	0.54	1.00
avg	0.59	0.50	0.57	0.53	0.29	0.54	0.57
Table 8: PAWS-X Pearson correlation of the performance between languages.
lang	de	es	fr	ja	ko	zh	avg
10	0.47	0.65	0.34	-0.22	0.53	0.56	0.48
50	0.81	0.56	0.57	-0.35	0.53	0.48	0.51
100	0.78	0.53	0.42	0.40	0.47	0.44	0.57
500	0.52	0.55	0.53	0.16	0.41	0.11	0.47
1000	0.75	0.77	0.30	-0.01	-0.02	0.35	0.45
all	0.66	0.56	0.41	0.11	0.37	0.36	0.50
Table 9: PAWS-X Pearson correlation between the performance on English and the performance on other languages using the same set of samples.
de	es	fr	ja	ko	zh
0.66	0.62	0.68	0.45	0.38	0.52
Table 10: PAWS-X Pearson correlation between the performance of machine translation and manual translation.
ar	de	zh	vi	en	es	hi	el	th	tr	ru	ro
ar	1.00	-0.14	0.03	0.07	0.12	-0.02	0.01	-0.03	0.07	0.25	0.12	-0.06
de	-0.14	1.00	-0.54	-0.18	0.35	0.57	0.42	0.22	-0.26	0.40	-0.09	-0.00
zh	0.03	-0.54	1.00	0.16	-0.37	-0.38	-0.21	-0.41	0.55	-0.17	-0.24	-0.22
vi	0.07	-0.18	0.16	1.00	-0.08	-0.02	-0.08	-0.01	0.02	-0.18	-0.12	-0.26
en	0.12	0.35	-0.37	-0.08	1.00	0.46	0.08	0.07	-0.17	0.06	-0.04	-0.06
es	-0.02	0.57	-0.38	-0.02	0.46	1.00	0.10	0.02	-0.31	0.09	-0.29	-0.24
hi	0.01	0.42	-0.21	-0.08	0.08	0.10	1.00	0.18	0.06	0.37	0.27	0.18
el	-0.03	0.22	-0.41	-0.01	0.07	0.02	0.18	1.00	-0.15	0.01	0.34	0.13
th	0.07	-0.26	0.55	0.02	-0.17	-0.31	0.06	-0.15	1.00	0.17	0.07	0.10
tr	0.25	0.40	-0.17	-0.18	0.06	0.09	0.37	0.01	0.17	1.00	0.33	0.27
ru	0.12	-0.09	-0.24	-0.12	-0.04	-0.29	0.27	0.34	0.07	0.33	1.00	0.56
ro	-0.06	-0.00	-0.22	-0.26	-0.06	-0.24	0.18	0.13	0.10	0.27	0.56	1.00
avg	0.12	0.15	-0.07	0.03	0.12	0.08	0.20	0.11	0.10	0.22	0.16	0.12
Table 11: XQuAD Pearson correlation of the performance between languages.
lang	ar	de	zh	vi	es	hi	el	th	tr	ru	ro	avg
10	0.54	0.41	0.08	-0.40	0.30	0.05	-0.10	0.43	0.44	-0.25	-0.23	0.19
50	0.37	0.24	-0.28	0.11	-0.01	0.19	0.27	0.12	0.21	-0.04	-0.08	0.18
100	-0.37	0.35	-0.54	-0.03	0.71	0.02	0.08	-0.09	-0.08	-0.40	-0.12	0.05
250	0.08	0.20	-0.25	0.03	0.65	-0.16	-0.38	-0.31	-0.45	-0.33	-0.34	-0.02
all	0.12	0.35	-0.37	-0.08	0.46	0.08	0.07	-0.17	0.06	-0.04	-0.06	0.12
Table 12: XQuAD Pearson correlation between the performance on English and the performance on other languages using the same set of samples.
XNLI	PAWS-X
MSE	RMSE	MSE	RMSE
avg (baseline)	1.05±0.56	0.99±0.26	1.26±0.76	1.08±0.34
model using features:
en performance	0.68±0.41	0.80±0.23	1.08±0.92	0.97±0.42
mt performance	0.34±0.28	0.56±0.20	0.92±0.56	0.93±0.28
en + mt performance	0.33±0.26	0.55±0.18	0.91±0.56	0.92±0.28
+ lang features	0.32±0.25	0.54±0.18	0.58±0.27	0.75±0.17
only lang features	0.93±0.47	0.93±0.24	1.01±0.45	0.98±0.25
Table 13: Performance prediction error. Predicting the few-shot performance gain using models with the English and MT performance as features. For each language, the average performance gain for all other languages is used as a baseline. +lang features further adds features from lang2vec. Having a combination of English and MT performance with language features achieves the best results.
en -	1.89%	1.17%	1.69%	1.29%	1.27%
ar -	2.30%	1.12%	1.80%	1.68%	1.08%
bg -	2.35%	1.59%	2.55%	1.83%	1.85%
de -	2.03%	0.94%	1.75%	1.28%	1.32%
el -	1.40%	0.92%	1.02%	1.08%	0.84%
es -	1.70%	0.78%	1.08%	1.40%	1.56%
fr -	1.06%	0.72%	0.84%	0.34%	0.52%
hi -	2.27%	1.95%	2.15%	1.80%	1.78%
ru -	2.01%	1.48%	2.03%	1.44%	1.06%
sw -	1.84%	0.90%	1.34%	1.62%	0.92%
th -	3.51%	3.27%	3.47%	2.81%	3.11%
tr -	2.52%	1.26%	2.50%	2.38%	1.76%
ur -	3.77%	2.29%	3.73%	2.99%	2.91%
vi -	3.45%	2.52%	2.66%	2.97%	2.32%
zh -	3.43%	2.94%	3.00%	3.02%	2.80%
avg -	2.44%	1.64%	2.18%	1.92%	1.73%
A	B	C	D	E
en	1.24%	0.68%	1.22%	0.84%	1.04%
ar	2.85%	2.21%	2.19%	2.37%	1.89%
bg	1.92%	1.54%	1.82%	1.54%	1.36%
de	2.87%	1.79%	2.63%	1.65%	2.45%
el	1.56%	1.46%	2.47%	1.36%	2.09%
es	2.32%	0.76%	1.84%	1.72%	1.58%
fr	0.90%	0.44%	0.82%	0.04%	0.46%
hi	1.24%	1.80%	1.86%	1.22%	1.70%
ru	1.89%	0.82%	1.12%	1.18%	1.10%
sw	2.08%	2.00%	2.31%	2.59%	0.96%
th	3.61%	3.91%	3.77%	3.39%	3.29%
tr	2.36%	1.14%	2.16%	1.84%	1.76%
ur	2.27%	1.87%	2.79%	1.87%	1.47%
vi	3.17%	2.46%	3.21%	3.09%	1.92%
zh	3.00%	2.92%	2.76%	2.68%	2.08%
avg	2.29%	1.79%	2.27%	1.90%	1.72%
A	B	C	D	E
en -	1.20%	1.06%	0.88%	1.00%	1.02%
ar -	2.95%	2.35%	2.83%	2.77%	2.29%
bg -	1.72%	0.82%	2.02%	1.38%	1.42%
de -	2.21%	1.13%	2.41%	2.09%	2.43%
el -	1.61%	1.50%	1.97%	1.69%	2.27%
es -	1.56%	0.82%	1.46%	1.76%	1.44%
fr -	1.78%	0.66%	1.56%	0.78%	1.28%
hi -	1.62%	2.31%	2.65%	1.90%	1.90%
ru -	1.40%	0.94%	1.36%	1.89%	1.55%
sw -	0.92%	1.18%	0.78%	1.04%	0.84%
th -	3.41%	3.19%	2.77%	3.11%	3.03%
tr -	2.20%	1.72%	2.38%	2.04%	2.42%
ur -	3.65%	3.05%	3.63%	2.23%	3.25%
vi -	3.61%	2.99%	3.65%	3.53%	3.17%
zh -	3.32%	2.82%	3.30%	2.52%	3.12%
avg -	2.28%	1.82%	2.34%	2.05%	2.17%
A	B	C	D	E
Figure 20: XNLI few-shot gain over zero-shot across 5 sets of samples (size=1000) for 3 different model initializations. Sets A and C yield better performance for the 3 different initializations. The English performance can be used as an indicator.
(a) XNLI chosen-shots gain using English performance
(b) XNLI chosen-shots gain using machine translation performance
(c) XNLI chosen-shots gain using (en + mt) model
(d) XNLI chosen-shots gain using  $(\mathrm{en} + \mathrm{mt} + \mathrm{lang}$  features) model
(e) PAWSX chosen-shots gain using English performance
(f) PAWS-X chosen-shots gain using machine translation performance
(g) PAWS-X chosen-shots gain using  $(\mathrm{en} + \mathrm{mt})$  model
(h) PAWS-X chosen-shots gain using  $(\mathrm{en} + \mathrm{mt} + \mathrm{lang}$  features) model
Figure 21: Chosen-shots gain in performance. The gain of choosing shots over the average of no-choosing (average over 5 random sets). The actual few-shot gain (compared to zero-shot) is shown in parenthesis as follows: chosen-shots-gain (few-shot-gain). When chosen-shots-gain is positive (green), choosing the shots results in more gain. When negative (red), it hurts and results in less gain.
(a) XQuAD chosen-shots gain using English performance
(b) XQuAD chosen-shots gain using en performance model
Figure 22: XQuAD chosen-shots gain in performance (no gain!). The gain of choosing shots over the average of no-choosing (average over 5 random sets). The actual few-shot gain (compared to zero-shot) is shown in parenthesis as follows chosen-shots-gain (few-shot-gain). We can see that there is no gain in choosing the shots. Experiments with adding language features to the model further decrease the performance.