SlowGuess/NAACL_2021_OCR huggingface dataset:
Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared A Dunnmon, James Zou, and Daniel L Rubin. 2020. Data valuation for medical imaging using Shapley value: Application on a large-scale chest x-ray dataset. arXiv preprint arXiv:2010.08006.
Tatiana Tommasi and Barbara Caputo. 2009. The more you know, the less you learn: from knowledge transfer to one-shot learning of object categories. In *British Machine Vision Conference*.
Sandhya Tripathi, N Hemachandra, and Prashant Trivedi. 2020. On feature interactions identified by Shapley values of binary classification games. arXiv preprint arXiv:2001.03956.
Guy Van den Broeck, Anton Lykov, Maximilian Schleich, and Dan Suciu. 2021. On the tractability of shap explanations. In Proceedings of the 35th Conference on Artificial Intelligence (AAAI).
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882-7926, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China. Association for Computational Linguistics.
Y. Yang, X. Li, P. Wang, Y. Xia, and Q. Ye. 2020. Multi-source transfer learning via ensemble approach for initial diagnosis of alzheimer's disease. IEEE Journal of Translational Engineering in Health and Medicine, 8:1-10.
Y. Yao and G. Doretto. 2010. Boosting for transfer learning with multiple sources. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1855-1862.
Joey Tianyi Zhou, Sinno Jialin Pan, Ivor W. Tsang, and Shen-Shyang Ho. 2016. Transfer learning for cross-language text categorization through active correspondences construction. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 2400-2406.
Supplementary Material: Appendices
A Details of UD Treebanks
We use the Flair framework provided version of UDTreebank.
The statistics of the Universal Dependency treebanks (v2.2) is summarized in Table 7. However, more accurate statistics can be found using the above link.
Table 7: Statistics of the UD Treebanks we used. For language family, "IE" stands for Indo-European and "(w/o) punct" means #tokens excluding "PUNCT" and "SYM".
Language Family	Languages
Afro-Asiatic	Arabic (ar), Hebrew (he)
Austronesian	Indonesian (id)
IE.Germanic	Norwegian (no), Danish (da), Dutch (nl), English (en), German (de), Swedish (sv)
IE.Indic	Hindi (hi)
IE.Altaic	Turkish (tr)
IE.Romance	Catalan (ca), French (fr), Portuguese (pt), Italian (it), Romanian (ro), Spanish (es)
IE.Slavic	Bulgarian (bg), Croatian (hr), Czech (cs), Polish (pl), Russian (ru), Slovak (sk), Slovenian (sl), Serbian (sr)
Japanese	Japanese (ja)
Korean	Korean (ko)
Sino-Tibetan	Chinese (zh)
Uralic	Finnish (fi)
Iranic	Persian (fa)
Isolate	Basque (eu)
Table 8: The selected languages from UDTreebank 2.2,used in our cross lingual POS tagging, grouped by language families. "IE" is the abbreviation of Indo-European.
B POS Taggong Dataset for Domain Transfer
Domain	WSJ	Emails	Newsgroups	Answers	Reviews	Weblogs
Train/Dev/Test	2976/1336/1640	4900/2450/2450	2391/1196/1195	3489/1745/1744	3813/1907/1906	2031/1016/1015
Table 9: Data Statistics of SANCL 2012 shared task dataset (Petrov and McDonald, 2012)
C Modified GLUE NLI Task for Domain Transfer
For NLI, we consider the 2-class classification (e.g., entailed or not) corpora used in Ma et al. (2019) that is made upon modification on 4 Glue benchmark (Wang et al., 2018) problems: SNLI, MNLI, QNLI, and QQP. We split the MNLI training set into a corpora of "fiction", "slate", "govt.", "travel", and "telephone" as in Williams et al. (2018) and always include them in the source corpora for all target domains. Here, As the annotations for GLUE test sets are publicly unavailable, for each target domain, we consider the original dev set as pseudo test set and randomly select 2k instances from training set for parameter tuning (i.e., pseudo dev set). For MNLI as target, we have two original dev set. Hence, we take the 2k instances from matched dev set as pseudo dev set and consider the miss-matched corpus as pseudo test set. Therefore, in zero-shot setting, the number of source corpora for target MNLI is 8, and for others it is 7.
Task Category	Dataset	Train Size	Dev Size
Natural Language Inference	SNLI	510,711	-
Multi-Genre	MNLI-Fiction	77348	-
MNLI-Travel	77350	-
MNLI-Slate	77306	-
Natural Language Inference	MNLI-Government	77350	-
MNLI-Telephone	83348	-
MNLI-Mismatched	-	9,832
Answer Sentence Selection	QNLI	108,436	5,732
Paraphrase Detection	QQP	363,847	-
Table 10: Data Statistics of Glue NLI tasks. We report the performance on the full dev set and to tune all the models, randomly select  $20\%$  examples from it.
D Sentiment Analysis Dataset for Domain Transfer
For sentiment analysis, following Cai and Wan (2019), we use the multi-domain sentiment datasets released by Liu et al. (2017) which has several additional domains than a popular sentiment analysis dataset Blitzer et al. (2007). For each domain, we use the same test set as in Liu et al. (2017). However, as train and dev data are released together, we simply consider the first section of this combined set as the train set and the last section as dev set. Statistics of the 14 domains in this dataset considered in our experiments are reported in Appendix.
Dataset	Train	Dev	Test	Avg length
Books	1400	200	400	159
Electronics	1398	200	400	101
DVD	1400	200	400	173
Kitchen	1400	200	400	89
Apparel	1400	200	400	57
Camera	1397	200	400	130
Health	1400	200	400	81
Toys	1400	200	400	90
Video	1400	200	400	156
Baby	1300	200	400	104
Magazine	1370	200	400	117
Software	1315	200	400	129
Sports	1400	200	400	94
MR	1400	200	400	21
E XNLI Results
Table 11: Data Statistics of multi-domain sentiment dataset
Model	es	de	el	bg	ru	tr	ar	vi	th	zh	hi	sw	ur	en	fr
All Sources	77.88	71.82	72.23	74.03	73.59	65.21	68.94	74.39	60.10	74.69	67.31	52.67	64.37	82.65	77.03
Baseline-r	77.88	71.82	72.23	74.69	74.53	65.85	68.68	75.03	60.10	74.69	66.69	52.97	63.69	82.65	77.03
Baseline-s	77.88	71.82	72.23	73.23	73.73	65.67	68.36	74.11	60.10	74.69	67.07	52.59	63.31	82.65	77.03
Ours	77.88	71.82	72.23	74.95	73.85	65.63	69.24	75.71	60.10	74.69	67.78	52.73	64.67	82.65	77.03
Table 12: Cross-lingual results on the XNLI test sets.
All model performances are same when selecting all source corpora as potential.
F How good is the approximation?
We consider three different datasets: XNLI (target: 'es'), Modified GLUE NLI dataset(target: 'MNLLmm'), and SANCL 2012 shared task for POS tagging (target: 'WSJ'). We use the corresponding full size source tasks except for the extremely large XNLI in which we randomly sample half of each source task (180k instances) and compute the Shapley value adopting the source code released by Ghorbani and Zou (2019). Then, on XNLI dataset we consider sample size 50k, on GLUE 20k, on SANCL 2012 sahred task 2k for each source task. Then we use Algorithm 1 (in the main paper) with tuned initial score to compute the approximate data Shapley value. Instead of full convergence, we do early stop by setting the Shapley value nepochoes to 10, 50, 30 on XNLI, GLUE, and SANCL DATsets respectively.
G Shapley Value Computation Time with/out different Factors:
We consider two example problems to transfer both language and domain: (i) UDPOS tagging for language transfer (ii) modified GLUE NLI for domain transfer. We consider the "initial score" to All Sources/2 and  $\mathcal{R}$ ; nepoch to 30, and 50 for these two respective target task, for the data Shapley computation as in Algorithm 1, we then switch different factors as in reported in Table 6 (in the main paper) as record the corresponding Shapley value computation time.
H Approximate Shapley Value with Different Seeds:
In Figure 9, we plot the SEAL-Shap value w.r.t same threshold  $(\theta_{k} = 0)$  by different seeds.
Figure 9: SEAL-Shap value with two different seeds.
I Adding sources according different source ranking/selection methods
Figure 10: Performance using top-k sources as per dfifferent source ranking/selection methods. (Task: XNLI, target: vi. Red colored line denotes Random or Baseline-S.
Figure 11: Performance using top-k sources as per different source ranking/selection methods (Task: Cross-domain Sentiment Analysis, Target: Electronics.) Red colored line denotes Random or Baseline-S.
Figure 12: Performance using top-k sources as per different source ranking/selection methods. (Task: Cross-domain NLI, target: MNLI-mm). Red colored line denotes Random or Baseline-S.
We consider a Bert model with certain model parameters. Then using the corresponding training and development dataset we compute SEAL-Shap values by adjusting the  $\rho$ , compute the ranks according to Baseline-s, Baseline-r, language distance from the target language etc., We consider the top-3 sources to compare. We also consider the top-3 sources in a greedy depth first search approach. Ours get consistent increase and best performance using top-3 sources. Here we plot figure with more top-3 sources. For large datasets, we do not plot the greedy DFS here as it takes extremely long time to compute due to the fact that the DFS search branches rarely overlap for different targets.
J Classifier and Data Preprocessing:
As for the underlying machine learning classifier, in our experiments on the domain transfer problem, we consider the BERT based cased model (Devlin et al., 2019) except for POS tagging. For POS tagging, we consider the state-of-the-art BiLSTM based Flair framework (Akbik et al., 2018). As for language transfer problem, we consider a generic state-of-the-art classifier: the multi-lingual version of BERT based cased model. For all bert models, we adopt Transformers implementation (Wolf et al., 2019). Number of model parameters BERT model 10 million parameters. For each task, no preprocessing is performed other than the tokenization of words into subwords with WordPiece except for cross-lingual POS for which we use an oppen-sourced multilingual preprocessing toolkit<sup>10</sup> to remove "strange control character" tokens. Following Wu and Dredze (2019), we also limit subwords sequence length to 128 to fit in a single GPU for all tasks. For all tasks, we use the accuracy metric.
$^{10}$ github.com/huggingface/transformers/tree/master/examples-token-classification
K Hyper-parameters Tuning:
11for GLUE NLI, pseudo test set
$^{12}$ github.com/flairNLP/flair/blob/master/resources/docs/EXPERIMENTS.md
13github.com/huggingface/transformers/
L Training a Direct Source Selection Ranker using SEAL-Shap
Figure 13: Transfer performance with the top-3 sources
In addition to the sampled single source performances Baseline-s in the main paper, here we also compare smapled SEAL-Shap with the full size (i.e., no sampling) single source performance. Lin et al. (2019) with full size Baseline-s results are found using the original ranker realised in (Lin et al., 2019). In most cases ours outperforms (Lin et al., 2019) such as 'hr', 'de', 'da', 'nl', 'en', 'fr', 'he', 'it', 'es', 'sv'. However, the margin is small and also there are multiple cases where Lin et al. (2019) outperforms ours such as 'ar', 'cs', 'zh', 'id', 'fi', 'ja', 'ko', 'sr'.
M Dev set Results
In Table 13, we report the dev set result for cross-lingual POS tagging. For French, German, Hebrew, Slovenian, we need
Lang	All Sources	Baseline-s	SEAL-Shap
en	85.21	87.39	88.50
no	90.05	90.05	90.05
sv	93.39	93.27	93.18
fr	95.52	95.68	95.71
pt	94.55	94.73	94.77
da	90.19	90.27	90.42
es	94.11	94.04	94.16
it	96.83	96.56	96.89
hr	96.36	96.36	96.36
ca	92.58	92.39	92.92
pl	91.64	91.40	91.62
sl	93.35	93.56	93.45
nl	91.47	91.55	91.55
bg	92.26	92.26	92.26
ru	92.87	92.79	92.92
de	91.42	91.65	91.42
he	77.09	76.16	77.30
cs	94.56	93.14	94.74
ro	90.41	90.41	90.41
sk	96.38	96.33	96.42
sr	97.18	97.27	97.35
id	83.98	84.63	85.58
fi	87.24	87.26	87.26
zh	71.31	71.31	71.31
ar	79.18	79.18	79.18
ko	63.58	63.76	64.31
hi	80.69	80.16	82.78
ja	69.28	69.72	69.93
tr	78.43	78.43	78.43
eu	80.90	80.90	80.90
fa	82.37	81.67	82.74
Table 13: Dev set results on cross-lingual POS tagging.
N Full Cross-lingual POS Tagging Results
Lang	en	All Source	Baseline-r	SEAL-Shap	Baseline-s
en	-	82.71	86.32	88.55*§	86.39
no	-	90.06	90.06	90.06	90.06
sv	83.6	93.26	93.26	93.26	93.26
fr	-	94.60	94.63	94.79	94.83
pt	82.1	94.33	94.33	94.33	94.33
da	88.3	88.94	89.30	89.47*	89.23
es	85.2	93.15	93.00	93.21§	93.04
it	84.7	96.58	96.43	96.67	96.71
hr	-	96.60	96.60	96.60	96.60
ca	-	91.54	91.64	92.08*§	90.78
pl	86.9	91.61	91.61	91.61	91.61
sl	84.2	93.28	93.50	93.52*	92.89
nl	75.9	90.10	90.19	90.26	90.14
bg	87.4	92.93	92.93	92.93	92.93
ru	-	92.98	92.91	93.13*§	92.71
de	89.8	90.79	91.07	91.06	91.44
he	-	76.67	75.75	76.73§	75.43
cs	-	93.89	93.04	94.81*§	93.94
ro	84.7	89.97	89.97	89.97	89.97
sk	83.6	95.68	95.62	95.81	95.53
sr	-	97.55	97.47	97.58	97.43
id	-	84.10	85.23	85.97*§	85.50
fi	-	87.13	86.89	87.05	86.86
zh	-	71.31	71.31	71.31	71.31
ar	-	80.07	80.07	80.07	80.07
ko	-	63.59	64.27	64.19	63.77
hi	-	81.49	80.27	82.41*§	79.94
ja	-	66.86	65.99	67.81*§	67.71
tr	-	78.43	78.43	78.43	78.43
eu	-	81.18	81.18	81.18	81.18
fa	72.8	81.03	80.69	81.79	82.37
Average	-	87.17	87.21	87.62
Table 14: Performance on universal POS tagging (test set) when using each of language as the target language and the rest as source languages.  $^\circ \ast$  and  $\$$ ' denote SEAL-Shap model is statistically significantly outperforms All Sources and Baseline-  $s$  respectively using paired bootstrap test with  $p \leq 0.05$ . 'en' refers to the best single source ('en') results, reported in Wu and Dredze (2019).
All model performances are same when selecting all source corpora as potential. (See line 2 in Table 13 and Table 14).
O Full Cross-domain Sentiment Analysis Results
Model	books	kitchen	dvd	electronics	apparel	camera	baby	health	magazines	MR	software	video	toys	sports	Avg
Cai and Wan (2019)	87.3	88.3	88.8	89.5	88.0	90.3	90.3	91.0	88.5	76.3	90.8	91.3	90.3	90.5	82.16
All Sources	87.3	90.3	88.3	90.8	91.0	91.5	92.3	92.0	90.5	79.3	90.3	85.3	91.3	90.5	89.33
Baseline-r	87.0	90.5	87.3	90.8	91.0	91.5	91.8	92.0	90.5	78.8	90.0	84.8	91.3	90.5	89.08
Baseline-s	86.8	89.8	87.0	90.8	91.0	91.5	92.5	92.0	90.5	77.5	90.0	84.8	91.3	90.5	-
SEAL-Shap	87.3	90.8	88.8	90.8	91.0	91.5	92.5	92.0	90.5	79.5	90.3	87.8	91.3	90.5	89.76
Table 15: Cross-domain Transfer performance on multi-domain sentiment analysis dataset (Liu et al., 2017). Cai and Wan (2019) leverages unlabelled data from the target domain.
P SEAL-Shap values for two similar targets
Figure 14: Similar SEAL-Shap value curvature of two close language English ("en") and French ("fr") on cross-lingual NLI.
Q Interpreting Source Shapley Values in Cross-domain NLI
Figure 15: SEAL-Shap value on cross-domain NLI, referring to relative contribution of source domains. For target domain MNLI-mm, source domain QQP has the lowest contribution, whereas for target domain QNLI, source domain QQP has the highest contribution.
R Data Statistics
Transfer	Task	Dataset	#target	#source
Language	POS tag	UD Treebank	31	30
NLI	XNLI	15	14
Domain	POS tag	SANCL 2012	6	5
NLI	mGLUE	4	7+
Sentiment Ana.	mlt-dom-senti	14	13
Table 16: Task statistics. #sources are for each target. In (m)odified GLUE, #sources is 8 for target MNLI, and 7 otherwise. "mlt-dom-senti" refers to Liu et al. (2017).
S Number of Sources Selected
Lang	#Sources Selected
en	9
fr	29
da	29
es	27
it	26
ca	25
sl	29
nl	28
ru	27
de	28
he	29
cs	27
sk	27
sr	27
id	26
fi	27
ar	30
ko	27
hi	27
ja	29
fa	27
Table 17: Number of sources selected from 30 different languages by SEAL-Shap for the task of cross-lingual POS tagging.