For the small multi-domain sentiments analysis dataset, we do a full search of the combination of learning rate, batch size and number epochs up to 5. For all other large scale datasets, we perform a greedy search. We first find the best combination of learning rate and batch size. Then we tune the number of epochs. For the extremely large XNLI, in which for any target task the multi-source training data size is  $\sim 5.5\mathrm{M}$ , we tune only when our framework selects a smaller subset of the source corpora for learning rate in  $\{3\times 10^{-5},5\times 10^{-5}\}$ , batch size  $\{32\}$  and epochs within 50k steps (i.e., no more than 3 epochs). On XNLI, when our framework selects all source corpora as potential, we do not further tune the hyper-parameters both baselines have the same training set as SEAL-Shap. Hence, we report the result using a default learning rate  $5^{-5}$ , batch size 32. All test results reported in this paper are performed on the corresponding test set using a singlegpu. All Shapley value calculations were performed on multi-gpus. After transfer source selection, all models for XLNI, and UDPOS, are trained, and tuned on multi-gpu distributed system and for SANCL 2012 POS tagging, and multistandard sentiment analysis datasets single gpu is used. As for modified GLUE NLI dataset, both singlegpu and distributed system is used. For UDPOS significance test, we use the default num_samples 10k, except for Polish we use 3k. As for Flair, the system does not support saving the prediction options, getting the model prediction even from a trained BiLSTM model is time consuming. Hence, we sample for no more than 50 times for SANCL 2012 dataset significance test. For Flair framework, after preliminary verification, we follow their configuration suggested for best performance on English Penn treebank POS tags<sup>12</sup> and tune each model up to 150 epochs with patience 4. All the approximated Shapley values (SEAL-Shap values) are computed within nepoch 30 and only for UDPOS and XNLI dataset, multiple seeds ( $< 3$ ) are used. For UDPOS nepoch within 30 or 46, for XNLI nepoch within 10 or 20. For any target task  $V_{k}$ , the corresponding threshold Shapley value  $\theta_{k}$  is chosen in  $\{1\times 10^{-2},1\times 10^{-3},5\times 10^{-3}\}$ , initial scores  $\rho$  in  $\{\mathcal{R},\mathcal{N},0.5,\text{All sources}/2,\text{All Sources},\mu\}$  where  $\mathcal{R}$  is a random baseline model performance (i.e., randomly initialized model performance); given the total number of sources  $n,\mathcal{N} = \frac{n - 1}{n}\times Score(C_{D_j},V_k)$ ;  $\mu = mean(\{\text{All Sources}\} \cup \{\text{Score}(C_{D_j},V_k)\forall D_j\in D\})$ , and  $D$  is all source tasks. This means we also tune SEAL-Shap value as the mean of a combination of SEAL-Shap values, leave one out values, and single source transfer values. For Shapley value computation, for multiple seeds run like on cross-lingual POS tagging and XNLI, seed 42, 43 is used. All the hyperparameter tuning is done with default seed in the open-sourced Transformer implementation<sup>13</sup> which is 42. All the SEAL-Shap values are calculated using single seed. Only for plotting Figure 6 in main paper, on cross-lingual POS tagging for target English two different seeds are used. The blue curve in Figure 6, and the results in Table 2 are using the same seed and all other plots uses the other seed. All the parameter configuration and the dev set performance will be reported here upon acceptance. All computations are performed on gpus; in general using (4,8,1) #gpus. Note that while tuning, if there is no  $\theta$  for which the corresponding subset of sources (i.e.,  $\subset \mathcal{D}$ ) achieves better result than using all of  $\mathcal{D}$ , then we select the set of all sources  $\mathcal{D}$  assuming each source is contributing positively.