SlowGuess/NAACL_2021_OCR huggingface dataset:
Wasserstein Distance: Wasserstein Distance (also called Earth Mover's distance) is another metric for two probability distributions. Intuitively, it measures the least amount of work done to transport probability mass from one probability distribution to another to make them equal. The work done in this case is measured as the mass transported multiplied by the distance of travel. It is known to be better than Kullback-Leibler Divergence and Jensen-Shannon Divergence when the random variables are high dimensional or otherwise. The Wasserstein metric is defined as:
$$
D _ {W a s s e r s t e i n} = \inf _ {\gamma \in \pi} \sum_ {x, y} \| x - y \| \gamma (x, y)
$$
Here  $\gamma \in \pi(P, Q)$  where  $\pi(P, Q)$  is the set of all distributions where the marginals are  $P$  and  $Q$ .
Maximum Mean Discrepancy (MMD): MMD is a non-parametric method to estimate the distance between distributions based on Reproducing Kernel Hilbert Spaces (RKHS). Given two random variables  $X = \{x_{1}, x_{2}, \dots, x_{m}\}$  and  $Y = \{y_{1}, y_{2}, \dots, y_{n}\}$  that are drawn from distributions  $P$  and  $Q$ , the empirical estimate of the distance between distribution  $P$  and  $Q$  is given by:
$$
M M D (X, Y) = \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \phi \left(x _ {i}\right) - \frac {1}{n} \sum_ {i = 1} ^ {n} \phi \left(y _ {i}\right) \right\| _ {\mathcal {H}} \tag {11}
$$
Here  $\phi : \mathcal{X} \to \mathcal{H}$  are nonlinear mappings of the samples to a feature representation in a RKHS. In this work, we map the contextual word representations of the text to RKHS. The different kinds of kernels we use in this work are given below. We use
the default values of  $\sigma = 0.05$  of the GeomLoss package (Feydy et al., 2019).
Rational Quadratic Kernel
$$
\phi (x, y) = \left(1 + \frac {1}{2 \alpha} (x - y) ^ {T} \Theta^ {- 2} (x - y)\right) ^ {- \alpha}
$$
$$
\phi (x, y) = - \left\| x - y \right\| _ {2}
$$
$$
\phi (x, y) = e x p (- \frac {\| x - y \| _ {2} ^ {2}}{2 \sigma^ {2}})
$$
$$
\phi (x, y) = e x p (- \frac {\| x - y \| _ {2}}{\sigma})
$$
Correlation Alignment (CORAL): Correlation alignment is the distance between the second-order moment of the source and target samples. If  $d$  is the representation dimension,  $\| \cdot \| _F$  represents Frobenius norm and  $Cov_{S},Cov_{T}$  is the covariance matrix of the source and target samples, then CORAL is defined as:
$$
D _ {C O R A L} = \frac {1}{4 d ^ {2}} \left\| C o v _ {S} - C o v _ {T} \right\| _ {F} ^ {2} \tag {12}
$$
Central Moment Discrepancy (CMD): Central Moment Discrepancy is another metric that measures the distance between source and target distributions. It not only considers the first moment and second moment, but also other higher-order moments. While MMD operates in a projected space, CMD operates in the representation space. If  $P$  and  $Q$  are two probability distributions and  $X = \{X_{1},X_{2},\dots,X_{N}\}$  and  $Y = \{Y_{1},Y_{2},\dots,Y_{N}\}$  are random vectors that are independent and identically distributed from  $P$  and  $Q$  and every component of the vector is bounded by  $[a,b]$ , CMD is then defined by:
$$
\begin{array}{l} C M D (P, Q) = \frac {1}{| b - a |} \| E (X) - E (Y) \| _ {2} \\ + \sum_ {k = 2} ^ {\infty} \frac {1}{| b - a | ^ {k}} \| c _ {k} (X) - c _ {k} (Y) \| _ {2} \tag {13} \\ \end{array}
$$
where  $E(X)$  is the expectation of  $X$  and  $c_{k}$  is the  $k - th$  order central moment, defined as:
$$
c _ {k} (X) = E \left(\prod_ {i = 1} ^ {N} \left(X _ {i} - E \left(X _ {i}\right)\right) ^ {r _ {i}}\right) \tag {14}
$$
and  $r_1 + r_2 + r_N = k$  and  $r_1\ldots r_N\geq 0$
A.4 Other Measures
Bhattacharya Coefficient: If  $P$  and  $Q$  are probability distributions, then the Bhattacharya coefficient and Bhattacharya distance are defined as:
$$
B h a t t a c h a r y a (P, Q) = \sum_ {x} \sqrt {P (x) Q (x)} \tag {15}
$$
$$
D _ {B h a t t a c h a r y a} = - \log (B h a t t a c h a r y a (P, Q)) \tag {16}
$$
Term Vocabulary Overlap (TVO): This measures the proportion of target vocabulary that is also present in the source vocabulary. If  $V_{S}$  is the source domain vocabulary and  $V_{T}$  is the target domain vocabulary, then the Term Vocabulary Overlap between the source domain  $(D_{S})$  and the target domain  $(D_{T})$  is given by:
$$
T V O \left(D _ {S}, D _ {T}\right) = \frac {\left| V _ {S} \cap V _ {T} \right|}{\left| V _ {T} \right|} \tag {17}
$$
Word Vector Variance: Different contexts in which a word is used in two different datasets can be used as an indication of the divergence between two datasets. Let  $\vec{w}_{src}^{i}$  denote the word embedding of word  $i$  in source domain and  $\vec{w}_{trg}^{i}$  is the word embedding of the same word in the target domain. Let  $d$  be the dimension of the word embedding. The word vector variance between the source domain  $(D_S)$  and the target domain  $(D_T)$  is given by:
$$
W V V \left(D _ {S}, D _ {T}\right) = \frac {1}{\left| V _ {S} \right| * d} \sum_ {i} ^ {\left| V _ {s} \right|} \left\| w _ {s r c} ^ {i} - w _ {t r g} ^ {i} \right\| _ {2} ^ {2} \tag {18}
$$
B Model Hyperparameters
For POS, NER and Sentiment Analysis models, we do a grid search of learning rate in  $\{1e - 01, 1e - 05, 5e - 05\}$  and dropout in  $\{0.2, 0.3, 0.4, 0.5\}$  and number of epochs in  $\{25, 50\}$ . PAD requires a domain discriminator. We sample as many samples in the target domain as the source domain (Ruder et al., 2017) and train a DistilBERT based classifier. For every domain discriminator we do a grid search of learning rate in  $\{1e - 05, 5e - 05\}$ , dropout in  $\{0.4, 0.5\}$  and number of epochs in  $\{10, 25\}$ . For POS and NER, we monitor the macro F-Score; for domain discrimination, we monitor the accuracy scores. We chose the best model after the grid
search for all subsequent calculations. For training the models we use the Adam Optimiser (Kingma and Ba, 2015) with the  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.99$  and  $\epsilon$  as 1e-8. We use HuggingFace Transformers (Wolf et al., 2019) for all our experiments.
C Cross-Domain Performances
C.1 Parts of speech tagging
Table 3 shows the hyper parameters for the best model for POS and Table 4 shows the cross domain performances.
C.2 Named Entity Recognition
Table 5 shows the hyper parameters for the best model for NER and Table 6 shows the cross domain performances.
C.3 Sentiment Analysis
Table 7 shows the hyper parameters for the best model for Sentiment analysis and Table 8 shows the cross domain performances.
D Silhouette Scores and t-SNE Plots
For calculating Silhouette scores we use a subset of domain divergence measures that are metrics (a requirement of Silhouette scores) and can be calculated between single instances of text. We sample 200 points for each dataset as the time complexity increases exponentially with number of points. We average the results over 5 runs.
We plot the t-SNE plots for POS (Figure 3a), NER (Figure 3b) and Sentiment Analysis (SA) (Figure 3c). We sample 200 points from each of the datasets for the plot. Wherever relevant, we use DistilBERT (Sanh et al., 2019) representations for calculations.
Dataset	Epochs	Learning Rate	Dropout	Fscore
EWT-answers	50	5 × 10-5	0.4	95.38
EWT-email	50	1 × 10-5	0.3	96.62
EWT-newsgroup	50	5 × 10-5	0.5	95.92
EWT-reviews	50	5 × 10-5	0.4	96.97
EWT-weblog	50	5 × 10-5	0.3	97.03
GUM	50	1 × 10-5	0.3	95.73
LINES	50	5 × 10-5	0.3	97.38
PARTUT	50	1 × 10-5	0.4	97.06
Table 3: Model performance and hyper-parameters producing the best model for Parts of Speech Tagging trained using DistilBERT as the base model. The datasets are from the Universal Dependencies Corpus (UD) (Nivre et al., 2016). 5 corpora are from the English Word Tree (EWT) portion which are EWT-answers, EWT-email, EWT-newgroup, EWT-reviews, EWT-weblog.
Source/Target	EWT- answers	EWT- email	EWT- newsgroup	EWT- reviews	EWT- blog	GUM	LINES	PARTUT
EWT-answers	95.38	93.96	94.02	95.83	95.64	93.58	93.86	92.06
EWT-email	94.11	96.62	94.40	95.42	95.37	93.08	93.98	93.47
EWT-newsgroup	94.71	95.07	95.92	95.31	96.80	93.82	93.83	92.74
EWT-reviews	94.99	94.51	94.56	96.97	95.55	93.07	94.27	92.62
EWT-weblog	95.38	93.96	94.02	95.83	95.64	93.58	93.87	92.06
GUM	91.63	92.59	91.75	93.55	93.56	95.73	93.54	93.12
LINES	89.79	89.77	88.76	92.39	90.77	91.75	97.38	92.68
PARTUT	89.27	89.54	89.56	91.28	92.27	90.65	92.97	96.65
Table 4: Cross-domain performance for POS tagging. The best model for each source domain is tested on the test dataset of the same domain and all other domains.
Dataset	Epochs	Learning Rate	Dropout	Fscore
CONLL-2003	50	5 × 10-5	0.5	0.90
WNUT	25	5 × 10-5	0.5	0.50
Onto-BC	50	5 × 10-5	0.5	0.82
Onto-BN	50	1 × 10-5	0.3	0.89
Onto-MZ	50	1 × 10-5	0.3	0.86
Onto-NW	25	5 × 10-5	0.4	0.89
Onto-TC	50	1 × 10-5	0.5	0.75
Onto-WB	50	5 × 10-5	0.4	0.63
Table 5: Model performance and hyper-parameters for Named Entity Recognition trained using DistilBERT as the base model. The datasets are CONLL-2003, Emerging and Rare Entity Recognition twitter dataset (WNUT), and six different sources of text in Ontonotes v5 (Hovy et al., 2006)
Source/Target	CONLL	WNUT	ONTO-BC	ONTO-BN	ONTO-MZ	ONTO-NW	ONTO-TC	WB
CONLL 2003	0.90	0.37	0.54	0.65	0.59	0.54	0.51	0.41
WNUT	0.66	0.50	0.40	0.44	0.49	0.42	0.49	0.33
ONTO-BC	0.48	0.31	0.82	0.81	0.77	0.74	0.72	0.45
ONTO-BN	0.53	0.37	0.77	0.89	0.76	0.79	0.76	0.47
ONTO-MZ	0.49	0.29	0.72	0.78	0.86	0.75	0.69	0.45
ONTO-NW	0.52	0.32	0.73	0.86	0.73	0.89	0.76	0.46
ONTO-TC	0.51	0.37	0.61	0.64	0.57	0.55	0.75	0.41
ONTO-WB	0.43	0.12	0.52	0.63	0.54	0.57	0.52	0.63
Table 6: Cross-domain performance for NER. The best model for each source domain is tested on the test dataset of the same domain and all other domains.
Dataset	Epochs	Learning Rate	Dropout	Fscore
Apparel	25	1 × 10-5	0.4	91.25
Baby	50	5 × 10-5	0.4	93.75
Books	50	1 × 10-5	0.4	92
Camera/Photo	25	1 × 10-5	0.4	92
MR	50	5 × 10-5	0.3	82.5
Table 7: Model performance and hyper-parameters for Sentiment Analysis with DistilBERT as the base model. We chose 5 out of 16 datasets from (Liu et al., 2017) which are Apparel, Baby, Books, Camera/Photo, and MR.
Source/Target	Apparel	Baby	Books	Camera/Photo	MR
Apparel	0.91	0.9100	0.85	0.87	0.77
Baby	0.89	0.9375	0.86	0.89	0.75
Books	0.88	0.8875	0.92	0.87	0.79
Camera/Photo	0.89	0.89	0.86	0.92	0.75
MR	0.76	0.76	0.8375	0.74	0.83
Table 8: Cross-domain performance for Sentiment Analysis. The best model for each source domain is tested on the test dataset of the same domain and all other domains.
(a) POS
(b) NER
(c) SA
Figure 3: t-SNE plots for different tasks.