SlowGuess/NAACL_2022_OCR huggingface dataset:
Asa Cooper Stickland, Xian Li, and Marjan Ghazvininejad. 2021. Recipes for adapting pre-trained monolingual and multilingual models to machine translation. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pages 3440-3453.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302-2315, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405–1418, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. CoRR, abs/2106.10199.
A.1 Corpus description
In Table 8, we present the sizes of the training and evaluation corpora used for the many-domain setup. Only one corpus is used for the few-domain but not the many-domain experimental setting, namely YELP.COM $^2$ . This corpus has 684M training tokens and 20M evaluation tokens. We randomly sub-sample 53M training tokens of this corpus for our first, few-domain setup, as we want to train a balanced model. Extensive documentation of the corpus is available from Dodge et al. (2021). We use the C4 corpus in accordance with the terms of use $^3$ .
A.2 Few-Domain Setup
In Table 3, we present out-of-domain evaluation perplexities for the first experimental setup. For the single adapters model, we present the best-performing models in the table. In order to allow for an exhaustive comparison, we also present the evaluation results of all the trained single adapter models on all held-out websites in Table 6. We see, for example, that when evaluating on the traveling website TRIPADVISOR, the single adapter model that was trained on either BOOKING or YELP provides the lowest perplexity scores, confirming our intuition that for a held-out website, we should use a model trained on a very similar domain.
single adapters trained on	baseline GPT-2
booking	yelp	frontiers	journals
ncbi	20.1	20.1	19.7	18.2	20.5
link.springer	27.1	27.1	24.5	24.5	27.7
scholars.duke	22.2	22.2	22.2	20.1	22.7
techcrunch	27.1	27.1	27.1	27.1	27.7
medium	30.0	30.0	30.0	30.0	29.1
tripadvisor	36.6	36.6	49.4	49.4	41.3
lonelyplanet	30.0	27.1	40.4	40.4	35.5
A.3 Many-Domain Setup
Confusion Matrix. Figure 4 depicts the confusion matrix of the GMM. We can observe visually that some clusters assign a high probability to multiple internet domains, while others remain empty. This shows that the intuition have for what a domain is
Table 6: Out-of-domain evaluation of single adapters in the few-domain setup (§3). We evaluate every set of single adapters in 7 different websites. The best results for every out-of-domain website (underlined) are shown in Table 3.
In-domain scores	GPT-2	multi adapters	hierarchical adapters
ign.com	30.0	25.5	23.8
insiderpages.com	30.0	19.7	18.4
eventbrite.com	34.5	27.4	25.5
androidheadlines.com	21.8	17.1	16.0
link.springer.com	27.9	22.6	21.5
librarything.com	29.4	17.6	16.9
csmonitor.com	29.4	25.8	24.8
city-data.com	36.2	31.2	30.3
forums.macrumors.com	37.0	27.7	26.0
glassdoor.com	20.7	7.9	7.5
oreilly.com	27.4	21.5	20.5
pcworld.com	24.3	19.7	18.9
express.co.uk	22.2	15.0	14.0
answers.sap.com	60.3	34.5	30.3
prweb.com	25.8	20.1	18.9
instructables.com	32.8	28.2	26.6
deviantart.com	42.5	33.1	31.2
entrepreneur.com	26.3	22.0	20.9
si.com	22.2	17.3	16.4
gsmarena.com	56.3	34.5	31.2
wired.com	30.0	24.3	23.8
medium.com	29.1	23.1	22.6
baltimoresun.com	27.1	20.9	20.1
npr.org	22.2	18.0	17.5
frontiersin.org	22.0	18.4	17.2
chicagotribune.com	27.1	21.1	20.7
foxnews.com	22.2	15.3	14.9
aljazeera.com	22.2	17.8	17.1
dailymail.co.uk	27.1	21.1	20.7
lonelyplanet.com	35.5	19.5	17.1
average	30.0	22.3	21.0
Table 7: In-domain evaluation perplexity for the many-domain setup (we note that the hierarchical model uses a single path).
does not correspond exactly to the cluster obtained by an unsupervised, data-driven approach. Our visualization is based on publicly available code<sup>4</sup>. Out-of-domain Evaluation. As mentioned in §4.5, to run evaluation on a given out-of-domain website  $i$ , we use two paths of the trained hierarchical model. The first path leads to the leaf node that corresponds to cluster  $m$  (with  $m$  assigning the highest probability to the largest fraction of  $N$  sequences from website  $i$ ) and the second path leads to cluster  $n$ , where  $n$  assigns the highest probability to the second-most number of the  $N$  sequences. We present the clusters  $m$  and  $n$  (and the websites they were mapped to during training) in Table 9.
Because we wanted to keep a modest computational budget, we did not perform multiple training runs for the hierarchical models and the baselines. Results are reported over a single run.
$^{2}$ www.yelp.com/dataset
3commoncrawl.org/terms-of-use/
$^{4}$ github.com/roeeaharoni/unsupervised-domain-clusters
Train (Eval.) Tokens	Train (Eval.) Tokens
TRAINING CORPUS	frontiersin.org	38M (6M)	journals.plos.org	53M (6M)
chicagotribune.com	31M (4M)	fool.com	34M (4M)
link.springer.com	28M (4M)	businessinsider.com	32M (4M)
aljazeera.com	26M (3M)	theatlantic.com	30M (4M)
instructables.com	25M (3M)	booking.com	30M (4M)
npr.org	25M (3M)	kickstarter.com	26M (3M)
dailymail.co.uk	25M (3M)	telegraph.co.uk	25M (3M)
csmonitor.com	23M (3M)	cnet.com	24M (3M)
baltimoresun.com	23M (3M)	ncbi.nlm.nih.gov	23M (3M)
city-data.com	22M (3M)	foxbusiness.com	23M (3M)
forums.macrumors.com	22M (3M)	cnbc.com	20M (2M)
medium.com	22M (3M)	ibtimes.co.uk	18M (2M)
foxnews.com	22M (3M)	reuters.com	17M (2M)
si.com	18M (2M)	bbc.com	17M (2M)
wired.com	18M (2M)	nypost.com	15M (2M)
prweb.com	17M (2M)	nydailynews.com	14M (2M)
express.co.uk	16M (2M)	fastcompany.com	14M (2M)
entrepreneur.com	16M (2M)	mashable.com	14M (2M)
androidheadlines.com	14M (2M)	thesun.co.uk	13M (2M)
pcworld.com	14M (2M)	techcrunch.com	13M (2M)
gsmarena.com	12M (2M)	inquisitr.com	13M (2M)
eventbrite.com	11M (1M)	youtube.com	11M (1M)
ign.com	10M (1M)	itunes.apple.com	11M (1M)
oreilly.com	9M (1M)	breitbart.com	10M (1M)
deviantart.com	9M (1M)	etsy.com	10M (1M)
insiderpages.com	8M (1M)	github.com	10M (1M)
lonelyplanet.com	6M (1M)	agreatertown.com	9M (1M)
answers.sap.com	6M (1M)	premium.wpmudev.org	9M (1M)
glassdoor.com	4M (500K)	deadline.com	9M (1M)
librarything.com	3M (500K)	dailystar.co.uk	9M (1M)
reference.com	7M (1M)
scholars.duke.edu	7M (1M)
tripadvisor.com	7M (1M)
simple.wikipedia.org	6M (1M)
nme.com	5M (1M)
homestars.com	3M (500K)
fineartamerica.com	2M (500K)
Table 8: Domains that make up our in-domain (training) and out of-domain (evaluation) corpus for the large setup, including the size of our training and evaluation data. All data is extracted from C4 (Raffel et al., 2020).
Figure 4: Confusion Matrix. The x-axis depicts the clusters that the internet domains are assigned to. If no data samples are added to a cluster (for example, cluster 2), the corresponding Gaussian distribution is not used for the hierarchical clustering. The y-axis depicts the internet domains used for training. The cluster numbers shown here are not the exact ones shown in the final dendrogram, but one can easily observe that, for example, the same cluster (in this example, cluster 7) assigns the highest probability to CITY-DATA.COM, BALTIMORESUN.COM and CHICAGOTRIBUNE.COM. This is mirrored in Figure 2 of the main paper.
Held-out Website	Path to Cluster m	Train. Website of Cluster m	Path to Cluster n	Train. Website of Cluster n
reuters.com	14	aljazeera.com	15	npr.org
ibtimes.co.uk	9	dailymail.co.uk	14	aljazeera.com
bbc.com	9	dailymail.co.uk	3	express.co.uk
tripadvisor.com	5	insiderpages.com	22	lonelyplanet.com
cnet.com	18	wired.com	17	androidheadlines.com
telegraph.co.uk	9	dailymail.co.uk	14	aljazeera.com
theatlantic.com	0	city-data.com	14	aljazeera.com
foxbusiness.com	11	prweb.com	15	npr.org
thesun.co.uk	9	dailymail.co.uk	3	express.co.uk
nydailynews.com	9	dailymail.co.uk	6	si.com
dailystar.co.uk	3	express.co.uk	9	dailymail.co.uk
fastcompany.com	1	entrepreneur.com	18	wired.com
nypost.com	9	dailymail.co.uk	6	si.com
businessinsider.com	1	entrepreneur.com	18	wired.com
deadline.com	8	librarything.com	9	dailymail.co.uk
breitbart.com	14	aljazeera.com	0	city-data.com
techcrunch.com	18	wired.com	1	entrepreneur.com
nme.com	8	librarything.com	9	dailymail.co.uk
fool.com	1	entrepreneur.com	15	npr.org
finance.yahoo.com	1	entrepreneur.com	15	npr.org
youtube.com	11	prweb.com	15	npr.org
ncbi.nlm.nih.gov	4	link.springer.com	19	frontiersin.org
scholars.duke.edu	4	link.springer.com	19	frontiersin.org
inquisitr.com	9	dailymail.co.uk	18	wired.com
simple.wikipedia.org	10	csmonitor.com	8	librarything.com
kickstarter.com	16	deviantart.com	18	wired.com
mashable.com	18	wired.com	9	dailymail.co.uk
booking.com	5	insiderpages.com	22	lonelyplanet.com
etsy.com	13	instructables.com	5	insiderpages.com
fineartamerica.com	16	deviantart.com	13	instructables.com
github.com	7	oreilly.com	2	answers.sap.com
journals.plos.org	4	link.springer.com	19	frontiersin.org
itunes.apple.com	20	gsmarena.com	8	librarything.com
agreatertown.com	22	lonelyplanet.com	5	insiderpages.com
premium.wpmudev.org	2	answers.sap.com	7	oreilly.com
homestars.com	5	insiderpages.com	13	instructables.com
reference.com	13	instructables.com	10	csmonitor.com
cnbc.com	15	npr.org	1	entrepreneur.com
Table 9: The two paths used for evaluation of the hierarchical adapter model on each held-out website.