SlowGuess/NAACL_2022_OCR huggingface dataset:
For hierarchical clustering, we use the symmetrized Kullback-Leibler (KL) divergence as a distance metric. Suppose we have two multivariate normal distributions (means  $\mu_0,\mu_1$ , covariance matrices  $\Sigma_0,\Sigma_1$ ) obtained by the GMM. To measure the difference between the two distributions, if they have the same dimension  $N$ , we compute the KL divergence. Because it is asymmetric, we cannot use it to measure the distance between distributions, so we compute the symmetrized version as follows:
$$
\begin{array}{l} D _ {K L} \left(\mathcal {N} _ {0} \| \mathcal {N} _ {1}\right) = \frac {1}{2} \operatorname {t r} \left(\Sigma_ {1} ^ {- 1} \Sigma_ {0}\right) + \ln \left(\frac {\det  \Sigma_ {1}}{\det  \Sigma_ {0}}\right) \\ + \frac {1}{2} \left(\left(\mu_ {1} - \mu_ {0}\right) ^ {\mathrm {T}} \Sigma_ {1} ^ {- 1} \left(\mu_ {1} - \mu_ {0}\right) - N\right) \tag {1} \\ \end{array}
$$
$$
\begin{array}{l} D _ {K L s y m} (\mathcal {N} _ {0}, \mathcal {N} _ {1}) = \\ \frac {1}{2} \left(D _ {K L} \left(\mathcal {N} _ {0} \| \mathcal {N} _ {1}\right) + D _ {K L} \left(\mathcal {N} _ {1} \| \mathcal {N} _ {0}\right)\right) \tag {2} \\ \end{array}
$$
Using Equation 2 as a distance metric, we use agglomerative clustering to infer the structure of our data. We start from 25 clusters, computed by
Figure 3: In-domain evaluation perplexity. Hierarchical adapters consistently outperform the multi-domain adapter on all websites used during training.
the GMM (5 are ignored because do not assign a high probability to data samples from any website, see Appendix A.3 for the confusion matrix). The clustering algorithm leads to a tree (see Figure 2). Nodes 0-24 correspond to the clusters of the GMM. Each website is assigned to a specific cluster.
For PCA, we use 100 dimensions. For the hierarchical clustering, we use distances computed using the symmetrized KL divergence. We get a tree of 49 nodes, shown in Figure 2. We add 49 adapters to GPT-2, one for each node. For a single training step, just one path in the tree is active (as in §3).
In this set of experiments, we used our computational budget to compare against our strongest baseline, multi-domain adapters, as that provided the most competitive results in §3. Comparing against single adapters could be relevant but we focus on our strongest baseline, as single adapters have shown to be less able to generalize to held-out domains. We train both our hierarchical model and the multi-domain adapter baseline for 4 epochs (11K steps), using 1 GPU per model and stopping after 51 hours. We oversample the low-resource domains to avoid overfitting. We use  $d = 64$  for hierarchical adapters, as the average path length is 8 and  $d = 512$  for the multi-domain adapter, since it adds just 1 adapter/transformer layer (Table 1).
4.4 In-Domain Results
Our in-domain results are shown in Figure 3. To evaluate our model in-domain, we use the path that leads to the cluster that assigns the highest probability to the domain of interest (the same as
during training). For example, to evaluate the performance of the model on PCWORLD, we use the path that leads to cluster 4. The average path length in the tree is 8, so we "activate" 8 adapters on average at every training step and also for in-domain evaluation. Our approach consistently outperforms multi-domain adapters, yielding +1.3 on average in terms of perplexity (see Appendix A.4 for details).
4.5 Out-of-Domain Results
We perform out-of-domain evaluation on 38 held-out websites (dataset sizes in Appendix A.1). We want to automatically find the best path in the tree for a held-out website. To this end, we use the fitted GMM to assign probabilities to data from the held-out websites. We intuitively want to place a held-out website close to similar training websites, so that it can benefit from positive transfer.
To do that, for a given out-of-domain website  $i$ , we assume we have a set of  $N$  sequences (in our experiments  $N = 1,000$ ) that we can use to find the best path; this path is used to evaluate the rest of the data from this website (e.g., for computing perplexity). Following a similar procedure to our training regime, we use GPT-2 to encode  $N$  sequences, then use the fitted GMM to find the probability assigned to each of the  $N$  vectors by each cluster (i.e., each leaf node). The single best path leads to the leaf node that corresponds to cluster  $m$ , where  $m$  assigns the highest probability to the largest fraction of the  $N$  sequences from website  $i$ . The second best path through the tree leads to cluster  $n$  that assigns highest probability to the second-most number of the  $N$  sequences from website  $i$ . Thus, using the GMM clusters and the hierarchical structure,
without training more parameters, we are able to evaluate out-of-domain data using the adapters that were trained on the most related domains. This is similar to the "cached" setting in Gururangan et al. (2021), and it does require a held-out set of  $N$  sequences that are only used for finding the best path through the tree (and not for computing perplexity). This is realistic setting when one has a significant amount of data from a single source, and we leave other approaches (e.g., finding the best path for every input sequence individually) to future work.
We show in Table 5 results of the out-of-domain evaluations. Our hierarchical adapter model outperforms the baseline of just evaluating GPT-2. We notice that using a single path, our approach provides worse results compared to multi-domain adapters. In this evaluation, the multi-domain adapters and the hierarchical model have the same number of active parameters, but the adapters in the hierarchical model are trained on less data (except the adapter associated with the root, which has the same number of updates as the multi-domain adapter but is significantly smaller). However, by having two paths through the tree active, the hierarchical adapter model leverages its modularity and surpasses multi-domain adapters, obtaining an improvement of  $+0.6$  in terms of perplexity.
At inference time, our approach with a single active path uses 122M parameters (112M of GPT-2 and  $\sim 10\mathrm{M}$  parameters for a path of average length). When two paths are active, at most 132M parameters are used. The overhead is thus quite small; if the two paths have some overlap, this computation is potentially significantly less. On average, active parameters in our model are trained on less data than multi-domain adapters (e.g., leaf nodes only see on average 400 updates, as shown in Table 1).
Our approach draws on prior work in domain adaptation and efficient language model fine-tuning.
Domain Adaptation. A large research area in NLP is domain adaptation (Jiang and Zhai, 2007; Daumé III, 2007). Training a masked language model on data from a specific domain (Lee et al., 2020; Beltagy et al., 2019) or fine-tuning a PLM using data from the target task (Howard and Ruder, 2018) or the target domain (Rietzler et al., 2020; Han and Eisenstein, 2019) has shown to be helpful to mitigate the domain shift between train and test data distributions of the same task. Gururangan
Out-of-domain scores	GPT-2	multi adapters	hierarchy 1 path	hierarchy 2 paths
reuters.com	20.9	16.0	16.4	16.3
ibtimes.co.uk	24.3	19.5	19.7	19.5
bbc.com	23.6	19.1	18.9	18.7
tripadvisor.com	40.4	34.8	35.9	33.8
cnet.com	26.8	23.3	22.2	22.9
telegraph.co.uk	30.9	23.6	24.5	22.2
theatlantic.com	28.5	23.6	23.8	23.6
foxbusiness.com	22.9	17.5	19.9	18.2
thesun.co.uk	26.8	19.9	19.9	18.2
nydailynews.com	24.5	19.3	19.5	18.2
dailystar.co.uk	20.7	13.9	12.2	12.2
fastcompany.com	27.9	21.3	21.5	20.9
nypost.com	26.3	18.9	18.9	18.7
businessinsider.com	24.3	20.5	20.7	20.9
deadline.com	33.1	26.3	33.1	26.8
breitbart.com	22.9	16.9	17.8	17.1
techcrunch.com	27.7	21.5	21.8	20.1
nme.com	28.2	20.1	23.8	20.5
fool.com	23.8	22.2	22.4	22.2
finance.yahoo.com	22.6	20.1	20.3	20.1
youtube.com	15.3	14.2	14.4	13.5
ncbi.nlm.nih.gov	20.7	18.5	18.4	18.2
scholars.duke.edu	22.6	20.7	20.3	20.3
inquisitr.com	22.4	17.5	16.4	16.4
simple.wikipedia.org	22.2	19.5	20.5	19.5
kickstarter.com	26.6	24.0	24.8	22.2
mashable.com	27.1	22.0	22.0	21.8
booking.com	29.7	22.9	24.5	22.0
etsy.com	28.8	26.3	26.8	24.5
fineartamerica.com	25.5	26.6	26.6	24.5
github.com	32.8	30.3	30.6	30.6
journals.plos.org	23.3	20.1	20.1	18.2
itunes.apple.com	34.8	28.8	33.1	30.0
agreatertown.com	44.7	40.0	39.6	35.9
premium.wpmudev.org	31.5	27.7	30.0	27.7
homestars.com	34.1	29.4	28.2	28.2
reference.com	28.5	24.5	25.3	24.5
cnbc.com	21.1	17.6	18.4	17.6
average	26.8	22.3	23.0	21.7
Table 5: Out-of-domain evaluation perplexity. With 1 path, our hierarchical model performs worse than the baseline. However, using paths of the 2 closest clusters to a held-out website, our approach yields better results. We show the paths used in detail in Appendix A.3.
et al. (2020) showed that a PLM can further improve by fine-tuning on data from a domain that is related to the domain of the task (DAPT). While this work suggests fine-tuning a different model to the domain of each task, our approach trains a single model to adapt to all domains. Also, although DAPT does not permit parameter sharing between domains, our hierarchical adapter model leverages domain similarities to improve adaptation.
Domain expert mixture (DEMix) layers (Gururangan et al., 2021) that condition a LM on the domain of input text have been recently proposed. DEMix layers replace feed-forward layers in a transformer and each of them is updated only using data from a specific domain. Then, a modular LM is trained from scratch. On the contrary, we
use a PLM and only train adapter layers on the target domains. Since each feed-forward layer is replaced with a mixture of experts, the parameters added grow linearly with the domains. In our approach, however, the number of parameters grows logarithmically, due to the hierarchical structure.
Adapters. Efficient fine-tuning using adapters (Rebuffi et al., 2017; Houlsby et al., 2019) is prevalent in many NLP taks, such as machine translation (Bapna and First, 2019), cross-lingual transfer (Pfeiffer et al., 2020) and dependency parsing (Ustun et al., 2020). Adapters can be trained on a single task or language (Pfeiffer et al., 2020), but also multilingually (Stickland et al., 2021). They have also been used to infuse factual and linguistic (Wang et al., 2021) as well as general-purpose and commonsense knowledge (Lauscher et al., 2020) into a PLM. To the best of our knowledge, we are the first to use them in a hierarchical structure for domain adaptation.
Efficient fine-tuning methods for PLMs. Besides adapters, multiple other parameter-efficient methods to adapt general-purpose PLMs to specific tasks have been recently proposed. Prefix tuning (Li and Liang, 2021), low-rank matrix approximation (Hu et al., 2022), as well as fine-tuning only the bias terms of a PLM (Zaken et al., 2021) are some of the lightweight alternatives to fine-tuning the entire PLM. He et al. (2022) show that these fine-tuning methods can be seen as modifications to some specific hidden states of PLMs and can thus be recast. We use adapters in this work, but our method could possibly also benefit from other parameter-efficient approaches.
In this paper, we present a novel approach for efficient domain adaptation on multiple domains using hierarchical adapters that encode the similarities and differences of domains, allowing parameter sharing but avoiding negative transfer. We start with a manually defined tree and then scale to a large tree, created in an unsupervised way. We also provide an evaluation-time algorithm that can combine paths to best adapt to an unseen domain.
In the future, we would like to investigate a more efficient evaluation-time approach, using only a few tokens of an unseen domain. It would also be interesting to extend our model to a multi-lingual setup. Finally, we would like to use our method to control language generation of PLMs, in order to
avoid generating hate speech or toxic text.
7 Limitations and Risks
Our work uses generative pretrained language models. As such models are trained on large datasets from text in the Internet, they encode biases that could harm marginalized populations (Bender et al., 2021). The specialized language model we propose could be used for propaganda or hate speech generation, same as any other language model. However, our hierarchical adapter model permits adding modular components and we believe that it could potentially be used to detoxify language generation, following Liu et al. (2021). This is in line with recent work on sparse models (Gururangan et al., 2021; Artetxe et al., 2021).
We thank the AllenNLP team and other researchers at the Allen Institute for AI for their thoughtful comments. We also thank Alex Fraser, Dario Stojanovski, and Suchin Gururangan for helpful discussions about this work.
Roeaharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online. Association for Computational Linguistics.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuhui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Ves Stoyanov. 2021. Efficient large scale language modeling with mixtures of experts.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355-362, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization.
Ankur Bapna and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538-1548.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Mithun Das Gupta, Srinidhi Srinivasa, J. Madhukara, and Meryl Antony. 2015. Kl divergence based agglomerative clustering for automated vitiligo grading. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2700-2709.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238-4248, Hong Kong, China. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 2790-2799.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264-271, Prague, Czech Republic. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. *ArXiv*, abs/2001.08361.
Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavaš. 2020. Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 43-49, Online. Association for Computational Linguistics.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pretrained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan First, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691-6706, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Antonis Maronikolakis and Hinrich Schütze. 2021. Multidomain pretrained language models for green NLP. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 1-8, Kyiv, Ukraine. Association for Computational Linguistics.
Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220-224, Uppsala, Sweden. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS 2017 Workshop on Autodiff.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Barbara Plank. 2016. What to do about non-standard (or non-canonical) language in nlp. In Proceedings of the 13th Conference on Natural Language Processing, KONVENS 2016, Bochum, Germany, September 19-21, 2016, volume 16 of Bochumer Linguistische Arbeitsberichte.
Barbara Plank and Gertjan van Noord. 2011. *Effective measures of domain similarity for parsing*. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1566–1576, Portland, Oregon, USA. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems.
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020. Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4933-4941, Marseille, France. European Language Resources Association.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.