SlowGuess/NAACL_2022_OCR huggingface dataset:
Alexandra Chronopoulou $^{1*}$ , Matthew E. Peters $^{2}$ , Jesse Dodge $^{2}$
<sup>1</sup>Center for Information and Language Processing, LMU Munich, Germany
$^{2}$ Allen Institute for Artificial Intelligence, Seattle, WA
achron@cis.lmu.de
{matthewp, jessed} @allenai.org
The remarkable success of large language models has been driven by dense models trained on massive unlabeled, unstructured corpora. These corpora typically contain text from diverse, heterogeneous sources, but information about the source of the text is rarely used during training. Transferring their knowledge to a target domain is typically done by continuing training in-domain. In this paper, we introduce a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. Our method is based on the observation that textual domains are partially overlapping, and we represent domains as a hierarchical tree structure where each node in the tree is associated with a set of adapter weights. When combined with a frozen pretrained language model, this approach enables parameter sharing among related domains, while avoiding negative interference between unrelated ones. Experimental results with GPT-2 and a large fraction of the 100 most represented websites in C4 show across-the-board improvements in-domain. We additionally provide an inference time algorithm for a held-out domain and show that averaging over multiple paths through the tree enables further gains in generalization, while adding only a marginal cost to inference.
Pretrained language models (PLMs) (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019), trained on massive general-domain corpora, have enabled great progress in many natural language processing (NLP) benchmarks (Wang et al., 2018). Nonetheless, continuing pretraining (as a dense model) a PLM on a narrower domain (Han and Eisenstein, 2019; Lee et al., 2019) is beneficial, although computationally expensive (Maronikolakis and Schütze, 2021), which indicates
Figure 1: We model domains as a hierarchical tree structure that associates adapters with nodes, allowing parameter sharing among related domains. Internet domains appear as leaf nodes. During training, we activate the adapters along the path to a leaf to specialize a language model to the domain corresponding to it.
that domain-relevant data is important for downstream tasks. Sparse models that use mixtures of experts (Lepikhin et al., 2021) have recently been proposed to allow efficient training.
Prior work typically assumes that individual domains are distinct, and models them accordingly. For example, Gururangan et al. (2020, 2021) train one model for each textual domain, either in a dense or sparse manner. This is related to data selection (Moore and Lewis, 2010; Axelrod et al., 2011; Plank and van Noord, 2011), which aims to select the best matching data for a new domain. This process does not scale to multiple domains efficiently, as the parameters grow linearly with the domains. It also does not allow sharing representations among related domains during training, as each domain is modeled with a separate set of parameters. At the other extreme, training one model on all domains as is common during unsupervised pretraining does not account for their similarities and differences and might hinder the model's generalization ability due to negative interference.
As an alternative, we start with the observation that the term "domain" typically denotes a distribution over language characterizing a given topic or genre, and that domains are partially overlapping.
* Work done while an intern at AllenAI.
Human Language Technologies, pages 1336 - 1351
For example, a sentiment model processing hotel reviews could be expected to benefit by also including data from restaurant reviews, which might in turn benefit from cooking recipes, but combing hotel reviews and recipes may be detrimental.
We want to model the relations between domains and selectively share information, so that we allow positive transfer and avoid negative interference. To this end, we propose a data-driven approach to modeling domains that automatically clusters them in a tree using PLM representations. We then introduce an efficient method that specializes a PLM in a number of domains leveraging their hierarchical structure. Our approach allows parameter sharing among related domains using adapters (Rebuffi et al., 2017; Houlsby et al., 2019), which are lightweight layers, added after each transformer (Vaswani et al., 2017) layer. Each node in the tree is associated with a separate set of adapters, that are only activated for a particular domain. For instance, data from BOOKING.COM activates parameters in nodes 3, 6, and 7, allowing parameter sharing with the highly related YELP.COM through nodes 6 and 7 (Figure 1).
We verify the efficacy of our approach in two settings. First, we manually define a tree structure, using websites as the leaves. In this first few-domain setting, our method outperforms prior work including single and multi-domain adapters added to GPT-2 (Radford et al., 2019) when tested in-domain. We further show that our method generalizes better to held-out websites than the baselines.
We then scale our model to a many-domain setting across almost 100 websites. We induce the hierarchical structure in an unsupervised way using representations from GPT-2 with a Gaussian Mixture Model (GMM) (Aharoni and Goldberg, 2020) and hierarchical clustering, similar to Das Gupta et al. (2015). In this way, the clusters model textual domains and the GMM provides a mechanism to automatically find the closest training websites to any held-out website. Empirical results show across-the-board improvements over strong baselines when evaluated in-domain. We also show that an efficient inference-time algorithm that averages over multiple paths through the tree improves generalization when tested on held-out websites.
2 Hierarchical Representation of Domains
In this section, we provide a formal problem definition and the intuition for a hierarchical ordering of domains. We then describe how we add a hierarchical structure to a PLM and present the training process. Additionally, we show how a path in the tree is selected to evaluate the in-domain and out-of-domain sets. We finally discuss the computational cost of our approach compared to the baselines and our experimental setup.
We formulate the task as follows: given a PLM, we aim at fine-tuning it in the task of language modeling using adapter modules, on a corpus consisting of  $k$  corpora for domain adaptation. The model is trained to minimize the cross-entropy loss on sentences from all  $k$  corpora, then is evaluated on both in-domain and out-of-domain test sets.
2.2 Textual domains and provenance
As there is no commonly-accepted definition of a "domain" in text (Plank, 2016), we take a practical approach and use the provenance of a piece of text (that is, the website from which the text was scraped) as a proxy for textual domain. To model how similar different textual domains are to each other, we fit a Gaussian Mixture Model (GMM) using PLM representations (Aharoni and Goldberg, 2020) using a small sample of text from each domain. After fitting this GMM, for some clusters there is a one-to-one correspondence between one cluster and text from one website, while for other clusters there is a one-to-many correspondence between one cluster and text from multiple, similar websites (see §4).
2.3 Hierarchical Structure
Domains generally overlap with each other and have different degrees of granularity. A model that encodes them should both capture domain-specific and general-domain information. To this end, we propose representing the data as a tree. An example of a tree structure is shown in Figure 1. Text from specific websites is encoded in the leaf nodes (such as FRONTIERSIN.ORG, JOURNALS.PLOS.ORG), while more general-domain knowledge is encoded in the upper nodes (SCIENTIFIC ARTICLES).
<sup>1</sup>Our code is publicly available at github.com/alexandrachron/hierarchical-domain-adaptation.
2.4 Model Architecture
Assuming a corpus with data from  $n$  domains, we consider the setting where we have a pretrained model  $M$ . We want to use  $M$  to adapt to  $n$  new domains. To this end, we can leverage adapters.
Adapter layer. Adapters are typically added to model  $M$  in each transformer layer and are trained to a task, while  $M$  remains unchanged. An adapter uses as input the output of the previous layer. It is formally defined as  $\mathbf{W}^U\mathrm{ReLU}(\mathbf{W}^D\mathrm{LN}(h_i)) + h_i$  ,where  $h_i$  is the output of the  $i$  -th layer, of dimension  $m$  , LN is a layer-normalization (Ba et al., 2016),  $\mathbf{W}^D$  is a down-projection in  $R^{m\times d}$  ,and  $\mathbf{W}^U$  is an up-projection in  $R^{d\times m}$  , and  $d$  is the bottleneck dimension of the adapter module.
Single Adapters. To adapt to  $n$  domains, one solution is to train  $n$  adapters (per transformer layer), one for each domain. The number of parameters added from single adapters grows linearly with the number of domains  $(\mathrm{O}(n))$ .
Multi-Domain Adapters. Another solution is to add just one set of adapters to model  $M$ . The adapter weights will be updated based on data from all  $n$  domains. This is a dense model that does not permit modular training. For  $n$  domains, the number of parameters added is constant.
Hierarchical Adapters. We propose associating each of the nodes in a tree that represents domains with a set of adapters and adding them to  $M$ . This sparse model adds parameters that scale logarithmically  $(O(log(n))$  with the number of domains because of the binary tree structure (Figure 1).
While Houlsby et al. (2019) insert adapters but re-train layer normalization parameters of  $M$  and Bapna and First (2019) introduce new layer normalization parameters for every adapter, we introduce just one set of layer normalization parameters in each transformer layer and these parameters are shared between all adapters of a transformer layer.
2.5 Training & Computational Cost
When our input consists of data from a particular domain, we only update the adapter layers of the path that leads to this domain (Figure 1).
Supposing we have a mini-batch from FRONTIERSIN.ORG, the hidden state  $h_i$  of the  $i$ -th layer is the input of  $\text{adapter}_i^1$  (the adapter of node 1 for transformer layer  $i$ ).  $h_i$  is also the input of  $\text{adapter}_i^5$  (parent) and  $\text{adapter}_i^7$  (root). Their outputs  $y_i^1, y_i^5, y_i^7$  are averaged. The final representation  $y_i$  is the input to the next transformer layer.
Using this simple training process, we allow sharing between related domains. Upper nodes in the tree are updated more often than leaves, thus they are better trained and encode more domain-general knowledge. More precisely, the root node of the hierarchical model in Figure 1 is updated for each sequence, but the leaf nodes are only updated using sequences from the associated domain.
In terms of computation, although our model adds a large number of trainable parameters (total parameters), only a small fraction of them is used for each forward pass (active parameters), as shown in Table 1. At inference time, to evaluate performance on a domain using the tree of Figure 1, our approach with a single path uses 126M parameters (GPT-2 has 112M and the adapters of each path account for 14M parameters). When we average two paths, 23M parameters are added to GPT-2.
Kaplan et al. (2020) provided a detailed breakdown of compute cost for transformer LMs. For a model with  $N$  non-embedding parameters, the approximate cost of a forward pass is  $2N$  flops per token. Extending their calculations to our setting, for a model with  $L$  layers, model dimension  $d_{\mathrm{model}}$ , adapter bottleneck size  $d$ , a single adapter adds  $4Ld_{\mathrm{model}}d$  flops per forward pass over the cost of running GPT-2. Our hierarchical method requires running  $T$  adapters per layer per forward pass, where  $T$  is the average tree depth. For the many-domain setting in  $\S 4$  with  $L = 12$ ,  $d_{\mathrm{model}} = 768$ ,  $d = 64$ ,  $N = 84\mathrm{M}$ ,  $T = 8$ , this gives an increase of  $\sim 11\%$  flops over GPT-2. At inference time, using two paths ( $\S 4.5$ ) the increase is  $22\%$  over GPT-2.
For fair comparison between our method and the baselines, we scale the adapter size so that our proposed model and the multi-domain adapters (the most related baseline) use the same number of flops. Following the previous paragraph, the adapter sizes in our hierarchical model are smaller by a factor of  $1 / T$  then those in the baseline models (Table 1).
2.6 In-domain/Out-of-domain Evaluation
At inference time, we need to define which path should be activated for each domain. When we perform in-domain evaluation, this is straightforward. We always activate the path that leads to the node that is assigned to this specific domain.
For out-of-domain evaluation, we need to find the path that better fits the held-out domain. We can also use multiple paths, as the computational cost is small. We describe in detail how we run
Few-Domain Setup	Many-Domain Setup
Hierarchical (ours)	Adapter Size	256	64
# Adapters	7	49
Average path length	3	8
Total parameters	33M	58M
Active parameters	14M	9.5M
Number of updates - root	22K	11K
Number of updates - leaf	5.5K	400
Multi-domain	Adapter Size	768	512
# Adapters	1	1
Average path length	1	1
Total parameters	14M	9.5M
Active parameters	14M	9.5M
Number of updates	22K	11K
out-of-domain evaluation in the following two sections, which present a manually defined (§3) and an automatically created hierarchical structure (§4).
2.7 Experimental Setup
We use GPT-2 (12 transformer layers; hidden size 768) as the pretrained model. GPT-2 has a vocabulary of 50,264 BPE (Sennrich et al., 2016) tokens and 112M parameters. Our code is built with PyTorch (Paszke et al., 2017), using the HuggingFace library (Wolf et al., 2020). We run all experiments on NVIDIA A100 GPUs with 40GB of RAM. We split our corpora in 800-token sequences. Models are trained with the Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of  $1e^{-3}$  and we accumulate gradients over 2 updates.
3 Hierarchical Domain Adaptation with a Manually Created Tree
In this section, we implement the model described in the prior section for a very limited number of domains (few-domain setup), to comprehensively examine design choices and verify the performance, before moving to a large-scale setting in  $\S 4$ .
We select four websites to be represented by leaf nodes in our tree: two that contain scientific articles (FRONTIERSIN.ORG, JOURNALS.PLOS.ORG) and two that contain reviews (BOOKING.COM and YELP.COM). We use text from the released version (Dodge et al., 2021) of C4 (Raffel et al., 2020), a web-scale corpus of English data; the first three internet domains are some of the largest sources of
text in C4. We also use YELP.COM, a publicly available dataset. Dataset sizes (training/evaluation) are shown in Appendix A.1.
3.2 Approach
We use the hierarchical structure shown in Figure 1, with two leaf nodes representing scientific articles sharing a parent, two leaf nodes representing reviews sharing a parent, and a single grandparent shared by the two parents. This tree structure was manually chosen using domain knowledge. We use a pretrained GPT-2 model as our base model, and add one set of adapters per node in the tree (one adapter per transformer layer for each node). We freeze the weights of GPT-2 and train the adapters on language modeling of the domains of interest. The training process is explained in detail in §2.5.
Our hierarchical model adds 7 sets of adapters to GPT-2, one for each node in the tree. Each adapter has a bottleneck dimension  $d$  of 256. For each training step, one path through the tree is active (so, 3 adapters) depending on which domain of text is represented in the current batch (see §2.5). Active nodes are used in the forward pass and updated in the backward pass (during training), while those that are not active are not used in the computation.
We evaluate two baselines: a multi-domain adapter, trained on all in-domain data, and single adapters, each trained on data from a different website. We ensure that the hierarchical model uses the same amount of compute for a forward pass as the multi-domain adapter baseline (using  $d = 768$  and 1 adapter/path). We also train each model to an equal amount of data from each domain. Results are shown after 20 epochs of training (22K steps).
Table 1: Parameters used by our approach and the multi-domain adapters. The few-domain and many-domain setup are explained in § 3 and § 4 respectively.
GPT-2	single adapters	multi adapters	hierarchical adapters
frontiersin	22.2	16.1	15.8	15.5
journals	24.5	16.6	16.3	15.8
booking	29.7	9.7	9.9	9.2
yelp	36.2	24.3	25.3	23.8
average	27.7	15.8	15.9	15.2
Table 2: In-domain evaluation perplexity for the few-domain setting (§3). Hierarchical adapters consistently provide better scores compared to the baselines.
3.4 In-Domain Results
In-domain evaluation scores are presented in Table 2. Our model clearly surpasses the multi-domain
GPT-2	single adapters	multi adapters	hierarchical adapters
ncbi	20.5	18.2	17.6	17.3
link.springer	27.7	24.5	22.7	22.6
scholars.duke	22.7	20.1	20.3	19.9
techcrunch	27.7	27.1	26.3	27.1
medium	29.1	30.0	27.9	28.5
tripadvisor	41.3	36.6	34.1	26.0
lonelyplanet	35.5	27.1	24.3	25.3
average	29.2	26.2	24.8	23.8
adapter baseline in all domains. On average, hierarchical adapters lower the perplexity by 0.7 compared to multi-domain adapters. Compared to just evaluating GPT-2, our model yields a large improvement, confirming prior work that suggests that further training a PLM in-domain is highly effective. Single adapters perform roughly equivalent to multi-domain adapters in this scenario.
3.5 Out-of-Domain Results
We perform evaluation on 7 unseen domains, some of which represent similar textual domains to our in-domain data, while others are quite different. For example, NCBI, LINK.SPRINGER, and SCHOLARS.DUKE contain text from scientific documents, similar to two of our in-domain sources of text, but TECHCRUNCH and MEDIUM are quite dissimilar to the in-domain text. All models outperform the baseline of just evaluating GPT-2, as shown in Table 3. We hypothesize that the pretraining data from GPT-2, which has not been publicly released, had a somewhat different distribution to C4, and thus further training on any data from C4 seems to improve performance. The best out-of-domain results are obtained with hierarchical adapters.
However, which set of single adapters we should use to evaluate a held-out domain is not obvious. For example, to evaluate on LONELYPLANET, it intuitively makes sense to use adapters trained on a reviews/travelling domain (BOOKING or YELP), but for LINK.SPRINGER, the model trained on a scientific articles (FRONTIERSIN or JOURNALS) might be more suitable. We have no a priori criterion to choose the most appropriate model. This is also true for our proposed model. We show the best evaluation scores using single adapters in Table 3 (full evaluation in Appendix A.2).
For the hierarchical adapter model, we show
Table 3: Out-of-domain evaluation perplexity for the small setting (§ 3). For the hierarchical model, 2 paths through the tree are used for the evaluation. The hierarchical model on average outperforms the baselines.
1 path	2 paths
journals	frontiers	booking	yelp	science	reviews
ncbi	17.6	18.7	34.8	26.0	17.3	26.3
link.springer	23.3	23.3	37.0	33.1	22.6	31.8
scholars.duke	20.7	20.7	35.5	29.4	19.9	28.8
techcrunch	27.7	27.9	34.8	32.8	27.1	29.4
medium	29.4	29.4	35.9	36.2	28.5	30.6
tripadvisor	47.9	47.9	37.0	38.1	45.6	26.0
lonelyplanet	39.6	40.0	25.5	38.9	38.5	25.3
average	29.5	29.7	34.4	33.5	28.5	28.3
Table 4: Out-of-domain evaluation of the hierarchical model using different paths. The left part of the table shows scores using a single path. The right part shows results using the average of two paths, corresponding to either the scientific articles or the reviews domain.
evaluation scores using various paths in Table 4. As expected, using a single path, the hierarchical model performs best leveraging the path of a website that is most similar to the unseen website. For example, the best evaluation score for NCBI is obtained with the path that leads to JOURNALS, while the best score for TRIPADVISOR using the path that leads to BOOKING. Using two paths (either the paths of FRONTIERS and JOURNALS, or BOOKING and YELP), results generally improve. For science or technology websites, using the paths of the science domain considerably boosts the hierarchical model's performance. For reviews/travelling websites, using both paths of the reviews domain is beneficial. This confirms our intuition that the hierarchical structure proposed adequately models domains, preventing negative transfer.
Comparing our hierarchical adapters to multi-domain adapters, using a single path, hierarchical adapters perform worse than multi-domain adapters (average scores of columns 1-4 in Table 4 are worse than the average score of column 3 in Table 3). However, with a second path active, hierarchical adapters outperform all other approaches (Table 3). This highlights an advantage: they are extensible even after training, allowing for flexible performance-efficiency trade-offs that dense approaches (like multi-domain adapters) do not.
4 Hierarchical Domain Adaptation with an Automatically Created Tree
In this section, we scale our approach to a many-domain setup, using a larger set of domains, and thus a much larger hierarchy, adding more adapters in our model. In the previous section, we manually selected a tree based on our domain knowledge, but in this section we automatically create a tree using
Figure 2: Dendrogram obtained from agglomerative clustering based on the average KL divergences of the GMMs. This diagram illustrates the hierarchical structure of 30 of the most high-resource websites on C4. The leaf nodes correspond to the cluster centers and are mapped to the websites they assign the highest probability to.
unsupervised methods. We leverage domain clusters obtained using Gaussian Mixture Models and hierarchical clustering and provide an algorithm for out-of-domain evaluation, leveraging the flexibility of hierarchical adapters, that can be combined to improve performance with a minimal cost.
As a training and evaluation corpus, we use data from C4. Specifically, we use text from 30 websites as our training corpus and we perform out-of-domain evaluation of our model and the baselines on 38 other websites. All websites used belong to the top 100 sites in C4 (details in Appendix A.1).
4.2 Approach
We want to create a hierarchical structure that represents relations between domains. To this end, we fit a Gaussian Mixture Model (GMM) and then use an agglomerative clustering algorithm on the GMM. A GMM assumes that all data points are generated from a mixture of a  $k$  Gaussian distributions and defines the probability for data points to belong to any of these distributions. We consider a GMM to be suitable choice because it accounts for the uncertainty of cluster assignment and provides soft assignments that we use at inference.
Similar to Aharoni and Goldberg (2020), we generate contextual representations of 1K sequences (uniformly sampled) from each of our 30 training websites using GPT-2. We use PCA for dimension
ality reduction. We then fit a GMM with 30 components to our data (i.e., 30 Gaussians/clusters). After that, we find the Gaussian which assigns highest probability to text from each website, and remove any Gaussian which does not assign the highest probability to any website (it can be the case that text from more than one websites could be drawn by the same Gaussian). The websites and their corresponding clusters are shown in Figure 2.