How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation? (2024)

Abstract: Recent breakthroughs in scale have enabled the emergence of powerful generative language models, and the ability to fine-tune these models on various tasks by casting them into prompts or instructions. In this landscape, the problem of Unsupervised Domain Adaptation (UDA), or the problem of leveraging knowledge from a labeled source domain to an unlabeled target domain, has been left behind, with recent UDA methods still addressing discriminative classification. In particular, two popular UDA approaches, involving Continued Pre-Training (CPT) and learning domain invariant representations, have been under-explored in the generative setting, signaling a gap. In this work, we evaluate the utility of CPT for generative UDA. We first perform an empirical evaluation to measure the trade-offs between CPT and strong methods promoting domain invariance. We further evaluate how well the benefits of CPT extend to different architectures, tuning methods and data regimes. We then motivate the use of CPT by studying to what degree it benefits classification performance on the target domain. Finally, we attempt to understand the mechanism behind which CPT improves classification performance on the unlabeled target domain. Our findings suggest that the model implicitly learns the downstream task while predicting masked words informative to that task. Our work connects the body of UDA research with that of instruction tuning, enabling an initial step towards a wider applicability of modern language models. Our code is available at https://github.com/Uppaal/ cpt-generative-uda.

heading: 1 Introduction
text: Recent advancements in the pre-training of language models have enabled the widespread use of powerful generative models, which can be leveraged across multiple domains with no training (Brown et al., 2020; Scao et al., 2022; Touvron
et al., 2023, inter alia). Despite these advancements, these autoregressive models are still fragile under certain kinds of data distribution shifts, making their applications across these domains challenging (Ribeiro et al., 2020; Bajaj et al., 2021; Chuang et al., 2023; Uppaal et al., 2024, inter alia). This is addressed, in part, by the concept of instruction tuning with templates (Zhang et al., 2023; Sanh et al., 2022; Ouyang et al., 2022; Wang et al., 2022; Wei et al., 2022), enabling the learning of new tasks without any randomly initialized parameters.
The problem of unsupervised domain adaptation (UDA) leverages learned knowledge from a labeled source domain to an unlabeled target domain (Pan and Yang, 2010; Ganin and Lempitsky, 2015; Long et al., 2015, inter alia). It is useful for adaptation to unlabeled domains with high labeling costs, where supervised instruction tuning does not suffice. Despite the pervasive need for models to generalize to such domains, recent UDA methods still address discriminative classification, barring the application of these approaches to recent generative models. In particular, Continued Pre-Training and Domain Invariance-based methods, two widely popular classes of UDA approaches (Ramponi and Plank, 2020), are completely unexplored for UDA in the generative setting.
The Continued Pre-Training (CPT) approach involves extended pre-training on a domain or task, followed by supervised training on the downstream task (Gururangan et al., 2020). This approach has been widely used for adaptation to labeled domains (Gao et al., 2021; Kim et al., 2021; Hung et al., 2023, inter alia), and in the UDA setup for unlabeled domains (Han and Eisenstein, 2019; Zhang et al., 2021b; Karouzos et al., 2021; Pfeiffer et al., 2020; Parović et al., 2023). Invariance-based approaches attempt to learn representations that are invariant across domains (Tzeng et al., 2014; Ganin et al., 2016; Wu and Shi, 2022; Guo et al., 2022), with the notion that when the learned representa-
99
tions from both domains cannot be distinguished by a classifier and the classifier performs well on the source domain, it will also exhibit strong performance on the target domain. These two classes of methods introduce a trade-off: invariance-based methods suffer from instability issues (Han and Eisenstein, 2019; Kashyap et al., 2021), while continued pre-training requires a larger computational budget. But how would this trade-off extrapolate to the generative setting? For example, invariancebased methods are well motivated in discriminative tasks, where there is a clear decision boundary; however, the same does not hold for generative tasks.
To address these gaps, we introduce the setting of Generative UDA, where an autoregressive model is trained to leverage knowledge from a labeled source domain to an unlabeled target domain, using only next word prediction as its objective. We formalize the use of CPT for this setting in Section 2, and then attempt to explore and understand the behaviour of CPT for Generative UDA. We begin by performing an empirical analysis on 40 real-world domain pairs to explore the tradeoff between continued pre-training and invariance-based approaches, and find vanilla CPT to be competitive with and significantly more stable than a state of the art invariance-based approach (Section 3). We then stress test CPT, by applying it across varying model architectures and scales, tuning approaches and data regimes. We find that CPT is robust across these settings, unlike our invariance-based approach (Section 4).
With recent language models being trained across vast corpora which may include domains similar to the target domain, the requirement for continued pre-training may be raised to question. In Section 5, we show that continued pre-training is indeed essential for strong downstream performance on the target domain, and this performance rapidly degrades with limited target domain exposure. Finally, we attempt to shed light on how masking plays a role in improving classification accuracy on the unlabeled target domain in Section 6. We find that the model may implicitly learn the downstream task as it predicts masked words that are informative to the downstream task.
Our work attempts to connect the body of UDA research with recent trends in language modeling, by providing a set of insights into the behaviour of the popular class of continued pre-training ap-
proaches, in the Generative UDA setting. We hope this enables an initial step towards a wider applicability of modern language models.

heading: 2 Continued Pre-Training for Generative UDA

heading: 2.1 Preliminaries: The UDA Problem
text: We consider a text classification task, where X is the input space of all text sentences and Y = {1, ...K} is the label space. In the UDA problem, we have access to a source labeled dataset Dsrc = {(xi, yi)}Ni=1 consisting of samples from a joint distribution Psrc, and a target unlabeled dataset Dtgt = {xj}Mj=1 sampling from a target input distribution PXtgt. We further denote P X src as the marginal distribution of Psrc on X , where PXsrc ̸= PXtgt. The goal of UDA is to learn a function f : X → Y that minimizes the error rate Ex∼PXtgt 1[f(x) ̸= y].

heading: 2.2 CPT for UDA as a Sequence of Prompt Based Tasks
text: We now formalize the extension of continued pretraining to the setting of generative UDA. We use the traditional two-phase training pipeline from Gururangan et al. (2020)1. The first phase uses templates to cast the source and target domain sequences into an autoregressive pre-training task.2 The second phase applies supervised instructiontuning of the model on source-labeled data.
Task 1: Autoregressive Continued Pre-training We reuse the input sequences from the sourcelabeled dataset Dsrc as the source-unlabeled dataset, denoted as DXsrc. Next, similar to Raffel et al. (2020); Song et al. (2019), for an unlabeled sequence x ∈ DXsrc and Dsrc, we use a prompt template to convert the sequence x to an input-output sequence pair, i.e., M(x) = (x̃, ỹ). For masked language modeling (MLM), an instruction is prepended to a randomly masked sequence x to create x̃. The output sequence ỹ is a concatenation of masked words from x. For example, given x =“The movie was so cool! Two hours of fun.”, we construct x̃ =“Fill in the blanks: "The _ cool!
1While we use the two-phase multi-task training pipeline (sequential) in our main experiments, in Appendix I, we show that an equivalent single-phase multi-task training pipeline (joint) results in similar performance.
2We investigate mask language modeling for T5 models and switch to causal language modeling for decoder-only models with a few simple template changes. We compare both in Section 4.
Two hours _”, and ỹ = “<sep> movie was so <sep> of fun. <sep>”. For causal language modeling (CLM), x̃ = x.
Given (x̃, ỹ), we train an autoregressive LM parameterized by θ to minimize the negative loglikelihood loss averaged over output words and the total loss over a corpus D = DXsrc ∪ Dtgt.
ℓ(x̃, ỹ; θ) = − 1|ỹ| ∑
t
logPθ(ỹt|x̃, ỹ1:t−1) (1)
LCPT(D; θ) = 1 |D| ∑
x∈D ℓ(M(x); θ)
Task 2: Source Supervised Instruction-tuning In the second phase, we use labeled data from the source domain to train the model on the downstream classification task. Similar to the first phase, we use prompts3 to generate input-output sequence pairs: C(x, y) = (x̃, ỹ) ∀ (x, y) ∈ Dsrc. For example, for sentiment classification, if x =“I like this movie.”, y = 1 ⇒ x̃ = “[x] Is this sentence positive or negative?”, ỹ = “Positive”.
Given the augmented sequence pair (x̃, ỹ) and the model trained after the first phase, we compute the same negative log-likelihood loss ℓ(x̃, ỹ; θ) in Eq. (1). Finally, we define the total loss on the source-labeled dataset in the second phase as:
LCLS(Dsrc; θ) = 1 |Dsrc| ∑
(x,y)∈Dsrc l(C(x, y); θ)
(2) After training, we follow the practice of Liu et al. (2022) to convert a label string ỹ to its corresponding label y at test time for evaluation.

heading: 3 Evaluating the Efficacy of Continued Pre-training for Generative UDA

heading: 3.1 Experimental Setup
text: Datasets We use the MNLI and Amazon Review classification datasets, which are widely used UDA benchmarks (Malik et al., 2023; Karouzos et al., 2021; Guo et al., 2020). The MNLI corpus (Williams et al., 2018) contains sentence pairs across five genres: Travel (T), Fiction (F), Government (G), Slate (S), and Telephone (Te). The task classifies every sentence pair as entailment, neutral, or contradiction. The Multi-Domain Sentiment Analysis Dataset (Blitzer et al., 2007) contains binary sentiment reviews for different types
3Prompt templates were selected from the Public Pool of Prompts (Bach et al., 2022).
of Amazon products. We use reviews from the Apparel (A), Baby (B), Books (Bo), Cameras (C), and Movies (M) domains. We evaluate a total of 40 pairs of source and target domains, across the two datasets. Appendix A contains more details about the datasets.
Models and Tuning Methods Our main experiments use the T5v1.1 base model and (IA)3 (Liu et al., 2022) PEFT method. T5v1.1 is an improved version of the original T5 model (Raffel et al., 2020), and unlike the original T5 model, it is not trained on any supervised datasets. We then extend our evaluation to different model architectures (T0, GPT-2), tuning methods (full fine-tuning, adapters) and data regimes (Section 4).
Training Each training phase is 30,000 steps long for MNLI and 15,000 steps for the Amazon dataset. We use Adam, a batch size of 8, and a learning rate of 0.003. We set the maximum sequence length to 256 tokens. We use length normalization during evaluation, as proposed by Liu et al. (2022). For each experiment, we report the mean and standard deviation across 3 runs. More details can be found in Appendix B.
Baselines Since our goal is to study the behaviour of CPT for generative UDA, we compare it with a simple supervised baseline, and a state-of-the-art invariance-based approach. • Src+Tgt (All labeled): We fine-tune the model
using labeled data from both the source and target domains. This serves as an upper bound on target domain performance.
• UDAPTER: Malik et al. (2023) propose an invariance-based method that measures the multi-kernel maximum mean discrepancy (MKMMD) (Gretton et al., 2012; Bousmalis et al., 2016) between source and target embeddings from each transformer layer and sums them to obtain an aggregate loss Ldiv. The final loss is the weighted sum of Ldiv and the classification loss, i.e., L = λ Lcls + (1 − λ) Ldiv, where λ gradually changes from 0 to 1 during training. Here, we use the embeddings from a model as it is being instruction tuned on the downstream classification task. Their method achieves stateof-the-art performance, and outperforms popular UDA approaches like DANN (Ganin et al., 2016) and DSN (Bousmalis et al., 2016); thus we only compare CPT with this approach.

heading: 3.2 Continued Pre-training is Competitive with Domain-Invariance Methods
text: Performance We compare CPT with other baselines over 40 domain pairs of the MNLI and Amazon Review datasets, and report the average accuracies over all pairs in Table 1. We see that CPT is competitive to UDAPTER. (Appendix C contains detailed results over 40 pairs and significance tests to check for competitiveness.) Interestingly, a visualization of sentence embeddings in Figure 8 (Appendix C) suggests that representations learned through CPT are not domain invariant. In addition to the MMD based method of Malik et al. (2023), we also compare CPT on one domain pair with other methods that promote domain invariance (DANN (Ganin et al., 2016), CORAL (Sun et al., 2017)) and weight interpolation (Ilharco et al., 2022) in Appendix C, further confirming the competitiveness of CPT.
Stability CPT performs more stably than UDAPTER, with the invariance-based method often reporting a variance of over 20% across runs (Table 6 in Appendix C). For example, for the MNLI pair Fiction (F) → Government (G), minimizing UDAPTER yields a variance of 23.4% across runs. This observation is consistent with existing findings (Kashyap et al., 2021; Han and Eisenstein, 2019) that minimizing divergence measures like MMD, when combined with auxiliary task-specific loss functions, result in training instabilities and vanishing gradients. We discuss this in more detail in Appendix J.

heading: 4 How General are the Benefits of CPT?
text: Instruction tuning for large models is often performed through parameter-efficient fine-tuning (PEFT) on limited data. This tuning also applies to models of different scales and architectures (decoder-only and encoder-decoder). In this section, we evaluate the utility of CPT across these factors, using the A→M domain pair from the Amazon Reviews dataset.
CPT Helps Decoder-only Models. We extend our analysis from MLM with encoder-decoder lan-
guage models, to causal language modeling (CLM) with decoder-only language models, using GPT-2 (medium) (Radford et al., 2019). We perform CLM in the first training phase by simply training the model for next-word prediction given the original sequence. Table 2 shows that CPT provides strong improvements on the target domain in comparison to the invariance-based baseline.
CPT Outweighs Invariance-based Methods for Instruction-tuned Models. We evaluate the performance of CPT over T5v1.1 XL (3B parameters) and the instruction-tuned T0 (3B parameters) (Sanh et al., 2022). Figure 1 (Table 8 of Appendix D) shows a wider gap between UDAPTER and CPT with higher model capacity, and this gap is further increased with instruction tuning. We hypothesize that this gap is due to the vast difference between the objectives of domain invariance and instruction tuning.
CPT Benefits are Consistent across Tuning Approaches. PEFT approaches have been shown to introduce resilience to domain shift (Fu et al., 2023). To isolate this effect from the CPT framework, we use T5v1.1 to evaluate CPT in a full finetuning setup. Additionally, we compare CPT with
two PEFT approaches4 : Adapters (Houlsby et al., 2019) and (IA)3 (Liu et al., 2022). We see in Figure 2 (Table 9 in Appendix E) that CPT continues to perform stronger than the domain invariancebased UDAPTER method.
CPT Outperforms Invariance-based Methods in Low-data Regimes. In this low-data experiment, we assume access to k labeled source-domain examples. For CPT, we assume access to the full unlabeled dataset in both domains for the first training phase, and k-shot access to labeled source-domain examples for the second phase of supervised training. For a fair comparison, we also introduce a two-phase version of the UDAPTER pipeline—the first phase minimizes MMD between unlabeled source and target domain embeddings (full data access), while the second phase optimizes supervised training on the source domain (k-shot). Figure 3 (Table 10 in Appendix F) showcases CPT clearly outperforming both variants of UDAPTER, across three different models for k = 256. Furthermore, Figure 4 (Table 11 in Appendix F) shows CPT providing consistent improvements in as low as 32-shots, unlike the unstable invariance-based approach.

heading: 5 To What Extent is Target-Domain Exposure Beneficial?
text: Given the vast distributions language models are pre-trained on, a natural assumption might be that
4We choose Adapters because He et al. (2022) present a unified view of PEFT approaches which shows that the operations applied by Adapters are very similar to those of Prefix Tuning (Li and Liang, 2021) and LoRA (Hu et al., 2022). We choose (IA)3 since it is a state-of-the-art PEFT approach that uses a fraction of the learnable parameters of Adapters (More in Appendix E).
the model has already been exposed to a domain similar to the target domain during pre-training. This would mean that the model could simply extrapolate the learned downstream task from the source to the target domain, questioning the need for CPT.
In this section, we establish that exposure to the target domain is helpful, even when similar domains may have been encountered during pretraining. Table 3 evaluates CPT on the A→M domain pair with the T5v1.1 model. We note that the performance on the target domain is strongly impacted by the presence of target-domain data during the first phase of training.
For a more fine-grained analysis, we investigate the impact of degree of exposure to the target domain, by varying the masking rates during the first phase of training. While masking 15% of a sequence is considered standard for random masking, previous work has shown that BERT-sized models (Devlin et al., 2019) can learn from as high as 80% masking rates during pre-training followed by adaptation to a labeled task (Wettig et al., 2023). The source-domain performance shown in Figure 5 (Table 12 in Appendix) matches this trend. However,
high masking rates effectively reduce the exposure of the model to target data, strongly deteriorating the performance on the target domain5. We hypothesize that since the model never sees any labeled data of the target domain, it heavily depends on the signal it gets from the unlabeled data through masking.

heading: 6 Why does Word Prediction Aid Classification for Generative UDA?
text: In this section, we examine why predicting masked words of the source and target domains through CPT boosts sentence classification on the unlabeled target domain for generative UDA. We hypothesize that by having to predict masked words that are informative to the downstream task during
5With masking rates under the optimal value of 15%, the semantic and background features learned through model prediction of masked words is limited, hurting performance on the target domain.
pre-training, the model implicitly learns information about the downstream task. For example, given the masked sentence, “I really _ the movie, it was a fascinating watch.”, the masked word is indicative of the downstream task, in this case sentiment analysis. The model can only predict this masked word (which would be a positive word like “loved” or “enjoyed”) by using other words informative to the task (“fascinating”). Through this process, the model is essentially learning features which are useful to the downstream task, despite having no direct supervision.
To test this hypothesis, we quantize the “informativeness” of each word to a classification task: an informative word is highly correlated with any of the labels in the downstream task.6 Specifically, we follow Gururangan et al. (2018) and use pointwise mutual information (PMI) (Fano, 1961) of the word with respect to the class label:
PMI(word, class) = log p(word, class) p(word)p(class) ,
where we count the frequency of a word-class pair on Dsrc to estimate p(word, class), and similarly count a word and a class individually on Dsrc to estimate p(word) and p(class).
We use two sets of words from a dataset: those with the top k% (informative) and bottom k% (uninformative) PMI with any inference label (k = 15). We also filter out low-frequency words from the selection.7 We compute these sets for the source and target domains individually, assuming access to target labels. We use the T5v1.1 model on the A→M pair for our analysis.
6These informative words are similar to pivot features (Blitzer et al., 2006; Ziser and Reichart, 2018; Ben-David et al., 2020, inter alia), with the exception that they are chosen based on information from the source domain only.
7Any word that occurs less than 10 times in the entire training corpus is considered to be low frequency.
How Masking helps Learn Classification We first confirm that label-informative words indeed impact the classification performance of the CPT model. We do this by masking informative words from a sentence at inference. Figure 7 (a) shows us that the performance of the model on the source domain is not impacted by masking uninformative words, but drops on masking informative words. However, how do we know how much of this bias towards label informative words was learned during the continued pre-training phase, rather than during supervised fine-tuning? To attempt to disentangle the impact of the training phases, we train the model through selective masking (informative or uninformative) in the first phase of training, and minimize the impact of the second phase by making it a few-shot task. Figure 7 (c) shows us that the model performs best on classification when trained to predict label informative words during masking, indicating that the model does indeed learn features relevant to the downstream task during the first phase of training.
The Interplay between CPT and Classification for Generative UDA We now extend this analysis to the target domain to understand how CPT plays a role in learning features from the unlabeled domain. Figure 7 (d) shows us that informative masking outperforms uninformative masking by a significant gap, once again signaling that the masking process helps the model implicitly learn the downstream task. However, unlike with the source domain, random masking results in the strongest performance. This is due to the domain mismatch: the informative words for the source and target domains are not identical (Figure 6), and the supervised training on the source domain adds a bias towards source-informative words. The mixture of these two sets of words are best predicted through random masking, explaining its strong performance.
This phenomenon also draws the observation that random masking is preferred to selective masking for generative UDA, contrary to single domain settings where informative masking is more useful (Levine et al., 2021; Gu et al., 2020).

heading: 7 Discussion
text: The Computational Trade-off of CPT Our results in Section 3.2 show that continued pretraining and methods promoting domain invariance are competitive with each other. Continued pre-
training suffers from the computational drawback of requiring an additional phase of training. Conversely, invariance-based methods are difficult to optimize, possibly requiring more runs to achieve a stable optimum, and having a higher amortized computational cost. Inspired by Karouzos et al. (2021), we introduce a simple single phase variant of continued pre-training which is equivalent in performance to its two phase variant, nullifying the additional computational overhead of the approach (more details in Appendix I).
UDA in the Age of LLMs Recent breakthroughs in scale have showcased that large language models (LLMs) are highly powerful, and can perform various downstream tasks with limited or no training. This may raise question on the relevance of the UDA problem as a whole — does a model even require expensive adaptation to a domain it may have already been exposed to during pretraining? In addition to our analysis in Section 5, we argue that the requirement to adapt small models to unseen domains still holds in specific cases. Small supervised models have been shown to be comparable with, or even outperform, zero-shot general-purpose LLMs on various downstream tasks (Huang et al., 2023; Zhu et al., 2023; Tang et al., 2024), serving as lightweight and customizeable (through fine-tuning) alternatives. Safety critical domains like healthcare and finance would benefit more from these models than a generalist LLM. Our study does not address how to better adapt to domains, rather we investigate ways a model may adapt to data unseen during pre-training. This is a question that holds for current LLMs, and will continue to hold as long as models are unable to access infinite data during pre-training.

heading: 8 Related Work
text: UDA through Promoting Domain Invariance A major class of approaches in Model-centric UDA methods (Ramponi and Plank, 2020) aims to minimize H∆H divergence (Ben-David et al., 2010) between the source and target domain features, through adversarial training (Tzeng et al., 2014; Ganin et al., 2016; Tzeng et al., 2017; Guo et al., 2022, inter alia) or through minimizing measures of domain similarity (Bousmalis et al., 2016; Ge et al., 2023). Malik et al. (2023) have shown the minimization of MMD to outperform other invariance-based methods. However, past work has shown that domain-invariance is a weak constraint
for adaptation (Zhao et al., 2019; Karouzos et al., 2021), could introduce domain-specific hyperparameters (Trung et al., 2022), and is also prone to instability issues (Han and Eisenstein, 2019; Sun et al., 2019; Wilson and Cook, 2020; Kashyap et al., 2021).
UDA through Continued Pre-Training The limitations of invariance-based model-centric methods have encouraged the emergence of alternate approaches, based on self-supervised learning through contrastive learning (Kumar et al., 2022; Shen et al., 2022; Long et al., 2022), pseudolabeling (Zhou and Li, 2005; Ruder and Plank, 2017, inter alia) or language model pre-training. Despite not being directly useful to certain downstream tasks (Uppaal et al., 2023), CPT has been used for adaptation to labeled tasks, in both full fine-tuning (Gururangan et al., 2020; Lee et al., 2020; Gao et al., 2021, inter alia) and PEFT setups (Kim et al., 2021; Hung et al., 2023). A smaller body of work has explored the utility of CPT in a UDA setup (Han and Eisenstein, 2019; Zhang et al., 2021b; Karouzos et al., 2021; Parović et al., 2023), identifying the class of methods to be more stable than invariance-based methods.
Generative UDA The emergence of large language models (Brown et al., 2020; Scao et al., 2022; Touvron et al., 2023, inter alia) introduced the concept of instruction tuning with templates (Zhang
et al., 2023; Sanh et al., 2022; Ouyang et al., 2022; Wang et al., 2022; Wei et al., 2022; Gao et al., 2021; Liu et al., 2023), enabling multi-task training without any task specific architectural changes. However, the framework of casting discriminative classification tasks into generative next word prediction tasks has not yet been extended to UDA. The closest work to this setting (Ben-David et al., 2021) uses a generative model to create domain identifier prompts and feed them back into the model, however the final task label prediction is still discriminative. In our work, we focus on gaining insights to extend the powerful class of CPT methods to purely generative UDA, where prediction on the downstream task is treated as a next word prediction task. Through this, we also present novel findings on the impact of CPT to prompt-based classifiers in the UDA framework, countering previous findings from other studies in a single-domain setting (Gu et al., 2020; Levine et al., 2021; Wettig et al., 2023).

heading: 9 Conclusion
text: We introduce the setting of Generative UDA, and perform an investigation on the utility of continued pre-training in this setting. We compare the approach with the popular class of domain-invariance based methods for UDA, showing that CPT is both competitive with, and more stable than invariancebased approaches. Our experiments show that the
benefits of CPT extend to different architectures, tuning methods and data regimes. We motivate the need for target domain exposure through CPT by showing that performance on the target domain gradually degrades with increasing masking rate. Finally, we shed light on the interplay between masking and classification performance, and how this aids UDA. Our analysis shows that in predicting masked words that are informative to the downstream task, the model implicitly learns about the downstream task, furthering the benefits of directly learning the task. Our work connects the body of UDA research with that of instruction tuning, enabling an initial step towards a wider applicability of modern language models.
Limitations
Our work presents an investigation into continued pre-training for UDA in a generative setting. Since generative UDA is an almost completely unexplored area, we establish a proof of concept by using sentence classification tasks for our analysis. We leave the extending our analysis to more complex tasks to future work.
In our study, we consider a class of PEFT methods that involve inserting learnable parameters between the layers of the model. Other classes of PEFT methods were not considered. However, we use Adapters and He et al. (2022) have shown connections between the method with Prefix Tuning (Li and Liang, 2021) and LoRA (Hu et al., 2022).
Due to the high variance across runs in PEFTbased learning, we note that the performance can vary significantly across random seeds. We attempt to make our findings reproducible by averaging every experiment over 3 seeds. Taking environmental costs into consideration, we reduce our computational budget by running a majority of our experiments with a smaller-sized model. Learning with larger models is discussed in Section 4.
Ethics Statement
Our project aims to extend the problem of unsupervised domain adaptation to the generative setting, matching current needs with large language models. This is an effort towards improving the reliability and safety of language models, which can be fragile under distribution shift (Ribeiro et al., 2020) and incur great costs over incorrect predictions (Ulmer et al., 2020; Zhang et al., 2021a).
Our study does not involve any human subjects or violation of legal compliance. We do not anticipate any potentially harmful consequences to our work. As detailed in Appendix A, all of our experiments are conducted using publicly available datasets. Our code shall be released for reproducibility. Through our study and releasing our code, we hope to raise stronger research and societal awareness toward the problem of unsupervised domain adaptation in natural language processing.

heading: A Preparation of Evaluation Benchmarks
text: We use two classification datasets, with 5 domains each. This results in a total of 40 pairs of source and target domains. For brevity, we include results of 24 domain pairs in the main paper, and the remaining 16 in Appendix C. For both datasets, we use the train, validation and test splits from (Malik et al., 2023). More statistics about each dataset is available in Table 4. The listed datasets are intended for research purposes only. We do not make any commercial use of them.
MNLI The Multigenre Natural Language Inference (MNLI) corpus (Williams et al., 2018) contains sentence pairs across multiple genres: Travel (T), Fiction (F), Government (G), Slate (S) and Telephone (Te). The NLI task involves classifying every premise-hypothesis sentence pair as Entailment, Neutral or Contradiction.
Amazon The Multi Domain Sentiment Analysis Dataset (Blitzer et al., 2007) contains Amazon product reviews for different type of products. We use reviews from the Apparel (A), Baby (B), Books (Bo), Cameras (C) and Movies (M) domains. Each review is labelled as positive or negative.

heading: B Details on Implementation
text: Models and Implementation We use T5v1.1, T0 and GPT-2 and LLaMA-2 from the HuggingFace library8, and use PyTorch9 to train our models.
Training We use the default hyperparameters from Liu et al. (2022), except for batch size and training duration. We perform a grid search for these values. We train each training phase for 30,000 steps on MNLI and 15,000 steps on the
8https://github.com/huggingface/transformers 9https://pytorch.org/
Training Steps Source Accuracy Target Accuracy
Amazon dataset, with a batch size of 8. For the T5v1.1 XL and T0 models (3B parameters each), we use a batch size of 1. We train with Adam and use a learning rate of 0.003. We set the maximum sequence length to 256 tokens. We use length normalization during evaluation, as proposed by Liu et al. (2022). For each experiment, we report the mean and standard deviation across 3 runs.
We choose the number of training steps based on early stopping on the validation set for one domain, and use that number of steps for all domains within that dataset. We report the test set performance after a varying number of training steps in the table below. For example, for the Apparel→Movies domain pair of the Amazon Reviews dataset, the performance saturates at 15,000 steps, as shown in Table 5.
Computations Using the (IA)3 PEFT framework, training the T5v1.1 Base model (60 million parameters) for 15,000 steps takes approximately two hours on a single NVIDIA RTX A6000 GPU. The T5v1.1 XL model and T0 model (3 billion parameters) take approximately 8 hours for 15,000 steps of training. For reproducibility, each experiment is repeated thrice, with changing random seeds. In total, we run 540 experiments with the Base model and 72 experiments with the larger models. This results in a total compute time of approximately 2400 GPU hours.

heading: C Detailed results with the Amazon and MNLI Datasets
text: Table 6 shows the performance of CPT on the Amazon and MNLI datasets.
On the Amazon dataset, CPT is competitive with the state of the art UDAPTER method from Malik et al. (2023) on average. We confirm this by checking for a significant difference in the performance of CPT and UDAPTER on the 20 dataset pairs. The Mann-Whitney U test and Student’s t-test both re-
sulted in non-significant p-values of 0.5516 and 0.8316, confirming the hypothesis that there is no significant difference between CPT and UDAPTER on the Amazon dataset.
However, on the MNLI dataset, where all domains have larger gaps, both significant tests showed a significant difference between CPT and UDAPTER, with CPT being more powerful. This is exemplified through cases like Travel (T) → Government (G), where CPT yields an accuracy of 83.6% on the target domain, equalling the upper bound of the Src+Tgt baseline.
Comparison with other Model Centric Approaches In addition to the MMD based method of Malik et al. (2023), we also compare CPT with other methods that promote domain invariance: 1) DANN (Ganin et al., 2016), which is the most widely used UDA method in NLP (Ramponi and Plank, 2020), but has been shown to be highly unstable; 2) CORAL (Sun et al., 2017), which minimizes second order statistics of the data embeddings. Additionally, with an emerging class of weight interpolation based methods, we make a comparison with task vector arithmetic (Ilharco et al., 2022). The use of task vectors with PEFT methods beyond LoRA (Hu et al., 2022) has been unexplored in the literature, and we find that the
method does not work with IA3. With fully finetuned models, the method improves in performance, but is still weaker than CPT.
CPT may learn representations that generalize across domains To better understand the improved UDA performance, we visualize the sentence embeddings learned by CPT in Figure 8. Using UMap (McInnes et al., 2018), the figure visualizes embeddings for the Apparel→Movies domain pair from the Amazon Product Review dataset. We see that CPT learns sentence embeddings that generalize across domains. For illustration, we draw a black line that cuts across both source and target domains. Note that the solid line suggests that there exists a classification hyperplane learned on the source labeled data (in blue and green). The same classifier can be potentially used to separate target data (in gray and orange). The visualization suggests that CPT achieves competitive UDA results without having to explicitly promote domaininvariant representations.

heading: D CPT across Model Architectures and Scales
text: We evaluate the performance of CPT over T5v1.1 XL and the instruction tuned T0 (3B) (Sanh et al., 2022) in Table 8.

heading: E PEFT Frameworks
text: The framework proposed in Section 2 is general and can be applied to fine-tune all model parameters. Additionally, our CPT framework is compatible with the parameter-efficient fine-tuning approach. The PEFT approach is desirable because it adds only a small amount of learnable parameters ϕ to a pre-trained language model θ, and finetunes only ϕ to perform prediction while keeping the other model parameters θ frozen. We use two instantiations in our implementations: Adapters (Houlsby et al., 2019) and (IA)3 (Liu et al., 2022).
(IA)3 is a state of the art PEFT learning method, and uses around a tenth of learnable parameters compared to popular methods like Adapters. (IA)3 works by element-wise multiplication (i.e. rescaling) of the model’s activations against a learned vector. In this case, the set of learnable parameters ϕ is a set of vectors {lv, lk, lff} applied to each attention mechanism and feed-forward layer as,
Amazon MNLI Src+Tgt UDAPTER CPT Src+Tgt UDAPTER CPT
h = σ ( Q(lk ⊚• KT )√
dk
) (lv ⊚• V )
h = (lff ⊚• γ(W1x)W2)
Here, K, Q and V are the key, query and value representations used in an attention block, and W1 and W2 are the weights in the feed-forward layer following an attention block. lk ∈ Rdk , lv ∈ Rdv , lff ∈ Rdff , σ is the softmax function while γ is any non-linearity.
Intuitively, each vector l simply learns weights measuring the importance of each feature in an activation of the pre-trained model, for the specific downstream task the model is trained on.
Adapters are a popularly used and high performing PEFT framework, and He et al. (2022) have shown equivalence in the operations applied by Adapters, Prefix Tuning (Li and Liang, 2021) and LoRA (Hu et al., 2022).
Adapters work by adding small learnable modules between transformer layers. Specifically, down and up projections Wdown ∈ Rd×r and Wup ∈ Rr×d are learnt such that ϕ =
{Wup,Wdown}. A residual connection and nonlinearity γ is added at every layer,
h = h+ γ(hWdown)Wup
Table 9 shows CPT beats UDAPTER across different tuning methods. We also note that finetuning yields slightly better performance for all UDA methods.

heading: F CPT in a Few-Shot Setup
text: Table 10 accompanies Figure 1 (Section 4), showing the 256-shot performance of CPT and other baselines, across model sizes. Similarly, Table 11 accompanies Figure 3, showing the relative performance of all baselines across varying k.
G Impact of Target Domain Exposure
The experiments in this section use the T5v1.1 base model on the Apparel→Movies domain pair of the Amazon reviews dataset.
Table 12 accompanies results from Figure 5, which show the impact of varying masking rates on CPT. Using the T5v1.1 base model, we train CPT using varying random masking rates on the Apparel → Movies domain pair, and report the mean and standard deviation over three runs. With high masking rates, the performance on the source domain is largely maintained, but the performance on the target domain rapidly deteriorates.

heading: H Understanding how CPT aids UDA
text: Table 13 (accompanies Figure 7) shows the impact of masking sequences at inference, on classification accuracy. Words are selected for masking based on their their “informativeness”, measured by their PMI to the inference class label. The performance of the model is best with the original unmasked sequences, indicating the presence of both informative and uninformative words are essential for strong classification performance.
Table 15 accompanies Figure 7 and shows the impact of varying masking strategies on classification performance, in a few-shot setting. We also consider two different few-shot setups: one with access to the full unlabelled datasets in phase 1 pretraining, and another where even the unlabelled data is few-shot.
To isolate any effects of PEFT methods or pretraining data, we repeat the analysis from Table 15 in Table 14 with fine-tuning Flan-T5 in a full data setting, and note similar trends.

heading: I Single Phase CPT Training
text: Our proposed approach in Section 2 involves two stages of training, which is more expensive than standard single phase UDA approaches. In this section, we propose a single training phase variant to CPT, and show that it performs similarly to the original method. We use the two phase pipeline in our experiments in the main paper, but note that the single and two phase pipelines are interchangeable.
We simply replace the two phase training with a joint multi-task objective as follows,
L(D,Dsrc; θ) = 1 |D| 1 |Dsrc| ∑
x′∈D
∑
(x,y)∈Dsrc
(λ l(C(x, y); θ) + (1− λ) l(M(x′); θ))
where l is the cross-entropy loss defined in Eq. (1), and M and C are the templates defined in Section 2. λ is the adaptation factor which gradually changes from 0 to 1 over the course of training. This results in the model being trained almost exclusively on the MLM task early on in training, and the CLS task towards the end of training.
Table 16 compares the performance of the single phase and two phase variants of CPT. We also compare with a vanilla joint single phase objective, where λ is fixed at 0.5 through training (called Single Phase Vanilla). The performance of the single and two phase variants are almost identical, and either can be used interchangeably. In comparison, the vanilla single phase method is significantly weaker on the target domain.
J Instability of Domain Invariance Methods for UDA
The Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) measures the difference between first order moments of variables in a Reproducing Kernel Hilbert Space (Aronszajn, 1950). Multiple lines of work have shown that minimizing divergence measures like MMD, when combined with auxiliary task-specific loss functions, results in training instabilities and vanishing gradients (Kashyap et al., 2021; Han and Eisenstein, 2019).
We also note that as minimizing MMD does not use any label information, there is a possibility for embeddings of the target domain to be aligned with the closest source domain class cluster. For example, Figure 9 shows us a setting where both
classes of the target domain (shown in green and gray) are mapped to the cluster of negative class
source embeddings (shown in blue). We compare variants of the UDAPTER method in Table 17 and show that the loss is sensitive to small changes in the loss design. Specifically we compare the UDAPTER method used in the main paper with:
• MMD over Logits: Measures the MMD between the logits of source and target domains, instead of using intermediate model outputs.
• Fixed Weight MMD: Instead of the multi-task loss for the MMD reduction and classification tasks, we use fixed weights for both tasks10.
• Two Phase MMD: The first training phase is used to minimize MMD between source and
10For the weighted loss, LCLS + 3 LMMD was found to be the best performing.
target embeddings, while the second phase is used to train the model for classification on the source domain.
CPT remains more powerful than all variants.

heading: Future Work
text: We leave the extending our analysis to more complex tasks to future work. In our study, we consider a class of PEFT methods that involve inserting learnable parameters between the layers of the model. Other classes of PEFT methods were not considered. However, we use Adapters and He et al. (2022) have shown connections between the method with Prefix Tuning (Li and Liang, 2021) and LoRA (Hu et al. , 2022). Due to the high variance across runs in PEFTbased learning, we note that the performance can vary significantly across random seeds. We attempt to make our findings reproducible by averaging every experiment over 3 seeds. Taking environmental costs into consideration, we reduce our computational budget by running a majority of our experiments with a smaller-sized model. Learning with larger models is discussed in Section 4