Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning
Authors: Quanyu Long
Wenya Wang
Sinno Jialin Pan
Venue: EMNLP 2023 Main (2023)
Keywords: Domain Adaptation
in-context learning
Retrieval Augmentation
TL;DR: We propose domain-adaptive in-context learning to acquire knowledge of both the target domain distribution and the discriminative task signal simultaneously.

Abstract: Large language models (LLMs) have showcased their capability with few-shot inference known as in-context learning. However, in-domain demonstrations are not always readily available in real scenarios, leading to cross-domain in-context learning. Besides, LLMs are still facing challenges in long-tail knowledge in unseen and unfamiliar domains. The above limitations demonstrate the necessity of Unsupervised Domain Adaptation (UDA). In this paper, we study the UDA problem under an in-context learning setting to adapt language models from the source domain to the target domain without any target labels. The core idea is to retrieve a subset of cross-domain elements that are the most similar to the query, and elicit language model to adapt in an in-context manner by learning both target domain distribution and the discriminative task signal simultaneously with the augmented cross-domain in-context examples. We devise different prompting and training strategies, accounting for different LM architectures to learn the target distribution via language modeling. With extensive experiments on Sentiment Analysis (SA) and Named Entity Recognition (NER) tasks, we thoroughly study the effectiveness of ICL for domain transfer and demonstrate significant improvements over baseline models.

Adapt in Contexts:
Retrieval-Augmenned Domain Adaptation via In-Context Learning
Quanyu Long' Wenya Wang' Sinno Jialin Pan 1.2
Nanyann Technological University, Singapore
"hh Chinese University of Hong Kong
{quanyuŁ01, wengwyJentu. edu. sinnopan@cuhk. edu. hk
Abstract
imput: Query
in study et Universitys Institute for Human Gene Therapy, rasaárcarrs altared common cold wrus carry version of (iee working dystrophin
Ently Prediction
Large language models (LLMs) have show- cased their capability with few-shot inference known as in-context learning. However, in- not always readily
News
Augmenting
domain demonstrations available in real scenarios, leading to cross- domain in-context learning. Besides, L.MSs are still facing challenges in long-tail knowl- edge in unseen and unfamiliar domains. The above limitations demonstrate the necessity of Unsupervised Domain Adaptation (UDA). In this paper, we study the UDA problem under in-context learning setting adapt language models from the source domain to the target domain without any target labels. The core idea is to retrieve subset of cross-domain ele- ments that are the most similar to the query, and elicit language model to adapt an in-context manner by learning both target domain distri- bution and the discriminative task signal simul- tancously with the augmented cross- coos-dommain in- context examples. We devise different prompt- ing and training strategies, accounting for dif- ferent LM architectures to learn the target dis- tribution via language modeling. With exten- sive experiments on Sentiment Analysis (SA) and Named Entity Recognition (NER) tasks, we thoroughly study the effectiveness of ICL. for domain transfer and demonstrate significant improvements over bascline models.
Target contexts: Rebrieved examples Our ogenization finds structural and devsiopmentel exprasston pattern of the ropeat gene DMR ND immediately Gpatleam Dystrophy Locus.
LMs
myotonic
Language Modeling
The XPR2 gene from Yarrows Applylica encodes an induciibe eikaline extracelluler protease.
Blomedical Literature
Figure 1: motivating example of retrieval-augmeente in-context adaptation for NER: biomedical texts re- trieved from the target unlabeled domain will serve demonstrative contexts to help LMs correctly pre- dict entities "Institute for Human Gene Therapy" and "dystrophin gene" (solid arrow). The language model transfers the knowledge to the target domain to identify unknown entities with a similar structure like "XPR2 gene" "dystrophy locus" by learning target distribu- tion with language modeling (dotted arrow).
LLMs are prone to generating unpredictable out- puts in undesired formats, and they are struggling with long-tail knowledge for unseen and unfamil- iar domains where topics and genres are less fre- quently encountered in the training corpus (Asai et al., 2023). Therefore, the limitations above call for effective adaptation strategies transfer knowl- edge of LMs from a labeled source domain the unlabeled target domain, known as Unsupervised Domain Adaptation (UDA).
1 Introduction
Large Language Models (LLMs) have demon- strated remarkable success in various tasks via in- context learning (ICL) with task instructions and few-shot demonstrations (input-label pairs) (Zhao et al., 2021; Liu et al., 2022; Min et al., 2022), elim- inating the need for fine-tuning from task-specific labels. Nevertheless, in-domain demonstrations are usually absent in real scenarios since the target domain labels are unavailable. Sourcing labeled ex- amples from other domains may suffer from huge syntactic and semantic domain shifts. Moreover,
To bridge the domain gap, UDA aims to adapt models that learn domain-agnostic features from labeled source samples and unlabeled target sam- ples. Some studies have proposed discrepancy mea- sures align source and target distributions in the representation space (Ganin et al., 2016; Ye et al., 2020; Long et al., 2022). However, these methods mainly focus on feature alignment and only ap- ply to encoder-based LMs. Other studies focus on adaptive pre-training including an additional post

---

pre-training phase of masked language modeling (MLM) on target unlabeled data to learn the tar- get domain distribution (Han and Eisenstein, 2019; Karouzos et al., 2021). However, different training phases make the learned diverse knowledge hard to remember, and such methods are also only applica- ble to encoder-only LMs which are usually smaller in scale. Therefore, few studies have investigated how to update knowledge of unfamiliar domains for larger LMs (e.g., decoder-only LMs). And few studies try to relate lource-labeled samples to tar- get unlabeled examples in a single training stage, while vast amounts of target unlabeled data can serve as a knowledge-rich datastore.
as well as the response output. Overall, we make the following contributions:
We propose domain-adaptive in-context learn- ing with retrieval augmentation in which we mix the source input and semantically rich tar- get contexts to learn two in-context objectives simultaneously;
We proposed unified framework with effi- cient prompting and fine-tuning strategies ac- counting for different architectures (encoder- only LMs and decoder-only LMs);
We thoroughly study the effectiveness of in-context learning for UDA. Our experi- ments surprisingly reveal that retrieving out- of-distribution demonstrations fails for LLMs' few-shot inference and fine-tuning is still ben- eficial for domain adaptation.
In this paper, we propose to retrieve similar ex- amples from the target unlabeled corpus to serve as the context of a source query and perform adap- tive in-context learning by concatenating the source query and target contexts the input prompt. The core idea is to elicit LMs to learn target distribu- tion and discriminative task signals simultaneously with the retrieved cross-domain examples. Fig. shows an illustrative example. For each input from the source domain, we compose its context with semantically similar texts retrieved from the target unlabeled domain to enrich semantics and reduce the domain difference in the surface form. Then the model will learn the task discrimination taking both the source input and the target context. To further mitigate domain shift, we propose to learn the target distribution using the language modeling mechanism (causal or masked language modeling) simultaneously by predicting tokens from the target context, which acts as proxy to the target distri- bution. Combining the two goals encourages the model to learn domain-agnen agnostic and task-aware in- formation which is beneficial for knowledge trans- fer.
Problem Definition
Consider scenario where we have access to two distinct domains: a source domain and a target domain. The source domain dataset, denoted as DS consists of n labeled data sampled i.i.d. from the source distribution, D = { M1 w........ where σ.द represents sequences of tokens, Yi repre- sents the corresponding label. On the other hand, the unlabeled target domain dataset, denoted D {xf) comprises m unlabeled data points, which are also sampled i.i.d. from the target domain. The primary objective of Unsupervised Domain Adaptation (UDA) is to adapt the knowl- edge learned from the source domain in such a way that allows it to generalize on the target domain effectively. This adaptation process involves lever- aging the unlabeled data from the target domain to learn the target distribution and mitigate the domain shift.
We propose domain-adaptive in-context Iearn- ing (DAICL) framework for different LM archi- tectures, including encoder-only and decoder-only models, and observe consistent advantages. To ac- count for the architectural difference, we devise distinct prompting and fine-tuning strategies. For the encoder-only model, we append contexts re- trieved from the target domain to each source input. The model is trained predict source input labels and masked tokens in the appended contexts. For the decoder-only model, we instead prepend exam- ples before the source input. The model is trained to predict each token autoregressively in the prompt
This paper focuses on the UDA problem over two application scenarios: Named Entity Recog- nition (NER)' and Sentiment Analysis (SA). We describe our method and pivot discussions around these two tasks in the following sections.
Method
We propose novel framework, Domain Adaptive In-Context Learning (DAICL), capable of training
'In our work, we only need to predict the entity spans, ignoring the entity type, because the label spaces for diffcrent domains are different when considering entity type. However, we still refer to this task as NER for short.

---

Source domain Input:
In the sludy the University's Institute for Human Gene Therapy, researchers allered common-cold carry version of the working dystrophin gene.
log Pra(t))
2-
1
CRF
log Pro(t(r)
query
Target domain unlabeled corpus:
Our oganization finds structural and developmental expression pallern mouse repeat gene immediately upstream of the myotonic Dystrophy Locus.
Trassformer-bssed Encoder
....*
The XPR2 from Yerrowia lipolytica encodes inducible extracellular protease.
(SEP)
MASK
Figure 2: An overview of training encoder-only NER models with retrieved context via in-context learning.
LMs to adapt with the help of retrieved contexts. We begin by introducing the overall framework in Section 3.1. Next, we present specific designs for Encoder-only language models in Section 3.2;
top-ranked (most similar) examples from the tar- get domain. For NER, we use BERTScore (Zhang et al., 2020; Wang et al., 2021), 1), because it gives a metric for each sentence based on the similarity of token representation, which is more crucial for the task of NER.
and Decoder-only language models in Section 3.3.
For decoder-only models, we present two settings: inference-only (Section 3.3.1) and fine-tuning (Sec- tion) 3.3.2.
Specifically, given a source sentence x paired with label y, we retrieve top-k relevant chunks of texts from the target unlabeled dataset DT. The retrieved examples are denoted as x. = {লī ... =xx² which will serve as the contexts to enrich the semantics for the source input.
3.1 In- In-Context Adaptation
The term In-Context Learning has been commonly referred to as few-shot prompting in LLMs. To be clear, in this work, instead use In-Context Learn- ing emphasize the idea of learning a model with semantically rich contexts. Here contexi should be differentiated with demonstration, the latter one represents input-label pairs in few-shot prompting. Under the setting of UDA where target labels are not accessible, context is composed of input-only examples from the unlabeled target domain. Next, we present overall framework to construct suit- able contexts and adapt LMs with suitable contexts,
3.1.2 Domain-Adaptive In-Conter Learning
With the retrieved context consisting most se- mantically similar examples to the source input, we seek a strategy to integrate this context into the source input and design a training objective that could learn target distribution and at the same time be able to discriminate the task label. To this end, we propose combine the following two objectives given the concatenated text sequences [x> ; IT). Objective 1: In-context Task Learning a supervised task to predict the task label y. Objec- tive 2: In-context Language Modeling a token prediction task to predict tokens from the target context b.:
3.1.1 Context Construction with Retrieval
Given input sentence from the source domain, we first search for semantically similar examples from the unlabeled target domain. This is analo- gous to retrieval and re-rank given a search query. Retreneel-augmented LM approaches (Guu et al., 2020; Lewis et al., 2020; Asai et al., 2023) ap- ply parametrized dense retriever to train with the task model. In this paper, we fix the retriever part and use the off-the-shelf scoring language models. For the SA task, al., 2021) which produces semantically meaning- ful sentence embeddings after being trained with contrastive learning (Chen et al., 2020; He et al., 2020). Here cosine similarity is used to retrieve
Ssu (0) - log Pro (,|x5, Fz'); (1)
LLM = log Pro (1 1x x ,"), C (2)
where represents the parameters for a language model. Ideally, the first objective (1) aims to learn task discrimination with the help of context. Note that unlike single-domain task prediction which only takes S as input, here we augment the source input with target contexts to learn mation across domains. The second objective (2)
SimCSE (Gao
infor-

---

encourages the model to learn the target distribu- tion by predicting tokens in the target context =". By mixing with a source input, the model learns to fuse the distributions from two different domains in order to bridge the domain gap. When combin- ing these two objectives, we expect that the model learns task-aware knowledge that is indistinguish- able from the two domains.
label y and masked tokens :y
min (a LD)
Σ
-[log Pro (yl x//))
(4)
+ A log (r [
where represents a scaling factor.
3.3 Decoder-only LMs with ICL
3.2 Encoder-only LMs with ICL
Recently, decoder-only LMs have received exces- sive attention and have motivated continuous devel- opments to scale up in order to solve various NLP tasks under zero-shot or few-shot settings, such as GPT-3 (Brown et al., 2020). LLaMA (Touvron et al., 2023), and ChatGPT. Despite the increasing scalability, they are still prone producing unpre- dictable outputs in undesired formats. For example, ChatGPT gives subpar performance for NER (see Table 1). This reflects the necessity of decoder-only LMs for learning to adapt to the target domain.
This section describes domain domain-adaptive in-context learning with encoder-only LMs, e.g.. BERT (De- vlin et al., 2019). As discussed in Section 3.1, for each input x we first retrieve top-k sentences x² from the target domain the context for x For encoder-only models, the retrieved sentences are then concatenated at the end of the source input.
[ 10,="" 1x (SEP) ; -'::.-.. =
(3)
where (SEP) is a separation token.
To perform in-context learning, recall from Sec- tion 3.1, two objectives (language modeling and task learning) are involved. An overview of the training process for encoder-only models on the NER task shown in Fig. 2. For the language mod- cling objective, we perform unsupervised Masked Language Modeling (MLM) on the target domain. We randomly sample 15% tokens from the target context :..;...: =x{]] and replace them with the [MASK] token. We denote the set of indices for the masked tokens M and the original ground-truth tokens for these masked positions are referred to as th [tyli= E M). The masked input becomes [x; M.l. where Jhe denotes the collection of tar- get contexts after masking. With the bidirectional structure of the encoder-only LMs, the represen- tation for each masked token in the target domain encodes both the target context and the source in- put. As such, the MLM objective encourages the encoder learn the target distribution that is indis- tinguishable from the source domain.
3.3.1 Cross-Domain Few-Shot Inference
Recent works show that providing few-shot ICL demonstrations (input-label pairs) contributes performance gains (Zhao al., 2021; Liu et al., 2022; Min et al., 2022). However, there are no in-distribution demonstrations available when per- forming inference on the unlabeled target domain. Therefore, in many real scenarios, we often select out-of cuttof--disribution (OOD) input-label pairs from another domain irrespective of the possible huge domain shift from the target query. In UDA, we have access the entire labeled source dataset, thus we could retrieve similar demonstrations from the source domain given a target query. We pro- vide prompts and examples in Fig. 3 showing how to use retrieved input-label pairs from the source domain as demonstrations.
In our experiments with ChaiGPT (see Table. I and Table. 2), surprisingly we find that re- trieving OOD demonstrations fails in most adap- tation scenarios; even randomly sampling cross- domain demonstrations can bring non-trivial per- formance gains comparing with the retrieval ap- proaches. However, fine-tuning much smaller LMs with in-context domain adaptation gives the best performances in most cases in our experiments. This phenomenon suggests we still need to fine- tune decoder-only LMs to update specific domain knowledge which will be discussed in the next sec- tion.
For the task objective, we use different predic- tion mechanisms for different tasks. For SA, we use average pooling on top of each token in the source input S before being fed into the classifier. For NER, we apply an additional CRF layer (Ma and Hovy, 2016; Lample et al., 2016) top of the LM feature which is common practice for token-level classifications.
Formally, the joint objective is to minimize the negative log- likeliieldoood of the ground truth task

---

Named Entity Span Prediction
Sentiment Analysis
Please identify all entities from the Input sentence. If there is no entity, please output None.
Givan the Input sentence, assign sontiment label from ('positive' 'neutraf', 'negative]. Return label only without any other text.
Swifenns: study Unioessy's institul
Gena Tharapy, carr version working
reserchers alterd common-cold
Sentence: This cable works well with my best thing is that is long so you don't have to be nghht noxt to your TV to reviow your photos and video.
dystaphhn gone.
Faily: fratiuut Humsi Gone Therary, dystnophii gane
Labal pralive
Sentence: The virus, which also was altered to misimioe its succoprbbiity to the immune system was then inyecled into the muscle calls of mice bred to lack dystraphin genes.
Sentence: Very good and normal quality
it Insteed onginats
cheap suliable
(haven't more words but the sito requlted wards
Faily dyatrophie genes
Label: postive
Sontence: Bath drugs are types intorforon. Eaily: None
Santence: dor't use the bendest every thy but asern not bad. Label: neutral
Source
Source
Sentence: Physical mapping 220 kb contromeric of the human MHC and DNA sequence analysis of the 43 Ab segment including the RINGY, HKES, and HKE4 genes.
Sentence: Spiritually and montally Inspiring! A book that allows you to question your morats and will help you discover who you really are! Label:
Target
Endty:
Target
Figure 3: Examples with prompts for inference. Different from fine-tuning setting that uses target unlabeled dataset as the retrieval corpus. for inference setting, we search input-label pairs from the source labeled dataset given a target test query. Dotted boxes contain demonstrations retrieved from the source.
Sentiment Analysis
anism, the objective is to predict the next token
Boltw instruatoon fuul dismrbbs Whlle resporse appropristayy completas requeest
4;
AAltestroollon hregative]] Return only without any
sentence, assign sentimmnn from ['positive; 'neutral,
min Σ, log log Pr(tilto. t1, ;.,1). (5)
8
:Sentence: lynce song so eowerrtul heartening
:Sontonce: Love albunt absolutoly unlitting would recommend to
Different from section 3.2, for decoder-only LMs, the retrieved target contexts x" need to be positioned before the source input x.S as the model will learn in an autoregressive manner. Morcover, instead of only calculating token prediction loss on the response/output y which is adopted for the In-context Task Learning objective as discussed in Section 3.1, we propose to compute the loss on every token within [prompt; x²;; ;x:::) v]- Objec- tive (5) can be decomposed into two objectives: 1) When l, E ,", the loss corresponds to token predictions in the target domain, analogous to the in-context language modeling objective; 2) When t; E y, the loss relates to in-context task learning which aims generate task label given both tar- get contexts and the source input. The objective (5) thus merges two proposed in-context objectives into a unified function.
Lanyono deserves stovs and
:Sentence: The CD banging
of hooks stopped
flom givin 2.5 but this a.must have
Input sentence:: This my must have books aince it a masterpiece spiritualiyy, the message behind so powerful that you read will take you enliggheemmen
#asRisponse: posttive
Figure 4: Illustration of crafted training example ppromp:: ==: .x.:v/. y). dotted box contains & = 3 input- only demonstrations from target unlabeled dataset.
3.3.2 Fine-tuning
In this work, we fine-tune LLaMA (Touvron et al., 2023) with a parameter efficient approach, i.e.. Low-Rank Adaptation (LoRA) (Hu et al., 2021). LoRA maintains the weights of pre-trained LMS while introducing trainable rank decomposition ma- trices into each transformer layer, making it feasi- ble fine-tune larger LMs with much fewer com- putational resources 2.
Similar to the method proposed in Section 3.2, we first retrieve top-k contexts from the target unla- beled set, given a source input query. We then insert these contexts in between the instruction and the source input sentence? (see an example in Fig. 4). Next, we finetune the decoder-only LMs given the crafted example [prompt; ; [fo. &1, t2, ...] and the source label. Specifically, with the Casual Language Modeling (CLM) mech-
4 Experiments
4.1 Datasets
NER datasets We experiment on 7 NER datasets covering four domains: News, Social media, Fi- nancial, and Biomedical. Under the News domain, CoNLL-03 English dataset (Sang and De Meulder, 2003) is the most popular NER dataset, and we treat the source domain dataset. The other three do- mains serve target domains. For the Social Me- dia domain, we use WN (Strauss et al., 2016) and WNUT-17 (Derezynski et al., 2017) collected from Twitter, For the Financial domain, we use
In our experiment, trainable parameters only account for 0.24% of the entire LLaMA-7B parameters.
We follow the template from Standford Alpaca (Taori ct al., 2023)

---

FIN (Alvarado et al., 2015) which is dataset of financial agreements. For the Biomedical domain: we use BC2GM (Smith al., 2008), BioNLPO9 (Kim et al., 2009), and BCSCDR (Li et al., 2016). Note that for different domains, entity types are different. For unsupervised domain adaptation, to ensure source and target domains share the same label space, we remove the entity types and convert all label formats to the BIO scheme", similar the problem of entity span prediction.
and Ba, 2015; Loshchilov and Hutter, 2018). In the experiments on NER datasets, the learning rate is set to le-5 for RoBERTa and 0.05 for CRF, For SA datasets, we set the learning rate to 5e-5 and use a linear scheduler with warm-up steps 10% of the total training steps. The weight factor À in (4) equals to 0.2.
For LLaMA-LORA, we the rank r to be 16, dropout rate to be 0.05. Trainable parameters only account for 0.24% of the entire LLaMA-7B param- eters. We fine-tune LLaMA-LoRA with batch size 256, learning rate 3e-4, and train 5 epochs with early stopping. With the help of LoRA, each adap- tation scenario only requires less than 1 hour of training time on single A A100 GPU.
Sentiment Analysis datasets We use the Amazon review dataset (He et al., 2018) which contains four domains: Book (BK), Electronics (E), Beauty (BT), and Music (M). The original crawled reviews con- tain star ratings (1 to 5 stars). Following previous work (He et al., 2018; Ye al., 2020), we label them with rating- < > 3, as negative, pos- itive, and neutral respectively. There are in total 12 adaptation scenarios, and we select of them in our experiment.
4.4 Results
We experiment with the following settings and base- lines. For Inference-only experiments: No demo performs zero-shot inference on each target test input without demonstrations.
Statistics and the data splits of all the datasets can be found in Appendix A.
Rand demo samples demonstrations randomly from the source domain.
4.2 Experiment Configurations
Retr demo retrieves top-5 demonstrations from the source domain, corresponding to the approach mentioned in Section 3.3.1.
For our retrieval system, we use SimCSE Roberta- Large (Gao et al., 2021) trained on NLI datasets the retrieval model for the SA task, and use RoBERTa-large (Liu et al., 2019) for BERTScore (Zhang et al., 2020) for the NER task We set k 3 for top-k retrieval from the target do- main. For the encoder-only model, we select XLM-RoBERTa-large (Conncau et al., 2020) as a basis which has 561M parameters. For the decoder-only model, we use LLaMA-7B² (Touvron al., 2023) and fine-tune it with LoRA (Hu et al., 2021). For inference-only LMs, we choose Chat- GPT and LLaMA- Alpaca For ChatGPT, we use gpt-3. 3pt-3.5-turie t LLaMA-Alpaca uses LoRA to fine-tune the LLaMA-7B model on Alpaca (Taori et al., 2023) dataset which is generated with Self- Instruct (Wang et al., 2022).
For fine-tuning experiments:
No-ICL does not retrieve any target context for each source input. The model is only trained on source inputs.
ICL-rand investigates the effectiveness of the task objective (1). Instead of retrieval, we randomly sample contexts from the target domain. In this case, the model is not exposed to semantically sim- ilar contexts in the target domain to enhance knowl- edge transfer via (1).
ICL-sup only trains the model via the task objec- tive (1). This investigates the effectiveness the language modeling objective (2). For the encoder- only model, we do not mask any token. For the decoder-only model, we calculate the loss corre- sponding to the response/output positions. ICL-source further investigates the effectiveness of the target contexts. Here we retrieve contexts solely from the source domain instead of the tar- get domain. Hence, the model learns to perform the task and language modeling within the source distribution.
4.3 Implementation Details
For training the RoBERTa model, fine-tune con- textualized embeddings using AdamW (Kingma
Forr example, entity "Los Angeles" has the label "B-LOC
I-LOC", convert "B "htts://github.com/ httpp//cctoonnp//Siiio. "htps://github.co httpe:///gthub.ooooo// com/ com/Tiiige/eeee Thiger//ert score
For computing efficiency, adopt the 7B model, but
DAICL is our proposed method domain- addaiveed in-context learning as shown in Section 3.2 and Section 3.3.2. As described in Section 3.1, this
method can be easily extended larger LM families.
"htpss//huggingface coltloen/alpaca-lore
MMay 24 version of ChatGPT's is used for experiments.

---

Financial
Social Media
Biomedical BioNLPO9
FIN
WNUT-16 WNUT-17
BC2GM
BC5CDR
Inference only
No demo
16.69
14.71
16.20
13.74
16.44
21.64
LLaMA-Alpaca
Rand demo
13.59
23.20
22.10
20.69
24.46
25.83
Retr demo
20.18
29,5
26.73
22.56
22.98
27.26
demo
19.60
32.10
33.45
19.90
15.44
37.16
ChatGPT
Rand demo
20.82
39.73
39.45
26.92
21.71
37.85
Retr demo
19.88
38.28
38.10
24.17
18.98
35.71
Fine-tuning
Vu ct al. (2020)
23.38
47.11
30.81
29.24
No-ICL
24.171.3
68.49h.2
63.18p.s
27.69
33.67
21.84..2
ICL-rand
24.912.9
69.26
64.66%
30.68
35.19)
26.93
RoBERTa
ICL-sup
26.24
70.89
65,40%
28.07
34.11a.a
23.20
ICL-source
24.911.2
68, 84g.a
63.38p.2
26.961.s
32.071.2
22.062.0
DAICL
27.22
71.79
65.79
32.51
36,81
25.92
No-ICL
15.20
45.22
$3,92
24.24
26,29
25,35
rCL-
12.68
42.09
51.08
23.06
21.66
21.28
LLaMA-LORA
ICL-sup
15.81
45.70
54.32
24.83
25.00
26.91
ICL-source
14.73
45.30
53.29
24.54
DAICL
14.82
46.51
55.08
26.02
24.21
28.96
Table results of Named Entity Span prediction tasks. The source domain is the CoNLL-03 dataset, and the target domains are financial, social media, and biomedical, For RoBERTa, results are reported with average and standard deviation in 5 runs, f represents the model is significantly stronger than the baseline model No-ICL with p < 0.05. For LLaMA, due the cost of inference computation, we only perform a single run.
method retrieves related contexts from the target domain and combines objectives perform domain-adaptiv ICL.
However, ICL-rand significantly impedes the per- formance of Sentiment Analysis. We conjecture that ICL-rand might select target contexts with op- posite sentiment labels from the source input, neg- atively affecting the learning process.
The experiment results for NER and SA are illus- trated in Table and Table 2, respectively. Below we conclude with some interesting findings.
Surprisingly, ChatGPT with random out-of- distribution (OOD) demonstrations achieves higher scores than retrieval in all NER and SA experi- ments (Rand demo vs. Retr demo). Previous work reveals that choosing demonstration examples that are close to the test input significantly enhances the effectiveness of ICL (Liu al., 2022; Rubin et al., 2022). However, they retrieve from a labeled training set in which the distributions of the text and label space are identical with the test input. contrast, in transfer setting which is close to the real-world scenario, we only have OOD input-label pairs from another labeled dataset. We make a hypothesis regarding this observation, for cross- domain ICI. providing diverse and distinct OOD demonstrations more beneficial for LLMs to un- derstand the task and generalize.
Adaptive ICL benefits UDA by learning two ob- jectives simultaneously. Given the results in Table and Table 2, we can observe that our proposed method DAICL which learns two objectives simul- tancously surpasses baselines with a large margin in most adaptation scenarios. From the result of ICL-sup, we find that training with the task objec- tive alone could slightly help UDA. As discussed in Section 3, the benefit originates from incorporating the target contexts for task discrimination. By com- paring DAICL with ICL-sup and ICL-source, we can conclude that the proposed in-context adapta- tion strategy enhances domain adaptation by jointly learning the task signal and language modeling si- multaneously.
Retrieving OOD examples could be disappoint- ing for LLMs. From the RoBERTa results of ICL- rand, find that random target contexts can im- prove NER (compared with No-ICL) by small margin. One possible reason that random con- texts from the target domain could still encourage the model to learn the target distribution via (2).
Fine-tuming is still beneficial for UDA. Under the UDA setting where labels only exist in the souree domain, we prompt LLMs with input-label pairs from the source domain infer the target la- bel (inference-only). Another option is to fine-tune smaller LMs to adapt task task-aware knowledge from the source to the target domains. natural ques-

---

E-BK
BT-BK Inference only
BK-B1
BK-M
BK
M-+T
Ave.
No demo
61.53
61.33
63.72
58.86
59.41
63.72
61.46
LLaMA-Alpaca
Rand demo
54,33
53,45
60.48
49.09
51.98
63.78
55,85
Retr demo
60.9
63.58
69.35
60.33
61.36
67.82
64,06
demo
72.68
72.68
72.27
70.06
69.83
72.27
71.63
ChatGPT
Rand demo
73.10
73.27
74.37
71.18
71.44
74.3
72.94
Retr demo
73.07
71.92
73.82
69.69
71.00
73.57
72.18
Fine-tuning
Long al. (2022)
70.330.3
70.92o.o
64.131.4
64.671.7
62.36a.7
65.40o.s
66.30
Ye et al. (2020)
70,90u.4
71,38p.8
67.48u.4
67.16u.u
64.001.2
70.710.3
68.61
No-ICL
68.33o.5
69.85p.6
65.921.1
61 1..4
61.36a.r
67.430.s
65.73
RoBERTa
ICI.-rand
67.61o.s
68.74p.6
64.80
61.59,.
61.44..a
66.72,.7
65.15
ICI.-sup
69.68
71.15
68.79
64.88
63.16
69.15
67.80
ICI--source
68. 70..s
70.64
65.29
61.812.2
61.75
66.89
65.84
DAICL
71.21
72.81
68.64
66.93
66.08
71.44
69.52
No-ICI.
74.15
74.30
72.97
70.42
70.08
70.15
72.01
ICL-rand
65,22
64.17
60.48
61.95
59.36
63.44
62,43
LLaMA-LORA
SCL-
76.10
75.20
72.25
71.63
71.78
70.54
72.75
ICI--source
70.18
68.45
68.46
63.27
67.23
67.94
67.59
DAICL
77.30
76.30
74.02
73.40
70.38
72.37
74.13
Table Accuracy(%) results of Amazon Review Sentiment Analysis. For example, E-BB represents training on Electronics (E) and adapting to Book (BK). There are 4 domains available, choose 6 out of 12 adaptation tasks.
tion to ask is can the few-shot prompting paradigm substitute the fine-tuning paradigm? In NER exper- iments, ChatGPT achieves very low performances, but fine-tuming a much smaller RoBERTa model achieves state-of-the-are scores in most adaptation scenarios. SA experiments, fine-tuning LLaMA with even fewer trainable parameters (1.7M) out- performs all the other methods. Hence, we hypothe- size that although LLMs have strong generalization ability, they cannot tackle problems in all domains. When it comes to UDA, designing an effective adaptation strategy is still beneficial.
the source input; 2) adaptive ICL learns two losses simultaneously, and for decoder-only model, we only have one type of task which merges these two losses intrinsically.
WNUT17
BC2GM
E E-BBK
M MBT
pre-train
54.62
25.78
74.20
70.45
No-ICL
53.92
24.24
74.15
70.15
DAICL 55.08
26.02
77.30
72.37
Table 3: A comparison of adaptive ICL and adaptive pre-training for ILaMA. On NER, we use CoNLL- 03 03-WN1T11 and CONLL-03-BC2GM For SA. we use BB and M-BBT.
To compare the two approaches, we conduct experiments on LLaMA-LoRA to perform adaptive pre-training. In the first stage, we pre-train LoRA weights using target unlabeled texts. In the second stage, we start from the LoRA checkpoint obtained in the previous stage and continue fine-tuning it with task supervision. We use the same Alpaca template but do not provide demonstrative context. Results can be found in Table 3. No ICL is identical to the second stage in adaptive pre-training.
4.5 Analysis
Adaptive ICL or Adaptive Pre-training? In Sec- tion 3.1, we propose to learn the two objectives simultaneously with the help of the target contexts. What if we separate the two objectives into dif- ferent training stages? In the first stage, we con- tinue pre-trainnn pre-training LMs on the unlabeled target do- main dataset with the language modeling objec- tive. In the second stage, supervised fine-tuning is performed on the labeled source domain dataset with the task objective. This two-step UDA pro- cedure is called adaptive pre-training or post pre- training (Han and Eisenstein, 2019; Vu et al., 2020; Karouzos et al., 2021). There are two differences between adaptive pre-training and adaptive ICL which we propose: 1) adaptive ICL mixes source input with target contexts when performing task predictions while adaptive pre-training only takes
We could observe that pre-training only gains marginal benefits for SA tasks compared with No- ICL. We conjecture that the domain gap is smaller in SA datascts than in NER datasets. The pro- posed adaptive ICL strategy outperforms adaptive pre-training, which could be attributed to the fact that the decoder-only model under adaptive ICL. can learn the two objectives with demonstrative contexts.

---

5 Related Work
and they also fine-tune the retrieval model, leading to stronger ICL performance. Rubin et al. (2022) trains a dense retriever select demonstrations that have a positive impact on the learning process.
Unsupervised Domain Adaptation
Traditional methods include Pseudo-labeling (Ye et al., 2020). Pivot-based approach (Pan et al., 2010), and adversarial neural network (Ganin et al., 2016). Recently, Adaptive pre-training on domain- specific corpora has proven to be an effective pro- cess for adaptation, such as BioBERT (Lee et al., 2019) which is specialized variant of BERT. Han and Eisenstein (2019) proposes AdaptaBERT, which includes a second phase of unsupervised pre- training for BERT in unsupervised domain adap- tation. Karouzos et al. (2021) proposes a mixed multi-task loss to learn classification and MLM. Chronopoulou et al. (2019) utilizes an auxiliary LM loss to prevent catastrophic forgetting in trans- fer learning.
Conclusion
In this work, we propose domain-adaptive in- context learning to acquire knowledge of both the target domain distribution and the discriminative task signal simultaneously. We develop different prompting and fine-tuning strategies that take into account various LM architectures and different lan- guage modeling mechanisms. Overall, our frame- work demonstrates significant performance gains over an extensive spectrum of cross- coos--dmmann ex- periments, and we perceive that fine-tuning is still effective and promising in the era of large language models when it involves domain shift.
Retrieval-Augmentted Language Models Retrieval bassd LMs have shown to effective in improving LM performance (Asai et al., 2023). The retriever with various knowledge datastores can provide up-to-date information since LMs cannot memorize all long-tail knowledge in the parame- ters. REALM (Guu al., 2020) pre-trains and fine- tunes encoder-only model jointly with a knowl- edge retriever by modeling documents as latent variables and marginalizing over all possible docu- ments. While RAG (Lewis et al., 2020) fine-tunes an encoder-decoder model with a non-parametric retriever by fixing the search index. Atlas (Izacard et al., 2022) combines RAG with pre-training on tasks. Replug (Shi et al., 2023) proposes adapting the dense retriever to the black-box large language models to reduce the generating perplexity.
Limitations
Our retrieval system is based on SimCSE and BERTScore to choose semantically similar con- texts following previous work. However, we do not explore other scoring and re-ranking metrics, or explore methods to train a dense retriever. On the other hand, it is hard tell what makes good demonstration simply based considering that the retrieval system does not have access to the inference task. We leave this for fu- ture work to explore what a good demonstrative example when encountering domain shift.
retrieval system,
open-domain QA and
Ethics Statement
To ensure the ethical use of Artificial Intelligence in the legal field, we have taken measures such anonymizing sensitive information in real-world datasets. In addition, our model's predictions should be served as supportive references for judges, assisting them in making judgments more efficiently, rather than solely deternining the judg- ments.
In-Context Learning
In the context of ICL, previous studies indicate that it primarily exposes the model's infrastructure learned during pre-training. Xie al. (2022) pro- vides evidence that ICL can be interpreted type Bayesian inference, where demonstrations act as noisy evidence. (Min et al., 2022) shows that the advantages of ICL mainly stem from having the appropriate distribution of inputs and labels, rather than solely focusing on the correctness of individual labels. Previous research has revealed that in scenarios where abundant training data is ac- cessible, retrieving examples that are similar the test input as demonstrations significantly enhances ICL performance. Liu et al. (2022) introduces a retrieval module for GPT-3 (Brown et al., 2020)
A Acknowledgements
This work is partially supported by the 2020 Mi- crosoft Research Asia collaborative research grant. Sinno J. Pan thanks for the support from HK Global STEM Professorship and the JC STEM Lab of Ma- chine Learning and Symbolic Reasoning.

---

References
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 6894-6910.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timo- thy Baldwin. 2015. Domain adaption named entity recognition to support credit risk assessment. In Pro- ceedings of the Australasian Language Technology Association Workshop 2015. pages 84-90.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023, Acl 2023 tutorial: Retrieval-basee based lan- guage models and applications. ACI. 2023.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsu- pervised domain adaptation of contextualized em- beddings for sequence labeling. In P Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNNLP) pages 4238 4248.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1 1877-9911
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational Conference on Machine Learning, pages 1597-1607. PMLR.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2018. Adaptive semi-supervised learning for cross-domain sentiment classification. ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3467- 3476.
Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019. An embarrassingly simple approach for transfer learning from pretrained language models. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 2089-2095.
Pro-
Edward Hu, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint 232.0920...
Leon Derezynski, Eric Nichols, Marieke Van Erp, and Nut Limsopatham. 2017. Results the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 140-147.
Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021. Udalm: Unsuper- vised domain adaptation through language modeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2579-2590.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Popers), pages 4171- 4186.
Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo. Yoshi- nobu Kano, and Jun ichi Tsujüi. 2009. Overview of bionlp' '09 shared task on event extraction. In Pro- ceedings of the BioNLP 2009 workshop companion volume for shared task, pages 1-9.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversari training of neural networks. The journal of machine learning research, 17(1):2096- 2030.
Guillaume Lample. Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition.

---

In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional L Linguistics: Human Language Technologies, pages 260-270.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Confer- ence on Empirical Methods in Natural Language Processing.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 12:1234- 1240.
Sinno Jialin Pan, Xiaochuan Ni, Tao Sun, Qiang Yang. and Zheng Chen. 2010. Cross-domain senti- ment classification via spectral feature alignment. In Proceedings of the 19th international conference World wide web, pages 751-760.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, et al. 2020. auggeetedd generation for knowledge-inteneee intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671.
Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sci- aky, Muih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. Biocreative cdr task corpus: a resource for chemical disease relation extraction. Database: The Journal of Biological Databases and Curation, 2016.
Erik Tjong Kim Sang and Fien De Meulder. 2003. In- troduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142 147.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James. Mike Lewis. Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301.12652.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114.
Larry L. Smith, Lorraine K. Tanabe, Rie Ando. Cheng- Ju Kuo, I-Fang Chung, Chun-Nan Hsu, Yu- Shi Lin, Roman Klinger, C. Friedrich, Kuzman Ganchev, Manabu Torii, Hongfang Liu, Barry Had- dow, Craig A. Struble, Richard J. Povinelli, An- dreas Vlachos, William A. Baumgartner, Lawrence E. Hunter, Bob Carpenter, Richard Tzong-Han Tsai, Hong-Jie Dai. Feng Liu, Yifei Chen, Chengjie Sun. Sophia Katrenko, Pieter W. Adriaans, Christian Blaschke, Rafael Torres, Mariana L. Neves, Preslav Nakov, Anna Divoli, Manuel Maña-López, Jacinto Mata, and W. John Wilbur. 2008 Overview of biocre- ative gene mention recognition. Genome Biology, 9:S2 S2.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.19099999
Quanyu Long, Tianze Luo, Wenya Wang, and Sinno Pan. 2022. Domain confused contrastive learning for unsupervised domain adaptation. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 9982-2995.
Benjamin Strauss, Bethany Toma, Alan Ritter, Marie- Catherine De Marneffe, and Wei Xu. 2016. Results of the w named entity recognition shared task. In Proceedings of the 2nd Workshop Noisy User- generated Text (WNUT), pages 138 144.
Nlya Loshchilov and Frank Hutter. 2018 Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end se- quence labeling via bi-directional -stm--nns-ee erf. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 1064-1074.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An cuatuuctionfoolooing llama model. https: // gi thubu. www.////tttttaaaaaoooooo stanford_alppac..
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image- based recom- mendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43-52.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint MxXiv:2302.13977

---

Thuy Vu, Dinh Phung, and Gholamreza Haffari. 2020. Effective unsupervised domain adaptation with ad- versarially trained language models. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 6163- 6173.
Xinyu Wang. Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang. Fei Huang. and Kewei Tu. 2021. Improving named entity recognition by external con- text retrieving and cooperative learning. In Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 1ih International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 1800-1812.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212 12. 10560.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. Interna- tional Conference on Learning Representations.
Hai Ye. Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng. and Lidong Bing. 2020. Feature adaptation of pre-trained language models across languages and domains with robust self-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7386- 7399.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Confer- ence on Learning Representations.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before Improv- ing few-shot performance of language models. In In- lernational Conference Machine Learning, pages 12697-12706. PMLR

---

A Datasets
by randomly sampling 1000 instances from the source labeled Set 1. For example, when perform- ing E->BK adaptation task, we use Electronics Set as the training set and validation set, we use Book Set as the target unlabeled set, and we retrieve similar examples from this set as contexts. The evaluation will be performed in Book Set 1.
Train
Dev
CONLL-03
14,987
3,466
3,684
FIN
1169
306
WNUT-16
2,394
1,000
3,849
WNUT-17
3,394
1,009
1.287
BC2GM
12574
2519
5038
BIONLPO9
7462
1448
2446
BC5CDR
4,560
4,581
4,797
B Example Input and Output Pairs of ChatGPT and LLaMA
Table 4: Statistics of the dataset split of NER dataset.
For NER datasets, we select CoNLL-03 training the source labeled dataset and CoNLL-03 dev set the validation set for adaptation. When adapt- ing to target domain, for example, WNUT16, we use WNUTI6 training set as the unlabeled target dataset by discarding training set. That is, in our approach, for the fine- tuning setting, we retrieve text-only examples from WNUT16 training dataset as the contexts of source input CoNLLO3. Statistics can be found in Table 4
labels from this
DOMAIN
Neg
Pos
Total
Sct
2000
2000
2000
6000
BOOK
Set 513
4824
6000
Set
2000
2000
2000
6000
ELEC
Set
694
489
4817
6000
Set
2000
2000
2000
6000
BEAUTY
Set
616
675
4709
6000
Set 2000
2000
2000
6000
MUSIC
Set 785
774
4441
6000
Table 5: Statistics of the dateset split of NER dataset.
For the Amazon review dataset, it does not re- move neutral labels, which is advantageous in un- supervised domain adaptation (UDA) scenarios where label information from the target domain is unavailable. A summary of this dataset can be found in Table 5. For SA. each dataset consists of two sets. Set I contains 6,000 instances with bal- anced class labels, while Set comprises instances randomly sampled from a larger dataset (McAuley et al., 2015), preserving the authentic label distribu- tion. It is important to note that there no overlap between the examples in these two sets. Following the approach outlined in (He et al., 2018), Set is used as the training set for the source domain. While the label distribution in the target domain is unpredictable and beyond control in real-life scenarios. it is more reasonable to use Set 2 as the unlabeled set for the target domain. Finally, the model is evaluated on Set from the target domain. Regarding the data split, a validation set is created

---

Table 6: Example input and output pairs for ChatGPT on NER dataset BC2GM. No demonstration.
Please identify all entities from the input sentence. If there no entity, please output None.
Sentence: Physical mapping 220 kb centromeric the human MHC and DNA sequence analysis of the 43 kb segment including the RINGI, HKE6 and HKE4 genes.
Entity: Physical mapping
human MHC
DNA sequence analysis
RING1 gene
НKЕ6 gene
HKE4 gene
Please identify all entities from the input sentence. If there no entity, please output None.
Sentence: DNA elements recognizing NF and Spl regulate the human multidrug resistance gene promoter.
Entity:
1. DNA elements
2. NF-Y
3. Spl
4. human multidrug-resistance gene promoter
Please identify all entities from the input sentence. If there is no entity, please output None.
Sentence:
Like other IAPs, ch IAP1 contains the terminal baculovirus IAP repeats and C- terminal RING finger motifs..
Entity:
IAPs, ch-IAP1, baculovirus IAP repeats, RING finger motifs.
Please identify all entities from the input sentence. If there no entity, please output None.
Sentence: To clarify the difference. both the Crk II and Crk II - 23, proteins were expressed in E coli and examined their binding capacity in vitro.
Entity:
Crk II
-Crk II- 23
-E. coli
Please identify all entities from the input sentence. If there is no entity, please output None.
Sentence: GT rich sequence binding the transcription factor Sp1 is crucial for high expression of the human type VII collagen gene (COL7AI) ) in fibroblasts and keratinocytes. Entity:
1. GT-rich sequence
2. Transcription factor Spl
3. Human type VII collagen gene (COL7A1)
Please identify all entities from the input sentence. If there no entity, please output None.
Sentence: Manual sample clean up procedures as well as the addition of an internal standard are not needed.
Entity:
None.

---

Table Example imput and output pairs for ChatGPT on NER dataset BC2GM. Retrieved demonstrations are from CoNLLO3.
Please identify all entities from the input sentence. If there is no entity, please output None.
Sentence: In the study at the University 's Institute for Human Gene Therapy researchers altered common-cold virus carry version of the working dystrophin gene. Entity: Institute for Human Gene Therapy
Sentence: The virus, which also was altered minimise its susceptibility to the immune system, was then injected into the muscle cells of mice bred to lack dystrophin genes.
Entity: None
Sentence: We agreed that following detailed scientific analysis using a methodology which would take out the maximum number of BSE cases possible.
Entity: BSE
Sentence: In the experiment, between 30 40 percent of the muscle fibers in group of mice produced dystrophin for two weeks before diminishing.
Entity: None
Sentence: Stork Hl results breakdown per sector. Entity: None
Sentence: Physical mapping 220 kb centromeric of the human MHC and DNA sequence analysis of the 43 kb segment including the RINGI, HKE6 and HKE4 genes.
Entity:
human MHC, RINGI, HKE6, HKE4 genes.
Please identify all entities from the input sentence. If there is no entity, please output None.
Sentence: In the study the University 's Institute for Human Gene Therapy researchers altered common-cold virus to carry version of the working dystrophin gene.
Entity: Institute for Human Gene Therapy
Sentence: Individuals with the disease have non working version of gene responsible for producing a crucial muscle protein called dystrophin.
Entity: None
Sentence: The virus which also was altered to minimise its susceptibility to the immune system, then injected into the muscle cells of mice bred to lack dystrophin genes.
Entity: BSE
Sentence: Both drugs are types of interferon.
Entity: None
Sentence: When it approved Avonex May the FDA said both Biogen 's product and Betaseron were developed under the incentives of the Ophran Drug Act which provides seven years of marketing exclusivity for products that treat rare diseases. Entity: Avonex, FDA, Biogen, Betaseron, Ophran Drug Act
Sentence: DNA elements recognizing NF- Y and Sp1 regulate the human multidrug resistance gene promoter.
Entity:
NF-Y, Y, Sp1, human resisaneu gene promoter.

---

Table 8: Example input and output pairs for ChatGPT on NER dataset BC2GM. Demonstrations are randomly sampled from CoNLLO3.
Please identify all entities from the input sentence. If there no entity, please output None. Sentence: The 32-year-old defender played seven seasons with Nantes and was with Paris St Germain for five seasons.
Entity: Nantes, Paris St Germain
Sentence: 29,582
Entity: None
Sentence: The Palestinian Authority was set up under the 1993 PLO-Israel interim peace deal.
Entity: Palestinian Authority, PLO-Isracl
Sentence: In Home Health said it previously recorded reserve equal to 16 percent of all revenue related to the community liaison costs. Entity: In Home Health
Sentence: realised this year, that without putting 99.9 percent of your mind into tennis I do n't think you successful, said the 22-year-old Medvedev..
Entity: Medvedev
Sentence:
Physical mapping 220 kb centromeric the human MHC and DNA sequence analysis of the 43 kb segment including the RINGI, HKE6 and HKE4 genes.
Entity:
human MHC, RINGI, HKE6, HKF4.
Please identify all entities from the input sentence. If there no entity, please output None.
Sentence: The blue-chip CAC-40 index ended 2.43 points or 0. percent lower at 2,017.99 points after brief foray into positive territory when the New York stock market opened higher.
Entity: CAC-40, New York
Sentence: American League
Entity: American League
Sentence: 1886 Skeleton Canyon in Arizona, Geronimo, Apache chief and leader of the last great Red Indian rebellion finally surrendered to General Nelson Miles
Entity: Skeleton Canyon, Arizona, Geronimo, Red Indian, Nelson Miles
Sentence: Formmula Shell leads best-of-seven series 1-0)
Entity: Formula Shell
Sentence: PRESS DIGEST- Jordan Aug 25_ Entity: Jordan Sentence: DNA elements recognizing NF and Spl regulate the human multidrug resistance gene promoter.
Entity:
NE-Y, Sp1, DNA elements.

---

Table 9: Example input and output pairs for LLaMA-Alpaca NER dataset BC2GM. No demonstration. Note that Alpaca-lora is not fine-tuned on NER datasets.
Below is instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please identify all entities from the input sentence. If there is no entity, please output None.
Input sentence: Physical mapping 220 kb centromeric of the human MHC and DNA sequence analysis of the 43 kb segment including the RING1, HKE6 and HKE4 genes.
### Response:
None
Below is instruction that describes task, paired with input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please identify all entities from the input sentence. If there no entity, please output None.
Input sentence: DNA elements recognizing NF Y and Sp1 regulate the human multidrug resistance gene promoter.
### Response:
DNA elements, NF Spl
Below is an instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please identify all entities from the input sentence. If there no entity, please output None.
Input sentence: Like other IAPs, ch IAPI contains the N terminal baculovirus IAP repeats and C- terminal RING finger motifs ### Response:
None
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please identify all entities from the input sentence. If there no entity, please output None.
Input sentence: GT rich sequence binding the transcription factor Sp1 is crucial for high expression of the human type VII collagen gene ( (COL7A1) ) in fibroblasts and keratinocytes.
### Response:
human type VII collagen gene

---

Table 10: Example input and output pairs for fine-tuned LLaMA-7B on NER dataset BC2GM. Inference with retrieved demonstration.
Below is instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please identify all entities from the input sentence. If there is no entity, please output None.
Sentence: In the study at the University 's Institute for Human Gene Therapy researchers altered common-cold virus carry version of the working dystrophin gene. Entity: Institute for Human Gene Therapy
Sentence: The virus, which also was altered minimise its susceptibility to the immune system, was then injected into the muscle cells of mice bred to lack dystrophin genes.
Entity: None
Sentence: We agreed that following detailed scientific analysis using a methodology which would take out the maximum number of BSE cases possible.
Entity: BSE
Sentence: In the experiment, between 30 40 percent of the muscle fibers in group of mice produced dystrophin for two weeks before diminishing.
Entity: None
Sentence: Stork Hl results breakdown per sector Entity: None
Input sentence: Physical mapping 220 kb centromeric of the human MHC and DNA sequence analysis of the 43 - kb segment including the RING1, HKE6 and HKE4 genes.
### Response:
Human MHC, RINGI, HKE6, HKE4 genes
Below is an instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please identify all entities from the input sentence. If there no entity, please output None.
Sentence: In the study at the University 's Institute for Human Gene Therapy, researchers altered a common-cold virus to carry version of the working dystrophin gene.
Entity: Institute for Human Gene Therapy
Sentence: Individuals with the disease have working version of gene responsible for producing crucial muscle protein called dystrophin. Entity: None
Sentence: The virus which also was altered minimise its susceptibility to the immune system, was then injected into the muscle cells of mice bred to lack dystrophin genes.
Entity: BSE
Sentence: Both drugs are types of interferon.
Entity: None
Sentence: When it approved Avonex in May the FDA said both Biogen 's product and Betascron were developed under the incentives of the Ophran Drug Act which provides seven years of marketing exclusivity for products that treat rare diseases. Entity: Avonex, FDA, Biogen, Betaseron, Ophran Drug Act
Input sentence: DNA elements recognizing NF Y and Sp1 regulate the human multidrug resistance gene promoter.
### Response:
NF Y. Spl


Reviews:
[{"review_id": "t2krvIkfq4", "reviewer": "Reviewer_9Foj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\n#### **comment #1**:\nIs it true that cross domain transfer has been achieved due to the similarity of sentences between the source domain and target domain?\n\n#### **response #1**:\nYes, a part of the domain transfer is accomplished through the similarity of the retrieved examples from the target domain. We compose the source domain input contexts from the target domain to enrich the semantics and reduce the domain difference in the surface form. Furthermore, these examples, served as demonstrative contexts, can help the model correctly predict labels. Additionally, in order to further mitigate the domain shift, we learn target distribution from a source-target joint view (where we mix the source and target, and the model learns to fuse the distribution and bridge the domain gap) with language modeling (masked or causal), this is different from previous adaptive pretraining approach which only learns target distribution from target sentences only (single view).\n\nTo investigate the effect of similarity, we conducted experiments involving random sampling. Observing Tables 1 and 2, it is evident that retrieved contexts with similar examples (referred to as ICL-retr) outperform randomly selected contexts (referred to as ICL-rand).\n\n#### **comment #2**:\nThe paper lack comparison with other research papers.\n\n#### **response #2**:\nWe have already included the previous SOTA results [1][2][3] for both tasks in our experiment tables. And our proposed method demonstrates superior performance compared to the prior approaches. To ensure a fair comparison, we also construct our baselines (No-ICL, ICL-rand, ICL-source, ICL-sup) using the same backbone model. Furthermore, we deliberately select more challenging datasets in order to demonstrate more meaningful results. The field of unsupervised domain adaptation commonly employs the Amazon Review Benchmark dataset. However, this benchmark proves to be  quite easy for previous works (achieving more than 0.9 in terms of accuracy with slight variations), since it discards the neutral instances. Thus, we consider a more challenging dataset in our paper which does not remove neutral labels.\n\nThank you for your insightful feedback again!\n\n[1] Effective unsupervised domain adaptation witg adversarially trained language models. Thuy Vu, Dinh Phung, and Gholamreza Haffari.\n \n[2] Domain confused contrastive learning for unsupervised domain adaptation. Quanyu Long, Tianze Luo, Wenya Wang, and Sinno Pan.\n\n[3] Feature adaptation of pre-trained language models across languages and domains with robust self-training. Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng, and Lidong Bing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has high reproducibility and has conducted experiments on the Named Entity Recognition and Sentient Analysis task, using different LM structures including encoder- and decoder-only models.", "reasons_to_reject": "1. For the input sentences given in the source domain, this paper retrieves similar sentences from the target domain as the context. Then, through a supervised task to predict the task label, and through a token prediction task to learn the target distribution. Is it true that cross domain transfer has been achieved due to the similarity of sentences between the source domain and target domain?\n2. The paper experiment is incomplete, lacks comparison with other research papers on Unsupervised Domain Adaptation method, and does not fully compare with existing research results on the dataset used in the experiment.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a Unsupervised Domain Adaptation method under in-context learning mechanism, which adapts language models from the source domain to the target domain. It mainly achieves the above goals through eliciting language models to learn both target domain distribution and the discriminative task signal via in-context examples.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7ryKtUnAM4", "reviewer": "Reviewer_cMth", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\n#### **comment #1**:\nIt would be worthwhile to understand if the framework can be extended to multiple source and target domains?\n\n#### **response #1**:\nThanks for pointing it out. Yes, our proposed framework can be extended to multiple source and target domains, since we could retrieve several examples from multiple unlabeled target domains. However, in the field of unsupervised domain adaptation, a common practice is to use one labeled source domain and one unlabeled target domain for easy comparisons and analysis.\n\n#### **comment #2**:\nWhen source and target domains are very different, would our method lead to a performance drop?\n\n#### **response #2**:\nWe conduct extensive experiments on sentiment analysis and named entity recognition tasks, including several transfer scenarios where the source and the target are very distinct. From our experiment results, the proposed method can consistently outperform baselines with a large margin across the majority of adaptation scenarios.\n\nFor NER, we quantify the similarity between source (CoNLL2003) and target domains [1]. This assessment was based on the vocabulary overlap between these domains. We discover that all the target domains are dissimilar to the source domain, with Financial domain having the lowest overlap (only 9.4%). Even within the challenging Financial domain, we can still observe performance gain over the No-ICL baseline.\n\nThank you for your insightful feedback again!\n\n[1] Effective unsupervised domain adaptation with adversarially trained language models. Thuy Vu, Dinh Phung, and Gholamreza Haffari.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles an interesting problem, wherein an LM is adapted in an unsupervised manner to a new target domain, by injecting contexts and training on an MLM learning task. This joint adaptation might help the LM to better understand the co-relations with the source and target domains, enabling it to learn better.\n\n2. Experimental results showcase improved performance of the proposed formulation.", "reasons_to_reject": "1. It would be worthwhile to understand if the framework can be extended to multiple source and target domains?\n2. If the source and target domains are radically different, would the injection potentially lead to performance drop, due to more noise?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes in-context learning to adapt LMs to a unlabeled target domain by using in-context examples (from the target domain) based on training examples from the source domain. The paper shows different training mechanisms for different LM architectures, and showcase improved results on two NLP tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yuL4a2hwDv", "reviewer": "Reviewer_gjmy", "summary": "", "questions": "Even for encode-only models the target example could be placed before the source example in the prompt. This could help reduce the gap between the solutions for encoder-only and decoder-only models. Did the authors consider this?\n\nIn decoder-only training (Eq. 5) would it help to weight  the prediction of $y$ higher than the prediction of $x^T$? In other words, weigh the relative importance of the supervised task objective vs. LM objective? \n\nLLaMA is less effective than RoBERTa in several cases. Could the authors give practical guidelines on which model & ICL technique to employ for each model class?\n\nAre the authors planning to share their code?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments, helpful suggestions and appreciation of our work!\n\n#### **comment #1**:\nThe first contribution is over-claimed.\n\n#### **response #1**:\nThanks for your suggestion! We are sorry about the confusion, and we would like to clarify and revise the statement \"We take the first step toward in-context domain adaptation\". To clarify, the previous approaches (Domain adaptive pretraining) leverage the masked language modeling to learn the target distribution. However, we present our methods with the discussion of in-context learning, and emphasize the idea of learning a model with the retrieved semantically-rich and demonstrative contexts. Moreover, we learn the target distribution from a source-target joint view (we mix the source and target, and the model learns to fuse the distribution and bridge the domain gap). In contrast, previous works only pretrain from a single view (target only). \n\nThen, we would like to draw your attention that we have compared our methods with the previous language modeling based UDA methods in Section 4.4 Analysis. We discuss the detailed differences between the two methods. Our proposed adaptive ICL learns the task discrimination and target distribution simultaneously with an in-context manner, however, the previous LM-based UDA learns these two objectives in two steps: first, pretraining on the target domain with MLM objective, and second, supervised training on the source domain. We conduct experiments on LLaMA and find that the proposed adaptive ICL outperforms adaptive pretraining.\n\nFollowing your suggestion, we will rewrite our first contribution to avoid the misunderstanding and over-claiming. \"We propose domain adaptive in-context learning in which we mix the source input and semantically-rich target contexts to learn two in-context objectives simultaneously.\"\n\n#### **comment #2**:\nFor LLaMA experiments, it is not clear that proposed method consistently boost the performance.\n\n#### **response #2**:\nThis is a good suggestion. Due to the cost of LLaMA inference computation (it may require 30 hours per task to get the evaluation result on a single A100, 6k sentences for inference), we only perform a single run. To address your concern, we conduct 5 runs for LLaMA-LoRA ICL-retr on WNUT-16 dataset, and we get the mean $46.39$ and standard deviation $0.7$, and this result is significantly stronger than the baseline model with $p < 0.05$.\n\n#### **comment #3**:\nFor encode-only models, the retrieved target examples could be placed before the source input.\n\n#### **response #3**:\nThis is a good suggestion! Intuitively, putting the target before or after the source both work in our framework. Following your advice, we conduct initial experiments on NER BC2GM dataset and find that the two approaches make slight differences in performance ($32.51\\pm 1.1$ and $32.39\\pm 1.3$). We would like to further run more experiments, and if the results remain the same, we are willing to include this prompt in our paper.\n\n#### **comment #4**:\nGiving supervised task objective and LM objective different weight different weight.\n\n#### **response #4**:\nThank you for your good suggestion. We will consider this as our future work.\n\n#### **comment #5**:\nCould the author give practical guidelines of which model and ICL technique to employ?\n\n#### **response #5**:\nFirst, for NER, LLMs exhibit poor performance in such sequence tagging task in our experiment, such phenomenon has also been discussed in recent evaluation papers [1]. So we would recommend to employ small expertise model such as Roberta + CRF to achieve the SOTA results. \n\nSecond, for SA, Chatgpt zero-shot inference (no demo) has already surpassed the performance of fine-tuned Roberta. However a fine-tuned LLaMA can outperform Chatgpt, which indicates fine-tuning is still beneficial in the era of LLMs.\n\nThird, our method ICL-retr can outperform the baselines with a large margin in most cases, indicating that retrieval is consistently advantageous for Roberta and LLaMA. However, retrieval does not yield positive results for ChatGPT. Interestingly, even random demonstrations can lead to non-trivial performance gains, and we discuss this observation in our paper starting from the line 444. One possible explanation is that when we have to search for out-of-distribution demonstrations for LLMs, providing diverse and distinct examples is more beneficial for the model to understand the task and generalize under UDA setting.\n\n#### **comment #6**:\nAre the authors planning to share their code?\n\n#### **response #6**:\nYes, we will release the code upon acceptance.\n\nThanks for your appreciation again!\n\n[1] Is ChatGPT a General-Purpose Natural Language Processing Task Solver? Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, Diyi Yang", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper shows that large language models, including ChatGPT have subpar performance, especially on NER and demonstrates practical ways to boost their accuracy both in inference-only and fine-tuning setups. Extensive experiments across datasets demonstrate the effectiveness of exemplar retrieval and further fine-tuning when possible. ", "reasons_to_reject": "It is not clear which part of the paper's first claimed contribution (\"To the best of our knowledge, we take the first step towards in-context domain adaptation, which takes advantage of LM architectures to learn target domain distribution while adapting in contexts\") is novel since there exist already approaches (mentioned in related work), which tackle UDA via LM objectives. I would suggest clarifying the paper's contribution specifically compared to previous work on LM-based UDA approaches. \n\nFor \"LLamMA-LoRA\", it is not clear whether any of the proposed methods lead to statistically significant performance improvements. It is concerning that different \"ICL-*\" setups do not consistently boost performance compared to \"No-ICL\".\n\n", "questions_to_authors": "Even for encode-only models the target example could be placed before the source example in the prompt. This could help reduce the gap between the solutions for encoder-only and decoder-only models. Did the authors consider this?\n\nIn decoder-only training (Eq. 5) would it help to weight  the prediction of $y$ higher than the prediction of $x^T$? In other words, weigh the relative importance of the supervised task objective vs. LM objective? \n\nLLaMA is less effective than RoBERTa in several cases. Could the authors give practical guidelines on which model & ICL technique to employ for each model class?\n\nAre the authors planning to share their code?", "paper_topic_and_main_contributions": "The paper proposes methods for unsupervised domain adaptation of language models via in-context learning. Different methods are developed for encoder-only and decoder-only method under an inference-only and a fine-tuning setup. The main idea under the inference-only setup is to augment target unlabeled examples with retrieved demonstrations from the source domain. The main idea under the fine-tuning setup is to augment source examples with semantically similar (unlabeled) examples from the target domain and fine-tune the model with language modeling and in-context task learning objectives. The proposed methods are evaluated on several datasets for NER and sentiment analysis.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y4pdlzXAnu", "reviewer": "Reviewer_PvYw", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! \n\n#### **comment #1**:\nLLMs are not suitable for domain adaptation research: \n- (1) LLMs lack a clear criterion to define in-distribution, we should not rely on criterion that distinguish the source and target as in traditional UDA using SLMs. \n- (2) Conducting inference experiments with LLMs is irrelevant to domain adaptation.\n\n#### **response #1**:\nWe think domain related problem is still worth studying in the era of LLMs. Below are our justifications:\n\n- (1) **We focus on fine-tuning instead of inference-only.** Unlike what the reviewer mentioned, inference-only experiments with LLMs are not the focus of this paper. Our main contribution lies in providing a unified framework with effective prompting and finetuning strategies for in-context domain adaptation accounting for different LM architectures. The reason for conducting inference-only experiments is to prove that fine-tunig is still beneficial for unsupervised domain adaptation compared with inference-only LLMs. In NER experiments, ChatGPT achieves very low performances, but fine-tuning a much smaller RoBERTa model achieves SOTA scores in most adaptation scenarios. In Sentiment Analysis experiments, fine-tuning LLaMA with even fewer trainable parameters (1.7M, Trainable parameters only account for 0.24\\% of the entire LLaMA-7B parameters) outperforms all the other methods. Hence, we hypothesize that even though LLMs have strong generalization ability, they cannot tackle problems in all domains. When it comes to UDA, designing an effective adaptation strategy (fine-tuning) is still promising, which is the goal of this work.\n\n- (2) **We can still distinguish source and target domains for inference using LLMs.** LLMs usually take a few input-output demonstrations as context (few-shot) to perform a task. However, in most real-life scenarios, we do not have such labeled in-distribution dataset (target domain, e.g., biomedical) to offer input-output demonstrations. As a result, we need to resort to other labeled datasets (source domain, e.g., social media) to construct those demonstrations. This has demonstrated better performances than not providing any demonstrations (zero-shot) as shown in Table 1 and 2. In such cases, the demonstrations (few-shot) come from the source domain and the target input to evaluate comes from the target domain.\n\n- (3) **Out-of-distribution demonstrations is worth studying for LLMs inference-only setting.** Since in most practical circumstances, we draw demonstrations from another labeled dataset with a different text distribution (e.g., changing topics and genres), thus we suffer from the out-of-distribution problem. Why this is worth studying? Previous studies [1] have shown that retrieval is always helpful under a single domain by selecting input-output demonstrations from the labeled in-distribution corpus (the train set). However, from our ChatGPT results, retrieval under the out-of-distribution setting fails, compared to randomly-sampled demonstrations. We make a hypothesis that providing diverse and distinct examples is more beneficial for LLMs to understand the task and generalize under the domain adaptation setting. While further in-depth experiments and studies are worthwhile studying under the OOD setting.\n\n- (4) **LLMs still face challenges with certain domains.** LLMs pretrianed over a huge corpus demonstrate generlizabilities towards a diverse set of tasks. They can be viewed as mastering fundamental knowledge, but that does not necessarily indicate they can perform well on any domain. There still exist many domains which are not frequently seen in the pretrained corpus, especially under real scenarios considering company-owned datasets. Hence, it is still crucial and necessary to investigate domain adaptation in the era of LLMs.  \n\n#### **comment #2**:\n- Utilizing basis loss functions for label prediction and language modeling is similar to previous works, similar methods have been attempted in various task before. \n- And the proposed method does not seem to demonstrate significant differences when comparing to these previous works, while the proposed approach comes with some noticeable drawbacks, such as long input length and an additional third-party retrieval module.\n\n#### **response #2**:\nWe provide the following justifications to address some misunderstandings:\n- First, **our method is not a simple utilization of two basis losses.** we present our methods with the discussion of in-context learning, we emphasize the idea of learning a model with the retrieved semantically-rich and demonstrative contexts. Besides, our approach is different from previous approaches (domain adaptive pretraining), which learns these two objectives in two steps: first, pretraining on the target domain with MLM objective, second, supervised training on the source domain. In contrast, our proposed adaptive ICL learns the task discrimination and target distribution simultaneously with an in-context manner, and we learn target distribution from a source-target joint view (we mix the source and target as context, the model learns to fuse the distribution and bridge the domain gap). Yet the previous works only pretrain from a single view (target only). It would be great if the reviewer could provide us some references that you think is similar to ours.\n\n- Second, in our paper, we present a unified framework with efficient prompting and fine-tuning strategies accounting for different LM architectures. **Specifically for the decoder-only model (LLaMA), our method is very different from previous works (encoder-only models, BERT).**\n\n- Third, in our experiment, **we have demonstrated that our method can surpass the baselines with a large margin.** We also conduct analysing experiments to compare our methods with the previous language modeling based UDA methods. We discuss the detailed differences between the two methods. We conduct experiments on LLaMA and find that the proposed adaptive ICL outperforms adaptive pretraining.\n\n- Fourth, **input length and the additional retrieval module should not be considered as drawbacks.** In the NLP community, both in-context learning and retrieval augmentation have garnered significant attention and discussion due to their potential to provide semantically-enriched contexts and exceptional performances. And it has become a common practice to use in-context demonstrations with the help of a retrieval module on various applications. Therefore we believe that exploring these aspects enhances the scope of domain adaptation and aligns with current trends in the field. Besides, we would like to clarify the effectiveness of our retrieval module. Given a corpus, building the indexes and search for top-k relevant sentences for 1k instances cost less than two minutes.\n\nThanks for your comments again!\n\n[1] What Makes Good In-Context Examples for GPT-3? Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Conducted rigorous experiments using diverse language models with varying methods.\n- The proposed methodology generally demonstrates better results compared to simply adding random samples from various datasets.", "reasons_to_reject": "- First of all, I am not entirely sure if large language models (LLMs) are suitable for domain adaptation research. Domain adaptation research aims to maintain the performance achieved by training on a source dataset when applied to a target distribution. However, when using LLMs only for inference, there are some issues. LLMs lack a clear criterion to define in-distribution (source and target distribution). Models like ChatGPT and LLAMA are already considered general-purpose language models, assuming they have been trained to some extent on both source and target domains. Therefore, when utilizing LLMs for domain adaptation, more thoughtful or convincing justifications are required rather than relying on methods that distinguish source and target as in traditional domain adaptation using smaller models. Similarly, conducting experiments with LLMs for inference only, without training them, appears irrelevant to domain adaptation research since there is no explicit learning for the source distribution.\n\n- The methodology lacks novelty and significance. Utilizing basic loss functions for label prediction and language modeling, it does not present an exceptionally innovative approach, as similar methods have been attempted in various tasks before. Furthermore, the proposed method does not seem to demonstrate significant differences when compared to using these methods alone without incorporating ICL. While the proposed approach comes with some noticeable drawbacks, such as long input length and an additional third-party retrieval module, its merits appear to be limited in comparison.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper conducted research to improve the performance of domain adaptation using in-context learning. Specifically, the proposed method involves finding and utilizing samples similar to the input through the process.\nFirstly, the input sentence and k semantically similar sentences are retrieved using third-party models (e.g., SimCSE). Next, the retrieved sentences are combined with the target input, and two types of losses are utilized: the basic loss for label prediction and the language modeling loss. The model was experimented with in NER and sentiment analysis research, and it demonstrated better performance compared to using the conventional ICL method.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents leverages in-context learning to adapt language models from the source domain to the target domain in the absence of target labels.  The paper presents various prompting and training strategies and shows on two tasks (Sentiment Analysis and NER) that the proposed approach works very effectively compared to existing baselines. The use of the ICL is very well-motivated and carefully done.  The authors are strongly advised to further polish the writing and reflect on the points (which are now clarified based on the discussion phase).
Decision: Accept-Main