Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features
Authors: Zihan Wang
Ziqi Zhao
Zhumin Chen
Pengjie Ren
Maarten de Rijke
Zhaochun Ren
Venue: EMNLP 2023 Findings (2023)
Keywords: Named entity recognition
Few-shot learning
Domain generalization
Prompt learning

Abstract: Few-shot named entity recognition (NER) has shown remarkable progress in identifying entities in low-resource domains. However, few-shot NER methods still struggle with out-of-domain (OOD) examples due to their reliance on manual labeling for the target domain. To address this limitation, recent studies enable generalization to an unseen target domain with only a few labeled examples using data augmentation techniques. Two important challenges remain: First, augmentation is limited to the training data, resulting in minimal overlap between the generated data and OOD examples. Second, knowledge transfer is implicit and insufficient, severely hindering model generalizability and the integration of knowledge from the source domain. In this paper, we propose a framework, prompt learning with type-related features (PLTR), to address these challenges. To identify useful knowledge in the source domain and enhance knowledge transfer, PLTR automatically extracts entity type-related features (TRFs) based on mutual information criteria. To bridge the gap between training and OOD data, PLTR generates a unique prompt for each unseen example by selecting relevant TRFs. We show that PLTR achieves significant performance improvements on in-domain and cross-domain datasets. The use of PLTR facilitates model adaptation and increases representation similarities between the source and unseen domains.

Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features
Zihan Wang Ziqi Zhao', Zhumin Chen', Pengjie Ren', Maarten de Rijke² and Zhaochun Ren 'Shandong University, Qingdao, China
University of Amsterdam, Amsterdam, The Netherlands
Leiden University, Leiden, The Netherlands
{zihanwang, sdu, ziqizhao work) com, chenzhumin@sdu..cm edu. cn jay. woy.reeeoutooko com, m. wedi nl, z. ren@l iacs. leidenuniv.ni
Abstract
its wide range of uses, such as question answering and document parsing, serving as erucial com- ponent in natural language understanding (Nadcau and Sekine, 2007; Ma and Hovy, 2016; Cui and Zhang, 2019; Yamada et al.. 2020). The availabil- ity of labeled data for NER is limited to specific domains, leading to challenges for generalizing models to new domains (Lee et al., 2022; Cui al., 2021; Ma al., 2022).
Few-shot named entity recognition (NER) has shown remarkable progress in identifying en- tities in low- low-resource domains. However, few- shot NER methods still struggle with out-of- domain (OOD) examples due to their reliance on manual labeling for the target domain. To address this limitation, recent studies enable generalization to an unseen target domain with only a few labeled examples using data augmen- tation techniques. Two important challenges remain: First, augmentation is limited to the training data, resulting in minimal overlap be- tween the generated data and OOD examples. Second, knowledge transfer is implicit and in- sufficient, severely hindering model generaliz- ability and the integration of knowledge from the source domain. In this paper, we propose framework, prompt learning with rype-related features (PITR), to address these challenges. To identify useful knowledge the source do- main and enhance knowledge transfer, PLTTR automatically extracts entity type-related fea- tures (TRFs) based on mutual information cri- teria. To bridge the gap between training and OOD data, PI.TR generates a unique prompt for each unseen example by selecting relevant TRFs. We show that PLTR achieves signifi- cant performance improvements in-domain and cross-domain datasets. The use of PLTR facilitates model adaptation and increases rep- resentation similarities between the source and unseen domains."
To overcome this issue, recent research focuses on enabling models to effectively learn from few labeled examples in new target domains (Lee et al., 2022: Ma et al., 2022; Das et al., 2022; Chen et al., 2022a; Wang et al., 2022, 2023) or on exploring data augmentation techniques, leveraging automati- cally generated labeled examples to enrich the train- ing data (Zeng et al., 2020). However, these meth- ods still require manual labeling for target domains, limiting their applicability zero-shot scenarios with diverse domains.
Recently, Yang et al. (2022) have explored a new task, few-shot cross-domain NER, aiming to generalize an entity recognizer to unseen target do- mains using small number of labeled in-domain examples. To accomplish this task, data augmen- tation technique, named FactMix, has been devised. FactMix generates semi-fact examples by replacing the original entity or non-entity words in training instances, capturing the dependencies between en- tities and their surrounding context. Despite its success, FactMix faces two challenges:
1 Introduction
Augmentation is limited to the training data. Since the target domain is not accessible during training, FactMix exclusively augments the train- ing data from the source domain. As result, there is minimal overlap between the generated exam- ples and the test instances both the entity and context levels. For instance, only 11.11% the entity words appear simultaneously in both the gen- erated data (by FactMix) and the Al dataset (target domain). At the context level, as demonstrated
Named entity recognition (NER) aims to detect named entities in natural languages, such lo- cations, organizations, and persons, in input text (Zhang et al., 2022; Sang and Meulder, 2003; Yang et al., 2017). This task has gained significant at- tention from both academia and industry due to
*Corresponding author.
!Our ende is available at petps://giihhe com/ WZH-NLP/PLTR.

---

0.25
Cross-domain
Similarity
0.2
FactMix
Science
In LaHaye helped establish Institute for Creation Research (ORG) at San Diego Christion College in Cajon, along Henry Momis (PER).
0.15
In-domain
[unseen]
ORG: The fledgling company established ground floor office here over the last two weeks, has received venture Park.
0.05
CONL12003 [source)
financing
0
AI
Music
Science
PER: Along with Mayfiar at 277 for the tournament were Stricker, had and Duffy Walderf, 66.
Datasets
(a) Average similarities between pains of sentences.
(b) Type-related Teatures the source domain.
Figure I: (a) Average SBERT similarities (Reimers and Gurevych, 2019) between pairs of sentences that contain the same type of entities. The source domain dataset is CoNLL2003 (Sang and Meulder, 2003); the target domain datasets include AL, Music, and Science (Liu et al., 2021). the "Cross-domain" setting, one sentence is from the source domain and the other is from the target domain. In the "FactMix" setting, one sentence is from augmented data by FactMix (Yang et al., 2022), and the other is from the target domain. In the "In-domain" setting, both sentences are from the target domain. (b) Examples of type-related features in the source domain.
in Fig. I(a), the average sentence similarity be- tween the augmented instances and the test exam- ples is remarkably low. These gaps pose severe challenges in extrapolating the model to OOD data. To address this problem, incorporate natural language prompts to guide the model during both training and inference processes, mitigating the gap between the source and unseen domains.
to effectively leverage knowledge from the source domain and bridge the gap between training and un- seen data, As Fig. 2 shows, PLTR is composed of two main phases: (i) type-related feature extraction, and (ii) prompt generation and incorporation. To identify valuable knowledge in the source domain, PLTR uses mutual information criteria extract entity type-related fcatures (TRFs). PLTR imple- ments a two-stage framework to mitigate the gap between training and OOD data. Firstly, given a new example, PLTR constructs unique sequence by selecting relevant TRFs from the source domain. Then, the constructed sequences serve prompts for performing entity recognition on the unseen data. Finally, a multi-task training strategy is em- ployed to enable parameter sharing between the prompt generation and entity recognition. Similar to FactMix, PLTR is a fully automatic method that does not rely on external data or human interven- tions. PLTR is able to seamlessly integrate with diffcrent few-shot NER methods, including stan- dard fine-tuning and prompt-tuning approaches.
Knowledge transfer implicit and insufficient. Intuitively, better generalization to unseen domains can be accomplished by incorporating knowledge from the source domain (Ben-David et al., 2022). However, in FactMix, the transfer of knowledge from the source domain occurs implicitly the rep- resentation level of pre-trained language models. FactMix unable to explicitly identify the type- related features (TRFs), i... tokens strongly asso- ciated with entity types, which play a crucial role in generalization. E.g., as illustrated in Fig. 1(b), the words "established" and "along with" exhibit a close relationship with organization and person entities, respectively, in both domains. This knowl- edge can greatly assist in recognizing organizations and persons in the target domain.
In summary, our contributions are: (i) to the best of our knowledge, ours is the first work study prompt learning for few-shot cross-domain NER: (ii) we develop a mutual information-based approach to identify important enttty type-related features from the source domain; (iii) we design two-stage scheme that generates and incorpo- rates prompt that is highly relevant to the source domain for each new example, effectively mitigat- ing the gap between source and unseen domains; and (iv) experimental results show that our pro- posed PLTR achieves ttat-of--hhe-ar performance on both in-domain and cross-domain datasets.
To tackle this limitation, we introduce mutual information criteria to extract informative TRF's from the source domain. Furthermore, we con- struct a unique prompt for each unseen instance by selecting relevant TRFs. Intuitively, these gener- ated prompts serve distinctive signatures, linking unfamiliar examples to the knowledge within the source domain.
Contributions. this paper, present frame- work, named prompt learning with rype-related features (PLTR) for few-shot cross-domain NER,

---

Type-related Feature Extraction
Prompt generation and incorporation
Person
PLM-based recognizer
Generated promot with TRFs (SEP) PER
Netanyahu and Levy's spakesmen said they could not confirm It.
Spakesmen (SEP) ORG: News, Corp|SEP]
[CLS] Bolton 's spooessperson told CBS News. (SEP] type related features: [MASK] (MASK)...
Organization
Input sentence
Prompt template
Seat posted 3,420 registrations compared
with 5522 reglstrations in July a year earller.
Named entities
PLM-based recognizer
Bolton CBS News ORG
Mutual Information
[CLS] Bolton's spokesperson told CBS [SEP] PER Spokesmen [SEP] ORG: Corp]SEP]...
TRF
PER: (spokesmen,
Input sentence
Generated prompt TRFs
ORGI( posted, registrations...
Prompt goneration
Prompt Incorporation
Figure 2: An overview PLTR. PLTR has two main phases: (i) type-related feature extraction, and (ii) prompt generation and incorporation. Besides, we utilize a multi-task training strategy to enahle parameter sharing hetween prompt generation and incorporation.
2
Related work
learning techniques with prompts to better capture label dependencies. Furthermore, Ma al. (2022) develop a template-free approach to prompt NER, employing an entity-oriente objective. Recently, several studies have conducted analyses of the per- formance of current large language models (LLMs), such as the GPT series (Brown et al., 2020; Ope- nAI, 2023), in the context of the few-shot NER task (Gutierrez et al., 2022; Han et al., 2023; Sun et al., 2023). Nevertheless, these investigations have revealed substantial performance gap be- tween recent LLMs and state-of-the-ar methods, Consequently, due to their high running costs and underwhelming performance, we do not consider recent L.MSs as the basic model of our proposed framework (refer to Sec. 3.2). As mentioned in Sec. 1, previous few-shot NER methods primarily focus on in-domain settings and require manual annotations for the target domain, which poses challenge for generalizing OOD examples.
Cross-domain NER. The task of cross-domain NER aims to transfer NER models across diverse text styles (Pan et al., 2013; Liu et al., 2021; Chen al., 2021; Lee al., 2018; Yang al., 2017; Jia et al., 2019; Jia and Zhang, 2020; Zheng et al.. 2022: Zhang et al., 2022; Hu al., 2023; Wang et al., 2021). Yang et al. (2017) train NER models jointly in the source and target domains, while Jia et al. (2019) and Jia and Zhang (2020) leverage lan- guage models for cross-domain knowledge transfer. Zhang et al. (2022) introduce modular learning approach that decomposes NER into entity span de- tection and type classification subtasks. However, these methods still rely on NER annotations or raw data in the target domain.
Few-shot NER and prompt-based learning. Few- shot named entity recognition (NER) is the task of identifying predefined named entities using only a small number of labeled examples (Wiseman and Stratos, 2019; Yang and Katiyar, 2020; Das et al., 2022: Zeng et al., 2020; Ma et al.. 2023). Var- ious approaches have been proposed to address this task. For instance, Huang et al. (2021) inves- tigate the effectiveness of self-training methods on external data using distance-based approaches, where the label of the nearest neighbors is copied. Zeng et al. (2020) involves generating counterfac- tual examples through interventions to augment the original dataset. Additionally, prompt-base based learning, which has gained prominence in natural language processing, has also been applied to few- shot NER (Cui al., 2021; Ma al., 2022; Lee et al., 2022; Das et al., 2022; Chen et al., 2022b; Li et al., 2022; Dong et al., 2023; Fang et al., 2023). In particular, Das et al. (2022) incorporate contrastive
The field of few-shot cross-domain learning is inspired by the rapid learning capability of humans to recognize object categories with limited exam- ples, known as rationale-based learning (Brown et al., 2020; Shen et al., 2021; Chen et al., 2022a; Baxter, 2000; Zhang et al., 2020). In the con- text of NER, Yang et al. (2022) introduce the few- shot cross- cross-domain setting and propose two-step rationale-centri data augmentation method, named FactMix, to enhance the model's generalization ability.
In this paper, we focus on few-shot cross-domain NER. The most closely related work is Fact- Mix (Yang et al., 2022). FactMix faces two chal- lenging problems: (i) augmentation is limited to the training data, and (ii) the transfer of knowledge from the source domain is implicit and insufficient.

---

In our proposed PLTR, to identify useful knowl- edge in the souree domain, mutual information criteria are designed for automatic type-related fea- ture (TRF) extraction. In addition, PLTR generates a unique prompt for each unseen example based on relevant TRFs, aiming to reduce the gap between the source and unseen domains.
word embeddings derived from the pre-trained lan- guage models. To recognize entities, optimize the cross-entropy loss NER as:
LER = Σ Yo log (Po,e)
(2)
where N denotes the number of classes, y is a binary indicator (0 or 1) indicating whether the gold label e the correct prediction for observation o, and is the predicted probability of c for o.
3 Preliminaries
3.1 Task settings
A NER system takes a sentence x 21,.... In as input, where x a sequence of n words. It pro- duces sequence of NER labels y = ........ Mn- where each belongs to the label set y selected from predefined tags {Br, I,, St, E,, O}. The labels B, I, F., and S indicate the beginning, middle, ending, and single-word entities, respec- tively. The entity type is denoted by E (PER, LOC, ORG, MISC while denotes non-entity tokens. The source dataset and out-of- domain dataset are represented by Din and D ooo- respectively. Following Yang et al. (2022), we con- sider two settings in our task, the in-domain setting and the out-of-domain (OOD) setting. Specifically, we first train model Min using small of labeled instances from Dmn Then, for in-domain and OOD settings, we evaluate the performance of Man on Din and Docd: respectively.
Prompt-tuning method. The prompt-tuning method for NER tasks involves the use of mask- and-infill techniques based on numan-defined tem- plates to generate label words. We adopt the recent EntLM model proposed by Ma et al. (2022) our benchmark for this method. First, label word set Vi constructed through label word engineering, which is connected the label set using a mapping function M y - V₁. Next, entity tokens at entity positions are replaced with the corresponding label word M(;)). The resulting modified input is then denoted as Ent = [xI, .M (2:) ...... #..). The language model is trained by maximizing the probability P (xEnt x)). The loss function for generating the prompt and performing NER for- mulated as:
N
LER = log P (x:
Σ
24
Ent
x),
(3)
3.2
Basic models
Since our proposed PLTR is designed to model- agnostic, we choose two popular NER methods, namely standard fine-tuning and prompt-tuning re- spectively, our basic models. As mentioned in Sec. 2, due to their high costs and inferior perfor- mance on the NER task, we do not consider recent large language models (e.g.. GPT series) our basic models.
where N represents the number of classes. The ini- tial parameters of the predictive model are obtained from PLMs.
Method
In this section, we present the two primary phases of the proposed PLTR method, as depicted in Fig. 2: (i) type-related feature extraction (see Sec. 4.1), and (ii) prompt generation and incorporation (see Sec. 4.2).
Standard fine-tuning method. We employ pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) to generate contextualized word em- beddings. These embeddings are then input into a linear classifier with softmax function to predict the probability distribution of entity types. The process involves feeding the input token into the feature encoder PLM to obtain the corresponding contextualized word embeddings h:
4.1 Type-related feature extraction
As mentioned in Sec. 1, type-related features (TRFs), which are tokens strongly associated with entity types, play a crucial role in the few-shot cross-domain NER task. To extract these features, propose a mutual information based method for identifying TRFs from the source domain. Here. we define S, a set that contains all sentences from the source domain where entities of the i- th type appear, and S²,S as a set that contains
h PLM(x),
(I)
where h represents the sequence of contextualized

---

sentences without entities of the i-th type. In our method, we consider a binary variable that indi- cates examples (texts) from S; as 1, and examples from S,,S, as 0. To find tokens closely related to S₁. we first calculate the mutual information be- tween all tokens and this binary variable, and then select the top tokens with the highest mutual in- formation scores. However, the mutual information criteria may favor tokens that are highly associated with S S',; rather than with Sj. Thus, we introduce a filtering condition as follows:
formulate the selection of relevant TRF's as a cloze-style task for our PLM-based basic model Mg (refer to Sec. 3.2). Specifically, we define the following prompt template function f(-) with [MASK] tokens:
((x
"x[SPP e (5)
By inputting (x) into Ma, we compute the hidden vector lεмASKI of [MASK]. Given a token r E R, we compute the probability that token can fill the masked position:
Csr,s(Wm)) s. Cs.(wm)) O, (4)
(wm)
p [MMSS]] = rxf(()))
exp(r h(HAS))
(6)
ERER cxp(F h((wss))
where Cs, (Wm) represents the count of the m- gram Wm = -ppp-- in St Csy.s, (Wm) represents the count of this m-gram Wm in all source domains except for S; and is an m-gram frequency ratio hyperparameter. By applying this criterion, we ensure that w.. is considered part of the TRF set of S; only if its frequency in S; is sig- nificantly higher than its frequency in other entity types (S), Since the number of examples in S; is much smaller than the number of examples in S%, we choose 2 1 but avoid setting it to a large value. This allows for the inclusion of fea- tures that are associated with S; while also being related to other entity types in the TRF set of S,. In experiments, we set p and only consider 1 I-gram texts for simplicity.
where the embedding of the token r the PLM Mg- For each [MASK], we select the token with the highest probability as the relevant TRF for x, while discarding any repeating TRI's For example, as illustrated in Fig. 2, for the sentence "Bolton's ss spokesperson told CBS News.", the most relevant TRFs include "Spokesmen" "News" and "Corp". To train M₁ for TRF selection, define the loss function Egen as follows:
Lyen
(7)
log p( [MASK]: = ((xx)))((x),
DDm
BEDm
where ddx, i) denotes the label for the i-th [MASK] token in x. To obtain ((x), compute the Eu- clidean distance between the PLM-based embed- dings of each E and each token in x, select- ing the top-K features. Note that our designed automatic selection process effectively filters out irrelevant TRFs for the given input sentence, sub- stantially reducing human interventions in TRF ex- traction (refer to Sec. 7).
Note that the type-related feature extraction mod- ule we designed is highly efficient with a compu- tational complexity of O(D 11 [T),, where (Diol laog. and T represent the number of sen- tences in the training dataset, the average sentence length, and the entity type set, respectively. This module is able compute the mutual information criteria in Eq. 4 for all entity types in T and each token by traversing the tokens in every training sentence just once.
Prompt incorporation. tity type information into prompts, we generate a unique prompt given the selected relevant TRFs R'(x) C R for input This is achieved using the following prompt template function f'(x):
To incorporate the en-
4.2 Prompt generation and incorporation
To connect unseen examples with the knowledge within the source domain, we generate and incorpo- rate unique prompt for each input instance. This process involves two-stage mechanism: first, rel- evant TRFs are selected form prompts, and then these prompts are input into the PbMe basic model for entity label inference.
((x
(8)
"' (x)))SSPP.. (it.:tiJ.SSS....SEEEI..... (/s (m,4))
T is the entity type name (c.g.. PER ORG). Given sentence x, R'(x, C R'(x) represents selected TRFs related to entity type 1,. Note that, if R'(x, t4) = 6, the entity type name, and relevant TRFs R'(x, are ex- cluded from f'(x). For example, as depicted
where
Automatic type-related feature selection. Given an input sentence x and the extracted TRF set R,

---

Instances
Meulder, 2003). consists of text in style similar to Reuters News and encompasses entily types such as person, location, and organization. Additionally, to examine whether PLTR is extensible to different source domains and entity types, we evaluate PLTR using training data from OntoNotes (Weischedel et al., 2013) (refer to Sec. A.3). OntoNotes is an English dataset consisting of text from wide range of domains and 18 types of named entities, such Person, Event, and Date.
Dataset
Train
Dev
Test
Entity types
CON1112003
14,987
3,466
3,684
4
OntoNotes
59,924
8,528
8,262
18
TechNews
2,000
4
AI
431
14
Literature
416
12
Music
456
13
Politics
651
9
Science
543
17
Table Statistics the datasets used.
Out-of-domain datasets. We utilize the OOD dataset collected by Liu et al. (2021), which in- cludes new domains such as AI, Literature, Music, Politics, and Science. The vocabulary overlaps between these domains are generally small, indicat- ing the diversity of the out-of-domain datasets (Liu et al., 2021). Since the model trained on the source domain dataset (CoNLL2003) can only predict per- son, location, organization, and miscellaneous tities, we assign the label O all unseen labels the OOD datasets.
Fig. 2, the unique prompt (x) corresponding to X = "Bolton's 's spokesperson told CBS News." can be represented as follows:
(xx = "Bolton's spokesperson told CBS News.
SEEPPPERRSSyoleeemeee1555700 Corp".
Then, we imput fx) into Mr to recognize entities in the given sentence x.
4.3 Joint training
To enable parameter sharing between prompt gen- eration and incorporation, train model using multi-task framework. The overall loss function is defined as follows:
5.2 Experimental settings and baselines
We compare PLTR with recent baselines in the following two experimental settings:
Fine-tuning, Following Yang et al. (2022), we em- ploy the standard fine-tuning method (Ori) based on two pre- pre-trained models with different parameter sizes: BERT-basc, BERT-large, RoBERT-base, and RoBERT-aagee All backbone models imple- mented using the transformer package provided by Huggingface." For fine-tuning the NER models a few-shot setting, we randomly select 100 instances per label from the original dataset (CoNLL2003) to ensure model convergence. The reported per- formance of the models is an average across five training runs.
= a L'NeR + (1) gen::
(10)
where NER denotes the normalized loss function for the NER task loss CNR (refer to Sec. 3.2). a is the weight assigned to S'NER with prompts as inputs. The weight 1 - a is assigned to the loss function Lyen for type-related feature selection. In our experiments, we optimize the overall loss func- tion using AdamW (Loshchilov and Hutter, 2019). Sec. A.I gives the detailed training algorithm of PLTR.
Prompt-tuning. Similar to Yang et al. (2022), adopt the EntLM model proposed by Ma et al. (2022) the benchmark for prompt-tuning. The EntLM model is built on the BERT-base or BERT- large architectures. We conduct prompt-based ex- periments using a 5-shot training strategy (Ma et al., 2022). Additionally, we select two representative datasets, TechNews and Science, for the OOD test based the highest and lowest word overlap with the original training domain, respectively.
5 Experiments
We aim to answer the following research questions: (RQ1) Does PLTR outperform state-of-the-art fine- tuning methods the few-shot cross-domain NER task? (Sec. 6.1) (RQ2) Can PLTR be applied to prompt-tuning NER methods? (Sec. 6.2) Micro Fl is adopted as the evaluation metric for all settings.
5.1 Datasets
Detailed statistics of both in in-domain and out-of- domain datasets are shown in Table
Additionally, we include a recent data augmenta- tion method CF (Zeng et al., 2020) and the state-
In-domain dataset. We conduct in-domain ex- periments on the CoNLL2003 dataset (Sang and
Phttps://huggingface.com co/nodelss

---

In-domain Fine-tuning Results
on in-domain and cross-domain datasets, respec- tively. (ii) Few-shot cross- dross-domain NER is notably more challenging than the in-domain setting, as all methods obtain considerably lower Fl scores. The performance decay in TechNews is smaller than in other domains, due to its higher overlap with the training set. In summary. PLTR demonstrates its effectiveness in recognizing named entities from both in-domain and OOD examples. The use of type-related features (TRFs), along with the incor- poration of prompts based on TRFs, are beneficial for in-domain and cross-domain few-shot NER.
Backbone
Ori
CF
FactMix
PLTR
FasT--baee-csedd
54.03
77.71
80.10
82.05*
BERT-large large-cased
65.38
81.11
83.04
83.75*
RoBEKTa-base
48.53
82.74
85,33
86,40*
RoBERTa-lage large
65.70
85,20
86,91
88.03*
Table 2: In-domain fine-tuning results (Micro FI) on CoNLL2003, indicates statistically significant dif- ference (1-test, p<00.55 when compared to FactMix
of-the-art cross-domain few-shot NER framework FactMix (Yang et al., 2022) as baselines in both of the above settings. Note that, we report the re- sults of FactMix's 's highest- sighest-performinn variant for all settings and datasets.
6.2 Results on few-shot prompt-tuning (RQ2)
To explore the generalizability of PLTR, we report in-domain and OOD results for the prompt-tuning setting in Table 4 and 5, respectively. We obtain the following insights: (i) Due to data sparsity, the over- all performance for the prompt-tuning setting is considerably lower than the results of 100-shot fine- tuning. (ii) Even with only 5-shot training instances per entity type. PLTR achieves the highest perfor- mance and outperforms the state-of-the the-tar base- lines by significant margin, demonstrating the effectiveness and generalizability of PLTR. For ex- ample, in the in-domain and cross-domain datasets, PLTR achieves an average improvement of 11.58% and 18.2 18.24% over FactMix, respectively. In sum- mary, the PLTR framework not only effectively generalizes fine-tuning-based NER methods to un- seen domains, but also attains the highest Fl scores in the prompt-tuning setting.
5.3 Implementation details
Following Yang et al. (2022), we train all models for 10 epochs and employ an early stopping crite- rion based on the performance on the development dataset. The AdamW optimizer (Loshchilov and Hutter, 2019) is used optimize the loss functions. We use a batch size of 4, a warmup ratio of 0.1, and a learning rate of 2e-5. The maximum input and output lengths of all models are set to 256. For PLTR, we search for the optimal loss weight a from (0.1, 0.25, 0.5, 0.75, 0.9). The frequency ratio hyperparameter is set to 3 for all domains.
6 Experimental results
To answer RQ1 and RQ2, we assess the perfor- mance of PLTR on both in-domain and cross- domain few-shot NER tasks. This evaluation is conducted two settings: a fine-tuning setting with 100 training instances per type, and a prompt- tuning setting with 5 training instances per type.
7 Analysis
Now that we have answered our research ques- tions, take closer look at PLTR to analyze its performance. We examine whether the prompts are designed appropriately. Besides, we study how the number of training samples and selected type-related features influence the performance (Sec. A.2), how PLTR affects representation simi- larities between the source and target domains, and whether PLTR is extensible to different source do- mains and entity types (Sec. A.3). Furthermore, we provide insights into the possible factors that limit further improvements.
6.1 Results on few-shot fine-tuning (RQ1)
Table and 3 show the in-domain and cross-domain performance the fine-tuning setting, respectively. Based on the results, we have the following obser- vations: (i) PLTR achieves the highest Micro Fl scores for all datasets and settings, indicating its superior performance. For instance, when using RoBERTa-large as the backbone, PLTR achieves an 88.03% and 75, 14% Fl score the CoNLL2003 and TechNews datasets, respectively. (ii) PLTR sig- nificantly outperforms the previous state-of-the-arn baselines in both in-domain and cross-domain NER. For example, PLTR exhibits a 1.46% and 10.64% improvement over FactMix, on average,
A Ablation studies. To investigate the appropriate- ness of our prompt design, we conduct ablation studies on few-shot cross-domain NER in both fine- tuning and prompt-tuning settings. The results are presented in Table In the "NP" variant, prompts

---

OOD Fine-tuning Results
OOD Fine-tuning Results
Backbone
Dataset
Ori
CF
FactMix
PLTR
Dataset
Ori
CF
FactMix
PLTR
BERT-base-cased
41.46
61.20
65.20
67.39
10.46
19.33
19.49
23.86
BERT-large-case
52.63
67.51
69.98
70.51
12.00
19.64
19,97
27.84*
TechNews
Music
RoBERTa-buse
44.88
71.83
73.62
75.06
11.78
22.24
23.75
30.52*
RoBERTa-large
51.76
73.11
74.89
75.14
14.44
21.13
22.93
30.26*
BFRT-base-cased
15.88
22.49
24.67
28.41
21.38
41.84
43,60
44.97
BERT-large-cased
18.62
26.00
26.25
30.25
29.77
43.37
43,84
45.85*
AI
Politics
RoBERTa-buse
18.63
32.03
32.09
33,87
26.81
44.12
44.66
47.56*
RoBERTa-large
23.27
28.76
30.06
31.97
28.56
45.87
45.05
48.35*
BERT-base-cased
12.85
22.89
25.70
27.39
12.41
25.67
29.72
31.78*
BERT-large-cased
17.53
24.96
26.25
27.83
16.05
28.75
27.88
31.19*
Literature
Science
RoBERTa-buse
15.05
28,21
28.89
30.80
14.17
33.33
34.13
34,87
RoBERTa-large
19.20
25.43
26.76
31.02"
17.25
31.36
32.39
35.08
Table 3: OOD fine-tuning results (Micro FI) over six datasets. x indicates a statistically significant difference (I-test,
p<00.05) when compared FactMix.
In-domain Prompt-tuning Results
FITR (ours)
P. (OunS) Foorld
FooM
0.34
Backbone
EntLM
CF
FactMix
PLTR
07
bEse-base-cased 54.00
55.61
59.19
63,50*
0.5
BERT-large-cased 60.37
56.49
60.80
70.46*
0.666
Table 4: In-domain prompt-tuning results (Micro FI) on CoNLL2003. indicates a statistically significant difference (I-test, p<0..05 when compared FactMix.
0.64
100
02
100
Numbee of training Instances per typa
Number of training inatances per bype
(a) TechNews.
(b) Science.
are removed during test-time inference. In this case, the F1 scores across all datasets and settings suffer significant drop compared to our proposed PLTR. This demonstrates the crucial role of in- corporating prompts during both the training and inference processes. In the "RDW" and "REW" variants, prompts are constructed using randomly selected words from the source domain and the given example, respectively. The performance of both the "RDW" and "REW" model variants con- sistently falls short of PLTR, indicating that PLTR effectively identifies important knowledge from the source domain and establishes connections be- tween unseen examples and the knowledge within the source domain.
Figure 3: Influence of training instances on TechNews and Science (BERT-base).
baseline FactMix. These results indicate that PLTR effectively identifies relevant TRFs for OOD exam- ples, considerably mitigating human interventions in the feature extraction process.
The influence of training samples. To examine the impact of the number of training samples, we compare the performance of PLTR and FactMix few-shot cross cross-domain NER using 100, 300, and 500 training samples per entity type. Fig. dis- plays the results based on the BERT-base-cased model. PLTR exhibits the largest improvements over FactMix when the dataset comprises only 100 training instances per entity type, as opposed to the 300 and 500 training instances scenarios. Fur- thermore, PLTR consistently outperforms the prior lfte-off-the-ar approach, FactMix, across all exper- imental settings with varying numbers of training examples, demonstrating its superiority.
Additionally, to explore the efficacy of type- related feature selection (refer to Sec. 4.2), we con- ducted an evaluation of PLTR (BERT-base) using various frequency ratios (in Eq. 4). The results are presented in Table 7. As the value of p in- creases, TRFs extracted using Eq.4 become less closely associated with the specified entity type but become more prevalent in other types. When 9, observed
the value of p is raised from only a slight decrease the Fl scores of PLTR. When the value is raised to 20, the Fl score of PLTR drops, but still surpasses the state-of-the-an
Analysis of sentence similarities. In our analysis of sentence similarities, we investigate the impact of PLTR on the representation similarities between the source and target domains. We compute the

---

OOD Prompt-tuning Results
OOD Prompt-tuning Results
Backbone
Dataset
EntLM
CF
FactMix
PLTR
Dataset
EntLM
CF
FactMix
PLTR
BERT-base-cased
47.16
52.36
52.44
60.99*
15.70
18.32
18.62
20.90
TechNews
Science
BERT-large larg--lage-casee
52.53
48.32
48.64
61.64
15.32
15.34
16.80
19.77
Table 5: OOD prompt-tuning results (Micro F1) on TechNews and Science. * indicates statistically significant difference (1-test, p<0..5) when compared to FactMix.
OOD Fine-tuning Results
OOD Prompt-tuning Results
Dataset
FactMix
NP
RDW
REW
PLTR
FactMix
NP
RDW
REW
PL.TR
TechNews
65.09
66.16
66.01
66.10
67.39
52.44
54.01
55.90
56.46
60.99
Science
29.72
29.84
30.02
30.06
31.50
18.62
18.78
18.72
19.19
20.90
Table 6: Ablation studies on TechNews and Science.
Frequency ratio
Sentence length
Model
Dataset
3
9
Dataset
25
25-35
35
Avg.
FactMix
24.67
24.67
24.67
24.67
24.67
AI
In-domain
80.12
81.25
84.12
82.05
PLTR
28.41
26.36
26.61
26.42
25.70
Al
25.86
24.71
29.65
28.41
FactMix PLTR
Science
29.72
29.72
29.72
29.72
29.72
Science
23.86
29.62
32.87
31.78
31.78
30.07
30.11
30.58
29.91
Table 8: Error analysis on sentence lengths in test sets (BERT-base, fine-tuning).
Table 7: Influence of frequency ratio (p) on Al and Science (BERRTbase, fine-tuning).
Table 8. We observe that the FI scores of PLTR on sentences with more than 35 words ("> 35") are substantially higher than the overall Fl scores. In contrast, the Fl scores on sentences with 25 to 35 words ("25-35") or less than 25 words ("< 25") consistently fall below the overall F1 scores. This suggests that it may more challenging for PLTR to select TRFs and generate appropriate prompts with less context.
Cross-domain
FactMix
0.2
Similarity
(ours)
0.15
In-domain
0.1
0.05
0
Music
Science
Datasets
Figure Analysis of sentence similarities on AI, Music, and Science (BERT-base, fine-tuning).
Conclusions
In this paper, we establish new state-of-the-an framework. PLTR, for few-shot cross-domain NER. To capture useful knowledge from the source do- main, PLTR employs mutual information criteria to extract type-related features. PLTT automat- ically selects pertinent features and generates a unique prompt for each unseen example, bridging the gap between domains. Experimental results show that PLTR not only effectively generalizes standard fine-tuning methods unseen domains. but also demonstrates promising performance when incorporated with prompt-tuiing-baee approaches. Additionally, PLTR substantially narrows the dis- parity between in-domain examples and OOD in- stances, enhancing the similarities of their sentence representations.
average SBERT similarities for sentence represen- tations in PLTR (BERT-base) between the source and target domains; the results presented in Fig. With the prompts generated by PLTR, the representation similarities between the source and unseen domains noticeably increase. This is, PLTR facilitates a more aligned and connected represen- tation space, mitigating the gap between the source and target domains.
Error analysis. Although our proposed PLTR out- performs state-of-the- baselines, we would like to analyze the factors restricting further improve- ments. Specifically, compare the performance of PLTR (BERT-base) on sentences of different lengths in the test sets of the CoNLL2003 (In- domain), AI, and Science datasets. The results the standard fine-tuni tune-tunigg setting are provided in

---

Limitations
Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022. PADA: example-base based prompt learning for on-the-ffy fly adaptation to unseen domains. Trans. Assoc. Comput. Lingnistics, 10:414-433.
While PLTR achieves a new state-of-the-art perfor- mance, it has several limitations. First, the number of type-related features for prompt construction needs to be manually preset. Second, PLTR re- lies on identifying TRFs, which are tokens strongly associated with entity types. Extracting and incor- porating more complex features, such as phrases, represents promising direction for future research. In the future, we also plan to incorporate PLTR with different kinds of pre-trained language mod- els, such autoregressive language models.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
Ethics statement
Jiawei Chen, Qing Liu, Hongyu Lin, Xianpei Han, and Le Sun. 2022a. Few-shot named entity recognition with self-describing networks. In ACL. pages 5711- 5722.
The paper presents prompt-based method for rec- ognizing named entities in unseen domains with limited labeled in-domain examples. However, the constructed prompts and model-predieted results still have considerable amount of misinformation. Besides, the reliance on black-box pre-trained lan- guage models raises concerns. Hence, caution and further research are required prior to deploying this method in real-world applications.
Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2021. Data augmentation for cross- domain named entity recognition. In EMNLP, pages 5356.
Yanru Chen, Yanan Zheng, and Zhilin Yang. 2022b. Prompt-based metric learning for few-shot NER. CoRR, 0bbs/2211.04331
Acknowledgement
Leyang Cui, Yu Wu, Jian Liu, Sen Yang. and Yue Zhang. 2021. Template-based named entity recognition us- ing BART. In ACL-IJCNLP, pages 1835-1845.
This work was supported by the National Key R&D Program of China (2020YFB1406704
2022YFC3303004) the Natural Science Founda- tion of China (62272274, 61972234, 62072279, 62102234, 62202271), the Natural Science Foun- dation of Shandong Province (ZR2021QF12) ZR2022QF004), the Key Scientific and Techno- logical Innovation Program of Shandong Province (20191ZY010129), the Fundamental Research Funds of Shandong University, the China Schol- arship Council under grant nr. 202206220085, the Hybrid Intelligence Center, a 10-year pro- gram funded by the Dutch Ministry of Educa- tion, Culture and Science through the Netherlands Organization for Scientific Research, https:// hybrid-inteellgenne---o centre.eeeeeeeeeeeeeeee nl, and project LESSEN with project number WWA.1389.20. of the research program NWA ORC 2020/21, which is (partly) financed by the Dutch Research Council (NWO).
Leyang Cui and Yue Zhang. 2019. Hierarchically- refined label attention network for sequence labeling. In EMNLP-IJCNLP, pages 4115-4128.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J. Passonneau, and Rui Zhang. 2022. Container: Few- shot named entity recognition via contrastive learn- ing. In ACL, pages 6338 6353.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT, pages 4171-4186.
Guanting Dong, Zechen Wang, Jinxu Zhao, Gang Zhao, Daichi Guo, Dayuan Fu, Tingfeng Hui, Chen Zeng. Keqing He, Xuefeng Li, Liwen Wang, Xinyue Cui, and Weiran Xu. 2023. A multi-task semantic decom- position framework with task-specific pre-training for few-shot NER. CORR, abs/2308. 145333.
Jinyuan Fang, Xiaobin Wang, Zaiqiao Meng, Pengjun Xie, Fei Huang, and Yong Jiang. 2023. MANNER: A variational memory- augmeeedd model for cross domain few-shot named entity recognition. In ACL, pages 4261-4276.
References
Jonathan Baxter. 2000. model of inductive bias learn- ing. J. Artif. Intell. Res., 12:149 1299
Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and

---

Yu Su. 2022. Thinking about GPT-3 in-context leam- ing for biomedical IE? think again. In EMNLP, pages 4497-4512.
Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022. Template- free prompt tuning for few-shot NER. In NAACI, pages 5721-5732.
Ridong Han, Tao Peng, Chaohao Yang. Benyou Wang, Lu Liu, and Xiang Wan. 2023. Is information extrac- tion solved by ChatGPT? An analysis of performance, evaluation criteria, robustness and emrors. CoRR, abs/2305.114500
Xuezhe Ma and Eduard Hovy. 2016. End-to-end se- quence labeling via bi-directional LITM--CNN-CRRE In ACL, pages 1074.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvis- ticae Investigationes, 30(1):3-28
Jinpeng Hu, Dandan Guo, Yang Liu, Zhuo Li, Zhihong Chen, Xiang Wan, and Tsung-Hui Chang. 2023, A simple yet effective subsequence-enhhance approach for cross-domain NER. In AAAI, pages 12890-
OpenAl, 2023. GPT-4 technical report. abs/2303.08774.
CoRR,
12898.
Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2021. Few- shot named entity recognition: An empirical baseline study. In EMNLP, pages 10408-10423.
Sinno Jialin Pan, Zhiqiang Toh, and Jian Su. 2013. Transfer joint embedding for cross-domain named entity recognition. ACM Trans. Inf. Syst. 31(2):1-27.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using siamese BERT- networks. In EMNLP-IJCNLP" pages 3980-3990.
Chen Jia, Xiaobo Liang, and Yue Zhang. 2019. Cross- domain NER using ICLC pages 2464-2474.
language modeling.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-indern independent named entity recognition. In CoNI., pages 142-147.
Chen Jia and Yue Zhang. 2020. Multi-cell composi- tional LSTM for NER domain adaptation. In ACL, pages 5906-5917.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2021. Towards out-of-distribution generalization: survey. CoRR, abs/2108. 13624.
Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2022. Good examples make a faster learner: Simple demonstration-basse learning for low-resource NER. In ACL, pages 2687-2700.
Xiaofei Sun, Linfeng Dong, Xiaoya Li, Zhen Wan, Shuhe Wang. Tianwei Zhang, Jiwei Li. Fei Cheng. Lingjuan Lyu, Fei Wu, and Guoyin Wang. 2023. Pushing the limits of ChatGPT on NLP tasks. CoRR, abs/2306.0974
Ji Young Lee, Franck Dernoncourt, and Peter Szolovits. 2018. Transfer learning for named-entity recognition with neural networks. In LREC.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang. Zhi Zhou, Lan-Zhe Guo. Heli Qi, Zhen Wu, Yu-Feng Li. Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. 2022. USB: A unified semi-supervised learning benchmark for classifica- tion. In NeurIPS.
Dongfang Li, Baotian Hu, and Qingeai Chen. 2022. Prompt-based text entailment for low-resource named entity recognition. In COLING, pages 1896- 1903.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1997.11192
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang. Marios Savvides. Takahiro Shinozaki, Bhiksha Raj. Bernt Schiele, and Xing Xie. 2023. Freematch: Self-adaptive threshold- ing for semi-supervised learning. In ICLR.
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Zi- wei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. CrossNER: Evaluating cross- domain named entity recognition. In AAAI, vol- ume 35, pages 13452-13460.
Zihan Wang, Hongye Song. Zhaochun Ren, Pengjie Ren, Zhumin Chen, Xiaozhong Liu, Hongsong Li. and Maarten de Rijke. 2021. Cross-domain contract element extraction with a bi-directional feedback clause-element relation network. In SIGIR, pages 1003-1012.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Ruotian Ma, Zhang Lin. Xuanting Chen, Xin Zhou,
Junzhe Wang, Tao Gui, Qi Zhang, Xiang Gao, and Yun Wen Chen. 2023. Coarse-to-fine few-shot learn- ing for named entity recognition. ACL, pages 41 4129.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni- anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin,

---

and Ann Houston. 2013. OntoNotes release 5.0. Linguistic Data Consortium, Philadelphia, PA. LDC2013T19.
Algorithm 1 Training Algorithm for PLTR.
Require: The source dataset Din; the basic model Md with parameters θ; the frequency ratio pt the number of selected type-related features
Sam Wiseman and Karl Stratos. 2019. Label-agnostic sequence labeling by copying nearest neighbors. In ACL, pages 5363-5369.
K; the loss weight a; the number of epochs epoch.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self- attention. In EMNLP, pages 6442-644 6454.
Ensure: The extracted type-related features R. and
the trained basic model MY;:
1: Establish the basic model MMg
2: Initialize model parameters O;
Linyi Yang, Lifan Yuan, Leyang Cui, Wenyang Gao, and Yue Zhang. 2022. Factmix: Using few labeled in-domain examples to generalize to cross-domain named entity recognition. In COLING, pages 5360- 5371.
Extract type-related features R. for all entity types from the source dataset Din (Eq. 4); while S epoch do for Sample batch X Din do
for all sentences x E 2' do
Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In EMNLP, pages 6365- 6375.
7:
Select relevant TRFs R'(x) for in- put x (Eq. 5 and 6);
8:
9:
Transform x into the prompt tem- plate f'(x) (Eq. 8);
10:
Zhilin Yang. Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In ICLR.
11:
Input f'(x) into MM for prediction;
12:
end for
13: Update + by optimizing C (Eq. 10);
Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020. Counterfactual generator: A weakly- supervised method for named entity recognition. In EMNLP, pages 7270-7280.
14: end for
15:
end while
Xinghua Zhang, Bowen Yu, Yubin Wang. Tingwen Liu, Taoyu Su, and Hongbo Xu. 2022. Exploring modular task decomposition in cross-domal named entity recognition. In SIGIR, pages 301-311.
f'(x for each input x, and these prompts are then fed into M₁ for entity recognition (lines 8-9). Fi- nally, we iteratively refine the parameters θ by jointly optimizing two loss functions: the NER task loss function L'EER and the TRF selection loss function Lgen (line 11). Note that, during in- ference, PLTR generates unique prompt for each sentence within the unseen target domain using ex- tracted TRFs R. In this way, knowledge from the source domain is explicitly integrated into both the training and inference phases.
Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang. 2020. Every document owns its structure: Inductive text classification via graph neural networks. In ACL, pages 334-339.
Junhao Zheng, Haibin Chen, and Qianli Ma. 2022. Cross-domain named entity recognition via graph matching. In ACL, pages 2670-2680.
Appendix
A.2 Influence of the number of selected type-related features
A.1 Training algorithm of PLTR
Algorithm gives the detailed training algorithm PLTR. To start, we establish a basic model Ms based on Pre-trained Language Models (PLM) and initialize its parameters θ (lines 1-2). To capture knowledge from the source domain, PLTR iden- tifies type-related features using mutual informa- tion criteria (line 3). Next, given input sen- tence Di... PLTR automatically selects rele- vant TRI's (x) R by formulating the selection process as cloze-style task for M2 (line 7). Fur- thermore, to incorporate entity type information into prompts, PLTR constructs a unique prompt
We evaluate PLTR based on BERT-base in fine- tuning setting, with the number of selected relevant type-related features varying from 10 to 60. The results are shown in Fig. 5. Our observations in- dicate that as the number of type-related features increases, the performance (F1 score) of PLTTR ini- tially improves because the model incorporated with more features is able to encode more useful knowledge from the source domain. But notice that the performance drops when the number of type- related features is too large. our experiments, we set the number of type-related features to 40 on all

---

Source: CoNL.12003
Source: OntoNotes
Setting
Model
Dataset
PER
LOC
ORG
MISC
Avg.
PER
LOC
ORG
EVENT
Avg.
FactMix PLTR
85.65
59.45
59.31
24.66
65.20
56.96
16.06
41.84
44.11
OOD
TechNews
86.00
71.34
59.93
26.25
67.39
86.14
18.33
57.71
65.31
Fine-tuning
Results
FactMix PLTR
35.43
31.28
24.46
23.58
29.72
13.60)
15.84
22.66
3.77
17.15
Science
36.27
39.38
36.57
29.99
31,78
38.51
16.79
23.20
7.61
27.46
FactMix PLTR
82.88
55.05
39,82
16.12
52.44
77.66
16.20
38.92
53.19
OOD
TechNews
87,91
56.73
42.69
29.21
60,99
79.27
20.26
42.67
54,91
Prompt-tuning
Results
FactMix PLTRR
29.87
23.17
8.47
14.30
18.62
33.14
3.96
7.54
2.90
19.83
Science
38.51
23.19
10.46
14.33
20.90
34.98
11.83
17.27
6.67
22.32
Table 9: Influence source domains (BERT-base). TechNews, there are no annotations for "EVENT" entities.
PLTR (ours)
FactMix
0.3
0.26
0.24
Number of relevant type-reiated features
Figure 5: Influence of the number of selected rele- vant type-related features (K) on Al (BERT-base, fine- tuning).
datasets.
A.3 Inftuence of source domains
We explore the performance of our proposed PLTR when trained on data from different source do- mains, i.e., CoNLL2203 and OntoNotes. Results in both the fine-tuning and prompt-tuning settings are shown in Table 9, Our observations indicate that our proposed PLTR consistently outperforms FactMix when trained on different domains. For in- stance, PLTR achieves an average improvement of 5.42% and 4.22% over FactMix for "LOC" entities when using CoNLL2003 and OntoNotes as source datasets, respectively. This highlights PLTR 's ca- pacity to extend to various source domains and entity types.


Reviews:
[{"review_id": "OESlmchAxs", "reviewer": "Reviewer_NpEC", "summary": "", "questions": "Do you think it would be extensible to other more challenging datasets and more target categories?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1**: The statistical approach that was followed may vary quite a bit depending on the target categories. PER, ORG, LOC, and MISC are arguably some of the simplest classes in this category. In more complex technical domains, I am not sure it will be so straightforward to do the type-related feature extraction. In particular, the value of $\\rho$ = 3, I think it will change wildly depending on some challenging domains, which may require human intervention defeating the purpose.\n\n**Response**: Thank you for your comment. Our proposed framework isn't confined to specific entity types. For comparability, we followed the approach outlined in FactMix [1] in our experiments, which led us to choose PER, ORG, LOC, and MISC as our target entity types. This choice was made not for simplicity, but because these types are shared between the source and target domains. Our paper does not address unseen entity types in the target domain.\n\nThe type-related feature extraction module we designed is highly efficient with a computational complexity of $\\rm{O} (N_{train} \\cdot l_{avg} \\cdot |\\mathcal{T}|)$, where $N_{train}$, $l_{avg}$, and $\\mathcal{T}$ represent the number of sentences in the training dataset, the average sentence length, and the entity type set, respectively. This module can efficiently compute the mutual information criteria in Eq. 3 for all entity types in $\\mathcal{T}$ and for each token by traversing the tokens in every training sentence just once. \n\nAdditionally, the value of $\\rho$ has limited influence on our proposed PLTR framework. As demonstrated in Section 4.2, PLTR automatically selects relevant type-related features from the extracted feature set while effectively filtering out irrelevant ones. This mechanism renders our framework highly resilient to variations in the value of $\\rho$. The following Table 1 shows the influence of $\\rho$ on the Science dataset. We can easily find that PLTR is highly robust to the value of $\\rho$. We will include this analysis in our camera-ready version.\n\n| | $\\rho = 3$ | $\\rho = 5$ | $\\rho = 7$ | $\\rho = 9$ | $\\rho = 20$ | \n| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | \n| **FactMix** | 29.72 | 29.72 | 29.72 | 29.72 | 29.72 |  \n| **PLTR** | **31.78** | **30.07** | **30.11** | **30.58** | **29.91** | \nTable 1: Influence of $\\rho$ on Science dataset (BERT-base, fine-tuning).\n\nIn summary, our proposed type-related feature extraction component is highly efficient, fully automatic, and can be readily extended to accommodate more complex domains with a broader range of entity types.\n\n[1] Yang et al. Factmix: Using a few labeled in-domain examples to generalize to cross-domain named entity recognition. In COLING 2022.\n\n**Comment 2**: Prompt incorporation with larger number of domains will be even more challenging. With four domains, sure it is simpler. However with datasets like OntoNotes, I guess it will take a lot of tokens to give all the TRFs and classes. This may in turn cause lower performance probably?\n\n**Response**: Thanks for your comment. In prompt incorporation, the length of prompts is controllable through the hyperparameter K for Eq. 4 (please see Section 4.2 for details), and it is not determined by either the number of domains or the number of entity types. This parameter allows us to manage prompt length based on the specific requirements of different datasets. To further mitigate the length of prompts, when the selected type-related feature set is null, PLTR removes the corresponding type-related features from the prompts. Therefore, our approach is readily extended to datasets with a larger number of domains and entity types.\n \nAdditionally, as Figure 5 shows, our proposed PLTR framework maintains its ability to achieve state-of-the-art performance for few-shot cross-domain named entity recognition with different numbers of type-related features used in prompts.\n\n**Comment 3**: While I see why the authors mainly using FactMix's evaluation criteria,\n ld be used in more domains and more categories to properly evaluate the efficacy. Otherwise it is not very easy to objectively say that this method may induce good performance in real life.\n\n**Response**: Thank you for your comments. PLTR is evaluated across a diverse range of real-world datasets from six domains, including Politics, Science, Music, Literature, Artificial Intelligence, and Reuter News [2]. This extensive evaluation showcases the strong effectiveness and generalizability of our approach (please see Section 6 for details).\n\nThe availability of public datasets for evaluating few-shot cross-domain NER with a substantial number of entity types is currently limited. For example, if we were to use OntoNotes [3] to replace CoNLL03 as the source domain dataset, this would lead to only five shared entity types across source and target domains, namely Event, LOC, ORG, PERSON, and Product. In future work, we intend to construct more complex datasets with a broader range of entity types and address the challenges posed by unseen types in the target domains. \n\n[2] Liu et al. CrossNER: Evaluating Cross-Domain Named Entity Recognition. AAAI 2021.\n[3] Pradhan et al. Towards robust linguistic analysis using OntoNotes. CoNLL 2013.\n\n**Question 1**: Do you think it would be extensible to other more challenging datasets and more target categories?\n\n**Response**: Thanks for the question. PLTR can readily be extended to handle more challenging datasets and a broader range of target categories. This is supported by the following observations:\n\n1. Our designed type-related feature extraction component is highly efficient and not limited by specific entity types (see our response to Comment 1 for details).\n2. PLTR showcases the capability to automatically select relevant type-related features, without relying on external data or extensive human interventions  (see our response to Comment 1 for details).\n3. The prompt length in our PLTR is controllable and PLTR maintains its ability to achieve state-of-the-art performance with different numbers of type-related features (see our response to Comment 2 for details).\n4. Our experimental results demonstrate the effectiveness of PLTR across real-world datasets from six diverse domains (see our response to Comment 3 for details).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new approach that statistically tries to find the most optimal tokens to transfer knowledge from source to target domain. \n2. OOD prompt tuning results show nice improvement over FactMix.", "reasons_to_reject": "1. Although I really like the idea of finding \"type related features\" I have some genuine concerns regarding the technique. The statistical approach that was followed may vary quite a bit depending on the target categories. PER, ORG, LOC, and MISC are arguably some of the simplest classes in this category. In more complex technical domains, I am not sure it will be so straightforward to do the type-related feature extraction. In particular, the value of rho = 3, I think it will change wildly depending on some challenging domains, which may require human intervention defeating the purpose. (Addressed in rebuttal)\n\n2. Prompt incorporation with larger number of domains will be even more challenging. With four domains, sure it is simpler. However with datasets like OntoNotes, I guess it will take a lot of tokens to give all the TRFs and classes. This may in turn cause lower performance probably?\n\n3. While I see why the authors mainly using FactMix's evaluation criteria, I think these category of methods should be used in more domains and more categories to properly evaluate the efficacy. Otherwise it is not very easy to objectively say that this method may induce good performance in real life.", "questions_to_authors": "Do you think it would be extensible to other more challenging datasets and more target categories?", "paper_topic_and_main_contributions": "This paper introduces PLTR framework for few-shot cross-domain NER that extracts and leverages type related features based on some mutual information criteria. Basically from a source domain this technique tries to generate some type related features, which they later use in inference example to find best type words. These type words are further used to prompt the model in giving the output NER class. Overall the proposed methodology shows some performance improvement against current baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LDhajKUaei", "reviewer": "Reviewer_2X7Q", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1**: The paper is over-length with one figure on Page 9.\n\n**Response**: Thanks for pointing this out. The figure on page 9 (i.e., Figure 5) serves to illustrate the impact of the number of type-related features and is included in the appendix section. Our main text spans less than 8 pages, ensuring the paper's overall length is within the specified limits.\n\n**Comment 2**: The repository is non-anonymous (see the files in pycache), where I can see the name in the repo.\n\n**Response**: Thanks for bringing this to our attention. We apologize for any confusion caused. The names appearing in the pycache files are actually server usernames, not the real names of the authors. To be on the safe side, we have removed this information from the repository.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons To Accept:\n\n- Regarding the writing: The paper is well written and easy to follow.\n- A mutual information-based approach and an instance-based incorporation approach are designed to improve the approach.\n- The authors conducted sufficient results to support their claim.\n- The proposed approach achieved statistically promising results in the benchmarks.", "reasons_to_reject": "Unfortunately, the paper should be **desk rejected** since it violates two submission policies:\n\n- The paper is **over-length** with one figure on Page 9.\n- The repository is **non-anonymous** (see the files in __pycache__), where I can see the name in the repo.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "Paper Topic: NER, OOD Generalization.\n\nMain contribution (NLP engineering experiment): The paper proposes a new prompt learning framework for few-shot cross-domain NER to alleviate two limitations in this area: (1) augmentation limited in the source domain; (2) insufficient knowledge transfer. Experiments demonstrate the effectiveness of the proposed approach, where state-of-the-art results are achieved.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "OyKPyjNLv8", "reviewer": "Reviewer_vj1P", "summary": "", "questions": "1. Can you provide more details on the limitations or failure cases of the proposed framework? How does the framework perform on different types of data or scenarios?\n2. How does the proposed framework compare with other state-of-the-art few-shot cross-domain NER methods? Can you provide a more comprehensive evaluation of the framework's performance against other methods, such as GPT-4?\n3. Can you provide more information on the impact of different hyperparameters or design choices on the proposed framework's performance? How can readers best apply the framework to different scenarios?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1**: One potential weakness of this paper is that it does not compare its proposed framework with other state-of-the-art few-shot cross-domain NER methods. A more comprehensive evaluation of the proposed framework's performance against other methods would provide a better understanding of its strengths and limitations.\n\n**Response**: Thanks for your comments. Our paper centers around the task of few-shot cross-domain NER, which is different from both cross-domain NER and few-shot NER tasks. In our few-shot cross-domain NER task, no prior knowledge of the target domain is available during training (please refer to Section 3.1 for details). The missing references you provide do not focus on our task but either on few-shot NER or on cross-domain NER. As mentioned in [1] and demonstrated in Section 6, it is not trivial to adapt current prompt-tuning or standard fine-tuning NER methods to effectively address the few-shot cross-domain NER task. FactMix [1] and our PLTR outperform the standard fine-tuning method and EntLM [2] by a large margin.\n\nFactMix [1] is the first work to formulate the few-shot cross-domain NER problem and the latest baseline for this task. Our PLTR significantly outperforms FactMix across a broad range of real-world datasets spanning 6 diverse domains (please refer to Section 6 for details). This demonstrates the strong effectiveness of our proposed framework.\n\n[1] Yang et al. Factmix: Using a few labeled in-domain examples to generalize to cross-domain named entity recognition. In COLING 2022.\n[2] Ma et al. Template-free Prompt Tuning for Few-shot NER. In NAACL 2022.\n\n**Comment 2**: Another potential weakness is that the paper does not provide a detailed analysis of the impact of different hyperparameters or design choices on the proposed framework's performance. A more thorough analysis of these factors would help readers understand how to best apply the framework to different scenarios.\n\n**Response**: Thanks for your comments. Concerning design choices, we evaluate the PLTR framework based on two kinds of pre-trained models with different parameter sizes. PLTR yields the highest F1 scores for all datasets and different design choices concerning backbones (see Section 6). In the fine-tuning setting, PLTR based on RoBERTa-large surpasses other variants in most cases, while in the prompt-tuning setting, PLTR based on BERT-large achieves the best performance in the majority of cases (again, see Section 6).\n\nRegarding the hyperparameters, we analyze the influence of the number of type-related features. The proposed PLTR framework consistently achieves state-of-the-art performance for few-shot cross-domain named entity recognition with different numbers of type-related features (see Figure 5).\n\nOther important hyperparameters whose influence we did not include in the paper due to space constraints are the threshold $\\rho$ in Eq. 3 and the loss weight $\\alpha$ in Eq. 8. As the results in the following Table 1 show, PLTR automatically selects relevant type-related features from the extracted feature set and is highly robust to the threshold $\\rho$. As the results in Table 2 below reveal, the optimal loss weight $\\alpha$ in our experiments is 0.5. We will add these additional analyses to the camera-ready version of our paper.\n\n|       | $\\rho = 3$ | $\\rho = 5$ | $\\rho = 7$ | $\\rho = 9$ | $\\rho = 20$ | \n| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | \n| **FactMix** | 29.72 | 29.72 | 29.72 | 29.72 | 29.72 |  \n| **PLTR** | **31.78** | **30.07** | **30.11** | **30.58** | **29.91** | \nTable 1: Influence of $\\rho$ on Science dataset (BERT-base, fine-tuning).\n\n|   | $\\alpha = 0.1$ | $\\alpha = 0.25$ | $\\alpha = 0.5$ | $\\alpha = 0.75$ | $\\alpha = 0.9$ |\n| - | ----------- | ----------- | ----------- | ----------- | ----------- | \n| **AI** | 24.46 | 24.79 | **28.41** | 25.27 | 24.73 | \n| **Science** | 29.63 | 29.60 | **31.78** | 29.96 | 27.91 |\nTable 2: Influence of $\\alpha$ on AI and Science datasets (BERT-base, fine-tuning).\n\n\n**Comment 3**: The paper does not provide a detailed analysis of the computational complexity of the proposed method, which could be a concern for large-scale applications.\n\n**Response**:  Thanks for your comments. The type-related feature extraction module we designed is highly efficient, with a computational complexity of $\\rm{O}(N_{train} \\cdot l_{avg} \\cdot |\\mathcal{T}|)$, where $N_{train}$, $l_{avg}$, and $\\mathcal{T}$ represent the number of sentences in the training dataset, the average sentence length, and the entity type set, respectively. This module can efficiently compute mutual information criteria in Eq. 3 for all entity types in $\\mathcal{T}$ and for each token by traversing the tokens in every training sentence just once. Additionally, our designed prompts can be incorporated with pretrained language models in a plug-and-play manner. Therefore, our proposed PLTR is well-suited for large-scale applications. We will include this analysis of computational complexity in our camera-ready version.\n\n**Comment 4**: The paper does not compare the proposed method to more recent LLM-based NER methods.\n\n**Response**: Thanks for your comment. As mentioned in our response to Comment 1, we have indeed included recent LLM-based baselines in our experiments, such as FactMix and EntLM, which utilize the BERT and RoBERTa architectures. \n\nRecent studies [e.g., 3, 4] indicate that NER models solely based on LLMs, such as the GPT series, tend to significantly underperform when compared to models fine-tuned from smaller pre-trained language models in low-resource scenarios. Therefore, we do not consider these models as baselines in our experiments.\n\nFurthermore, GPT-NER [4] requires both a fine-tuned NER model and GPT-3 to effectively address cross-domain NER and few-shot NER. Consequently, it is unfair to directly compare GPT-NER with other fine-tuning or prompt-tuning approaches.\n\nBesides, GPT-4 is not publicly available and GPT-NER was published in May 2023 (the month before the EMNLP 2023 deadline), making it challenging to reproduce and adapt it to our task settings within the limited time frame.\n\n[3] Gutiérrez et al. Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. In EMNLP 2022.\n[4] Wang et al. GPT-NER: Named Entity Recognition via Large Language Models. In Arxiv 2023.\n\n**Question 1**: Can you provide more details on the limitations or failure cases of the proposed framework? How does the framework perform on different types of data or scenarios?\n\n**Response**: Thanks for your interesting questions. We would like to provide an analysis of errors and conduct fine-grained performance comparisons on different entity types, as presented in the following tables.\n\nIn terms of error analysis, we examine the framework's performance across different sentence lengths: sentences with less than 25 tokens (<25), sentences with 25 to 35 tokens (25-35), and sentences with more than 35 tokens (>35). The results in the following Table 3 illustrate that the performance of PLTR significantly decreases as sentence length becomes shorter. The reason may be that PLTR faces challenges in selecting relevant type-related features with less context information. Additionally, the results in Tables 4 and 5 demonstrate that PLTR achieves notably lower F1 scores on \"MISC\" types, likely due to the high diversity of \"MISC\" entity mentions.\n\nFor fine-grained comparisons on different entity types, as the following Tables 4 and 5 show, our proposed PLTR consistently outperforms the SOTA baseline FactMix, highlighting the superiority of our framework.\n\nWe appreciate your feedback, and we will incorporate the above-mentioned analyses into our camera-ready version.\n\n|   | <25 | 25 - 35 | >35 | Average\n| -|-|-|-|-|\n| **CoNLL03** | 80.12 | 81.25 | 84.12 | 82.05 |\n| **AI** | 25.86 | 24.71 | 29.65 | 28.41 |\n| **Science** | 23.86 | 29.62 | 32.87 | 31.78 |\nTable 3: Performance on different sentence lengths on CoNLL03, AI, and Science datasets (BERT-base, fine-tuning).\n\n|   | PER | LOC | ORG | MISC | Average |\n|-| ----------- | ----------- | ----------- | ----------- |  ----------- |\n| **FactMix** | 31.38 | 29.57 | 30.04 | 12.28 | 24.67 |\n| **PLTR** | **33.05** | **38.97** | **37.06** | **12.58** | **28.41** |\nTable 4: Fine-grained comparisons on AI datasets (BERT-base, fine-tuning).\n\n|   | PER | LOC | ORG | MISC | Average |\n|-|-|-|-|-|-| \n| **FactMix** | 35.43 | 31.28 | 24.46 | 23.58 | 29.72 |\n| **PLTR** | **36.27** | **39.38** | **36.57** | **29.99** | **31.78** |\nTable 5: Fine-grained comparisons on Science datasets (BERT-base, fine-tuning).\n\n**Question 2**: How does the proposed framework compare with other state-of-the-art few-shot cross-domain NER methods? Can you provide a more comprehensive evaluation of the framework's performance against other methods, such as GPT-4?\n\n**Response**: Thanks for your question. Please see our response to Comments 1 and 4.\n\n**Question 3**: Can you provide more information on the impact of different hyperparameters or design choices on the proposed framework's performance? How can readers best apply the framework to different scenarios?\n\n**Response**: Thanks for your questions. Please see our response to Comment 2. \n\n**About missing references**: Thanks for bringing these references to our attention. Both of the mentioned references focus on cross-domain NER or few-shot NER, that is, on tasks that differ from the problem setting addressed in our paper. Please see our response to Comments 1 and 4 for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction of a novel approach to few-shot cross-domain NER that leverages type-related features (TRFs) and unique prompts, which could inspire further research in this area.\n2. The demonstration of the effectiveness of the proposed framework on several benchmark datasets, which could serve as a baseline for future research in few-shot NER.", "reasons_to_reject": "1. One potential weakness of this paper is that it does not compare its proposed framework with other state-of-the-art few-shot cross-domain NER methods. A more comprehensive evaluation of the proposed framework's performance against other methods would provide a better understanding of its strengths and limitations.\n2. Another potential weakness is that the paper does not provide a detailed analysis of the impact of different hyperparameters or design choices on the proposed framework's performance. A more thorough analysis of these factors would help readers understand how to best apply the framework to different scenarios.\n3. The paper does not provide a detailed analysis of the computational complexity of the proposed method, which could be a concern for large-scale applications.\n4. The paper does not compare the proposed method to more recent LLM-based NER methods.", "questions_to_authors": "1. Can you provide more details on the limitations or failure cases of the proposed framework? How does the framework perform on different types of data or scenarios?\n2. How does the proposed framework compare with other state-of-the-art few-shot cross-domain NER methods? Can you provide a more comprehensive evaluation of the framework's performance against other methods, such as GPT-4?\n3. Can you provide more information on the impact of different hyperparameters or design choices on the proposed framework's performance? How can readers best apply the framework to different scenarios?", "paper_topic_and_main_contributions": "The paper specifically addresses the problem of few-shot named entity recognition (NER) in low-resource scenarios, where there is limited labeled data available for training models. The paper proposes a novel framework, PLTR, that leverages type-related features (TRFs) to generate unique prompts for each unseen example, thereby reducing the gap between the source and unseen domains. The paper's main contributions are the use of mutual information criteria to extract TRFs, the development of a two-stage framework for few-shot cross-domain NER, and the demonstration of promising performance on several benchmark datasets. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Wang et al. Gpt-ner: Named entity recognition via large language models.\n\n[2] Li et al. Prompt-based Text Entailment for Low-Resource Named Entity Recognition"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for few-shot cross-domain NER. It mines templates for prompt generation depending on entity types. The type-dependent prompts are shown to improve the effectiveness over several baselines, in particular, FactMix.

The reviewers have found the work interesting. There is not extensive work on few-shot cross-domain NER. This work enriches the literature on this topic.
The reviewers have questioned about the comparison with few-shot NER and cross-domain NER. The authors replied that the comparison is irrelevant. Nevertheless, it would be interesting to compare the proposed method with the existing few-shot methods by considering cross-domain as a single domain. This would show how an in-domain approach can or cannot be generalized to cross-domain scenario.

The proposed prompt mining approach can also be compared to template mining in traditional NER studies based on rules. There is a large body of research work on template mining, which the paper does not refer to. It would also be possible to compare with such traditional approaches in the experiments.
Decision: Accept-Findings