MANNER: A Variational Memory-Augmented Model for Cross Domain Few-Shot Named Entity Recognition
Year: 2023

Abstract:
This paper focuses on the task of cross domain few-shot named entity recognition (NER), which aims to adapt the knowledge learned from source domain to recognize named entities in target domain with only a few labeled examples. To address this challenging task, we propose MANNER, a variational memoryaugmented few-shot NER model. Specifically, MANNER uses a memory module to store information from the source domain and then retrieve relevant information from the memory to augment few-shot tasks in the target domain. In order to effectively utilize the information from memory, MANNER uses optimal transport to retrieve and process information from memory, which can explicitly adapt the retrieved information from source domain to target domain and improve the performance in the cross domain few-shot setting. We conduct experiments on both English and Chinese cross domain fewshot NER datasets, and the experimental results demonstrate that MANNER can achieve superior performance 1.

heading: 1 Introduction
text: Named Entity Recognition (NER) is a fundamental NLP task that aims at classifying mention spans into entity types. Previous works mainly study the NER task in a supervised setting (Chiu and Nichols, 2016; Devlin et al., 2019; Yamada et al., 2020). However, supervised learning requires largescale annotated datasets, which can be difficult to obtain in some scenarios (e.g., annotating biomedical named entities always requires domain expertise (Ogren et al., 2008)). In this paper, we focus on a more practical and challenging setting in realworld applications, namely cross domain few-shot NER — given a source domain with sufficient labeled data and a target domain with a few labeled
∗Work done when interning at Alibaba DAMO Academy. †Yong Jiang is the corresponding author.
1 Our code is publicly available at: https://github. com/Alibaba-NLP/MANNER
data, the goal is to correctly recognize named entities in the target domain (Hou et al., 2020). This is achieved by adapting the knowledge learned from the source domain to the target domain based on few-shot examples available in the target domain. Figure 1 provides an illustration of the cross domain few-shot NER task.
Recent work demonstrated that learning prototype representations for each label class could be effective to address few-shot tasks (Snell et al., 2017), and this idea has also been applied to fewshot NER tasks (Fritzler et al., 2019; Huang et al., 2021; Ma et al., 2022). Specifically, when dealing with a few-shot task in the target domain, these models learn prototypes for each entity type based on a few labeled data available in the support set and then assign labels to tokens in the query set by measuring their distances to the prototypes. However, since there are only a few labeled examples for each entity type in the support set, the prototypes obtained from the support set only may not be accurate and representative, leading to the suboptimal and unstable performance of prototype-based few-shot NER models (Huang et al., 2020).
To this end, we propose a variational MemoryAugmeNted cross domain few-shot NER model, abbreviated as MANNER. It introduces an external
4261
memory module that utilizes information from the source domain to augment the support set in the target domain, so as to learn more accurate prototypes. The basic idea of introducing the memory module is that the entity type information from the source domain can provide additional background knowledge for learning prototypes in the target domain (Zhen et al., 2020). For example, in Figure 1, the information of the entity type “Person” in the source domain can provide guidance for recognizing the entity type “Scientist” in the target domain. Specifically, MANNER stores token representations of entity types from the source domain in a memory module. For each entity type in the target domain, MANNER first retrieves the most similar entity types from the memory and then leverages the retrieved information to learn prototype for the entity type.
One critical issue when using the memory module is how to utilize the information from the memory to augment few-shot tasks in the target domain. Recent research indicates that the performance of memory-augmented methods which directly use neural networks to fuse information from the memory and the task (He et al., 2020; Zhen et al., 2020), is suboptimal when dealing with cross domain tasks (Du et al., 2022), such as our cross domain fewshot NER task. This is because the knowledge (i.e., entity type information) of the source domain stored in the memory can be inconsistent with that of the target domain. Therefore, in cross domain few-shot NER tasks where the entity types of the source domain and target domain are disjoint, directly utilizing the information retrieved from the memory may be suboptimal. Actually, we empirically found that this could degrade the model performance (see § 4.2). To address this problem, we take inspiration from domain adaption and leverage optimal transport (Villani, 2009) to retrieve and process information from the memory. One benefit of using optimal transport is that we can adapt the retrieved information from the source domain to the target domain via the optimal transport plan. This adaption process helps alleviate the inconsistency problem between the two domains.
Our contributions can be summarized as follows: (1) We propose MANNER, a novel cross domain few-shot NER model, which uses a memory module to utilize the information from the source domain to augment few-shot NER tasks in the target domain. (2) We leverage optimal transport to re-
trieve and process information from the memory, which is conducive to improve the performance of MANNER in the cross domain setting. (3) Experimental results on English and Chinese cross domain few-shot NER datasets demonstrate that MANNER can achieve superior performance compared with existing few-shot NER models.

heading: 2 Preliminaries
text: In this section, we formalize the cross domain fewshot NER task, and provide a brief introduction to optimal transport, which serves as the foundation of our model.
Task Formulation. NER is a sequence labeling task, where each token in the sequence is assigned a label representing an entity class or “O” (not an entity). In this paper, we focus on a practical setting of NER, namely corss domain few-shot NER (Hou et al., 2020; Yang and Katiyar, 2020), where the NER model is first pretrained on data-sufficient source domain(s) Ds and then tranferred to target domain(s) Dt with only a few labeled examples.
Formally, we denote a sentence and its labels as x = {x1, x2, . . . , xn} and y = {y1, y2, . . . , yn}, respectively. Following previous works (Hou et al., 2020; Ma et al., 2022), we adopt the episode learning paradigm in this paper, where we first pretrain the model on a set of tasks Ds = {(Ss,Qs)} from the source domain and then adapt the model to another set of tasks Dt = {(St,Qt)} from the target domain. Each task consists of a support set S = {(x(i),y(i))}N×Ki=1 for task adaption, and a query set Q = {(x(j),y(j))}N×K′j=1 for evaluation, where N denotes the number of entity types in a task, K and K ′ denote the number of few-shot samples that belong to each entity type in the support set and the query set, respectively. Given a task in the target domain, the goal of our model is to predict the labels of sentences in the query set after adapting the model to the task with its support set (i.e., finetuning the model with the support set). Figure 1 provides an illustration of the cross domain few-shot NER task.
Optimal Transport. Cross domain few-shot NER task can be considered as a domain adaption task. Optimal transport (OT) is a widely used method to solve the domain adaption tasks in the field of computer vision (Courty et al., 2017a,b; Damodaran et al., 2018; Fatras et al., 2021). Specifically, OT is a metric that measures the distance
between two probability distributions. In this paper, we focus on the discrete OT for two discrete empirical distributions, i.e., νs and νt:
W(νs,νt) = min T∈Σ(νs,νt) ⟨C,T⟩ , (1)
where Σ(νs,νt) = {T ∈ Rn×m+ : T1m = νs,T
⊤1n = νt} is a set of joint probabilities, 1m and 1n denote m-dimensional and n-dimensional vectors of ones respectively, ⟨·, ·⟩ is the Frobenius dot product, and C = [cij ] ∈ Rn×m+ is a cost matrix with each element representing the distance between the i-th data point of νs and the j-th one of νt. The optimal solution of T is called optimal transport plan, denoted as T∗, which can be efficiently obtained through the Sinkhorn algorithm (Cuturi, 2013) by solving an entropy regularized version of Equation (1) (see Appendix A).

heading: 3 Methodology
text: The overall framework of our MANNER is shown in Figure 2. In this section, we first introduce the details of MANNER in §3.1, and then introduce the pretraining of MANNER on the source domain in §3.2. We finally introduce how to adapt the model to the target domain in §3.3.
3.1 The MANNER Model Following previous prototype-based NER models (Fritzler et al., 2019; Wang et al., 2021c), we learn a prototype for each entity type, which is the mean of representations of tokens that belong to this type in the support set. However, compared with vanilla prototype-based methods which model prototypes
as deterministic vectors, we employ a probabilistic framework by modeling prototypes as stochastic variables, which is conducive to learn more informative prototypes and improve the robustness of few-shot models by capturing the uncertainties of prototypes (Allen et al., 2019; Zhen et al., 2020).
Moreover, following previous two-stage fewshot NER models (Wang et al., 2021b; Ma et al., 2022), we decompose the label prediction of NER into two sub-tasks: span detection which aims to predict the position tags of tokens, such as “B” and “I”, and entity typing which aims to predict the entity types of tokens. Accordingly, for each sentence, we additionally introduce two types of labels, namely position tags a = {a1, a2, . . . , an} and entity types e = {e1, e2, . . . , en}. We adopt the BIOES tagging scheme in this paper. Therefore, for a few-shot task τ = {S,Q}2, the position tags are chosen from {O,B, I, E, S}, while the entity types are chosen from the entity set E in the task, such as {Person, Location, ... }. We define the joint probability distribution of our model as:
pθ(y,a, e,Z | x,S,M) = pθ(y,a, e, | x,Z) pθ(Z | S,M) , (2) pθ(y,a, e, | x,Z) = pθ(y | a, e) pθ(a | x) pθ(e | x,Z) (3)
where Z ∈ R|E|×D denotes prototypes of all entity types in the task, which are obtained from the support set S and a memory module M (detailed below), i.e., pθ(Z | S,M). These prototypes are
2As the model applies to both source and target domain, we drop subscripts s and t in this section for clarity.
then used to predict the entity types of tokens, i.e., pθ(e | x,Z), which is further combined with the predicted position tags, i.e., pθ(a | x), to obtain the distribution over labels, i.e., pθ(y | a, e). Figure 3 illustrates the probabilistic graphical model of MANNER. In what follows, we will introduce the details of the joint probability distribution.
Memory-Augmented Prototypes pθ(Z | S,M): Since there are only a few labeled data in the support set, the prototypes that are obtained from the support set may not be accurate and representative. Therefore, we leverage an external memory module to store entity type information from the source domain to augment the support sets of few-shot tasks in the target domain. Specifically, we denote the memory as M, which contains key-value pairs that correspond to different entity types in the source domain. The keys are different entity types and the values are representations of tokens that belong to the corresponding entity types. For efficient retrieval from memory, we limit the number of token representations of each entity type to be m, which is referred to as the memory size.
In order to adapt the retrieved information (i.e., token representations) from the source domain to the target domain, we leverage optimal transport to retrieve and process information from the memory. Specifically, for an entity type k in a few-shot task τ , we first retrieve its most similar entity types k∗ in the memory based on the OT distance:
k∗ = argmin k′∈E W(Mk′ ,Hk)
= argmin k′∈E min T∈Σ( 1
m 1m, 1 nk 1nk ) ⟨C,T⟩ , (4)
where Hk = fθ(S(k)), S(k) = {xk,1, . . . , xk,nk}, is the contextualized representations of tokens that
belong to entity type k in the support set, fθ is a token encoder such as BERT (Devlin et al., 2019), Mk′ denotes the token representations of entity type k′ stored in the memory, and C is a cost matrix with each element computed as: c(Mk′,i,Hk,j) = ||Mk′,i − Hk,j ||22. We denote the retrieved information for entity type k as Mk∗ , and the optimal transport plan between token representations Hk and Mk∗ as T∗k, which is obtained through the Sinkhorn algorithm (Cuturi, 2013) in this paper.
We next follow previous works (Courty et al., 2017a,b) to adapt the retrieved information from the source domain, i.e., Mk∗ , to the domain of task τ through the following barycentric mapping:
ĥi = arg min h∈RD
∑
j
T∗k(i, j) · c(h,Hk,j), (5)
for all i = 1, . . . ,m, where ĥi denotes the projected representation of the i-th item in Mk∗ , and T∗k(i, j) represents an element of the optimal transport plan T∗k. It has been shown that when the cost function is squared Euclidean norm, the solution to above barycenter mapping corresponds to a weighted average of Hk (Courty et al., 2017b), which is given by:
Ĥk = diag(T ∗ k1nk) −1T∗kHk, (6)
where diag(·) is a diagonal matrix. After obtaining the adapted memory, i.e., Ĥk, we combine it with token representations in the support set to get the prototype distributions:
pθ(Z | S,M) = ∏
k∈E pθ
( zk | S(k),Mk∗ )
= ∏
k∈E N
( zk | gθ(Ĥk,Hk), σ21I ) , (7)
where we model the distributions over prototypes as Gaussian distributions, whose mean is obtained through a mean function gθ and the covariance is given by σ21I. We define the mean function as:
gθ(Ĥk,Hk) =γ ·Neural([r̂k, rk]) + (1− γ) · rk , (8)
where r̂k = 1m ∑ i Ĥk,i and rk = 1 nk ∑ j Hk,j are the mean of token representations in the memory and the support set respectively, [·, ·] is the concatenation operation, and Neural(·) is a feed-forward neural network with Relu activation function. We introduce a hyperparameter γ to interpolate the information from the memory and the support set.
Span Detection pθ(a | x): We formulate span detection as a sequence labeling task, i.e., predicting the position tags of tokens. Note that we use an encoder function fθ, e.g., BERT, to obtain the contextualized representations of tokens when computing prototypes. Based on these token representations, we use a linear classifier to compute the probability distributions of position tags. Specifically, for a sentence x, its distribution of position tags is:
pθ(a | x) = Softmax(fθ(x)W + b) , (9)
where W ∈ RD×5 and b are model parameters. Entity Typing pθ(e | x,Z): We follow the principle of prototypical networks (Snell et al., 2017) to compute the probability distributions of entity types. Specifically, for a sentence x, we compute its distribution of entity types as:
pθ(e | x,Z) = Softmax(fθ(x)Z⊤) , (10)
where Z is the sampled prototypes from prototype distribution defined in Equation (7).
Label Prediction pθ(y | a, e): Finally, we combine the results of span detection and entity typing to get the label distributions of a sentence x, i.e., pθ(y | a, e) = ∏n i=1 pθ(yi | ai, ei), where the predicted label distribution of each token is:
pθ(yi | ai, ei) ∝ pθ(ai | x) · pθ(ei | x,Z). (11)
For example, for a token xi with label “B-Person”, the probability of the token being classified as “BPerson” is proportional to the product of pθ(ai = B | x) and pθ(ei = Person | x,Z).

heading: 3.2 Learning in Source Domain
text: We next introduce how to learn our model on source domain. The goal of learning is to maximize the likelihood of observations in source domain3: pθ(Ds | M) = ∫ pθ(Ds | Zs)pθ(Zs | Ss,M)dZs
where Ds = {Ss,Qs} = {(x,y,a, e)} represents the set of observed variables in both support set and query set. The learning of such probabilistic models requires inferring the posterior distributions of stochastic variables, i.e., inferring the distribution of prototypes after seeing both the support set and
3We use a subscript s and t to denote variables in the source domain and the target domain respectively.
the query set in our case. However, exact inference is intractable due to the non-Gaussian likelihood function in our model. Therefore, we resort to variational inference to approximate the posteriors and learn the model (Kingma and Welling, 2014).
Specifically, we approximate posteriors of prototypes with the following variational distributions:
qθ(Zs | Ss,Qs) = ∏
k∈E qθ
( zs,k | S(k)s ,Q(k)s )
= ∏
k∈E N
( zs,k | gθ ( fθ(Q(k)s ), fθ(S(k)s ) ) , σ22I ) ,
where Q(k)s represents tokens that belong to entity type k in the query set, and σ22I is the covariance. For parameter efficiency, we use the same inference network gθ in Equation (7) to infer the posteriors of prototypes. However, the variational distributions are different from Equation (7), which can be regarded as prior distributions, in that the variational distributions are obtained based on both support and query sets while the prior distributions are obtained based on support set and the memory.
With the above variational distributions, we can derive the Evidence Lower BOund (ELBO) of loglikelihood function of our model as:
LELBO = −DKL [qθ(Zs | Ss,Qs) || pθ(Zs | Ss,M)] + ∑
o∈Ds Eqθ(Zs|Ss,Qs) [log pθ(y,a, e, | x,Zs)]
+ const. , (12)
where o = (x,y,a, e) and DKL[·||·] is the Kullback–Leibler (KL) divergence. Please refer to Appendix B for the detailed derivation of ELBO. We learn the model parameters θ by maximizing the ELBO defined in Equation (12), where KL divergence has a closed-form solution while the expectation term is approximated with Monte Carlo method by sampling from the variational distributions. At each training iteration, in order to remain the accuracy of the memory, we use the token representations, which is obtained through fθ, in both the support and query sets to update the information in the memory. Specifically, for each entity type in a task, we randomly select m representations of tokens that belong to this entity type to update token representations stored in the memory. We leave the exploration of selecting representative token representations as our further research.

heading: 3.3 Adaption in Target Domain
text: Finally, we introduce how to adapt our model to the target domain. Similar to previous work (Ma et al., 2022), we finetune the model with few-shot examples in the target domain. However, since we do not have access to the query set in the target domain, we can not use the ELBO in Equation (12) to finetune our model. Therefore, we propose to adapt our model to the target domain by maximizing the likelihood function of the support set in the target domain. Formally, the objective function is:
min θ
Epθ(Zt|St,M) [log pθ(St | Zt)] . (13)
After adapting our model to the target domain, we make prediction for a sentence x in the query set with pθ(y,a, e | x, Z̃t), where Z̃t represents the mean of prototype distributions. The pseudo code of the training and adaption process of our model is provided in Appendix C.

heading: 4 Experiments

heading: 4.1 Experimental Setups
text: Datasets. We conduct experiments on two groups of datasets: (1) Cross-Dataset (Hou et al., 2020): It is an English cross domain few-shot NER dataset constructed from four datasets: Ontonotes (Pradhan et al., 2013), WNUT-2017 (Derczynski et al.,
2017), GUM (Zeldes, 2017), CoNLL-2003 (Sang and De Meulder, 2002). For fair comparison, we use the same sampled episodes and dataset splits as in (Hou et al., 2020), where two of the four datasets are used for training, one for validation and the other for test. For example, to evaluate the performance on Ontonotes, we take WNUT and GUM as the training sets and CoNLL as the validation set. (2) Chinese Cross-Dataset: We also construct a Chinese cross domain few-shot NER dataset using five publicly available datasets: CCKS4, Address5, Medical (Zhang et al., 2022), Weibo (Peng and Dredze, 2015) and Cluener (Xu et al., 2020). Following the settings in (Yang and Katiyar, 2020), we first train few-shot models on the training set of the CCKS dataset and then evaluate their performance on the other four datasets. For each test dataset, we sample K-shot data from their training set as the support set and use the whole test set as the query set to construct a test episode. We repeat the sampling process for five times and obtain five test episodes for each dataset. We compare the average performance on the five test episodes. More details about our datasets are provided in Appendix D.1.
4https://www.biendata.xyz/competition/ccks_ 2020_el/
5https://tianchi.aliyun.com/dataset/109339
Baselines. On Cross-Dataset, we take the following models as our baselines: DecomposedMetaNER (Ma et al., 2022), CONTaiNER (Das et al., 2022), L-TapNet+CDT (Hou et al., 2020) and those baselines used in (Hou et al., 2020), such as TransferBERT, SimBERT, Matching Network, and ProtoBERT (Fritzler et al., 2019). On Chinese Cross-Dataset, we compare against some strong few-shot NER models such as DecomposedMetaNER, CONTaiNER, ProtoBERT, NNShot and StructShot (Yang and Katiyar, 2020).
Evaluation. We employ the episode evaluation as in (Hou et al., 2020) where we calculate micro F1 score within each test episode and then average over all test episodes. We repeat each experiment for 5 times with different seeds and report average micro F1 scores with their standard deviations.
Settings. Following previous works (Hou et al., 2020; Ma et al., 2022), we use bert-base-uncased (Devlin et al., 2019) to obtain contextualized token representations for Cross-Dataset. Similarly, bertbase-chinese is utilized for Chinese Cross-Dataset. We instantiate the mean function of the inference network, i.e., gθ, with a two-layered feed-forward neural network with the ReLU activation function and set the number of hidden units as 128. Moreover, to effectively optimize the ELBO, we follow previous works (Osawa et al., 2019; Zhang et al., 2021) to introduce an additional hyperparameter λ to down-weight the KL-divergence in the ELBO. Throughout the experiments, we set λ as 1e−3, and sample 5 times from the variational distributions to approximate the expectation term in the ELBO.
We set the maximum sequence length of the BERT models as 128 and the hyperparameter γ as 0.5. To optimize the parameters, we use AdamW (Loshchilov and Hutter, 2019) with a 1% linearly scheduled warmup as the optimizer and freeze the embedding layers of bert during optimization. Moreover, we perform grid search to select hyperparameters. Additional details about hyperparameter settings are in Appendix D.2.

heading: 4.2 Results and Analysis
text: Overall Performance. The performance of MANNER and baselines on Cross-Dataset and Chinese Cross-Dataset are reported in Table 1 and Table 2, respectively. The results show that MANNER performs better than all the baselines on F1 score in all settings and surpass the second best models by a large margin in most cases. Particularly, on Cross-Dataset, MANNER achieves an average performance improvement of 5.37% and 7.58% in 1- shot and 5-shot settings respectively compared with the best baselines. Similarly, on Chinese CrossDataset, the average performance improvement of MANNER is 6.65% (1-shot) and 7.38% (5-shot). The experimental results well demonstrate the effectiveness of MANNER in handling both English and Chinese few-shot NER tasks. Moreover, compared with DecomposedMetaNER, a strong baseline, MANNER achieves performance improvement up to 9.48% (Ontonotes 1-shot) on Cross-Dataset and 13.09% (Address 1-shot) on Chinese CrossDataset, which suggests that MANNER can achieve superior performance even with very few labeled data (e.g., 1-shot).
Ablation Studies. We conduct ablation studies to investigate the effect of different components, i.e., memory module, optimal transport and probabilistic framework, in our model. We introduce three variants of MANNER for the ablation study: (1) MANNER w/o Memory, where the memory module is removed and the prototype distributions are inferred from the support set only. Note that this variant does not use OT either as it is unnecessary to adapt the retrieved information from the memory to the target domain. (2) MANNER w/o OT, where we remove the OT module and use cosine similarity to retrieve the most similar entity type from memory. The retrieved information is directly used to infer prototype distributions without any processing. (3) Deterministic, where we remove probabilistic framework and model prototypes as deterministic vectors.
The results of ablation studies are reported in Table 3. It is shown that MANNER consistently outperforms MANNER w/o Memory in all settings, which indicates the effectiveness of our memory module in improving the performance. This is because with appropriate processing, e.g., OT in this paper, the information stored in memory can provide background knowledge for quickly and accurately learning new classes from a few examples and therefore brings performance improvement.
Table 3 also shows that MANNER outperforms MANNER w/o OT, which demonstrates the effectiveness of OT in MANNER. Moreover, we found that MANNER w/o Memory outperforms MANNER w/o OT which directly use the information from the memory without any processing. This is because in our cross domain few-shot setting, the information from the memory (source domain) is different from that of the test tasks (target domain), i.e., the entity types of two domain are disjoint, and therefore directly utilizing the information from the memory may introduce noises to the test tasks, leading to the performance degradation. The results demonstrate the necessity of leveraging OT to adapt information from the memory to current task to achieve satisfactory performance.
Table 3 shows that MANNER achieves better or comparative results compared with its deterministic counterpart, especially on the 1-shot settings. The improvement can be explained by the fact that MANNER introduces small noises to the prototypes by sampling from the prototype distributions to prevent the model from overfitting the few-shot data during the finetuning stage.
Effect of Decomposed Framework. It is worth noting that MANNER decomposes label prediction of NER into two-subtasks: span detection and entity typing. To investigate the effect of the decomposed framework, we additionally introduce a
variant of MANNER: VM-ProtoNet where we only remove the decomposed framework and learn prototypes for each entity label, which is similar to ProtoBERT. Note that we also use memory and OT to augment few-shot NER tasks in VM-ProtoNet. We compare the performance of MANNER and VMProtoNet on Cross-Dataset, which is presented in Figure 4. The results show that MANNER surpasses VM-ProtoNet in all settings and achieves noticeable performance improvement in most cases, especially on the CoNLL dataset. The success of the decomposed framework maybe because it avoids learning prototype for non-entities (i.e., “O” class) which is noisy and meaningless. Overall, experimental results demonstrate the effectiveness of the decomposed framework in few-shot NER tasks, which is consistent with the results of previous works (Wang et al., 2021b; Ma et al., 2022).
Effect of Memory Size. In MANNER, we limit the number of token representations of each entity type stored in the memory. We further conduct experiments to understand the effect of the memory size on the performance of MANNER. Specifically, we vary the memory size from 1 to 30 and report the performance of MANNER on Ontonotes and Address. The results in Figure 5 show that MANNER can achieve decent performance even with a low memory size and the performance converges with the increase of memory size. These findings suggest that MANNER is insensitive to memory size, which brings another benefit: it is sufficient for MANNER to achieve satisfactory performance by storing only a small number of token representations in the memory, which is efficient for both retrieving and processing information from the memory.

heading: 5 Related Work
text: Few-Shot NER. Recently, few-shot NER has received growing interest. Previous works mainly address few-shot NER with meta-learning methods (Fritzler et al., 2019; Wang et al., 2021c;
Huang et al., 2021; Tong et al., 2021; Ma et al., 2022). These methods build few-shot models either upon prototypical network (Snell et al., 2017), which learns prototypes for entity types (Fritzler et al., 2019; Huang et al., 2021; Tong et al., 2021; Wang et al., 2022b; Ji et al., 2022; Wang et al., 2022a), or MAML (Finn et al., 2017), which adapts the model parameters to few-shot tasks through inner-update on the support set (Li et al., 2022; Ma et al., 2022). Another line of work adopts the transfer learning paradigm, where they first learn a feature extractor on the source domain and then transfer the pretrained model to the target domain (Hou et al., 2020; Yang and Katiyar, 2020; Das et al., 2022). These methods make predictions through the nearest neighbor inference (Wiseman and Stratos, 2019). In addition, some recent works focus on the two-stage few-shot NER model (Ziyadi et al., 2020; Wang et al., 2021b; Ma et al., 2022), where they decompose the NER task into two-subtasks: span detection and entity typing. Moreover, prompt-based techniques (Cui et al., 2021; Ding et al., 2022; Chen et al., 2022) have also been proposed to address few-shot NER tasks. In contrast, MANNER stores the information from the source domain in the memory, which is then used to augment few-shot task in target domain.
Memory. Memory-augmented methods have been widely studied in the field of computer vision (Santoro et al., 2016; Bornschein et al., 2017; Ramalho and Garnelo, 2019; Munkhdalai et al., 2019; Zhen et al., 2020; Du et al., 2022). Particularly, Santoro et al. (2016) propose to augment neural network with Neural Turing Machine (Graves et al., 2014) for few-shot learning, which enables quickly encoding and retrieving new information. Ramalho and Garnelo (2019) further introduce a memory controller to select the minimum samples to be stored in the memory. Memory-augmented methods have also been successfully applied in NLP tasks, such as question answering (Das et al., 2017), text classification (Geng et al., 2020), text generation (He et al., 2020) and slot tagging (Wang et al., 2021a). Compared with above methods, our model utilizes optimal transport to adapt the retrieved memory to the target domain instead of using neural networks, which is more effective.
Optimal Transport. In domain adaption, optimal transport is a widely used method to transport data from the source domain to the target do-
main (Courty et al., 2017a,b; Damodaran et al., 2018; Fatras et al., 2021; Nguyen et al., 2021; Fatras et al., 2022). Theoretical guarantees have been provided in (Redko et al., 2017) to justify the use of OT in domain adaption. In Courty et al. (2017b), they propose to transport features from the source domain to the target domain through a barycentric mapping. However, they only consider transporting feature distributions. In contrast, some works propose to align the joint distributions of features and labels in source and target domains (Courty et al., 2017a; Damodaran et al., 2018).

heading: 6 Conclusion
text: This paper proposes MANNER to handle the cross domain few-shot NER task. MANNER uses a memory module to store information from the source domain, which is then leveraged to augment few-shot task in the target domain. To effectively utilize the information from the memory, MANNER uses optimal transport to retrieve and process information from the memory, which enables explicitly adapting the retrieved information to the target domain and improve the performance in the cross domain few-shot setting. Experimental results on both English and Chinese few-shot NER datasets show that MANNER can achieve superior performance over existing methods.
Limitations
One limitation of our work is that MANNER only explicitly utilizes the memory to enhance the performance of the entity typing module in target domain. However, we argue that the memory could also implicitly enhances the span detection module through the shared pretrained language model with entity typing module. We leave how to explicitly leverage memory to enhance both entity typing and span detection modules as future work.

heading: Acknowledgements
text: We thank all the reviewers for their valuable feedback and constructive suggestions. We would also like to thank Zeqi Tan for helping constructing the Chinese cross domain few-shot NER dataset and Xuming Hu for helpful discussions on improving the quality of the paper.

heading: A Sinkhorn Algorithm
text: Sinkhorn algorithm (Cuturi, 2013) is an efficient method to approximate the optimal transport (OT) distance. It aims to solve an entropy regularized optimal transport problem, which is defined as:
Wϵ(νs,νt) = min T∈Σ(νs,νt) ⟨C,T⟩+ ϵh(T) , (14)
where h(T) = ∑
i,j Tij logTij denotes the entropy regularizer and ϵ is the regularization parameter. The optimization problem in Equation (14) can be efficiently solved through the following iterative Bregman projections (Benamou et al., 2015):
a(l+1) = νs
Gb(l) , b(l+1) =
νt
G⊤a(l+1) , (15)
starting from b0 = 1m1m, where G = [Gij ] and Gij = e
−Cij/ϵ. After L iterations, the optimal transport plan T∗ is calculated as T∗ij = a L i Gijb L j .

heading: B Derivation of ELBO
text: Note that the joint probability distribution of our model on source domain is given by:
pθ(Ds,Zs | M) = pθ(Ds | Zs)pθ(Zs | Ss,M),
where Ds = {Ss,Qs} = {(x,y,a, e)} represents the set of observed variables. We further define a
Algorithm 1: Variational Memory-Augmented Few-Shot NER (MANNER). Input : Tasks from source domain Ds, few-shot tasks from target domain Dt, training steps T ,
finetune steps J , training learning rate η, finetune learning rate ξ. 1 Initialize model parameters θ and memory M; 2 /* Part I: Training on source domain. */ 3 for s = 1, . . . T do 4 Sample a batch of tasks Dbatch from Ds; 5 for each task T = (Ss,Qs) ∈ Dbatch do 6 for each entity type k in T do 7 Retrieve the most similar entity type k∗ from memory based on Equation (4); 8 Adapt the retrieved content Mk∗ to current task based on Equation (6);
9 Calculate the prior distributions of prototypes, i.e., pθ(Zs | Ss,M) based on Equation (7); 10 Calculate the variational distributions of prototypes, i.e., pθ(Zs | Ss,Qs); 11 Sample prototypes Zs from pθ(Zs | Ss,Qs); 12 Calculate the ELBO based on Equation (12); 13 Accumulate gradients of model parameters θ which are obtained by maximizing the ELBO; 14 Update memory with token representations in both support and query sets.
15 Update model parameters θ with learning rate η;
16 /* Part II: Finetuning on target domain. */ 17 for each task T = (St,Qt) ∈ Dt do 18 Initialize model parameters θ′ = θ ; 19 for s = 1, . . . , J do 20 for each entity type k in T do 21 Retrieve the most similar entity type k∗ from memory based on Equation (4); 22 Adapt the retrieved content Mk∗ to current task based on Equation (6);
23 Calculate the prior distributions of prototypes, i.e., pθ(Zt | St,M) based on Equation (7); 24 Sample prototypes Zt from pθ(Zt | St,M); 25 Calculate the objective function based on Equation (13); 26 Update model parameters θ′ with learning rate ξ;
variational distribution qθ(Zs | Ss,Qs) to approximate the posteriors of latent variables. Therefore, we can derive the ELBO as follows:
log pθ(Ds | M)
= log ∫ pθ(Ds,Zs | M)
qθ(Zs | Ss,Qs) qθ(Zs | Ss,Qs) dZs
≥ ∫
qθ(Zs | Ss,Qs) log pθ(Ds,Zs | M) qθ(Zs | Ss,Qs) dZs
= Eqθ(Zs|Ss,Qs) [log pθ (Ds | Zs)] −DKL [qθ(Zs | Ss,Qs) || pθ(Zs | Ss,M)]
≜ LELBO , (16) where the inequality is obtained via the Jensen’s inequality. Since the likelihood function of our model is given by:
pθ(Ds | Zs) = ∏
o∈Ds pθ(y,a, e, | x,Zs) p(x).
We can put this function into Equation (16) and further derive the ELBO as:
LELBO = −DKL [qθ(Zs | Ss,Qs) || pθ(Zs | Ss,M)] + ∑
o∈Ds Eqθ(Zs|Ss,Qs) [log pθ(y,a, e, | x,Zs)]
+ const. , (17)
where const. = ∑
o∈Ds log p(x) is a constant. The KL divergence in Equation (17) has a closed form solution, which is given by:
−DKL [qθ(Zs | Ss,Qs) || pθ(Zs | Ss,M)]
= −1 2
∑
k∈E
1
σ21 (µk −mk)⊤(µk −mk)
− |E| 2 ((s− 1)D − log s) , (18)
where µk, mk denote the mean of the prior and variational distributions of prototypes, respectively, s = σ22/σ 2 1 , and D is the dimension of prototypes. Moreover, we sample Zs from variational distributions qθ(Zs | Ss,Qs) to approximate the expectation term in ELBO.

heading: C Pseudo Code
text: The training and inference process of our model is provided in Algorithm 1. In the training process, we randomly sample a small batch of tasks Dbatch, accumulate the gradients of their objective function and then update the model parameters with the AdamW optimizer. In the inference process, for each task, we first initialize the model parameters θ′ with the learned model parameters in the source domain, i.e., θ′ = θ, and then finetune the parameters by maximizing the likelihood function in the support set for J steps.

heading: D Experimental Details
text: D.1 Datasets Table 5 shows the statistics of original datasets used to construct the experimental datasets and statistics of the constructed few-shot datasets.
Cross-Dataset is an English cross domain fewshot NER dataset, which is constructed to evaluate the performance of meta-learning based few-shot models. We use the public episodes6 constructed by (Hou et al., 2020) in our experiments, where
6https://github.com/AtmaHou/FewShotTagging.
the training, validation and test episodes for each dataset are provided.
We additionally construct a Chinese cross domain few-shot NER dataset from five public Chinese NER datasets: CCKS, Address, Medical, Weibo and Cluener. We follow the experimental settings in (Yang and Katiyar, 2020), where they first train few-shot models on a source domain and then transfer the model to target domain with fewshot data. We take CCKS as source domain and the other four datasets as target domains. To construct few-shot data in target domains, we use the sampling method in (Ding et al., 2021) to sample K-shot data from the training set of each test dataset as support set and use the original test data as query set. We repeat the sampling process for five times to obtain accurate experimental results.
D.2 Hyperparameter Settings We set the memory size m, i.e., number of token representations of each entity type in the memory, as 15. The hyperparameter γ and the standard deviation of prototype distributions is set to be 0.5 and e−10 respectively. The dropout rate and weight decay coefficient is set to be 0.1 and 1e − 3, respectively. On Cross-Dataset, we choose batch size from {1, 16, 32}, learning rate from {1e-5, 3e-5, 1e-4}, training steps from {300, 500, 1000}, and finetune steps from {30, 50}. We perform grid search to choose hyperparameters that have the best performance on the validation set. The optimal hyperparameter settings on Corss-Dataset are provided in Table 4. On Chinese Cross-Dataset, we set the batch size as 1, training steps as 1000, finetune steps as 50, training learning rate as 3e-5 and finetune learning rate as 3e-5 for all settings. During training, we evaluate our model on the validation set every 100 steps and select the checkpoint with best f1 scores on the validation set as the final model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Please see the Limitations Section.
7 A2. Did you discuss any potential risks of your work? There is not obvious risk regarding our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Please see the Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Please see Section 4.
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Our model is built based on the existing pretrained model, and the computational budget might vary depending on the used backbone model.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Please see Section 4.1 and Appendix D.2.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Please see Section 4.2.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Please see Appendix D.2.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave how to explicitly leverage memory to enhance both entity typing and span detection modules as future work