SlowGuess/NAACL_2022_OCR huggingface dataset:
Peiyi Wang $^{1*}$ , Runxin Xu $^{1*}$ , Tianyu Liu $^{2}$ , Qingyu Zhou $^{2}$ , Yunbo Cao $^{2}$ , Baobao Chang $^{1}$ , Zhifang Sui $^{1\dagger}$
<sup>1</sup> Key Laboratory of Computational Linguistics, Peking University, MOE, China  
<sup>2</sup> Tencent Cloud Xiaowei
{wangpeiyi9979, runxinxu}@gmail.com
{rogertyliu, qingyuzhou, yunbocao}@tencent.com
{chbb, szf}@pku.edu.cn
Few-Shot Sequence Labeling (FSSL) is a canonical paradigm for the tagging models, e.g., named entity recognition and slot filling, to generalize on an emerging, resource-scarce domain. Recently, the metric-based meta-learning framework has been recognized as a promising approach for FSSL. However, most prior works assign a label to each token based on the token-level similarities, which ignores the integrality of named entities or slots. To this end, in this paper, we propose ESD, an Enhanced Span-based Decomposition method for FSSL. ESD formulates FSSL as a span-level matching problem between test query and supporting instances. Specifically, ESD decomposes the span matching problem into a series of span-level procedures, mainly including enhanced span representation, class prototype aggregation and span conflicts resolution. Extensive experiments show that ESD achieves the new state-of-the-art results on two popular FSSL benchmarks, FewNERD and SNIPS, and is proven to be more robust in the nested and noisy tagging scenarios. Our code is available at https://github.com/Wangpeiyi9979/ESD.
Many natural language understanding tasks can be formulated as sequence labeling tasks, such as Named Entity Recognition (NER) and Slot Filling (SF) tasks. Most prior works on sequence labeling follow the supervised learning paradigm, which requires large-scale annotated data and is limited to pre-defined classes. In order to generalize on the emerging, resource-scare domains, Few-Shot Sequence Labeling (FSSL) has been proposed (Hou et al., 2020; Yang and Katiyar, 2020). In FSSL, the model (typically trained on the source domain
Figure 1: A 2-way 2-shot few-shot NER task. The model needs to learn new entities with few examples.
data) needs to solve the  $N$ -way ( $N$  unseen classes)  $K$ -shot (only  $K$  annotated examples for each class) task in the target domain. Figure 1 shows a 2-way 2-shot target domain few-shot NER task, where 'PER' and 'ORG' are 2 unseen entity types, and in the training set  $\mathcal{S} = \{S_1, S_2\}$ , both of them have only 2 annotated entities. The tagging models should annotate 'Albert Einstein' in the test sentence  $q$  as a 'PER' according to  $\mathcal{S}$ .
Recently, metric-based meta-learning (MBML) methods have become the mainstream and state-of-the-art methods in FSSL (Hou et al., 2020; Ding et al., 2021), which train the models on the tasks sampled from the source domain sentences in order to mimic and solve the target task. In each task of training and testing, they make the prediction through modeling the similarity between the training set (support set) and the test sentence (query). Specifically, previous MBML methods (Ding et al., 2021; Hou et al., 2020; Yang and Katiyar, 2020) mainly formulate FSSL as a token-level matching problem, which assigns a label to each token based on the token-level similarities. For example, the token 'Albert' would be labeled as 'B-PER' due to its resemblance to 'Steve' and 'Isaac'.
However, for the MBML methods, selecting the appropriate metric granularity would be fundamental to the success. We argue that prior works that focus on modeling the token-level similarity are sub-optimal in FSSL: 1) they ignore the integrality of named entities or dialog slots that are composed of a text span instead of a single token. 2) in FSSL,
$^\ddagger$ Contribution during internship in Tencent.
Human Language Technologies, pages 5012 - 5024
the conditional random fields (CRF) is an important component for the token-level sequence labeling models (Yang and Katiyar, 2020; Hou et al., 2020), but the transition probability between classes in the target domain task can not be sufficiently learned with very few examples (Yu et al., 2021). The previous methods resort to estimate the values with the abundant source domain data, which may suffer from domain shift problem (Hou et al., 2020). To overcome these drawbacks of the token-level models, in this paper, we propose ESD, an Enhanced Span-based Decomposition model that formulates FSSL as a span-level matching problem between test query and support set instances.
Specifically, ESD decomposes the span matching problem into three main subsequent span-level procedures. 1) Span representation. We find the span representation can be enhanced by information from other spans in the same sentence and the interactions between query and support set. Thus we propose a span-enhancing module to reinforce the span representation by intra-span and interspan interactions. 2) Span prototype aggregation. MBML methods usually aggregate the span vectors that belongs to the same class in the support set to form the class prototype representation. Among all class prototypes, the O-type serves as negative examples and covers all the miscellaneous spans that are not entities, which poses new challenges to identify the entity boundaries. To this end, we propose a span-prototypical module to divide O-type spans into 3 sub-types to distinguish the miscellaneous semantics by their relative position with recognized entities, together with a dynamically aggregation mechanism to form the specific prototype representation for each query span. 3) Span conflicts resolution. In the span-level matching paradigm, the predicted spans may conflict with each other. For example, a model may annotate both "Albert Einstein" and "Albert" as "PER". Therefore, we propose a span refining module that incorporates the Soft-NMS (Bodla et al., 2017; Shen et al., 2021) algorithm into the beam search to alleviate this conflict problem. With Beam Soft-NMS, ESD can also handle nested tagging cases without any extra training, which previous methods are incapable of.
We summarize our contribution as follows: (1) We propose ESD, an enhanced span-based decomposition model, which formulates FSSL as a span-level matching problem. (2) We decompose the span matching problem into 3 main subsequent
procedures, which firstly produce enhanced span representation, then distinguish miscellaneous semantics of O-types, and achieve the specific prototype representation for each query, and finally resolve span conflicts. (3) Extensive experiments show that ESD achieves new state-of-the-art performance on both few-shot NER and slot filling benchmarks and that ESD is more robust than other methods in nested and noisy scenarios.
2.1 Few-Shot Learning
Few-Shot Learning (FSL) aims to enable the model to solve the target domain task with a very small training set  $D_{train}$  (Wang et al., 2020). In FSL, people usually consider the  $N$ -way  $K$ -shot task, where the  $D_{train}$  has  $N$  classes, and each class has only  $K$  annotated examples ( $K$  is very small, e.g., 5). Training a model only based on  $D_{train}$  from scratch will inevitably lead to over-fitting. Therefore, researchers usually introduce the source domain annotated data to help train models, e.g., Few-Shot Relation Classification (FSRC) (Han et al., 2018), Few-Shot Text Classification (FSTC) (Geng et al., 2019) and Few-Shot Event Classification (FSEC) (Wang et al., 2021a). The source domain data do not contain examples belonging to classes in  $D_{train}$ , and thus the FSL setting can be guaranteed. Specifically, FSSL tasks in our paper also have the source domain data.
2.2 Metric-Based Meta-Learning
Meta-learning (Hochreiter, Younger, and Conwell, 2001a) is a popular method to deal with Few-Shot Learning (FSL). Meta-learning constructs a series of tasks sampled from the source domain data to mimic the target domain task, and trains models across these sampled tasks. Each task contains a training set (support set) and a test instance (query). The core idea of meta-learning is to help model learn the ability to quickly adapt to new tasks, i.e., learn to learn (Hochreiter, Younger, and Conwell, 2001b). Meta-learning can be combined with the metric-learning (Kulis et al., 2013) (metric-based meta-learning), which makes predictions based on the similarity of the support set and the query. For example, Prototypical Network (Snell, Swersky, and Zemel, 2017) first learns prototype vectors from a few examples in the support set for classes, and further uses prototype vectors for query prediction. Specifically, ESD (our model) follows this
Figure 2: The architecture (5 span modules) of ESD with a 2-way ('PER' and 'ORG') 2-shot input task. We only list spans with lengths less than 2 for clarity. ESD assigns label 'PER' to the span "Albert Einstein" and 'O' to the other spans in query  $q$  based on the support set  $\{S_1, S_2\}$ .
metric-based meta-learning paradigm.
2.3 Few-Shot Sequence Labeling
Recently, few shot sequence labeling (FSSL) has been widely explored by researchers. For example, (Fritzler, Logacheva, and Kretov, 2019a) leverages the prototypical network (Snell, Swersky, and Zemel, 2017) in the few-shot NER. (Yang and Katiyar, 2020) further proposes a cheap but effective method to capture the label dependencies between entity tags without expensive CRF training. (Wang et al., 2021b) utilizes a large unlabelled dataset and proposes a distillation method. (Cui et al., 2021) introduces a prompt method to tap the potential of BART (Lewis et al., 2020). (Hou et al., 2020) extends the TapNet (Yoon, Seo, and Moon, 2019) and proposes a collapsed CRF for few-shot slot filling. (Ma et al., 2021) and (Athiwaratkun et al., 2020) formulate FSSL as a machine comprehension problem and a generation problem, respectively. (Yu et al., 2021) proposed to retrieve the most similar exemplars in the support set for span-level prediction. Although their methods also involve span matching, their main focus is on the retrieval-augmented training. Our work differs from (Yu et al., 2021) in that we propose an effective actionable pipeline to get enhanced span representation, handle miscellaneous semantics of O-types and resolve the potential span conflicts in both non-nested and nested situations, where the last two modules are essential to align the candidate spans with class prototypes in the support set but missing in (Yu et al., 2021). Besides, our enhanced span representation greatly improves the batched softmax objective of (Yu et al., 2021) by inter- and
intra-span interactions.
We define a sentence as  $\mathbf{x}$  and its label as  $\mathbf{y} = \{(s_i, y_i)\}_{i=1}^M$ , where  $s_i$  is a span of  $\mathbf{x}$  (e.g., 'Steve Jobs'),  $y_i$  is the label of  $s_i$  (e.g., 'PER') and  $M$  is the number of spans in the  $\mathbf{x}$ . Following the previous FSSL setting (Hou et al., 2020; Ding et al., 2021), we have data in source domain  $\mathcal{D}_{source}$  and target domain  $\mathcal{D}_{target}$ , and models are evaluated on tasks from  $\mathcal{D}_{target}$ . Meta-learning based FSSL has two stages, meta-training and meta-testing. In the meta-training stage, the model is trained on tasks sampled from  $\mathcal{D}_{source}$  to mimic the test situation, and in the meta-testing stage, the model is evaluated on test tasks. A task is defined as  $\mathcal{T} = \{S, q\}$ , consisting of a support set  $S = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^I$ , and a query sentence  $q = \mathbf{x}$ .  $S$  includes  $N$  types of entities or slots ( $N$ -way), and each type has  $K$  annotated examples ( $K$ -shot). For spans that do not belong to the  $N$  types, e.g., 'studied at' in Figure 1, we set their label to O. The types except O in each test task are guaranteed to not exist in the training tasks. Given a task  $\mathcal{T} = \{S, q\}$ , the model needs to assign a label to each span in the query sentence  $q$  based on  $S$ .
Our ESD formulates FSSL as a span-level matching problem and decomposes it into a series of span-related procedures for a better span matching. Figure 2 illustrates the architecture of ESD.
4.1 Span Initialization Module
Given a task  $\mathcal{T} = \{S,q\}$ , we use BERT (Devlin et al., 2019) to encode sentences in  $S$  and  $q$ , and utilize the output of the last layer to represent tokens in the sentence. Therefore, for a sentence with  $N$  tokens  $\mathbf{x} = (x_{1},x_{2},\dots,x_{N})$ , we can achieve representations  $(\mathbf{h_1},\mathbf{h_2},\dots,\mathbf{h_N})$ , where  $\mathbf{h_i}\in \mathbb{R}^{d_w}$  is the hidden state corresponding to the token  $x_{i}$ . Then, for a span  $s = (l,r)$ , where  $l$  and  $r$  are the start index and end index of span  $s$  in the sentence  $\mathbf{x}$ , we obtain its initial representation  $\mathbf{s}_{(l,r)} = [\mathbf{h_l};\mathbf{h_r}]\mathbf{W_s}$ . We enumerate spans in the sentence with a maximum length of  $L$ .
4.2 Span Enhancing Module
4.2.1 Intra Span Attention
Intuitively, the meaning of specific spans can be inferred from other spans in the same sentence. We thus design an Intra Span Attention (ISA) mechanism. Given all the span representations of a sentence  $\mathbf{S} \in \mathbb{R}^{B \times d}$  ( $B$  is the number of spans). We denote the  $i$ -th row of  $\mathbf{S}$  as  $\mathbf{s}_i$ , which represents the  $i$ -th span in the sentence. For  $\mathbf{s}_i$ , we first get its ISA span representation  $\bar{\mathbf{s}}_i = \sum_{j=1}^{B} \alpha_i^j \mathbf{s}_j$ , where  $\alpha_i = \text{softmax}(\mathbf{s}_i \mathbf{S}^{\mathbf{T}})$ . For clarity, we denote this attention aggregation operation as  $\phi$ , i.e.,
$$
\bar {\mathbf {s}} _ {\mathbf {i}} = \phi (\mathbf {s} _ {\mathbf {i}}, \mathbf {S}) \tag {1}
$$
then, a Feed Forward Neural Networks (FFN) (Vaswani et al., 2017) with residual connection (He et al., 2016) and layer normalization (Ba, Kiros, and Hinton, 2016) are used to get the final ISA enhanced feature  $\tilde{\mathbf{s}}_{\mathrm{i}}$
$$
\tilde {\mathbf {s}} _ {\mathbf {i}} = \operatorname {L a y e r N o r m} \left(\mathbf {s} _ {\mathbf {i}} + \mathrm {F F N} _ {\text {i s a}} (\bar {\mathbf {s}} _ {\mathbf {i}})\right) \tag {2}
$$
$$
\mathrm {F F N} _ {\mathrm {i s a}} \left(\bar {\mathbf {s}} _ {\mathbf {i}}\right) = \operatorname {G E L U} \left(\bar {\mathbf {s}} _ {\mathbf {i}} \mathbf {W} _ {\mathbf {i s a}} ^ {\mathbf {1}}\right) \mathbf {W} _ {\mathbf {i s a}} ^ {\mathbf {2}} \tag {3}
$$
4.2.2 Cross Span Attention
After ISA, to accommodate the span matching between the test query and supporting sentences, and facilitate the inter-span interaction, we propose Cross Span Attention (CSA) to enhance query spans  $\tilde{\mathbf{Q}}\in \mathbb{R}^{B_q\times d}$  with the support set spans  $\{\tilde{\mathbf{S}}_{\mathbf{i}}\in$ $\mathbb{R}^{B_i\times d};i = 1,\dots,I\}$ , and vice versa. We first concatenate all span representations in the support set into one matrix  $\tilde{\mathbf{S}} = [\tilde{\mathbf{S}}_1,\tilde{\mathbf{S}}_2,\dots,\tilde{\mathbf{S}}_{\mathbf{I}}]\in \mathbb{R}^{B_s\times d}$ . We denote the  $n$ -th row of  $\tilde{\mathbf{S}}$  as  $\tilde{\mathbf{s}}_{\mathbf{n}}$  and the  $m$ -th row of  $\tilde{\mathbf{Q}}$  as  $\tilde{\mathbf{q}}_{\mathbf{m}}$ , and obtain their CSA span representations  $\hat{\mathbf{s}}_{\mathbf{n}} = \phi (\tilde{\mathbf{s}}_{\mathbf{n}},\tilde{\mathbf{Q}})$  and  $\hat{\mathbf{q}}_{\mathbf{m}} = \phi (\tilde{\mathbf{q}}_{\mathbf{m}},\tilde{\mathbf{S}})$ . Then,
we get the final CSA enhanced representation  $\check{\mathbf{s}}_{\mathbf{n}}$  and  $\check{\mathbf{q}}_{\mathbf{m}}$  as follows,
$$
\tilde {\mathbf {s}} _ {\mathbf {n}} = \text {L a y e r N o r m} (\tilde {\mathbf {s}} _ {\mathbf {n}} + \mathrm {F F N} _ {\mathrm {c s a}} (\hat {\mathbf {s}} _ {\mathbf {n}})) \tag {4}
$$
$$
\tilde {\mathbf {q}} _ {\mathbf {m}} = \text {L a y e r N o r m} \left(\tilde {\mathbf {q}} _ {\mathbf {m}} + \mathrm {F F N} _ {\mathrm {c s a}} \left(\hat {\mathbf {q}} _ {\mathbf {m}}\right)\right) \tag {5}
$$
$$
\mathrm {F F N} _ {\mathrm {c s a}} (\mathbf {x}) = \operatorname {G E L U} (\mathbf {x} \mathbf {W} _ {\mathrm {c s a}} ^ {1}) \mathbf {W} _ {\mathrm {c s a}} ^ {2} \tag {6}
$$
4.3 Span Prototypical Module
4.3.1 Instance Span Attention
Since different support set spans play different roles for a query span, we propose multi-IN stance Span Attention (INSA) to get class representations. For the  $i$ -th class that contains  $K$  annotated spans with enhanced representations  $\tilde{\mathbf{S}}_{\mathbf{i}} = [\tilde{\mathbf{s}}_{\mathbf{i}}^{1},\dots,\tilde{\mathbf{s}}_{\mathbf{i}}^{K}]$  in the support set, given a query span  $\check{\mathbf{q}}_{\mathbf{m}}$ , INSA gets the corresponding prototypical representation  $\mathbf{z}_{\mathbf{m}}^{\mathrm{i}} = \phi (\check{\mathbf{q}}_{\mathbf{m}},\check{\mathbf{S}}_{\mathbf{i}})$ .
4.3.2 O Partition and Prototypical Span Attention
The O-type spans have huge quantities and miscellaneous semantics, which is hard to be represented by only one prototypical vector. In the span-based framework, considering the boundary information is essential for a span, we divide the O-type spans into 3 sub-classes according to their boundary, to alleviate their miscellaneous semantics problem. Specifically, given a sentence with  $I$  annotated spans  $\{(l_i,r_i)\}_{i = 1}^I$ , where  $l_{i}$  and  $r_i$  are the left and right boundary of the  $i$ -th annotated span. For each of the other spans  $(l_o,r_o)$ , we assign it a sub-class  $\mathrm{O}_{sub}$  as follows,
$$
O _ {s u b} = \left\{ \begin{array}{l l} O _ {1}, & \forall i, s. t. r _ {o} <   l _ {i} \vee l _ {o} > r _ {i} \\ O _ {2}, & \exists i, s. t. l _ {o} \geq l _ {i} \wedge r _ {o} \leq r _ {i} \\ O _ {3}, & \text {O t h e r s} \end{array} \right. \tag {7}
$$
where  $O_{1}$  denotes the span that does not overlap with any entities or slots in the sentence, e.g., "study at" in  $S_{2}$  of Figure 2, and  $O_{2}$  represents the span that is the sub-span of an entity or slot, e.g., "Isaac" in  $S_{2}$  of Figure 2. After O Partition, we get the prototypical representation of each  $\mathrm{O}_{\mathrm{sub}}$  via INSA, thus for a query span  $\check{\mathbf{q}}_{\mathbf{m}}$ , we have 3 sub-class representations  $\mathbf{Z}_{\mathbf{m}}^{\mathrm{o}} = [\mathbf{z}_{\mathbf{m}}^{\mathrm{o}_1},\mathbf{z}_{\mathbf{m}}^{\mathrm{o}_2},\mathbf{z}_{\mathbf{m}}^{\mathrm{o}_3}]$  for the class O. Then, we utilize Prototypical Span Attention (PSA) to achieve the final O representation  $\mathbf{Z}_{\mathbf{m}}^{\mathrm{o}} = \phi (\check{\mathbf{q}}_{\mathbf{m}},\mathbf{Z}_{\mathbf{m}}^{\mathrm{o}})$ .
4.4 Span Matching Module
Given a task  $\mathcal{T} = (\mathcal{S},q)$ , for the  $m$ -th span in  $q_{m}$ , we achieve its enhanced representation  $\check{\mathbf{q}}_{\mathbf{m}}$
<sup>1</sup>We omit the bias term in this paper for clarity.
Figure 3: A demonstration for Beam Soft-NMS (BSNMS). For clarity, we set the filter threshold  $\delta$  to 0.6, and suppose the span score is always decayed by the overlapped spans with a constant decayed score  $-0.1$  (Note that the example is simplified for demonstration). BSNMS filters false positive spans (colored in orange) in this demonstration. A more clear step by step demonstration is included in Appendix D.
and corresponding prototypical vectors  $\mathbf{Z}_{\mathbf{m}} = (\mathbf{z}_{\mathbf{m}}^{\mathrm{o}},\mathbf{z}_{\mathbf{m}}^{\mathbf{1}},\dots,\mathbf{z}_{\mathbf{m}}^{\mathbf{N}})$  through previous span modules. Then, we predict  $q_{m}$  as the type  $z_{k}$  in the support set with probability,
$$
p \left(y _ {m} = z _ {k} \mid q _ {m}\right) = \frac {\exp \left(- L _ {2} \left(\check {\mathbf {q}} _ {\mathbf {m}}, \mathbf {z} _ {\mathbf {k}} ^ {\mathbf {m}}\right)\right)}{\sum_ {k ^ {\prime}} \exp \left(- L _ {2} \left(\check {\mathbf {q}} _ {\mathbf {m}}, \mathbf {z} _ {\mathbf {k} ^ {\prime}} ^ {\mathbf {m}}\right)\right)} \tag {8}
$$
where  $L_{2}$  is the euclidean distance. During training, we use cross-entropy as our loss function,
$$
\mathcal {L} = - \frac {1}{B _ {q}} \sum_ {m = 1} ^ {B _ {q}} \log p \left(y _ {m} ^ {*} \mid q _ {m}\right) \tag {9}
$$
where  $y_{m}^{*}$  is the gold label of  $q_{m}$  and  $B_{q}$  is the number of spans in the query  $q$ .
4.5 Span Refining Module in Inference
During inference, spans outputted by the matching module may have conflicts, we thus propose a refining module that incorporates the SoftNMS into beam search for the conflicts resolution. For the  $m$ -th span with the left index  $l_{m}$  and the right index  $r_{m}$  in the query, we obtain its prediction probability distribution  $\mathbf{p}_{\mathbf{m}}$ , label  $y_{m} = \mathrm{argmax}(\mathbf{p}_{\mathbf{m}})$  and score  $score_{m} = \max (\mathbf{p}_{\mathbf{m}})$ . Figure 3 illustrates a simplified post-processing process. In each step, we first expand all beam states (e.g., states 1-3 and 1-2 in step2 of Figure 3), and then prune new states according to the beam size. Specifically, given a beam state  $S$  containing spans  $\{l_t,r_t, score_t,y_t\}_{t=1}^T$ , for
each non-contained span  $s_i = (l_i, r_i, score_i, y_i)$ , we first calculate its decayed score  $score_i^{decay}$
$$
\operatorname {s c o r e} _ {i} ^ {\text {d e c a y}} = \operatorname {s c o r e} _ {i} * u ^ {\eta} \tag {10}
$$
$$
\eta = \sum_ {t = 1} ^ {T} \mathbb {I} (\operatorname {I o U} \left(s _ {i}, s _ {t}\right) \geq k) \tag {11}
$$
where  $\mathrm{IoU}(s_i,s_t) = \frac{|\{l_i,\ldots,r_i\} \cap \{l_t,\ldots,r_t\}|}{|\{l_i,\ldots,r_i\} \cup \{l_t,\ldots,r_t\}|}$  is the overlap ratio of two spans. The decay ratio  $u$  and threshold  $k$  are hyperparameters. Then, we expand the beam state  $S$  with the non-contained span  $s_i$  if  $score_i^{decay} > \delta$ . For example, in the step2 of Figure 3, we expand the state 1-3 to 1-3-4, while the state 1-3-2 fails to be expanded since  $score_2^{decay} = 0.58 <= \delta$ . After expanding all states, we prune available beam states with lower path scores. For example, we prune states 1-4, 3-4 and 3-2 in step2 of Figure 3. In addition, as our needed output is order-independent, we also prune duplicate states, e.g., the state 3-1 (equivalent to the state 1-3) for the diversity of beam states. When all states in the beam can not be expanded or have been expanded before but failed, we select the beam with the largest path score as the final output.
5.1 Experiments Setup
Datasets We evaluate ESD on FewNERD (Ding et al., 2021) and SNIPS (Coucke et al., 2018). FewNERD designs an annotation schema of 8 coarse-grained (e.g., 'Person') entity types and 66 fine-grained (e.g., 'Person-Artist') entity types, and constructs two tasks. One is FewNERD-INTRA, where all the entities in the training set (source domain), validation set and test set (target domain) belong to different coarse-grained types. The other is FewNERD-INTER, where only the fine-grained entity types are mutually disjoint in different sets. For the sake of sampling diversity, FewNERD adopts the  $N$ -way  $K \sim 2K$ -shot sampling method to construct tasks (each class in the support set has  $K \sim 2K$  annotated entities). Both FewNERD-INTRA and FewNERD-INTER have 4 settings, 5-way  $1 \sim 2$ -shot, 5-way  $5 \sim 10$ -shot, 10-way  $1 \sim 2$ -shot and 10-way  $5 \sim 10$ -shot. SNIPS is a slot filling dataset, which contains 7 domains  $\mathcal{D} = \{\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_7\}$ . (Hou et al., 2020) constructs few-shot slot filling task with the leave-one-out strategy, which means when testing on the target domain  $\mathcal{D}_i$ , they randomly choose  $\mathcal{D}_j (i \neq j)$
Models	FEW-NERD (INTRA)	FEW-NERD (INTER)
1 ~ 2 shot	5 ~ 10 shot	Avg.	1 ~ 2 shot	5 ~ 10 shot	Avg.
5 way	10 way	5 way	10 way	5 way	10 way	5 way	10 way
ProtoBERT	20.76±0.84	15.04±0.44	42.54±0.94	35.40±0.13	28.44	38.83±1.49	32.45±0.79	58.79±0.44	52.92±0.37	45.75
NNShot	25.78±0.91	18.27±0.41	36.18±0.79	27.38±0.53	26.90	47.24±1.00	38.87±0.21	55.64±0.63	49.57±2.73	47.83
StructShot	30.21±0.90	21.03±1.13	38.00±1.29	26.42±0.60	28.92	51.88±0.69	43.34±0.10	57.32±0.63	49.57±3.08	50.53
ESD (Ours)	36.08±1.6	30.00±0.70	52.14±1.5	42.15±2.6	40.09	59.29±1.25	52.16±0.79	69.06±0.80	64.00±0.43	61.13