Venue: NAACL 2022

# An Enhanced Span-based Decomposition Method for Few-Shot Sequence Labeling

Peiyi Wang\({}^{1}\), Runxin Xu\({}^{1}\), Tianyu Liu\({}^{2}\), Qingyu Zhou\({}^{2}\),

**Yunbo Cao\({}^{2}\), Baobao Chang\({}^{1}\), Zhifang Sui\({}^{1}\)**

\({}^{1}\) Key Laboratory of Computational Linguistics, Peking University, MOE, China

\({}^{2}\) Tencent Cloud Xiaowei

{wangpeiyi9979, runxinxuu}@gmail.com

{rogertyliu, qingyuzhou, yunbocao}@tencent.com

{chbb, szf}@pku.edu.cn

Equal contribution.Corresponding author.Contribution during internship in Tencent.

###### Abstract

Few-Shot Sequence Labeling (FSSL) is a canonical paradigm for the tagging models, e.g., named entity recognition and slot filling, to generalize on an emerging, resource-scare domain. Recently, the metric-based meta-learning framework has been recognized as a promising approach for FSSL. However, most prior works assign a label to each token based on the token-level similarities, which ignores the integrality of named entities or slots. To this end, in this paper, we propose **Esd**, an **E**nhanced **S**pan-based **D**ecomposition method for FSSL. Esd formulates FSSL as a span-level matching problem between test query and supporting instances. Specifically, Esd decomposes the span matching problem into a series of span-level procedures, mainly including enhanced span representation, class prototype aggregation and span conflicts resolution. Extensive experiments show that Esd achieves the new state-of-the-art results on two popular FSSL benchmarks, FewNERD and SNIPS, and is proven to be more robust in the nested and noisy tagging scenarios. Our code is available at [https://github.com/Wangpieyij9979/ESD](https://github.com/Wangpieyij9979/ESD).

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

## 1 Introduction

Many natural language understanding tasks can be formulated as sequence labeling tasks, such as _Named Entity Recognition_ (NER) and _Slot Filling_ (SF) tasks. Most prior works on sequence labeling follow the supervised learning paradigm, which requires large-scale annotated data and is limited to pre-defined classes. In order to generalize on the emerging, resource-scare domains, Few-Shot Sequence Labeling (FSSL) has been proposed Hou et al. (2020); Yang and Katiyar (2020). In FSSL, the model (typically trained on the source domain data) needs to solve the \(N\)-way (\(N\) unseen classes) \(K\)-shot (only \(K\) annotated examples for each class) task in the target domain. Figure 1 shows a \(2\)-way \(2\)-shot target domain few-shot NER task, where _'PER'_ and _'ORG'_ are \(2\) unseen entity types, and in the training set \(\mathcal{S}=\{S_{1},S_{2}\}\), both of them have only \(2\) annotated entities. The tagging models should annotate _'Albert Einstein'_ in the test sentence \(q\) as a _'PER'_ according to \(\mathcal{S}\).

Recently, _metric-based meta-learning (MBML) methods_ have become the mainstream and state-of-the-art methods in FSSL Hou et al. (2020); Ding et al. (2021), which train the models on the tasks sampled from the source domain sentences in order to mimic and solve the target task. In each task of training and testing, they make the prediction through modeling the similarity between the training set (support set) and the test sentence (query). Specifically, previous MBML methods Ding et al. (2021); Hou et al. (2020); Yang and Katiyar (2020) mainly formulate FSSL as a token-level matching problem, which assigns a label to each token based on the token-level similarities. For example, the token _'Albert'_ would be labeled as _'B-PER'_ due to its resemblance to _'Steve'_ and _'Isaac'_.

However, for the MBML methods, selecting the appropriate metric granularity would be fundamental to the success. We argue that prior works that focus on modeling the token-level similarity are sub-optimal in FSSL: 1) they ignore the integrality of named entities or dialog slots that are composed of a text span instead of a single token. 2) in FSSL,

Figure 1: A \(2\)-way \(2\)-shot few-shot NER task. The model needs to learn new entities with few examples.



the conditional random fields (CRF) is an important component for the token-level sequence labeling models Yang and Katiyar (2020); Hou et al. (2020), but the transition probability between classes in the target domain task can not be sufficiently learned with very few examples Yu et al. (2021). The previous methods resort to estimate the values with the abundant source domain data, which may suffer from domain shift problem Hou et al. (2020). To overcome these drawbacks of the token-level models, in this paper, we propose Esd, an Enhanced Span-based Decomposition model that formulates FSSL as a span-level matching problem between test query and support set instances.

Specifically, Esd decomposes the span matching problem into three main subsequent span-level procedures. 1) **Span representation**. We find the span representation can be enhanced by information from other spans in the same sentence and the interactions between query and support set. Thus we propose a span-enhancing module to reinforce the span representation by intra-span and inter-span interactions. 2) **Span prototype aggregation**. MBML methods usually aggregate the span vectors that belongs to the same class in the support set to form the class prototype representation. Among all class prototypes, the O-type serves as negative examples and covers all the miscellaneous spans that are not entities, which poses new challenges to identify the entity boundaries. To this end, we propose a span-prototypical module to divide O-type spans into \(3\) sub-types to distinguish the miscellaneous semantics by their relative position with recognized entities, together with a dynamically aggregation mechanism to form the specific prototype representation for each query span. 3) **Span conflicts resolution**. In the span-level matching paradigm, the predicted spans may conflict with each other. For example, a model may annotate both "Albert Einstein" and "Albert" as _"PER"_. Therefore, we propose a span refining module that incorporates the Soft-NMS Bodla et al. (2017); Shen et al. (2021) algorithm into the beam search to alleviate this conflict problem. With Beam Soft-NMS, Esd can also handle nested tagging cases without any extra training, which previous methods are incapable of.

We summarize our contribution as follows: (1) We propose Esd, an enhanced span-based decomposition model, which formulates FSSL as a span-level matching problem. (2) We decompose the span matching problem into 3 main subsequent procedures, which firstly produce enhanced span representation, then distinguish miscellaneous semantics of O-types, and achieve the specific prototype representation for each query, and finally resolve span conflicts . (3) Extensive experiments show that Esd achieves new state-of-the-art performance on both few-shot NER and slot filling benchmarks and that ESD is more robust than other methods in nested and noisy scenarios.

## 2 Related Work

### Few-Shot Learning

Few-Shot Learning (FSL) aims to enable the model to solve the target domain task with a very small training set \(D_{train}\)Wang et al. (2020). In FSL, people usually consider the \(N\)-way \(K\)-shot task, where the \(D_{train}\) has \(N\) classes, and each class has only \(K\) annotated examples (\(K\) is very small, e.g., \(5\)). Training a model only based on \(D_{train}\) from scratch will inevitably lead to over-fitting. Therefore, researchers usually introduce the source domain annotated data to help train models, e.g., Few-Shot Relation Classification (FSRC) Han et al. (2018), Few-Shot Text Classification (FSTC) Geng et al. (2019) and Few-Shot Event Classification (FSEC) Wang et al. (2021a). The source domain data do not contain examples belonging to classes in \(D_{train}\), and thus the FSL setting can be guaranteed. Specifically, FSSL tasks in our paper also have the source domain data.

### Metric-Based Meta-Learning

Meta-learning Hochreiter et al. (2001) is a popular method to deal with Few-Shot Learning (FSL). Meta-learning constructs a series of tasks sampled from the source domain data to mimic the target domain task, and trains models across these sampled tasks. Each task contains a training set (support set) and a test instance (query). The core idea of meta-learning is to help model learn the ability to quickly adapt to new tasks, i.e., learn to learn Hochreiter et al. (2001). Meta-learning can be combined with the metric-learning Kulis et al. (2013) (metric-based meta-learning), which makes predictions based on the similarity of the support set and the query. For example, Prototypical Network Snell et al. (2017) first learns prototype vectors from a few examples in the support set for classes, and further uses prototype vectors for query prediction. Specifically, Esd (our model) follows this 

metric-based meta-learning paradigm.

### Few-Shot Sequence Labeling

Recently, few shot sequence labeling (FSSL) has been widely explored by researchers. For example, Fritzler et al. (2019) leverages the prototypical network Snell et al. (2017) in the few-shot NER. Yang and Katiyar (2020) further proposes a cheap but effective method to capture the label dependencies between entity tags without expensive CRF training. Wang et al. (2021) utilizes a large unlabelled dataset and proposes a distillation method. Cui et al. (2021) introduces a prompt method to tap the potential of BART Lewis et al. (2020). Hou et al. (2020) extends the TapNet Yoon et al. (2019) and proposes a collapsed CRF for few-shot slot filling. Ma et al. (2021) and Athiwaratkun et al. (2020) formulate FSSL as a machine comprehension problem and a generation problem, respectively. Yu et al. (2021) proposed to retrieve the most similar exemplars in the support set for span-level prediction. Although their methods also involve span matching, their main focus is on the retrieval-augmented training. Our work differs from Yu et al. (2021) in that we propose an effective actionable pipeline to get enhanced span representation, handle miscellaneous semantics of O-types and resolve the potential span conflicts in both non-nested and nested situations, where the last two modules are essential to align the candidate spans with class prototypes in the support set but missing in Yu et al. (2021). Besides, our enhanced span representation greatly improves the batched softmax objective of Yu et al. (2021) by inter- and intra-span interactions.

## 3 Task Formulation

We define a sentence as \({\bf x}\) and its label as \({\bf y}=\{(s_{i},y_{i})\}_{i=1}^{M}\), where \(s_{i}\) is a span of \({\bf x}\) (e.g., '_Steve Jobs_'), \(y_{i}\) is the label of \(s_{i}\) ( e.g., '_PER_') and \(M\) is the number of spans in the \({\bf x}\). Following the previous FSSL setting Hou et al. (2020); Ding et al. (2021), we have data in source domain \(\mathcal{D}_{source}\) and target domain \(\mathcal{D}_{target}\), and models are evaluated on tasks from \(\mathcal{D}_{target}\). Meta-learning based FSSL has two stages, meta-training and meta-testing. In the meta-training stage, the model is trained on tasks sampled from \(\mathcal{D}_{source}\) to mimic the test situation, and in the meta-testing stage, the model is evaluated on test tasks. A task is defined as \(\mathcal{T}=\{\mathcal{S},q\}\), consisting of a support set \(\mathcal{S}=\{({\bf x}_{i},{\bf y}_{i})\}_{i=1}^{I}\), and a query sentence \(q={\bf x}\). \(\mathcal{S}\) includes \(N\) types of entities or slots (\(N\)-way), and each type has \(K\) annotated examples (\(K\)-shot). For spans that do not belong to the \(N\) types, e.g., '_studied at_' in Figure 1, we set their label to O. The types except O in each test task are guaranteed to not exist in the training tasks. Given a task \(\mathcal{T}=\{\mathcal{S},q\}\), the model needs to assign a label to each span in the query sentence \(q\) based on \(\mathcal{S}\).

## 4 Methodology

Our Esd formulates FSSL as a span-level matching problem and decomposes it into a series of span-related procedures for a better span matching. Figure 2 illustrates the architecture of Esd.

Figure 2: The architecture (5 span modules) of Esd with a \(2\)-way (‘PER’ and ‘ORG’) \(2\)-shot input task. We only list spans with lengths less than \(2\) for clarity. Esd assigns label ‘PER’ to the span “Albert Einstein” and ‘O’ to the other spans in query \(q\) based on the support set \(\{S_{1},S_{2}\}\).

 



### Span Initialization Module

Given a task \({\cal T}=\left\{{\cal S},q\right\}\), we use BERT Devlin et al. (2019) to encode sentences in \({\cal S}\) and \(q\), and utilize the output of the last layer to represent tokens in the sentence. Therefore, for a sentence with \(N\) tokens \({\bf x}=(x_{1},x_{2},...,x_{N})\), we can achieve representations \(({\bf h_{1}},{\bf h_{2}},...,{\bf h_{N}})\), where \({\bf h_{i}}\in\mathbb{R}^{d_{w}}\) is the hidden state corresponding to the token \(x_{i}\). Then, for a span \(s=(l,r)\), where \(l\) and \(r\) are the start index and end index of span \(s\) in the sentence \({\bf x}\), we obtain its initial representation \({\bf s}_{(l,r)}=[{\bf h_{j}};{\bf h_{r}}]{\bf W_{s}}\).1 We enumerate spans in the sentence with a maximum length of \(L\).

Footnote 1: We omit the bias term in this paper for clarity.

### Span Enhancing Module

#### 4.2.1 Intra Span Attention

Intuitively, the meaning of specific spans can be inferred from other spans in the same sentence. We thus design an Intra Span Attention (ISA) mechanism. Given all the span representations of a sentence \({\bf S}\in\mathbb{R}^{B\times d}\) (\(B\) is the number of spans). We denote the \(i\)-th row of \({\bf S}\) as \({\bf s_{i}}\), which represents the \(i\)-th span in the sentence. For \({\bf s_{i}}\), we first get its ISA span representation \(\mathbf{\bar{s}_{i}}=\sum_{j=1}^{B}\alpha_{ij}{s_{\textbf{j}}}\), where \(\alpha_{\textbf{i}}=\mathrm{softmax}({\bf s_{i}S^{T}})\). For clarity, we denote this attention aggregation operation as \(\phi\), i.e.,

\[\mathbf{\bar{s}_{i}}=\phi({\bf s_{i}},{\bf S}) \tag{1}\]

then, a Feed Forward Neural Networks (FFN) Vaswani et al. (2017) with residual connection He et al. (2016) and layer normalization Ba Kiros and Hinton (2016) are used to get the final ISA enhanced feature \(\mathbf{\tilde{s}_{i}}\),

\[\mathbf{\tilde{s}_{i}}=\mathrm{LayerNorm}({\bf s_{i}}+\mathrm{ FFN}_{\text{isa}}(\mathbf{\bar{s}_{i}})) \tag{2}\] \[\mathrm{FFN}_{\text{isa}}(\mathbf{\bar{s}_{i}})=\mathrm{GELU}( \mathbf{\bar{s}_{i}W^{1}_{isn}})\mathbf{W^{2}_{isn}} \tag{3}\]

#### 4.2.2 Cross Span Attention

After ISA, to accommodate the span matching between the test query and supporting sentences, and facilitate the inter-span interaction, we propose Cross Span Attention (CSA) to enhance query spans \(\mathbf{\tilde{Q}}\in\mathbb{R}^{B_{q}\times d}\) with the support set spans \(\{\mathbf{\tilde{S}_{i}}\in\mathbb{R}^{B_{i}\times d};i=1,...,I\}\), and vice versa. We first concatenate all span representations in the support set into one matrix \(\mathbf{\tilde{S}}=[\mathbf{\tilde{S}_{1}},\mathbf{\tilde{S}_{2}},..., \mathbf{\tilde{S}_{1}}]\in\mathbb{R}^{B_{s}\times d}\). We denote the \(n\)-th row of \(\mathbf{\tilde{S}}\) as \(\mathbf{\tilde{s}_{n}}\) and the \(m\)-th row of \(\mathbf{\tilde{Q}}\) as \(\mathbf{\tilde{q}_{m}}\), and obtain their CSA span representations \(\mathbf{\hat{s}_{n}}=\phi(\mathbf{\tilde{s}_{n}},\mathbf{\tilde{Q}})\) and \(\mathbf{\hat{q}_{m}}=\phi(\mathbf{\tilde{q}_{m}},\mathbf{\tilde{S}})\). Then, we get the final CSA enhanced representation \(\mathbf{\tilde{s}_{n}}\) and \(\mathbf{\tilde{q}_{m}}\) as follows,

\[\mathbf{\tilde{s}_{n}}=\mathrm{LayerNorm}(\mathbf{\tilde{s}_{n}}+ \mathrm{FFN}_{\text{csa}}(\mathbf{\hat{s}_{n}})) \tag{4}\] \[\mathbf{\check{q}_{m}}=\mathrm{LayerNorm}(\mathbf{\tilde{q}_{m}}+ \mathrm{FFN}_{\text{csa}}(\mathbf{\hat{q}_{m}}))\] (5) \[\mathrm{FFN}_{\text{csa}}(\mathbf{x})=\mathrm{GELU}(\mathbf{x} \mathbf{W^{1}_{esa}})\mathbf{W^{2}_{esa}} \tag{6}\]

### Span Prototypical Module

#### 4.3.1 Instance Span Attention

Since different support set spans play different roles for a query span, we propose multi-INstance Span Attention (INSA) to get class representations. For the \(i\)-th class that contains \(K\) annotated spans with enhanced representations \(\mathbf{\tilde{S}_{i}}=[\mathbf{\tilde{s}_{i}^{1}},...,\mathbf{\tilde{s}_{i}^{ K}}]\) in the support set, given a query span \(\mathbf{\check{q}_{m}}\), INSA gets the corresponding prototypical representation \(\mathbf{z_{m}^{i}}=\phi(\mathbf{\check{q}_{m}},\mathbf{\tilde{S}_{i}})\).

#### 4.3.2 O Partition and Prototypical Span Attention

The O-type spans have huge quantities and miscellaneous semantics, which is hard to be represented by only one prototypical vector. In the span-based framework, considering the boundary information is essential for a span, we divide the O-type spans into \(3\) sub-classes according to their boundary, to alleviate their miscellaneous semantics problem. Specifically, given a sentence with \(I\) annotated spans \(\{(l_{i},r_{i})\}_{i=1}^{I}\), where \(l_{i}\) and \(r_{i}\) are the left and right boundary of the \(i\)-th annotated span. For each of the other spans \((l_{o},r_{o})\) , we assign it a sub-class O\({}_{sub}\) as follows,

\[O_{sub}=\left\{\begin{array}{ll}O_{1},&\forall i,s.t.~{}r_{o}<l_{i}\lor l_{ o}>r_{i}\\ O_{2},&\exists i,s.t.~{}l_{o}\geq l_{i}\wedge r_{o}\leq r_{i}\\ O_{3},&\mathrm{Others}\end{array}\right. \tag{7}\]

where \(O_{1}\) denotes the span that does not overlap with any entities or slots in the sentence, e.g., _"study at"_ in \(S_{2}\) of Figure 2, and \(O_{2}\) represents the span that is the sub-span of an entity or slot, e.g., _"Isaac"_ in \(S_{2}\) of Figure 2. After O Partition, we get the prototypical representation of each O\({}_{sub}\) via INSA, thus for a query span \(\mathbf{\check{q}_{m}}\), we have 3 sub-class representations \(\mathbf{Z_{m}^{o}}=[\mathbf{z_{m}^{o_{1}}},\mathbf{z_{m}^{o_{2}}},\mathbf{z_{m}^ {o_{3}}}]\) for the class O. Then, we utilize Prototypical Span Attention (PSA) to achieve the final O representation \(\mathbf{z_{m}^{o}}=\phi(\mathbf{\check{q}_{m}},\mathbf{Z_{m}^{o}})\).

### Span Matching Module

Given a task \({\cal T}=\left({\cal S},q\right)\), for the \(m\)-th span in \(q_{m}\), we achieve its enhanced representation

and corresponding prototypical vectors \({\bf Z_{m}}=({\bf z^{o}_{m},z^{1}_{m},...,z^{N}_{m}})\) through previous span modules. Then, we predict \(q_{m}\) as the type \(z_{k}\) in the support set with probability,

\[p(y_{m}=z_{k}|q_{m})=\frac{exp(-L_{2}(\bar{\mathbf{q}_{m}},\mathbf{z^{m}_{k}}))}{ \sum_{k^{\prime}}exp(-L_{2}(\bar{\mathbf{q}_{m}},\mathbf{z^{m}_{k^{\prime}}}))} \tag{8}\]

where \(L_{2}\) is the euclidean distance. During training, we use cross-entropy as our loss function,

\[\mathcal{L}=-\frac{1}{B_{q}}\sum_{m=1}^{B_{q}}log~{}p(y^{*}_{m}|q_{m}) \tag{9}\]

where \(y^{*}_{m}\) is the gold label of \(q_{m}\) and \(B_{q}\) is the number of spans in the query \(q\).

### Span Refining Module in Inference

During inference, spans outputted by the matching module may have conflicts, we thus propose a refining module that incorporates the SoftNMS into beam search for the conflicts resolution. For the \(m\)-th span with the left index \(l_{m}\) and the right index \(r_{m}\) in the query, we obtain its prediction probability distribution \(\mathbf{p_{m}}\), label \(y_{m}=\operatorname{argmax}(\mathbf{p_{m}})\) and score \(score_{m}=\max(\mathbf{p_{m}})\). Figure 3 illustrates a simplified post-processing process. In each step, we first expand all beam states (e.g., states 1-3 and 1-2 in step2 of Figure 3), and then prune new states according to the beam size. Specifically, given a beam state \(S\) containing spans \(\left\{l_{t},r_{t},score_{t},y_{t}\right\}_{t=1}^{T}\), for each non-contained span \(s_{i}=\left(l_{i},r_{i},score_{i},y_{i}\right)\), we first calculate its decayed score \(score_{i}^{decay}\),

\[score_{i}^{decay}=score_{i}*u^{\eta} \tag{10}\] \[\eta=\sum_{t=1}^{T}\mathbb{I}(\mathrm{IoU}(s_{i},s_{t})\geq k) \tag{11}\]

where \(\mathrm{IoU}(s_{i},s_{t})=\frac{\left|\{l_{i},...,r_{t}\}\cap\{l_{t},...,r_{t} \}\right|}{\left|\{l_{i},...,r_{t}\}\cup\{l_{t},...,r_{t}\}\right|}\) is the overlap ratio of two spans. The decay ratio \(u\) and threshold \(k\) are hyperparameters. Then, we expand the beam state \(S\) with the non-contained span \(s_{i}\) if \(score_{i}^{decay}>\delta\). For example, in the step2 of Figure 3, we expand the state 1-3 to 1-3-4, while the state 1-3-2 fails to be expanded since \(score_{2}^{decay}=0.58<=\delta\). After expanding all states, we prune available beam states with lower path scores. For example, we prune states 1-4, 3-4 and 3-2 in step2 of Figure 3. In addition, as our needed output is order-independent, we also prune duplicate states, e.g., the state 3-1 (equivalent to the state 1-3) for the diversity of beam states. When all states in the beam can not be expanded or have been expanded before but failed, we select the beam with the largest path score as the final output.

## 5 Experiments

### Experiments Setup

DatasetsWe evaluate Esd on FewNERD Ding et al. (2021) and SNIPS Coucke et al. (2018). **FewNERD** designs an annotation schema of \(8\) coarse-grained (e.g., 'Person') entity types and \(66\) fine-grained (e.g., 'Person-Artist') entity types, and constructs two tasks. One is **FewNERD-INTRA**, where all the entities in the training set (source domain), validation set and test set (target domain) belong to different coarse-grained types. The other is **FewNERD-INTER**, where only the fine-grained entity types are mutually disjoint in different sets. For the sake of sampling diversity, FewNERD adopts the \(N\)-way \(K\sim 2K\)-shot sampling method to construct tasks (each class in the support set has \(K\sim 2K\) annotated entities). Both FewNERDNINTRA and FewNERD-INTER have \(4\) settings, \(5\)-way \(1\sim 2\)-shot, \(5\)-way \(5\sim 10\)-shot, \(10\)-way \(1\sim 2\)-shot and \(10\)-way \(5\sim 10\)-shot. **SNIPS** is a slot filling dataset, which contains \(7\) domains \(\mathcal{D}=\left\{\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{7}\right\}\). Hou et al. (2020) constructs few-shot slot filling task with the leave-one-out strategy, which means when testing on the target domain \(\mathcal{D}_{i}\), they randomly chose \(\mathcal{D}_{j}(i\neq j)\)

Figure 3: A demonstration for Beam Soft-NMS (BSnMS). For clarity, we set the filter threshold \(\delta\) to \(0.6\), and suppose the span score is always decayed by the overlapped spans with a constant decayed score \(-0.1\) (Note that the example is simplified for demonstration). BSNMS filters false positive spans (colored in orange) in this demonstration. A more clear step by step demonstration is included in Appendix D.

 

as the validation domain, and train the model on source domains \(\mathcal{D}-\left\{\mathcal{D}_{i},\mathcal{D}_{j}\right\}\). In the sampling task of SNIPS, all classes have \(K\) annotated examples in the support set, but the number of them (\(N\)) is not fixed. The few-shot slot filling task in each domain of SNIPS has two settings, 1-shot and 5-shot. FSSL models are trained and evaluated on tasks sampled from the source and target domain, respectively. To ensure the fairness, we use the public sampled data provided by Ding et al. (2021) for FewNERd2 and data provided by Hou et al. (2020) for SNIPS, to train and evaluate our model.

Footnote 2: The FewNERDA data we used is from [https://cloud.tsinghua.edu.cn/f/0e38bd108d7b49808cc4/2dl=1](https://cloud.tsinghua.edu.cn/f/0e38bd108d7b49808cc4/2dl=1), which corresponds to the results reported in [https://arxiv.org/pdf/2105.07464v6.pdf](https://arxiv.org/pdf/2105.07464v6.pdf).

Parameter SettingsFollowing previous methods Hou et al. (2020); Ding et al. (2021), we use uncased BERT-base as our encoder. We use Adam Kingma and Ba (2015) as our optimizer. We set the dropout ratio Srivastava et al. (2014) to \(0.1\). The maximum span length \(L\) is set to \(8\). For BSNMS, the beam size \(b\) is 5. Because FewNERD and SNIPS do not have nested instances, we set the threshold to filter false positive spans \(\delta\) to \(0.1\), the threshold to decay span scores \(k\) to \(1e\text{-}5\) and the decay ratio \(u\) to \(1e\text{-}5\) to force the refining results have no nested spans. More details of our parameter settings are provided in Appendix A.

Evaluation MetricsFor FewNERD, following Ding et al. (2021), we report the micro F1 over all test tasks, and the average result of \(5\) different runs. For SNIPS, following Hou et al. (2020), we first calculate micro F1 score for each test episode (an episode contains a number of test tasks), and then report the average F1 score for all test episodes as the final result. We report the average result of \(10\) different runs the same as Hou et al. (2020).

BaselinesFor systematic comparisons, we introduce a variety of baselines, including **ProtoBERT**Ding et al. (2021); Hou et al. (2020), **NNShot**Ding et al. (2021), **StructShot**Ding et al. (2021), **TransferBERT**Hou et al. (2020), **MN+BERT**Hou et al. (2020), **L-TapNet++CDT**Hou et al. (2020), **Retriever**Yu et al. (2021), **ConVEx**Henderson

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**FewNERD (intra)**} & \multicolumn{3}{c}{**Few-NERD (inter)**} \\ \cline{2-10} \multirow{3}{*}{**Models**} & \multicolumn{3}{c}{\(1\sim 2\) shot} & \multicolumn{3}{c}{\(5\sim 10\) shot} & \multicolumn{3}{c}{\(1\sim 2\) shot} & \multicolumn{3}{c}{\(5\sim 10\) shot} \\ \cline{2-10}  & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} \\ \cline{2-10}  & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} \\ \cline{2-10}  & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} & \multicolumn{3}{c}{—} \\ \hline ProtoBERT & 20.76\({}_{\pm 0.94}\) & 15.04\({}_{\pm 0.44}\) & 42.54\({}_{\pm 0.44}\) & 35.40\({}_{\pm 0.13}\) & 28.44 & 38.83\({}_{\pm 1.00}\) & 32.45\({}_{\pm 0.79}\) & 58.79\({}_{\pm 0.64}\) & 52.92\({}_{\pm 0.77}\) & 45.75 \\ NNShot & 25.78\({}_{\pm 0.91}\) & 18.27\({}_{\pm 0.61}\) & 36.18\({}_{\pm 0.79}\) & 27.38\({}_{\pm 0.53}\) & 26.90 & 47.42\({}_{\pm 1.00}\) & 38.87\({}_{\pm 0.21}\) & 55.64\({}_{\pm 0.64}\) & 49.57\({}_{\pm 2.73}\) & 47.83 \\ StructShot & 30.21\({}_{\pm 0.60}\) & 21.03\({}_{\pm 1.19}\) & 38.00\({}_{\pm 1.39}\) & 26.42\({}_{\pm 0.60}\) & 28.92 & 51.88\({}_{\pm 0.60}\) & 43.34\({}_{\pm 0.50}\) & 57.32\({}_{\pm 0.64}\) & 49.57\({}_{\pm 0.56}\) & 50.53 \\ Esd (Ours) & **36.08\({}_{\pm 1.6}\)** & **30.00\({}_{\pm 0.49}\)** & **52.14\({}_{\pm 1.85}\)** & **40.09** & **59.29\({}_{\pm 1.85}\)** & **52.16\({}_{\pm 0.79}\)** & **69.06\({}_{\pm 0.88}\)** & **64.00\({}_{\pm 0.63}\)** & **61.13** \\ \hline \hline \end{tabular}
\end{table}
Table 1: F1 scores with standard deviations on FewNERD. The best results are in **boldface**.

 

and Vulic, 2021) and **Ma2021**(Ma et al., 2021). Please refer to the Appendix A for more details.

### Main Results

Table 1 illustrates the results on FewNERD. As is shown, among all task settings, Esd consistently outperforms ProtoBERT, NNShot and StructShot by a large margin. For example, compared with StructShot, Esd achieves \(11.17\) and \(10.60\) average F1 improvement on INTRA and INTER, respectively. Table 2 shows the results on SNIPS. In the \(1\)-shot setting, L-TapNet+CDT is the previous best method. Compared with L-TapNet+CDT, Esd achieves comparable results and \(4.58\) F1-scores improvement in the \(1\)-shot and \(5\)-shot settings, respectively. We think the reason is that the information benefits brought by our cross span attention mechanism in \(1\)-shot setting is much less than that in \(5\)-shot setting. In addition, compared with L-TapNet+CDT, Esd performs more stable, and also has a better model efficiency (Please refer to Section 6.3). In \(5\)-shot setting, Ma2021 is previous best method, and Esd outperforms it \(1.14\) and \(0.42\) F1-scores in \(1\)-shot and \(5\)-shot settings, respectively. These results prove the effectiveness of Esd in few-shot sequence labeling.

## 6 Analysis

### Ablation Study

To illustrate the effect of our proposed mechanisms, we conduct ablation studies by removing one component of Esd at a time. Table 3 shows the results on the validation set of SNIPS (\(1\)-shot, domain "We"). Firstly, we remove ISA or CSA, which means the span cannot be aware of other spans within the same sentence or spans from other sentences. As shown in Table 3, the average F1 scores drop \(0.7\) and \(2.6\) without ISA and CSA, respectively. These results suggest that our span enhancing module with ISA and CSA is effective. In addition, CSA is more effective than ISA, since CSA can enhance query spans with whole support set spans, while ISA enhances spans only with other spans in the same sentence. CSA brings much more information than ISA. Secondly, when we remove INSA and achieve the prototypical representation of a class through an average operation, the average F1 score would drop \(3.5\). When we do not consider the sub-classes of O-type spans (_r.m._ OP), the average F1 score would drop \(1.1\). These results show that our span prototypical module is necessary. At last, the result without BSNMS suggests the importance of our post-processing algorithm in this span-level few-shot labeling framework. Moreover, with BSNMS, Esd can also easily adapt to the nested situation.

### Robustness in the Nested Situation

Sequence labeling tasks such as NER can have nested situations. For example, in the sentence "Isaac Newton studied at Cambridge University", where "Cambridge" is a location and "Cambridge University" is an organization. However, both FewNERD and SNIPS do not annotate nested examples. To explore the robustness of Esd on such a challenging but common situation, we construct **FewNERD-nested**, which has the same training tasks as FewNERD-INTRA, but different test tasks with a nested ratio \(r_{nested}\). In FewNERD-nested, we sample each test task either from FewNERD or from GENIA Ohta et al. (2002) with the probability \(1-r_{nested}\) and \(r_{nested}\), respectively, and all tasks sampled from GENIA are guaranteed to have the query with nested entities. We sample validation tasks with nested instances from

Figure 4: The performance drops in nested situations when increasing \(r_{nest}\). \(\Delta\)F1% is the F1 reduction percent over \(r_{nested}=0\).

\begin{table}
\begin{tabular}{l l} \hline \hline
**Ablation Models** & **F1** \\ \hline Esd & 61.7\({}_{\pm 1.3}\) \\ \hline _r.m._ intra span attention (ISA) & 61.0\({}_{\pm 0.4}\) \\ _r.m._ cross span attention (CSA) & 59.1\({}_{\pm 1.1}\) \\ \hline _r.m._ instance span attention (INSA) & 58.2\({}_{\pm 1.3}\) \\ _r.m._ Op partition (OP) & 60.6\({}_{\pm 1.3}\) \\ \hline _r.m._ beam soft-nms (BSNMS) & 57.3\({}_{\pm 1.4}\) \\ \hline \end{tabular}
\end{table}
Table 3: The effect of our proposed mechanisms on the validation set of SNIPS (1-shot, domain “We”). We report the average result of 3 different runs with standard deviations. _r.m._ denotes _remove_.

 

ACE05 (Walker et al., 2006) to tune \(k\), \(u\) and \(\delta\) in BSNMS. Please refer to the Appendix B for more details about FewNERD-nested. Figure 4 shows the results of Esd and several typical baselines in FewNERD-nested with different \(r_{nested}\). When \(r_{nested}\) increases, Esd is more stable than previous methods, since Esd with BSNMS can easily extend to nested tagging cases without any extra training while previous methods are incapable of. In addition, we also compare different post-processing methods when \(r_{nested}=1\). As shown in Table 4, the beam search method can not handle the nested situation, and thus it harms the performance. SoftNMS (Shen et al., 2021) and BSNMS can both improve the model performance. However, when incorporating beam search into SoftNMS, the model can be more flexible and avoid some local optimal post-processing results achieved by SoftNMS (e.g., spans \(1\),\(3\) and \(4\) in Figure 3), and thus BSNMS outperforms SoftNMS 3.

Footnote 3: Esd is also more robust than baselines in the noisy situation. Please refer to Appendix C for details.

### Model Efficiency

Compared with the token-level models, Esd needs to enumerate all spans within the length \(L\) for a sentence. Therefore, the number of spans is approximately \(L\) times that of tokens, which may bring extra computation overhead. To evaluate the efficiency of Esd, we compare the average inference time per task of Esd (including the BSNMS post-processing process), L-TapNet+CDT (the state-of-the-art token-level baseline model with the open codebase in the SNPIS) and ProtoBERT (an extremely simple token-level baseline). As shown in Table 5, with exactly the same hardware setting, in the domain "We" of SNIPS 1-shot setting, Esd (avg. \(8.53\) ms per task) is nearly 3 times faster than L-TapNet+CDT (avg. \(24.67\) ms per task). We see a similar tendency in the 5-shot setting. Although Esd (avg. \(8.53\) and \(18.47\) ms in 1- and 5-shot per task) is slower than ProtoBERT (avg. \(3.13\) and \(5.27\) ms), it outperforms ProtoBERT by \(31.53\) and \(16.68\) F1 scores in 1- and 5-shot settings (reported in Table 2) with a bearable latency. Besides the inference time, we also compare the parameter number of these models. As is shown, the added parameter scale (span enhancing and prototypical modules) is very small (2M) compared with the ProtoBERT and L-TapNet+CDT (110M). These results show that Esd has an acceptable efficiency.

### Error Analysis

To further explore what types of errors the model makes in detail, we divide error of model prediction into \(2\) categories, 'FP-Span' and 'FP-Type'. As shown in Table 6, Esd outperforms baselines and has much less false positive prediction errors. 'FP-Span' is the major prediction error for all models, showing that it is hard to locate the right span boundary in FSSL for existing models. Therefore, we should pay more attention to the span recognition in the future work. However, compared with previous methods, Esd has less ratio of the 'FP-span' error. We think the reason is that our span-level matching framework with a series of spam-related procedures has a better perception of the entity and slot spans than that of our baselines.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Models** & **F1** & **Total** & **FP-Span** & **FP-Type** \\ \hline Esd & 59.29 & 9.4k & 72.8\% & 27.2\% \\ \hline ProtoBERT & 38.83 & 30.4k & 86.7\% & 13.3\% \\ NNShot & 47.24 & 21.7k & 84.7\% & 15.3\% \\ StructShot & 51.88 & 14.5k & 80.0\% & 20.0\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Error analysis of \(5\) way \(1\sim 2\) shot on FewNERD-INTER. ‘FP-Span’ denotes extracted entities with the wrong span boundary, and ‘FP-Type’ represents extracted entities with the right span boundary but the wrong entity type. ’Total’ denotes the total wrong prediction of two types.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Models** & **F1** \\ \hline Esd (with BSNMS) & 31.25 \\ \hline Esd (with SoftNMS) & 31.09 \\ Esd (with Beam Search) & 28.94 \\ Esd (without Post-processing) & 30.34 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison between different post-processing methods in the nested situation with \(r_{nested}=1\).

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**\#Para.**} & \multicolumn{2}{c}{**Inference Time**} \\ \cline{3-4}  & & **1-shot** & **5-shot** \\ \hline Esd & 112M & 8.53 ms & 18.47 ms \\ \hline ProtoBERT & 110M & 3.13 ms & 5.27 ms \\ L-TapNet+CDT & 110M & 24.67 ms & 54.19 ms \\ \hline \hline \end{tabular}
\end{table}
Table 5: The parameter number and inference time per task of Esd and L-TapNet+CDT in the domain “We” of SNIPS. Although Esd is slower than ProtoBERT, Esd outperforms ProtoBERT by \(31.53\) and \(16.68\) F1 scores in 1- and 5-shot settings with a bearable latency.

 

Conclusion

In this paper, we propose Esd, an enhanced span-based decomposition model for few-shot sequence labeling (FSSL). To overcome the drawbacks of previous token-level methods, Esd formulates FSSL as a span-level matching problem, and decomposes it into a series of span-related procedures, mainly including span representation, class prototype aggregation and span conflicts resolution for a better span matching. Extensive experiments show that Esd achieves the state-of-the-art performance on two popular few-shot sequence labeling benchmarks and that ESD is more robust than previous models in the noisy and nested situation.

## Acknowledgements

The authors would like to thank the anonymous reviewers for their thoughtful and constructive comments. This paper is supported by the National Key Research and Development Program of China under Grant No. 202AAA0106700, the National Science Foundation of China under Grant No.61936012 and 61876004, and NSFC project U19A2065.

## References

* Athiwaratkun et al. (2020) Athiwaratkun, B.; Nogueira dos Santos, C.; Krone, J.; and Xiang, B. 2020. Augmented Natural Language for Generative Sequence Labeling. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, 375-385. Online: Association for Computational Linguistics.
* Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016. Layer normalization. _arXiv preprint arXiv:1607.06450_.
* Bodla et al. (2017) Bodla, N.; Singh, B.; Chellappa, R.; and Davis, L. S. 2017. Soft-NMS-improving object detection with one line of code. In _Proceedings of the IEEE international conference on computer vision_, 5561-5569.
* Coucke et al. (2018) Coucke, A.; Saade, A.; Ball, A.; Bluche, T.; Caulier, A.; Leroy, D.; Doumouro, C.; Gisselbrecht, T.; Caltagirone, F.; Lavril, T.; et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. _arXiv preprint arXiv:1805.10190_.
* Cui et al. (2021) Cui, L.; Wu, Y.; Liu, J.; Yang, S.; and Zhang, Y. 2021. Template-Based Named Entity Recognition Using BART. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, 1835-1845. Online: Association for Computational Linguistics.
* Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, 4171-4186. Minneapolis. Minnesota: Association for Computational Linguistics.
* Ding et al. (2021) Ding, N.; Xu, G.; Chen, Y.; Wang, X.; Han, X.; Xie, P.; Zheng, H.; and Liu, Z. 2021. Few-NERD: A Few-shot Named Entity Recognition Dataset. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, 3198-3213. Online: Association for Computational Linguistics.
* Fritzler, Logacheva, and Kretov (2019a) Fritzler, A.; Logacheva, V.; and Kretov, M. 2019a. Few-shot classification in named entity recognition task. In _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_, 993-1000.
* Fritzler, Logacheva, and Kretov (2019b) Fritzler, A.; Logacheva, V.; and Kretov, M. 2019b. Few-shot classification in named entity recognition task. In _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_, 993-1000.
* Geng et al. (2019) Geng, R.; Li, B.; Li, Y.; Zhu, X.; Jian, P.; and Sun, J. 2019. Induction Networks for Few-Shot Text Classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, 3904-3913. Hong Kong, China: Association for Computational Linguistics.
* Han et al. (2018) Han, X.; Zhu, H.; Yu, P.; Wang, Z.; Yao, Y.; Liu, Z.; and Sun, M. 2018. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, 4803-4809. Brussels, Belgium: Association for Computational Linguistics.
* He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, 770-778.
* Henderson and Vulic (2021) Henderson, M.; and Vulic, I. 2021. ConVEx: Data-Efficient and Few-Shot Slot Labeling. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, 3375-3389. Online: Association for Computational Linguistics.
* Hochreiter, Younger, and Conwell (2001a) Hochreiter, S.; Younger, A. S.; and Conwell, P. R. 2001a. Learning to learn using gradient descent. In _International Conference on Artificial Neural Networks_, 87-94. Springer.
* Hochreiter, Younger, and Conwell (2018) Hochreiter, S.; Younger, A. S.; and Conwell, P. R. 2018. Learning to learn using gradient descent. In _International Conference on Artificial Neural Networks_, 87-94. Springer 

Hou, Y.; Che, W.; Lai, Y.; Zhou, Z.; Liu, Y.; Liu, H.; and Liu, T. 2020. Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, 1381-1393. Online: Association for Computational Linguistics.
* Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015. Adam: A Method for Stochastic Optimization. _CoRR_, abs/1412.6980.
* Kulis et al. (2013) Kulis, B.; et al. 2013. Metric Learning: A Survey. _Foundations and Trends\(@\) in Machine Learning_, 5(4): 287-364.
* Lewis et al. (2020) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, 7871-7880.
* Ma et al. (2021) Ma, J.; Yan, Z.; Li, C.; and Zhang, Y. 2021. Frustratingly Simple Few-Shot Slot Tagging. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, 1028-1033. Online: Association for Computational Linguistics.
* Ohta et al. (2002) Ohta, T.; Tateisi, Y.; Kim, J.-D.; Mima, H.; and Tsujii, J. 2002. The GENIA corpus: An annotated research abstract corpus in molecular biology domain. In _Proceedings of the human language technology conference_, 73-77. Citeseer.
* Shen et al. (2021) Shen, Y.; Ma, X.; Tan, Z.; Zhang, S.; Wang, W.; and Lu, W. 2021. Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, 2782-2794.
* Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017. Prototypical networks for few-shot learning. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, 4080-4090.
* Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural networks from overfitting. _The journal of machine learning research_, 15(1): 1929-1958.
* Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is all you need. In _Advances in neural information processing systems_, 5998-6008.
* Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; kavukcuoglu, k.; and Wierstra, D. 2016. Matching Networks for One Shot Learning. In Lee, D.; Sugiyama, M.; Luxburg, U.; Guyon, I.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc.
* Walker et al. (2006) Walker, C.; Strassel, S.; Medero, J.; and Maeda, K. 2006. ACE 2005 multilingual training corpus. _Linguistic Data Consortium, Philadelphia_, 57: 45.
* Wang et al. (2021a) Wang, P.; Xun, R.; Liu, T.; Dai, D.; Chang, B.; and Sui, Z. 2021a. Behind the Scenes: An Exploration of Trigger Biases Problem in Few-Shot Event Classification. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 1969-1978.
* Wang et al. (2021b) Wang, Y.; Mukherjee, S.; Chu, H.; Tu, Y.; Wu, M.; Gao, J.; and Awadallah, A. H. 2021b. Meta Self-training for Few-shot Neural Sequence Labeling. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 1737-1747.
* Wang et al. (2020) Wang, Y.; Yao, Q.; Kwok, J. T.; and Ni, L. M. 2020. Generalizing from a few examples: A survey on few-shot learning. _ACM Computing Surveys (CSUR)_, 53(3): 1-34.
* Yang and Katiyar (2020) Yang, Y.; and Katiyar, A. 2020. Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, 6365-6375. Online: Association for Computational Linguistics.
* Yoon, Seo, and Moon (2019) Yoon, S. W.; Seo, J.; and Moon, J. 2019. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In _International Conference on Machine Learning_, 7115-7123. PMLR.
* Yu et al. (2021) Yu, D.; He, L.; Zhang, Y.; Du, X.; Pasupat, P.; and Li, Q. 2021. Few-shot Intent Classification and Slot Filling with Retrieved Examples. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, 734-749. Online: Association for Computational Linguistics.

 

Experiments

### Baselines

We compare Esd with a variety of baselines as follows:

* **TransferBERT**Hou et al. (2020) is a fine-tuning based model, which is a direct application of BERT Devlin et al. (2019) to the few-shot sequence labeling.
* **ConVEx**Henderson and Vulic (2021) is a fine-tuning based model, which is first pre-trained on the Reddit corpus with the sequence labeling objective tasks and then fine-tuned on the source domain and target domain annotated data for final few shot sequence labelingm
* **Ma2021**Ma et al. (2021) formulates sequence labeling as the machine reading comprehension problem, and proposes some questions to extract slots in the query sentence.
* **ProtoBERT**Fritzler et al. (2019b) predicts the query labels according to the similarity of BERT hidden states of support set and query tokens.
* **Matching Net (MN)+BERT**Hou et al. (2020) is similar to ProtoBERT. The only difference is that MN uses the matching network Vinyals et al. (2016) for token classification.
* **L-TapNet-CDT**Hou et al. (2020) utilizes the task-adaptive projection network Yoon et al. (2019), pair-wise embedding and collapsed dependency transfer mechanisms to do classification.
* **NNShot**Yang and Katiyar (2020) is similar to ProtoBERT, while it makes the prediction based on the nearest neighbor.
* **StructShot**Yang and Katiyar (2020) adopts an additional Viterbi decoder during the inference phase on top of NNShot.
* **Retriever**Yu et al. (2021) is a retrieval based method which does classification according to the most similar example in the support set.

### Parameter Setting

In our implementation, we utilize _BERT-base-uncased_ as our backbone encoder the same as Hou et al. (2020); Yu et al. (2021); Ding et al. (2021). We use Adam Kingma and Ba (2015) as our optimizer. In FewNERD, the learning rate is \(2e-5\) for BERT encoder and \(5e-4\) for other modules. In the 1-shot setting of SNIPS, for the domain "Mu", the learning rate is \(5e-6\) for BERT encoder and \(1e-4\) for other modules. For the domain "Bo", the learning rate is \(1e-5\) for BERT encoder and \(1e-4\) for other modules. For other settings of SNIPS, the learning rate is \(5e-5\) for BERT encoder and \(5e-4\) for other modules. We set the dropout ratio Srivastava et al. (2014) to \(0.1\). The dimension of span representation \(d\) and the maximum span length \(L\) is set to \(100\) and \(8\), respectively. For BSNMS, the beam size \(b\) is 5. Since these is no nested instances in FewNERD and SNIPS, we set the threshold to filter false positive spans \(\delta\) to \(0.1\), the threshold to decay span scores \(k\) to \(1e-5\) and the decay ratio \(u\) to \(1e-5\) to force the refining results have no nested spans. We use the grid search to search our hyperparameters, and the scope of each hyperparameter are included in Table 7. We train our model on a single NVIDIA A40 GPU with 48GB memory.

## Appendix B FewNERD-nested

We construct our FewNERD-nested via ACE0 Walker et al. (2006), GENIA Ohta et al. (2002) and the origin FewNERD datasets. GENIA is a biological named entity recognition dataset, which contains 5 kinds of entities, _'DNA'_, _'Protein'_, _'cell_type'_, _'RNA'_ and _'cell_line'_, and all of these entity types are not included in the Few NERD. We

\begin{table}
\begin{tabular}{l l} \hline \hline learning rate & [5e-5, 1e-4, 3e-4, 5e-4] \\ dropout & [0.1, 0.3, 0.5] \\ bert learning rate & [5e-6, 1e-5, 2e-5, 3e-5, 5e-5] \\ span dimension & [50, 100, 150, 200] \\ beam size & [1, 3, 5, 7] \\ \hline \hline \end{tabular}
\end{table}
Table 7: The searching scope of hyperparameters.

\begin{table}
\begin{tabular}{l c} \hline \hline \(k\) & [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] \\ \(\delta\) & [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] \\ \(u\) & [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] \\ \hline \hline \end{tabular}
\end{table}
Table 8: The searching scope of BSnMS hyperparameters.

 

partition the sentences in GENIA into two groups, \({\cal G}_{1}\) and \({\cal G}_{2}\). \({\cal G}_{1}\) consists of sentences with nested entities (\(4,744\) sentences in total), and \({\cal G}_{2}\) consists of sentences without nested entities (\(11,924\) sentences in total). We utilize sentences in \({\cal G}_{1}\) and \({\cal G}_{2}\) to construct the query and support set of the GENIA task, respectively. Our FewNERD-nested contains 2000 5-way 5\(\sim\)10-shot test tasks in total, where \(r_{nested}\) percent tasks are from GENIA, and the remained tasks are from FewNERD-INTRA. We use ACE05 to construct the validation set. ACE05 is a widely used named entity recognition dataset, which contains \(7\) coarse-grained entity types, '_FAC_', '_PER_', '_LOC_', '_VEH_', '_GPE_', '_SPE_', '_WEA_' and '_ORG_'. The '_LOC_', '_PER_', _ORG_' and '_FAC_' are not included in the training set of FewNERD-INTRA, and therefore we use them (\(28\) fine-grained entity types in total) and sample \(1000\) nested tasks as the validation dataset to tune the \(k\), \(\delta\) and \(u\) of BSNMS. The search scope is included in the Table 8. In this nested situation, we finally set the \(k\), \(\delta\) and \(u\) to \(0.1\), \(0.1\) and \(0.4\) respectively.

## Appendix C Robustness in the Noisy Situation

FSSL methods tend to be seriously influenced by the noise in the support set, since they make the prediction based on only limited annotated examples. To explore the robustness of Esd in the noisy situation, we construct **FewNERD-noise**. In FewNERD-noise, we disturb each \(5\) way \(5\sim 10\) shot FewNERD-INTRA task with a noisy ratio \(r_{noise}\), which means there are nearly \(r_{noise}\) percent entities in the support set are mislabeled. As illustrated in the right part of Figure 5, with \(r_{noise}\) increasing, the performance of Esd drops less than baselines, which furthuer shows the superiority of our methods.

## Appendix D A Detail Case Study of BSNMS

For span conflicts resolution, we propose a post-processing method BSNMS. A step by step demonstration of BSNMS is shown in Figure 6.

Figure 5: The performance drops in noisy situations when increasing \(r_{noise}\). \(\Delta\)F1% is the F1 reduction percent over \(r_{noise}=0\).

 

* [16] A. Krizhevsky, S. Goyal, and R. L. Krause, "A fast-time algorithm for the estimation of the parameters of a given model," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2593-2598, 2014.

[MISSING_PAGE_POST]