SlowGuess/NAACL_2022_OCR huggingface dataset:
Ruotian Ma $^{1*}$ , Xin Zhou $^{1*}$ , Tao Gui $^{2\dagger}$ , Yiding Tan $^{1}$ , Linyang Li $^{1}$ , Qi Zhang $^{1\dagger}$ , Xuanjing Huang $^{1}$
$^{1}$ School of Computer Science, Fudan University, Shanghai, China  
 $^{2}$ Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China  
{rtma19,xzhou20,tgui,qz,xjhuang}@fudan.edu.cn
Prompt-based methods have been successfully applied in sentence-level few-shot learning tasks, mostly owing to the sophisticated design of templates and label words. However, when applied to token-level labeling tasks such as NER, it would be time-consuming to enumerate the template queries over all potential entity spans. In this work, we propose a more elegant method to reformulate NER tasks as LM problems without any templates. Specifically, we discard the template construction process while maintaining the word prediction paradigm of pre-training models to predict a class-related pivot word (or label word) at the entity position. Meanwhile, we also explore principled ways to automatically search for appropriate label words that the pre-trained models can easily adapt to. While avoiding the complicated template-based process, the proposed LM objective also reduces the gap between different objectives used in pre-training and fine-tuning, thus it can better benefit the few-shot performance. Experimental results demonstrate the effectiveness of the proposed method over bert-tags and template-based method under few-shot settings. Moreover, the decoding speed of the proposed method is up to 1930.12 times faster than the template-based method.
Pre-trained language models (LMs) have led to large improvements in NLP tasks (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020). Popular practice to perform downstream classification tasks is to replace the pretrained model's output layer with a classifier head and fine-tune it using a task-specific objective function. Recently, a new paradigm, prompt-based learning, has achieved great success on few-shot classification tasks by reformulating classification tasks as cloze
Figure 1: An example of template-based prompt method for NER. Predicting all labels in sentence "Obama was born in America." requires enumeration over all spans.
questions. Typically, for each input [X], a template is used to convert [X] into an unfilled text (e.g., "[X] It was ____"), allowing the model to fill in the blank with its language modeling ability. For instance, when performing sentiment classification task, the input "I love the milk." can be converted into "I love the milk. It was ____". Consequently, the LM may predict a label word "great", indicating that the input belongs to a positive class.
Two main factors contribute to the success of prompt-based learning on few-shot classification. First, re-using the masked LM objective helps alleviate the gap between different training objectives used at pre-training and fine-tuning. Therefore, the LMs can faster adapt to downstream tasks even with a few training samples (Schick and Schütze, 2021a,b; Brown et al., 2020). Second, the sophisticated template and label word design helps LMs better fit the task-specific answer distributions, which also benefits few-shot performance. As proved in previous works, proper templates designed by manually selecting (Schick and Schütze, 2021a,b), gradient-based discrete searching (Shin et al., 2020), LM generating (Gao et al., 2021) and continuously optimizing (Liu et al., 2021) are able to induce the LMs to predict more appropriate answers needed in corresponding tasks.
However, the template-based prompt methods are intrinsically designed for sentence-level tasks, and they are difficult to adapt to token-level clas
Human Language Technologies, pages 5721 - 5732
sification tasks such as named entity recognition (NER). First, searching for appropriate templates is harder as the search space grows larger when encountering span-level querying in NER. What's worse, such searching with only few annotated samples as guidance can easily lead to overfitting. Second, obtaining the label of each token requires enumerating all possible spans, which would be time-consuming. As an example in Fig.1, the input "Obama was born in America." can be converted into "Obama was born in America. [Z] is a __ entity.", where [Z] is filled by enumerating all the spans in [X] (e.g., "Obama", "Obama was") for querying. Fig.1 shows that obtaining all entities in "Obama was born in America." requires totally 21 times to query the LMs with every span. Moreover, the decoding time of such an approach would grow catastrophically as sentence length increasing, making it impractical to document-level corpus.
In this work, we propose a more elegant way for prompting NER without templates. Specifically, we reformulate NER as an LM task with an Entity-oriented LM (EntLM) objective. Without modifying the output head, the pre-trained LMs are fine-tuned to predict class-related pivot words (or label words) instead of the original words at the entity positions, while still predicting the original word at none-entity positions. Next, similar to template-based methods, we explore principled ways to automatically search for the most appropriate label words. Different approaches are investigated including selecting discrete label words based on the word distribution in lexicon-annotated corpus or LM predictions, and obtaining the prototypes as virtual label words. Our approach keeps the merits of prompt-based learning as no new parameters are introduced during finetuning. Also, through the EntLM objective, the LM are allowed to perform NER task with only a slight adjustment of the output distribution, thus benefiting few-shot learning. Moreover, well-selected label words accelerate the adaptation of LM distribution towards the desired predictions, which also promotes few-shot performance. It's also worth noting that the proposed method requires only one-pass decoding to obtain all entity labels in the sentence, which is significantly more efficient compared to the time-consuming enumeration process of template-based methods. Our codes are publicly available at https://github.com/rtmaww/EntLM/.
To summarize the contribution of this work:
- We propose a template-free approach to prompt NER under few-shot setting.
- We explore several approaches for label word engineering accompanied with intensive experiments.
- Experimental results verify the effectiveness of the proposed method under few-shot setting. Meanwhile, the decoding speed of the proposed method is 1930.12 times faster than template-based baseline.
In this work, we focus on few-shot NER task. Different from previous works that assume a richresource source domain and available support sets during testing, we follow the few-shot setting of (Gao et al., 2021), which supposes that only a small number of examples are used for fine-tuning. Such setting makes minimal assumptions about available resources and is more practical. Specifically, when training on a new dataset  $\mathbf{D}$  with the label space  $\mathcal{V}$ , we assume only  $K$  training examples for each class in the training set, such that the total number of examples is  $K_{tot} = K \times |\mathcal{V}|$ . Then, the model is tested with an unseen test set  $(X^{test}, Y^{test}) \sim \mathbf{D}_{test}$ . Here, for NER task, a training sample refers to a continual entity span  $\mathbf{e} = \{x_1, \ldots, x_m\}$  that is labeled with a positive class (e.g., "PERSON").
In this work, we propose a template-free prompt tuning method, Entity-oriented LM (EntLM) fine-tuning, for few-shot NER. We first give a description of the template-based prompt tuning. Then we introduce the EntLM method along with the label word engineering process.
3.1 Template-based Prompt Tuning
The standard fine-tuning process for NER is replacing the LM head with a token-level classification head and optimizing the newly-introduced parameters and the pre-trained LM. Different from standard fine-tuning, prompt-based tuning reformulates classification tasks as LM tasks, and fine-tunes LM to predict a label word.
Formally, a prompt consists of a template function  $T_{prompt}(\cdot)$  that converts the input  $x$  to a prompt input  $x_{prompt} = T_{prompt}(x)$ , and a set of
(a) Standard fine-tuning.
(b) Entity-oriented LM fine-tuning.
(c) Template-based prompt tuning.
Figure 2: Comparison of different fine-tuning methods for NER. (a) is the standard fine-tuning method, which replace the LM head with a classifier head and perform label classification. (c) is the template-based prompt learning method, which induces the LM to predict label words by constructing a template. (b) is the proposed Entity-oriented LM fine-tuning method, which also re-uses the LM head and leads the LM to predict label words through an Entity-oriented LM objective. (For entities with multiple spans, the model predicts the same label word at each position, which is similar to the "IO" labeling scheme.)
label words  $\mathcal{V}$  which are connected with the label space through a mapping function  $\mathcal{M}:\mathcal{V}\to \mathcal{V}$ . The template is a textual string with two unfilled slot: a input slot [X] to fill the input  $x$  and an answer slot [Z] that allows LM to fill label words. For instance, for a sentiment classification task, the template can take the form as "[X] It was [Z].". The input is then mapped to "  $x$  It was [Z].". Specifically, when using a masked language model (MLM) for prompt-based tuning, [Z] is filled with a mask token [MASK]. By feeding the prompt into the MLM, the probability distribution over the label set  $\mathcal{V}$  is modeled by:
$$
\begin{array}{r l} P (y | x) & = P ([ M A S K ] = \mathcal {M} (\mathcal {Y}) | x _ {\text {p r o m p t}}) \\ & = \operatorname {S o f t m a x} \left(\mathbf {W} _ {l m} \cdot \mathbf {h} _ {[ M A S K ]}\right) \end{array} \tag {1}
$$
where  $\mathbf{W}_{lm}$  are the parameters of the pre-trained LM head. Unlike in standard fine-tuning, no new parameters are introduced in this approach, therefore the model can easier fit the target task with few samples. Also, the LM objective reduces the gap between pre-training and fine-tuning, thus benefiting few-shot training (Gao et al., 2021).
3.1.1 Problems of Prompt-based NER
However, when applied to NER, such prompt-based approach becomes complicated. given an input  $X = \{x_{1},\ldots ,x_{n}\}$  , we need to obtain the label sequence  $Y = \{y_{1},\dots ,y_{n}\} ,y_{i}\in \mathcal{V}$  corresponding to each token of  $X$  . Therefore, an additional slot [S] is added in the template to fill a token  $x_{i}$  or a continual span  $\mathbf{s}_j^i = \{x_i,\dots ,x_j\}$  that starts from  $x_{i}$  and ends with  $x_{j}$  . For example, the template can take the form as "[X] [S] is a [Z] entity.", where the LMs are fine-tuned to predict an entity label word at [Z] (e.g., person) corresponding to an entity label (e.g., PERSON). During decoding, obtaining the labels  $Y$  of the
whole sentence requires enumeration over all the spans:
$$
\bar {Y} = \{\arg \max _ {y \in \mathcal {Y}} P ([ Z ] = \mathcal {M} (\mathcal {Y}) | T _ {p r o m p t} (X, s _ {j} ^ {i})),
$$
$$
s _ {j} ^ {i} = \operatorname {E n u m e r a t e} \left(\left\{x _ {i}, \dots , x _ {j} \right\}, i, j \in \{1.. n \}\right), \tag {2}
$$
Such a decoding way is time-consuming and the decoding time increasing as the sequence length getting longer. Therefore, although efficient in few-shot setting, template-based prompt tuning is not suitable for NER task.
3.2 Entity-Oriented LM Fine-tuning
In this work, we propose a more elegant way to prompt NER without templates, while maintaining the advantages of prompt-tuning. Specifically, we also reformulate NER as a LM task. However, instead of forming templates to re-use the LM objective, we propose a new objective, Entity-oriented LM (EntLM) objective for fine-tuning NER. As shown in Fig. 2 (b), when fed with "Obama was born in America", the LM is trained to predict a label word "John" at the position of the entity "Obama" as an indication of the label "PER". While for none-entity word "was", the LM remains to predict the original word.
Formally, to fine-tune the LM with EntLM objective, we first construct a label word set  $\mathcal{V}_l$  which is also connected with the task label set through a mapping function  $\mathcal{M}: \mathcal{V} \to \mathcal{V}_l$ . Next, given the input sentence  $X = \{x_1, \ldots, x_n\}$  and the corresponding label sequence  $Y = \{y_1, \ldots, y_n\}$ , we construct a target sentence  $X^{Ent} = \{x_1, \ldots, \mathcal{M}(y_i), \ldots, x_n\}$  by replacing the token at the entity position  $i$  (here we assume  $y_i$  is an entity label) with corresponding label word  $\mathcal{M}(y_i)$ , and maintaining the original words at none-entity positions. Then, given the original input
$X$ , the LM is trained to maximize the probability  $P(X^{Ent}|X)$  of the target sentence  $X^{Ent}$ :
$$
\mathcal {L} _ {E n t L M} = - \sum_ {i = 1} ^ {n} \log P \left(x _ {i} = x _ {i} ^ {E n t} \mid X\right) \tag {3}
$$
where  $P(x_{i} = x_{i}^{Ent}|X) = \text{Softmax}(\mathbf{W}_{lm} \cdot \mathbf{h}_{i})$ . Noted that  $\mathbf{W}_{lm}$  are also the parameters of the pre-trained LM head. By re-using the whole pretrained model, no new parameters are introduced during this fine-tuning process. Meanwhile, the EntLM objective serves as a LM-based objective to reduce the gap between pre-training and finetuning. In this way, we avoid the complicated template constructing for NER task, and keep the good few-shot ability of prompt-based method.
During testing, we directly feed the test input  $X$  into the model, and the probability of labeling the  $i^{th}$  token with class  $y \in \mathcal{V}$  is modeled by:
$$
p \left(y _ {i} = y \mid X\right) = p \left(x _ {i} = \mathcal {M} (y) \mid X\right) \tag {4}
$$
Noted that we only need one-pass decoding process to obtain all labels for each sentence, which is intensively more efficient than template-based prompt querying.
3.3 Label Word Engineering
Previous template-based studies have verified the significant impact of template engineering on few-shot performance. Similarly, in this work, we explore approaches for automatically selecting proper label words. Since the EntLM object lead all entities that belong to a class to predict the same label word, we believe that the purpose of label word searching is to find a pivot word that can mostly represent the words in each class.
3.3.1 Low-resource Label word selection
When selecting label words with only few annotated samples as guidance, the randomness of sampling will largely affect the selection. In order to obtain more consistent selection, we explore the usage of unlabeled data and lexicon-based annotation as a resource for label word searching. This is a practical setting since unlabeled data of a target domain or a general domain is usually available, and for NER, the entity lexicon of target classes are usually easy to access.
To obtain annotation via entity lexicon, we adopt the KB-matching approach proposed by Liang et al. (2020), which leverages an external KBs, wikidata,
Figure 3: Searching for two types of label words: the discrete label words and the continuous vectors as virtual label words. To search for the discrete label words, we select the high-frequency words in data or LM output distribution, or combine these two ways. To search for virtual label words, we calculate the mean vectors of the high-frequency words of each class as prototypes.
as the source of lexicon annotation. Such lexicon-based annotation is inevitably noisy. However, our approach do not suffers a lot from the noise since we only regarded it as an indication of the data distribution and do not train the model directly with the noisy annotation.
3.3.2 Label word searching
With the help of lexicon-annotated data  $\mathcal{D}_{\text{lexicon}} = \{(X_i, Y_i^*)\}_{i=1}^N$ , we explore three methods for label word searching.
Searching with data distribution (Data search) The most intuitive method is to select the most frequent word of the given class in the corpus. Specifically, when searching for label words for class  $C$ , we calculate the frequency  $\phi(x = w, y^* = C)$  of each word  $w \in \mathcal{V}$  labeled as  $C$  and select the most frequent words by ranking:
$$
\mathcal {M} (C) = \arg \max  _ {w} \phi (x = w, y ^ {*} = C) \quad (5)
$$
Searching with LM output distribution (LM search) In this approach, we leverage the pretrained language model for label word searching. Specifically, we feed each sample  $(X,Y^{*})$  into LM and get the probability distribution  $p(\hat{x}_i = w|X)$  of predicting each word  $w\in \mathcal{V}$  at each position  $j$ . Suppose  $\mathcal{I}_{topk}(\hat{x}_i = w|X,Y^*)\to \{0,1\}$  is the
Datasets	Domain	# Class	# Train	# Test
CoNLL'03	News	4	14.0k	3.5k
OntoNotes*	General	11	60.0k	8.3k
MIT Movie	Review	12	7.8k	2.0k
Table 1: Dataset details. OntoNotes* denotes the Ontonotes5.0 dataset after removing value/numerical/time/date entity types.
indicator function indicating whether  $w$  belongs to the top  $k$  predictions of  $x_{i}$  in sample  $(X, Y^{*})$ . The label word of class  $C$  can be obtained by:
$$
\mathcal {M} (C) = \arg \max  _ {w}
$$
$$
\sum_ {(X, Y ^ {*}) \in \mathcal {D}} \sum_ {i} ^ {| X |} \phi_ {\text {t o p k}} \left(\hat {x} _ {i} = w, y _ {i} ^ {*} = C\right) \tag {6}
$$
where  $\phi_{topk}(\hat{x}_i = w, y^* = C) = \mathcal{I}_{topk}(\hat{x}_i = w|X, Y^*) \cdot \mathcal{I}(y_i^* = C)$  denotes the frequency of  $w$  occurring in the top  $k$  predictions of the positions labeled as class  $C$ .
Searching with both data & LM output distribution (Data&LM search) In this approach, we select label words by simultaneously considering the data distribution and LM output distribution. Specifically, the label word of class  $C$  can be obtained by:
$$
\begin{array}{l} \mathcal {M} (C) = \arg \max  _ {w} \{\sum_ {(X, Y ^ {*}) \in \mathcal {D}} \sum_ {i} ^ {| X |} \phi (x _ {i} = w, y _ {i} ^ {*} = C) \\ \cdot \sum_ {(X, Y ^ {*}) \in \mathcal {D}} \sum_ {i} ^ {| X |} \phi_ {t o p k} (\hat {x} _ {i} = w, y _ {i} ^ {*} = C) \} \tag {7} \\ \end{array}
$$
3.3.3 Removing conflict label words
The selected high-frequency label words are potentially high-frequency words among all the classes. Using such label words will result in conflicts when training for different classes. Therefore, after label word selection, we remove the conflict label words of a class  $C$  by:
$$
w = \mathcal {M} (C), i f \frac {\phi (x = w , y ^ {*} = C)}{\sum_ {k} \phi (x = w , y ^ {*} = k)} > T h \tag {8}
$$
where  $Th$  is a manually set threshold.
In this section, we conduct few-shot experiments to verify the effectiveness of the proposed method. We also conducts intensive analytical experiments for label words selection.
As mentioned in Section 2, in this work, we focus on few-shot setting that no source domain data yet only  $K$  samples of each class are available for training on a new NER task. To better evaluate the models' few-shot ability, we conduct experiments with  $K \in \{5,10,20,50\}$ . For each  $K$ -shot experiment, we sample 3 different training set and repeat experiments on each training set for 4 times. Few-shot data sampling. Different from sentence-level few-shot tasks, in NER, a sample refers to one entity span in a sentence. One sampled sentence might include multiple entity instances. In our experiments, we conduct an exact sampling strategy to ensure that we sample exactly  $K$  samples for each class. The details of the algorithm can be found at Appendix A.2.
4.2 Datasets and Implementation Details
We evaluate the proposed method with three benchmark NER datasets from different domains: the CoNLL2003 dataset (Sang and De Meulder, 2003) from the newswire domain, Ontonotes 5.0 dataset (Weischedel et al., 2013) from general domain and the MIT-Movie dataset (Liu et al., 2013)<sup>1</sup> from the review domain. As we focus on named entities, we omit the value/numerical/time/date entity types (e.g., "Cardinal", "Money", etc) in OntoNotes 5.0. Details of the datasets are shown in Table 1.
Labeling multi-span entities. For entities with multiple spans (including multiple words or subtokens after tokenization), we let the model predict the same label word at each position. This labeling method is the same with the "IO" labeling schema, which is consistent to our baseline implementation.
To ensure a few-shot scenario, we didn't use a development set for model choosing. Instead, we use the model of the last epoch for predicting. For lexicon-based annotation, we use the KB-matching method of Liang et al. (2020)<sup>2</sup>. For more implementation details (e.g., the learning rate, etc.), please refer to Appendix A.1 or our codes.
4.3 Baselines and Proposed Models
In our experiments, we compare our method with competitive baselines, involving both metric-learning based and prompt-based approaches.
BERT-tags (Devlin et al., 2019) The BERT-based baseline which fine-tunes the BERT model
<sup>1</sup>https://groups.csail.mit.edu/sls/downloads/
$^{2}$ https://github.com/criang1453/BOND
Datasets	Methods	K=5	K=10	K=20	K=50
CoNLL03	BERT-tags (IO)	41.87 (12.12)	59.91 (10.65)	68.66 (5.13)	73.20 (3.09)
NNShot	42.31 (8.92)	59.24 (11.71)	66.89 (6.09)	72.63 (3.42)
StructShot	45.82 (10.30)	62.37 (10.96)	69.51 (6.46)	74.73 (3.06)
Template NER	43.04 (6.15)	57.86 (5.68)	66.38 (6.09)	72.71 (2.13)
EntLM (Ours)	49.59 (8.30)	64.79 (3.86)	69.52 (4.48)	73.66 (2.06)
EntLM + Struct (Ours)	51.32 (7.67)	66.86 (3.01)	71.23 (3.91)	74.80 (1.87)
OntoNotes 5.0	BERT-tags (IO)	34.77 (7.16)	54.47 (8.31)	60.21 (3.89)	68.37 (1.72)
NNShot	34.52 (7.85)	55.57 (9.20)	59.59 (4.20)	68.27 (1.54)
StructShot	36.46 (8.54)	57.15 (5.84)	62.22 (5.10)	68.31 (5.72)
Template NER	40.52 (8.62)	49.89 (3.66)	59.53 (2.25)	65.15 (2.95)
EntLM (Ours)	45.21 (9.17)	57.64 (4.18)	65.64 (4.24)	71.77 (1.31)
EntLM + Struct (Ours)	46.60 (10.35)	59.35 (3.24)	67.91 (4.55)	73.52 (0.97)
MIT-Movie	BERT-tags (IO)	39.57 (6.38)	50.60 (7.29)	59.34 (3.66)	71.33 (3.04)
NNShot	38.97 (5.54)	50.47 (6.09)	58.94 (3.47)	71.17 (2.85)
StructShot	41.60 (8.97)	53.19 (5.52)	61.42 (2.98)	72.07 (6.41)
Template NER	45.97 (3.86)	49.30 (3.35)	59.09 (0.35)	65.13 (0.17)
EntLM (Ours)	46.62 (9.46)	57.31 (3.72)	62.36 (4.14)	71.93 (1.68)
EntLM + Struct (Ours)	49.15 (8.91)	59.21 (3.96)	63.85 (3.7)	72.99 (1.80)
Table 2: Main results of EntLM on three datasets under different few-shot settings (K=5,10,20,50). We report mean (and deviation in brackets) performance over 3 different splits (4 repeated experiments for each split).
with a label classifier.
NNShot and StructShot (Yang and Katiyar, 2020) Two metric-based few-shot learning approaches for NER. Different from Prototypical Network, they leverage a nearest neighbor classifier for few-shot prediction. StructShot is an extension of NNShot which proposes a viterbi algorithm during decoding. We extend these two approaches to our few-shot setting. Noted that the viterbi algorithm in the original paper calculates the data distribution of a source domain, yet in our setting, the source domain is unavailable. Therefore, we also use the lexicon-annotated data for performing this method.
TemplateNER (Cui et al., 2021) A template-based prompt method. By constructing a template for each class, it queries each span with each class separately. The score of each query is obtained by calculating the generalization probability of the query sentence through a generative pre-trained LM, BART(Lewis et al., 2020).
EntLM The proposed method.
EntLM+Struct Based on the proposed method, we further leverages the viterbi algorithm proposed in (Yang and Katiyar, 2020) to boost the performance. For more details please refer to (Yang and Katiyar, 2020) or our codes.
In Appendix A.5, we also compare with the roberta-base baselines from (Huang et al., 2020).
4.4 Few-shot Results
Table 2 show the results of the proposed method and baselines under few-shot setting. From