SlowGuess/NAACL_2022_OCR huggingface dataset:
the table, we can observe that: (1) On all the three datasets, for all few-shot settings, the proposed method performs consistently better than all the baseline methods, especially for 5-shot learning. Also, the performance of the proposed method is more stable (according to the deviation) than the compared baselines. (2) BERT-tags method shows poor ability of few-shot learning, and the proposed method achieves up to  $9.45\%$ ,  $11.83\%$ ,  $9.58\%$  improvement over BERT-tags on CoNLL03, OntoNotes 5.0 and MIT-Movie, respectively. These results show the advantages of the proposed method over standard fine-tuning, which introduces no new parameters and uses an LM-like objective to reduce the gap between pre-training and fine-tuning. (3) The proposed method consistently outperforms the template-based prompt method, Template NER, which shows the advantage of the proposed method over standard template-based method. (4) When no rich-resource source domain is available, the metric-based methods (NNShot) do not show advantages over BERT-tags, which shows the limitation of these method under more practical few-shot scenarios. (5) Among all baselines, the StructShot is a competitive baseline that also leverages lexicon and unlabeled data for structure-based decoder, yet our method can also benefit from the viterbi decoder and outperform StructShot.
4.5 Efficiency Study
In this section, we perform an efficiency study on all the three datasets. We calculate the decoding
Methods	CoNLL03	OntoNotes	MIT-Movie
K=5	K=10	K=5	K=10	K=5	K=10
DataSearch	50.00 (9.75)	61.31 (4.73)	36.94 (5.04)	49.54 (5.02)	39.25 (4.83)	51.65 (5.52)
LMSearch	48.40 (6.81)	59.39 (5.50)	36.98 (6.71)	48.20 (5.46)	39.12 (4.18)	48.30 (3.76)
Data&LMSearch	49.55 (7.76)	61.00 (6.98)	36.60 (7.90)	50.64 (6.12)	38.86 (11.43)	50.42 (6.45)
Data + Virtual	49.25 (4.96)	63.40 (5.13)	45.61 (10.51)	55.13 (4.95)	45.59 (8.25)	55.10 (4.42)
LM + Virtual	42.65 (12.58)	59.39 (5.50)	45.29 (7.77)	54.50 (3.66)	46.23 (5.60)	54.92 (6.15)
Data&LM + Virtual	49.59 (8.30)	64.79 (3.86)	45.21 (9.17)	57.64 (4.18)	46.62 (9.46)	57.31 (3.72)
Table 3: Comparison of our label word selection methods. We report mean (and standard deviation) performance.
Methods	CoNLL	OntoNotes	MIT-Movie
BERT-tags	8.57	23.89	6.46
TemplateNER	6,491.00	50,241.00	5254.00
NNShot	16.03	82.62	15.98
StructShot	19.84	98.67	17.66
EntLM	9.26	26.03	6.64
EntLM + Struct	13.40	34.92	7.38
Table 4: The decoding time (s) of different methods.
time of each method on a TiTan XP GPU with batch size=8. (The source codes of Template NER do not allow us to change the batch size, so we keep the original batch size=45, which is the enumeration number of a 9-gram span.) From Tab.4, we can observe that: 1) EntLM can achieve comparable speed with BERT-tags, as only one pass of token classification is required for decoding each batch. 2) The decoding speed of TemplateNER is severely slow, while EntLM is up to 1930.12 times faster than TemplateNER. These results show the advantages of EntLM over template-based prompt tuning methods in NER task.
4.6 Label Word Selection
In Sec.3.3, we have presented different ways for label word selection. In this section, we conduct experiments on these methods and the results are reported in table 3. We can observe that: 1) The virtual word selection approach is always better than the discrete word selection. While among all virtual selection methods, choosing high-frequency words with the combination of data and LM distribution shows advantages over other methods. The reason of these results might be that simultaneously considering both data distribution gives not only the data prior in the target dataset, but also the contextualized information from the PLM, thus benefiting the performance. 2) Searching only with LM distribution leads to poor results especially under 5-shot setting, showing that the general knowledge learned from pretrained might be less helpful than the data-specific knowledge under few-shot settings.
Figure 4: Impact of different lexicon sizes.
4.6.1 Impact of Lexicon Quality on Label Word Selection
Note that we leverage unlabeled data and lexicon annotation for label word selection. In this experiment, we study how the quality of lexicon impacts the performance on the OntoNotes* dataset. Specifically, we obtain different sizes of lexicon (5% to 80% of the original lexicon size) by sampling entity words in the original lexicon with the weights of entity frequency. This sampling method follows the real-world situation since high-frequency entities are easier to obtain. Fig.4 shows the results of EntLM and baseline methods against lexicon size. We can observe that: (1) EntLM with the Data&LM+Virtual selection method illustrates consistent high performance even with 5% lexicon. This means our method is not limited to the lexicon quality, and we only require a small lexicon to reach acceptable few-shot performance. (2) Compared with Data&LM+Virtual method, the Data&LM is much more fragile regarding the lexicon quality. However, it still performs better than the compared baselines.
We further conduct experiments on different sizes of the unlabeled dataset by uniformly sampling  $5\% - 80\%$  of the original data. As shown in Fig.5, the proposed method also shows high robustness to the amount of unlabeled data.
4.7 Effect of Further Pre-training
When predicting label words on task-specific data during fine-tuning, there is an intrinsic gap between
Figure 5: Impact of the amount of unlabeled data.
Methods	CoNLL03
K=5	K=10
BERT-tags	41.87 (12.12)	59.91 (10.65)
EntLM	49.59 (8.30)	64.79 (3.86)
EntLM + Struct	51.32 (7.67)	66.86 (3.01)
BERT-tags (further)	41.16 (10.41)	61.70 (5.15)
EntLM (further)	56.82 (12.27)	66.82 (4.65)
EntLM + Struct (further)	58.77 (12.16)	68.96 (4.41)
Table 5: Impact of further pre-training.
the LM output distribution and the target data distribution. Therefore, it is natural to conduct a further pre-training approach on the target-domain unlabeled data to boost the LM predictions towards target distribution. In Table 5, we show the results of our method and BERT-tags trained after further pre-training with MLM objective on domain-specific unlabeled data. As seen, the further pre-training practice can largely boost the few-shot learning ability of EntLM, while showing less helpful for classifier-based fine-tuning method. This might because the LM objective used in EntLM can benefit more from a task-specific LM output distribution, showing the superiority of EntLM in better leveraging the pre-trained models.
5.1 Template-based prompt learning
Stem from the GPT models (Radford et al., 2019; Brown et al., 2020), prompt-based learning have been widely discussed. These methods reformulate downstream tasks as cloze tasks with textual templates and a set of label words, and the design of templates is proved to be significant for prompt-based learning. Schick and Schütze (2021a,b) uses manually defined templates for prompting text classification tasks. Jiang et al. (2020) proposes a mining approach for automatically search for templates. Shin et al. (2020) searches for optimal discrete templates by a gradient-based approach. (Gao et al., 2021) generates templates with the T5 pre-trained model. Meanwhile, several approaches
have explore continuous prompts for both text classification and generation tasks Li and Liang (2021); Liu et al. (2021); Han et al. (2021). Also, several approaches are proposed to enhance the templates with illustrative cases (Madotto et al., 2020; Gao et al., 2021; Brown et al., 2020) or context (Petroni et al., 2020). Although template-based methods are proved to be useful in sentence-level tasks, for NER task (Cui et al., 2021), such template-based method can be expensive for decoding. Therefore, in this work, we propose a new paradigm of prompt-tuning for NER without templates.
5.2 Few-shot NER
Recently, many studies focuses on few-shot NER (Hofer et al., 2018; Fritzler et al., 2019; Li et al., 2020; Ding et al., 2021; Chen et al., 2021). Among these, Fritzler et al. (2019) leverages prototypical networks for few-shot NER. Yang and Katiyar (2020) propose to calculate the nearest neighbor of each queried sample instead of the nearest prototype. Huang et al. (2021) experimented comprehensive baselines on different datasets. Tong et al. (2021) proposes to mine the undefined classes for few-shot learning. Cui et al. (2021) leverages prompts for few-shot NER. However, most of these studies follow the manner of episode training or assume a rich-resource source domain. In this work, we follow the more practical few-shot setting of Gao et al. (2021), which assumes only few samples each class for training. We also adapt previous methods to this setting as competitive baselines.
In this work, we propose a template-free prompt tuning method, EntLM, for few-shot NER. Specifically, we reformulate the NER task as a Entity-oriented LM task, which induce the LM to predict label words at entity positions during fine-tuning. In this way, not only the complicated template-based methods can be discarded, but also the few-shot performance can be boosted since the EntLM objective reduces the gap between pretraining and fine-tuning. Experimental results show that the proposed method can achieve significant improvement on few-shot NER over BERT-tags and template-based method. Also, the decoding speed of EntLM is up to 1930.12 times faster than the template-based method.
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No. 61976056, 62076069), Shanghai Municipal Science and Technology Major Project (No.2021SHZDX0103).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Xiang Chen, Ningyu Zhang, Lei Li, Xin Xie, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021. Lightner: A lightweight generative framework with prompt-guided attention for low-resource ner. arXiv preprint arXiv:2109.00720.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using BART. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 1835–1845, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3198-3213, Online. Association for Computational Linguistics.
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021.Ptr: Prompt tuning with rules for text classification.
Maximilian Hofer, Andrey Kormilitzin, Paul Goldberg, and Alejo Nevada-Holgado. 2018. Few-shot learning for named entity recognition in medical text.
Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2020. Few-shot named entity recognition: A comprehensive study.
Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2021. Few-shot named entity recognition: An empirical baseline study. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10408-10423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Jing Li, Billy Chiu, Shanshan Feng, and Hao Wang. 2020. Few-shot named entity recognition via meta-learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-1.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision, page
1054-1064. Association for Computing Machinery, New York, NY, USA.
Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass. 2013. Query understanding enhanced by hierarchical parsing structures. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 72-77. IEEE.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. CoRR, abs/2103.10385.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Andrea Madotto, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020. Language models as few-shot learner for task-oriented dialogue systems.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Roektaschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. In *Automated Knowledge Base Construction*.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Meihan Tong, Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, and Juanzi Li. 2021. Learning from miscellaneous other-class words for few-shot named entity recognition. In Proceedings of
the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6236-6247, Online. Association for Computational Linguistics.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6365-6375, Online. Association for Computational Linguistics.
We implement our method based on the huggingface transformers<sup>3</sup>. For all our experiments except TemplateNER, we use "bert-base-cased" pre-trained model as the base model for fine-tuning, and no new parameters are introduced in the proposed method. For both bert-base baselines and our method, we set learning rate=1e-4 and batch size=4 for few-shot training. For all experiments, we train the model for 20 epochs, and AdamW optimizer is used with the same linear decaying schedule as the pre-training stage. These hyperparameter settings are as the same with (Huang et al., 2021). For other hyper-parameter settings of the baseline methods, we simply follow the default settings. When implementing all methods, we adopt the "IO" labeling schema since we found that the "IO" schema is better than "BIO" schema under few-shot setting.
As for label word selection, we use the Data&LM searching along with the virtual method (Data&LM+Virtual) for all dataset and set the conflict ratio to  $Th = 0.6$ . When selecting the top  $k$  high-frequency words for virtual method, we set  $k$  to 6.
A.2 Sampling Algorithm
We conduct an exact sampling algorithm to ensure sampling exactly  $K$  samples for each class, which is different from the greedy sampling method used in previous methods (Yang and Katiyar, 2020). The algorithm is detailed in Algorithm 1. For all of the three datasets we used, we exactly obtained  $K$  samples for each class under all the  $K$ -shot setting.
A.3 Effect of Conflict threshold
Fig. 6 shows the impact of conflict threshold on 5-shot performance. As seen, for Data&LMSearch, lower conflict threshold results in improper label words that bring noisy annotated entities. Therefore, the performance promoting as the conflict threshold increasing. As for Data&LM+Virtual method, the impact of conflict words are less significant since multiple words are selected to construct the virtual vector.
A.4 Effect of  $k$  in virtual method
Fig.7 shows the impact of the choice of top  $k$  number for virtual method. We conduct
Figure 6: Impact of the conflict threshold.
Figure 7: Effect of the choice of top  $k$  number for virtual method.
experiments using the Data&LMSearch+Virtual method on CoNLL 5-shot dataset. We can see that the performance of the proposed method is robust to the choice of  $k$ , since it can consistently achieve good results when  $k >= 3$ . In our main experiments, we simply choose  $k = 6$  for all datasets.
A.5 Comparison with Comprehensive few-shot NER benchmark
We also conduct experiments on the few-shot benchmark provided by (Huang et al., 2021), in order to compare with the competitive baselines in the paper. These methods are implemented with the "Roberta-base" pretrained model. Therefore, we also implement our method based on "Robertabase" for fair comparison. Since the sampled data of OntoNotes is not available, we only experimented on the CoNLL'03 and MIT-Movie datasets. The results are shown in Table 6.
The results show that, our method outperforms over all baselines. Notice that the NSP method leverages the 6.8GB WiFiNE dataset for pretraining, and that the ST method performs self-training on the unlabeled data. However, our method still shows better results, which illustrates the effectiveness of the proposed objective over standard fine-tuning. Also, the proposed method can be further boosted with NSP and ST. We leave
Methods	CoNLL 5-shot	MIT-Movie 5-shot
LC	53.5	51.3
LC+NSP	61.4	53.1
Proto	58.4	38.0
Proto+NSP	60.9	43.8
LC+ST	56.7	54.1
LC+NSP+ST	65.4	55.9
EntLM	68.6	55.2
EntLM (Struct)	69.9	57.1
this for future works.
In Table 7, we show the label words selected with the Data&LM+Virtual method as examples.
Algorithm 1 Few-shot Sampling
Table 6: Comparison with the methods presented in (Huang et al., 2021). LC is linear classifier fine-tuning method. P is prototype-based training using a nearest neighbor objective. NSP is noisig supervised pretraining and ST is self-training. Notice that our method shows better results even without NSP and ST, and can also be further boosted by these two methods.
Require: # of shot K, labeled training set D with a label set Y.
1: S ← φ // Initialize the support set
2: for each class i ∈ Y do
3: Count[i] ← 0 // Initialize the counts of each entity class
4: end for
5: Shuffle D
6: for (X, Y) ∈ D do
7: Add ← True
8: for i ∈ Y do
9: Calculate Temp_count[i] // Calculate the mention number of class i in (X, Y)
10: if Count[i] + Temp_count[i] > K then
11: Add ← False // Skip current sample that violates the K-shot rule
12: end if
13: end for
14: if Add is True then
15: S ← S ∪ {(X, Y)}
16: Update {Count[i] ← Count[i] + Temp_count[i]} ∀i ∈ Y
17: end if
18: if Count[i] == K, ∀i ∈ Y then
19: break // Finish sampling
20: end if
21: end for
22: return S
Datasets	Label words (Data&LM+Virtual Search)
CoNLL'03	{"I-PER": ["Michael", "John", "David", "Thomas", "Martin", "Paul"], "I-ORG": ["Corp", "Inc", "Commission", "Union", "Bank", "Party"], "I-LOC": ["England", "Germany", "Australia", "France", "Russia", "Italy"], "I-MISC": ["Palestinians", "Russian", "Chinese", "Russians", "English", "Olympic"]}
OntoNotes*	{"I-EVENT": ["War", "Games", "Katrina", "Year", "Hurricane", "II"], "I-FAC": ["Airport", "Bridge", "Base", "Memorial", "Canal", "Guantanamo"], "I-GPE": ["US", "China", "United", "Beijing", "Israel", "Taiwan"], "I-LANGUAGE": ["Mandarin", "Streetspeak", "Romance", "Ogilvyspeak", "Pentagonese", "Pilipino"], "I-LAW": ["Chapter", "Constitution", "Code", "Amendment", "Protocol", "RICO"], "I-LOC": ["Middle", "River", "Sea", "Ocean", "Mars", "Mountains"], "I-NORP": ["Chinese", "Israeli", "Palestinians", "American", "Japanese", "Palestinian"], "I-ORG": ["National", "Corp", "News", "Inc", "Senate", "Court"], "I-PERSON": ["John", "David", "Peter", "Michael", "Robert", "James"], "I-PRODUCT": ["USS", "Discovery", "Cole", "Atlantis", "Coke", "Galileo"], "I-WORK_OF_ART": ["Prize", "Nobel", "Late", "Morning", "PhD", "Edition"]}
MIT-Movie	{"I-ACTOR": ["al", "jack", "bill", "pat", "der", "mac"], "I-CHARACTER": ["solo"], "I-DIRECTOR": ["de", "del", "stone", "marks", "bell", "dick"], "I-GENRE": ["fantasy", "adventure", "romance", "comedy", "action", "thriller"], "I-PLOT": ["murder", "death", "vampires", "aliens", "zombies", "suicide"], "I-RATING": ["13"], "I-RATINGS_AVERAGE": ["very", "nine", "well", "highly", "really", "popular"], "I-REVIEW": ["comments", "regarded", "opinions", "positive"], "I-SONG": ["heart", "favourite", "loves"], "I-TITLE": ["man", "woman", "night", "story", "men", "dark"], "I-TRAILER": ["trailers", "trailer", "preview", "glipse", "clips"], "I-YEAR": ["last", "past", "years", "decades", "ten", "three"]}