Less than One-shot: Named Entity Recognition via Extremely Weak Supervision
Authors: Letian Peng
Zihan Wang
Jingbo Shang
Venue: EMNLP 2023 Findings (2023)
Keywords: extremely weak supervison
few-shot learning
named entity extraction

Abstract: We study the named entity recognition (NER) problem under the extremely weak supervision (XWS) setting, where only one example entity per type is given in a context-free way. 
While one can see that XWS is \emph{lighter than one-shot} in terms of the amount of supervision,
we propose a novel method X-NER that can outperform the state-of-the-art one-shot NER methods.
We first mine entity spans that are similar to the example entities from an unlabelled training corpus.
Instead of utilizing entity span representations from language models, we find it more effective to compare the context distributions before and after the span is replaced by the entity example.
We then leverage the top-ranked spans as pseudo-labels to train an NER tagger.
Extensive experiments and analyses on 4 NER datasets show the superior end-to-end NER performance of X-NER, outperforming the state-of-the-art few-shot methods with 1-shot supervision and ChatGPT annotations significantly.
Finally, our X-NER possesses several notable properties, such as inheriting the cross-lingual abilities of the underlying language models.

Less than One-shot: Named Entity Recognition via Extremely Weak Supervision
Letian Peng and Zihan Wang and Jingbo Shang"
University of California, San Diego
(lepeng, ziw224, jshang]@ucsd.edd edu
Abstract
Table I: Different NER setups. We only show one entity type "Artist" here for simplicity. Our XWS setting shall be considered weaker than 1-shot since the example entity is given a context-free way.
We study the named entity recognition (NER) problem under the extremely weak supervision (XWS) setting, where only one example entity per type is given in a context-fiee free way. While that XWS is lighter than one-shot in terms of the amount of supervision, we pro- pose novel method X-NER that can outper- form the state-of-the-ae one-shot NER meth- ods. We first mine entity spans that are simi- lar the example entities from an unlabelled training corpus. Instead of utilizing entity span representations from language models, we find it more effective to compare the context distri- butions before and after the span is replaced by the entity example. We then leverage the top- ranked spans as pseudo-labels train an NER tagger. Extensive experiments and analyses on 4 NER datasets show the superior end-to- end NER performance of X-NER, outperform- ing the state-of-the-art few-shot methods with 1-shot supervision and ChatGPT annotations significantly. Finally, our X-NER possesses several notable properties, such as inheriting the cross-lingual abilities of the underlying lan- guage models."
Sctup
Example Supervision from Human
love the works by [Davinci]Anin: I'm big fan of [Van Goghlani art.
one
FEW-SHOT
Paintings by [Monet]asa are breathtaking.
[Michelangelo]] Joma incredible sculptures.
ONE-SHOT
love the works by [Duvincilame.
XWS (1-ENT)
Artist: Davinii
60-
50-
40-
A 30-
Tagger
FFF-NER
CharGPT
10-
(UU
Figure 1: Recent SOTA few-shot NER methods fall dra- matically as the available shots decrease on CoNLLO3, much weaker than our X-NER under the XWS setting.
on raw texts as pscudo labels. However, existing few-shot NER works typically achieve decent ac- curacy with a default setting using 5 shots (Huang et al., 2020); when the number of shots approaches 1, their performance drops catastrophicallyy Con- cretely, vary the number of shots for a recent state-of-the-arl (SOTA) few-shot NER method FFF- NER (Wang et al., 2022) and a standard sequence labeling NER tagger (Devlin et al., 2019) and show their performance in Figure 1. Under the stan- dard Fj metric in the CoNLLO3 dataset, they only achieved under-20 F1 scores in the 1-shot sce- nario, much lower than the desirable 60 F, they can achieve with 5-shots. We argue that such low performance is likely inevitable for methods that involve fine-tuning language models (LMs) on the limited training data, which a common practice in strong performing few-shot NER methods (Yang and Katiyar, 2020; Fu et al., 2021; Cui et al., 2021;
1 Introduction
Named Entity Recognition (NER) with Extremely Weak Supervision (XWS) is the task of perform- ing NER using only one entity example per entity type the supervision. As shown in Table 1, 1- shot NER is arguably the closest setting to XWS, however, XWS shall viewed as even weaker supervision, making it very compelling when the annotation is difficult or expensive.
To develop effective NER models under the XWS setting, the keystone is to find sufficiently large, high-quality pseudo labels from raw texts to train the NER model. A straightforward solu- tion is to adapt few-shot NER methods (with uncer- tainty quantification) to select confident predictions
Conresponding author.
'Our code is released at KomeijiForcc/X-NER

---

Feerral was Gremes
Eallyy Eype
XMS Suparsidan
Prefic topetatee
"essot NWSSSpprrdin
"Taing foort"
By 243,, Soll
Eatby Tuachndd - "Atter XWSS
"Frsla Karke"
Rewilag Aoir
beveanglt mary
surben minneritrnn viter thargo
Bowe
(Gneary) (nevnd)
Aulocastoad (s22ea scamcl
KL Livegaan
Fudiazza wel intollsaannt
Peeeh N831 SL. Dncagscc spuz)
Suoee SS4
Tepence
wm
Beyons
Win
Rolkkkktwad (obgral settecc
Divegacc
Fafiemntt wetl rntoafockeid
Figure 2: An overview of X-NER.
Ma et al., 2022). Therefore, we propose to obtain high-quality pseudo labels using the pre-trained LM's intrinsic understanding of entities, without any fine-tuning. We need to find metric (based on the LM) quantify the similarity of a span in an unlabelled corpus to target example entity.
standard sequence labeling NER tagger as a proof- of of-concept, while we note that noise-robust NER methods might bring even better performance. We conduct extensive experiments to compare our X-NER with various few-shot NER methods and ChatGPT for direct inference or pseudo data annotation. Remarkably, X-NER outperforms all these methods with a relatively small LM (i.c.. RoBERTa) as the backbone. Further analyses show that X-NER has several interesting properties: (1) it favors bidirectional masked LMs over similarly sized unidirectional CLM-based LMs. (2) it scales well with the capabilities of language models, and (3) it inherits the cross-lingual abilitics of the un- derlying language models.
is tempting to directly calculate a span's 's simi- larity with an entity based on their LM representa- tions, however, this approach faces two challenges: (I) entities are likely phrases of different lengths and it is non-trivial to define fixed-length vector representation for multiple tokens, and (2) context understanding is crucial for the generalization ca- pability of an NER model, however, the LM repre- sentations of an entity or a span emphasizes more on their surface tokens (Gu et al., 2021).
Our contributions are summarized follows.
In this paper, we propose X-NER as shown in Figure 2. Specifically, to remedy both challenges, we propose span test, which measures the change of the span's context when the span is swapped with the example entity. A lower change indicates that the span is of a higher similarity to the entity example. To quantify the change, we consider the context token-by bo-nn--bytokeen and gather the LM's predic- tion distribution of the tokens before and after the swap. Precisely, for a masked language modeling trained language model, the prediction distribution is obtained by replacing the token with mask and obtaining the LM's masked prediction probabili- ties; for a casual language modeling (CLM)-based LM, we consider all text before the desired token, and we gather the LM's prediction of that token. We then measure the -ettbacckLeibbeer (KL.) di- vergence between the prediction distributions be- fore and after replacing the span. We average the divergence over the context (only tokens to the right of the span for CLM obtain the final quan- tification of the change. We apply a cerpus-level ranking among all spans and select top- K entities to construct the pseudo-labels, We finally train a
We explore the challenging Extremely Weakly Supervised NER problem, and achieve substan- tially higher performances than previous SOTA few-shot methods.
We propose to use a distributional difference of an entity span's 's context when it is replaced with an entity example, which is much more effective than the span's 's representation itself.
We empirically studied method extensively. Besides strong performances, we additionally demonstrate its suitable language model types (prefer Masked LM over Causal LM), the effect of language model scaling (the larger the better), and its inherited cross-lingual NER ability.
Related Work
2.1 Few-shot NER
Current studies few-shot Named Entity Recog- nition (NER) can be divided into two categories: recognition from scratch and transferring from other domains. The latter category, known as do- main transferring NER (Huang et al., 2022; Li and Qian, 2023; Han et al., 2023), trains models on an-

---

notated NER datasets and then uses meta-learning learn new entity labels with limited data. Our work focuses on NER from scratch and there are two main approaches in this field. The first approach involves prompting by templates (Cui et al., 2021; Lee et al., 2022), while the second approach involves aligning NER with pre-training in the learning objective (Paolini et al., 2021; Wang al., 2022). These two approaches utilize either the definition of entities or unsupervised learning objectives to outperform traditional sequence tag- ging models. Unfortunately, with 1-shot or even less supervision, their performances suffer from overfitting and are not different from the tagger. To address this issue, our method applies training-free mining to collect entities with high quality and then trains the NER model with them.
et al., 2020) is another metric developed by Google Research that uses a fine-tuned BERT model predict human evaluation scores on a combined dataset of existing evaluation resources.
Explicit predictions from pre-trained language models (PLMs) have been widely used for various tasks beyond just generating implicit word repre- sentations. Perplexity, which is measure of the PLM's language modeling ability, has been applied not only for evaluating language models, but also for natural language inference (Holtzman et al.. 2021), question answering (Holtzman et al., 2021; Kumar, 2022), and text compression (Niu et al., 2019). Recently, Peng et al. propose to use di- vergence between masked prediction as semantic distance. We adapt this idea to span test and imple- ment entity mining with a high precision.
2.2 Extrememly Weak Supervision
Preliminary and Definitions
Extremely Weak Supervision (XWS) refers to the setting where a model expected to solve a cer- tain task with only the minimal supervision that a human requires to solve the task. It was initially proposed for text classification where extremely weak supervision would require the model to clas- sify texts into classes when the only information is the class name of each class (Wang et al., 202 2021b). The typical workflow of XWS in text classification involves aligning texts classes via Seed Match- ing (via surface forms or representations) on unla- belled text and then training a supervised model on such pseudo- pseudo-annotaate annotated dataset (Meng et al., 2020; Park and Lee, 2022; Zhang et al., 2021; Zhao et al.. 2022). Language model (zero-shot) prompting is another solution to XWS tasks, however, has been shown to be sub-par against these pseudo-dataset training approaches which can effectively learn on an unsupervised training set (Wang et al., 2023). Our work is the first that addresses NER in an XWS setting, where prior methods usually require about 5-shot sentence annotations per class. We show that our method can achieve reasonable performance with only one single example entity per class.
Named Entity Recognition is a sub-task of infor- mation extraction that seeks to locate and classify named entities in text into pre-defined categories. Given a sentence X = [_i..... .xn] where æ, is the i-th token in the sentence, and n is the total number of tokens. The objective of the NER task is to identify entities in the sentence. For specific entity type, each entity is represented by pair of boundaries (i, where Tzj [.,,... ..y] signi- fies that the tokens from i-th to j-th, inclusive, form a named entity of that type.
Pretrained Language Models such as BERT (De- vlin et al., 2019), GPT (Radford et al., 2019), and others (Lewis et al., 2020; Raffel et al., 2020) learn to predict the distribution of token æ; based on its context X through their training objective, rep- resented as fLM (arg X). The context, contingent on the model's architecture, could encompass to- kens around x; in BERT-like Masked Language Modeling (MLM) where x; is replaced by a special mask token during training, or just the preceding tokens in casual language modeling (CLM) models such as GPT. The distribution f(x X) reflects the probable occurrence of tokens in the vocabulary at position i given the context and is an effective dense representation of the context information. We will use this distribution to tell the difference between two contexts X and X'.
2.3
Plae- Metric
In recent years, several metrics have been proposed that leverage PLMs metries. BERTScore (Zhang al., 2020) computes token-level F1 scores be- tween generated and reference texts using contex- tualized word embeddings from BERT, achieving a better correlation with human judgments compared to traditional metrics like BLEU, BLEURT (Sellam
4 Our X-NER
In this section, we introduce our method X-NER to train a NER model with a raw corpus and given

---

only one entity example per type as supervision.
tokens in X with Z, and retrieve and average token representations for Z similarly as what we did for Enc(X).
4.1 Overview
We show an illustration of X-NER in Figure 2. We start with XWS Supervision and extract model annotated, or called pseudo, entities on the raw text data. We refer to the model as a miner. Pre- cisely, for each pscudo-annotater sentence X [ , there several (i.j) pairs iden- tified by the miners that signify pseudo entities Pej = [a...... . After that, new NER model is trained on the gpeudo-annotatee data. The major reason of needing a pseudo-anmnn annotation step, instead of directly serving the miner as the final model, is of inference efficiency. This pscudo-annotation then training step abstracts the latency of the miner pre processsss,ng, therefore, potentially slow min- ers can still be used efficiently.
Our proposed Divergence Distance: A more context-aware solution. While cosine similar- ity is straightforward method, it is not necessarily consistent with the primary objective for which the PLM encoder Enc(.) is optimized for. We propose divergence distance evaluation inspired by Peng et al. (2023) that accounts the language modeling objective more, The core insight is that instead of comparing the spans and Zey, which may have different lengths, directly, we compare their im- pact on a shared context. Since language models capture contextualized information, their context would reflect nuanced differences between the two spans. Formally, decompose the similarity
(x,7) [
Σ -D((xl|X), (#t)) X (kg-+)))
4.2 Context-Aware Entity Span Ranking
For pseudo NER labeling, our objective is to ex- tract spans Tty = [.,,... ... xy, from an unlabeled sentence X = [21, ... .n]] that are most similar to seed entity Z = [.,,... 2m] Thus, the span test is designed to evaluate the similarity S(-) of each possible span Taj to Z conditioning the context X.
where function that calculates the difference of a token Tk in context versus X =z-z-- after replacing the testing span with the seed entity; and, the negative sign translates difference to similarity.
Recall that in Section 3 we defined fLA (xrg X) as a language model's prediction of token for the context X. The prediction is a distribution over the vocabulary, which is nuanced quantification of the context. Therefore, we employ standard KL Divergence on the language model predictions as the distance metric D. That is,
(X.Z) 2.j
S(Zij. X)X).
We extract entities by selecting the spans with sim- ilarity scores 8 (X,2) above a threshold value.
Cosine Similarity: A straightforward solution. As traditional metric for vector representation similarity, we first use cosine similarity to introduce the workflow of our span testing, and to provide reasonable baseline.
(X.2)
Σ
kelimiud
Σ Σ plez = e
plae x
p(z= = =| [
kelmmkjjeE
Two-way testing w/ annotation-free context. Our span testing can be further extended to a two- way testing, where the seed span is provided to- gether with a seed context Y where Z Yu.w- We use the label name of the seed span to create an annotation context background to enable the woo-way testing: "The (ENTITY LABEL]: [SEED SPANJ". We don't view this additional supervi- sion, but rather akin an instruction template used in prompting methods.
Spus (Zty Z|X) Cosine Cosine(Enc((:e) (Enc(xag) (X), Enc (nc(
Specifically, we first use the PLM encoder Enc(-) to incorporate context X. To calculate Enc(x,,y X). we get the output representations from the i-th to j-th elements of Enc(X) from a language model, and then apply mean-pons mean-pooling (Wang et al., 2021a) over the tokens to get fixed length representation. For Enc(2) X ) replace the i-th to j-th
The similarity (X.Z) can be thus transformed
(X,Y) with the seed into the following expression $1 context Y.
In our case, our span-test; in other cases, it could be a NER model trained elsewhere or generative model that answer NER annotation prompts
(X,Y) i.aa
(X,Y,,)
(Y.X.))
8zj
+ Sue

---

(X.Ya,w) side, we only calculate two For the $ij neighboring elements & E (i- [ to regular- ize the imbalance in sentence length, which are ver- ified to be effective according to Peng et al. (2023). We avoid using mean- meaan-pooling.o as the decay of word interdependency in long sentences could inadver- tently cause the metric to favor short sentences. For
PromptMine ranks entities by the language model prompting probability using "X, Xtj is a/an <mask> entity." for MLM or "X, Xzg is a/an" for CLM. This method only requires the label names of the entity types, which has less su- pervision than XWS, that requires an additional example entity. We denote this setting as L. InstructMine queries 5-turbo (OpenAI, 2023) with instruction to annotate entities. All seed spans and templates are included in Ap- pendix A.
the Sane (x) side, as the tests on the seed context
use the same neighboring elements, Suew (Y,Xi.) calcu- lates all neighbors in the annotation- context.
Thus, the two-way testing adds | - [Z] calcula- tions to each span test.
X-NER w/ Cosine is a variant of X-NER that uses cosine similarity of the span and the entity mentioned in Section 4.2.
5 Experiments
We also compare with few-shot NER methods. Since they can not operate on the XWS setting, we provide them with 1-shot supervision, which is more supervision than XWS.
5.1 Datasets
To verify the effectiveness of our X-NER, we con- duct experiments four NER datascts ranging from general to specific domains.
ETAL (Chaudhary et al., 2019) is a method with pseudo-labeling to search for highly-confident entities that maximize the probability of BIO sequences.
CoNLLO3 (Sang and Meulder, 2003) is promi-
nent NER corpus on news corpus that encom- passes four general entity types: person (PER), location (LOC), organization (ORG), and miscel- laneous (MISC).
SEE-Few (Yang et al., 2022) expands seeded entities and applies an entailment framework to efficiently learn from a few examples.
Tweebank (Jiang et al., 2022) shares the same
four entity types as CoNLLO3 on tweet messages. WNUT17 (Derczynski et al., 2017) centers on emerging and rare entity types in user-generated content, such as social media posts.
TemplateNER (Cui et al., 2021) utilizes human- selected templates decoded by BART for each sentence-span pair during training.
EntLM (Ma et al., 2022) employs an MLM pre-trained model to predict tokens the in- put sentence, minimizing the gap between pre- training and fine-tuning but potentially introduc- ing context-related issues.
MIT Restaurant (Liu et al., 2013) features a variety of entities in the restaurant domain, such as dish names, restaurant names, and food types. A summary of the datasets along with correspond- ing dataset statistics can be found in Appendix B.
SpanNER (Fu et al., 2021) separates span detec- tion and type prediction, using class descriptions to construct class representations for matching detected spans, though its model designs differ from the backbone pre-trained model BERT.
When mining entity spans from unlabeled train-
ing sets, for all compared methods that need can-
didate spans, we focus on spans with length of
no more than 3 tokens and exclude any containing stop words. 3
FFF-NER (Wang et al., 2022) introduces new token types to formulate NER fine-tuning as masked token prediction or generation, bringing it closer to pre-training objectives and resulting in improved performance on benchmark datasets.
5.2 Compared Methods
We denote our method as X-NER. The default ver- sion of our X-NER is divergence distance-basse 2-way span testing with the annotation-free context technique. We also include 1 1-way variant which only test the seed entity as an ablation study. Under the XWS setting, our major baselines, which all share the same pseudo dataset training pipeline as X-NER, are as follows.
SDNET (Chen et al., 2022) pre-trains a T5 on silver entities from Wikipedia and then fine-tunes it on sew-shot examples.
In addition to the few-shot NER methods in pre- vious research, we additionally consider two base- lines that can help us ablate X-NER:
'Empirically, this simple rule can boost the efficiency while enjoying a 90% recall of all ground-truth entities, except 83.4% for Restaurant.
PredietMine follows a similar procedure as X- NER where it also first makes pseudo annotations

---

Table 2: NER results under XWS. implies using label names only.
CoNLL03
Tweebank
WNUT17
Restaurant
Method
Model
P.
R.
F.
R.
E
P.
R.
E
P.
R.
F
GPT2-XL
18.3
18.8
18.5
0.5
0.2
0.3
1.7
1.4
1.5
24.8
13.9
17.8
PromptMine
RoB-I.
23.7
20.4
22.0
0.4
0.1
0.2
1.2
0.4
0.6
21.7
11.1
14.7
XWS
InstructMine
ChatGPT
61.9
51.6
56.3
23.5
39.7
29.6
22.1
18.0
19.8
29.3
35.5
32.1
RoB-L (Cos)
37.7
20.3
19.5
19.1
7.9
11.2
X-NER
RoB-L (Div)
71.4
52.1
60.2
50.4
40.5
44.9
51.5
34.6
41.4
58.8
27.3
37.3
Table 3: Comparison between method's XWS per- formance and previous few-shot method's 1-shot perfor- mance on CoNLL03. Methods with geeudo-labeling t: SDNET uses language model pre-trained silver entities from Wikipedia.
'able 4: Precision @ 1000 of different entity ranking
methods on CONILO3.
Method
Setup
PER
L.C
ORG
MISC
PrompiMine
GPT2-XI.
7.3
32.2
28.4
3.4
Rob-1.
40.1
Method
P.
Cos-2 way
27.1
41.0
15.1
X-NER
Div-1 way
88.6
64..
24.2
12.9
XWS
X-NER
71.4
52.1
60.2
Div-2 98.4
97.9
62.3
82.2
Tagger
17.2
12.8
14.6
PredictMine"
13.5
17.2
15.1
3pt-3.5-turbo. X-NER has slight edge on CoNLL03 and Restaurant, while much better on
CertMine"
20.2
14.7
17.0
ETAI."
26.4
17.1
20.8
1-shot
SEE-Few
21.5
26.7
23.8
Tweebank and WNUT17. We suspect that this in- dicates preference for X-NER in noisy user text. Finally, the results also reveal that using the co- sine similarity of averaged token representations between the entity and the span as the metric leads to much worse performance. This justifies the im- portance of our divergence distance metric.
TemplateNFR
12.6
6.7
8.8
EaILM
13.6
29.0
18.5
SpanNER
56.1
3.8
7.1
NER
39.1
3.6
6.5
SDNET
56.6
Tagger
58.9
55.1
56.9
5-shot
FIT-NER
70.2
63.3
67.7
Table 3 presents the comparison of X-NER to methods that require 1-shot data. Due to limita- tion of resources, we run the experiments only on the standard CoNLLO3 benchmark. X-NER out- performs all previous few-shot baselines by very significant gap, indicating its substantially stronger ability in handling NER under extremely weak supervision. We also include 5-shot results of a vanilla tagger and the SOTA few-shot method FFF- NER. We find the X-NER can rival strong 5-shot NER methods while requiring much less supervi- sion (5 fully annotated sentences per entity type v.s. example entity for each entity type). This result also implies that there should be much more room for few-shot NER methods improve. Finally, PredictMine and CertMine have a similar (poor) performance as prior methods, indicating that the performance advantage of X-NER is not mainly be- cause of mining on unlabelled corpus, but rather our insights using the divergence metric.
on an unlabeled corpus, and then trains another NER model on the pseudo-annotated corpus. The difference is that PredictMine does not use our divergence distance, instead, it uses model pre- dictions from NER model trained on 1-shot data;
CertMine is similar to PredictMine but addi- tionally includes a sequential uncertainty estima- tion of predicted entities by calculating the token- wise average of negative log probability on each predicted span. CertMine only pscudo-annotates entities with the lowest 10% uncertainty.
Our X-NER by default uses roberta-large. For all the compared methods, use compara- ble size, if more favorable, language model for them (e.g., bert-large, bart-large, GPT-XL).
5.3 NER Performance Comparison
Table 2 presents the end-to-end NER evaluation results of different methods that operate under the XWS setting. While arguably PromptMine requires one less entity example than X-NER, its performance is much subpar compared to ours on all four datasets. When compared with
5.4 Span Mining Performance Analysis
To better understand why our divergence metric is better in mining entities, we conduct fine-grained
"The experiments are conducted in the timeframe May 23rd June 6th, which corresponds to the 0313 version

---

analysis.
XANER (tatEiTT))
Span Ranking Method Comparison Table presents the performance of various XWS meth- ods for entity mining on the NER datasets, mea- sured using Precision @ 1000. The results reveal that X-NER significantly outperforms other meth- ods across all datasets and shows remarkably high precision on specific entity types, particularly PER and MISC entities.
PER
L,00
ORG
MISC
Rocall
Figure 3: Comparison with k-shot tuning and ChatGPT.
LOC
SaNd
In the 2-way configuration with divergence dis- tance, X-NER outperforms other setups such as 1-way span testing and the cosine similarity met- ric. This verifies the mining precision to provide solid support X-NER's performance. X-NER also achieves an outstanding Precision @ 1000 score 98.4 for PER and 97.9 for LOC entities in the CoNLLO3 dataset. Comparatively, other methods such CertMine, PromptMine, and CosMine ex- hibit lower performance. CertMine achieves an av- erage Precision @ 1000 of 16.5, while PromptMine performs better with average scores of 22.9 (GPT2- XL) and 24.3 (RoBERTa-L). However, these scores still fall significantly short of X-NER's s perfor- mance.
464
IERISALEM ARESALLM
Mak
Sple
1100NA
Laske
scopTast
Sallame
Aertylion
Torket Rond
Nepelar
Nanber
Figure 4: Fine-grained entity results on LOC and MISC CoNLL03. The entities listed in order from right decomposed fine- fine-grained entity types for loca- tion and miscellaneous.
type.²
We present an example of decomposing LOC from CoNLLO3, in which we find a few representa- tive fine-grainer location entities. We first sample 1000 sentences with their ground truth annotations from the CoNLLO3 dataset and denote them by E. Next, we randomly sample 50 seed LOC entities from E: the goal is to identify a few representative entities from these 50 entities. For each sampled entity ej we use it as reference to mine enti- ties from & using X-NER. Because of fine- mne-ggained awareness of X-NER, the mined entities with the lowest distance not only location entities, but also locations of similar type as eq. We denote E, as the set of mined entities.
Span Extraction Method Comparison As illus- trated in Figure 3, precision-recall graphs compare our model's performance in 1-shot setting with NER extractors. When compared to PredictMine, X-NER performs similarly to a model trained with 10 ~ 20 shots, except for the ORG cat- egory. This can be attributed to the challenging di- versity of ORG entities. In comparison to ChalGPT, X-NER demonstrates competitive performance on LOC entities and superior performance MISC entities. A single-shot example isn't t enough to guide ChatGPT extract the correct entity phrases, due the difficulty in switching its memorized con- cept of MISC. However, ChatGPT maintains high performance in PER and ORG categorics, thanks to its comprehensive knowledge of entities. For these labels, X-NER's advantage lies in its ability to maintain high precision while using smaller sampling rate.
Then, we employ a maximal coverage algorithm to help us identify the unique E,'s that span over the location entities in E. We simply use greedy algorithm that progressively selects E, that con- tains the most entities that not contained in previous selections.
The results are shown in Figure 4, where we ad- ditionally include an experiment with the MISC en- tity type. By looking at the first selected entities for Location, we can find different fine-grained en- tity types. For instance, the first five seed spans for
6 Case Studies
6.1
Span Coverage and Subset Decomposition
As X-NER is a similarity-based method, we il- lustrate how we can use X-NER to discover finc- grained entity entity types a given coarse- coarse-grained entity
More discussion about fine-grained awareness can he found in Appendix D.

---

Preause ca, Accall (Late Longzaee
Proase to Accallio thep Langrage
RallCCh-Lary GPI-Maliam
Rocall
34
30
100
40
10
PER
LOC
Figure 5: Cross-lingual span mining Performance of X-NER on the MultiCoNTRR training dataset.
Prediim
Figure 6: The correlation between model scale and mining ability.
the LOC category include the country (Sri Lanka, Mexico, Syria), city (VIENNA), and region (Cen- tral Asia). Thus, we conclude that our similarity evaluation successfully decomposes coarse-grained categories into fine-grained ones.
Ability vs. Model Size the entity mining ability of our X-NER commen- surately improves with the backbone model size. It's noteworthy that even models of small size dis- play decent mining capacity, suggesting that more compact models can serve viable starting points for this task. As these models grow into base and large sizes, there is a noticeable enhancement in their mining ability. This indicates direct, posi- tive correlation between model size and extraction efficiency, validating the scalability of our method and potentially guiding future development with larger models. Furthermore, our analysis reveals a consistency in performance rankings across differ- ent entity types, irrespective of model scale. This suggests that some entities may inherently present more challenges for extraction than others, a con- sistent factor that could inform future strategies for mining optimization.
The figure highlights that
6.2 Cross-Lingual Results
Our entity mining technique holds promising po- tential as a useful tool for languages with fewer resources. This is because it can efficiently ex- tract entities within single span, eliminating the need for expert annotators. Furthermore, we ve illustrated how method can be successfully ap- plied in cross-lingual contexts. For example, as shown in Figure 5, we utilized an English span to extract entities in various other languages, lever- aging the cross-lingual masked language model, XML-Roberta-L. We test the quality of pseudo en- tities labeled in the cross- looss-l dataset. During our comprehensive experimental process, choose to utilize the MultiCoNER (Malmasi al., 2022) dataset, which encompasses data from 12 languages, as platform to extract multilin- gual person entities. Our choice of the dataset was driven by its diversity and comprehensiveness, enabling us to test our approach across a broad range of linguistic contexts. When compared to En- glish, X-NER demonstrates remarkable resilience and effectiveness in most Latin-based languages. This proved the robustness of entity mining technique, reinforcing its potential universally applicable tool in a broad array linguistic en- vironments. While the performance is somewhat reduced in non-Latin languages, our X-NER still manages to identify relevant entities, showcasing the universality of our technique.
MILM vs. CLM In our experimental sctup, we generally employ bidirectional PLMs as opposed to their unidirectional counterparts. This strategic decision was primarily motivated by our belief that bidirectional context information is of paramount importance when assessing divergence distance, critical factor in our methodology. This section aims draw a comparison between the informa- tion extraction capabilities of these two types of PLMs. For the unidirectional, CLM-based GPT2, we adapted our label context to follow the structure of "[SEED SPAN] is a/an JENTITY LABEL] name" to make it compatible with the unidirectional con- text understanding. The results showed the MLM- based roberta-le tabbert--lagee large (355M) to considerably out- performed the CLM-based gpt2-mediuu um (345M). This marked performance difference lends support to our initial hypothesis, demonstrating the inherent advantage of bidirectional language modeling
6.3 Backbone Analysis
We compare different backbone models and present their performances in Figure 6.

---

Method
Sampling
Training
Inference
its efficacy across a broader array of NER scenar- ios, and refining the span test for more nuanced linguistic contexts.
Tagger
O
O(N)
PredictMine
O(N)
O
O(N)
CertMine
O(N)
O
O(N)
TemplateNER
O
O(NL)
Limitations
EatLM
O
O(N)
SpanNER
O(N)
O(N)
FFE-NER
O(VL)
O(N
While our proposed method, X-NER, has shown commendable performance in NER under ex- tremely weak supervision setting, it does come with certain limitations. Firstly, the time complexity of generating pscudo-labels is significant, largely due the need for similarity computation between the context distributions of entity spans and exam- ple entities an unlabelled training corpus. This makes the process less time-efficient, particularly in cases of extensive training data. Secondly, our approach heavily depends on the understanding of the PLM used. If the PLM fails to adequately com- prehend or interpret the entity, the method might fail to generate effective pseudo-labels, leading less accurate NER tagging. Lastly, our method may sometimes wrongly classify entity spans that are encapsulated within another entity. For in- stance, in the case of the entity "New York Univer- sity", the method might incorrectly identify "New York" as a separate location entity. Despite these challenges, our method demonstrates substantial promise, though it points future work for refining the approach to mitigate these limitations.
X-NER
O(NL)
O(N) O(N)
Table Time complexity analysis of different few-shot NER frameworks.
Method
#Instance
#Label
Time
CoNLLO3
14.0K
4
3.7H
Tweehank
1.6K
4
0.5H
WNUTI7
3.4K
6
1.7H
Restaurant 7.7K
8
1.8H
Table 6: The time cost of X-NER for sampling on dif- ferent training datasets.
effectively capturing context and improving entity extraction.
6.4 Time Complexity
We present a time complexity analysis that com- pares the time consumed by different methods in Table 6, which will be added to the new version of our paper. N refers to the number of instances for sampiing/tainnngiinfereeee. refers to the length sentences, where the average length can be used for approximation.
Ethical Consideration
From the table above we can see our X-NER is efficient as normal tagger and faster than some baselines like FFF-NER during inference since train tagger based the mined corpus. The main time burden for X-NER is from mining where test about 3/. spans for N instances in the unla- beled corpus.
Our work focuses on the traditional NER task, which generally does not raise ethical consider- ation.
References
Aditi Chaudhary, Jiateng Xie, Zaid Sheikh, Graham Neubig, and Jaime G. Carbonell, 2019. little anno- tation does a lot of good: A study in bootstrapping low-resource named entity recognizers. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference Natural Language Pro- cessing, EMNLP-IJCNLPP 2019, Hong Kong, China, November 3-7, 2019, pages for Computational Linguistics.
7 Conclusions and Future Work
In summary, our research offers a groundbreaking Extremely Weak Supervision method for Named Entity Recognition, which significantly enhances I-shot NER performance. We generate high-quality pseudo annotations without requiring fine-tuning by capitalizing pre-trained language models' inherent text comprehension. Additionally, we ad- dress challenges related span representation and context integration with novel span test, leading to superior results on NER tasks compared to many established methods. Future work will focus on im- proving the scalability of this approach, enhancing
Association
Jiawei Chen, Qing Liu, Hongyu Lin, Xianpei Han, and Le Sun. 2022. Few-shot named entity recognition with self-describing networks. Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Popers), ACL 2022, Dublin, Ireland. May 22-27, 2022, pages 5711- 5722. Association for Computational Linguistics.

---

Leyang Cui. Yu Wu. Jian Liu, Sen Yang, and Yue Zhang. 2021. Template- based named entity recognition us- ing BART. In Findings of the Association for Com- putational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/JCCLLP 2021 of Findings of ACI., pages 1835-1844 1855. Association for Computational L Linguistics.
Youcheng Huang, Wenqiang Lei, Jie Fu, and Jiancheng Lv. 2022. Reconciliation of tre-trained models and prototypical neural networks in few-shot named en- tity recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arub Emirates, December 7-11, 2022, pages 1793-1807. Association for Computational Linguistics.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@ EMNLP 2017, Copen- hagen, Denmark, September 7, 2017, pages 140-147. Association for Computational Linguistics.
Hang Jiang, Yining Hua, Doug Beeferman, and Deb Roy. 2022. Annotating the tweebank corpus named entity recognition and building NLP mod- els for social media analysis. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 7199-7208. European Language Resources Association.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-4186. Association for Computational Linguisties.
Sawan Kumar. 2022. Answer-level calibration for free- form multiple choice question answering. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 665-679. Association for Computational Linguistics.
Dong- Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mi- tani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2022. Good examples make A faster learner: Sim- ple demonstration-bant NER. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2687-2700. Association for Computational Linguistics.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. Spanner: Named entity re-frecognition as span pre- diction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing, ACL/UCNNP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 7183-7195. Association for Computational Linguis- tics.
learning for low-resource
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising for natural language generation, tramslation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, ACI. 2020, Online, July 5-10, 20220, pages 787 7871-7880. Association for Computational Linguisties.
Xiaotao Gu, Zihan Wang, Zhenyu Bi, Yu Meng, Liyuan Liu, Jiawei Han, and Jingbo Shang. 2021. Uephrase: Unsupervised context-aware quality phrase tagging. In Proceedings of the 27th ACM SIGKDD Confer- ence on Knowledge Discovery & Data Mining, pages 478-486.
to-nuueee---eeeeeee pre-training
Chengeheng Han, Renyu Zhu, Jun Kuang, FengJiao Chen, Xiang Li. Ming Gao, Xuezhi Cao, and Wei Wu. 2023. Meta-learning triplet network with adap- tive margins for few-shot named entity recognition. CoRR, abs/2302.07739.
Yongqi Li and Tieyun Qian. 2023. Type-aware decom- posed framework for few-shot named entity recogni- tion. CoRR, abs/2302.063977.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competi- tion: Why the highest probability answer isn't 't always right. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event/ Punta Cana, Domini- can Republic, 7-17 November, 2021, pages 7038- 7051. Association for Computational Linguistics.
Jingjing Liu, Panupong Pasupat, Scott Cyphers, and James R. Glass. 2013. Asgard: A portable archi- tecture for multilingual dialogue systems. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pages 8386-8390. IEEE.
Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li. Qi Zhang, and Xuanjing Huang, 2022. Template- free prompt tuning for few-shot NER. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022,
Jiaxin Huang, Chunyuan Li. Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng. Jianfeng Gao, and Jiawei Han. 2020. Few-shot named entity recognition: A comprehensive study. CoRR, abs/2012, 14978.

---

pages 5721-5732. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder, 2003, Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with IILT-NAACL 2003, Edmonton, Canada, May 31 June 1, 2003, pages 142-147, ACI..
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. Semeval-2022 task 11: Multilingual complex named entity recog- nition (multiconer). In Proceedings of the 16th In- lernational Workshop on Semantic Evaluation, Se- mEval
@ NAACL 2022, Seattle, Washington, United States, July 14-15, 2022, pages 1412-1437. Associa- tion for Computational Linguisties.
Thibault Sellam, Dipanjan Das. and Ankur P. Parikh. 2020. BLEURT: learning robust metries for text generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, 2020, Online, July 5-10, 2020, pages 788 1-89-. Association for Computational Linguistics.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji. Chao Zhang. and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 9006-9017. Association for Computational Linguistics.
Zihan Wang, Chengyu Dong, and Jingbo Shang. 2021a. "average" approximates "first principal component"? an empirical analysis on representations from neural language models. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5594-5603, Association for Computational Linguis- tics.
Tong Niu, Caiming Xiong, and Richard Socher. 2019. Deleter: Leveraging BERT to perform unsupervised successive text compression. CoRR, abs/1909.03223,
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08777.
Zihan Wang. Dheeraj Mekala, and Jingbo Shang. 2021b. X-class: Text classification with extremely weak su- pervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisticy: Human Language Technologies. NAACL-HLI 2021, Online, June 6-11, 2021. pages 3043-3055 Association for Computa- tional Linguistics.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma. Alessandro Achille, Rishita Anubhai. Cícero Nogueira dos Santos, Bing Xiang, and Ste- fano Soatto. 2021. Structured prediction as transla- tion between augmented natural languages. In 9th International Conference on Learning Representa- tions, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. Open
Zihan Wang, Tianle Wang, Dheeraj Mekala, and Jingbo Shang. 2023. A benchmark on ex- tremely weakly supervised text classification: Rec- oncile seed matching and prompting approaches. hutps://araix.or//ubb// 127.7
Seongmin Park and Jihwa Lee. 2022.
LIME:
text classification without seeds. In Proceedings of the 29th International Confer-
ence on Computational Linguistics, COLING 2022,
Zihan Wang, Kewen Zhao, Zilong Wang. and Jingbo Shang. 2022. Formulating few-shot fine-tuning to- wards language model pre-training: A pilot study on named entity recognition. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3186-3199, Association for Computa- tional Linguistics.
Gyeongju. Republic of Korea, October 12-17, 2022, pages 1083-1088. International Committee on Com- putational Linguistics.
Letian Peng. Zuchao Li, and Hai Zhao, 2021. Contextu- alized semantic distance between highly overlapped texts. CoRR, abs/21 10.01 176.
Letian Peng. Zuchao Li, and Hai Zhao. 2023. Contextu- alized semantic distance between highly overlapped texts. In Findings of the Association for Compu- tational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 10913-10931. Association for Computational Linguistics.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 2365-6375 Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Zeng Yang, Linhai Zhang, and Deyu Zhou. 2022. See- few: Seed, expand and entail for few-shot named entity recognition. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, COLING 2022, Gyeongju. Republic of Korea, Oc- lober 12-17, 2022, pages 2540-2550 International Committee on Computational Linguistics.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach, Learn Res,, 21:140: 1-14677

---

Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021. Weakly-supervinne text clas- sification based on keyword graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Procexsing, EMNLP 2021, Vir- uual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 2803-2813. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Xuandong Zhao, Siqi Ouyang. Zhiguo Yu, Ming Wu, and Lei Li. 2022. Pre-trained language models be fully zero-shot learners. arXiv preprint arXiv:2212.06950.

---

A
Seed Entity and Template
Dataset
CoNLLO3
Tweebonk
WNUTI7
Restaurant
Domain
News
Social Media
Social Media
Review
#Train
140K
1.6A
3.1K 1.3K
7.7K
#Test
abel 3.5K
1.2K
1.5K
Entity Label
Seed Span
CoNLL03
PER
Masaki Yoshida
Table 9: The statistics of the NER dataset in our experi- ments.
LOC
Singapore
ORG
Boston Celtics
MISC
Japanese
Reaction Supervisors Famous <mask> (Ala: Turing]
Divergenes Disancce
20
Tweebank
PER
Masaki Yoshida
John Neumann
13
Los Angeles
LOC
Singapore
Microsoft
ORG
Boston Celtics
Japaness
MISC
Japanese
Person
Masaki Yoshida
mattermate
sefeester
plosinn
anvotey
prigemmnne
Authar
mighonr
phinoapten
Pemia
WNUT17
Location
Singapore
Creative work
#BattlsttrGalactica
Resctin of Supervison (amora Smake? [Abe Turing]
Group
Boston Celtics
Diverpenes Disance
20
John Neumaan
Corporation
Microsoft
Albert Einstcia
Product
iPhone
Thomas Edison
Napulcon Bonaparte
Restaurant name
McDonald's
Restaurant
Location
nearby
6
Dish
chicken sandwich cheap
Rue nowfrmmntiio
pipaloot
invenoy
antibot
migeverr
phlkaapler
Percier
Price
Rating
five star
Cuisine
Japanese
Figure 7: Supervisor reaction to span test for coarse and fin-grained entities.
Table 7: Entity Labels and Seed Spans
sentence into the pscudo dataset, we avoid too- long sentences which more likely suffer from miss- ing annotation. The maximal length threshold for CoNLLO3 and Tweebank is 30, while for WNUTI7 and Restaurant is 20.
Template
Based on these examples:
<example>
Bracket and label the named entities in the sen-
For the similarity threshold, we keep it constant for different labels the same data. For divergence distance, the threshold for CoNLLO3 and WNUT17 is < 0.15, while for Tweebank and Restaurant is < 0.20. This policy is also applicable to cosine sim- ilarity where the threshold is > 1.999. For Prompt- Mine, since the score is not evenly distributed, we label the top 2% span for each entity category.
tence:
<sentence>
Table 8: The XWS template used for ChatGPT in In- structMine.
B Dataset Statistics
D Fine-grained Interpretation
The statistics of the datasets in our experiments presented in Table 9.
Reaction of Neighbors (Supervisors) We fur- ther explore how the pre-trained LM under- stands the semantics of the seed entity. In Figure 7, present ing how the masked predictions words (supervisors) change when Alan Turing is substituted with few selected entities of the same coarse-grained. Here, supervisor dis- tance refers to the weighted divergence p(rx
Pseudo Dataset Construction
We also include several constraints to construct high-quality pseudo dataset. These constraints are applied to the span testing-based meth- ods: PromptMine and X-NER. First, we only test sentences under certain lengths (token numbers). Since each discovered entity will add the whole
studies demonstrat- neighbor

---

Artiet likeihoodd
NER's ability distinguish fine-grained entities. Notably, Leonardo da Vinci, who is both an artist and scholar, is positioned closest to the origin, indicating that X-NER effectively capturing the dual nature of Leonardo da Vinci's 's expertise.
Antist
Scbolar Likellhoo
Songro Botiecoll
Scholar
Pickggr Mickeo
Robert Bojle
Fdlgara Minick
Johgnnes Schole Grultherr
Goufyheershheelinn Schredinger
Ergin
Leonardo Vinci
Nitaold
Ingac
Afbert Finkeein
10
Figure 8: Decomposed probability distributions. Nega-
tive log probability axes.
of x) p(ax=o(X)
X of supervisor We plot the distance of supervisors with the top-K K weight p(#x= x()).
For entities with distinct coarse-grasd -gaiine cate- gories, most supervisors are activated to identify the differences, showcasing their strong ability for coarse-grained categorization. For other PER enti- ties, a supervisor is activated only when the entities differ in the attribute it represents. For example, Albert Einstein activates the mathematician super- visor but not the scientist supervisor. Conversely, Thomas Edison activates both the mathematician and scientist supervisors but not inventor su- pervisor. Interestingly, Napoleon Bonaparte, the PER entity most dissimilar to Alan Turing, activates the most supervisors, highlighting the sensitivity of supervisors to fine- fine-grained differences among entities.
Decomposed Divergence Distance We further explore how the performance of our X-NER is supported by the pre-trained LM's understanding of named entities. By examining the varying re- sponses supervisors to different entities, we can break down the divergence distance for each su- pervisor. This decomposition is demonstrated for artist and scholar in Figure 8. In this case, we select 20 renowned artists and scholars, position- ing them based on the negative log probability of the artist and scientist predictions from the famous <mask> [NAME] prompt. Interestingly, we can ob- serve that the artists and scholars automatically cluster into two distinct groups, showcasing X-


Reviews:
[{"review_id": "NTliaH1vm2", "reviewer": "Reviewer_c5pW", "summary": "", "questions": "QA Will different seed spans have significant differences in results?\n\nQB Why does two-way have such a significant impact on MISC and ORG in Table 4?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for expressing interest in our work! We appreciate your curiosity and are excited to delve deeper into our framework for named entity recognition under extremely weak supervision. We have incorporated more experiments and comparisons to support the claims and contributions of our work. \n\n- **Pseudo-labeling Baselines (For A, B)**\n\n For pseudo-labeling baselines, we have already included mining with a predictor **PredictMine** and **CertMine** and variants with cosine similarity or prompting probability (**PromptMine**) in the current version. We are also glad to add more distantly-supervised or self-training baselines in the following table. \n\n| Method     | P.     | R.     | F.     |\n|------------|-------|-------|-------|\n| **XWS**        |        |        |       |\n| X-NER      | 71.4  | 52.1  | 60.2  |\n| **1-shot**     |        |        |       |\n| Tagger    | 17.2  | 12.8  | 14.6  |\n| PredictMine  | 13.5  | 17.2  | 15.1  |\n| CertMine    | 20.2  | 14.7  | 17.0  |\n| **ETAL**      | 26.4  | 17.1  | 20.8  |\n| SEE-Few   | 21.5  | 26.7  | 23.8  |\n| TemplateNER | 12.6  | 6.7   | 8.8   |\n| EntLM     | 13.6  | 29.0  | 18.5  |\n| SpanNER   | 56.1  | 3.8   | 7.1   |\n| FFF-NER   | 39.1  | 3.6   | 6.5   |\n| **SDNET**   | 56.6  | 44.1  | 51.3  |\n| **5-shot**    |        |        |       |\n| Tagger    | 58.9  | 55.1  | 56.9  |\n| FFF-NER   | 70.2  | 65.3  | 67.7  |\n\n**Bold Methods** are newly added to the next version. **ETAL** is a method with pseudo-labeling to search for highly-confident entities that maximize the probability of BIO sequences. **SEE-Few** expands seeded entities and applies an entailment framework to efficiently learn from a few examples. **SDNET** pre-trains a T5 on silver entities from Wikipedia and then fine-tunes it on few-shot examples.\n\nOur results add more distantly-supervised and self-training baselines and our X-NER significantly outperforms them. Remarkably, our method outperforms SDNET, which uses pre-trained models on silver NER labels from Wikipedia. \n\n- **Time Complexity (For C)**\n\nWe add a time complexity analysis that compares the time consumed by different methods in the following table, which will be added to the new version of our paper. $N$ refers to the number of instances for sampling/training/inference. $L$ refers to the length of sentences, where the average length can be used for approximation. \n\n| Method      | Sampling | Training | Inference |\n|-------------|----------|----------|-----------|\n| Tagger      | -        | O(N)     | O(N)      |\n| PredictMine | O(N)     | O(N)     | O(N)      |\n| CertMine    | O(N)     | O(N)     | O(N)      |\n| TemplateNER | -        | O(N)     | O(NL)     |\n| EntLM       | -        | O(N)     | O(N)      |\n| SpanNER     | -        | O(N)     | O(N)      |\n| FFF-NER     | -        | O(NL)    | O(NL)     |\n| X-NER       | O(NL)    | O(N)     | O(N)      |\n\nFrom the table above we can see our X-NER is as efficient as a normal tagger and faster than some baselines like FFF-NER during inference since we train a tagger based on the mined corpus. The main time burden for X-NER is from mining where we test about $3L$ spans for $N$ instances in the unlabeled corpus. \n\n| Method      | #Instance | #Label | Total Time  |\n|-------------|-----------|--------|-------|\n| CoNLL03     | 14.0K     | 4      | 3.7H  |\n| Tweebank    | 1.6K      | 4      | 0.5H  |\n| WNUT17      | 3.4K      | 6      | 1.7H  |\n| Restaurant  | 7.7K      | 8      | 1.8H  |\n\nThe result is from runs on a single RTX-3090. The time costs are controlled under 4 hours on all datasets, which verifies our X-NER to be an applicable method for NER.\n\n- **Seed Span Analysis (For D and QA)**\n\nWe present an analysis of the impact of different seed examples. As we cannot present a precision-recall curve on OpenReview, we describe the setting and results from our experiments. We test four seed examples \"Masaki Yoshida\", \"John Smith\", \"Haruki Murakami\", and \"Alan Turing\". We find the seed examples do not have a significant impact on the precision-recall curve. Additionally, general names like \"Masaki Yoshida\" and \"John Smith\" outperform famous names, which might be attributed to their less bias to entities with similar properties like career and title.\n\n- **Similarity Threshold (For E)**\n\nBased on our experiments, we find a threshold between $0.15$ to $0.20$ works well across all tested datasets for mining on the unlabeled corpus. Thus, one can try these two thresholds when facing a new dataset. \n\n- **Two-way Testing (For QB)**\n\nThe most significant improvement of two-way testing on ORG and MISC can be attributed to the contextual diversity of these entities and the replacement of tested spans to seed examples provides a fair context to calculate the distance between them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. This paper presents a challenging Extremely Weak Supervision NER setting.\n\nB. The method described in this paper is reasonable and has achieved excellent results.\n\nC. Experimental results have demonstrated the excellent cross-lingual effectiveness of the method proposed in this paper.", "reasons_to_reject": "A. The experiment in Table 3 seems unfair because compared to incorporating pseudo-labeled data, 1-shot fine-tuning clearly performs worse.\n\nB. The setting and methodology of this article are similar to distant supervision and self-training, but the baseline lacks comparison with such methods.\n\nC. The method described in this paper seems to be time-consuming and lacks a comparison of the time spent on each method.\n\nD. This paper did not analyze the potential impact of different seed spans.\n\nE. There is no mention of how to set a similarity threshold for a new dataset.", "questions_to_authors": "QA Will different seed spans have significant differences in results?\n\nQB Why does two-way have such a significant impact on MISC and ORG in Table 4?", "paper_topic_and_main_contributions": "This paper presents a challenging Extremely Weak Supervision NER setting, i.e., only one entity mention is provided for a entity type, and proposes a pseudo label method to resolve it. This paperutilizes the ability of language model to contextually model and obtain pseudo labels from unlabeled text, forming training data. Experimental results show that the method proposed in this paper has demonstrated excellent performance, surpassing both the 1-shot approach and the method of generating pseudo-labels through chatgpt.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LBeAYSwqdb", "reviewer": "Reviewer_o4z4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely thankful for your appreciation of our work! You can refer to our responses to other reviewers to view the updates we make to further improve the quality of the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1. The proposed approach is novel and effective.\n2. It is amazing that the proposed method for the XWS setting outperforms many methods designed for 1-shot or few-shot settings.\n3. The experiments are solid, and the analysis is convincing and informative.", "reasons_to_reject": "The paper appears to be free of significant weaknesses.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper targets the extremely weak supervision (XWS) setting of named entity recognition (NER), a setting that is rarely studied in previous research. One core step of the proposed approach is the construction of pseudo-labels from the training corpus. The authors propose a smart way to perform span test based on language models, by comparing the context distributions before and after the span is replaced by the entity example.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CdD0qR5oSy", "reviewer": "Reviewer_anZR", "summary": "", "questions": "** In line 315, “We don’t view this as additional supervision, but rather akin to an instruction template used in prompting methods “ ==> This statement is debatable. Prompts are a form of additional supervision. Not labeled does not mean they are not additional information.\n\n** In Table 3, \"Due to a limitation of resources, we run the experiments only on  the standard CoNLL03 benchmark.\"    Can you be more specific about the resource limitation preventing doing 1-shot experiment for other datasets? It seems to be easy to do. For instance, you can take one sentence for each entity type from the train or dev split. \n \n** In Table 3, : Do PredictMine and CertMine (slight variations of X-NER) use the same unlabeled corpus and K as X-NER to generate pseudo labels?\n\n** It is more conventional to call the provided example \"seed example\" rather than  \"seed span\" as it is just a term not a sentence.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for showing enthusiasm towards our research! We value your inquisitiveness and look forward to diving further into our framework for named entity recognition with minimal supervision (XWS). To bolster our claims, we have added the comparison with more baselines and provided a detailed breakdown of our implementation.\n\n- **More Comparisons and References** \n\nWe have added the comparison with your mentioned works. We are also going to add descriptions and citations of those works in the related works section.\n\n| Method     | P.     | R.     | F.     |\n|------------|-------|-------|-------|\n| **XWS**        |        |        |       |\n| X-NER      | 71.4  | 52.1  | 60.2  |\n| **1-shot**     |        |        |       |\n| Tagger    | 17.2  | 12.8  | 14.6  |\n| PredictMine  | 13.5  | 17.2  | 15.1  |\n| CertMine    | 20.2  | 14.7  | 17.0  |\n| **ETAL**      | 26.4  | 17.1  | 20.8  |\n| **SEE-Few**   | 21.5  | 26.7  | 23.8  |\n| TemplateNER | 12.6  | 6.7   | 8.8   |\n| EntLM     | 13.6  | 29.0  | 18.5  |\n| SpanNER   | 56.1  | 3.8   | 7.1   |\n| FFF-NER   | 39.1  | 3.6   | 6.5   |\n| **SDNET**   | 56.6  | 44.1  | 51.3  |\n| **5-shot**    |        |        |       |\n| Tagger    | 58.9  | 55.1  | 56.9  |\n| FFF-NER   | 70.2  | 65.3  | 67.7  |\n\n**Bold Methods** are newly added to the next version. **ETAL** is a method with pseudo-labeling to search for highly-confident entities that maximize the probability of BIO sequences. **SEE-Few** expands seeded entities and applies an entailment framework to efficiently learn from a few examples. **SDNET** pre-trains a T5 on silver entities from Wikipedia and then fine-tunes it on few-shot examples.\n\nOur results add more distantly-supervised and self-training baselines and our X-NER significantly outperforms them. Remarkably, our method outperforms SDNET, which uses pre-trained models on silver NER labels from Wikipedia.\n\n- **Implementation Details**\n\n (1) For swapping candidates, we mention in \\textbf{5.1} that all spans with no more than $3$ tokens and without stop words are selected for each sentence. (2) The construction of the pseudo-labeling dataset is in \\textbf{Appendix C}, we set a limitation of similarity and length for datasets. (3) The $K$ is determined by a similarity threshold that is also mentioned in \\textbf{Appendix C} and we will move these into the main paper in the updated version. (4) You are correct that prompts usually provide extra supervision for the framework, but the prompt in our work is simply \"<label name>: <span example>\", which provides very little extra supervision. (5) In Table 3, as 1-shot methods clearly lag behind our X-NER, running all compared methods on all the datasets takes a lot of GPU resources, with little additional value. (6) **PredictMine** and **CertMine** use exactly the same unlabeled corpus (the corresponding training dataset for each test dataset) and the same $K$ as our X-NER. (7) Your suggestion on replacing \"seed example\" with \"seed span\" clearly captures the meaning of this element in our experiment, we apply the replacement to the version of our paper.\n\n- **Multiple Seed Examples** \n\nWe study single seed examples rather than multiple ones due to two reasons: (1) Single seed example has lesser burden in providing supervision lessens users effort. One can definitely explore the case where more supervision is allowed, building upon our results. (2) Single seed example is the arguably minimum supervision necessary for a human to solve NER, so we design a method to make machines work under this setting. (3) As an example of incorporating multiple seed examples, we experiment with pseudo-labeling with multiple seed examples by averaging the similarity scores. Experiment results show our X-NER benefits from more seed examples. For instance, using two person seed examples \"Masaki Yoshida'' and \"John Smith\" will achieve a better precision-recall curve on CoNLL03 than only using ``Masaki Yoshida'' in our current experiments. We will add this result to the next version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "** Few-shot NER is an important area to apply NER in real-world problems.\n** The use the KL divergence between the context words before and after replacing a candidate span with the seed example to decide the likelihood of the span belonging to the same category as the seed example is new and promising. \n** The paper shows several ablation experiments providing good insights.", "reasons_to_reject": "** The idea is few (one)-shot NER has been studied by many researchers both with and without LLMs (See missing references below) . The comparisons provided in this paper are not sufficient.  This paper compares X-NER against pure few-shot methods (e.g, FFF-NER) and non few-hot method (SpanNER) but does not compare X-NER method against existing methods which also utilize unlabeled data.\n\n** The paper does not provide important details on how X-NER works. (1) how do you generate  the candidate spans to swap with the seed entity? do you consider all possible spans up to a certain length as in SpanNER or do you use some heuristics?  (2) how did you construct the unlabeled corpus for the experiments? the performance of X-NER will depend on the quality of the unlabeled data. (3) what was the value of K in top-K entities selected by the entity miner?\n\n** The argument of using only one example is not convincing, as providing a few examples with the class name is very easy to do in practice.", "questions_to_authors": "** In line 315, “We don’t view this as additional supervision, but rather akin to an instruction template used in prompting methods “ ==> This statement is debatable. Prompts are a form of additional supervision. Not labeled does not mean they are not additional information.\n\n** In Table 3, \"Due to a limitation of resources, we run the experiments only on  the standard CoNLL03 benchmark.\"    Can you be more specific about the resource limitation preventing doing 1-shot experiment for other datasets? It seems to be easy to do. For instance, you can take one sentence for each entity type from the train or dev split. \n \n** In Table 3, : Do PredictMine and CertMine (slight variations of X-NER) use the same unlabeled corpus and K as X-NER to generate pseudo labels?\n\n** It is more conventional to call the provided example \"seed example\" rather than  \"seed span\" as it is just a term not a sentence.", "paper_topic_and_main_contributions": "This paper studies NER under the extremely weak supervision setting, where only one example per entity type is given in the \"The [Class LABEL]; <example>\" structure.\nThe proposed system (X-NER)  uses the one-shot examples to obtain high-quality pseudo labels from an unlabeled data using a pre-trained LM and train an NER model using the standard sequence tagging method.\n\nThe steps to generate pseudo-labels using a large unlabelled data and one example per entity type:\n    (1) It finds a candidate span in a sentence for the target entity type\n    (2) Swap the span with the example entity\n    (3) Measure the LM’s prediction distribution of the tokens in the context and computer the KL divergence between the contexts before and after the swap \n    (4) Rank all spans and select top-K entities to construct pseudo labels for the entity type\n\nThe main contribution is to use the KL divergence of the context words, instead of the cosine similarity between entity spans, to choose new entity mentions from unlabeled sentences to generate pseudo labeled sample. Their experiments and ablation studies show the effectiveness of this approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "This is a small subset of existing works. \n* Few-Shot Named Entity Recognition: An Empirical Baseline Study\n* CONTAINER: Few-Shot Named Entity Recognition via Contrastive Learning\n* Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning\n* Few-shot Named Entity Recognition with Self-describing Networks\n* SEE-Few: Seed, Expand and Entail for Few-shot Named Entity Recognition\n* A Little Annotation does a Lot of Good: A Study in Bootstrapping Low-resource Named Entity Recognizers"}}]

Author rebuttal:

Meta-review: Reviewers recognized the contribution of this paper to low resource NER settings, and the efficacy of the proposed data augmentation on NER benchmarks. The reviewers also suggested additional baseline comparisons and related work which provide important comparisons for the specific application, which were added during the rebuttal discussion. Reviewers also had several questions about the methodology, scalability, and more precise description of the approach which should be addressed in revision. This paper will likely be of moderate interest to the EMNLP community with contributions to few-shot learning and data augmentation strategies.
Decision: Accept-Findings