Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning
Authors: Minghao Tang
Yongquan He
Yongxiu Xu
Hongbo Xu
Wenyuan Zhang
Yang Lin
Venue: EMNLP 2023 Findings (2023)
Keywords: fine-grained entity typing
natural language processing
noisy labels
co-prediction prompt tuning
large language model
TL;DR: This paper present a simple and effective approach to correct noisy labels in Fine-grained entity typing (FET) by using Co-Prediction Prompt Tuning.

Abstract: Fine-grained entity typing (FET) is an essential task in natural language processing that aims to assign semantic types to entities in text. However, FET poses a major challenge known as the noise labeling problem, whereby current methods rely on estimating noise distribution to identify noisy labels but are confused by diverse noise distribution deviation. To address this limitation, we introduce Co-Prediction Prompt Tuning for noise correction in FET, which leverages multiple prediction results to identify and correct noisy labels. Specifically, we integrate prediction results to recall labeled labels and utilize a differentiated margin to identify inaccurate labels.  Moreover, we design an optimization objective concerning divergent co-predictions during fine-tuning, ensuring that the model captures sufficient information and maintains robustness in noise identification. Experimental results on three widely-used FET datasets demonstrate that our noise correction approach significantly enhances the quality of various types of training samples, including those annotated using distant supervision, ChatGPT, and crowdsourcing.

Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning
Minghao T Yongquan He Yongxiu Xu 12; Hongbo Xu', Wenyuan Zhang" and Yang Lin
Institute of Information Engineering, CAS, China
School of Cyber Security, UCAS, China MMettunn, China
[tangminghao, xuyongxiu, iuyurgxiu,hbxuueiiee. hww.)/iie......oooooo...iii heyongquanêmei tuan. com
Abstract
Misnoy Corporation Hong Kong Chinese Disney park.
Fine-grained entity typing (FET) is an essential task in natural language processing that aims to assign semantic types to entities text. How- ever, FET poses major challenge known as the noise labeling problem, whereby current meth- ods rely on estimating noise distribution to iden- tify noisy labels but are confused by diverse noise distribution deviation. To address this limitation, we introduce Co-Prediction Prompt Tuning for noise correction in FET, which lever- ages multiple prediction results to identify and correct noisy labels. Specifically, we integrate prediction results to recall labeled labels and utilize a differentiated margin to identify in- accurate labels. Morcover, we design op- timization objective concerning divergent co- predictions during fne-ttuning ensuring that the model captures sufficient information and maintains robustness in noise identification. Ex- perimental results on three widely-used FET datasets demonstrate that our noise correction approach significantly enhances the quality of various types of training samples, including those annotated using distant supervision, Chat- GPT, and crowdsourcing.
organization, company, person, artist, author
Knowledge
Base
The Weihhl agricaltural introduced potato variety, cultivation and breeding, has provided 50,000 kgef [potatoes].
tivin
CharGPT
Figure I: Examples from OntoNotes dataset, with inac- curate labels in red and unlabeled labels in gray. (a) A distantly labeled example. (b) A weakly labeled exam- ple by ChatGPT.
may exhibit superficial resemblances but possess distinct underlying meanings. For example, the term "apple" could pertain to a technology corpo- ration or a fruit. Achieving high performance in FET typically requires considerable amount of annotated data, resource that is both costly and tine-consuming to acquire. To tackle this issue, re- searchers have explored various approaches for au- tomatically labeling training samples. One particu- larly popular method is distant supervision (Ling and Weld, 2012; Gillick et al., 2014), which as- sumes that any text mentions an entity in existing knowledge bases (e.g.. Freebase (Bollacker et al., 2008)) is related the corresponding entity types. However, this technique can be susceptible to noise and ambiguity (refer to Figure la), since it fails consider the contextual information surrounding the entities.
1 Introduction
Fine-grained entity typing (FET) is fundamen- tal task in the field of natural language processing (NLP). involves assigning specific types to en- tity mentions based on the contextual information surrounding them. The results of FET can be lever- aged to enhance various downstream NLP tasks, in- cluding entity linking (Raiman and Raiman, 2018; Chen et al., 2020a), relation extraction (Shang al., 2020, 2022), question answering (Wei et al., 2016) and other tasks (Jiang et al., 2020; Liu et al., a02
Meanwhile, the large language models (LLMs), such as ChatGPT1, has drawn significant interest among researchers in exploring their applications in text annotation tasks. Recent studies (Gilardi et al., 2023; Törnberg, 2023) indicate that zero- shot ChatGPT classifications outperform crowd- workers. However, despite their remarkable suc-
FET poses a formidable challenge that entails not only the classification of entities across diverse domains but also the distinction of entities that
Yongxiu Xu is the corresponding author
hhtp:://opemai.ccmmbbboohhagggg

---

cess in other domains, the weakly labels ChatGPT generated for FET still pose challenges. As shown Figure lb, it can generate accurate weakly labels, but its coverage fine-grained types may be lim- ited. This difficulty may arise from the challenge of designing prompt that can accurately handle fine-grained entity types across multiple domains simultancously. However, this incomplete recogni- tion also results in noisy labels, as it mislabels the correct type as a negative sample.
We conduct experiments on three public FET datasets: OntoNotes and WiKi have distantly an- notated training data; Ultra-Fine has crowdsourced training data. The experimental results show that the performance of baseline model is significantly improves after training on the corrected data, in- dicating the effectiveness of our noise correction method. Additionally, relabel subset of training data from OntoNotes and Wiki and then optimized this data using our method. The experimental results demonstrate that our method can further improve the quality of the data annotated by ChatGPT.
utilized ChatGPT to
Training deep models on data with noisy labels can result in the learning of incorrect patterns and subsequent incorrect predictions. As a result, sev- eral approaches have been proposed to tackle the issue of noisy labels in FET, including designing robust loss functions (Ren et al., 2016; Xu and Barbosa, 2018), estimating noise transition matri- ces (Wu al., 2021; Pang et al., 2022), and correct- ing noisy labels (Onoe and Durrett, 2019; Zhang al., 2021; Pan et al., 2022). Among these ap- proaches, correcting noisy labels is often consid- ered the most desirable, as it leads to more accurate and informative labels. However, the methods for noise correction often rely on estimating the distri- bution of noise, which can be challenging due to the diverse deviations arising from different labeling methods. For instance, distant supervision tends to generate inaccurate labels, while labels generated by ChatGPT are accurate but incomplete.
2 Related Work
2.1 Automatic Labeling Techniques for FET
Due to the scarcity of manually labeled data, auto- matic labeling techniques have gained significant attention in FET. Some studies (Ling and Weld, 2012; Gillick et al., 2014) utilize distantly super- vised learning to generate labeled training samples by mapping entity mentions to entities from pub- lic knowledge bases and selecting reliable map- ping types labels. However, the labels obtained through this method are independent of sentence semantics. On the other hand, some studies (Choi et al., 2018; Dai et al., 2021) use head supervision to label training samples, where head words are used to label entities. However, both distant su- pervision and head supervision have limitations in producing accurate labels.
In this paper, we propose a novel noise cor- rection method for FET using the co-prediction prompt-tuning technique, which leverages multiple prediction results to identify and correct noisy la- bels. Specifically, we first fine-tunes a pre-trained masked language model (PLM) on data with noisy labels using co-prediction prompt. This prompt contains two mask tokens, which focus on differ- ent predictive capabilities and produce multiple predictions for each entity mention. Note that the presence of noisy labels may make it difficult for the two masks to arrive at a consensus on the out- puts. In this scenario, one mask may start fitting the noise before another, leading to divergent co- predictions. Hence, we design an optimization objective concerning divergent co-p co-predictions dur- ing fine-tuning to maintain the fitting difference of masks on the noise labels and robustness to noise identification. Afterward, we integrate the multi- ple prediction results recall unlabeled labels and use a differentiated margin to identify inaccurate labels.
Recently, large language models such Chat- GPT have shown promising results in weakly su- pervised learning, even surpassing crowdsoureed annotators in some domains (Gilardi et al., 2023; Törnberg, 2023). However, due to the large num- ber of entity types, designing effective prompts for ChatGPT to generate comprehensive label sets can be challenging in FET. In this paper, our goal is to develop a noise correction method that enhances the quality of weakly labeled data, instead of de- signing perfect prompts for ChatGPT or rules for distantly supervised learning.
2.2 Noise Learning Method for FET
The problem of noisy labeling presents a significant challenge in FET research. Several research studies have explored label dependencies. Ren al. (2016) proposes a hierarchical partial-label loss to handle noisy labels. Wu et al. (2019) leverages a random walk process on hierarchical labels to weight out

---

noisy labels. Wu et al. (2021) learns the noise confusion matrix by modeling the hicrarchical label structure. However, these approaches require a predefined hierarchical label structure, which can be difficult to establish in practice.
3.2 Co-Prediction Prompt Tuning
The accurate identification of inaccurate or unla- beled labels is a crucial challenge in noise cor- rection, given that manual identification of such labels is time-consuming and expensive. Recent studies (Han et al., 2018; Nguyen et al., 2020) have highlighted the "memory effect" (Arpit al., 2017), which suggests that deep models tend memorize clean labels before fitting to noisy la- bels when trained on data with noise. Building on this insight, we propose leveraging a co-prediction prompt-tuning technique for noisy label detection that explicitly captures the fitting difference.
Some studies assume that a subset of clean la- bels is available. Onoe and Durrett (2019) trains noise filter and relabel modules on the clean data. Pang et al. (2022) treats the clean labels as anchors to estimate noise transition matrices. However, the efficacy of these methods is heavily reliant on the size of the clean labels, which may not always be guaranteed. Hence, some approaches try to esti- mate noise distribution to produce pseudo-truth labels (Zhang et al., 2021) or filter out noisy la- bels (Pan et al., 2022). Nonetheless, umifying the noise distribution from different automatic label- ing methods can be challenging. Differ from them, our approach employs co-prediction prompt tuning for noise correction, thus avoiding the pitfalls of estimating noise distribution for various types of weakly labels.
3.2.1 Model Structure
There are two primary components in the prompt- based model: prompt and verbalizer. The prompt provides task-related information to guide the pre- trained language model generating relevant and accurate output. The verbalizer converts the output of the model into natural language.
Co-Prediction Prompt Differ from traditional prompts, we adopt co-prediction prompt for FET that contains two different mask tokens: [PMASK] and [NMASK]:
3 Methodology
In this section, we introduce the problem formu- lation and describe our noise correction method. As shown in Figure 2, our method mainly consists of two main steps: 1) Applying co-prediction prompt fine-tune PLMs with weakly labeled cor- pora; 2) Correcting noisy labels by analyzing the co-prediction results this model.
T(c, m) = {c} [P] {m} belongs to
(1)
[PMASK] rather than [NMASK],
where [P] represents set of soft words that are randomly initialized trainable special tokens, and [PMASK] and [NMASK] are initialized by the em- bedding of original mask token [MASK]. Clearly, artificially words (e.g., "belongs to", "rather than") are used guide the two mask tokens focus on different predictive abilities. The objective is to extract diverse knowledge from PLMs and create varying difficulty in fine-tuning the representations of the two mask tokens.
3.1 Problem Formulation
The FET task aims to assign appropriate semantic types an entity mention based on contextual information provided by sentence xr The seman- tie types are subset of a pre-defined set of fine- grained entity types Y = {y1, 1/, ... }. If the training corpora is labeled by distant or weak su- pervision, two types of noisy labels may arise. The first type is inaccurate labeling, which occurs when annotator assigns incorrect semantic type to entity mention. The second type is unlabeled labeling, which happens when the annotator fails identify an appropriate semantic type.
Verbalizer Selection Following the previous prompt-based model (Ding et al., 2021), we uti- lize the soft verbalizer that stores a set of mappings from soft word v C V, to a label C Y. As fine- grained entity types typically have hierarchical type structure, such as "/organizzaiion/oomannyeeee use the average embedding of type tokens initialize soft words.
In our approach, the goal is improve the qual- ity of the training corpus by correcting noise labels, specifically by removing inaccurate labels and re- calling unlabeled labels.
To determine the probability distribution of en- tity mentions m with respect to the label set Y, the co- co- prediction model generates two distinct scores for each entity type E Y by using [PMASK] and

---

(a) Co-Prediction Prompt Tuning
(b) Noisy Label Correction
Training Samples
Pre-trained Masked Language Model (PLM)
Co-Prediction Model
[CIS] "Harrisburg International Airport" is an in Pennsylvania. (SEP)
[P
[Pa] Pennsylvania belongs to [PMASK] rather than [NMMASK]
[SEP]
Recalling Unlabeled Labels
MLM
Label Word
Cearnable geediaTTokan
Original Golden Labels
Head
Embeddings
Meak Tukaro
Early Stop
Org
PP
[PMASK] Label Scores
[NMASK] Label Scores
Loc: Location
09
[PMASK] Predicted
[NMASK]
Pro: Proviace
Gov: Government
00 08
0.2
Lahels
Prodicted
Labels
Org Organization
10
104
0.4
Art: Artint
0.2
Eliminating Inaccurate Labels
Enlarged Labels
Chaamed Tabsll
Diveegenn Co prodicctionss Recognition
о Gov
Lass
Consistent
Divergent
0 Org
B Loc
Gopendactio Diftrwace
Loc
Pro
Adjustment
Prediction Losses
Prediction Losses
0 Pro
Figure 2: Overview of our noise correction method, which first fine-tunes pre-trained masked language model via co-prediction prompt-tuning and then use this model for noise correction. "GT" denotes the ground-truth. "PP" and "PN" are prediction scores of [PMASK] and [NMASK].
[NMASK]. The prediction score for each label y can be calculated as follows:
model's learning from the labels that exhibit diver- gent co- co-predictions. Hence, we adjust the training loss function as follows:
Pp.ul (c, m) P[[PMASK] T(c, m)),
(2)
L c) - E
Eyy.okk Yo-h
(5)
Pag (c, m) ><[NMASK] T(C m)),
Y4
where Y2 denotes the labels with divergent co- predictions, le is the number of these labels; Y;- denotes the labels with consistent co-pre co-prediction, the total number labels; by is the co- co-prediction loss for each label y C Y: y is a hyper-parameter that represents the loss weight.
where Ppy and Prayy are the prediction scores of [PMASK] and [NMASK] for label y,.
3.2.2 Training with Noise
Drawing on insights from "memory effect" (Arpit et al., 2017), it is possible for one mask to fit noisy labels more quickly than another due to differences in their fine-tuning speeds. This fitting discrepancy can result in divergent predictions on noisy labels. Figure 2a provides an example in which the pre- dictions generated by [PMASK] and [NMASK] for entity types "Gov" and "Org" are inconsistent. In accordance with the semantics of the co-prediction prompt, we define divergent co-predictions that satisfy one of the following criteria:
As [PMASK] and [NMASK] focus opposite learning abilitics, the co-prediction loss Ly is cal- culated follows:
Ly BCB( (m, c), Py) + BCE((Png) (m, c), py),
(6)
where by {0, 1} denotes the ground-truth for whether the entity mention m should be classified as label y given the context c, and BCE(-) denotes the binary cross-entropy loss function.
{ Pp. 2 0.5 and Pn, 2 0.5
During the initial training stage, it is crucial to recognize that labels with divergent co-predictions may include large number of clean labels. There- fore, we start the model training with y 1. As the fine-tuning process progresses, we gradually de- crease 7 until reaches the marginal value. Once the co-prediction model achieves the peak general- ization performance, we terminate the model train- ing process. We evaluate the effectiveness of this
(4)
Ppy < 0.5 and Pay < 0.5
Optimization Loss Function As the fine-tuning process progresses, both [PMASK] and [NMASK] will eventually fit all noisy labels, leading a de- crease of divergent co-pre co-predictions. To maintain the ability co- preddiiiition model to generate divergent results on noisy labels, we propose constraining the

---

training strategy through empirical experiments in Section 5.3.
Datasets
Wiki
OntoNotes
Ultra-Fine
hierarchy depth entity types mentions-tran
2
1
3.3 Noisy Label Correction
112
89
2519
2009898
253241
2000
this section, we will detail how to correct noisy labels in the training data by recalling unlabeled labels and eliminating inaccurate labels.
mentions-dev
2202
2000
mentions-test
563
8963
2000
Table 1: Statistics of datasets.
Recalling Unlabeled Labels The co- co-prediction model utilizes two mask strategies, [PMASK] and [NMASK], to extract knowledge from PLMs. By combining the predictions generated by both strate- gies, we can produce more comprehensive set of predicted labels, some of which might unlabeled in the training data. As shown in Figure 2b, the predicted label "Pro" is not included in the origi- nal golden labels, which suggests that it may be a potentially unlabeled label.
Spotlight (Gillick et al., 2014), with 253K training samples and 89 entity types. WiKi (Ling and Weld, 2012) used sentences from Wikipedia articles and news reports, and was distantly annotated using Freehase (Bollacker et al., 2008), with 2M training samples and 112 entity types. Ultra-Fine Choi et al. (2018) collected 6K samples through crowd- sourcing, which contains 2,519 entity types. Statis- tics datasets are in Table
Eliminating Inaccurate Labels After obtaining a more comprehensive label set by recalling unla- beled labels, the next step to optimize this set by removing inaccurate labels. As the co-prediction model retains the ability to generate divergent pre- dictions on noisy labels, we can identify inaccurate labels by measuring the absolute difference in the co- co-prediction scores. Specifically, we can calculate the divergence score dy for each label y as follows:
Moreover, we employ ChatGPT to relabel 3k and 6k training samples from OntoNotes and WiKi, respectively. For Ultra-Fine, it is challenging design a comprehensive prompt for data relabeling with a huge number of entity types. More details are in Appendix A.
4.2 Baseline Methods
We first consider previous noise relabeling or cor- rection methods for FET, including LDET (Onoe and Durrett, 2019), NFETC-AR (Zhang et al., 2021), and ANLC (Pan al., 2022). Meanwhile, we also compare our method with some noise learning methods for FET, including AFET (Ren et al., 2016), NFETC (Xu and Barbosa, 2018), and FCLC (Pang et al., 2022). Finally, we compare our method with other competitive FET systems, such as UFET (Choi et al., 2018), ML-L2R (Chen et al., 2020b), MLMET (Dai et al., 2021), LRN (Liu et al., 2021b), UNIST (Huang et al., 2022), and PKL (Li et al., 2023).
8y Pp. (1 pn.y)
We can then classify each label y as either a clean or inaccurate label by comparing its divergence score by margin threshold e. If ō, e, classify the label y as inaccurate and remove it from the label set.
Different weak supervision methods may result in different levels of label noise, so important to choose an appropriate margin threshold for each method. For example, distant supervision often results in inaccurate labels, so it may be more appropriate use small to remove such noise. In contrast, ChatGPT may produce accurate but incomplete labels, in which case relatively larger e could be adopted to ensure that more potentially relevant labels are retained.
4.3 Experimental Details
In our co-prediction model, we choose the BERT- base (Devlin et al., 2019) model as backbone. Then, the parameters of BERT are optimized by Adam optimizer (Kingma and Ba, 2015) with the learning rate of 2e-6, 2 and 2e-5 on WiKi and OntoNotes, and Ultra-Fine, respectively. Other hyperparame- ters are in Appendix B.
4 Experimental Settings
4.1 Datasets
We evaluate our noise correction method on three publicly FET datasets. OntoNotes (Weischedel et al., 2013) used sentences from OntoNotes cor- pus, which was distantly annotated using DBpedia
After obtaining the corrected training sets of each dataset, we train basic prompt-based model using supervised learning to evaluate the effective-

---

Model
Ace
Results on Distantly Annotated Data Table 2 and 3 present the results on OntoNotes and WiKi with distantly annotated training set. Compared with previous SOTA methods for FET, the simple baseline (BERT-base + Prompt Tuning) achieves on-par or inferior performance when training on the original training set. Afterward, this baseline Macro F1
Micro
Model
F1
With Distantly Annotated Training Samples
With Crowdsourced Annotated Training Samples
2216))0166
53.3
71.2
64.6
UFET(2018)
47.1
24.2
32.0
(F
60.2+0.2
76.4+0.1
70.2+0.2
LDET(201))
51.5
33,0
40.2
ML-LL2((2000)
58.7
73.0
68.1
LRN(2021b)
54.5
38.9
45.4
LRN(2021b)
56.6
77.6
71.8
MLMET(2021)
53.6
45.3
49.1
NFETC-ARmee(2021)
64.0=0.3
78.8+0.3
73.0+10.3
UNIST(2022)
49.2
49.4
49.3
(202))))((022))
65,310.2
79.6:2..3
74.0:20.3
Den0
59.2::0.2
81,3110.3
75.3+00.
DenoiseFET(2022)
55.6+0..4
44.7:10.3
49.5+0.1
PKL((023)
52.7
49.2
50.9
Raseline
0.7
78.1+1.2
71,2+009
Baseline (corrected)
58.8+0.8
83.740.2
76,3+00.3
Baseline
57.0+0.7
41.2+003
48.0+0.2
w/ co-predict
57.020.5
83.120.1
75.2:00.
Baseline (corrected)
53.2.10.7
48.110.5
50.510.1
w/ co-predict
52.7.10.6
48.210.5
50.310.1
With ChatGPT Annotated Training Samples
Baseline
64.4+0.7
85,8+0.3
80.5:10.5
Table 4:
Performance Ultra-Fine set. denotes the previous noise relabeling or correction approach.
Baseline (corrected)
64.620.7
87.1-0.1
81.7=0.1
w/ predict
64.3::0.6
86.5
81.3:10.1
Table 2: Performance OntoNotes test set. denotes the previous noise relabeling or correction approach.
Results and Discussion
5.1 Main Results
Model
Ace
Macro F1
Micro Fl
With Distantly Annotated Training Samples
2FFT(20016)
53.3
69.3
66.4
(2018))
68.920.6
81.90.77
79.010.77
ML-L2R(20206)
69.1
82.6
80.8
AFETC-ARA..;202111
70.1+00.9
83.2+-0.7
80.1+00.6
FCLCne
71.3=111
82.2+0.7
81.1:0.66
D
67.520.3
82.310.6
78.5100.
Baseline
58.910.5
81.110.2
76.7:002
model achieves the best Macro Fl scores over all datasets when training on the corrected training set, outperforming previous SOTA methods by a magnitude from 1.1 to 2.4 percentages.
Baseline (corrected)
66.4+0.2
84.3+ 0.2
80.1+00.2
w/ co-predict
66.920.4
84.2+0.11
79.9=0.1
With ChatGPT Annotated Training Samples
Baseline
62.2=0.6
$0.8+0.1
77.5=0.33
Baseline (corrected) 65.8
82.610.4
79.1:10.4
Results on ChatGPT Annotated Data Tables 2 and 3 also present the results on the OntoNotes and Wiki using ChatGPT annotated training set. It is noteworthy that the baseline model achieves comparable superior performance with limited number of ChatGPT annotated training samples, compared to significantly larger number of dis- tantly annotated training samples (3k vs. 253k OntoNotes, and 9k vs. 2M in Wiki). Furthermore, we apply our noise correction approach on these training samples, resulting in significant perfor- mance improvements of the baseline model across all metrics. These results provide further evidence of the effectiveness of our approach in accurately identifying and correcting noisy labels.
co-predict 6..1
82.4+0.3
78.6+004
Table 3: Performance on WiKi test sets. denotes previous noise relabeling correction approach. means our re-implementation via their code.
ness of our approach², For this baseline, we finc- tune the BERT-base model with standard prompt (without "rather than [NMASK]").
4.4 Evaluation
Following prior works, we use the strict accu- racy (Acc), Macro FI, and Micro Fl scores for OntoNotes and WiKi. As for Ultra-Fine, we use the Macro precision (P), recall (R), and Fl scores. Moreover, we conduct our experiment five times and report the mean and standard deviation values.
Results Crowdsourced Annotated Data Ta- ble presents the results Ultra-Fine. The per- formance of the baseline model is significantly improved after training on the corrected training set, achieving comparable results with the previ- ous best method. This finding indicates that our
"Our code link:

---

noise correction method can enhance the quality of -amaan-nnnttttd training data, even in a challeng- ing dataset such as Ultra-Fine.
Dataset
Model
P
With Distantly Annotated Training Samples
Baseline
87.8=0.9
80.0=:066
83.7:10.2
Furthermore, compared with previous best noise correction method (DenoiseFET),, our approach has the advantage of being straightforward to imple- ment, which does not require counting the noise distribution of each type and training additional models for noise correction.
OntoNotes
w/o unl.
89.3=0.2
76.0+1.0
81.8+0.3
w/o ina
83.9=0.6
82.1-0.8
83.0:0.2
Baseline
82,9=0.1
86.110.2
84.3:10.2
WiKi
w/o unl.
83,5=0.1
84.220.3
83.8:0.1
w/o ina
70.7=0.4
90.9=0.4
79.540.2
With ChatGPT Annotated Training Samples
Baseline
88.8=0.9
85.4:10.9
87.1+0.11
5.2 Ablation Study
OntoNotes
w/o unl.
90.2=0.5
80.1
84.9:0.2
To evaluate the contribution of each component in our noise correction method, we conduct ablation studies on the three datasets.
w/o ina. 85,4=0.6
86.1:10.2
85.7+0..
Baseline
82.2=0.5
83.1
82.6:0.4
WiKi
w/o unl.
83.0=0.4
79.820.4
81.3+0.11
w/o ina.
80.5=0.2
83.3:0.2
81.8+0.1
As the co- ro-prediction prompt plays a crucial role for noise correction, we first assessed its impact on the model's performance. To be specific, the case "w/ co-predict" denotes a baseline model (BERT-base + Co-Prediction Prompt Tuning) are trained on the corrected training set. Performance of [PMASK] are reported in Table 2, and 4. How- ever, the results show a slight decline in perfor- mance when the co-prediction prompt are used, which suggests that it may not enhance the peak generalization performance of the model.
With Crowdsourced Annotated Training Samples
Baseline
53.2=0.7
48.1
50.5:1.2
Ultra-Fine
w/o unl.
59.5=1.2
41.0=0.7
48.5:0.3
w/o ina.
51.5=0.6
48.8:104
50.1+0.1
Table 5: Ablation studies on three different types of weakly label.
Next, we investigated the impact of each step in the noise correction process $ 3.3. As shown in Table 5, the case "w/o wnl." indicates that the noisy labels are corrected without recalling unlabeled la- bels. As we can see, the baseline model achieves higher precision but lower recall scores over the all datasets. These results demonstrate the effective- ness of this step in recalling unlabeled labels and can be applied to various common types of weak labels. The case "w/o ina." indicates that the noisy labels corrected without eliminating inaccurate labels. Similarly, shown in Tables 5, the base- line model achieves relatively lower precision but higher recall scores over all datasets.
Mang
200
2003
Troreg Betlhy
mg Stides
Figure Left:
The rate
of divergent
predictions on ChatGPT annotated training set in WiKi dataset. Right:
The performance curves of our co- prediction model in WiKi test set, and macro Fl scores of [PMASK] are reported.
losses for labels that demonstrate divergent co- predictions. This approach aims to help the model maintain its ability to predict divergent results on noisy labels. To assess the efficacy of this strat- egy, we present the evolution trend of divergent co-predictions during the training process.
Overall, these ablation studies demonstrate the comprehensive effectiveness of our method cor- recting noisy labels by combining the two essential steps: recalling unlabeled labels and eliminating in- accurate labeled labels. By utilizing the knowledge from PLMs, our method can effectively address the issue of insufficient coverage of fine-grained entity types in the training samples weakly annotated by the large language model ChatGPT.
Figure displays the evolution curve of diver- gent co-predictions generated on the WiKi training set (weakly labeled by ChatGPT). As the model training progresses, the number of divergent co- predictions decreases gradually. According to the memory effect theory, the reason may be that both [PMASK] and [NMASK] are gradually fitting more noisy labels. However, our loss adjustment strat- egy allows the model to generate more divergent co-predictions on the training data.
5.3 Detailed Analysis
Effect of Loss Adjustment As previously dis- cussed, we have developed a strategy to adjust

---

677
40.
06.
40.0
100
96.0
40.5
05.3
40.0
64.5
0.055
E41
0.5
01
05
10
0.22
0.0
5.25
0.1
05
10
Ray9g
10
Wwyeig yoa
(a) Original
Precaon
Precaas
Recal
Meod
40
MaDo
Naroo
0.05
0.3
544
63
055
05
5:55
0.1
02
6.3
05
0.
10
Waylngg
Figure 4: Performances Ultra-Fine and OntoNotes when using different loss weight 7 and threshold c.
(b) Corrected
Figure 5: T-SNE virtualization of [PMASK] representa- tions on OntoNotes test set: optimized by (a) original and corrected distantly annotated training The entity type "/other" is removed, well as the types with less than samples.
Training deep models noisy labels neg- atively impact their performance unseen data. Therefore, we report the model's performance on WiKi test set. As seen, our strategy leads to better peak generalization performance. This finding sug- gests that the labels with divergent co-predictions may indeed contain potentially noisy labels, and our strategy can mitigates their negative impact by restricting the model learning them.
To demonstrate the effectiveness of our noise correction method, we visualize [PMASK] representations optimized separately by the original and corrected training sets. We choose to display the fine-grained type since types have only a few samples. Upon comparing Figure 5(a) and (b), we observe more prominent margins and clearer decision boundaries between different types after noise correction. However, some clusters still contain confusing samples, We attribute this to the fact that many samples have multiple fine-graines entity types (such as "/organi- zation/ nation/goveerment" and "/lccation/coumtyy)) but only one can be reported. This also highlights the complexity of noise correction in FET, as it requires determining the correctness of each type in the bel set, rather than simply selecting one correct label.
Visualization
Sensitivity of Hy perparameters We explore the performance changes with respect different y (Eq.5) and (E...7)
Selecting appropriate y is crucial for the co- prediction model to effectively identify noisy la- bels. Since it is difficult directly evaluate the noise detection ability, selecting an appropriate y relies on the model's generalization performance. Figure 4(a, b) present the results on Ultra-Fine and OntoNotes. To achieve optimal performance, hyper-paramette tuning for is necessary when training the co-prediction model in real-world sce- narios.
The performance analysis of corrected training sets generated with different threshold 6 is pre- sented in Figure 4(c, d). The results clearly demon- strate the impact of varying values on the cor- rected training sets. A small precision score significantly, but it may also ad- versely affect the recall score. Therefore, choosing a reasonable e value requires striking balance be- tween precision and recall scores to ensure optimal model performance.
Conclusion
This paper presents a novel noise correction method for FET through the utilization of co-prediction prompt tuning. Our approach is not only straight- forward but also highly effective, leveraging pre- trained language model that has been fine-tuned with a co-prediction prompt. This fine-tuned model is capable identifying and rectifying noisy labels. To evaluate the performance of our method, we con-
improve the

---

ducted experiments on three publicly available FET datasets, each containing different types of training samples, including those annotated through distant supervision, ChatGPT, and crowdsoureing. The re- sults of our experiments demonstrate the versatility of our noise correction method, as it significantly enhances the quality of the training data in all three scenarios. Consequently, it leads to substantial improvement in the overall performance of the FET model.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collabo- ratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIG- MOD international conference on Management of data, pages 1247-1250.
Shuang Chen, Jinpeng Wang. Feng Jiang. and Chin-Yes Lin. 2020a. Improving entity linking by modeling latent entity type information. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7529-7537.
Tongfei Chen, Yunmo Chen, and Benjamin Van Durme, 2020b. Hierarchical entity typing via multi-level learning to rank. Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, ACL 2020, Online, July 5-10, 2020, pages 8465 8475, Association for Computational Linguis- tics.
Limitations
We have developed a co-prediction prompt specifi- cally for fine-tuning pre-trained masked language models. However, it is important to note that this is just one example of co-prediction prompt tuning, and there exist numerous possibilities for design- ing more engaging prompts. For instance, one can consider increasing the number of [MASK] tokens or designing prompts with more instructive words. These alternative approaches can be complemented by a novel process that utilizes the co- co-prediction results to further enhance the accuracy of noise cor- rection. In addition, our method does not rely on prior knowledge or information about entity types. such as predefined hierarchical structures or de- tailed type introductions. While this information is often not readily available, it can be useful for identifying noisy labels.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87-96, Melbourne, Australia, Association for Computational Linguistics.
Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021. Ultra-fine entity typing with weak supervision from masked language model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACI/IJCNLP 2021. ( (Volume 1: Long Papers). Virtual Event, Au- gust 1-6, 2021, pages 1790-1799. Association for Computational Linguistics.
Ethics Statement
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 4171-4186. Association for Compu- tational Linguistics.
To address ethical concerns, we provide the two detailed description: 1) All experiments were con- ducted on existing datasets derived from public scientific papers or technologies. 2) Our work does not contain any personally identifiable information and does not harm anyone.
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu. Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. 2021. Prompt-learning for fine-graine ane-ggri entity typing. arXiv preprint arXiv:2 10604.
Acknowledgements
This work was supported by Strategic Priority Re- scarch Program of Chinese Academy of Sciences (NO. XDC02040400).
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint arXiv:2303.15056
References
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context- dependent fine-grained entity type tagging. CoRR, abs/1412.11200.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017. A closer look at mem- orization in deep networks. In International confer- ence on machine learning, pages 233-242, PMLR.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural

---

networks with extremely noisy labels. Advances in neural information processing systems, 31.
Weiran Pan, Wei Wei. and Feida Zhu. 2022. Automatic noisy label correction for fine- fineggrained entity typing. In Proceedings of the Thirty-Firyy International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022. pages 4317-4323. ijcai.org.
James Y Huang, Bangzheng Li, Jiashu Xu, and Muhao Chen. 2022. Unified semantic typing with mean- ingful label inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computaaional Linguistics: Human Language l'ccnoloogess pages 2642-2654.
Kunyuan Pang, Haoyu Zhang. Jie Zhou, and Ting Wang. 2022. Divide and denoise: Learning from noisy la- bels in fine-grann fine-grained entity typing with cluster-wise loss correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Yolume 1: Long Papers), pages 1997-2006.
Haiyun Jiang, Yanghua Xiao, and Wei Wang. 2020, Ex- plaining bag of words with hierarchical conceptual labels. World Wide Web, 23(3): 1793-1333.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations. ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Jonathan Raiman and Olivier Raiman. 2018. Deeptype: multilingual entity linking by neural type system evo- lution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016. Afet: Automatic fine-grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1369- 1378.
Na Li, Zied Bouraoui, and Steven Schockaert. 2023. Ultra-inee entity typing with prior knowledge about labels: A simple clustering based strategy. arXiv preprint arXiv:2305. 12802.
Xiao Ling and Daniel Weld. 2012. Fine-grained entity recognition. In Proceedings of the Twenty- Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada. AAAI Press.
Yu- Ming Shang, Heyan Huang, Xin Sun, Wei Wei, and Xian-Ling Mao. 2022. Learning relation ties with a force-directed graph in distant supervised relation ex- traction. ACM Transactions on Information Systems (TOIS).
Lemao Liu, Haisong Zhang, Haiyun Jiang, Yangming Li. Enbo Zhao, Kun Xu, Linfeng Song, Suncong Zheng, Botong Zhou, Dick Zhu, et al. 2021a. Texs- mart: A system for enhanced natural language under- standing. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing: System Demonstrations, pages 1-10.
Yuming Shang. He- Yan Huang, Xian-Ling Mao, Xin Sun, and Wei Wei. 2020. Are noisy sentences use- less for distant supervised relation extraction? In Proceedings of the AAAI conference on artificial in- telligence, volume 34, pages 8799-8806.
Petter Tömberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588.
Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. b02 Fine-grained entity typing via label reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. pages 4611-4622. Association for Computa- tional Linguistics.
Wei Wei, Zhao Ming. Liqiang Nie, Guohui Li, Jian- jun Li, Feida Zhu, Tianfeng Shang, and Changyin Luo. 2016. Exploring heterogeneous features for query-focused summarization of categorized commu- nity answers. Information Sciences, 330:403 423..
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy. Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0 ldc2013119. Linguistic Data Consortium, Philadelphia. PA, 23.
Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi- Phuong-Nhos Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. 2020. SELF: leam- ing filter noisy labels with self-ensembling. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Junshuang Wu, Richong Zhang, Yongyi Mao, Hongyu Guo, and Jinpeng Huai 2019. Modeling noisy hierar- chical types in fine-grained entity typing: A content- based weighting approach. In IJCAI, pages 5264- $270.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-abelled data for entity typing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume (Long and Short Papers), pages 2407- 2417. Association for Computational Linguistics.
Junshuang Wu, Richong Zhang, Yongyi Mao, Ma- soumeh Soffaei Shahrbabak, and Jinpeng Huai, 2021. Hierarchical modeling of label dependency and label noise fine fine-grained entity typing. Proceedings of the Thirtieth International Joint Conference on

---

Artificial Intelligence, IJCAI-21, pages 3950-3956. International Joint Conferences on Artificial Intelli- gence Organization. Main Track.
Peng Xu and Denilson Barbosa. 2018. Neural fine- grained entity type classification with hicrarchy- aware loss. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. NAACL-HLI 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages 16-25, Association for Compu- tational Linguistics.
ChatGPT
Fine-graimed entity typing aims assign semantic types to entities in texts. The type set contains: /person, / location, For example:
location.forganizznn /organization, /other, /location/cityy. ....
Columbia Savings is major holder so called junk types of "Columbia Savings" contain: /organization, /organization/comee company.
Haoyu Zhang, Dingkun Long, Guangwei Xu, Muhua Zhu, Pengjun Xie, Fei Huang, and Ji Wang. 2021. Learning with noise: improving distantly-sup supeeviied fine-grained entity typing via automatic relabeling. In Proceedings of the Twenty-Ninth Ninh International Conference on International Joint Conferences on Artificial Intelligence, pages 3808-3815 15.
The
Question:
President
Taiwan joins
World Trade Organization-LRB- -ER---L-----R--EEEE The semantic
types of "President Chen" contain:
/person, http://soo//oooiiaaliiiuuu
A Automatic Labeling with ChatGPT
Figure example of using ChatGPT label entity mentions.
In order to reduce both the cost and time required for labeling, we choose to relabel only a subset of high-quality training samples OntoNotes and Wiki datasets.
Firstly, we apply a frequency-based filtering ap- proach to remove entity mentions from the training samples, specifically discarding those with fre- quency lower than 10 in OntoNotes and 20 in Wiki. Additionally, nonsensical non-entity mentions such as "yes" and "please" are also removed. Finally, we randomly extract 3,000 and 9,000 training samples for OntoNotes and Wiki datasets, respectively.
These samples are merged with an artificially designed prompt and then fed into GPT-3.5 text- davinci-Ofl model via its official API for relabel- ing. For parameters, ature 0.7. An example of the labeling process is shown in Figure
Parameters
OntoNotes
WiKi
Ultra-Fine
Batch size
16
16
16
top-p 1, with temper-
Learning Rate
3e-6
3e-6
2e-5
Adam epsilon
le-8
le-8
le-8
Warmup ratio
0.0
0.0
0.0
Emb. dropout
0.2
0.2
0.2
B Hyperparameters
Weight decay
0.01
0.01
0.01
We use a grid search to find the best hyperparame- ters depending development set performances. The hyperparameters used to fine-tuning BERT and correct noisy labels in three datasets are listed in Table 6.
Clipping grad
0.1
0.1
0.1
Loss weight y Threshold
0.1
0.1
0.005
0.2
0.05
0.8
ChatGPT
0.3
0.5
Table 6: Hyper-parameters in OntoNotes, WiKi and
Ulura-Fine datasets.
/htpps//openaa.com//ap//


Reviews:
[{"review_id": "0vmiUdXtWL", "reviewer": "Reviewer_FZzX", "summary": "", "questions": "A. In Table 7 of the MLMET paper [2], a vanilla BERT-base model is able to achieve 75.9 Micro-F1 on Ontonotes, which is significantly higher than the reported 71.2 in the paper. Is there any reason that might have caused the discrepency?\n\nB. Details of decreasing \\gamma during the training process are not mentioned.\n\nC. Some details in the “verbalizer selection” section are not clear (e.g. how do you compute the probability for \"/organization/company/news” which consists of multiple words and each word might consist of multiple tokens?)\n\nD. How is the “/other” category in Ontonotes handled during label correction?\n\n[2] Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model. ACL2021", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for providing your valuable comments. We would like to respond to each question as follows:\n\nQ1:About why a BERT-base model is capable of correcting labels annotated by humans (e.g., Ultra-fine dataset, [1] Ultra-fine entity typing. 2018 ACL).\n\n\nA1: The Ultra-fine dataset, specifically the Ultra-fine entity typing dataset from the 2018 ACL [1], consists of annotations provided by five crowd workers on Mechanical Turk. This dataset includes a total of 2519 entity types. Due to the limited domain expertise of the crowd annotators, accurately labeling all 2519 types can be challenging, often resulting in missing annotations. In our noise correction method, we start by  fine-tuning a pre-trained masked language model, such as BERT, using a co-prediction prompt. Throughout the training process, we devise an optimization objective that allows multiple prompts (e.g., [PMASK] and [NMASK]) to adapt to varying degrees of noise. And the co-prediction prompt is designed to guide the BERT to generate multiple prediction results that target diverse predictive abilities. This training approach enables the BERT model to identify and correct noisy labels based on the co-prediction results.  For Ultra-fine dataset, these different prediction results can be combined to recover unlabeled labels and address the issue of missing annotations.\n\n\nQ2: About the results on OntoNotes are lower than the recent method ([2] Ultra-fine entity typing with indirect supervision from natural language inference. 2022 TACL)\n\n\nA2: It is important to note that the OntoNotes dataset has two versions: the original version and the augmented version introduced by the paper [1] (Ultra-fine entity typing. 2018 ACL). The experimental results presented in paper [2] were conducted on the augmented version, while our experiments were exclusively conducted on the original dataset. To ensure a fair comparison, we did not take the method presented by paper [2] as our baseline. In next version, we will retest the method presented by paper [2] on the original OntoNotes dataset and present the comparative results in the paper.\n\n\nQ3:MLMET ([3] Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model. 2021 ACL) is used as a baseline for Ultra-fine but its results on Ontonotes are not reported (80.35). \n\n\nA3: Similarly, MLMET conducted experiments on the augmented version of the OntoNotes dataset, while we conducted experiments on the original dataset. So,we did not reported the results of MLMET on OntoNotes dataset.  Additionally, the performance drops can be attributed to the variations in training the vanilla BERT-base model on different versions of the OntoNotes dataset. This is the primary reason for the decrease in performance, rather than any other unidentified factors.\n\n\n\n\nQ4: About the details of decreasing γ during the training process.\n\n\nA4: The parameter \\gamma is gradually decreased during the training process until it reaches the marginal value \\delta. This process occurs in the first training epoch, and \\gamma decreases linearly.  For instance, if an epoch consists of 1000 batches and \\delta is set to 0.1, then \\gamma = 1 - 0.0009 * (number of batches).  We genuinely appreciate your meticulous, and we will add these details in the next version of our paper.\n\n\nQ5: About the details in the \"verbalizer selection\" section.\n\n\nA5:  As mentioned in our paper, we utilize the average embedding of type tokens to initialize soft words. For example, when decomposing \"/organization/company/news,\" we obtain three individual words: \"organization,\" \"company,\" and \"news.\" These word embeddings can be acquired from the embedding layer of the BERT model by using their corresponding IDs in the BERT tokenizer's vocabulary. By summing and averaging the embeddings of \"organization,\" \"company,\" and \"news,\" we derive the representation for the type \"/organization/company/news.\"\n\n\nQ6: About how the \"/other\" category in OntoNotes is handled during label correction.\n\n\nA6: In our method, we capture noisy patterns within the noisy dataset through co-prediction prompt tuning BERT. Subsequently, for each instance of entity classification, we propose a label correction process that involves recalling unlabeled labels and eliminating inaccurate ones. When it comes to the \"/other\" category, we do not treat it differently. If \"/other\" is a missing label for an instance of entity typing, it would be corrected through the process of recalling unlabeled labels. Conversely, if \"/other\" is an incorrectly assigned label, it may be rectified through the process of eliminating inaccurate labels.\n\n\nWe genuinely hope that our responses address the queries and contribute to a positive outcome. We highly appreciate your time, effort and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is overall well written and easy to follow. The high-level intuition of the method is clear. The method is novel and seems to be a general method that can be applied to other classification tasks.\n- The proposed approach achieve improvements on different types of annotated data (distantly/ChatGPT/human).\n- The authors experiment with three datasets and carry out detailed analysis.", "reasons_to_reject": "- While experimental results shows clear improvements in terms of end task performance (results of correction + finetuning), no evaluations (either by case study or quantative experiments) on the corrected labels themselves (results of correction only) are conducted. It is also unclear why a BERT-base model is able to correct labels annoated by human. Additionally, the training set and test set of Ultra-fine are sampled from the same distribution and the results in Table 4 indicates that altering the training set leads to improvement. This result is unintuitive and requires further explanation.\n- While the authors claim that the proposed approach outperform previous SOTA, the results on OntoNotes are siginificantly lower than some of the recent methods [1] (76.3 vs 81.4). MLMET is used as a baseline for Ultra-fine but its results on Ontonotes are not reported (80.35).\n\n[1] Ultra-fine entity typing with indirect supervision from natural language inference. TACL2022", "questions_to_authors": "A. In Table 7 of the MLMET paper [2], a vanilla BERT-base model is able to achieve 75.9 Micro-F1 on Ontonotes, which is significantly higher than the reported 71.2 in the paper. Is there any reason that might have caused the discrepency?\n\nB. Details of decreasing \\gamma during the training process are not mentioned.\n\nC. Some details in the “verbalizer selection” section are not clear (e.g. how do you compute the probability for \"/organization/company/news” which consists of multiple words and each word might consist of multiple tokens?)\n\nD. How is the “/other” category in Ontonotes handled during label correction?\n\n[2] Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model. ACL2021", "paper_topic_and_main_contributions": "The paper proposed a co-prediction prompt tuning approach to correct incorrect labels in entity typing data. The proposed approach achieves improvement over vanilla prompt tuning. Improvements are shown on three types of noisy training data, with the most significant improvement seen on distantly annotated data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xmEhhR8CBm", "reviewer": "Reviewer_yPJ4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for providing your valuable comments. We genuinely appreciate your feedback, and we would like to respond to each question as follows:\n\nQ1: About a SOTA backbone network can be adopted rather than the BERT-base, such as LLaMa or ChatGLM.\n\n\nA1: Our research focuses on addressing the issue of noisy labels in datasets obtained through automatic annotation techniques. In our noise correction method, we fine-tune BERT using co-prediction prompts to generate multiple prediction results. These prediction results are then utilized to identify and correct the noisy labels. By utilizing a SOTA backbone, such as LLaMa or ChatGLM, our method may potentially achieve more reliable prediction results, thereby improving the accuracy of noise label detection and correction. This, in turn, enhances the quality of datasets obtained through automatic annotation techniques. \n\n\nQ2：About the proposed \"co-predict\" model is similar to contrastive learning.\n\n\nA2：It is important to note that our proposed \"co-predict\" model may have inherent differences from \"contrastive learning.\" Contrastive learning aims to learn meaningful representations by maximizing similarity between similar instances and minimizing similarity between dissimilar instances. But, our \"co-prediction prompt tuning\" is designed to guide BERT in generating multiple prediction results that target diverse predictive abilities. Therefore, their motivations are different. Furthermore, in the field of fine-grained entity typing, there have been no prior works that specifically employ contrastive learning for noise correction. Therefore, it is challenging to find corresponding studies as baselines for comparison. We hope the reviewer understands and forgives this limitation.\n\n\nQ3：About modifying the prompt without the \"rather than\" part as an baseline.\n\n\nA3: During the noise correction process, the prediction results generated by co-prediction prompts form the core of recalling unlabeled labels and eliminating inaccurate labels. Therefore, we can not remove the \"rather than\" part as a baseline to achieve noise correction independently.\nIn evaluating of the quality of our method's corrected dataset, as mentioned in the \"Experimental Details\" section, we fine-tune vanilla BERT using the corrected dataset based on standard prompts (without \"rather than [NMASK]\"). Additionally, we have fine-tuned vanilla BERT on the corrected dataset by using our co-prediction prompts as a baseline.\n\n \nWe genuinely hope that our responses address the queries and contribute to a positive outcome. We highly appreciate your time, effort, and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed \"co-predict\" mechanism is interesting which includes both positive and negative label in one prompt.\n\n2. The proposed model outperforms some existing works on public data sets.\n", "reasons_to_reject": "1. Since the performance of the proposed model heavily rely on the capability of the pretrained language model, a SOTA backbone network can be adopted rather than the BERT-base. For example, the author can use LLaMa or ChatGLM as the backbone pre-train model.\n\n\n2. The proposed \"co-predict\" model is similar to contrastive learning, the author needs to include contrastive learning-based approaches as baselines in the experiment section.\n\n3. Since the major contribution of this paper is the \"co-predict\" mechanism, the authors need to evaluate the performance \"co-predict\" in detail. For example, the authors can modify the prompt without the \"rather than\" part as an baseline.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper targets the fine-grained entity typing task which classifies the mention of an entity given the context information. Different from typical classification approaches, this paper formulate entity typing as a masked token prediction task. Firstly, the authors compose a textual input which includes the context information, the mention of the entity and the typing information (masked tokens); Secondly, the authors use a pre-trained language model to predict the masked token; Thirdly, the predicted masked token as well as the probabilities are used to infer the entity type. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lc6MTVjDYY", "reviewer": "Reviewer_MVhU", "summary": "", "questions": "Pls refer to reason to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for providing your valuable comments. We genuinely appreciate your feedback, and we would like to respond to each point with utmost sincerity and kindness:\n\nQ1: About negative learning.\n\n\nA1: We appreciate the reviewer's point regarding negative learning. We will carefully consider the relevance of negative learning to our method and further improve the related work section accordingly.\n\n\nQ2：About the recalling procedure of the prediction is divergence for some unlabeled labels.\n\n\nA2: If the prediction is divergence of [PMASK] and [NMASK] for a certain unlabeled label, it is important to consider whether this label is correct. Therefore, as described in the section \"Eliminating Inaccurate Labels,\" we calculate the divergence score (Eq.7) and compare it with a threshold value, \\epsilon . If the divergence score is greater than \\epsilon , we consider this recalled label from the model's prediction to be potentially incorrect. If it is less than \\epsilon, we consider the predicted label to be correct. The choice of threshold value \\epsilon is flexible, and we discuss its sensitivity in the \"Sensitivity of Hyperparameters\" section.\n\nQ3：About the  ablation study of loss adjustment.\n\n\nA3: We will add additional experimental data on using different decreasing schedules of \\gamma.\n\n\nQ4: About the analysis of \\epsilon.\n\n\nA4: In Figure 4, we present the impact of different values of \\epsilon  and draw the following conclusion: A small \\epsilon can significantly enhance the precision score; however, it may have a negative impact on the recall score. We appreciate the thoroughness of the reviewer and, as a result, we intend to provide further analysis on the parameter \\epsilon in the \"Sensitivity of Hyperparameters\" section. \n\nQ5: About how to determine the parameter \\gamma to make sure terminating the training process at the appropriate time.\n\n\nA5: During the co-prediction prompt tuning process, the model stops training once it reaches early stopping criteria, which is typically determined by achieving the best generalization performance on the development set. The parameter \\gamma primarily affects the fitting differences of [PMASK] and [NMASK] to the noise labels. By adjusting the value of \\gamma, we can control the degree of alignment between the prediction results of [PMASK] and [NMASK]  on the noise labels, thereby influencing the model's ability to effectively correct the noise in the dataset.\n\n\n\nWe genuinely appreciate your feedback and suggestions, and we will incorporate them into our work to improve its quality. We highly appreciate your time, effort, and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and organized;\n2. The proposed method is simple but effective, and may be easy to implement in real world applications, which can recall the unlabeled labels as well as eliminating inaccurate labels without estimating noise distributions, which can be used in various noise scenarios\n3. This paper proposes a novel prompt-base noise correction model: (1) leverage the knowledge of PLM for noise detection and correction; (2) design two kinds of prompts with opposite objectives which provide comprehensive predictions, and highlights the effectiveness of using LLM (e.g., ChatGPT) to generate weakly labeled training data, which may be better than the tradition methods like crowdsourcing, it may be inspire how to take advantage of LLMs to handle the similar problem for NLP research community. \n", "reasons_to_reject": "1. The proposed co-prediction method is related to negative learning, suggest to add some references in this domain.\n2. The model reall the unlabeled labels if the prediction of [PMASK] and [NMASK] is consensus, I wonder how to handle if the prediction is divergence for some unlabeled labels in the recalling procedure.\n3. The experiments are incomplete, for example, (1) The ablation study of loss adjustment, the coefficient γ is gradually decreasing during the fine-tuning process, and thus I wonder whether the performance is sensitive to the decreasing schedule, and how to obtain a proper decreasing schedule in practice? (2) The proposed method is partially sensitive to the threshold, however the analysis of ϵ is missing; (3) How to determine the parameter δ to make sure terminating the training process at the appropriate time? (4) The threshold in Eq. (7) should be small if there are many inaccurate labels, otherwise it should be relatively large when having many unlabeled data, suggest to add more experiments for further analysis. \n", "questions_to_authors": "Pls refer to reason to reject.", "paper_topic_and_main_contributions": "This paper focuses on noise label correction for fine-grained entity typing (FET), and proposes a label-cleaning method based on co-prediction prompt to address the problem, and utilizes LLM (such as ChatGPT) to generate weakly labeled training data for FET. Extensive experiments demonstrate the effectiveness of the proposed method. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The proposed co-prediction method is related to negative learning, suggest to add some references in this domain.", "Justification_for_Ethical_Concerns": "N.A."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. There are some type errors, for example:\n(1) Line 021, “demonstrates”  “demonstrate”\n(2) Line 096, “fine-tunes”  “fine-tune”\n(3) Line 118, “improves”  “improved”\n(4) Line 159, “proposes”  “propose”\n(5) Line 160, “leverages”  “leverage”\n(6) Line 162, “learns”  “learn”\n(7) Line 168, “trains”  “train”\n(8) Line 170, “treats”  “treat”\n(9) Line 213, “labels”  “a label”\n\n2. Eq (1): the meaning of c and m are needed\n3. The model conducts a pipeline which first-recalls and then-eliminates, and treats unlabeled label and inaccurate labels differently. However, when considering the results of both [PMASK] and [NMASK] together, these two types of noise are symmetric and may can be processed uniformly.\n4. It can be observed that the utility of PLM’s knowledge largely determines the performance of the model. It is better to consider how to better extract their knowledge, for example, the design of prompts.\n\n"}}]

Author rebuttal:

Meta-review: The paper proposed a co-prediction prompt tuning method for the entity typing task. The method could correct the noisy labels through masked token prediction. The experimental results on three datasets showed that the proposed model mostly outperformed existing work.

Strength: The proposed "co-predict" mechanism is interesting and somewhat novel. The whole paper is well-organized and clear.

Weakness: 1) Actually, BERT is also based on the masked token prediction. So it is not very clear whether BERT can actually correct human-annotated labels, which should be investigated further. 2) co-prediction prompt is similar to contrastive learning. The authors should explain their differences through illustration or experimental comparisons. 3) More experimental analyses/discussions are needed.
Decision: Accept-Findings