SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition
Year: 2023

Abstract:
Distantly-Supervised Named Entity Recognition effectively alleviates the burden of timeconsuming and expensive annotation in the supervised setting. But the context-free matching process and the limited coverage of knowledge bases introduce inaccurate and incomplete annotation noise respectively. Previous studies either considered only incomplete annotation noise or indiscriminately handle two types of noise with the same strategy. In this paper, we argue that the different causes of two types of noise bring up the requirement of different strategies in model architecture. Therefore, we propose the SANTA to handle these two types of noise separately with (1) Memory-smoothed Focal Loss and Entity-aware KNN to relieve the entity ambiguity problem caused by inaccurate annotation, and (2) Boundary Mixup to alleviate decision boundary shifting problem caused by incomplete annotation and a noisetolerant loss to improve the robustness. Benefiting from our separate tailored strategies, we confirm in the experiment that the two types of noise are well mitigated. SANTA also achieves a new state-of-the-art on five public datasets.

heading: 1 Introduction
text: As a fundamental task in NLP, Named Entity Recognition (NER) aims to locate and classify named entities in text, which plays an important role in many tasks such as knowledge graph construction (Peng et al., 2022; Li et al., 2022c) and relation extraction (Zeng et al., 2021; Wang et al., 2022). To alleviate the burden of annotation in the supervised setting, Distantly-Supervised Named Entity Recognition (DS-NER) is widely used in real-world scenarios. It can automatically generate labeled training data by matching entities in existing knowledge bases with snippets in plain
*Equal contribution. †Corresponding author.
text. However, DS-NER suffers from two inherent issues which introduce many noisy samples: (1) inaccurate annotation: the entity with multiple types in the knowledge bases may be labeled as an inaccurate type in the text, due to the context-free matching process, and (2) incomplete annotation: the knowledge bases with limited coverage of entities cannot label all entities in the text. As shown in Figure 1, the entity types of “Amazon" and “Washington" are wrongly labeled owing to context-free matching, and “410 Terry Ave N" is not recognized due to the limited coverage of knowledge bases.
Due to the sensitivity to the noise, the original supervised methods achieve poor performances in DS-NER. Therefore, many works have been proposed to handle the issue. Some works attempted to focus on solving incomplete annotation noise in DS-NER, including positive-unlabeled (PU) learning (Peng et al., 2019; Zhou et al., 2022b), negative sampling (Li et al., 2021, 2022a), and retrieval augmented inference with contrastive learning (Si et al., 2022). However, the ignorance of inaccurate annotation noise limits the model to further improve the performance. Recently, Zhang et al. (2021a) jointly trained two teacher-student networks to handle the two types of noise with the same strategy. Meng et al. (2021) adopted a noise-robust learning scheme and self-training for the whole training set to avoid overfitting in noise. However, by handling both types of noise with the same strategy, these
3883
methods failed to address the unique characteristics of different type of noise, thereby limiting their ability to effectively handle the noise in DS-NER.
As the causes of the two types of noise are different, both of two noise may lead to different problems in DS-NER task. Inaccurate annotation noise in training data can lead to serious entity ambiguity problem in the DS-NER. As exampled in Figure 1, when a model is trained on data where “Washington" is consistently labeled as “PER" (person) due to context-free matching, the model may continue to predict “Washington" as a "PER" even in contexts where it should be labeled as a “LOC" (location), such as when it refers to the city of Washington. Incomplete annotation noise can lead to the decision boundary shifting problem (Si et al., 2022) in the DS-NER task. This problem occurs when the model is trained on data where some entity spans are not labeled, causing the model to shift its decision boundary, making it more likely to predict an entity span as a non-entity type. Therefore, the noise in the spans labeled as entities by distant supervision leads to the ambiguity problem, and the noise in the spans labeled as non-entities leads decision boundary shifting problem. To further improve the performance in DS-NER, we argue that the two types of noise should be handled separately with specialized designs in model architecture. This can help the model to address the specific problems posed by each type of noise and lead to the better overall performance of the model.
In this paper, we propose the Separate strAtegies for iNaccurate and incompleTe Annotation noise in DS-NER (SANTA). Unlike previous works in DS-NER, we introduce different strategies to handle the two types of noise respectively. For inaccurate annotation, we propose Memory-smoothed Focal Loss (MFL) and Entity-aware KNN. These strategies aim to address the entity ambiguity problem posed by inaccurate annotation noise. For incomplete annotation, we propose Boundary Mixup to handle decision boundary shifting problem caused by incomplete annotation, which generates augmented instances by combining the instances around the boundary and the entity instances. Due to further training on the augmented instances, the biased decision boundary can be pushed towards right (fully supervised) side as the augmented instances exist between the biased boundary and the right boundary. Meanwhile, we empirically analyze the characteristics of incomplete annotation
noise, then adopt a noise-tolerant loss to further improve the model’s robustness to this type of noise.
Experiments show SANTA achieves state-of-theart on five public DS-NER datasets. Further analysis shows the effectiveness of each designed module and separate handling.

heading: 2 Related Work
text: To address data scarcity problem, several studies attempted to annotate datasets via distant supervision. Using external knowledge bases can easily get training data through string matching, but introduces two issues: inaccurate annotation noise and incomplete annotation noise. To address these issues, various methods have been proposed.
Only Focusing on Incomplete Annotation. Several studies (Shang et al., 2018; Yang et al., 2018; Jie et al., 2019) modified the standard CRF to get better performance under the noise, e.g., Partial CRF. LRNT (Cao et al., 2019) leaved training data unexplored fully to reduce the negative effect of noisy labels. AdaPU (Peng et al., 2019) employed PU learning to obtain unbiased estimation of the loss value. Furthermore, Conf-MPU (Zhou et al., 2022b) used multi-class PU learning to further improve the performance. Li et al. (2021) performed uniform negative sampling to mitigate the misguidance from unlabeled entities. Li et al. (2022a) then proposed a weighted sampling distribution to introduce direction to incomplete annotation when negative sampling. Si et al. (2022) adopt supervised contrastive-learning loss and retrieval-augmented inference to mitigate the decision boundary shifting problem. However, these studies only addressed incomplete annotation noise, ignoring inaccurate annotation noise, which also exists in DS-NER.
Handling Two Types of Noise With the Same Strategy. To further exploring the information in DS-NER text, many studies attempted to consider both inaccurate and incomplete annotation noise. BOND (Liang et al., 2020) designed a teacher-student network to drop unreliable labels and use pseudo labels to get more robust model. SCDL (Zhang et al., 2021b) further improved the performance by jointly training two teacher-student network and refining the distant labels. RoSTER (Meng et al., 2021) adopt a noise-robust learning scheme and self-training to improve the robustness. CReDEL (Ying et al., 2022) trained an automatic distant label refinement model via contrastive learn-
ing as a plug-in module for other DS-NER models. Although these works jointly considered the two types of noise, they did not take into account the difference of the two types of noise. Due to the different causes and frequencies of the two different types of noise, we argue that they should be handled separately with specialized strategies, which have not been considered in previous work.

heading: 3 Method
text: As shown in Figure 2, SANTA separately handles the spans labeled as entities and labeled as non-entities. SANTA uses the MFL and Entityaware KNN to address the entity ambiguity problem caused by inaccurate annotation. SANTA adopts noise-tolerant GCE + SR loss and Boundary Mixup to handle the incomplete annotation.

heading: 3.1 Span-based NER Model
text: We follow the same span-based NER model as Li et al. (2022b, 2020) and Si et al. (2022). For sentence [x1, x2, ..., xn], we use a pre-trained language model as an encoder to get the representations for every token x in the sentence:
[h1,h2, ...,hn] = Encoder([x1, x2, ..., xn]) (1)
where hi is the representation for token xi. For each span si,j ranging from i-th token to j-th token, the span representation si,j is calculated as:
si,j = hi ⊕ hj ⊕ (hi − hj)⊕ (hi ⊙ hj) (2)
where ⊕ is the concatenation operation and ⊙ is the element-wise product operation.
Then, we use linear layer and activation function to get a more dense representation:
ri,j = tanh(W si,j) (3)
Finally, we can obtain the entity label distribution oi,j for every span si,j as:
oi,j = softmax(V ri,j) (4)
where W and V are trainable parameter.

heading: 3.2 Against Inaccurate Annotation Noise
text: We observe that the inaccurate annotation in spans labeled as entities leads to two problems: (1) it can cause fluctuations in the training process and make it difficult to achieve consistent predictions; (2) it can lead to the entity ambiguity problem.

heading: 3.2.1 Memory-smoothed Focal Loss
text: In the training process, the inaccurately labeled entities and the similar true entities supervise the model back and forth and cause fluctuations in the model’s learning. For example, the “Washington" in Figure 1 is inaccurately labeled as “PER" (person), the model trained with it tends to predict “Washington" as “PER" instead of “LOC" (location). However, if the model is also exposed to similar and correctly labeled entities, such as “Seattle" labeled as “LOC", the model may also learn to generalize "Washington" as a "LOC". This back and forth supervision can make the model’s performance being less consistent and less accurate.
Intuitively, if we can smooth out such fluctuations, the performance of model can be further improved.
This observation motivates us to propose Memory Label Smoothing (MLS) to reduce the fluctuation. MLS memorizes model predictions to continuously update the soft labels during the training stage. The memorized soft labels are used to smooth the training of every sample, which tries to enable the model jointly consider the label information from previous predictions, so that learning for a particular sample does not fluctuate tremendously. Given each span xi,j labeled as entity, if its prediction is correct, the soft label corresponding to the entity type yi,j will be updated using the model output oi,j . Specifically, in training total T epochs, we construct Ŷ = [Ŷ 0, ..., Ŷ t, ..., Ŷ T ] as the memory soft labels at different training epochs. Ŷ t is a matrix with |L| rows and |L| columns, and each column in Ŷ t corresponds to the vector with latitude |L| as soft label for one category. L denotes the set of entity types. For epoch t > 0, Ŷ t is defined as:
Ŷ tyi,j ,l = 1
N
∑ l∈L I{l = yi,j}oi,j (5)
where N denotes the number of correctly predicted entities with label yi,j . I is indicator function. Ŷ 0 is initialize as identity matrix.
Then the updated soft labels will be utilized to supervise the model in the next epoch and smooth the learning curve of the model. In training epoch t, given a span (xi,j , yi,j), we use the previous G soft label [Ŷ t−G, ..., Ŷ t−1] to supervise the model:
Y tfinal = ∑
l∈L (λI{l = yi,j}+ (1− λ)
1
G
G∑
g=1
Ŷ t−gyi,j ,l)
(6)
where G and λ are hyperparameters. For t < G, we use all the previous soft labels to get Y tfinal.
Meanwhile, we adopt Focal Loss (Lin et al., 2020) to handle the entity ambiguity problem, which can be calculated as:
LFL = − ∑
l∈L α(1− oi,j)γ log(oi,j) (7)
where α and γ are hyperparameters. It can be seen that Focal Loss has a greater weight for ambiguous samples with low confidence.
Combined with the capability from MLS to reduce the fluctuation and the capability from Focal Loss to handle the ambiguous samples, the
Memory-smoothed Focal Loss (MFL) can be defined as:
LMFL = ∑
xi,j∈De
∑ l∈L α(Y tfinal)l(1− oi,j) γ log(oi,j) (8)
In addition, MFL is only performed on spans labeled as entities De to focus on the problems caused by inaccurate annotation.

heading: 3.2.2 Entity-aware KNN
text: The proposed MFL method may not completely solve the entity ambiguity caused by inaccurate annotation noise, and therefore relying solely on the output of the trained model may still result in limited performance. To address this issue, we propose Entity-aware KNN during the inference stage to facilitate the decoding process by retrieving similar labeled samples in the training set, which further improves the overall performance.
Entity-aware KNN consists of two parts, including Entity-aware Contrastive Learning (Entityaware CL) and KNN-augmented Inference. Specifically, we use Entity-aware CL to close the distance between the span labeled as entity with the same type, pull the distance of different types. Therefore, we could get the better representation to easily use KNN-augmented Inference to get a retrieved distribution oknn. Finally, we interpolate the output oi,j from model with oknn to further handle the entity ambiguity problem.
To improve the performance of KNN retrieving, Entitiy-aware CL pulls spans belonging to the same entity type together in representation space, while simultaneously pushing apart clusters of spans from different entity types. Therefore, the type of entities could be better distinguished. We use the cosine similarity as metric between the representations ri,j and rî,ĵ of span xi,j and xî,ĵ :
dsi,j ,sî,ĵ = ri,j · rî,ĵ |ri,j ||rî,ĵ |
(9)
Then the Entity-aware CL LCL is defined as:
LCL = − ∑
l∈Le
∑
ri,j∈El
1 Nl − 1 ∑
rî,ĵ∈El̄
F (ri,j , r̂i,ĵ)
(10)
where Le is the entity label set; Nl is the total number of spans with the same entity label l in the batch; El is the collection of all training spans with
l-th entity label. F (ri,j , rî,ĵ) is calculated as:
F (ri,j , rî,ĵ) = log exp(dri,j ,rî,ĵ/τ)∑
rm,n∈El̄ exp(dri,j ,rm,n/τ)
(11)
where τ is the temperature. El̄ is the collection of labeled entity spans not with entity label l.
After we get the easily distinguishable entity span representations, we propose KNN-augmented Inference to further relief the ambiguity problem by augmenting the trained model output. KNNaugmented Inference can be split into three parts:
(i) Firstly, we cache each entity representation rkey in training set and its label lvalue to construct a pair (key,value) ∈ DataStore.
(ii) We calculate the cosine similarity as Eq. 9 between the representation ri,j from span xi,j and each cached representation from DataStore.
(iii) Then, we select the top K most similar retrieved entities DK and then convert them into a one-hot distribution based on the KNN majority voting mechanism.
yknn = argmax l
∑
DK
I(lvalue = l),∀l ∈ L
oknn = onehot(yknn)
(12)
(iv) Finally, we interpolate the oi,j from model with oknn to get the final distribution ofinal:
ofinal = (1− µ) ∗ oi,j + µ ∗ oknn (13)
where µ is a hyperparameter to make a balance between two distributions.
In this way, we could use cached similar entities in the training set as memory to adjust the output of the trained model, therefore further mitigating the entity ambiguity problem.

heading: 3.3 Against Incomplete Annotation Noise
text: We observe that the incomplete annotation in spans labeled as non-entities leads to two problems: (1) the decision boundary shifting problem, and (2) the high asymmetric noise rate.

heading: 3.3.1 Boundary Mixup
text: When the model is trained on data where some entity spans are not labeled, the learned decision boundary can be biased and the model tends to predict an entity span as a non-entity type. As shown in Figure 3, the learned decision boundary tends to
shift from the fully supervised boundary (our desired boundary) towards the entity side. Therefore, we propose Boundary Mixup to push the decision boundary to the unbiased (desired) side.
The model always makes a wrong prediction to the instances around the learned decision boundary due to the decision boundary shifting. Motivated by this, if we can find the instances around the learned decision boundary, we can further find the location of the learned decision boundary. Then we can utilize Mixip (Zhang et al., 2018) to generate the augmented instances to modify the location of the learned decision boundary. Specifically, if the span xi,j is predicted as non-entity ln with a confidence lower than ε, it may actually be an instance around the learned decision boundary. Therefore, we randomly sample a entity e with the most possible entity label le according to oi,j . Then use Mixup between (re, le) and (ri,j , li,j) to generate an augmented instance (r̂, ŷ):
(r̂, ŷ) = (θ′ri,j + (1− θ′)re, θ′ln + (1− θ′)le) (14)
where θ′ is calculated as:
θ′ = max(θ, 1− θ) (15) θ ∼Beta(α′, α′), α′ ∈ (0,∞) (16)
where α′ is a hyperparameter. As shown in Figure 3, due to further training on the augmented instances, the biased decision boundary can be pushed towards the fully supervised side, mitigating the decision boundary shifting problem.

heading: 3.3.2 Noise-Tolerant Loss
text: To study the characteristics of incomplete annotation noise, we conduct an empirical analysis on BC5CDR and CoNLL2003. We use the knowledge bases provided by Zhou et al. (2022a) to relabel
the two training sets using string matching, then compare the matching results with original welllabeled training sets to get the rate of inaccurate and incomplete annotation for different entity types. As shown in Table 1, incomplete annotation noise accounts for the majority of the noise in DS-NER as the Incomplete Rate is much higher than Inaccurate Rate. Meanwhile, the incomplete annotation noise is asymmetric, which means the incomplete annotation noise is unbalanced in different entity types, because the knowledge bases always have different limited coverage of different entity types.
Recently, Meng et al. (2021) adopts generalized cross entropy (GCE) (Zhang and Sabuncu, 2018) instead of cross entropy (CE) to improve the robustness in DS-NER. GCE calculated on spans labeled as non-entities Dn can be described as following:
LGCE = ∑
xi,j∈Dn
1− oqi,j q
(17)
where q denotes a hyperparameter; when q → 1 GCE approximates Mean Absolute Error Loss (MAE), which is widely used in regression task; when q → 0, GCE approximates CE (using L’Hôpital’s rule).
Ghosh et al. (2017) theoretically proved that a loss function that satisfies the symmetric condition would be inherently tolerant to symmetric label noise. However, the derived loss functions according to this design principle such as MAE, suffer from the underfitting (Charoenphakdee et al., 2019). GCE attempts to perform the trade-off between CE and symmetric loss MAE, but is performs poorly on asymmetric noise as it attempts to satisfy symmetric condition (Ghosh et al., 2017). As the incomplete annotation noise is always asymmetric, we adopt Sparse Regularization (SR) to improve the GCE performance, which is proved to be more robust to asymmetric noise (Zhou et al., 2021). GCE + SR only calculated on Dn can be defined as:
LGCE+SR = ∑
xi,j∈Dn
1− oqi,j q + ∥(oi,j)∥pp (18)
where p ≤ 1 denotes a hyperparameter. In this way, we can achieve better robustness to asymmetric noise caused by incomplete annotation.

heading: 3.4 Training
text: We weighted the losses as follows:
LFinal = ηLMFL + (1− η)LCL + LGCE+SR (19)
η is hyperparameter to control the weight between LMFL and LCL only calculated on spans labeled as entities. As introduced before, LGCE+SR is only calculated on spans labeled as non-entities.

heading: 4 Experiment
text: Compared with extensive baselines, SANTA achieves significant improved performance in five datasets. We also conduct experiments and provide analyses to justify the effectiveness of SANTA.

heading: 4.1 Dataset
text: We conduct experiments on five benchmark datasets, including CoNLL2003 (Tjong Kim Sang and De Meulder, 2003), Webpage (Ratinov and Roth, 2009), OntoNotes5.0 (Weischedel et al., 2013), BC5CDR (Li et al., 2015) and EC (Yang et al., 2018). For CoNLL2003, OntoNotes5.0, Webpage, Liang et al. (2020) re-annotates the training set by distant supervision, and uses the original development and test set. We keep the same knowledge bases as Shang et al. (2018) in BC5CDR. For EC, Yang et al. (2018) uses the distant supervision to get training data, and labels development and test set by crowd-sourcing. Statistics of five datasets are shown in Table 2.

heading: 4.2 Evaluation Metrics and Baselines
text: To compare with baselines, we use Precision (P), Recall (R), and F1 score as the evaluation metrics. We compare SANTA with different groups of baseline methods, including supervised methods and distantly supervised methods. We also present the results of KB-Matching, which directly uses knowledge bases to annotate the test sets.
Supervised Methods We select BiLSTM-CRF (Ma and Hovy, 2016) and RoBERTa (Liu et al., 2019) as original supervised methods. As trained on noisy text, these methods achieve poor performance on DS-NER datasets.
Distantly-Supervised Methods We compare several DS-NER baselines, including: (1) methods that only consider the incomplete annotation: Conf-MPU (Zhou et al., 2022b) uses multi-class PU-learning loss to better estimate the loss. AutoNER (Shang et al., 2018) modifies the standard CRF to get better performance under the noise. LRNT (Cao et al., 2019) tries to reduce the negative effect of noisy labels, leaving training data unexplored fully. NegSampling (Li et al., 2021) and Weighted NegSampling (Li et al., 2022a) uses down-sampling in non-entities to relief the misleading from incomplete annotation. SCL-RAI (Si et al., 2022) uses span-based supervised contrastivelearning loss and designed inference method to improve the robustness against incomplete annotation. (2) methods that consider two types of noise equally: Co-teaching+ (Yu et al., 2019) and JoCoR (Wei et al., 2020) are two classical methods to handle noisy labels in computer vision area. BOND (Liang et al., 2020) and SCDL (Zhang et al., 2021b) proposes teacher-student network to reduce the noise from distant labels. RoSTER (Meng et al., 2021) adopts GCE loss, self-training and noisy label removal step to improve the robustness. CReDEL (Ying et al., 2022) proposes an automatic distant label refinement model via contrastive-learning as a plug-in module.

heading: 4.3 Experimental Settings
text: For a fair comparison, (1) we use BERT-base (Devlin et al., 2019) as the encoder the same as Si et al. (2022) and Zhang (2022) for CoNLL2003,
OntoNotes5.0, Webpage and EC; (2) for BC5CDR in the biomedical domain, we use BioBERT-base (Lee et al., 2020) the same as Zhang (2022). We use Adam (Kingma and Ba, 2015) as our optimizer. We list detailed hyperparameters in Table 7. Experiments are run on NVIDIA-P40 and NVIDIA-A100.

heading: 4.4 Main Results
text: Table 3 presents the main results of SANTA. From these results, the following four insights can be drawn. (1) on all five datasets, SANTA achieves the best F1 performance among all DS-NER baselines, and strikes a good balance between precision and recall, demonstrating superiority when trained on distantly-supervised text; (2) compared to original supervised methods, including BiLSTM-CRF and RoBERTa, SANTA improves the F1 score with an average increase of 18.48% and 7.47% respectively, which demonstrates the necessity of DS-NER models and the effectiveness; (3) compared with methods only focusing on incomplete annotation such as NegSampling, SANTA achieves more balanced precision and recall, showing the necessity to handle two types of noise. (4) compared with methods handling two types of noise with the same strategy such as RoSTER, SANTA achieves better performance both in precision and recall, demonstrating the effectiveness of separate handling.

heading: 4.5 Analysis
text: Ablation Study. To further validate the effectiveness of each component, we compare SANTA with the fine-grained ablations by removing one component at a time in Table 4. (1) MFL improves the precision and recall compared with using noisetolerant GCE + SR, indicating that MFL can help model to avoid underfitting. When using CE instead of MFL, percision is significantly reduced showing MFL can better handle entity ambiguity problem. Meanwhile, designed MLS can reduce
the fluctuation of Focal Loss to achieve better performance, and better than static Label Smoothing; (2) Entity-aware KNN consists of KNN-augmented Inference and Entity-aware CL, the significantly reduction of F1 score and only using one of them achieves poor performance, both showing the design of Entity-aware KNN is necessary; (3) compared with GCE, CE and MFL, GCE + SR shows the strong capability to handle the incomplete annotation noise, due to the improvement of robustness and asymmetric noise condition; (4) Boundary Mixup can significantly improve the recall, which indicates it can help model to make more correct prediction of the samples around the decision boundary and alleviate the decision boundary shifting problem. Boundary Mixup reduces randomness of sampling in Mixup and focus on examples around the decision boundary to better handle the incomplete annotation in DS-NER.
Case Study. We also perform case study to better understand the advantage of SANTA in Table 5. We show the prediction of Conf-MPU only focusing on
incomplete annotation noise, and SCDL handling two types of noise with same strategy. Conf-MPU is able to learn from labeled DS-NER text and slightly learn to generalize, such like “Johnson" and “John McNamara" can be recognized correctly. But the limited generalization capability leads to memorize “Columbia" and “Angels" with wrong labels. Meanwhile, Conf-MPU can not handle entity ambiguity well because it ignores the inaccurate annotation noise, such as wrongly predicting the type of span “California Angels" and “Columbia Presby Hospital". SCDL is able to generalize better and slightly handle entity ambiguity, due to the comprehensive consideration of two types of noise. But it is still impacted by entity ambiguity problem in difficult span “California Angels" . SANTA can further detect the noisy labels via separate strategies and correctly recognize all the entities in this sentence from CoNLL2003 training set.
Strategies Exploration. We further explore our strategies as following: (1) as shown in Table 4, we simply use Focal Loss, CE, GCE + SR in Spanbased NER model as baselines. Meanwhile, we use CE & GCE + SR separately for the spans labeled as entities suffered from inaccurate annotation & the spans labeled as non-entities suffered from incomplete annotation. CE & GCE + SR and SANTA achieve better performance, showing that our motivation of separately handing is helpful and well-designed separate strategies is powerful. (2) as shown cases in Table 6, we can observe that only use Entity-aware KNN and only use Boundary Mixup can both help model to make more accurate predictions. Entity-aware KNN augments the score from model and finally get the right label prediction, alleviating the entity ambiguity problem. Boundary Mixup correctly help model to recall span “British", which indicates decision boundary shifting problem is relieved.

heading: 5 Conclusion
text: Inaccurate and incomplete annotation noise are two types of noise in DS-NER. We propose SANTA to use separate strategies for two types of noise. For inaccurate annotation, we propose Memorysmoothed Focal Loss and Entity-aware KNN to relief the ambiguity problem. For incomplete annotation, we utilize noise-robust loss GCE + SR and propose Boundary Mixup to improve the robustness and mitigate the decision boundary shifting problem. Experiments show that SANTA achieves
state-of-the-art methods on five DS-NER datasets and the separate strategies are effective.
Limitations
Our proposed work is dedicated to considering the noise in DS-NER, and our noise-specific analyses are all based on this task. Therefore, if it were not for DS-NER task, our model would not necessarily be robust compared to other task-specific methods. Also, our approach is based entirely on previous experimental settings in DS-NER, so we do not consider how to reduce noise from the distant supervision process, e.g., designing models to help the annotation process rather than learning to reduce noise from the distantly-supervised text. Designing models to help the distant supervision process could be a direction for future study.

heading: Acknowledgements
text: This paper is supported by the National Key R&D Program of China under Grand No.2018AAA0102003, the National Science Foundation of China under Grant No.61936012. Our code can be found in https://github.com/PKUnlpicler/SANTA.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We discuss the limitations in Sec. Limitations
7 A2. Did you discuss any potential risks of your work? Our work focuses on Distantly-Supervised Named Entity Recognition and public datasets, which does not pose a risk.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: We discuss abstract and introduction in Sec. Abstract and Sec. Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We report them in appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We discuss them in Sec. Experiment and Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We discuss them in Sec. Experiment.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. we do not use them.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Designing models to help the distant supervision process could be a direction for future study