CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition
Year: 2023

Abstract:
Cross-lingual named entity recognition (NER) aims to train an NER system that generalizes well to a target language by leveraging labeled data in a given source language. Previous work alleviates the data scarcity problem by translating source-language labeled data or performing knowledge distillation on target-language unlabeled data. However, these methods may suffer from label noise due to the automatic labeling process. In this paper, we propose CoLaDa, a Collaborative Label Denoising Framework, to address this problem. Specifically, we first explore a model-collaboration-based denoising scheme that enables models trained on different data sources to collaboratively denoise pseudo labels used by each other. We then present an instance-collaboration-based strategy that considers the label consistency of each token’s neighborhood in the representation space for denoising. Experiments on different benchmark datasets show that the proposed CoLaDa achieves superior results compared to previous methods, especially when generalizing to distant languages.1

heading: 1 Introduction
text: The named entity recognition (NER) task aims to locate and classify entity spans in a given text into predefined entity types. It is widely used for many downstream applications, such as relation extraction and question answering. Deep neural networks have made significant progress on this task leveraging large-scale human-annotated data for training. However, fine-grained token-level annotation makes it costly to collect enough high-quality labeled data, especially for low-resource languages. Such scenarios motivate the research on zero-shot cross-lingual NER, which attempts to leverage labeled data in a rich-resource source language to
*Work during internship at Microsoft. †Corresponding author. 1Our code is available at https://github.com/
microsoft/vert-papers/tree/master/papers/CoLaDa.
solve the NER task in a target language without annotated data.
Recent attempts at cross-lingual NER can be roughly categorized from two aspects: learning language-independent features via feature alignment (Huang et al., 2019; Keung et al., 2019) and learning language-specific features from automatically labeled target-language data (Wu et al., 2020c,b). Despite bringing great success to crosslingual NER, the former line of research misses exploiting language-specific features and thus shows substandard performance, especially when transferring to distant languages, e.g., from English to Arabic (Fu et al., 2023). Hence, a series of studies focuses on the latter category, which typically creates pseudo-labeled target-language data and uses it to perform conventional supervised learning or teacher-student learning. For example, as shown in Fig 1(a), earlier studies (Ehrmann et al., 2011; Mayhew et al., 2017; Xie et al., 2018; Jain et al., 2019), such as TMP (Jain et al., 2019), first translate labeled data in the source language and then perform label projection. Recently, several approaches have
5995
utilized a weak model, which could be an NER model either trained on the source language’s labeled data as in TSL (Wu et al., 2020c), or further finetuned on the generated translation data as in UniTrans (Wu et al., 2020b), to annotate the unlabeled target-language data for improvement, as shown in Fig 1(b) and Fig 1(c).
Unfortunately, these methods inevitably suffer from the label noise induced by inaccurate translation and label projection, or the weak model’s limited capability. Although some methods are proposed to mitigate the label noise problem by additionally training an instance selector (Liang et al., 2021; Chen et al., 2021) or designing heuristic rules for data selection (Ni et al., 2017), they independently manipulate either the translation data (Dtrans) (Ni et al., 2017) or the target-language data (Dtgt) pseudo-labeled by NER models trained in the source language (Liang et al., 2021; Chen et al., 2021). Hence, all these methods ignore the complementary characteristics between both for denoising. Particularly, from the text view, Dtgt is collected from a natural text distribution of the target-language data, while Dtrans can be regarded as a way of data augmentation to provide more lexicon variants. From the labeling function view, labels of Dtrans are obtained via the label projection algorithm, which have little association with those of Dtgt generated by NER models.
With such consideration, we propose a modelcollaboration-based denoising scheme, which incorporates models trained on both data sources to mutually denoise the pseudo-labels of both data sources in an iterative way. As shown in Fig 1(d), we first leverage Mtgt trained on the pseudolabeled target-language data Dtgt to denoise the translation data annotated by label projection. In this way, the learned model Mtrans will be less affected by noise in the translation data. We then employ the improvedMtrans to re-label the targetlanguage unlabeled data Dtgt. It is expected that there is less noise in the relabeled data, and thus we can produce a more powerfulMtgt. We perform this procedure for several iterations, so that all the involved data sources and models can be improved in an upward spiral.
Moreover, borrowing the idea from anomaly detection (Gu et al., 2019) that a given data point’s neighborhood information can be used to measure its anomalism, here we find that the similar tokens in the feature space can also collaborate for denois-
ing. Previous studies (Zhai and Wu, 2019; Xu et al., 2020) have shown that instances with the same label are more likely to locate close to each other in the representation space. Our intuition is that, if a token’s label conflicts a lot with labels of other tokens in its neighborhood, then this label is probably noisy. Therefore, we further propose an instancecollaboration-based denoising strategy to explore the neighborhood structure of each token for denoising, as shown in Figure 2. Specifically, we utilize the label consistency of each token’s neighborhood in the representation space to re-weight the soft-labeled examples in knowledge distillation.
We integrate the instance-collaboration-based denoising strategy into the model-collaboration-based denoising scheme and propose a Collaborative Label Denoising framework, i.e., CoLaDa, for cross-lingual NER. We conduct extensive experiments on two popular benchmarks covering six languages for evaluation. Experimental results show that our method outperforms existing state-of-theart methods. Qualitative and quantitative analyses further demonstrate the effectiveness of our framework in reducing the data noise.

heading: 2 Problem Formulation
text: Here we take the typical sequence labeling formulation for the named entity recognition task. Given a sequence with L tokens x = (x1, . . . , xL) as the input text, an NER system is excepted to assign each token xi with a label yi.
In this paper, we assume to have the labeled training data Dsrc = {(xs,ys)} in the source language, the unlabeled data Dtgt = {xu} from the target language, and translation dataDtrans = {(xt,yt)} obtained by data projection from Dsrc. Our goal is to train an NER modelM that can generalize well to the target language utilizing these resources.

heading: 3 CoLaDa Framework
text: Figure 3 depicts an overview of the CoLaDa framework. It is an iterative model-collaboration-based denoising framework which consists of two steps: noise-robust learning on translation data and noiserobust learning on unlabeled target-language data. An instance-collaboration-based denoising strategy (Sec 3.1) is then integrated into the modelcollaboration-based denoising procedure (Sec 3.2).

heading: 3.1 Instance Collaboration for Denoising
text: Previous work (Zhai and Wu, 2019; Xu et al., 2020) indicates that tokens with the same labels are more likely to locate close to each other in the representation space of a deep neural network. If the label of a given token is inconsistent with lots of its neighbors, this token would be isolated from other tokens with the same label in the feature space, and hence its label is more likely to be noisy. Therefore, we propose instance-collaboration-based denoising, which evaluates the reliability of a given token’s label by measuring the label consistency of its neighborhood, and then uses the reliability score to weight the noisy labels from a teacher model M for knowledge distillation on data D = {x}. Noisy labels are expected to have lower weights than clean ones.
Create a memory bank. We leverage the feature extractor F of the NER modelM to obtain the hidden representations h = {hi}Li=1 of each sentence x = {xi}Li=1 ∈ D:
h = F(x). (1) We then construct a memory bank BD = {h} to store the hidden representations of all tokens in D. Compute label consistency. Given a token xi, we retrieve its K-nearest neighbors NK(xi) in BD using cosine similarity. Let pi denote the soft label (i.e., the probability distribution over the entity label set) assigned by the teacher model M for xi. We measure the label consistency of xi, i.e., λ(pi;xi), by calculating the fraction of xi’s neighbors that are assigned with the same labels as xi in NK(xi):
λ(pi;xi) = 1
K
∑
xj∈Nk(xi) I(ŷj = ŷi), (2)
where ŷi = arg max (pi) is the pseudo entity label corresponding to the maximum probability in pi. Similarly, ŷj is the pseudo entity label corresponding to xj . I is the indicator function.
Produce a reliability score. We use the label consistency λ(pi;xi) to compute the reliability score of the soft label pi, which is further used as the weight of pi during model learning (see 3.2). Considering that different entity types may contain different levels of label noise and show different statistics on label consistency, here we present a class-adaptive reliability score for weighting:
w(pi;xi) = Sigmoid (α (λ(pi;xi)− µ(ŷi))) , (3)
where µ(ŷi) denote the mean of all λ(pj ;xj) where arg max (pj) = ŷi and xj ∈ D. α > 0 is a hyper-parameter that controls the sharpness of the weighting strategy. If α→ 0, all tokens have equal weights. If α → ∞, tokens whose label consistency is larger than the average label consistency w.r.t. its pseudo label will be weighted with 1 and those with smaller consistency will be dropped.

heading: 3.2 Model Collaboration for Denoising
text: Here we elaborate on the details of the two noiserobust training processes. Algorithm 1 depicts the overall training procedure of CoLaDa.
Noise-robust training on translation data. Assuming the availability of a collaborator Mtgt2 trained on pseudo-labeled target-language data Dtgt, here we focus on leveragingMtgt to reduce the label noise in the translation data Dtrans = {(xt,yt)}, with which we further deliver a more powerful modelMtrans.
Specifically, given a sentence (xt,yt) ∈ Dtrans, we first obtain the soft label pui of each x t i ∈ xt from the collaboratorMtgt. Then, we take both the one hot label yti and the soft label p u i as the supervision to train the model Mtrans.3 Denote the output probability distribution ofMtrans for xti as p̂ t i. The loss function w.r.t. x t is defined as:
Lxt = 1 L
L∑
i=1
( β1CE(p̂ti, p u i ) + β2CE(p̂ t i, y t i) ) ,
(4) where CE(·, ·) denotes the cross-entropy loss, L is the sentence length, β1 and β2 are weighting scalars. Here we further incorporate the instancecollaboration-based denoising strategy (3.1) to provide a token-level reliability evaluation to the supervision from the collaboratorMtgt via:
β1(x t i)← β1 ∗ w(pui , xti), (5)
where w(pui , x t i) is calculated by Eq. (3).
Noise-robust training on target-language unlabeled data. Here we leverageMtrans obtained via the above noise-robust training on translation data to provide high-quality supervision for
2For the first iteration, we use an NER model trained on the source language labeled data Dsrc. For the later iterations (i > 1), we use the model from the noise-robust-training on target-language unlabeled data in the previous iteration (i−1).
3The student modelMtrans is initialized fromMtgt to equip the knowledge of real target-language text distribution for better generalization during test.
Algorithm 1 Pseudo code of CoLaDa. Input: an NER modelMsrc trained onDsrc, translation data Dtrans, the unlabeled data Dtgt, the maximum iteration T. 1: M(0)tgt ←Msrc ▷ Initialization 2: for i = 1, 2, . . ., T do 3: # Step 1: Noise-robust training on Dtrans 4: InferenceM(i−1)tgt onDtrans = {(xt,yt)} to get the
predictions D̂trans = {(xt,pu)} 5: Get w for (xt,pu) ∈ D̂trans withM(i−1)tgt , Eq.(3) 6: TrainM(i)trans with loss on (xt,yt,pu,w), Eq.(4) 7: # Step 2: Noise-robust training on Dtgt 8: InferenceM(i)trans on Dtgt = {xu} to get the predic-
tions D̂tgt = {(xu,pt)} 9: Get w′ for (xu,pt) ∈ D̂tgt withM(i)trans, Eq.(3)
10: TrainM(i)tgt with loss on (xu,pt,w′), Eq.(6) 11: end for Output: an NER modelM(T )tgt .
Dtgt = {xu}. By performing knowledge distillation on Dtgt, the student modelMtgt is supposed to benefit from the unlabeled data drawn from the real text distribution in the target language with the knowledge from the teacher modelMtrans.
Specifically, given a sentence xu ∈ Dtgt, we first utilizeMtrans to predict soft label pti for each token xui ∈ xu. Then, we integrate the instancecollaboration-based denoising technique into the learning process. The loss function w.r.t. xu to train the student modelMtgt can be formulated as:
Lxu = 1 L
L∑
i=1
w(pti, x u i ) · CE(p̂ui , pti), (6)
where p̂ui denotes the output probability distribution ofMtgt for the i-th token xui and w(pti, xui ) is calculated by Eq. (3).

heading: 4 Experiments

heading: 4.1 Experiment Settings
text: Datasets We conduct experiments on two standard cross-lingual NER benchmarks: CoNLL (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003) and WikiAnn (Pan et al., 2017). CoNLL contains four languages: English (en) and German (de) from the CoNLL-20034 NER shared task (Tjong Kim Sang and De Meulder, 2003), and Spanish (es) and Dutch (nl) from the CoNLL-20025 NER shared task (Tjong Kim Sang, 2002). This dataset is annotated with four entity types: PER, LOC, ORG, and MISC. WikiAnn
4https://www.clips.uantwerpen.be/conll2003/ner/ 5https://www.clips.uantwerpen.be/conll2002/ner/
contains an English dataset and datasets in three non-western languages: Arabic (ar), Hindi (hi), and Chinese (zh). Each dataset is annotated with 3 entity types: PER, LOC, and ORG. All datasets are annotated with the BIO tagging scheme. We use the train, development, and test splits as previous work (Wu and Dredze, 2019; Wu et al., 2020b).
We take English as the source language and other languages as the target language, respectively. We remove the labels of the training data for the target language and take it as the unlabeled target language data. For the CoNLL benchmark, we use the word-to-word translation data provided in UniTrans (Wu et al., 2020b) for a fair comparison. For the WikiAnn benchmark, we translate the source data to the target language with the public M2M100 (Fan et al., 2020) translation system and conduct label projection with the marker-based alignment algorithm as Yang et al. (2022).
Evaluation The entity-level micro-F1 on test set of the target language is used as the evaluation metric. We report the mean value of 5 runs with different seeds for all the experiments.
Implementation Details For the base NER model, we stack a linear classifier with softmax over a base encoder such as mBERT. We implement our framework with Pytorch 1.7.16, the HuggingFace transformer library (Wolf et al., 2020), and use FAISS (Johnson et al., 2019) for embedding retrieval. Following Wu and Dredze (2019) and Zhou et al. (2022), we use the multilingual BERT base model (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) large model as our base encoders. Most of our hyper-parameters are set following Wu et al. (2020b). We use AdamW (Loshchilov and Hutter, 2019) as optimizer and train the model on source NER data with the learning rate of 5e-5 for 3 epochs. The dropout rate is 0.1. For teacher-student learning, we train the model with a learning rate of 2e-5 for 10 epochs. We freeze the bottom three layers as Wu and Dredze (2019). Following Keung et al. (2019), we choose other hyper-parameters according to the target language dev set. We set K in Eq. (2) to 500 and α in Eq. (3) to 6. For the first iteration, we start with an NER model trained on the source-language data to denoise the translation data with β1 and β2 in Eq. (5) setting to 0.5. For the following iterations, β1 is set to 0.9 and β2 is set to 0.1. The maximum number of iterations is 8.
6https://pytorch.org/

heading: 4.2 Main Results
text: Baselines We compare our method to previous start-of-the-art baselines as follows: i) feature alignment based methods: mBERT (Wu and Dredze, 2019), XLM-R (Conneau et al., 2020), BERTalign (Wu and Dredze, 2020), AdvCE (Keung et al., 2019), and AdvPicker (Chen et al., 2021); ii) translation based methods: MulDA (Liu et al., 2021), UniTrans (Wu et al., 2020b), and TOF (Zhang et al., 2021)); iii) knowledge distillation based methods: TSL (Wu et al., 2020c), RIKD (Liang et al., 2021), and MTMT (Li et al., 2022); iv) consistency based methods: xTune (Zheng et al., 2021) and ConNER (Zhou et al., 2022).
Performance Comparison Tables 1 and 2 show the performance comparison of the proposed CoLaDa and prior start-of-the-art baselines on CoNLL and Wikiann, respectively. It can be seen that
CoLaDa outperforms prior methods with both encoders, achieving a significant improvement of 2.70 F1 scores on average for CoNLL and 10.14 F1 scores on average for WikiAnn with XLM-R as the encoder. This well demonstrates the effectiveness of our approach. Interestingly, CoLaDa shows more significant superiority when transferring to distant target languages in WikiAnn. The knowledge distillation based baselines (i.e., TSL, RIKD, MTMT) struggle on distant languages such as Chinese (zh) due to the noisy predictions from the weak teacher modelMsrc trained in the source language. UniTrans, which is developed with the same data sources as ours, shows poor performance, especially in distant languages such as Arabic (ar). We conjecture that the problem of label noise is even more critical in these distant languages. Our CoLaDa can better handle noise in both translation data and unlabeled target-language data, thus leading to significant performance gains.

heading: 5 Analysis

heading: 5.1 Ablation Study
text: To further validate the effectiveness of each mechanism in the proposed framework, we introduce the following variants of CoLaDa in an ablation study: 1) CoLaDa w/o instance collaboration, where we directly set the reliability score in Eq. (3) to 1 for all tokens. 2) CoLaDa w/o translation data denoise, where we set β1 in Eq. (4) to 0. 3) CoLaDa w/o iteratively denoise, where we remove the iterative enhancement and only conduct the denoising process for one iteration. 4) CoLaDa w/o model collaboration, where we set β1 in Eq. (4) to 0, remove the iteration mechanism, and directly take the model finetuned on Dtrans as the teacher model to train a student model with instance-collaborationbased denoising on Dtgt. 5) CoLaDa w/o instance & model collaboration, which further drops the instance-collaboration-based denoising from 4).
Table 3 shows the ablation results. We can draw
some in-depth conclusions as follows. 1) CoLaDa outperforms CoLaDa w/o instance collaboration, which highlights the effectiveness of leveraging neighborhood information to reduce label noise in knowledge distillation.
2) CoLaDa outperforms CoLaDa w/o translation data denoise, which emphasizes the importance of using the collaboratorMtgt to refine labels of translation data, especially in distant languages where the translation data is noisier (e.g., 12.8 F1 drop on Arabic and 5.7 F1 drop on Chinese).
3) CoLaDa outperforms CoLaDa w/o iteratively denoise, which indicates the necessity of iterative learning: models obtained from the previous iteration should be re-used as the collaborator to further improve label quality in the next iteration.
4) CoLaDa w/o instance & model collaboration, which eliminates all denoising strategies from CoLaDa, leads to a significant performance drop, demonstrating the essentiality of label denoising for cross-lingual NER.

heading: 5.2 Analysis of Model Collaboration
text: Here we attempt to understand how the two models, i.e., Mtrans and Mtgt, collaboratively improve each other.
As shown in Figure 4, F1 scores ofMtrans and Mtgt consistently improve as iterations go on, and finally converge at the last iteration. This indicates that both models benefit from the proposed model collaboration scheme. Two reasons are speculated: i) An improvedMtgt can provide more accurate labels on the translation data, which further help to improve Mtrans via noise-robust learning on such translation data. For example, at the initial step (i = 0), the F1 score of the model M0trans trained on the original translation labels is 50.0. With the additional supervision from the collaboratorM0tgt,M1trans achieves a performance gain of 5.7 F1. ii) An improvedMtrans predicts pseudo labels with higher quality on the target-language unlabeled data, which further benefits the learning of Mtgt. As in Figure 4, the quality of pseudo-labeled Dtgt (the green line) grows as Mtrans improves. In this way, bothMtrans andMtgt are providing more and more reliable labels for each other to learn as the iterations progress.

heading: 5.3 Analysis of Instance Collaboration
text: This subsection dives into the working mechanism of the instance-collaboration-based denoising.
Reliability scores v.s. label quality. To study the relationship between reliability score and label quality, we partition tokens in the target-language unlabeled data, xi ∈ Dtgt into several bins according to their reliability scores w(pti, xi) calculated via M(1)trans. Then, we compute the tokenlevel F1 over each bin by comparing pseudo labels ŷi = argmax(p t i) to the ground-truth ones. As shown in Figure 5, the label quality is proportional to the reliability score, which well demonstrates the effectiveness of our instance-collaboration-based denoising strategy.
Analysis of Label Consistency. We also study the characteristics of label consistency w.r.t. different entity types and representation spaces of the memory bank. Figure 6 shows the results. We can draw some in-depth observations as follows.
i) Clean tokens show a larger average consistency than noisy tokens w.r.t. all entity types, demonstrating the effectiveness of our label consistency based denoising strategy again.
ii) Different entity types lead to different distributions of label consistency, which validates the necessity of our design for class-adaptive reliability score for weighting as Eq.(3).
iii) Label consistencies calculated with token representations from the upper layers are generally larger than those corresponding to the bottom layers. Also, the label consistency gap between clean tokens and noisy tokens gets larger from the bottom to the top (e.g., the gap between two orange lines). This may be attributed to the fact that representations from upper layers are more taskspecific (Muller et al., 2021), hence they can better discriminate between noisy and clean tokens.
Choice of K for neighborhood information. Figure 7 shows the performance of CoLaDa using different K in Eq. (2). Generally speaking, CoLaDa is robust to the choice of K. Any value for K > 0 leads to a better performance compared with removing the instance collaboration, i.e., K = 0. A smaller K may lead to a slight performance drop due to limited neighborhood information.

heading: 5.4 Case Study
text: To better illustrate the kinds of label noise presented in the data and the capability of CoLaDa to address such noise, we conduct a case study on the Chinese translation data from the WikiAnn English data. As shown in Figure 8, there are three typical cases of noisy labels in the translation data: noisy labels induced by inaccurate translations, alignment errors, and annotation errors in the original source-language data.7 Figure 8 shows that the translate-train model, finetuned on the original translation data, overfits the noisy labels. However, CoLaDa is less affected by such noise and makes correct predictions.

heading: 6 Related Work

heading: 6.1 Cross-lingual NER
text: Prior work on cross-lingual NER mainly falls into two major categories: feature-based and data-based transfer.
Feature-based These methods learn languageindependent features so that the model trained on the source language can directly adapt to the tar-
7Due to the short entity context information in many sentences in WikiAnn, the translation quality of entity mentions with M2M100 is less than satisfactory on the dataset.
get language. Earlier work exploits word clusters (Täckström et al., 2012), gazetteers (Zirikly and Hagiwara, 2015), Wikifier features (Tsai et al., 2016), and cross-lingual word embedding (Ni et al., 2017), etc. More recently, with the fast growth of multilingual pre-trained language models (Devlin et al., 2019; Conneau et al., 2020) and their promising results on cross-lingual transfer (Wu and Dredze, 2019), lots of studies build upon such pre-trained models and further promote the learning of language-independent features via meta-learning (Wu et al., 2020d), contrastive alignment (Wu and Dredze, 2020), adversarial learning (Keung et al., 2019; Chen et al., 2021), and by integrating other resources (Fetahu et al., 2022). Despite the great success, they mostly ignore language-specific features, which are especially important when transferring to distant languages (Fu et al., 2023).
Data-based These approaches learn languagespecific features via automatically labeled targetlanguage data and can be further divided into translation-based and knowledge distillation-based methods.
Translation-based methods first translate the source-language data to the target language, then perform label projection from the source side to the target side. Some prior studies have proposed to use cheap translation such as word-to-word (Xie et al., 2018) or phrase-to-phrase (Mayhew et al., 2017) translation. Jain et al. (2019) propose an entity projection algorithm to utilize the Google translation system. Recently, Liu et al. (2021) and Yang et al. (2022) propose to translate sentences with pre-defined markers for label projection. And Ni et al. (2017) design heuristic rules to select high-
quality translation data. However, both data noise and artifacts (Artetxe et al., 2020) in the translation data still limit the performance of such methods (García-Ferrero et al., 2022).
Knowledge distillation-based methods train a student model on unlabeled target-language data with the soft labels from a teacher model (Wu et al., 2020c). Li et al. (2022) improve the single task based teacher-student learning with entity similarity as an auxiliary task. To mitigate the label noise from the teacher model, Chen et al. (2021) propose AdvPicker, which trains a language discriminator to select the less language-dependent unlabeled data for knowledge distillation; Liang et al. (2021) design a reinforcement learning algorithm to train an instance selector according to features such as model confidence to select reliable pseudo labels iteratively.
While most previous work leverages either translation data or unlabeled data, UniTrans (Wu et al., 2020b) utilizes the model trained on translation data to perform teacher-student learning on unlabeled data. But it still suffers from the data noise problem. More recently, consistency training (Zheng et al., 2021; Zhou et al., 2022) has also been explored to leverage both unlabeled data and translation data without explicit label annotation.
To the best of our knowledge, we are the first to propose a unified denoising framework to handle data noise in both translation and unlabeled data collaboratively from the model and instance levels for cross-lingual NER.

heading: 6.2 Learning with Label Noise
text: Previous studies mainly address the label noise via re-weighting examples (Shu et al., 2019), designing noise-robust loss functions (Ma et al., 2020), and selecting clean instances (Bahri et al., 2020; Wu et al., 2020a), etc. However, these methods only consider the corrupted labels that naturally occur in one data source. In this work, we consider the complementary characteristics of translation and unlabeled data, and design a model-collaborationbased denoising scheme. While Xu et al. (2023) target at the few-shot learning scenario and leverage the neighborhood information among the labeled examples to hard-select the reliable pseudo labels in self-training, we focus on the zero-shot cross-lingual setting and softly re-weight the noisy pseudo-labels in knowledge distillation without any clean labeled data in target language.

heading: 7 Conclusion
text: To address the problem of label noise in crosslingual NER, this paper presents CoLaDa, a collaborative label denoising framework. We propose a model-collaboration-based denoising scheme to make two models trained on different data sources to denoise the labels of each other and hence promote each other’s learning. We further propose an instance-collaboration-based strategy that collaboratively considers the label consistency among similar tokens in the feature space to re-weight the noisy labels assigned by a teacher model in knowledge distillation. By integrating the instancecollaboration strategy into the model-collaboration denoising scheme, our final framework CoLada achieves superior performance over prior start-ofthe-art methods by benefiting from better handling the data noise.
Limitations
Our framework relies on the availability of translation system and unlabeled data in the target language, which can not be applied to languages without any unlabeled text or translation text. The knowledge distillation step requires a certain amount of unlabeled text, while it may struggle in cases where only few hundreds of unlabeled sentences are available. It would be interesting to combine our label denoising framework with data augmentation techniques in such scenarios. Besides, the boarder application to other low-resource languages, such as MasakhaNER 2.0 (Adelani et al., 2022), and other cross-lingual sequence labeling tasks are left for exploration in future work.

heading: A Appendix
text: A.1 Dataset Statistics
Table A.1 reports the dataset statistics for CoNLL and WikiAnn.
A.2 Other Implementation Details
All experiments are conducted on a Tesla V100 (32GB). The total of trainable parameters (Mtrans andMtgt) for our model with mBERT-base-cased as the encoder is 172M and the training time is about 35 mins for one iteration. With XLM-Rlarge as our base encoder, the total of trainable parameters are 822M and the training takes about 90 mins for one iteration.
A.3 Baselines
We consider the following start-of-the-art baselines:
mBERT (Wu and Dredze, 2019) and XLMR (Conneau et al., 2020) directly train an NER model on the labeled data in the source language, with mBERT and XLM-R as the basic encoder, respectively.
BERT-align (Wu and Dredze, 2020) tries to explicitly add word-level contrastive alignment loss to enhance the mBERT representation.
AdvCE (Keung et al., 2019) exploits adversarial learning on source- and target-language text to avoid learning language-specific information.
AdvPicker (Chen et al., 2021) leverages adversarial learning to learn language-shared features and then selects the less language-specific sentences in target-language unlabeled text for knowledge distillation.
MulDA (Liu et al., 2021) proposes the labeled sequence translation method for data projection from source-language NER data, a generative model is further applied to augment more diverse examples in the target language.
UniTrans (Wu et al., 2020b) unifies model- and translation-data-based-transfer via knowledge distillation.
TOF (Zhang et al., 2021) leverages the labeled data for machine reading comprehension task on target language to help the NER task in crosslingual transfer.
TSL (Wu et al., 2020c) proposes knowledge distillation to use unlabeled target-language data for cross-lingual NER.
RIKD (Liang et al., 2021) proposes a reinforcement learning algorithm to iteratively select reliable pseudo-labels for knowledge distillation.
MTMT (Li et al., 2022) proposes multi-task multi-teacher knowledge distillation, which further leverages the entity similarity task.
xTune (Zheng et al., 2021) leverages unlabeled translation text and other word-level data augmentation techniques for consistency training.
ConNER (Zhou et al., 2022) conducts spanlevel consistency training on unlabeled targetlanguage data using translation and further applies dropout-based consistency training on the sourcelanguage data.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: "Limitations" section.
A2. Did you discuss any potential risks of your work? Not applicable. We study cross-lingual NER task on public datasets, our work doesn’t have potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract & Sec 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sec 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sec 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Sec 4
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. These datasets are all public for research purpose.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We conduct experiments on public datasets.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sec 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix.
C 3 Did you run computational experiments? Sec 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sec 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sec4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Sec 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: , 2022), and other cross-lingual sequence labeling tasks are left for exploration in future work