Federated Domain Adaptation for Named Entity Recognition via Distilling with Heterogeneous Tag Sets
Year: 2023

Abstract:
Federated learning involves collaborative training with private data from multiple platforms, while not violating data privacy. We study the problem of federated domain adaptation for Named Entity Recognition (NER), where we seek to transfer knowledge across different platforms with data of multiple domains. In addition, we consider a practical and challenging scenario, where NER datasets of different platforms of federated learning are annotated with heterogeneous tag sets, i.e., different sets of entity types. The goal is to train a global model with federated learning, such that it can predict with a complete tag set, i.e., with all the occurring entity types for data across all platforms. To cope with the heterogeneous tag sets in a multi-domain setting, we propose a distillation approach along with a mechanism of instance weighting to facilitate knowledge transfer across platforms. Besides, we release two re-annotated clinic NER datasets, for testing the proposed method in the clinic domain. Our method shows superior empirical performance for clinic NER with federated learning.

heading: 1 Introduction
text: Federated learning for Named Entity Recognition (NER) is the task of collaboratively learn with NER datasets from multiple platforms, while not violating data privacy, i.e., without sharing data across different platforms (Ge et al., 2020). A platform can be an institution, e.g., a hospital or a drug company, where a private collection of clinic NER dataset are locally stored. In reality, data from different platforms are usually sampled from different clinic domains, due to different patient groups, etc. Additionally, different schemes may be used when annotating for different platforms. This happen when healthcare providers use customized tag sets to create their own datasets (Beryozkin et al., 2019). As an example, a hospital may hold a dataset of
†Corresponding Author
clinical reports from doctors annotated with entities of patient Disease and Drugs prescribed by the doctors, while a drug company may have text data of patient feedback, annotated with Drugs and their adverse drug effects (ADE). In this case, it would be mutually beneficial for the hospital and the drug company if they can train in a federated manner a shared (global) NER model with both datasets. The global model should in principle predict with the complete tag set, i.e., {Disease, Drugs, ADE}, enabling the hospital to also recognize the ADE in their clinic reports and the drug company to identify Disease from their patient feedback, without sharing or re-annotating their local datasets. This can be regarded as a problem of domain adaptation, since the key challenge is to efficiently transfer knowledge of locally unlabeled entity types, i.e., Disease and ADE across domains/platforms, so that the resulting global model can work for both the hospital and the drug company.
So motivated, we study federated domain adaptation for clinic NER in the multi-domain setting where datasets from multiple platforms representing different domains. Further, we address a more challenging scenario in which different platforms also annotate with different tag sets, i.e., set of entity types. The goal is to benefit all platforms from federated learning, via training a global model that predicts with the complete tags set, including all the encountered entity types, for text of different domains/platforms. Note there are previous works studying federated NER in a multi-domain setting (Ge et al., 2020; Zhao et al., 2021). However, these works generally presume that the NER model for one platform only predicts with the entity types annotated in the local training data, unlike our setting that requires predicting on a larger tag set (the complete tag set). Here, we claim that such an assumption might not be practical in a multi-domain setting. To illustrate this, suppose there is a platform with annotations of training data
7449
that sufficiently cover enough entity types for its own propose of evaluation. For this platform, with enough amount of training data locally, joint training with data from other distant domains may harm the performance of the resulting model on its local data, i.e., there is no guarantee that data of other platforms is similar enough to be beneficial to its own domain. As a result, such a platform might be reluctant in joining federated learning, further considering the potential risk of data leakage in any federated learning system (Li et al., 2021). On the contrary, we require to predict with the complete tag set, while annotating with incomplete (subset of the complete tag set) and heterogeneous tag sets locally. This motivates a platform to participate in federated learning, so that it can benefit from knowledge of locally unlabeled entity types transferred from other platforms.
To address the heterogeneous tag sets and facility knowledge transfer across platforms, with regards to the locally unlabeled entity types, we propose a distillation approach, that distills knowledge of unlabeled entity types from other platforms via pseudo-annotations with the complete tag set. Based on the proposed distillation, we further propose a instance weighting mechanism, so that knowledge learned with local data is more transferable across platforms. We adopt a promptbased NER model (Chen et al., 2022) with superior performance for cross-domain NER, and only transmit prompt-related parameters (7% of the model size) for each round of federated learning to reduce the communication cost. We should note that a comprehensive evaluation of the global model in
the setting considered requires testing data with the complete tag set for each domain/platform. However, existing public clinical datasets of different domains are usually annotated using different tag sets (with small overlap), i.e., they lack evaluation data that is consistently annotated with the complete tag sets for multiple domains. Therefore, we re-annotate the ADE-Corpus (Gurulingappa et al., 2012) and SMM4H (Weissenbacher et al., 2019) datasets using the annotation scheme of CADEC (Karimi et al., 2015), resulting in datasets of multiple domains that are annotated consistently for evaluation. Our contributions are as follow:
• We study federated learning for clinic NER, where data in different platforms can be from multiple domains and annotated with heterogeneous tag sets.
• We propose a distillation approach along with a weighting mechanism to facilitate knowledge transfer across different platforms.
• We release two re-annotated clinic datasets for evaluation in clinical settings and to encourage future research. Empirical results show that our method delivers superior performance in the considered setting.

heading: 2 Preliminaries

heading: 2.1 Problem Formulation
text: Figure 1 illustrates our considered setting for clinic NER. Suppose there are K platforms in federated learning with datasets {Dk}Kk=1, Dk =
Algorithm 1 Our Federated Learning Algorithm. Assume the non-trainable parameters of the NER model are available in each platform. Randomly initialize the trainable parameters, θ1, for the NER model on the server of federated learning. Initialize the instance weights wki,1 = 1, for i = 1, . . . , Nk and k = 1, . . . ,K (see Section 3.3). for the tth round of federated learning do
% Update instance weighting Compute wki,t+1 according to (12). % Local training Download θt to each local platform. Train locally on each platform, via distilling with the pseudo-complete annotation set resulting in {θkt }Kk=1 (see Section 3.2). % Aggregation Upload {θkt }Kk=1 to the server and aggregate with (1), generating θt+1 for the next round of federated learning.
end for
{Xi,Y Tki }Nki=1, with Nk being the size of Dk. Xi is a text sequence and Y Tki is its NER label sequence, annotated with tag set Tk. In Figure 1, we have T1 = {Drug,ADE}, T2 = {Disease}, etc. We assume Xi of different platforms are from different text domains. The goal is to train an NER model that predicts with the complete tag set i.e. T = ∪Kk=1Tk, for all platforms, without data being shared across different platforms.

heading: 2.2 Federated Learning
text: As illustrated in Figure 1, federated learning involves periodical communication between the server and platforms involving the trainable parameters of the model. Specifically, let θt be the trainable parameters of the NER model before the tth communication round of federated learning. We assume the non-trainable parameters, e.g., the pretrained parameters of a PLM, are available locally in each platform. A typical training cycle of federated learning includes:
Local Training: θt is transferred to each platform and is then trained/updated locally with the private data of each platform. Specifically, θt is trained for Eloc epochs separately on different platforms. We denote {θkt }Kk=1 as the trainable parameters of different platforms from local training.
Aggregation: After local training, each platform will transfer their updated parameters {θkt }Kk=1 to
the server. Since the goal of our federated learning setting is to training a global model for all platforms, the server will aggregate the {θkt }Kk=1, generating θt+1 for the next round of communication. The aggregation is usually performed via weighted averaging, i.e.,
θt+1 = K∑
k=1
mkθ k t , (1)
where ∑
k mk = 1. Since aggregation is not the focus of this work, we will discuss the values of mk in the Appendix. Algorithm 1 shows the complete procedure of federated learning. The proposed distillation and instance weighting mechanism are described in Sections 3.2 and 3.3, respectively.

heading: 3 Methodology

heading: 3.1 Model Architecture
text: In order to efficiently train a global model for all the participants, we need to i) Facilitate knowledge transfer across different platforms/domains, so that each client can benefit from knowledge regarding locally unlabeled entity types, transferred from other platforms. ii) Reduce the communication cost of federated learning. With these considerations, we adopt LightNER (Chen et al., 2022) as our NER model for federated learning. Below, we briefly describe the LightNER model, along with the rationale that we adopt it for our setting.
Sequence-to-Sequence NER: NER is conventionally identified a sequence labeling problem, which predicts with a label-specific output layer (Luo et al., 2020; Lee et al., 2020) on top of a Pretrained Language Model (PLM), e.g., BERT. However, such models may have inferior performance for cross-domain problems, since the label-specific output layer that is trained from scratch cannot benefit from the pretrained knowledge for generalization (Chen et al., 2022). To solve this, recent works (Cui et al., 2021; Chen et al., 2022) adopt a sequence-to-sequence framework for NER based on the pretrained BART model (Lewis et al., 2019), where the entity labels are predicted as natural language tokens in the output sequence, leveraging the pretrained semantics knowledge from BART token embeddings for better generalization. By formulating NER as sequence-to-sequence generation, LightNER achieve superior performance for crossdomain NER tasks, a merit that we value for our setting involving multiple domains. Given a lengthL text sequence Xi = [xl]Ll=1 from platform k,
the model should generate the following label sequence Y T k
i , indicating the start/end positions and entity types of each entity within Xi,
Y T k = [pT k 1 ; · · · ;pT k n ], p T k c = [sc, ec, tc], (2)
where c = 1, · · · , n and [ ; ] denotes concatenation. n is the number of entities in Xi. pTc denotes the cth entity annotated within Xi, where sc/ec denotes its start/end position in Xi, and tc ∈ T k is the entity type.
LightNER follows the encoder-decoder architecture of BART, generating the label sequence Y Ti autoregressively, given Xi. The LightNER model for platform k can be trained via minimizing the following loss of cross-entropy,
LT ki =− 3n∑
l=1
logPθ(y T l |Xi,yT k 1 , · · · ,yT k l−1), (3)
where yT k l is the l th element of Y T k
i and θ denotes the trainable parameters of LightNER.
Prompt Tuning: To preserve the pretrained knowledge from BART for better generalization across domains/platforms, we follow LightNER that freezes the pretrained parameters of BART and inserts tunable prompt parameters for training. Specifically, let q ∈ RNq×D denote an array of Nq prompt tokens, where we have Nq = 10 and d = 768. q is projected by a trainable layer into the keys and values of the self-attention in each pretrained transformer layer, with q being shared by all layers. The projection on q follows (Chen et al., 2022) and is detailed in Appendix B. As a result, the number of trainable parameters in the model is significantly reduced, i.e., only 7% of the total model size, compared with fine-tuning all the model parameters. This leads to reduced communication cost for federated learning, considering that we only need to communicate trainable parameters between the server and platforms.

heading: 3.2 Distillation with Pseudo-Complete Annotation
text: The local datasets of each platform only contain annotations of {T k}Kk=1, T k ⊂ T . For platform k, we denote entity types that are not annotated locally as T \k, with T ∪ T \k = T . During local training, if the local trainable parameter θkt (Algorithm 1) is trained solely with the local annotations of T k, the resulting NER model will learn to ignore the entities of T \k from the input text sequences. This
contradicts our goal of predicting with the complete tag set T . To solve this problem, we notice that the parameter θt in Algorithm 1 is aggregated from updates of different platforms ({θkt−1}Kk=1) with {T k}Kk=1. Thus, the NER model with θt should be able to predict with the complete tag set T , including T k from each platform k. Additionally, considering that θt is downloaded to each platform before the local training of the tth round of federated learning (Algorithm 1), the model with θt should be locally available for each platform. Inspired by this, we propose to distill from the model with θt while training locally with θkt , so that θ k t can be trained with T instead of T k. Specifically, we extract predictions regarding T \k from the the model with θt and combine them with the local annotations of T k, constituting the pseudo-complete annotation. θkt will be trained with the pseudo-complete annotation of the complete tag set T . Let Xi be the ith text sequence from platform k and Y T k
i = [p T k 1 ; · · · ;pT k nloc ] be
the local annotation regarding T k. nloc is the number of locally annotated entities of T k. Given Xi, we first greedly decode the prediction Ŷ Ti from θt,
ŷTl = argmaxy Pθt(y|X1, ŷT1 , · · · , ŷTl−1). (4)
As mentioned above, the prediction Ŷ Ti should have the complete tag set T . Predictions regarding T \k within Ŷ Ti represents the knowledge of un-annotated entity types in platform k, which is transferred from other platforms. We extract such predictions from Ŷ Ti , denoted as, Ŷ T \k i = [pT \k
1 ; · · · ;pT \k
ntrans ]. ntrans is the number of entities in Ŷ T
\k i . {pT \k c }ntransc=1 is defined as in (2), rep-
resenting entities that are predicted as types from T \k with θt. We combine Ŷ T \ki with the existing annotation Y T k
i from platform k, generating the pseudo-complete annotation,
Y Ti =[Y T k i ; Ŷ T \k i ] = [p T 1 ; · · ·;pTnloc+ntrans ], (5)
where each entity is from either Y T k i or Ŷ T \k i . Y Ti is constructed to cover the complete tag set T , illustrated in Figure 2. For local training, θkt is trained with the pseudo-complete annotation Y Ti instead of Y T k
i , with the loss of (3).

heading: 3.3 Instance Weighting
text: With (5), θkt is expected to be trained with {Xi,Y Ti }Nki=1 during local training. Let yl be the lth element of Y Ti , which can be categorized as
Clozapine, Drug, Drug, akathisia, ADR, with schizophrenia, Disease
akathisia, ADR
1) extract
Clozapine, Drug, schizophrenia, Disease
Clozapine, Drug, akathisia, ADR, schizophrenia, Disease
2) combine with
either yl ∈ Y T k or yl ∈ Ŷ T \k
i . The training loss can be decomposed as,
LT = LT k + LT \k (6)
= 1
Nk
Nk∑
i=1
LT ki + 1
Nk
Nk∑
i=1
LT \ki , (7)
where,
LT ki =− ∑
yl∈Y T ki
logPθkt (yl|X,yT1 ,· · ·,yTl−1), (8)
LT \ki =− ∑
yl∈Ŷ T \k
i
logPθkt (yl|X,yT1 ,· · ·,yTl−1). (9)
{LT k}Kk=1 represents training with the local annotations of T k. The knowledge learnt from LT k will be transferred to other platforms where annotations of T k is not available. Correspondingly, {LT \k}Kk=1 represents how platform k can benefit from knowledge of T \k that is transferred from the other platforms where annotations for T \k are available. For platform k, the model is expected to benefit from the knowledge learnt with {LT k ′ }k′ ̸=k, regarding entity types that are not locally annotated (T \k), so that it can identify entities of T \k via training with LT \k . With this perspective, we denote {LT k}Kk=1 and {LT
\k}Kk=1 as the source and target loss, respectively, in terms of the direction of knowledge transfer.
To facilitate the knowledge transfer across platforms discussed above, we propose a weighting mechanism for the training instances of the source loss {LT k}Kk=1, so that the knowledge learnt from the source loss can be more transferable for the
target loss {LT \kt }Kk=1. Specifically, we want to upweight instances that are more beneficial for the training in other platforms and vise versa. Formally, we rewrite the source loss as,
LT k = 1 Nk
Nk∑
i=1
wki,t × LT k i , (10)
where wki,t = 1 reduces to (7). w k i,t is the weight for the ith sample for platform k at the tth federated learning round, measuring how the knowledge from training with LT ki (source) is transferable for the target loss in other platforms, i.e., {LT \k}k′ ̸=k (target). For conciseness, we omit the subscript t that denotes the number of federated learning round in presenting the loss functions, but only showing it for the weight wki,t.
The question remaining is how to measure the transferablility of knowledge learnt from LT ki in the federated learning setting. Since the federate learning is a privay-preserving framework that only allows communicating model updates between the server and platforms, we define wki,t according to the gradient similarity between the source and target loss. Specifically, for the ith sample of platform k, we first compute the gradients of its source loss and mean of the target loss from other platforms, which we denote as gsrci and g tgt, respectively,
gsrci = ∂LT ki ∂q , gtgt = ∑
k′ ̸=k
∂(LT \k) ∂q , (11)
q is the prompt embeddings as introduced in Section 3. wki,t is updated with the cosine similarity
between the two gradients,
wki,t+1 = α·wki,t+(1−α)· < gsrci , g tgt >
||gsrci ||2||gtgt||2 , (12)
where α is a momentum value. < ·, · > denotes the dot product and || · ||2 is the L2 norm. wki,t is computed before local training (Algorithm 1). For platform k, we save gsrci locally and upload the gradient of the target loss LT \k to the server for computing gtgt. gtgt is computed on the server side, then downloaded to each platform for updating wki,t with (12). We further elaborate the procedures of updating wki,t in Algorithm 2. Note that updating wki,t does not involving training of the NER model and wki,t is not shared to the server or other platforms. We should also notice that the above uploading and downloading of gradients introduce additional communication cost. With such concern, we only compute gradients with respect to q ∈ RNq×d (as in (11)), which has only several thousand parameters (Section 3), inducing only minor communication cost. We use q to calculate the gradient similarity, because q is shared by each pretrained layer in BART (Section 3), thus it should correspond to the general information regarding prompt tuning.

heading: 4 Related Works
text: NER with Heterogeneous Tag Sets. Greenberg et al. (2018); Beryozkin et al. (2019) investigate on training over NER datasets with heterogeneous tag sets. However, they assume these datasets are available in a centralized location. Such an assumption is not practical in training with clinical data, for which privacy preservation is of primary concern (Hassan Mahlool and Hamzah Abed, 2022). Additionally, they do ot explicitly consider the differences in data distribution for the text from different datasets. Our work is orthogonal to these works, since we assume decentralized training, i.e., federated learning, where we account for the issues of privacy and communication costs that do not exist in training with centralized datasets. Federated Domain Adaptation Peng et al. (2019) is the first work studying domain adaptation for federated learning. Recently, Hong et al. (2021) further studies the fairness and debasing problem in federated domain adaptation. These works adopt a discriminator module for adversarial domain adaptation, which increases the communication cost of
federated learning. Yao et al. (2022) studies federated domain adaptation via sharing statistics of data distributions of the local platforms. However, such an approach may be vulnerable to membership inference attacks (Shokri et al., 2017), resulting in data leakage, thus may not be applicable to clinical data for which data privacy is the primary concern. Additionally, these work only consider the task of image classification. Our work studies federated domain adaptation for clinical NER. Note that federated domain adaptation is different from federated learning with non-IID (Independent and Identically Distributed) data (e.g.,, (Li et al., 2020)). The latter focus on the problem with slow convergence or diverged results in aggregating with updates from non-IID data. Instead, we targets at effectively transferring knowledge across platforms/domains, so that each platform can benefit from knowledge of locally unannotated entity types transferred from other platforms. Federated Learning for NER. Ge et al. (2020) presents a pilot study of federated learning for clinical NER. Zhao et al. (2021) introduces adversarial training to solve the adversarial attack problem for federated NER. One of the major problems is that these approaches require sharing or communicating the whole NER model (or its encoder) between the server and platforms of federated learning. This will induce huge communication cost in training with the recent Pretrained Langauge Models (PLMs) (Kenton and Toutanova, 2019; Lewis et al., 2019), i.e., containing hundreds of millions of parameters. In this work, we study using a promptbased pretrained NER model (Chen et al., 2022) for our federated learning, thus only communicating prompt-related parameters. This significantly reduces the communication cost compared to fine tuning all the pretrained parameters. Further, different from Ge et al. (2020); Zhao et al. (2021), we focus on federated domain adaptation that efficiently transfer knowledge among platforms of different domains. (Wu et al., 2021) investigates knowledge distillation in federated learning with NER, but is not targeting the federated domain adaptation problem as in our setting.

heading: 5 Experiments

heading: 5.1 Baselines and Ablations
text: We first compare with the classic adversarial domain adaptation with (Ganin et al., 2016), and two more recent works of federated domain adapta-
tion (Peng et al., 2019; Hong et al., 2021). Note that these methods are originally designed for image classification. We re-implement them with our NER model, i.e., LightNER, for comparison. Please refer to Appendix A for details.
Note that these approaches generally require an additional domain discriminator for adversarial domain matching. Such discriminator is trained and communicated along with the NER model. This introduces additional communication cost, as with uploading and downloading the gradients of q in Section 3.3. In the Appendix A, we compared the communication cost of our instance weighting with q with that of the discriminator. Our communication cost is lower, while achieving better performance as in Table 1 and 2.
We denote training with Algorithm 1 as Ours. For the ablation study, we consider: (i) Ours w/o distill&weight. This is to train the LightNER model without distillation in Section 3.2 and instance weighting in Section 3.3. Specifically, the model is trained with only the local annotation Y T k instead of (5) , and wti,k is always set to 1. (ii) Ours w/o weight. It trains the NER model with (5) (as in Ours), while setting wti,k = 1, i.e., no instance weighting. Please refer to the appendix C for implementation details.

heading: 5.2 Experiments with OntoNote 5.0
text: Before evaluating with clinic data, we first demonstrate our method with OntoNote 5.0, a classic NER dataset of 18 entity types (|T | = 18), with data from six domains: nw, tc, wb, bn, bc and mz. We have the number of platforms K = 6, with each platform representing data of a different domain. To simulate the heterogeneous tag sets, we assume the training data of each domain/platform is annotated with 3 entity types (|T k| = 3), which are randomly sampled from the 18 entity types without replacement. For OntoNote 5.0, we study the challenging scenario of federated domain adaptation that each entity type is only annotated in one of the six platforms, i.e., T k1 ∩ T k2 = ∅, for k1, k2 ∈ {1, · · · ,K} and k1 ̸= k2. We randomly sample five time and report the F1 score for each domain, averaged over different samplings. For each domain, the F1 score is computed via evaluating the global model on the testing dataset of this domain with all the 18 entity types.
Table 1 shows the resulting F1 score with OntoNote 5.0. Our method outperforms the base-
lines with a large margin. Instead of communicating domain discriminators as baselines, we communicate the gradients of prompt embeddings, which has a smaller size (Appendix A). Additionally, the performance gain from Ours w/o distill&weight to Ours w/o weight shows the effectiveness of our distillation with pseudo-complete annotation (Section 3.2), which allows the NER model being trained with the complete tag set during local training. Similarily, the the performance gain from Ours w/o weight to Ours validates the usefullness of our proposed instance weight mechanism. Both of these techniques contribute to the superior performance of our trained NER model.

heading: 5.3 Experiments with the Clinic Datasets
text: As mentioned in Section 1 and 5.2, the evaluation of our NER model requires testing data of different domains with complete tag sets. However, existing public clinic datasets are generally created with different annotation schemes. For example, datasets may be annotated with different tags sets (Beryozkin et al., 2019; Karimi et al., 2015), and even the same entity type can have various definitions in different datasets (Karimi et al., 2015). Such a lack of consistent annotations for clinic data of different domains poses challenges to the evaluation of our considered setting. Broadly speaking, this also add to the difficulty in studying general transfer learning problems with clinic NER. For instance, the classic domain adaptation (Long et al., 2015) generally involves transferring knowledge from a labeled source domain to an unlabeled target domain. The resulting model is evaluated with testing data of the target domain, annotated with the same classes/entity types as in the source domain, i.e., requiring consistent annotation for data of the source and target domain, which is hardly fullfilled with public datasets when dealing with clinic NER.
To solve this problem, we take three clinic datasets: CADEC (Karimi et al., 2015), ADE (Gurulingappa et al., 2012) and SMM4H (Weissenbacher et al., 2019), which contains text from three distinct text domains, i.e., formal costumer report, medical case report and casual tweets, respectively. We provide some samples of the three dataset in the Supplymentary data. These datasets are originally annotated with different tag sets. To have consistent annotation across domains. We re-annotate ADE and SMM4H with the tag sets defined in CADEC (with the largest tag set). As
a result, the three datasets are consistently annotated with the same tag set of 5 entity types, T = {Drug,ADE,Disease,Finding, Symptom}, as defined in (Karimi et al., 2015). In Appendix D, we also elaborate our annotation procedure and dataset statistics. In simulating our setting of federated domain adaptation with the above datasets, we set the number of local platform K = 3. Each platform holds text data of a different domain/dataset. Unlike OntoNote 5.0, we consider a more flexible and practical scenario that allows overlapping among tags set of different platforms. Please refer to Appendix D on the tag sets of annotation for each local platform in experiments.
Table 2 shows the results of federated domain adaptation with our clinic datasets. Our method has the highest F1 score averaged over the three considered datasets/domains. Among the three client datasets, CADEC is larger and more diverse than ADE and SMM4H. Thus, CADEC may contain samples that are quite different from those in ADE and SMM4H, and knowledge learnt with such samples may not be transferable for the training of ADE and SMM4H. From our weighting mechanism (10), such samples can be downweighted during training to facilitate knowledge transfer across platforms. Since such downweighted samples may be important for the local training with CADEC,
the improvement for CADEC with our weighting mechanism is slightly smaller than that on the other two clinic datasets. However, we should note that our proposed method can consistently provide improvement over the ablations for different datasets. Table 2 also shows that our annotations for ADE and SMM4H are meaningful, and can be leveraged for the training of existing advanced NER model (Chen et al., 2022). To faciliate future research, we have released our annotated clinic datasets†.

heading: 5.4 Hyperparameter Analysis
text: Let η be the percentage of trainable parameters in the NER model, which is proportional to the communication cost during federated learning. In order to investigate the relation between the communication cost of federated learning and the model performance, we vary the value of η and plot η with the averaged F1 score on OntoNote 5.0 in Figure 3 (a). η is varied by changing the hidden dimension h of the projection on q, explained in Appendix B. Results in Figure 3 (a) shows that, when η is not large (η ≤ 10), the model performance can be improved with larger communication cost (larger η). However, when the value of η gets large enough (e.g., η ≥ 10), the model may overfit to the domain specific information of each client during local training, hindering the further improvement of model performance.
Figure 3 (b) shows the F1 score on OntoNote 5.0, with varing values of Eloc, i.e. epoches of local training per round. All the points share the same communication cost, with the same η and communication rounds for federated learning. The model performance generally improves with longer local training (larger Eloc). We should note that increasing Eloc corresponds to larger computation
†https://github.com/RayWangWR/ClinicDataset
cost in local training. The performance gets saturated when Eloc get too large, i.e., Eloc ≥ 2, which indicated that the local training may have reached convergence after 2 epoches.

heading: 6 Conclusion
text: In this work, we study the problem of federated domain adaptation for clinic NER. We consider the practical setting with heterogeneous tag sets for different platforms of federated learning. To cope with the heterogeneous tag sets and facilitate knowledge transfer among different platforms, we propose distillation with pseudo-complete annotation and an instance weighting mechanism. In addition, we will release two re-annotated clinic datasets for our considered setting. In experiments, our trained NER model show superior performance in the considered setting.

heading: 7 Limitations
text: Our work is base on the existing sequence-tosequence NER model, since its way of decoding has been shown effective for knowledge transfer between different classes (Chen et al., 2022). However, it might also be valuable to consider other token-classification-based or CRF-based (Sutton et al., 2012) NER models. Especially, it would be interesting to employ the existing CRF-based distillation method (Wang et al., 2020b) to cope with the problem of heterogeneous tag sets for NER.

heading: 8 Acknowledgements
text: This research was supported by ONR N00014-18-12871-P00002-3. The student involved was also supported by Adobe gift research funding. We would like to thank the anonymous reviewers for their insightful comments. Moreover, we want to thank Billy I. Kim for his dedicated efforts in screening the annotations.

heading: B Details of Prompt Implementation
text: We following (Chen et al., 2022) in implementing the prompt in Section 3. Generally speaking, (Chen et al., 2022) insert an array of key embeddings and value embeddings into the self-attention module of each transformer layer in BART (Lewis et al., 2019). The inserted key embeddings and value embeddings are denoted as ΦK ∈ RNq×d and ΦV ∈ RNq×d, respectively. Let X l be the input of a transformer layer in BART. The selfattention module first projects X l into embeddings of the key (K l), query (Ql) and value (V l),
K l = X lWK ,Ql = X lWQ,V l = X lW V
(13) where WK ,WQ,W V ∈ Rd×d are the project matrices. The self-attention output with inserted ΦK and ΦV can be computed as,
outputl = softmax( Ql[K l;ΦK ]
⊺ r
d )[V l;ΦV ]r
(14) where outputl denote the output from selfattention. [; ]r denotes row concatenation. ΦK and ΦV are projected from the prompt q ∈ RNq×d in Section 3,
[ΦK ;ΦV ]c = W l 2Tanh(W l 1q) (15)
where [; ]c denotes column concatenation. Tanh is the tangent activation. W l1 ∈ Rd×h and W l1 ∈ Rh×2d are two trainable linear projections for a transformer layer. h is the hidden dimension, controlling the size of trainable parameters.

heading: C Experiment Details
text: In the experiments, we show results with Nq = 10, d = 768 and h = 400. With such configuration, the trainable parameters (those need to be communicated) only takes up 7.04% (η = 7.04) of the model size, significantly reducing the communication cost compared to finetuning the full model. The model is locally trained for 1 epoch before being upload for aggregation, i.e., Eloc = 1 (Section 2.2), and train with 25 rounds of communication. We fix the pretrained BART parameters in LightNER, only training and communicating the trainable parameters for federated learning. Our model is trained with learning rate 3e-5 and batch size 8. We empirically set the momentum value α = 0.9. We train with a single GPU with pytorch 1.7.0 and python 3.8. For the weights of aggregation in equation (1), {mk}Kk=1, we initially tried with FedAvg
(Wang et al., 2020a) that set mk as proportional to the size of the dateset in its corresponding domain. However, we found this will lead to inferior results for platforms whose dataset is small in size. Therefore, we set the weights {mk}Kk=1 as uniform.

heading: D The Clinic Datasets
text: The labeling procedure: We annotate the text corpous of ADE (Gurulingappa et al., 2012) and SMM4H (Weissenbacher et al., 2019) with a tag set of 5 entity types, i.e., T = {Drug,ADE,Disease,Finding, Symptom}, following the definition as in the original paper of CADEC (Karimi et al., 2015). Following (Gurulingappa et al., 2010),we have two annotators that can discuss on the disagreement. We split the text of ADE (Gurulingappa et al., 2012) and SMM4H (Weissenbacher et al., 2019) into batches of 100 sentences. The annotators will work on streaming of batches, and annotating each batch takes about an hour. To ensure the quality of the resulting annotation, we also include a medical student from a clinical institution, in addition to the two annotators, to decide on sample for which the two annotators are not confident. The medical student and the two annotators are all student volunteers, who are also contributing to the methodology and experiments of this research project and credited with their names included in the paper author list. Table 3 show the statistics of our annotations, re-
garding the number of sentences and identified entities. We have also removed some of the duplicated sentences in SMM4H.
Simulating heterogeneous tag sets for different platforms: As in Section 5.3, our experiments with the clinic datasets consider three platforms for federated learning. During the experiments, we specify different sets of annotated entity types (T k) for different platforms to simulate local training with heterogeneous tag sets. For instance, if T k is specified as annotated in platform k, then annotations of T \k will be ignored in this platform. {T k}Kk=1 are specified such that each platform contains at least one annotated entity types whose annotations are not available in the other platforms. Formally, for each platform k, there exist at least one s ∈ T k, s.t., s /∈ T k′ , k′ ̸= k. In this way, we simulate a practical scenario that each platform will have its unique contribution to the federated learning system, via enabling the global model to recognize at least one entity types whose annotations are only available in this platform. Such a setting is based on the consideration that including more platforms in the federated learning system may increase the risk of backdoor attack (Bagdasaryan et al., 2020) and privacy leakage (Li et al., 2021). Therefore, it is realistic that a platform is allow to participate in federated learning only if it can make unique contributions to the global model, i.e., enabling the global model to recognize entity types that are not
annotated in other platforms. Additionally, since there are 3 platforms, we allow each entity types to be annotated in at most 2 platforms. This is because it is less necessary for knowledge of a certain entity type to be transferred across platforms, if all the three platforms have already had its annotation.
As in Section 5.3, we experiment with 3 platforms (K = 3) using the clinic datasets, with text of each platform being from a unique clinic dataset. In determining the T k for each platform, we first randomly (uniformly) sample three different entity types (Drug, ADE and Disease as an example) from T , one for each platform. Each of the sampled entity types is specified as uniquely annotated in its associated platform. Then, for each of the rest of the entity types, denoted as s, (s ∈ {Finding, Symptom} in this example), we first randomly decide whether the it is annotated in n ∈ {1, 2} platforms, with a bernoulli distribution of probability 0.5 for each case. Then, we randomly (uniformly) sample n platforms, and assume s is annotated within these platforms. We randomly sample 5 sets of {T k}Kk=1 with the above process. Since the three clinic datasets do not come with training and testing splits. We follow (Ge et al., 2020) that randomly sample 10% of the data in each dataset for testing, while the rest is for local training. We have 3 random split per sampled {T k}Kk=1, and run the experiment with each split and sampled {T k}Kk=1. Following (Ge et al., 2020; Chen et al., 2022) We report the average F1 score of all the experiment runs.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: To faciliate future research, we have released our annotated clinic datasets†