Federated Nearest Neighbor Machine Translation
Authors: Yichao Du
Zhirui Zhang
Bingzhe Wu
Lemao Liu
Tong Xu
Enhong Chen
Venue: ICLR 2023 poster (2023)
Keywords: Machine Translation
Federated Learning
Memorization Augmentation

Abstract: To protect user privacy and meet legal regulations, federated learning (FL) is attracting significant attention. Training neural machine translation (NMT) models with traditional FL algorithm (e.g., FedAvg) typically relies on multi-round model-based interactions. However, it is impractical and inefficient for machine translation tasks due to the vast communication overheads and heavy synchronization. In this paper, we propose a novel federated nearest neighbor (FedNN) machine translation framework that, instead of multi-round model-based interactions, leverages one-round memorization-based interaction to share knowledge across different clients to build low-overhead privacy-preserving systems. The whole approach equips the public NMT model trained on large-scale accessible data with a $k$-nearest-neighbor ($k$NN) classifier and integrates the external datastore constructed by private text data in all clients to form the final FL model.  A two-phase datastore encryption strategy is introduced to achieve privacy-preserving during this process.  Extensive experiments show that FedNN significantly reduces computational and communication costs compared with FedAvg, while maintaining promising performance in different FL settings.

Published as a conference paper ICLR 2023
FEDERATED NEAREST NEIGHBOR MACHINE TRANSLATION
Yichao Dult, Zhirui Zhang', Bingzhe Wu', Lemao Liu", Tong Xu and Enhong Chen
Universitt of Science and Technology of China
State Key Laboratory of Cognitive Intelligence *Tencent Al Lab
tt #duyichao@@aall ustc. edu.cn {tongxu, cheneh (thhnnh)@usttee @usthh edu. com
pruuscc1@@mmati com (bingzhewu,
ABSTRACT
To protect user privacy and meet legal regulations, federated learning (FL. is attracting significant attention. Training neural machine translation (NMT) models with traditional FL. algorithms (e.g., FedAvg) typically relies on multi-round model-based interactions. However, is impractical and inefficient for translation tasks due to the vast communication overheads and heavy synchronization. In this paper, we propose novel Federated Nearest Neighbor (FedNN) machine translation framework that, instead of multi-round model-based interactions, leverages roenround memorization- based interaction to share knowledge across different clients and build low-overhead
systems. The whole approach equips the public NMT model trained on large-scale accessible data with k-nearest- neighbor (ANN) classifier and integrates the external datastore constructed by private text data from all clients to form the final FL model. A two-phase datastore encryption strategy is introduced to achieve privacy-preservie show that FedNN significantly reduces computational and communication costs compared with FedAvg, while maintaining promising translation performance in different FL. settings.
during this process. Extensive experiments
1 INTRODUCTION
In recent years, neural machine translation (NMT) has significantly improved translation quality (Bahdanau ct al. 2015 Vaswani et al. 2017, Hassan et al. 2018 and has been widely adopted in many commercial systems. The current mainstream system is first built on a large-scale corpus collected by the service provider and then directly applied to translation tasks for different users and enterprises. However, this application paradigm faces two critical challenges in practice. On the one hand, previous works have shown that NMT models perform poorly in specific scenarios, especially when they are trained the corpora from very distinct domains (Koehn & Knowles 2017] Chu & Wang 2018). The fine-tuning method is a popular way mitigate the effect of domain drift, but brings additional model deployment overhead and particularly requires high-quai high-quality in-domain data provided by users or enterprises. On the other hand, some users and enterprises pose high data security requirements due to business concerns or regulations from the government (e.g., GDPR and CCPA), meaning that we cannot directly access private data from users for model training. Thus, a conventional centralized trainning manner is infeasible these scenarios.
In response to this dilemma, natural way is to leverage federated learning (FL) (Li et al. 2019) that enables different data owners to train a global model in a distributed manner while leaving raw private data isolated to preserve data privacy. Generally, standard FL workflow, such as FedAvg (McMahan et al.) 2017), contains multi-round model-based interactions between server and clients. At each round, the client first performs training on the local sensitive data and sends the model update to the server. The server aggregates these local updates to build an improved global model. This straightforward idea has been implemented by prior Passban et al. 2022) that directly apply FedAvg for machine translation tasks
works (Roosta et al. 2021

---

Published as a conference paper at ICLR 2023
and introduce some parameter pruning strategies during node communication. Despite this, multi-round model-based interactions are impractical and inefficient for NMT applications. Current models heavily rely on deep neural networks as the backbone and their parameters can reach tens of millions or even hundreds of millions, bringing vast computation and communication overhead. In real-world scenarios, different clients (i.e., users and enterprises) usually have limited computation and communication capabilities, making it difficult to meet frequent model training and node communication requirements in the standard FL workflow. Further. due to the capability differences between clients, heavy synchronization also hinders the efficacy of FL workflow. Fewer interactions may ease this problem but suffer from significant performance loss.
Inspired by the recent remarkable performance of memorization- -unnonntuumedd techniques (c.g., the k- k-nearest- neighbor, kNN) in natural language processing (Khande wal et al. 2020 2021 [Zheng al. 2021a b) and computer vision (Papernot & Mcdaniel [0188) Orhan (018), we take a new perspective to deal with above federated NMT training problem. In this paper, we propose novel Federated Nearest Neighbor (FedNN) machine translation framework, which equips the public NMT model trained on large-scale accessible data with a ANN classifier and integrates the external datastore constructed by private data from all clients form the final FL model. In this way, we replace the multi- houldd model- based interactions in the conventional FL paradigm with the one-round encrypted memorization-biae based interaction to share knowledge among different clients and drastically reduce computation and communication overhead.
Specifically, FedNN follows a similar server-client architecture. The server holds large-scale accessible data to construct the public NMT model for all clients, while the client leverages their local private data to yield an external datastore that is collected to augment the public NMT model via &NN retrieval. Based on this architecture, the key is to merge and broadcast all datastores built from different clients, while avoiding privacy leakage. We design a two-phase datastore encryption strategy that adopts an adversarial mode between server and clients to achieve privacy privacy-preserve preserving during the memorization-base one hand, the server builds (AC, -)-encyyption model for clients to increase the difficulty of reconstructing the private text from the datastores constructed by other clients. The KC-encryption model is coupled with the public NMT model to ensure the correctness of ANN retrieval. On the other hand, all clients use shared content-encryption model for local datastore during the collecting process so that the server can not directly access the original datastore. During inference, the client leverages the corresponding content-decryption model obtain the final integrated datastore.
interaction process. On the
We set up several FL scenarios (i.e., Non-lID and IID settings) with multi- English- (En-De) translation dataset, and demonstrate that FedNN not only drastically decreases computation and commu- nication costs compared with FedAvg, but also achieves the ofte-of-the-ard translation performance in the Non-III setting. Additional experiments verify that FedNN easily scales large-scale clients with sparse data scenarios thanks to the memorization- interaction across different clients. Our code is open-sourced on htttts:/// htwp://www..ccco.ol.yca///////aa MT
2 FEDNMT: FEDERATED NEURAL MACHINE TRANSLATION
Current commercial NMT systems are built large-scale corpus collected by the service provider and directly applied to different users and enterprises. However, this mode is difficult to flexibly satisfy the model customization and privacy protection requirements of users and enterprises. In this work, focus more general application scenario, where users and enterprises participate in collaboratively training NMT models with the service provider, but the service provider cannot directly access the private data.
Formally, this application scenario consists of |C| clients (i.e., user or enterprise) and a central server (i.e., (P-1 service provider). The central server holds vast accessible translation data D, {(x,, v() where x' xz. .... x(x1)) and yi (ví v2,, =fy.) ) (for brevity, we omit the subscript here) are text sequences in the source and target languages, respectively. The central server easily train public NMT
2

---

Published as a conference paper ICLR 2023
model fo based on this corpus, where 0 denotes model parameters. For each client c, contains private data
which is usually sparse in practice (i.e., [Drl << (D,) and only accessible to itself. This setting actually falls into the federated learning framework. The straightforward idea is to apply the vanilla FL method (i.e., FedAvg) or its variants (Roosta et al., [021]] Passban et al. (022). Generally, FedAvg contains multi-round model-based interaction updates between server and clients. At each round r, each client e downloads global model for from the server and optimizes it using De- Then the local updates 8% are uploaded to the server, while the server aggregates these updates to form a new model fø--1 via a simple parameter averaging technique: 0++1 = >m m= n
D. = (x y:)
y') !P:],
Thm 6" where 7m denotes the number of data points
in the m-th client's private data, and n is the total number of all training data. However, such FI. workflow is inefficient for the above application scenario because the parameter of NMT models typically reaches tens of millions or even hundreds of millions, bringing vast computation and communication overhead. The system heterogeneity between server and clients, i.e., mismatch of bandwidth, computation resources, etc., also makes it difficult to satisfy frequent updates and communication requirements the standard FL workflow.
3 FEDNN: FEDERATED NEAREST NEIGHBOR MACHINE TRANSLATION
Inspired by the advanced memorization-augue -aggioonedd techniques, e.g., İNN-MT (Khandelwal et al. 202 that has shown the promising capability of directly incorporating the pre- pre-trained NMT model with external knowledge via ANN retrieval, we explore leverage one-round memorization baase interaction rather than multi-round model-based interactions to achieve knowledge sharing across different clients. In this work, we design novel Federated Nearest Neighbour (FedNN) machine translation framework, which extends the promising capability of 6NN-MT in the federated scenario and introduces two-phase datastore encryption strategy avoid data privacy leakage. The whole approach complements the public NMT model built by the central server with kNN classifier and safely collects the local datastore constructed by private text data from all clients to form the global FL model. The entire workflow of FedNN is illustrated in Figure] [ consisting of initialization, one-round memorization-bat interaction and model inference on clients.
3.1 INITIALIZATION
FedNN starts with the public NMT model and encryption models. The central server is responsible for opti- mizing the public NMT model fe with D., Following ANN-MT (Khandelwal et al. 2021), the memorization (also called as datastore) is a set of key-value pairs. Given a sentence pair (Xs, ys E D we gain the context representation fx vs,<ce) in the last decoder layer at each timestep t. The whole datastore M, (K,,, Vo) is constructed by taking the representation (x,,, Ms,ct)) as key and ground-truth yt as value:
M, = (KC,
U (X..y))D,
{(f(xx, Mon), Vya.e E y.).
(I)
Based on M,, the central server further builds K-Enneyptiion model fice(.) that is coupled with the public NMT model. This design is for clients, which aims to increase the difficulty of reconstructing the private text from datastores constructed by other clients. The fic((-) should also satisfy the correctness of ANN retrieval during inference and the detailed K--Eneryption algorithm selection is described Section 3.4 All clients prepare the shared content-encee content-encryption model fcE(.) and corresponding content-decryption model fDE(-), which are applied to the local datastore so that the server cannot directly access the original datastore. The content-eneryption algorithm selection is relatively loose, which is detailed in Section 3.4
3.2 MEMORIZATION- INTERACTION
The entire memorization- based interaction is decomposed into two steps: private memorization construction and global memorization aggregation. The central server broadcasts fo and fxx(() for all clients to build the
3

---

Published as conference paper ICLR 2023
Ciient 1
V
AB
Ag
Federated Nearest Neighbor
uncelo
Server
Passing
R EnEeryptiooi Process
Content Encryption
Private
Public Model
II
SoAft
C Gbobal
@ Privato Memorization Construction
Data
Memorization
Aggregation
Runde
Client 71
Fittaale Sienesizatoon
polax
V-Encryption
Chete
Mannsiiosioon
JAN
Folge
foz
ragerdo
Ki
V
Public Model
LEEeryplion
Content Encryption
hundsfor
Dalor
Private Data
Ragen
6
karos
rp
y
Salix
Fundo
Velum
Dio
20
40
200
i
Initialization
Public Mamoriisatonn
P
Decryplion
Content
100
Ich
Valcom
Da
Paarx
Polpen
Weight Prediiction
Passing Public Mode
Cooa Un
adtop
Query
Public
+00
nunde
Passing Public Model
Final Prediction
fYE
Learning X-lincryption
Figure I: The overall workflow of our proposed federated framework (FedNN).
local encrypted datastore. Specifically, for each client adopt similar construction way Equation to yield the local datastore Maka via De, with difference that f'cE(.) )) used to preserve private information:
MIKE (K//EE Ve)
U (ffc Ve,c), Vye, E ycl-
(X..
order to ensure that above datastore is not explicitly available to the server, we encrypt key-value pairs by fcu(-) before uploading them to the server, formalized as:
fKE = (ICfxa. V.a = U (Je[(fx((f(xx. (e) Mo,r), Vyo. E yet. (x
(3)
Once the central server has received the private memorization from all clients, it directly aggregates all datastores via simple key-value pair concatenation and performs V-encryption operation (i.e., shuffling on key-value pair avoid clients identifying the source of datastore) obtain the global memorization Mg f.K.vn vn,c, which is sent to all clients for model inference. Then each client decrypts the contents the received M fEE. to gain an accessible integrated datastore M fEL.VE
3.3 MODEL INFERENCE ON CLIENTS
For model inference on clients, we follow the adaptive ANN-MT (AK-MT) Zheng et al. 2021a to incorporate fe with K via adaptive ANN retrieval. AK-MT introduces a lightweight Meta-k Network fMeu-k to dynamically determine the number of retrieved tokens to consider at each step, and has promising generalization ability. Thanks to this, could train fMew- with small data in any vertical scenario and then directly apply it to other scenarios. Since the parameters in f are negligible, we ignore the additional training and communication costs of AK-MT in FedNN.
4

---

Published as a conference paper ICLR 2023
Concretely, given the already generated words d<t and source input x, AK-MT augments the probability distribution of t-th target token Y/ via kNN retrieval based on the context representation fo fo(x, δet). It considers a set of possible ks that are smaller than pre-defined K, i.e. h E S where S {0) U {k;
log2 K2 E N, k; I } (k- = 0 indicates ignoring ANN retrieval and only utilizing the public NMT model).
Then K nearest neighbors of the current representation (x, Uet) are retrieved from Mg M according to the squared L2 distance -). The L2 distances from fo (x, Det) to each neighbor (h3, 14) is denoted as d; = d(h;, fo (x, D)c+)) and the count of distinct values top-i is denoted as c₁. The normalized weights of applying different kNN retrieval results are computed as:
PMeta (A) = softmax (Id1,.., dx; Cl Cl.,.., ckl)).
The final prediction probability p (yo x (<1) is weighted ensemble differnt &NN retrieval distributions:
Σ 1m exp (-d (hi, fo (x, Bec))
PKNNN (2d/x, I<c)
T
(4)
p (v6|x, Pcc) =
Σ
PMeta (k.) PK,NN (yrl d<c).
kiES
where T is the temperature to control the sharpness of softmax function.
3.4 ENCRYPTION AND PRIVACY DISCUSSIONS
X-Encryption. In this work, adopt the Product Quantizer (Jégou al. 2011 algorithm to build K-- encryption model, which decomposes the space into Cartesian product of low-dimensional subspaces and quantifies each subspace separately into segment code representations. Further, we map the representation to the above shortcode representation, which cannot be reversed to the original one, further reducing the possibility of reverse-constructing private data. This way also satísfies the correctness of kNN retrieval after encryption. Note that any algorithm that makes representation distorted and irreducible could be adopted in FedNN, such as PCA.
pententtEnnryytionn. We require to generate the ciphertext by content-nneryption model and ensure that it is indistinguishable from the chosen plaintext attacks (Oded, 2004). Since each record of the memorization can be regarded as a string, can be encrypted and decrypted using any asymmetric encryption algorithm, such as Paillier, Elgamal (Gamal 1984) and RSA (Rivest et al. 1978), etc.
Threat Models and Leakage Quantifying. We consider that all clients involved in the training process are semi-honest following prior works (Zhang et al.) 2021 Bonawitz et al. 2016). In this semi-honest setting, each client adheres to the designed protocol but it may attempt to infer information about other participant's input (i.e., memorization). Under this setting, our method has achieved different protection levels for the client and server side. Specifically, for the server side, our mechanism achieves the same protection level to the conventional Public-key cryptography system (e.g.. RSA). Thus, the server cannot obtain any useful information from the encrypted data. For the client side, since the client gets shared information generated by Product Quantizer (ke.ffceJea (Xe; (e)< Uc,,)) from other clients, the aim of this paper is to prevent the shared information from reconstruction attacks (i.e., recovering private text data from the datastore). To this end, we introduce some metrics to quantify the privacy leakage of the shared datastore information (see more details in Section 4.5).
4 EXPERIMENTS
4.1 SETUP
We adopt WMT14 En-De data (Bojar et al.) simulate two typical FL scenarios for model evaluation: 1) the non-independently identically distribution
(014) and multi-domain En-De dataset (Koehn & Knowles 2017)
5

---

Published as a conference paper ICLR 2023
(Non-IID setting) where each client distributes data from different domains; 2) the independently identically distribution (IID setting) where each client contains the same data distribution from all domains. In our experiments, WMT14 and multi-domain En-De dataset are viewed the server's s data and clients' private data, respectively. The multi-domain data provides a natural division for exploring the Non-IID setting, of which we assign IT, Medical, and Law domain data to each of the three clients. For the IID setting, we mix the above domain data and randomly sample the same number of sentence pairs from it for each client. More dataset and implementation details can be found in Appendix A
We compare our method FedNN with several basclines: (i Centralized Model (C-NMT): A standard centralized-training method uses all clients and server data to obtain a global NMT model. (ii) Public Model (P-NMT): A generic NMT model is trained on only server data and used for initializing the client-side model. (iii) FedAvg: A vanilla FL approach (McMa et al. 2017) 7) that iteratively optimizes a global model through multi-round moodll-based interactions. (iv) FT- FTT-Ensemble: We the client's private data to fine-tune P-NMT and ensemble the output probability distributions of all fine-tuned models during inference.
Table I: BLEU score [%] of different methods on clients and server test sets. "A3" refers to the improvement of methods compared with P-NMT. The subscript "1/oc" indicate that the server perform model aggregation after one or infinite epochs (i.c., client model convergence) of client model updates, respectively. The superscript "s" indicates that the server data also involved in model training of FedAvg. "Comm." " and "Comp." refer to communication and computational overhead in "GB" and "FLOPs" respectively.
Client Test
Server Test WMT14
Overall Performance
Cost
Inference Speed
Methods
IT
Law Medical
Client A
Global A
Comm.
Comp.
C-NMT
37..0
49.72
47.40
26.58
44.81
40.25
1.00x
P-NMT
26.62
35.91
30.27
26.63
30.93
29.86
1.00x
FedAvgi
26.99
37.65
32.36
26.63
32.33
+1.40
30.91
+1.05
1.72 x 10t9
388.12
1.00x
FedAvg:
28.26
53.00
45.90
13.45
42.39
+11.45
35.15
+5.30
3.23 10ts
Non- IID
FedAvgio
27.04
38.37
32.32
26.33
32.66
+1.72
31.08
+1.22
7.02 x 10"7
FedAvg..
17.03
47.06
30.61
13.33
31.57
+0.63
27.01
2.85
4.85
7.02 x 10
1.00x
FTT-Ensemble
30.11
38.14
39.15
17.13
35.80
+4.87
31.13
+ +1.28
4.85
7.02 x 10
0.39x
FedNN
35.62
55.57
49.21
22.29
46.80
+15.87
40.67
+10.82
5.08
6.72 10
0.75x
FedAvgì
30.83
43.47
39.22
26.36
37.84
+6.91
34.97
+5.11
1.72 x 10t9
FedAvg:
37.99
54.53
50.23
14.80
47.58
+16.65
39.39
+9.53
388.12
3.23 10t8
1.00x
FedAvg
29.23
39.67
34.49
26.28
34.46
+3.53
32.42
+2.56
7.02 x 10
FedAvg...
34.71
48.68
44.79
16.23
42.73
+11.79
36.10
+6.25
4.85
7.02 x 10
1.00x
F'T-Ensemble
36.74
51.34
47.49
16.42
45.19
+14.26
38.00
+8. 14
4.85
7.02 x 10"
0.39x
FedNN
34.64
54.45
47.98
23.15
45.69
+14.76
40.06
+10.20
5.08
6.72 10
0.75x
4.2
MAIN RESULTS
Table iillstrates the performance of all methods. We observe an average 13.88 BLEU gap on the client test set between P-NMT and C-NMT. Restricted by privacy protection, it is not always possible to access private data for centralized training, Specifically, we evaluate the performance of different FL methods in both Non-IID and IID settings.
attempt build perfooreformance global models using FL techniques.
For the Non-III setting, have the following findings: (i) All FL methods outperform P-NMT the client test set, but show degradation on the server test set. This indicates that FL fuses helpful information from multiple parties, but suffers from varying degree of knowledge conflict and catastrophic forgetting. (ii)
6

---

Published as a conference paper at ICLR 2023
FedAvg is heavily affected by whether the server data is used in training and model aggregation frequeney. When server data is exploited during the training of FedAvg, the overall performance of FedAvg is similar to P-NMT, but FedAvg significantly improves performance on client test set without involving the server data, In addition, the performance gain has positive correlation with the size of the client dataset (Law> Medical>IT), since FedAvg utilizes the size of different datasets as weights to aggregate client models. For the aggregation frequency, FedAvg1 is much better than FedAvg and more details can be found in Appendix C.2 We find that frequent aggregation significantly reduces the parameter conflicts between different models, but it brings high communication cost. (iii) FT-Ensemble is better than FedAvg indicating that the fusion of output probabilities leads to less knowledge conflict compared with model aggregation. (iv) FedNN achieves an average 4.41/.99 BLEU score improvement on the client test compared to FedAvgı and C-NMT respectively, and maintains a competitive performance on the server test set. It demonstrates the effectiveness of FedNN in capturing client-side knowledge by memorization and integrating with P-NMT. (v) Although FedNN slightly increases inference time, it not only improves translation quality, but also significantly reduces communication and computation overhead compared with other FL baselines, which is tolerable for clients.
For the IID setting, we have some different findings: (i) Some FL methods that do not leverage server data in their training process (i.e.. FedAvgı. FT-Ensemble and FedNN) outperform C-NMT on the client test set. The reason is that there is no statistical data heterogeneity among clients. resulting in fewer parameter conflicts and less conflict of probability outputs. (ii) The performance of FedNN is slightly weaker than that in the Non-IID setting. It demonstrates that the benefit of the memorization-basee interaction is more significant when the data distribution more heterogeneous.
Overall, FedNN shows stable performance with less communication and computational overhead in Non-IID and IID settings, which verifies the practicality of memorization- based interaction mechanisms. More results and analysis are shown in Appendix B
4.3 THE IMPACT OF CLIENT'S S NUMBER
We further verify the effectiveness of FedNN on a larger number of clients. We adopt the number of clients ranging from (3, 6, 12, 18) for quick experiments The detailed results are shown Figure 2
Comparisons with FL Methods. As the number of clients increases, we observe that: (i) Both FedAvgı and FT-Ensemble show varying degrees of performance degradation on the client test sets, especially for FT-Ensemble. We conjecture that the limited local data cannot support the training of local models and retain most of the knowledge of P-NMT. (ii) FedNN outperforms FL baselines on both private and global test sets for the Non-IID setting, while for the IID setting it maintains a similar performance to FedAvgi i on private test sets and keeps a higher global performance. These results show that FedNN, benefiting from the memorization based interaction, could quickly scale to large-scale client scenarios and avoid performance loss due to insufficient local private data. The more analysis of FL methods is described in Appendix E Comparisons with Personalized Methods. We also compare FedNN with the personalized methods, including FT (fine-tuning P-MMT with only local client-side data) and AK-MT (constructing a datastore with only local client-side data and decoding with assisted Meta-k network). AK-MT and FT perform similarly, as AK-MT is able to capture the personalized knowledge by local memorization. The performance of both AK-MT and FT tends decrease in the Non-IID setting the number of clients increase, while FedNN hardly decreases. For the IID setting, although the performance of all methods degrades, FedNN still achieves the best performance on all clients test sets. It because that the global memorization capture more similar patterns than the local memorization to assist in inference.
'Due to the limited resources in our experiments, there are no more domains to ensure the Non-IID setting when the number of clients inereases. Thus, we directly separate the Non-IID and IID data distributions with the ratio of (1, 1, 1, -.). Note that the Non-IID setting here not the strictly one, but it is worth exploring.
7

---

Published as conference paper ICLR 2023
Methods:
FedAAg
PT Ersemble
FedNN
Personalized Methods:
MT
Nodlos
60
60
60
18 Nodes
30-
50-
50-
50-
No) Setting 1SUI Sooe
S0LE11 Scute
55.511 Scrre
DUL Sceen
-0-
1.00 Maticl WMTT1
1,3 Medice
WMM111
Menical WXIT14
Mohcal WMT34
NNodess
12Nodds
RUA Soor
NUEU Sour
MURU Soors
11E00 Soor
ID
20
20-
10
Law
Malial WMT14
10
17
Law
Medical WMT14
10
17
Lane
Medical WM774
10
17
Law
Molkal WMT14
Figure 2: The translation performance of FL and personalized methods when the number of clients increases.
FedAvg:
FT-Ensemble
FT
AK-MT
FedNN
Client IT
Client Law
Client Medical
Server WMI14
55
36-
50-
22
B1.EU Seare
50
28-
45
40
16
40
20
35
30
10
Figure 3: The impact of data distribution heterogeneity for different FL and personalized methods.
4.4 THE IMPACT OF DATA HETEROGENEITY
To further investigate the effect of data heterogeneity between three clients FL performance, adopt a mixed ratio E {0, 0.2, 0.4, 0.6, 0.8} to construct the data distribution that we want: we randomly take a proportion of from each domain to construct the IID dataset, and then remain domain data is mixed with one-third of this IID dataset to form the final data distribution. As - O, partitions tend to be more heterogeneous MoDn-IID,, and conversely, the data distribution more uniform. As shown in Figure 3 the performance of personalized methods (FT and AK-MT) is degraded as data heterogeneity decrease, which is caused by the reduction of available domain-specifie data the client. FT-Ensemble also decreases across all client test sets and is worse than FT, while FedAvg, shows opposite performance trends between Law and IT, Medical. This is because when FedAvg aggregates, the model weight of each client is proportional to the data size, and as increases, the data size between clients tends from [D > (Deedioull > [Drrl to equally to 3 ([D + (DMecicall + 'Drrl). Our FedNN maintains stable and remarkable performance across all client test sets and significantly outperforms other methods in the server test set. It indicates that the memorization-ban -based interaction mechanism could capture and retain the knowledge of all clients, avoiding the knowledge conflict based on traditional model-based interaction.
4.5 QUANTITATIVE ANALYSIS OF PRIVACY
We quantify the potential privacy-leaking risks of global memorization. Since all clients obtain the public NMT model, they could utilize their own datastore train reverse attack model to reconstruct the private data in
8

---

Published as a conference paper at ICLR 2023
global memorization. In this experiment, we task one client (e.g., IT) as the attacker and others as the defenders (e.g., Medical and Law). The reconstruction BLEU (Papineni et al.
scores are used to evaluate the degree of privacy leakage. The more expermeenal details are shown Appendix [D] As illustrated in T whether the input is an unencrypted key or key encrypted by fKF, the threat model has very low scores in all defenders, especially for the recall score, meaning that it is difficult to recover and identify valuable information from global memorization. Furthermore, the SIEE increases the difficulty of reconstructing the private text from the memorization constructed by other clients. We also provide some case studies in Appendix D.4 which better help qualitatively assess the safety of FedNN.
Table 2: The reconstruction BLEU/Precision(PP) (etprrccisioo(PPecaaalRRR (R) score [%] of the attack model.
Metric
Datastore
IT MedMMllaal
LL
Medical-ITT
Med
Law-II
Law-cMeiical
BLEU
(C.
8.21
5.09
9.90
6,16
7.33
8,30
(K/S.V)
6.52
4.27
7,88
5.58
6.35
6.86
P/R/FI (C.
14.55/2 90/4.4
43/223
12.18/7.5/9.31
23.18/11.65/1555
11.15/7 00/.004//..
9,75/4,85/6
(K/ V)
01/33/2.0//399
41.26/5.28/9.66
11.88/7.43/91144
12.18/7.53/9.31
11.81/6.35/8.26
9.62/4.04/5.69 69
5 RELATED WORK
The FL. algorithm for deep learning (McMahan et al. 2017) is first proposed for language modeling and image classification tasks. Then theory and framework of FL are widely applied many fields, including computer vision (Lim et al. 2020), data mining (Chai et al. 2021), and edge computing (Ye et al. 2020). Recently, researchers explore applications of FL srivacy-preserving NLP, such as next word prediction (Hard et al. 2018: Chen et al. 2019), aspect sentiment classification (Qin et al. (2221), relation extraction (Sui et al. 2021), and machine translation (Roosta et al.) 202T Passban et al. [ 2022). For machine translation, previous works directly apply FedAvg for this task and introduce some parameter pruning strategies during node communication. However, multi-round model-based interactions are impractical and inefficient for NMT because of the huge computational and communication costs associated with large NNT models. Different from them, we design an efficient federated nearest neighbor machine translation framework that requires only one-round memorization interaction to obtain a high-quality global translation system.
Mlemorization-augenntt methods have attracted much attention from the community and achieved remark- able performance on many NLP tasks, including language modeling (Khandelwal et al.) 2020: He et al. 2021) named entity recognition (Wang et al. 2022). few-shot learning with pre-trained language model (Bari al. 2021 Nie et al. 2022). , and machine translation (Khandelwal et al., [021 Zheng et al.. [021 by Wang
et al. 2021 Du et al. [2022). For the NMT system, Khandelwal et al. (2021) first propose ANN-MT, a
simple and efficient non- non-parametric approach that plugs kNN classifier over large datastore with traditional
NMT models (Vaswani et al., 2017 Zhang et al.) 2018a b. Guo et al. 2020: Wei et al. 2020) to achieve
significant improvement. Our work extends the promising capability of ANN-MT in the federated scenario
and introduces two-phase datastore encryption strategy to avoid data privacy leakage.
6 CONCLUSION
In this paper, we present a novel federated nearest neighbor machine translation framework to handle the federated NMT training problem. This FL framework equips the public NMT model trained on large-scale accessible data with a kNN classifier and safely collects all local datastores via a two-phase datastore encryption strategy to form the global FL model. Extensive experimental results demonstrate that our proposed approach significantly reduces computational and communication costs compared with FedAvg, while achieving promising performance in different FL settings. In the future, we would like explore this approach on other effectiveness of our method on larger number of clients, such hundreds of clients with more domains.
tasks. Another interesting direction is to further investigate the
9

---

Published as conference paper ICLR 2023
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for helpful feedback early versions of this work. We appreciate Wenxiang Jiao, Xing Wang. Longyue Wang and Zhaopeng Tu for the fruitful discussions. This work was done when the first author an intern Tencent AI Lab and supported by the grants from National Natural Science Foundation of China 612.62222,3,, 62072423), and the USTC Research Funds of the Double First-Class Initiative (No. 5000000)10002009 Zhirui Zhang, Tong Xu and Enhong Chen are the corresponding authors.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In EMNLP, 2015.
M Saiful Bari, Batool Haider, and Saab Mansour. Nearest neighbour few-shot learning for cross-lingual classification. ArXiv, abs/21 100.2211,, 2021.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamehyna. Findings of the 2014 workshop on statistical machine translation. In WMT@ACL, 2014.
Kallista A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. CoRR, abs/ 1.0422, 2016. [RLttt/://aaxx wwwp:x/wwwrg/aa///ooaoaaa/b org. a61l
611. 1611.04482
Di Chai, Leye Wang, Kai Chen, and Qiang Yang. Secure federated matrix factorization. IEEE Intelligent 3ystems,36:11-20 2021.
Mingqing Chen, Ananda Theertha Suresh, Rajiv Mathews, Adeline Wong, Cyril Allauzen, Franccoise Beaufays, and Michael Riley. Federated learning of n-gram language models. In CoNLJ,, 2019.
Chenhui Chu and Rui Wang. survey of domain adaptation for neural machine translation. In COLING, 2018.
Yichao Du, Weizhi Wang, Zhirui Zhang, Boxing Chen, Tong Xu, Jun Xie, and Enhong Chen. Non-parametric domain adaptation for end-to-end speech translation. In EMNLP, 2022.
Taher El Gamal. public key cryptosystem and a signature scheme based on discrete logarithms. IEEE Trans. Inf. Theory, 31:469 4722, 1984.
Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. Incorporating bert into parallel sequence decoding with adapters. In NewrlPS, 2020.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. ArXiv, abs/1811.03360 2018.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William D. Lewis, Mu Li, Shujic Liu, Tic-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. Achieving human parity on automatic chinese to english news translation. ArXiv, abs/ 803.803.500 005.05567, 2018
10

---

Published as conference paper ICLR 2023
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. Efficient nearest neighbor language models. In EMNLP, 2021.
Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33:117-128, 2011.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7:535-547, 2021.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. machine translation. In ICLR, 2021.
Nearest neighbor
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. CoRR, abs/1 412.6980, 2015.
Philipp Kochn and Rebecca Knowles. Six challenges for neural machine translation. In NMT
Tian Li, Anit Kumar Sahu, Ameet S. Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37:50-60, 2019.
Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Tao Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 22:2031-2063, 2020.
H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication- efficient learning of deep networks from decentralized data. In AISTATS, 2017.
Feng Nie, Meixi Chen, Zhirui Zhang, and Xuan Cheng. Improving few-shot performance of language models via nearest neighbor calibration. ArXiv, abs/2212.00211,, 2022.
Goldreich Oded. Foundations of cryptography: Volume 2, basic applications. 2004.
A. Emin Orhan. A simple cache model for image recognition. ArXiv, abs/ 1005180.0000 805.08709, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. NAACL, 2019.
Nicolas Papernot and Patrick Mcdaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. ArXiv, abs/1803. 1otp//803.0475, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Peyman Passban, Tanya Roosta, Rahul Gupta, Ankit R. Chadha, and Clement Chung. Training mixed-domain translation models via federated learning. In NAACL, 2022.
Han Qin, Guimin Chen, Yuanhe Tian, and Yan Song. Improving federated learning for aspect- based sentiment analysis via topic memories. EMNLP, 2021.
Ronald L. Rivest, Adi Shamir, and Leonard M. Adleman. A method for obtaining digital signatures and public-key cryptosystems. Commun. ACM, 21:120-122 1978.

---

Published as conference paper ICLR 2023
Tanya Roosta, Peyman Passban, and Ankit R. Chadha, Communication--lffciiine federated learning for neural machine translation. ArXiv, abs/21 12.06113,,, 2021.
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh, Bleurt: Learning robust metrics for text generation. In ACL, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. ArXiv,
2016.
Dianbo Sui, Yubo Chen, Kang Liu, and Jun Zhao. Distantly supervised relation extraction in federated settings. In EMNLP, 2021.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Dongqi Wang. Hao-Ran Wei, Zhirui Zhang. Shujian Huang, Jun Xie, Weihua Luo, and Jiajun Chen. Non- parametric online learning from human feedback for neural machine translation. In AAAI Conference on Artificial Intelligence, 2021.
Shuhe Wang, Xiaoya Li, Yuxian Meng. Tianwei Zhang, Rongbin Ouyang, Jiwei Li, and Guoyin Wang. knn-ner: Named entity recognition with nearest neighbor search. ArXiv, abs/2203.17103, 2022.
Hao-Ran Wei, Zhirui Zhang, Boxing Chen, and Weihua Luo. Iterative domain-repaired back -translaattio. In EMNLP, 2020.
Yunfan Ye, Shen Li, Fang Liu, Yonghao Tang. and Wanting Hu. Edgefed: Optimized federated learning based on edge computing. IEEE Access, 8:209191-209198, 2020.
Junpeng Zhang, Mengqian Li, Shuiguang Zeng, Bin Xie, and Dongmei Zhao. A survey on security and privacy threats federated learning. In 2021 International Conference on Networking and Network Applications, NaNA 2021, Lijiang City, China, October 29 Nov. 1, 2021, pp. doi: 10.1109/NNA3336422000 00002.. URL
IEEE, 2021. 1. 1109 N 2021
00062
Zhirui Zhang, Shujie Liu, Mu Li, M. Zhou, and Enhong Chen. Joint training for neural machine translation models with monolingual data. In AAAI Conference on Artificial Intelligence, 2018a.
Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. Regularizing neural machine translation by bidireeidii agreement. In AAAI Conference on Artificial Intelligence, 2018b.
Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen. Adaptive nearest neighbor machine translation. In ACI., 2021a.
Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. Non- parametric unsupervised domain adaptation for neural machine translation. EMNLP(findings) 2021b.
12

---

Published as a conference paper ICLR 2023
A IMPLEMENTATION DETAILS AND EVALUATION
The statistics of the dataset and datastore used by the server/clients are listed in Table 3 and Table respectively. We follow the recipe to perform data pre-processing. The Moses toolkit is used to tokenize all sentences and learn bpe-code in the publicly available corpus WMT14, Based on this, we split all the words of the above datasets into subword units (Sennrich et al.) 2016). All experiments
implemented
based on the FAIRSEQ toolkit (Ott et al. 2019). We train the public model on the WMT14 14 En-De dataset and use it the initialization model for methods. We adopt Transformer (Vaswani et 2017 model structure of all baselines. in which it consists of 6 transformer encoder layers, and 6 transformer decoder layers. The input embedding size of the transformer layer 512, the FFN layer dimension is 2048, and the number of self-attention heads is 8. During training. we deploy the Adam optimizer (Kingma & Ba (015) with learning rate of 5e-4 and 4K warm-up updates to optimize model parameters. Both label smoothing coefficient and dropout rate are set to 0.1. The batch size is set to 16K tokens. We train all models with 4 Tesla-V V100 GPU and set patience to to select the best checkpoint on the validation set. The FAISS (Johnson 2021) is leveraged to construct the datastore and we use its IndexIVFPQ strategy implement Product
ct al. Quantizer K-encryption and fast nearest neighbor search. We utilize the FAISS to learn 4096 cluster centroids public datastore, and apply it to client's 's datastore. During inference, the beam size and length penalty are set to 5 and for all methods and we search 64 clusters for each target token when using FAISS. In all experiments, we report the case-sensitive BLEU score (Papineni al. 2002) using sacreBLEU" We estimate the number of floating-point operations (FLOPs) used to train the model by multiplying the training time, the number of GPUs used, and an estimation of the sustained single-pre preccisonn floating -pintt capacity of each G
Table 3: The statistics of datasets for server and clients.
Server
Client
WMT14
Medical
Law
Train
4,475,414
222,927
248,009
467,309
Dev
45,206
2,000
2,000
2,000
Test
3,003
2,000
2,000
2,000
Table 4: The statistics of datastores for server and clients.
Server
Client
Global
WMT14
IT
Medical
Law
(IC, V) size Hard Disk Space (Datastore) Hard Disk Space (Faiss Index)
117,427.00 427,034
3,085,523
5,858,648
25,812,236 16,868,065
114 GB
3,938 MB
6,890 MB
17,717 MB
28,545 MB
8,988 MB
244 MB
451 MB
1,266 MB
1,978 MB
http://iiiiooiippppssssssreeeeee with a configuration of 13a tokenizer, case-sensiiiveensss and full punctuation
'The kinle-precision floating- point capacity for Tesla-V Milla-V100 1000 GPU 14 TFLOPs.
13

---

Published as conference paper ICLR 2023
B MORE RESULTS FOR THE NON-IID SETTING
B.1 PERFORMANCE COMPARISONS WITH CONTROLLER
We compare the performance (BLEU) and overhead of FedNN, FedAvg, and Controller in the Non-IID setting of the En2De translation task. For the Controller model, as shown in Roosta et al. (202I)'s study, 6E-6D/C-C((-3)) model achieves the best trade-off of performance and efficiency among all FL methods. Thus, we follow their setup and adopt layers 0 and 3 (both for the encoder and decoder) as controllers to participate in the parameter interaction of FedAvg training. The experimental results are shown in the Table We find that the Controller has a significant performance improvement compared to P-NMT, but still worse than FedAvgı and FedNN. In addition, since the Controller falls into the multi-round model-based FL interaction paradigm, its communication overhead is still much higher than FedNN.
Table The performance and overhead comparison with Controller. "Comm." and "Comp." refer to communication and computational cost in "GB" and "FLOPs" respectively
Client Test
Server Test
Overall Performance
Cost
Methods
IT
Law Medical
WMT14
Client A Global
A
Comm.
Comp.
P-NMT
26.62
35.91
30.27
26.63
30.93
29.86
Controller
27.78
46.30
35.62
18.72
36.57
+5.63
32.11
+2.25
10.86
6.77
10
FedAvgi
28.26
53.00
45.90
13.45
42.39
+11.45
35.15
+5.30
388 12
3.23
10as
FedNN
35.62
55.57
49.21
22.29
46.80
+15.87
40.67
+10.82
5.08
6.72
10
B.2 EVALUATION WITH BLEURT
We evaluate the two settings in Table Juing the neural metric BLEURT (Sellam et al. 2020). The detailed results are shown in Table 6 We can get similar conclusions when using the BLEU score as an evaluation metric, i.., for the Non-IID setting, our FedNN significantly outperforms all other FL methods; for the IID setting, our FedNN also achieves comparable performance to the FedAvg: and FT-Ensemble.
Table 6: BLEURT score [%] of different methods in Table 1
Client Test
Server Test
Overall Performance
Methods
IT
Law Medical
WMT14
Client
A Global
C-NMT
70.46
78.49
74.97
71.62
74.64
73.89
P-NMT
62.00
72.65
65.64
71.93
66.76
68.06
FedAvgi
63.02
73.71
67.06
72.19
67.93
+1.17
69.00
+0.94
FedAvgı
64.09
78.05
72.74
53.61
71.63
+4.86
67.12
-0.93
FedAvg
62.87
74.11
67.50
72.20
68.16
+1.40
69.17
+1.11
Non-III
FedAvg
52.63
77.37
65.02
56.26
65.01
-1.76
62.82
-5.24
FT--Ensemble
64.08
72.47
70.01
59.17
68.85
+2.09
66.43
-1.62
FedNN
68.86
78.12
72.74
67.38
73.24
+6.48
71.78
+3.72
FedAvgi
66.38 76.31
71.65
72.46
71.45
+4.68
71.70
+3.65
FedAvgı
69.93
78.67
75.68
55.78
74.76
+8.00
70.02
+1.96
FedAvg
64.92
74.32
68.68
72.15
69.31
+2.54
70.02
+1.96
IID
FedAvg.
68.96
78.08
74.39
59.44
73.81
+7.05
70.22
+2.16
FT-Ensemble
70.28
78.80
75.04
58.78
74.71
+7.94
70.73
+2.67
FedNN
67.69 77.74
72.31
68.16
72.58
+5.82
71.48
+3.42
14

---

Published as conference paper ICLR 2023
B.3 SIGNIFICANT TEST FOR TABLE
We use the bootstrap re-sampling method to test the significant difference between FedNN and other methods. Table shows the significance test results of English-German German direction under the Non-IID setting. The "." means that FedNN is not significantly better than the method. We can find that FedNN significant outperforms all FL methods, including FedAvg and FT-Ensemble.
Table 7: The significant test between FedNN and other methods for the Non-IID setting in Table !
Client Test
Server Test
Methods
IT
Law
Medical
WMT14
C-NMT
<0.01
$0.05
P-NMT
$0.01
50.01
<0.01
$0.01
FedAvgı
30.01
<0.01
30.05
$0.01
F'T-Ensemble
<0.01
50.01
50.01
$0.05
B.4 PERFORMANCE COMPARISONS GERMAN-ENGLISH DIRECTION
As illustrated in Table 8 we report the performance of different FL methods in the Non-IID setting of German-Englin Direction. We observe that the findings in the German-English direction remain consistent with the English-Germal direction (shown in Table 1b. in which FedNN outperforms other methods in terms overall performance both client-side and globally.
Table 8: Performance of different methods in the Germann-Englise direction for the Non-IID setting.
Client Test
Server Test
Overall Performance
Methods
IT
Law Medical
WMT14
Client A Global A
P-NMT
31.70
39.86
34.37
31.64
35.31
34.39
FedAvg:
32.22
58.32
48.56
16.83
46.37
+11.06
38.98
+4.59
FT-Ennemblee
35.76
44.07
43.20
21.48
41.01
+5.70
36.13
+1.74
FedNN
41.11
60.18
53.44
27.12
51.58
+16.27
45.46
+11.07
ABLATION STUDY ON THE NON-IID SETTING
C.I THE IMPACT OF CLIENT DATA SIZE ON DIFFERENT FL METHODS
We carry out ablation study to verify the impact of client data size different FL methods, including FedAvgı, FT-Ensemble and FedNN. For each domain, we adopt a ratio range of 3 E (0.0, 0.2, 0.4, 0.6, 0.8) randomly sample from its complete data to constitute the client data of different scales. The detailed results are shown in the Table We can observe that the performance and cost all methods increase as the size of the client data increases. Moreover, FedNN significantly outperforms other FL methods in terms of performance, communication and computational overhead.
C.2 THE IMPACT OF INTERACTION FREQUENCY ON FEDAVG
We conduct experiments to analyze the impact of model interaction frequency on the FedAvg performance. We set the frequency to & E {1,2, 5, 10, 20, oo), i.e., the client interacts the model with the server after k
15

---

Published as conference paper ICLR 2023
Table The impact client data size on the performance and cost of different methods. "Á" refers to the improvement of methods compared with P-NMT. "Comm." and "Comp." refer communication and computational cost "GB" and "FLOPs" respectively.
Client Test
Server Test
Overall Performance
Cost
B
Methods
IT
Law Medical
WMT14
Client
A Global
A
Comm.
Comp.
0.2
P-NMT
26.62
35.91
30.27
26.63
30.93
29.86
FedAvgı
20.27
47.42
33.92
15.73
33.87
+2.94
29.34
-0.52
111.59
9.27
10
0.2
FT-Ensemble
29.95
40.25
37.41
20.81
35.87
+4.94
32.11
+2.25
4.85
1.40
10'7
FedNN
29.65
46.89
41.46
22.72
39.33
+8.40
35.18
+5.32
2.00
1.34 x 1046
FedAvg:
20.99
48.98
35.58
16.58
35.18
+4.25
30.53
+0.67
116.44
9.68 x 10t7
0.4
FT-Ensemble
29.60
39.50
38.45
21.43
35.85
+4.92
32.25
+2.39
4.85
2.81
1047
FedNN
30.41
50.30
43.89
23.32
41.53
+10.60
36.98
+7.12
2.77
2.69
1015
FedAvg:
22.36
52.38
42.04
12.83
38.93
+7.99
32.40
+2.55
245.00
2.04
10
0.6
FT-Ensemble
28.32
40.55
40.19
18.61
36.35
+2.48
31.92
+2.06
4.85
4.21 x 1017
FedNN
31.65
52.29
45.65
23.68
43,20
+12.26
38.32
+8.46
3.54
4.03 x 1046
FedAvgı
24.11
52.71
43.72
12.76
40.18
+9.25
33.33
+3,47
354.74
2.92 x 1017
0.8
FT-Ensemble
29.01
38.16
39.30
16.87
35.49
+4.56
30.84
+0.98
4.85
5.62
10"7
FedNN
32.15
54.11
46.62
22.22
44.29
+13.36
38.78
+8.92
4.31
5.38 x 1015
rounds of local updates. We set the total computation overhead to be the same for a fair comparison of translation performance and communication overhead. The detailed results are shown in the Table We find that the performance decreases significantly as k increases, especially when & = oo (i.c., copy the server model is trained locally until convergence), and the average performance drops to level similar to that P-NMT. The reason is that too many local updates suffer from catastrophic forgetting of knowledge from the previous aggregated models, resulting in strong knowledge conflicts in the new round of interactions. The optimal performance is presented at k = 1, which means that frequent interactions are essential to alleviate the knowledge conflicts for FedAvg.
Table 10: BLEU score [%] and communication cost of FedAvg with different interaction frequency. "Comm. Cost" refer to communication cost in "GB".
Client Test
Server Test
Overall Performance
Comm. Cost
Methods
IT
Law Medical
WMT14
Client
A
Global
A
P-NMT
26.62
35.91
30.27
26.63
30.93
29.86
FedAvg:
28.26
53.00
45.90
13.45
42.39
+11.45
35.15
+5.30
388.12
FedAvga
26.37
52.92
44.22
13.07
41.17
+10.24
34.15
+4.29
194.06
FedAvgs
24.93
52.23
41.08
12.85
39.41
+8.48
32.77
+2.92
77.62
FedAvgıo
22.63
51.11
38.60
12.65
37.45
+6.51
31.25
+1.39
38.81
FedAvgoo
21.82
49.53
36.24
12.75
35.86
+4.93
30.09
+0.23
19.41
FedAvg.
17.30
47.06
30.61
13.33
31.57
+0.63
27.01
-2.85
4.85
C.3 THE IMPACT OF ENSEMBLE STRATEGY ON FT-ENSEMBLE
The ensemble strategy of FT-Ensemble could be implemented in two ways: the first to directly average the probability distribution of each client model (as used in the Table T. i.e., FT-Ensemble; the second, similar
16

---

Published as conference paper ICLR 2023
Table 11: BLEU score [%]] of FT-Ensemble with different aggregation strategy.
Client Test
Server Test
Overall Performance
Methods
IT
Law
Medical
WMT14
Client A Global
A
Public Model
26.62
35.91
30.27
26.63
30.93
29.86
F'T-Ensemble
30.11
38.14
39.15
17.13
35.80
4.87
31.13
1.28
FT-Ensemble- Wei
24.09
48.58
34.11
16.06
35.59
4.66
30.71
0.85
FedNN
35.62
55.57
49.21
22.29
46.80
15.87
40.67
10.82
Table 12: The impact of P-NMT's quality. "A" refers to the improvement of the method compared with the model mentioned Table
IT
Law
Medical BLEU
WMT14
Clients Avg. BLEU
Global Avg. BLEU
Methods
BLEU
A
BLEU
A
A
BLEU
A
A
P-NMT
30.72
+4.10
38.69
+2.78
35.90
+5.63
29.77
+3.14
35.10
+4.17
33.77
+3.91
FedAvg.
28.63
+0.37
58.32
+5.32
49.08
+3.18
16.05
+2.60
45.34
+2.95
38.02
+2.87
FedNN
38.24
+2.62
55.76
+0.19
50.65
+1.44
22.65
+0.36
48.22
+1,42
41.83
+1.16
to FedAvg, is to weight the probability distribution of each client model's output by assigning weights to it according to its data size, i.e., FT-Ensemble-War The performance comparison of these two ways the Non-IID setting shown in the Table II] We find that FT-Ensemble outperforms FT-Ensemble-Wer in both client-side and global overall performance. FT-Ensemble has a more balanced performance on the client side, while FT-Ensemble- Wei is similar FedAvg in that the performance is more biased towards the client Law's model, which has more local data. Our FedNN outperforms both of these methods on all clients. Note that the two implementations of FT-Ensemble described above in the IID setting are equivalent since the data size is the same for all client.
C.4 THE IMPACT OF PUBLIC MODEL'S QUALITY
Since FedNN performs federated learning based the P-NMT, we investigate the impact of the P-NMT's quality on performance. We introduce WMT20 En-De data to train the P-NMT, which contains 40 million parallel pairs, and conduct fast experiments in the Non-IID setting. From Table 12 quality of the P-NMT improves, all methods show better performance.
observe that the
D THE DETAILS OF PRIVACY LEAKAGE ANALYSIS
DATASET CONSTRUCTION
Given a local parallel sentence pair (X) E De of client attacker, the public NMT model generates the context representation & fe (x(x,, (e,et) in the last decoder layer at each timestep t, and the ground-truth e = Ve,x- has two forms, i.e., whether it is encrypted by K- K-Encryption or not. Next, we concat them to obtain training sample Fe. = @ a <2sr<> x @ <2tgt> @ yo <r v, where is the concatenation operation. The language tag <2src> and <2tgt> are used to identify the generation of source and target languages, respectively. By traversing the entire D., obtain the whole dataset R (r1,72,.... In) used to train the threat model, where n Σ [Pal ly, + [De]- The detailed statistics of dataset used for threat model are shown in Table 13
17

---

Published as conference paper ICLR 2023
Table 13: The statistics of datasets for threat model.
IT
Medical
Law
Train
3,085,523
5,858,648
16,868,065
Dev 34,737
55,577
51,423
Test
2,000
2,000
2,000
Prefix Token:
Text Predicted by Autoregressive Paradigm:
hunde
<2re>>
like
dogs
<ttt>
Ich
mag
hunde
Reconstruction Transformer Decoder
Linear
Embd
Embd
Embd
Embd
Embd
Embd
Embd
Embd
key
value
hunde
<2sre>
like
dogs
<2tgr>
mag
Figure 4: The threat model based the autoregressive paradigm.
D.2 THE ARCHITECTURE OF THREAT MODEL
The goal of the threat model is reconstruct the corresponding original text from the memorization (k, v) of client defender. As shown in Figure 4 we use a transformer decoder as the architecture of the threat model, which is similar the left-to-right language model based on the auto-regressive paradigm. consists of 6 transformer layers. The input embedding size is 512, the FFN layer dimension is 2048, and the number of self- aff-aatention heads 8. We first transform the first input token to the same dimension the word embedding using linear layer, and then auto-regressive perform left-to-right reconstruction modeling.
D.3 EVALUATION OF PRIVACY LEAKAGE
We quantify the privacy information leaked by global memorization using sentence-level and word-level metrics, i.e., reconstruction BLEU and privacy word hitting Precision/RecallFF.. Assuming that the text recovered by the threat model from memorization K e is h; (h,,1 hip,, ..pp, } and the ground-truth is is test sample index. Then calculate the reconstruction BLEU score using sacreBLEU Before evaluating the word-level privacy leakage, we require to extract the privacy dictionary of the client defender. The privacy dictionary is obtained by computing the difference between the word distribution of the defender's 's private dataset and the server public dataset. Further, we filter h; and g, according to this dictionary obtain sentences h. and g that contain only privacy words. The
g; 99.,1. .9. J. where
18

---

Published as conference paper ICLR 2023
word-Ievel metric then is computed as follows:
E. Ex
1 Σ h
Count So hpp)
Precision
Recall Σ, Σj |b
Count g
(5)
lg
Fl x Precision Recall Precision + Recall
where Countra y) represents that æ has appeared in y.
D.4 QUALITATIVE ANALYSIS OF PRIVACY
Some qualitative cases illustrated in Table and we find that the style of all reconstructed texts remained consistent with the attacker's s training data, including text length and domain style. For example, in Casel and Case2, the reconstructed texts from the attack model trained on the Law domain exhibit domain style of law client. This means that it is difficult to recover and identify valuable information, such as domain and private words, from global memorization.
COST ANALYSIS OF FL METHODS ON DIFFERENT CLIENT'S NUMBER
The communication and computational costs of different FL methods illustrated in Table 14 For communication, the cost of FedAvg is much higher than that of FedNN and FT-Ensemble. The reason that FedAvg requires multi-round communication based the model, while both FedNN and FT- FT-Ensemble require only one-round memorization- linearly related the number of nodes. It cannot be extended practical applications because of the number of local models that need to be integrated for inference. In contrast, the cost of FedNN is only 1/60 and N/2 of FedAvgı and FTEEnsemble, respectively. Considering many clients' limited communication bandwidth and computational resources, FedNN is promising framework selection to save lot of communication time and computational consumption.
communication. For computation, the cost of FT-Ensemble
Table 14:
The communication cost and computation cost of different methods, where "M, N, R and D" respectively represent the model size (4 (414MB), number of client, rounds of communication (160) and the total size of all encrypted datastores (1978MB).
Communication Cost (GB)
Computation Cost (FLOPs))
Compl.
6
3
18
FedAvg FT-Ensemble
MxNxRx2
388.12
776.25
1552.50
3105.00
3.23x 18
3,23x10'8
3,23x10'
3
MXNx(N+1)
4.85
16.98
63.07
138.27
7.02x 17
1.40x
2
1
FedNN
(D+M)xN
5.08
12.08
26.10
40.12
6.72x 15
6.72x
607
6.72x10'5
F LIMITATIONS
In this paper, we utilizes one round of memorization- interaction share knowledge among different clients, thus building low-overhead method as follows.
translation systems. We discuss limitations of our
Despite our proposed approach achieves strong performance when exploiting global memorization sharing. it leads to reduced inference efficiency due the need for I:NN retrieval. As shown in Table T the inference
19

---

Published as a conference paper ICLR 2023
speed of FedNN about 0.75x that of P-NMT. In practice, these costs may be acceptable since we employ FAISS to speed up KNN retrieval. We encourage future work to improve the efficiency of kNN retrieval.
The communication overhead required for memorization-bass interaction is positively correlated with the client data size. Extremely large client data will make our approach inapplicable because it leads to higher communication overhead. Our approach is more applicable to the generic scenario described in Section 2 i... private data is sparse ([P,) < [D&). We also encourage further exploration of how to build a smaller and more accurate memorization further to mitigate this problem.
This paper still very preliminary in the privacy leakage analysis of memorization interaction. Although the threat model on shared global memorization has very low reconstruction scores, privacy leakage is still potential risk. How to better evaluate and mitigate the privacy leakage of memorization remains an open question, which we leave for future work.
Table 15: Examples of qualitative analysis for privacy leakage. Text in green / blue represent the defender-
specific ground-truth and private words, respectively. Text in red represents the hit private words by attacker. The bold word represents the threat model of the client-side attacker, where the superscript "fxE" represents the input of training data & is encrypted K-Encryption, otherwise it is not.
Case Examples
Case I: Defender
<2are> cursor qu' ckly moving to object
<2tgt>
carsor
schnell su efmem Objekt bewegen
Medical/KI; <2sre> curves fointing, salivation, vomiting, diarrhoes, fainting, fainting, or vomiting, diarchoea. <2tgi> Kleben, Painting, Speichelfluss, Rrbrechen, Durchfall, Ohnnacht Erbre- chen odez Durchfal: oder Erbrecher.
Medical: <arre> curoniun or vecurontum: <2tgt>> Vecurontum um oder Vecurontum:
Law/ke; <Zere> palm ofl falling within CN code 2710 30 90 <2tst> Pa1m81 des KN-Codea 2710 00 90 Law: <2aru> curbiling Uhe use of the designatiin 'butler" in Annex Lo Regulation (EBC) No 3143
<2tgt> curbitration der Bezeichnung 'Butter' in Anhang der Verordnung (EW3) Nr. 3142 85
Case 2: Defender Medical
<2erc> Entravenous infusion after reconet itut fon
and d'lution. <Ztgt> Intravenčse
Tnfustor
Auflösung Verdülnuug.
IT/KE; Insects placeholder. <2tgt> Niermit Eügen Sie Platzhaltes
<2are> Teserts a new row, <2tgt> Fügtt aiine neue Selll ein.
Law/ke; <2erc>
Appot ntment of the date of mintmum purability shall be given. <Ztet> Die Angaba des
Law:
mininum
daLe durabilily <2LgL> Angabe Mindes berkeesssss
Case 3: Defender is Law
<2orc> The Commission consistently Lakes Cavourable view of such hilfen werden voa der Kommission stets befürwortet.
aid
<ZLgt> Derartige Bei-
IT/EE; <2sru> The kappnames Hannbbook <2.gL> Das Handbbch zu keppname;
IT: <2arc> Follewing packages depand on the installed packages: <2tgt> Dia fol tolgenden von den installlerten Pakate ab:
Pakete hängen
Medical/EE: <2sre> Most common side offects with Azarga (seen in between 1 and 13 patients in 100) zrehaadache, dizzinesa, sommolence (sleepinesal, nausaa (feeling sick), diarrhoea, abdominal tunmy pein, diarrhona, flatulence (gas), abdominsl (tunmy) psin, dyspepsia (indigestion), diarrhoea, nau- sea (feeling sick), voniting, abdominal (turmy) pain, dyspopsia (Indigastion), flatulence (wind) ...
Medical: <2arc> Puropean Commission granted marketing authorisation valid throughout the 24 April 2009, <Zogt> April 2009 erteilte die Europäische Kemnission Unternehmen Intervet Insernational By eine Gens -migung für das Inverkehabringe: von Nobilis Influenza IIEN5 in der gesemten Europälschen Unien.
Buropean Union for Nobilis Influersa H5N6 to Intervet International
20


Reviews:
[{"review_id": "yl38r2D8Kde", "reviewer": "Reviewer_acCF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the federated nearest neighbor (FedNN) machine translation framework, which is built on top of the nearest neighbor machine translation method (kNN-MT). FedNN extends kNN-MT in a federated learning setting, where a server provides a global model and clients keep private data to improve the model. The paper proposes that to avoid privacy leakage, the clients send out an encrypted datastore which is built on a private dataset. By doing this, the system avoids multi-round model-based interactions, so that the communication overhead is made less. The paper conducts experiments on WMT14 and multi-domain en-de dataset. The experiments show that FedNN outperforms several baselines while keeping the communication and computational overhead low.\n\nThe main contribution of this paper is to apply kNN-MT to a federated learning setup, which is a reasonable application given kNN-MT is a cheap and effective method for domain adaptation. \n", "summary_of_the_review": "In summary, I think the paper studies an important problem and the proposed solution makes sense to me. However, I also think the paper can still be improved in some places.\n", "strength_and_weaknesses": "Strengths:\n- The paper studies an intersection of two important topics -- machine translation and federated learning. I believe this topic can be potentially impactful.\n- The paper is well-motivated and the general idea of building domain-specific datastores without updating the model in the federated learning setting makes sense to me.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- The main concern of this paper is that it may not make enough technical contributions. The FedNN framework is an extension of the kNN-MT in the federated learning setup, but only this contribution doesn’t guarantee the paper to be published in ICLR in my opinion. The two-phase datastore encryption is interesting, however it is not studied extensively in the paper but built on off-the-shelf tools. The paper can be made stronger by considering different encryption methods.\n- The paper compares the performance and communication/computational overhead of FedNN and existing methods. However, the paper does not discuss the privacy preserving ability of different approaches. For example, how do multi-round model-based interaction methods preserve the privacy of clients compared to FedNN?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iZ87INkpAkP", "reviewer": "Reviewer_1QjM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an adaptation of the kNN MT model into the federated learning framework. The motivation is to achieve client-side machine translation with minimal communication costs with the central server NMT model and while maintaining the privacy of client data. The authors propose the kNN federated learning approach paired with (K,V) encryption of the data-stores as an alternative to federated learning approaches that use multi-round model updates to integrate client-side data to the central model. The authors compare their model to other baseline approaches regarding BLEU score as well as communication, computational and inference costs and they experiment with IID and non-IID setups, while they also perform ablation tests regarding the number of clients and the heterogeneity degree of data.", "summary_of_the_review": "Well-motivated approach with interesting insights, but some questionable choices in the experimental setup that could be improved or further clarified/justified.", "strength_and_weaknesses": "The paper is well motivated and the authors perform comparisons across different aspects. They draw useful insights on the impact of the data distribution, the number of clients etc. They also do show that their approach outperforms the other models, especially for the non-IID setup. Overall, it is interesting to see the integration of the kNN MT approach in the federated learning workflow and the authors do a good job at addressing privacy and computational/communication concerns.\n\nHowever, I have a few concerns regarding the experimental setup and decisions. The authors seem to compare against the FedAvg method which is an implementation of the McMahan et al. (2017) method, and they point out the vast communication cost. However, there have been more recent papers (that are cited by the authors as related work: Roosta et al., 2021; Passban et al., 2022) that show improvements in terms of communication. It would make more sense to compare to these methods instead, or at least make some attempt to calculate an optimal update interval for the FedAvg method instead of considering only the FedAvg_1 and FedAvg_infinity. Similarly it is unclear how the authors aggregate the prediction outputs for the FT-Ensemble method: in this case too, the aggregation approach (e.g. is it a weighted fusion?) and potential optimisation of the parameters could impact the performance of the FT-Ensemble.\n\nMoreover, it is not clear why the authors did not use the WMT14 De-En language pair (but use the WMT14 En-De instead) that was used by the previous works and would allow for direct comparisons with reported results in other work (especially since the authors seem to have not optimised the methods they compare against). \n\nAdditionally, the authors mention significantly outperforming other methods (e.g.: \"FedNN has a significant performance improvement of 4.41/1.99 BLEU scores...\") but it is unclear what is the statistical significance test that was performed (if any). \n\nWhile my main concerns are listed above, I would like to add some further suggestions:\n- While I appreciate the already performed ablation tests, I am missing some ablation regarding the impact of the size of the data on the client and the server side. It is hinted in the paper that the size of the client data impacts the performance of the FedAvg methods but there is no specific experiment about it. Still it seems to be a crucial parameter that could impact the performance and cost of most of the compared methods and it would be very interesting to see a more detailed analysis about this.\n- I would propose that the authors consider comparing the performance of different systems using some more recent, neural metric (e.g. BLEURT, COMET) in addition to BLEU as it would allow to draw better comparisons.\n\nSome formatting/grammar/typo comments:\n- Some fonts of figure 1 are way too small\n- In the mention: \"the server builds (K, V)-encryption model\" I think it should be:  \"the server builds a (K, V)-encryption model \" . Additionally, you mention (K,V) a couple of times before explaining what (K,V) stands for. I would propose to add the definition/explanation in the first mention.\n- \"vis kNN retrieval based on the context representation\" —> via instead of vis?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UyG-HyuRhaH", "reviewer": "Reviewer_Ljh8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a federated nearest neighbor (FedNN) method for machine translation. Instead of multi-round model-based interactions, this mothed leverages one-round memorization-based interaction to share knowledge across different clients to build low-overhead privacy-preserving systems.", "summary_of_the_review": "Authors propose a federated nearest neighbor (FedNN) method for machine translation. The proposed method combines federated learning and neural machine translation to achieve SOTA results. However, the innovation is limited and do not give much context to federated learning.", "strength_and_weaknesses": "Strength\n\nIn general, this paper is well organized. The method is reasonable and technically sound, the experiments are well designed and basically sufficient to support the conclusions.\n\nWeakness\n\n1. The innovation in the approach proposed by the authors is incremental. It is a combination of machine translation methods (e.g., kNN-MT) and federated learning methods.\n\n2. There is too little related works about federated learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CRceaSbW3N", "reviewer": "Reviewer_61jW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles developing neural machine translation models in the federated learning scenario.\nThere have been several methods that attempt to solve the identical scenario.\nThe authors point out that the conventional methods require vast communication overheads and heavy synchronization that makes the methods impractical when we consider the actual use-case for training machine translation models, including the way to use training data with privacy.\nThis paper proposed a method that can be built only with low overhead while preserving privacy to mitigate inefficiency.\nThe key idea is to leverage the technique of nearest-neighbor neural machine translation (kNN-MT).\nThe experimental results on two datasets in the federated learning scenario show significantly better performance over several conventional methods, with fewer communication costs.", "summary_of_the_review": "As I pointed out above, the method itself is not super novel, but this paper carefully considers and tackles a practical scenario in actual use.\nTherefore, this paper has sufficient contributions to be accepted to the conference.\n", "strength_and_weaknesses": "Strength:\n\n* This paper is well-organized and easy to read.\n* Figure 1 is neat and concise, and can help a deeper understanding of the proposed method.\n* The claims and the proposed method seem reasonable.\n* The experiments show that the proposed method can perform better than a few conventional methods in practical federated learning scenarios.\n\nWeakness:\n\n* The primary idea of the proposed method is just borrowing the previous method of kNN-MT. From this perspective, the proposed method can be considered just applying the existing method (with slight modifications) in a different learning configuration.\n* most of the sub-modules that consist of the proposed method are also a combination of the existing methods. Therefore, the method is not innovative, so this is somewhat of an incremental study.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster