Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present an adaptation of the kNN MT model into the federated learning framework. The motivation is to achieve client-side machine translation with minimal communication costs with the central server NMT model and while maintaining the privacy of client data. The authors propose the kNN federated learning approach paired with (K,V) encryption of the data-stores as an alternative to federated learning approaches that use multi-round model updates to integrate client-side data to the central model. The authors compare their model to other baseline approaches regarding BLEU score as well as communication, computational and inference costs and they experiment with IID and non-IID setups, while they also perform ablation tests regarding the number of clients and the heterogeneity degree of data.
strength_and_weaknesses: The paper is well motivated and the authors perform comparisons across different aspects. They draw useful insights on the impact of the data distribution, the number of clients etc. They also do show that their approach outperforms the other models, especially for the non-IID setup. Overall, it is interesting to see the integration of the kNN MT approach in the federated learning workflow and the authors do a good job at addressing privacy and computational/communication concerns.

However, I have a few concerns regarding the experimental setup and decisions. The authors seem to compare against the FedAvg method which is an implementation of the McMahan et al. (2017) method, and they point out the vast communication cost. However, there have been more recent papers (that are cited by the authors as related work: Roosta et al., 2021; Passban et al., 2022) that show improvements in terms of communication. It would make more sense to compare to these methods instead, or at least make some attempt to calculate an optimal update interval for the FedAvg method instead of considering only the FedAvg_1 and FedAvg_infinity. Similarly it is unclear how the authors aggregate the prediction outputs for the FT-Ensemble method: in this case too, the aggregation approach (e.g. is it a weighted fusion?) and potential optimisation of the parameters could impact the performance of the FT-Ensemble.

Moreover, it is not clear why the authors did not use the WMT14 De-En language pair (but use the WMT14 En-De instead) that was used by the previous works and would allow for direct comparisons with reported results in other work (especially since the authors seem to have not optimised the methods they compare against). 

Additionally, the authors mention significantly outperforming other methods (e.g.: "FedNN has a significant performance improvement of 4.41/1.99 BLEU scores...") but it is unclear what is the statistical significance test that was performed (if any). 

While my main concerns are listed above, I would like to add some further suggestions:
- While I appreciate the already performed ablation tests, I am missing some ablation regarding the impact of the size of the data on the client and the server side. It is hinted in the paper that the size of the client data impacts the performance of the FedAvg methods but there is no specific experiment about it. Still it seems to be a crucial parameter that could impact the performance and cost of most of the compared methods and it would be very interesting to see a more detailed analysis about this.
- I would propose that the authors consider comparing the performance of different systems using some more recent, neural metric (e.g. BLEURT, COMET) in addition to BLEU as it would allow to draw better comparisons.

Some formatting/grammar/typo comments:
- Some fonts of figure 1 are way too small
- In the mention: "the server builds (K, V)-encryption model" I think it should be:  "the server builds a (K, V)-encryption model " . Additionally, you mention (K,V) a couple of times before explaining what (K,V) stands for. I would propose to add the definition/explanation in the first mention.
- "vis kNN retrieval based on the context representation" —> via instead of vis?
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written and clear with the exception of some typos etc. It is easy to follow but I would perhaps clarify the privacy/encryption related information sooner.

There is no novel method proposed per-se, but the novelty lies in integrating the kNN MT method with federated learning and addressing the privacy and computational concerns. It is a solid contribution with nice insights, but some issues that could be improved in the experimental setup and the implementation of the methods comparing against.

In terms of reproducibility, the method proposed by the authors could probably be reproduced with some effort and using the information in the appendix, but some details for the methods compared against are less clear (see also my comments in the previous section).
summary_of_the_review: Well-motivated approach with interesting insights, but some questionable choices in the experimental setup that could be improved or further clarified/justified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a federated nearest neighbor (FedNN) method for machine translation. Instead of multi-round model-based interactions, this mothed leverages one-round memorization-based interaction to share knowledge across different clients to build low-overhead privacy-preserving systems.
strength_and_weaknesses: Strength

In general, this paper is well organized. The method is reasonable and technically sound, the experiments are well designed and basically sufficient to support the conclusions.

Weakness

1. The innovation in the approach proposed by the authors is incremental. It is a combination of machine translation methods (e.g., kNN-MT) and federated learning methods.

2. There is too little related works about federated learning.

clarity,_quality,_novelty_and_reproducibility: Clarity: clear

Quality：high

Novelty: incremental

Reproducibility: low

summary_of_the_review: Authors propose a federated nearest neighbor (FedNN) method for machine translation. The proposed method combines federated learning and neural machine translation to achieve SOTA results. However, the innovation is limited and do not give much context to federated learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the federated nearest neighbor (FedNN) machine translation framework, which is built on top of the nearest neighbor machine translation method (kNN-MT). FedNN extends kNN-MT in a federated learning setting, where a server provides a global model and clients keep private data to improve the model. The paper proposes that to avoid privacy leakage, the clients send out an encrypted datastore which is built on a private dataset. By doing this, the system avoids multi-round model-based interactions, so that the communication overhead is made less. The paper conducts experiments on WMT14 and multi-domain en-de dataset. The experiments show that FedNN outperforms several baselines while keeping the communication and computational overhead low.

The main contribution of this paper is to apply kNN-MT to a federated learning setup, which is a reasonable application given kNN-MT is a cheap and effective method for domain adaptation. 

strength_and_weaknesses: Strengths:
- The paper studies an intersection of two important topics -- machine translation and federated learning. I believe this topic can be potentially impactful.
- The paper is well-motivated and the general idea of building domain-specific datastores without updating the model in the federated learning setting makes sense to me.
- The paper is well-written and easy to follow.

Weaknesses:
- The main concern of this paper is that it may not make enough technical contributions. The FedNN framework is an extension of the kNN-MT in the federated learning setup, but only this contribution doesn’t guarantee the paper to be published in ICLR in my opinion. The two-phase datastore encryption is interesting, however it is not studied extensively in the paper but built on off-the-shelf tools. The paper can be made stronger by considering different encryption methods.
- The paper compares the performance and communication/computational overhead of FedNN and existing methods. However, the paper does not discuss the privacy preserving ability of different approaches. For example, how do multi-round model-based interaction methods preserve the privacy of clients compared to FedNN?

clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: the paper is well-written and easy to follow. There are a few places that I am a bit confused about and I would suggest the authors make them more clearly or include more discussions.
- In Table 1, there is too much content. It is better to highlight some numbers.
- In Table 1, why is the communication/computation of FedAvg_1 more than FedAvg_inf?
- In Table 1, the centralized model shows slightly worse performance compared to FedNN. This is surprising to me, as the centralized model is trained with client data. Could you provide more discussion about this?

Novelty: See my evaluation in the previous section.

Reproducibility: I believe the results can be reproduced easily. I encourage authors to include more implementation details.

summary_of_the_review: In summary, I think the paper studies an important problem and the proposed solution makes sense to me. However, I also think the paper can still be improved in some places.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles developing neural machine translation models in the federated learning scenario.
There have been several methods that attempt to solve the identical scenario.
The authors point out that the conventional methods require vast communication overheads and heavy synchronization that makes the methods impractical when we consider the actual use-case for training machine translation models, including the way to use training data with privacy.
This paper proposed a method that can be built only with low overhead while preserving privacy to mitigate inefficiency.
The key idea is to leverage the technique of nearest-neighbor neural machine translation (kNN-MT).
The experimental results on two datasets in the federated learning scenario show significantly better performance over several conventional methods, with fewer communication costs.
strength_and_weaknesses: Strength:

* This paper is well-organized and easy to read.
* Figure 1 is neat and concise, and can help a deeper understanding of the proposed method.
* The claims and the proposed method seem reasonable.
* The experiments show that the proposed method can perform better than a few conventional methods in practical federated learning scenarios.

Weakness:

* The primary idea of the proposed method is just borrowing the previous method of kNN-MT. From this perspective, the proposed method can be considered just applying the existing method (with slight modifications) in a different learning configuration.
* most of the sub-modules that consist of the proposed method are also a combination of the existing methods. Therefore, the method is not innovative, so this is somewhat of an incremental study.
clarity,_quality,_novelty_and_reproducibility: These points are all written in the above Strength and Weakness text box.
Please refer it.

summary_of_the_review: As I pointed out above, the method itself is not super novel, but this paper carefully considers and tackles a practical scenario in actual use.
Therefore, this paper has sufficient contributions to be accepted to the conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper puts together retrieval-augmented machine translation (via the kNN-MT) and federated learning to achieve privacy-preserving client-side machine translation with minimal communication costs with the central server NMT model. The authors compare their model to other baseline approaches regarding BLEU score as well as communication, computational and inference costs and perform several ablation tests. In their updated version, the authors report also BLEURT scores (trained neural metrics) which correlate better with human judgments than BLEU. This is a somewhat a borderline paper, with all reviewers inclined to accept, but pointing out the limited technical novelty as a concern, as the paper essentially puts together two lines of research, kNN-MT and federated learning. In my opinion, despite the lack of technical novelty, putting these two things together is still a valuable contribution and the result is of interest to the research community, therefore I lean towards acceptance. I urge the authors to incorporate the suggestions made by the reviewers when preparing the final version of their paper.
justification_for_why_not_higher_score: I don't think the paper presents enough novelty for being accepted as spotlight or oral.
justification_for_why_not_lower_score: Despite the lack of technical novelty, the paper puts two lines of research together and it does it well. Experiments are convincing and include ablation tests.
Decision: Accept: poster