Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work is to improve the efficiency of the KNN-MT framework. In this work, the author proposed a simple and scalable framework to speed up the KNN search. By utilizing sentence level retrieval approach (BM25), the author reduced the search pool for each input. The experiment results shows comparable performance with KNN-MT and reduced time/storage cost.
strength_and_weaknesses: Strength: The proposed method is reasonable and effective to improve the kNN-MT framework by reducing the search space. 
Weakness: More analysis on m and k choice is needed. If k=1 or 2, is the always best choice for kNN-MT, which means the rest examples are all far away, then we just keep top-1 token pair is ok and does not need sentence retrievel.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, and the quality is good. The proposed idea is reasonable and novel but simple.
summary_of_the_review: Overall, the work is simple but effective but needs further justification mentioned in the k setup to verify if the method is solid or it is due to the task dataset.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents SK-MT which makes storage efficiency improvements over the kNN-MT model. The efficiency comes in the form of a dynamic datastore construction strategy that uses token-level retrieval (BM25) to first get a set of relevant translation pairs, and then construct a datastore from this smaller subset. To alleviate noise from a smaller datastore, an adaptive interpolation parameter is computed based on the ratio of the top-1 neighbor’s distance and the softmax temperature used with kNN-MT. This is a simpler adaptation approach than adaptive kNN-MT that involves learning a separate network for adaptation.

Experiments are conducted on two MT tasks, reporting BLEU and character n-gram f-scores.  
- Domain adaptation using the multi-domains data: the proposed SK-MT models are on par with the full-datastore baselines (kNN-MT and adaptive kNN-MT). The proposed SK-MT method reduces storage size of the datastore considerably and eliminates the need for a faiss index, but doesn’t necessarily run faster than the full datastore baselines when they search the faiss index on the GPU.
- MT with humans in the loop providing post-edits and corrections: this involves growing the datastores dynamically as the human edits become available. The two corpora include EMA and JRCAquis. The proposed SK-MT method is on par with the KoK baseline that operates over the full datastore with an adaptive interpolation parameter.

strength_and_weaknesses: Strengths:  
1. This paper brings the approach from Gu et al. (2018) to modern transformers and kNN-MT style approaches that don’t require training the model, while having much lower storage costs for the datastore which makes brute-force search faster.
2. It proposes a simplification to adaptive kNN-MT that does not require training a separate network. Ablations show that the proposed adaptation is especially important for the dynamically constructed datastores.
3. The proposed approach is on par with the full datastore kNN-MT and variants.
4. The paper provides latency scores as well as an analysis of word accuracy across methods based on the frequency of the words.
5. The paper provides an analysis of the rate at which various approaches adapt to domain-specific words.

Weaknesses: 
1. The paper’s primary proposed approach for building the index dynamically seems very similar to Gu et al. (2018) and specifically their shallow fusion variant. Yet, the paper doesn’t really explore this connection or even attribute this approach to them. The results and analyses are interesting, but a discussion on this connection seems important and is currently missing. 

2. While the decrease in storage costs is nice, the decoding speed improvements aren’t clearly there due to faiss-gpu, especially when the number of retrieved translation pairs increases from 2 to 16. The current story and discussion should be revised to remove the claim that this approach improves decoding speed, or it should be revised to clearly state that a trade-off exists – better decoding speed comes at the cost of lower performance. Adding a plot on latency vs. performance would go nicely with a discussion on the trade-off. 

3. The paper calls this approach scalable but it is unclear how the number of retrieved translation pairs (m) scales with the size of the corpus. Is this claim justified somewhere in the paper through trends based on corpus size? The original kNN-MT paper shows that larger datastores provide diminishing returns so perhaps there is some evidence of this in the literature but missed the discussion in the paper.

4. The discussion on online learning experiments needs more detail. It’s not very clear what the setup is for the different approaches, especially SK-MT. Is there some constraint that selects human provided translations, or are the token-level retrievals only over the human provided translations?

5. Related to the previous discussion, the analysis on R-values is also a bit confusing. Given the current explanation of the online learning experiments, shouldn’t the R-value for all kNN approaches converge to the same value for tokens that have appeared 9+ times? Why do the full datastore retrievals ignore the human feedback but SK-MT doesn’t, or is something else going on? Also how important is it to upweight the retrievals from human feedback? What if there were a completely separate kNN function operating on the human feedback combined with kNN over the dataset and the base MT system?

6. Overall, given the connection to Gu et al. (2018) and the lack of decoding speed improvements, it isn’t clear if the approach makes enough of a case for added value even though the results and analysis are interesting – at least not in the current narrative of the paper. Why is this approach better than just adaptive kNN-MT if it doesn’t run faster? Both sets of experiments show similar performance to full datastore but adaptive versions of kNN-MT. Maybe one argument is that lower storage makes this viable for running kNN-MT on-device where there are storage constraints? 

Questions:
1. Why are numbers in Tables 2 and 4 for IT different?

Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. Search engine guided neural machine translation. In AAAI, 2018.

clarity,_quality,_novelty_and_reproducibility: The paper could improve on clarity. See suggestions in the previous section. Also add descriptions of the baselines to at least the appendix (or main paper if there is space). 

The experiments and analysis are interesting, even given the connection to previous work which hurts novelty a bit. But, the framing and discussion are not clear or comprehensive which currently hurts the quality of the work. 

There is no mention of code release, but the paper includes details on hyperparameters and setup that would probably help reproducibility, at least on the domain adaptation experiments.

summary_of_the_review: Recommending rejection because the current version of the paper is missing an important connection to prior work and a number of claims are unclear and/or unjustified. Open to revising the recommendation after the discussion period.

EDIT after author response:  
The authors did a good job of addressing the primary concerns raised in the initial review. While the novelty of their approach is still limited, this paper makes contributions on the empirical side and provides data points that could be valuable for the community. For this reason, I’m raising my score to a 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes two very simple methods to improve the efficiency and effectiveness of k-nearest neighbor machine translation by 1) dynamically creating a datastore of bilingual sentences given an input sentence using BM25, and 2) dynamically adjusting a coefficient to scale the contribution from the similarity scores of extracted instances.

Experiments are carried out in two scenarios, MT domain adaptation and online learning. In the domain adaptation task, the proposed work achieved significant speed up without loosing the MT qualities. In the online learning task, e.g., iteratively enlarging the training data, the proposed method also achieved better results when compared with prior baselines.
strength_and_weaknesses: Strength

* The use of data retrieval based on BM25 sounds a simple yet efficient method to solve the computational problem in kNN-MT. The experiments in Table 3 shows reduced latencies by extracting only relevant bilingual sentences without loss in translation qualities as shown in Table 2.

* The dynamic adaptation for the coefficient in Equation 6 is very simple and experiments demonstrate the gains.

Weakness

* This work presents experiments in two tasks, domain adaptation and online learning, and does not present any results regarding translation in general, e.g., training data on news domain and testing on news domain. This work should also include those results to discuss whether the proposed method also works in a standard task setting. Even if the results were negative, they should be provided for further discussion.

  - The additional experimental results comparing for generic dataset sounds reasonable to me. 

* The proposed dynamic coefficient scaling seems to be very effective in adjusting the tradeoff of the extracted dataset, but it is not clear whether the approach is only effective to the BM25-based filtering or not. This work should compare the use of the dynamic scaling for other baselines, e.g., the original kNN MT and its variations, to investigate whether the approach is orthogonal to the retrieval methods or not.

  - I also like the additional experiment, and that should be included in the main body.
clarity,_quality,_novelty_and_reproducibility: This paper is easy to follow and the proposed method is very clear. The proposed method will be easily reproduced given the simplicity of the proposed approach.
summary_of_the_review: This work presents an interesting and very simple approach to solve the efficiency problem in kNN MT. Experiments are well designed, but it is missing an important question on how well the proposed approach might work on a standard setting, i.e., training and testing on the same domain with large data. Also, the dynamic scaling sounds a nice contribution to the field, but it is not clear whether the approach could be employed in other retrieval methods or not.

- Additional experiments will be useful for this community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: $k$NN-NMT is a straightforward yet powerful approach for fast domain adaptation, but is burdened with massive storage requirements
and high computational complexity. This paper propose an efficient variant of $k$NN-MT that drastically increase speed in two folds:
1) search similar sentences by search engine instead of searching over all tokens in training datasets
2) use a distance-aware adaptor to adaptively incorporate the kNN retrieval results into the pre-trained NMT models.


strength_and_weaknesses: **Strength**
* The method is simple yet effective on multiple adaptation tasks.
* The writing is clear and the paper is easy to follow.

**Weaknesses**
* $k$NN-MT is not only good at (narrow-)domain adaptation, but also good at improving performance on large-scale open-domain task like WMT en-de, however there is no corresponding experiments in this paper. Actually, I suspect that the sentence-level retrieval may only work well on narrow-domain data as reported in [2].
* Regarding the two major contribution of this paper, i.e. 1) search similar sentences by search engine and 2) distance-aware adaptor, there are existed works [1][2][3]. I would give authors the credit of combining of them together in $k$NN-MT setting, but I think the contribution may not be significant enough. Directly comparing the proposed method with [1] and [2] and siginificant improvements could address my concern.
* Although the compuation cost of searching $k$NN is alleviated, the proposed method still need to compute hidden representation of $k$ nearest sentences on-the-fly, this is actually another heavy compuational burden on GPU.

**References**

[1] Search engine guided neural machine translation. Jiatao Gu et al., AAAI 2018

[2] Guiding Neural Machine Translation with Retrieved Translation Pieces. Zhang et al., NAACL 2018 

[3] Adaptive nearest neighbor machine translation. Zheng et al., ACL2021
clarity,_quality,_novelty_and_reproducibility: See strengths and weaknesses
summary_of_the_review: Overall, the experimental results on adaptaion tasks seems good, but lack of experiments on large-scale open-domain dataset like WMT.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission introduces a method to improve the efficiency of kNN-MT, by first retrieving a set of relevant translation pairs with BM25 and using these as a kNN datastore. The authors also introduce a simple but effective technique for dynamically choosing interpolation weights. 

The method is well motivated, and makes kNN-MT dramatically more practical by improving efficiency. The method for adaptive interpolation weights also simplifies previous work. The results are strong, and match or outperform the original kNN-MT.

One weakness is that there are not larger scale experiments on WMT19, which leaves it uncertain how well the method will scale. The authors note in their response that the baseline kNN-MT is too resource-intensive to re-implement in this setting, although a reviewer notes that they have re-implemented it on CPU. The technical contribution is also a bit limited compared to [1] and [2].

[1] Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Nearest neighbor machine translation. In ICLR 2021
[2] Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. Search engine guided neural machine translation. In AAAI, 2018.


justification_for_why_not_higher_score: The technical contribution is somewhat limited, and the paper doesn't evaluated the scalability on larger corpora. 
justification_for_why_not_lower_score: The proposed method is simple and scalable as advertised, and achieves strong empirical results, so I think it is a useful contribution to the community.
Decision: Accept: poster