Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a language model that can autoregressively choose tokens from a fixed vocabulary, or multi-token phrases from a pre-encoded corpus. The paper builds on previous work on phrase encoding for question answering, but extends this work to freeform text generation. The model (Copy-generator) is evaluated on three separate language modeling tasks. Two that use Wikipedia text and one that involves domain transfer to a legal domain. When compared to a vanilla LM and a KNN-LM that retrieves memories of individual tokens, Copy-generator performs better according to the recently proposed MAUVE metric. It does less well, in comparison to KNN-LM, according to diversity and Rep-N metrics on some tasks, though. Traditional perplexity scores are not particularly meaningful here because the model can choose different segmentations of the text than those used by the baselines.

The phrase encoder is trained along with the language model in the following fashion. First a greedy segmentation algorithm is run to choose a single segmentation of each training example into phrases that exist elsewhere in the training corpus. Then, for each phrase, a piplined retrieval system is run. First a document retrieval system retrieves a small set of documents that are similar to the prefix. Then, a phrase search is applied over all phrases in this retrieved set of documents. Phrases are represented using encodings of their start and end position, following previous work on phrase retrieval. The phrase retrieval model and language model are trained using a combination of InfoNCE loss over phrase candiates, as well as language model loss. The document retrieval model is fixed (as far as I can make out).

At inference time, phrases or individual tokens are selected from the union of all phrases in a large retrieval corpus and the fixed length vocabulary. Examples show that Copy-generator often chooses to select long phrases instead of individual tokens (I would like to also see statistics illustrating how often different length n-grams occur), and the generated text is deemed significantly better than the baseline systems according to the automatic MAUVE metric, which has been trained to correlate with human judgements. An analysis of performance shows that quality increases with the number of documents in the phrase corpus, and this increase is particularly pronounced when greedy search is used during decoding. 

strength_and_weaknesses: #### Strengths
- This is a novel and interesting architecture that is significantly different to previous work.
- Copy generator performs well in comparison to reasonable baselines.
- While the model is only applied to language modeling tasks, I can imagine extensions to other forms of generation where retrieving and copying text is likely to be a good strategy (e.g. multi-document summarization).

#### Weaknesses
- The paper relies heavily on the recently proposed MAUVE metric. It would be great to also have some human judgement of quality, no matter how small, to get more intuitions of where Copy-generator does better or worse than the baselines.
- Copy-generator makes use of context-dependent phrase encodings. However, the examples in Figures 2 and 5-8 seem to show that large phrases are being drawn from very different contexts to the target context. It'd be good to see some analysis of how important the is here.
- It would be interesting to see a comparison to retrieval augmented language models (e.g. RETRO, ATLAS) which also include document retrieval at inference time (but also have a greater inference-time computational cost than copy-generator).

clarity,_quality,_novelty_and_reproducibility: #### Clarity
If I understand this paper properly, the pipelined approach presented in Section 4.2 as 'implementation details' are essential during both inference and training. In particular, I believe that the BERT phrase encoder parameters are only ever updated for the small set of retrieved documents, and there is no need to re-index the phrase table during training (the phrase representation is operating more like a re-ranker). If my understanding is correct, then I suggest this detail is moved out from a paragraph focusing on 'inference efficiency' into the description of the core approach. Without this detail, the description of the training procedure is very hard to understand.

Otherwise, this paper is well written and easy to follow.

#### Reproducability
More should be said about the construction of the InfoNCE loss. How exactly are the in-batch negatives constructed? Are these all phrases in all retrieved documents for all prefixes in the batch? Also, what happens when there are multiple separate 'correct' representations of the target phrase $p_k$ coming from different contexts in the retrieved documents?


#### Novelty & Quality
Phrase encoding and retrieval has been studied before, as has contextualized token representation retrieval for language modeling. But, I believe this is the first work to do phrase retrieval for language modeling, or any type of long-form generation. That seems significant.

summary_of_the_review: This paper presents a novel approach to language modeling, that can copy large phrases from a retrieval corpus rather than decoding them one token at a time. The performance on three language modeling datasets are positive, according to the MAUVE metric and in relation to the similar KNN-LM baseline that performs retrieval of single token encodings. It would be nice to see some more analysis of the outputs, and how the copy mechanism is being used during inference time. It also would be very interesting to see how well this model does on a wider variety of text generation tasks. However, the current paper is sufficiently novel and interesting for acceptance at ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper formulates text generation as copying from existing text segments. The authors decompose text generation as a series of copy-and-paste operations by seeking text spans from existing articles and copying them as the generation result at each step. Experimental results on WikiText-103 show that the proposed method can achieve higher MAUVE scores and also support effective domain adaption and better inference efficiency.
strength_and_weaknesses: Strengths:

1) Reformulating text generation as a series of actions that fully depend on copying is an interesting idea, which may work well in some NLG scenarios.

2) From the experimental results, copy-generator can achieve better generation quality while having lower latency compared with Transformer-based baselines, which shows its effectiveness.

3) This paper is well-organized and easy to follow.

Weaknesses:

1) The idea which combines vanilla token-level generation and phrase copying has been already discussed in the existing works such as [1]. In my view, the main difference only falls into the format of knowledge sources (unstructured documents in this paper and structured knowledge in [1]). Thus, the authors should further clarify the difference between copy-generator and existing works to show their novelty.

2) The metrics used in the experiment are somewhat questionable. First, this paper lacks the fluency metric such as perplexity. From my experience, if copy operations dominate the generation process compared with token-level generation, the fluency of the whole sentence may be degraded. Then, MAUVE as a distribution-based metric may be biased to evaluate copy-generator because copy-generator largely copies source texts. If the distributions of source texts and the test set are similar enough, it’s not surprising that copy-generator can achieve a high MAUVE score.

3) The latest baseline selected in this paper is in 2020. The authors should add state-of-the-art baselines about retrieval-augmented language models such as [2].

4) Since the experiment is mainly conducted on WikiText-103, I wonder whether there is a principle to collect the source texts for a specific downstream task / dataset, such as machine translation, text summarization or dialogue generation. It’s important because the generation quality of copy-generator largely depend on the quality of source texts.

5) Typo: “all the e” in the second paragraph of Section 3 should be removed.

[1] Latent Relation Language Models. AAAI 2020.

[2] GNN-LM: Language Modeling based on Global Contexts via GNN. ICLR 2022.

clarity,_quality,_novelty_and_reproducibility: The overall quality of this paper is OK. But the authors should further clarify the differences between the proposed method and existing works to highlight their novelty. The reproducibility may be degraded due to the lack of codes.
summary_of_the_review: Although generating texts via a series of copy operations is interesting, I’d recommend the authors to solve the concerns about the novelty and experimental settings before making this paper ready for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The proposed Copy-Generator architecture is an attempt to depart from the classical probabilistic token-by-token language modelling perspective for language generation. Instead, generation is framed as gluing together phrases that are retrieved from a collection of documents. Searching for phrases at inference time is implemented in a heuristic coarse-to-fine-grained fashion to reduce the computational cost. The issue of computing a partition function over a potentially ginormous set of phrases is side-stepped by using an NCE loss in training, and either using greedy decoding or nucleus sampling (normalization over top-k items) for inference.
strength_and_weaknesses: I think that the idea is a reasonable follow-up to the kNN-LM approach of Khandelwal et al. The ability of adding in-domain data without retraining is very appealing. But there are some practical disadvantages compared to vanilla LMs that are brushed over to some extent. First, Copy-Generator does not define a probability distribution over a fixed vocabulary (as the authors admit in Sec. 4.3), and that limits its applicability to pure generation - it does not work in combination with other models (e.g. Bayesian, (re-)scoring...). More importantly, the idea of retrieving phrases from the training data contradicts the trend towards using bigger and bigger training sets. In fact, while latency is reported on Wiki-103 (which has 100M tokens, so rather small by today's standards), it is not reported on the En-Wiki corpus. But even En-Wiki with 3B tokens is not outrageously large nowadays.

A potential way around this latency concern would be a separate treatment of the training set and the document collection used at inference time. A good data point would be the En-Wiki Copy-Generator with an inference-time document collection trimmed down to match the latency of the baselines. More details about the sensitivity regarding the size/domain of the inference-time phrase table would be interesting. I think the note made in Sec. 5.3 "we can use learn COPY-GENERATOR with a smaller corpus but leverage
additional information in a larger corpus in a plug-and-play fashion" misses the point: we want to train with a large corpus, and plug in a small corpus at for inference. 

A better sense for the importance of the phrase table would also be provided by reporting Wiki-103 performance of the Copy-Generator with Law-MT phrase table. Is Copy-Generator more/less prone to catastrophic forgetting?

Minor comments:
- It is nice to know that the average length of phrases is 3.95, but more details (variance, full histogram..) would be useful.
- Phrase extraction by segmentation seems limiting as it does not allow any overlap between phrases.
- Why is the Copy-Generator latency for nucleus smaller than for greedy
- Abstract: coping -> copying 
- Sec. 3 "We denote all the e all the phrases"
- Sec. 3 "all phrases are context-sensitive" clarify that they depend on the source document context, not on the target prefix context at inference time.
- Sec 4.2: "does not suit work well"
clarity,_quality,_novelty_and_reproducibility: Novel architecture, good and clear writing style.

summary_of_the_review: The idea of tackling neural language generation on the phrase-level is neat, but the practical limitations (latency, sensitivity regarding the size and content of phrase table) are not fleshed out enough in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a retrieval-based LM that can generate phrases. Their base model is a standard transformer, and the retrieval component uses BERT. Their model is trained using in-batch positives / negatives. Retrieval is actually done at document-level, then phrase extracted using a segmentation algorithm. For evaluation, rather than focus solely on language modeling, they evaluate in a text generation setting. They primarily evaluate with MAUVE but also report other metrics useful for analysis. The results are strong compared to baselines, but text generation is very hard to evaluate and I have some questions / suggestions about this.
strength_and_weaknesses: # Strengths

- Strong results using MAUVE in all settings (although I find it hard to interpret MAUVE here, and there should be more effort to help readers calibrate their understanding of the reported metrics).

- The method can generate phrases instead of only single words (although I have questions about efficiency).

- The retrieval component uses rich context information.

# Weaknesses

- It is not ideal that COPY uses BERT for retriever and kNN-LM uses vanilla transformer trained from scratch. To me this is a major confounder, and maybe having a strong retrieval component (i.e. BERT) is more important than using phrase-based vocab.

- The rep-N metric is very strict (measures contiguous duplicate n-grams), and perhaps is useful for debugging, but I imagine it does not tell use much about generation quality. It would be more useful to include distribution novel n-grams with respect to ground truth. Can review RAVEN paper for guidance (McCoy et al., 2021. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN).

- The results are very reliant on the MAUVE metric, but for wikipedia text, factual correctness is very important and I am not sure how will this is reflected by MAUVE. Although there is much discussion about perplexity being a poor measure of text generation, here, I think it would have been helpful since it could reflect factual correctness to some degree. Unfortunately, authors mentioned that PPL is not reliable because of the phrase-based vocabulary --- I wonder if it would be possible to still compute PPL in some settings if not all (certainly the baselines support PPL at least). In any case, some human eval may be appropriate here so we can better calibrate the interpretation of MAUVE in this setting.

- It would help considerably to have other baselines. This might alleviate concerns about MAUVE too.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and the idea is clear. I am not sure of other methods that so simply enables phrase-based vocabulary. I have some questions about method and efficiency. My major concerns are about evaluation with MAUVE which I mentioned as weaknesses.

Is phrase encoder updated every batch?

How large is the "effective" vocab during training vs test time? I suppose this is function of the number of retrieved documents, k = 50.

How do you handle duplicate phrases in the softmax?

I am surprised to hear COPY is faster than vanilla transformer. Sure, can generate more tokens at once, but don't these tokens still need to be processed by the LM after generation? Also, I imagine the size of the softmax is much larger in the COPY approach, although I am not sure. I suppose with K = 50 documents, if there are only a few thousand phrases added to the vocab then maybe it is not such a problem. Along these lines, what size is the word-level vocab used here?

How is it possible to retrieve phrases from local context (like in Fig 2, "Boulter is an")? This was not explained in the main text AFAICT.

One minor comment about the intro... It should be more clear in the intro that Wikitext-103 is not being used in the standard way, and in fact, is not being used as a language modeling benchmark since there is no perplexity eval.

"average inference time cost" --- is it average over words in the test set?

typo: "We denote all the e"

typo: "i is the maximum valid value", I think "i" is not supposed to be the max value here.

nit: Can be confusing that top-k is used in so many different ways. Maybe k with different subscripts will help?
summary_of_the_review: The work is very interesting and relevant to the community. The main issue is the authors have taken a language modeling task for Wikipedia and converted it into an unconditional text generation task. In unconditional text generation, there are many valid and diverse continuations of text. But in Wikipedia especially I believe that factual information is important to get right, making it hard to interpret some of the evaluation. Human evaluation is one way to immediately address this concern, but perhaps other evaluations besides MAUVE could be useful too. I mentioned perplexity, but maybe BLEU with small sliding window could help understand factual correctness of the model AND leverage the phrase-based aspect.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose a new method for text generation, relying on retrieval and copy of chunks of tokens that are relevant to continue the text. The method is inspired by retrieval augmented LM, such as KNN-LM, but introduces the following technical contributions: (1) the method retrieves full chunks, meaning that it can generates multiple tokens in one step (2) a new training algorithm, based on contrastive learning.
The reviewers enjoyed this paper, as it opens a new research direction for text generation. They found the paper well written, and believe that this method will lead to interesting follow-up works. However, there is a large concern over evaluation of the method, which mostly rely on the recently proposed automatic MAUVE metric. I am really on the fence for this paper, as I believe that ICLR should publish exploratory work, even if the evaluations are a bit weak. However in this case, I also believe that the authors could have done more to convince readers that the method is promising, by reporting more automatic metrics, and performing more careful human evaluations. I also believe that additional baselines, which copy large amount of text, should be included in the automatic metric.
justification_for_why_not_higher_score: I have concerns about the evaluation of the method, and believe that more careful evaluation should be conducted to make the paper stronger.
justification_for_why_not_lower_score: The paper propose a novel and interesting research direction, and some reviewers expressed a strong interest in building on top of the method.
Decision: Accept: poster