Copy is All You Need
Authors: Tian Lan
Deng Cai
Yan Wang
Heyan Huang
Xian-Ling Mao
Venue: ICLR 2023 poster (2023)
Keywords: neural text genertion

Abstract: The dominant text generation models compose the output by sequentially selecting words from a fixed vocabulary. In this paper, we formulate text generation as progressively copying text segments (e.g., words or phrases) from an existing text collection. We compute the contextualized representations of meaningful text segments and index them using efficient vector search toolkits. The task of text generation is then decomposed into a series of copy-and-paste operations: at each time step, we seek suitable text spans from the text collection rather than selecting from a standalone vocabulary. Experiments on the standard language modeling benchmark (WikiText-103) show that our approach achieves better generation quality according to both automatic and human evaluations. Besides, its inference efficiency is comparable to token-level autoregressive models thanks to the reduction of decoding steps. We also show that our approach allows for effective domain adaptation by simply switching to domain-specific text collection without extra training. Finally, we observe that our approach attains additional performance gains by simply scaling up to larger text collections, again without further training.\footnote{Our source codes are publicly available at \url{https://github.com/gmftbyGMFTBY/Copyisallyouneed}.}

Published as a conference paper at ICLR 2023
COPY IS ALL YOU NEED
Tian Lan ,, Deng Cai
Yan Wang
9.1
Heyan Huang
Xian-Ling Mao
Tencent Lab
School of Computer Science and Technology, Beijing Institute of Technology
(lantiangmf tby, chisisjcyked, yanwang..rrrrr hrttwangbbanneeneee. branden)@@gaill com
(hhy63, maox1 yebi edu.
ABSTRACT
The dominant text generation models compose the output by sequentially selecting words from a fixed vocabulary. In this paper, we formulate text generation as progressively copying text segments (e.g., words phrases) from an existing text collection. We compute the contextualized representations of meaningful text segments and index them using efficient vector search toolkits. The task of text generation then decomposed into a series of copy-and-paste operations: at each time step, we seek suitable text spans from the collection rather than selecting from a standalone vocabulary. Experiments on the standard language modeling benchmark (WikiText- 103) show that our approach achieves better generation quality according to both automatic and human evaluations, Besides, its inference efficiency comparable token-level autoregressive models thanks to the reduc- tion of decoding steps. We also show that our approach allows for effective domain adaptation by simply switching to -omainn text collection without extra training. Finally, observe that our approach attains additional performance gains by simply scaling up to larger text collections, again without further training
INTRODUCTION
Most neural language models (LMs) process text generation tasks by making series of next-token predictions in an autoregressive manner (Radford et al. 2019 Dai et al. 2019 Khandelwal et al. 2020, [hi al. 2022). Specifically, LMSs generate the next-token distribution over fixed vocabulary for any given prefix. Then, the next token is selected by a chosen decoding method, such as greedy search and nucleus sampling (Holtzman al. 2020). This process continues until some stop condition is reached. For example, a special end-of -oneeerteenertionn token is emitted, or the generated text reaches the maximum length limit.
Unlike traditional neural language models, we reformulate text generation by copying text segments from existing text collections. The text segments can be of variable lengths, including single words and multi-word phrases. For clarity, we will use the term "phrase" to refer any contiguous text segments, and single word can also seen as a phrase of length 1. We compute a contextualized vector representation for each phrase and pack them into offline index. At each decoding step, a suitable phrase is retrieved from the offline index and appended to the current prefix. In other words, the next-token predictions in traditional neural language models are replaced by series of copy-and-pee paste operations.
Our proposed model, named CoG (short for COPY- KOPY-GENRRATOR,, enjoys the following advantages. First, our method selects phrases in specific contexts rather than standalone tokens in fixed vocab- ulary. It potentially allows for method allows training-free adaptation to new knowledge sources because the text collection can be updated plug-and-play fashion. It could benefit application scenarios such domain adaptation and data expansion/ filteringg. Third, our method allows sequence of multiple tokens (i.c., multi-word
accurate candidate representation and selection. Second, our
Contributed Equally.
1 Corresponding authors.
Our source codes are publicly available at httpt/////.tttt.cc.a/a//////////

---

Published as a conference paper at ICLR 2023
phrase) to be generated in one single step. It could reduce the total number of decoding steps, leading to improved inference efficiency.
We conduct extensive experiments verify the effectiveness of our proposed CoG. On the standard language modeling benchmark (WikiText- 103), our proposed CoG substantially outperforms standard baselines automatic metrics (26.14 14 23.43 MAUVE (Pillutla et al. 2021 and human evaluation (48% vs. 28% human preference). Moreover, when we directly switch the text collection from the WikiText-105 103 corpus to a domain-specific corpus, Law-MT (Kochn & Knowles 2017), our proposed CoG outperforms strong baselines on this domain adaption setting (28.14 vs. 26.85 MAUVE and 52% vs. 36% human preference) without any domain-spectin up the text collection of CoG to larger one, the En- Wiki dataset, we obtain additional gain (26.97 vs. 23.43 MAUVE), again without any further training. Our contributions can be summarized as follows:
training. Furthermore, when we scale
We propose CoG, a method that reformulates text generation tasks as series of copy-and- paste operations from existing text collections.
We show that CoG can outperform standard neural language model baselines on existing language modeling benchmarks.
We demonstrate that CoG allows for training-free adaptations to larger text collections and domain-specific text collections.
2 BACKGROUND: NEURAL TEXT GENERATION
Neural text generation can be divided into two categories: (1) unconditional text generation; (2) conditional text generation. Unconditional text generation (or language modeling) aims to generate coherent text continuation given a prefix. In this case, language models perform generation using density estimation over sequences po(x)) (ar). Conditional text generation aims to generate text with some condition c and instead estimates the probability of pa (x]c). Its typical applications include machine translation (Sutskever al. 2014 [Bahdanau et al. 20155,, summarization (See et al. (2017). Throughout this paper, our discussion will be focused on unconditional text generation, however, our approach can be readily adapted to conditional text generation as well.
The canonical approach to language modeling factors the generation in an autoregressive left-to-riee manner Po (= = II= (xx)) In this case, text generation is reduced the task of repeatedly predicting the next token conditioned on the partial sequence (i.e., prefix) generated so far p(z; ((ra)))))) The model often consists of two parts: (1) prefix encoder and (2) set of token embeddings. The prefix encoder is often parameterized by the Transformer architecture (Vaswani et al. 2017). which transforms any prefix into a fixed-sized vector representation h4 E IRd PrefixEEncderr (F Then, the probability of the next token being is calculated as
exp(tw h4)
pa (a( = w 2)) Ewev exp(Uw h:))
where Ua. is the context-independdee token embedding representing the token w, and V is the pre- defined vocabulary consisting of all possible tokens. Based the chosen decoding method, such as greedy search and nucleus sampling (Holtzman et al.) (020), the next token is selected according to the probability distribution over the fixed vocabulary This process is repeated an autoregressive manner, until some stop condition is reached, e.g., the maximum length of generation is reached.
3 COPY-GENERATOR
Unlike traditional language models that compute the next token distribution over a fixed vocabulary that is usually composed of words or sub-words (Sennrich et al.) 2016) Kudo & Richardson 2018), our proposed CoG has a dynamic "vocabulary" that dependent on the available source text collections. Each item in the "vocabulary" corresponds to text segment (termed phrase in this paper) in the source text collection. Importantly, all phrases are context-sensitrve That is, the same phrases in different contexts considered to be different. The overall framework is depicted in Figure
Formally, our approach assumes a set of source documents (D',..., } is available. For each document D', a phrase k: Dt of length + can be extracted, where and mark the start
2

---

Published as a conference paper at ICLR 2023
Phrase Table
Phruse Encoder
MLP
MLP
... in theaters on October 22, 2021 in United States
and extremely well releireceived by crities and audiences
the film premiered 78th
gaan 0000
BERT model
September 3, 2021. [ O the
[ in
[
O0
0000
and audiences
Source Text
Token Embeddings
Collection
Copy
Prefix Encoder
Maximum Inner Product Search
[...]
I...II...]
The released thenters October 22, 2021 United States] [and extremely well-roceived critics
and audiences] [Before] [thad]
[the film premiered at the 78th International Film Festival September 2021.]
The overview of our proposed CoG, Given the prefix The Dune film was released, CoG retrieve 3 phrases (in different colors) from the documents and generates 3 tokens (Before, that, and the comma.) .) from the fixed vocabulary form the whole generation.
Figure 1:
and end positions of the phrase in the document, respectively. We denote all the phrases the source text collection as P. For given prefix Teg we aim to select the best phrases that can form a coherent prefix. To this end, we compute a contextualized representation for each phrase Pk: _ PhraseEncoder((s
text continuation following
D') using phrase encoder. Thus, phrase table {(k, |k) E P can be constructed. Similar to traditional language models, at test time, CoG also employs prefix encoder to map the prefix Z<i into vector representation q:- The fitness of phrase to the prefix dςi is then measured by the dot product of their vector representations pt and gi:
p(k( x exp(p= 4)).
At each time step, a suitable phrase is selected and appended to the current prefix accordingly.
Note that the size of the phrase table can be up billions. To search over this large candidate pool, we pre-compute the phrase representations and use a coarse-to- search pipeline based on maximum inner product search (MIPS) (Johnson al. 2019). The details deferred to Section 4.2 Moreover, to support the scenarios where no suitable phrases are available, we also add the context-independent token embeddings (ww, (w) V} in standard LMs the phrase table.
Ethical Consideration The text generated by CoG contains text segments copied from other documents, which may cause copyright disputes in real-world applications. Therefore, there are few things to be considered: (1) The copyright of the source text documents needs to be carefully checked. One should not use documents with strict copyright protection and/or private information; (2) is recommended to quote the original source explicitly, especially when the retrieved phrases are long.
3.1 MODEL ARCHITECTURE
As illustrated Figure 1 our proposed model consists of three major components: (1) a prefix encoder that maps prefixes to fixed-sized representations; (2) context-dependent phrase encoder that computes the vector representations of the phrases in the source text collection; (3) set of context-indeppeddnt token embeddings similar to the one used in standard neural language models.
Prefix Encoder The prefix encoder is responsible for encoding the prefix x<ii into a vector represen- tation for the next-phrase prediction. We treat the prefix sequence of tokens (previously predicted phrases are split into tokens as well) and encode them using the standard Transformer architecture with causal attention (Vaswani et al. 2017 Radford et al.) 2019). Causal attention only allows each position the input sequence to attend to its preceding positions. Therefore, the prefix representation the generation progresses, leading faster inference. Concretely,
can be computed incrementally the prefix encoder transforms a prefix Ø2i of length i into matrix H. E gixdL where d is the hidden dimension and L is the number of Transformer layers. The computation can be written as:
H4I = PrefixEncoder(2,, H4))
We the hidden state of the last token the prefix representation /j-
3

---

Published as a conference paper at ICLR 2023
Phrase Encoder
Given set of source documents (D'...,, D"), the phrase encoder computes the Vector representations of all the phrases in the documents. Inspired by previous work (Lee et al.) 2016)] construct context-dependent phrase representations follows. For a document D D1.-.. Di...., Dm of length m, we first apply deep bidirectional Transformer (Devlin al. 2019) obtain contextualized token representations 'D E Rmxxd, where de is the dimension of token representations. Then, we apply two MLPs models, MLPstart and MLP end. to
Sco et al.) 2018 Lee et al. 202I),
convert D into start and end token representations Dstart; Dend E m respectively:
D start MLP, start (D)) Dend MLP end (D).
For each phrase see that starts at s and ends at in the document, we use the concatenation of the corresponding start and end vectors the phrase representation.
PhrascEncoder(ss: e, D) = [Dstart [s]; Denu[e]] E IR.
The advantages of the above representation method are that (1) we only need to encode the document once obtain all phrase representations; and (2) we only need to store all the token representations instead of all phrase representations.
contextInndpendee Token Embeddings Although CoG can copy phrases from other documents, would like retain the generalization capability compose output with standalone tokens. This can be especially useful when there is no suitable phrase in the source text collection. Therefore, we also add the traditional context-independent token embeddings V E RIV xd to our phrase table. These tokens can be seen as phrases of length without any context information.
3.2 MODEL TRAINING
CoG decomposes the task of text generation into series of copy-and-paste operations: at each time step, it selects the next phrase either from the source text collection or the fixed token vocabulary. In other words, phrases document in the training set is chunked into sequence of phrases in a similar spirit. Specifically, we propose a greedy segmentation algorithm based forward maximum matching. Taking document D Di...., Dm of m tokens as an example, our algorithm segments the document from left to right. The first tokens will be cut out as a phrase if it can be found documents and is the maximum valid value. The above process is repeated until all tokens are cut out. Note that some resultant phrases can be single tokens in the fixed token vocabulary when no proper matching can be found. Detailed explanations of the phrase segmentation algorithm can be found in Appendix
used the basic building blocks for text generation. To train CoG, each
sub-sequence in other
Suppose that document D has been split into phrases D PI....: Pn. If the k-th phrase Ps is copied from another document, let D* be the source document and let 8k, 6g be the start and end positions of pę. in D the phrase encoder is used to extract its context-dependent phrase representations Eucodernc,,,r(Ss, Ek, D*) (Eq. 2). On the other hand, we directly retrieve the context-independent token embedding of pı if it copied from the fixed token vocabulary. As illustrated by Eq. D CoG relies on shared vector space of prefix and phrase representations, where the representations of semantically coherent prefixes and phrases should be closer each other while others should be pushed apart. We define the training loss for next-phrase predictions by using the InfoNCE loss with in-batch negatives (Karpukhin al. 2020):
exp(yx Pk)
C,
log
EpEp, exp(at Pp) Ewev exp(q)- U.e)
k=
where Px consists of all the phrases the source document D* the set of all tokens in the token vocabulary, and yj denotes the representation of the prefix preceding the phrase ps in D.
Additionally, to retain the capability of token-level generation, we also train CoG with the standard token-level autoregressive loss.
m
exp(gi, UD.) log
L = m Ewev explgi Uw)
4

---

Published as a conference paper at ICLR 2023
where di denotes the prefix representation preceding the token D, in D. Finally, the training loss is the sum of these two losses:
L = Lp Le
EXPERIMENTAL SETUP
4.1 BASELINES
We compare CoG with the following three baselines:
Transformer (Vaswani et al. 2017 has been the de facto model for neural language models. Concretely, we fine-tune the pre-trained GPT2 model (Radford et al. 2019) in our experiments.
&NN-LM (Khandelwal al.
2020 is retrieval-augmenner generation model, which ex-
tends a pre-trained neural language model by linearly interpolating its next token distribution with k-nearest neighbors (kNN) model.
RETRO (Borgeaud et al. 2022 another retrieval-an -agaral---meeedddd generation model which combines a frozen BERT retriever, a differentiable encoder and a chunked cross-attention mechanism predict next tokens. Since there is pre-trained RETRO model that could be accessed, we train it from scratch on the WikiText- -00T dataset.
4.2 IMPLEMENTATION DETAILS
All the baselines and our source codes are based on the popular Huggingface transformers package (Wolf et al. 2020). For fair comparison, the prefix encoders in Transformer, ANN-LM, and CoG use the same model architecture as the pre-trained GPT2 model (12 layers, 12 heads, and 768 hidden dimensions) (Radford al. (2119). For the phrase encoder in CoG, we fine-tune the pre-trained BERT-base-cased model (Devlin et al. 2019) (12 layers, 12 heads, and 768 hidden dimensions). We train baselines and CoG for 400,000 steps on 8 Tesla- V100 GPUs. For all the baselines, the learning rate, dropout rate, and gradient clipping are set as 5--5, 0.1, and 1.0. respectively. Due to memory limitation, the batch size set to contain 256 phrases. For the BERT model the phrase encoder, the maximum sequence length is set as 256. For the GPT2 model in the prefix encoder, the maximum sequence length set as 512. Our proposed CoG contains overall 248M parameters from BERT and GPT2 models, and other baselines contain over 124M parameters. As suggested by |Boreaaud et al. (2022), the parameters and a of kNN-LM are set as 0.118 and 0.00785, respectively.
To improve the inference efficiency of CoG, encode all the documents in the source text collections offline. Note that retrieving from such a super large phrase collection faces severe challenges on the engineering side. This paper uses coarse-to tere-t--fin to-feee pipeline address this challenge. Specifically, first use a document retriever to retrieve top-k related documents for each given prefix. Then, their corresponding phrase representations matching model, DPR (Karpukhin et al. (2200) and vector search toolkit, FAISS (Johnson et al.) 2019) are used as the document retriever, which can recall documents that have similar topics with the prefix. The value & is empirically set to 1024.
collected for selection. In this paper, popular semantic
CoG can be used with both greedy search and nucleus sampling. For greedy search, CoG selects the phrase that has the highest fitness score at each time step. As for nucleus sampling, we first obtain the next-phrase distribution by using the sof tmax function over the fitness scores of all candidate phrases. Then, the next phrase is sampled over this distribution.
More details of the implementation can found in Appendix A and
4.3 AUTOMATIC EVALUATION METRICS
For each document in the set, we use the first 32 tokens as the prefix. The baselines and our proposed CoG generate text continuations of length 128 based on the same prefix. Following
5

---

Published as a conference paper at ICLR 2023
conventions (Welleck et al. 2020 Su et al. 2022), greedy search and nucleus sampling
(Holtzman et al. 2020) (p- 0.95) throughout our experiments. Following previous work (Welleck
et al. 2020 Su et al. 2022) and report the results the following evaluation metrics:
MAUVE (Pillutla et al. 2021), efficient, interpretable, practical automatic evaluation, is highly coherent with human judgments and widely used to evaluate modern text generation models (Su et al. 2022 Krishna et al., 2022). In this paper, MAUVE leverages the GPT2- large model to generate the scores, and the scaling factor is set as 2.0.
Rep-n Welleck et al. 2020) measures the sequence-level repetition as the portion of duplicate n-grams the generated (Welleck al. 2020). For a generation text x, Rep-n can be formulated as: 100 x (1.0
lunique gram(x)
Higher Rep-n denotes the severe
[total a- gg)mm(m]
degeneration problem in generations.
Diversity (Welleck et al. 2020) measures the diversity of the generations, which is formu- lated as 2 Rep-1
Generations that have higher Diversity scores usually are more
100
informative.
Note that previous work (Khandelwal et al. 2020 Dai et al. 2019) often uses perplexity the primary evaluation metric to measure the performance of language modeling. However, since our proposed CoG does not calculate next-token distributions perplexities is not reliable and thus omitted. However, we can test the perplexity of generated text using external language model, and the results are shown in Appendix C
fixed vocabulary, the comparison of
EXPERIMENTAL RESULTS
In this paper, we evaluate baselines and our proposed CoG three different settings: (1) standard language modeling; (2) domain adaption; (3) enlarged phrase index.
5.1 LANGUAGE MODELLING ON WIKITEXT-LO3
In this setting, models trained the training set of the WikiText- 103 dataset and evaluated on its test set. The WikiText-103 dataset (Merity et al. 2017) contains an extensive collection of Wikipedia articles with over 100 million words, which widely used to evaluate the performance of universal
language modeling KKandeewal et al. 2020 Dai et al. 2019 Su et al. 2022).
Model
Decoding MAUVE
Rep-2.
Rep-3]
Rep-4
Diversity?
Latency (s)
greedy
19.87
43.56
38.55
35.5
22.37
1.32
Transformer
nucleus
23.43
5.10
1.33
0.50
93.22
1.48
greedy
19.92
43.79
38.76
35.69
22.13
10.36
ÀNN-LM
nucleus
22.50
3.33
0.69
0.21
95.8
10.42
greedy 21.19
44.65
39.63
36.6
21.19
4.39
RETRO
nucleus
22.86
6.21
1.93
0.86
91.19
4.51
greedy 26.01
28.14
23.80
21.40
43.03
1.29
CoG
nucleus
26.14
7.31
2.66
1.28
89.07
1.54
Table 1: The automatic evaluation on the test set of WikiTeext-003. As for each model with nucleus sampling, we run 10 times and recorded the average MAUVE and Diversity scores.
Results Table T shows the performance comparison between the baselines and our proposed CoG on the test set of the WikiText- 103 corpus. can be found that our proposed CoG substantially outperforms the Transformer and &NN-LM baselines most metrics. Specifically, CoG improves MAUVE score over the best baseline (Transformer with nucleus sampling) from 23.43 to 26. 14 an improvement of 2.71. Interestingly, although it is well known that greedy search could raise severe degeneration problems (Welleck et al.) 22200), CoG with greedy search still outperforms the standard Transformer baseline with nucleus sampling, with 2.58% improvements on MAUVE. This

---

Published as a conference paper at ICLR 2023
a staning nol the tolesision poetrayed Scot Parr the spieods"
The Man Top
Ho alsii
inselved in
in inorgiing
Decausst
Dareewal
Input Prefix
The regular qpisods The Trp Took: Origisl
Metellias Heleebo abu inflaeneed by onchoeral and clasical composer io.
However. Jenyy Huddin, was incalcaal ininether morph to claracio Deep Thrns ...
acunne fation
drmm scrics
Callst ls
Magic Boundhheur ielestsioo scries
The Man Trap He is also invoired in in ceusing acience fictimn drama sario called the Magic Roustabou Ielesision sofice
Decement
Ther with the high tech science fiction fron Man...
she hozame the youngoal actiess ln druma series when ste Stured
Eric Thompson who sarrated The Magic
Roundabout televisoo sciies, wis bocs in lasae un Jemyy Sioce
Generated Continuation
Figure 2: An example generated by CoG on the test set of WikiText-103, The dotted squares denote that the content (highlighted in red jis copied from the token vocabulary, and the solid squares denote that the content (highlighted in blue is copied from other documents.
observation demonstrates that CoG is more robust and less prone to the degeneration problem, which can be considered additional bonus.
Inference Speed Furthermore, we also compare the average time cost of different methods for completing the generation on the test set. Since the phrase representations in CoG are pre- computed offline, its encoding time cost is not included. The results are reported in Table [ As seen, CoG still achieves comparable inference efficiency with the standard Transformer baseline. The reason is that the copied phrases usually contain multiple tokens (the statistics of phrase length are shown in Table 2 As a result, CoG uses fewer decoding steps when generating the text of the same length. Unlike CoG that uses coarse-to-fine search pipeline, kNN-LM conducts large-scale vector search at every decoding step. Its inference latency is much higher than Transformer, and CoG, which is aligned with previous work(Alon al. 2022).
Method Uni-gram
2-gram
3-gram
4-gram
5-gram
6-gram
Greedy Nucleus
0.583
0.195
0.121
0.056
0.029
0.017
0.434
0.219
0.181
0.09
0.048
0.028
Table 2: The statistics the length of the copied phrases (on the test set of WikiText-103).
Human Evaluation To ensure the reliability of our evaluations, we also run human evaluation with three native-speaker graders from third-party grad- ing platform. Specifically, we randomly select 100 test prompts. For each test prompt. the annotators are given two continuations, in random order, which generated by CoG and Transformer respectively. The annotators are asked to decide which one is better by considering the following aspects:
Comparison CoG vs. Transformer 48%
Better No Prefer.
Worse
24%
28%
Table 3: Human evaluation the WikiTeet-- 103 corpus.
Fluency: Whether the generated text is fluent and easy to understand.
Informativeness: Whether the generated text is diverse and contains interesting content.
When annotators make different decisions on the same sample, we ask them to have discussion and make the final decision. As shown in 3 our proposed CoG model significantly outperforms strong Transformer bascline, indicating its better generation quality.
Case Study For a better understanding of the performance of CoG, we present an example of the text continuations generated by our proposed CoG in Figure 2 It can be found that CoG can retrieve phrases that are semantically coherent and fluent for given prefixes. For example, at the second decoding step. CoG generate the punctuations [".-] from the pre-defined vocabulary to close the film name "The Man Trap" and the sentence. Besides, the ninth decoding step, CoG directly copied the named entity Magic Roundabout television series from the related document. More examples can be found in Appendix E
7

---

Published as a conference paper at ICLR 2023
5.2 DOMAIN ADAPTION ON LAW-MT
In the domain adaption setting, the models trained on the WikiText- 103 dataset are tested on a specific domain. Following previous work (He et al. [022]] Alon et al. 2022), we use the English part of Law-MT (Kochn & Knowles 2017). which is an English-German translation dataset for law documents. The memory of KNN-LM, RETRO and CoG are constructed from the training set of Law-MT. We also present the performance of Transformer baselines with or without further fine-tuning on the training set of Law-MT.
Results As shown in Table it can be ob- served that CoG even outperforms the Trans- former model further fine-tuned on the Law- MT corpus (Transformer w/ FT). Specifically, CoG outperforms Transformer w/ FT by 2.93% MAUVE score. The results indicate that CoG allows a single model to be specialized in dif- ferent domains, by simply switching the source text collection. Although &NN-LM brings in higher Diversity scores, CoG surpasses it by 3.39% MAUVE score, which shows CoG has higher generation quality in general.
Decoding MAUVE
Diversity
greedy
20.32
70.66
Transformer w/o FT
nucleus
25.21
93.88
greedy
23.00
80.52
Transformer FT
nucleus
26.85
90.14
greedy 23.31
19.85
ANN-LM
nucleus
greedy 18.70
71.14
RETRO
nuclous 20.35
94.81
greedy
21.31
84.32
CoG
nucleus
28.14
92.56
Table 4: The automatic evaluation on Law-MT.
Human Evaluation We also conduct the hu- man evaluation on the Law-MT corpus, which has a similar setup to that in ($5.1). Table 5 shows that most of CoG's generations are bet- ter than strong Transformer baseline. This observation demonstrates that CoG can even outperform the fine-tuned Transformer bascline without any domain-specific training.
Comparison
Better
No Prefer.
Worse
CoG vs.
52%
12%
36%
Transformer w/
Table 5: Human evaluation Law-MT.
5.3 ENLARGED PHRASE INDEX WITH EN-WIKI
In the enlarged phrase index setting, make use of a large text collection, the En-Wiki corpus, and test baselines on the test set of WikiText--00 103. The En- Wiki corpus contains large-scale collection of Wikipedia articles with over 3 billion words, whose size is much larger than the WikiText-103 dataset. The memory of &NN-LM, RETRO, and CoG are built from the training set of En-Wik Similar to the domain adaption setting, we also present the results of Transformer baselines with or without further fine-tuning on the En Wiki corpus.
Results The experimental results are shown in Table 6 CoG with En-Wiki memory surpasses other strong baselines and CoG with WikiTextt 103 memory. This is especially remarkable because CoG does not require any additional training, suggesting we can train CoG with a smaller corpus but leverage additional information in larger corpus in plug-and-pla domain adaption setting, we also notice that, although kNN-LM baseline improves Diversity scores, it obtains much lower MAUVE score than CoG (23.39 26.97). Note that the Transformer w/ FT is slightly worse than that without fine-tuning on the En-Wiki dataset. This phenomenon is mainly because there are deviations between En En-Wiki and WikiText-10 103 datasets.
fashion. Similar to the
Effects of Index Size To further investigate how the size of the phrase index affects the generation quality, we randomly sample several subsets of the En-Wiki dataset with proportions from 0.1% to 100%. As shown in F when the proportion is less than 1%. CoG exhibits similar quality, which is unsurprising since few enlarged documents are added the phrase index. contrast, once the proportion is larger than 1%, the larger the phrase index becomes, the better generation quality the model achieves.
Due the hardware limitation, RETRO uses the subset of the En-Wiki corpus (over million chunks).
8

---

Published as a conference paper at ICLR 2023
Model
Decoding M
Diversity
27.0
greedy
19.87
22.37
Transformer w/o FT
nucleus
23.43
93.22
26.5
greedy
20.21
19.62
26.0
Transformer FT
nucleus
92.92
25.5
ANN-LM
greedy
23.21
20.33
ASUVI
25.0
nucleus
23.39
96.37
24.5
greedy 19.75
21.15
RETRO
24.0
nucleus 22.87
91.09
Transformer w/o
23.5
greedy 24.68
40.45
CoG
nucleus
26.97
90.00
0.0 0.001 0.003 0.01
0.03
0.1 0.3 1.0
The proportion enlarged En Wile Documents
Table 6: The automatic evaluation on the test set of WikiText-I0F, the memory built on the train set of En-Wiki. Transformer w/ FT and Trans- former w/o FT denote the Transformer baseline with and without further fine-tuning on the train set of En- Wiki, respectively.
Figure 3: Generation quality of CoG with differ-
ent sizes of the phrase index. For each proportion (point in the X-axis), sample 10 times and record the averaged MAUVE score. A propor- tion of 0.0 indicates that only documents from WikiText- 103 are used.
RELATED WORK
The dense retrieval technique (Karpukhin et al. 2020) has been widely used in many downstream NLP tasks, such open-domain question answering (Karpukhin et al. 2020 Lee et al.] (02I), open-domain dialogue systems (Lan et al. (021) and machine translation (Cai ct al., 1). Different from the traditional sparse retrieval system, such BM25 and TF-IDF
Dense Retrieval
(Robertson & Zaragoza 2009), dense retrieval learns a shared vector space for queries and documents, where relevant pairs of query and document have smaller distances (i.e., higher similarities) than the irrelevant pairs.
The most closely related work to study is DensePhrase (Lee et al. 2021). DensePhrase reformu- lates the muestion-ansserrine task as a phrase retrieval problem, where phrases are directly retrieved and returned as answers to factual questions. Differently, work aims generate coherent text continuations through multiple rounds of phrase retrieval. Since the connection between two adjacent phrases should be coherent and fluent in the text generation task, is much more difficult.
Aaerivval-Augmentee Text Generation (RAG) Retrieval- -urieeeee- text generation has gained increasing interest recently. Most prior work improves the generation quality (c.g., informativeness) of language models by grounding the generation on a set of retrieved materials (e.g., relevant documents)
(Li et al. 2022 Guu et al., 2020; Hashimoto et al., 2018 Weston et al.) 2018 Cai et al. 2019a b Khandelwal ct al. 2020 Wu et al. 2019] Guu et al. 2020 Lewis et al. 2020 Borgeaud et al. 2022
Yang et al. 2023 Our work is this line of research but takes radical step forward. Unlike prior
work that builds the combinations of retrieval and generation, retrieval is generation in CoG.
One contemporary work to work is Min et al. (2022), which shares the idea of replacing the fixed vocabulary with a nonparametric phrase table. However, Min et al. (2022) focuses on masked language modeling while our focus is causal language modeling and text generation.
CONCLUSION
In this paper, we reformulated text generation as progressively copying phrases from the massive text collection. Following this formalization, we proposed novel neural text generation model, named CoG, which generates text by retrieving semantically coherent and fluent phrases from other documents. Experimental results proved the advantages of CoG over the strong baselines three experimental settings: standard language modeling (WikiText- 103), domain adaptation (Law-MT), and enlarged phrase index (En-Wiki).

---

Published as a conference paper at ICLR 2023
JUSTIFICATION OF CHANGES
Note that the experimental results in the current version have some changes from the previous version that has been reviewed. We made number of revisions the experiments according to the valuable suggestions from the reviewers.
ACKNOWLEDGEMENT
The authors thank the anonymous reviewers for their valuable suggestions and comments our paper, which significantly improves the quality of our paper.
REFERENCES
Uni Alon, Frank Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig. Neuro- Neuro-symbolic language modeling with automaton-auu -angeett-- retrieval. In International Conference on Machine Learning. PMLR, 2022.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning. PMLR, 2022.
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. Skeleton-to- response: Dialogue generation guided by retrieval memory. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019a.
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. Retrieval-guided dialogue response generation via matching- tonge-eratonno framework. In Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 1866 1875, Hong Kong, China, November 2019b. Association for Computational Linguistics. doi: 10. 1/1.8866/1//D11-1199 URL htttps://aclanthollle ortg/19-11/D9
Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. Neural machine translation with monolingual translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), 2021.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-Xe XLL Attentive language models beyond a fixed-length context. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wer Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.0880 08909, 2020.
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. A retrieve-and and-edit framework for predicting structured outputs. Advances in Neural Information Processing Systems, 2018.
Junxian He, Graham Neubig, and Taylor Berg- Kirkkarikt..... Efficient nearest neighbor language models. arXiv preprint arXiv:2 109. 022 2021.
10

---

Published as a conference paper at ICLR 2023
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3), 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation, Proceedings of the First Workshop on Neural Machine Translation, 2017.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit lyyer. Rankgen: Improving text generation with large ranking models. arXiv preprint arXiv:2205.009 099000 2022.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018
Tian Lan, Deng Cai, Yan Wang, Yixuan Su, Xian-Ling Mao, and Heyan Huang. Exploring dense retrieval for dialogue response selection. arXiv preprint arXiv:2110.0660 2021.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), 2021.
Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Be- rant. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436. 2016.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. retrieval-augmenter generation for knowledge-intensee
NLP tasks. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmeened text generation. ArXiv, abs/2202.01 100, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zeutlemoer. Nonparametric masked language modeling. arXiv preprint arXiv:2212.01344 2022.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8), 2019.
S. Robertson and H. Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trendy Inf. Retr., 3, 2009.
11

---

Published as a conference paper at ICLR 2023
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer-generator generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), 2016.
Minjoon Seo, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannanch Hajishirzi. Phrase- indexed question answering: A new challenge for scalable document comprehension. In Proceed- ings of the 2018 Conference on Empirical Methods Natural Language Processing, 2018.
Shuming Shi, Enbo Zhao, Duyu Tang, Yan Wang. Piji Li, Wei Bi, Haiyun Jiang, Guoping Huang, Leyang Cui, Xinting Huang, Cong Zhou, Yong Dai, and Dongyang Ma. Effidit: Your ai writing assistant. ArXiv, ftss/2208.0181.. 2022.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems, 2022.
llya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Jason Weston, Emily Dinan, and Alexander H. Miller. Retrieve and refine: Improved sequence generation models for dialogue. In SCAI@ EMNLP, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: state-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, and Ming Zhou. Response generation by context-aware prototype editing. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2019.
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models, 2023.
A DATASET STATISTICS
The experiments in this paper include three benchmarks: (1) WikiText-103: (2) English part of
Law-MT; (3) Wiki.... The statistics of these benchmarks are shown in Table 7 En-Wiki corpus used for the enlarged phrase index settings in this paper, containing over 4,848,348 long English Wikipedia documents.
12

---

Published as a conference paper at ICLR 2023
Benchmarks WikiText-103 Law-MT
Train
Dev
Test
1,801,350
3,760
4,358
389,292
2,000
2,000
Table The number of sentences in the WikiText- and Law-MT datasets.
B MORE IMPLEMENTATION DETAILS
During training, the dynamic vocabulary of CoG contains two parts: (1) word-level vocabulary size (50257 in GPT2 vocabulary); (2) the phrases in a batch of training documents. During inference, the dynamic vocabulary consists of the word-level vocabulary and the phrases extracted from the Top-k retrieved documents (k=1024 in this paper). The size of the pre-defined word-level vocabulary contains 50257 subwords. Since there are only a few documents encoded to extract the phrase representations, the average number of the phrase representations is 950,942.4 in the WikiTextt-103 test when K 1024.
C PERPLEXITY OF GENERATED TEXT
We calculate the perplexity of the generated texts un- der a large pre-trained language model (GPT2-Large). As shown in Table 8 can be found texts gener- ated by greedy search can achieve very low perplex- ity scores (even much lower than the ground-truih This is expected greedy search targets at likelihood maximization. Sampling-based decoding methods give much higher perplexity scores. Moreover, it is worth noting that CoG achieves the closest perplexity score ground-truth.
Models
Perplexity
greedy
nucleus
Transformer
3.26
37.11
kNN-LM
3,48
78.01
RETRO
3.27
36.40
CoG
10.41
27.24
Ground-Truth
18.64
The perplexity on the test set of WikiText-1033.
Table
D THE PHRASE SEGMENTATION ALGORITHM
CoG takes phrases the minimum units that be put together form a coherent document. To train CoG, we design a phrase segmentation algorithm to split each document in the training set into a sequence of phrases. This algorithm makes use of forward maximum matching strategy to identify phrases. Maximum matching is one of the most popular structural segmentation algorithms. This method favors long phrases and is greedy algorithm by design. Specifically, we treat each document as a sequence of tokens and scan each document from left to right. At each step, we search for the longest prefix of the unsegmented part that also a sub-sequence of other documents other than the current document. If the length of that prefix is bigger than 2, we take that prefix as the next phrase. Otherwise, we take the first token as the next phrase and is labeled as coming from the fixed token vocabulary. In both cases, process the rest part of the current document recurrently. The algorithm can be very time-consuming because exhaustive searches over millions documents are compute-intensive Therefore, propose efficient approximation follows. First, we retrieve the top-k most similar documents for each document using the popular DPR model (Karpukhin et al.) 2020 and vector search toolkits, FAISS (Johnson et al. 2019). Then, the phrase search only runs on the corresponding top-k documents. The relevant documents usually have similar topics to the current document. The value of k is set as 1024 in our experiments. The details of our proposed phrase segmentation algorithm can be found in Algorithm|] I SearchPhrase is function that searches the cached token sequence (i.e.. the current candidate for the next phrase) among the most relevant documents. It returns label that denotes whether the phrase can be found and its position in the relevant documents.
NNoe that the original perplexity of GPT2-I .arge model the test of WikiText- 103 is 22.05 (Radford al. 2019). The gap between it and our results caused hy the different number samples. this study, we only use samples that have more than 32 tokens to generate text.
https::// https://huggingffae co/ co/facebook/dpr-ctae www.eeennderriiineee. l--n----n--e
13

---

Published as a conference paper at ICLR 2023
Algorithm I: Phrase Segmentation Algorithm
1N Data: Document set: D (d4, (dj) i=1 where d; denotes the i-th document. K denotes the number of retrieved documents. N denotes the number of documents in the training set. The pre-defined maximum and minimum phrase lengths are Immaz and Timin-
Result: Segmented document set by phrase granularity: D' = (((pi.as (dj, pos;)) |idelle V.V where Pi,æ denotes the z-th phrase in d; that also appears in another document dj in position J. ||d/|p denotes the number of the collected phrases in d,.
Ji=1
Preprocess: split each document into token-level pieces by using the off-the-shelf tokenizer.
The preprocessed document can be formulated as D {(1i,=)==1 tie the z-th token of d, which consists of ||d:]/= tokens. Prepare the empty list D U. empty phrase cache cache,=(), and cached search success label labeltast.
الاdاl { J.V. where
for 1 N do
cursor=0 PhraseCollection=(
while cursors ||dill do
if Linin I (len(cuche,)) < Lmmaz then
7
labelnows ress=SearchPirase(ccche))
else
len(cachep)) > Lmaz then ca end end
12
if labehast is True and label is False then
14
cursor - 1 ChraseColllctiio.appencccc cache,=()
rest)
16
17
else
if labeliast is False and label is False then PhraseCollection..e append) cachep, None) ca end end
19
cursor
label =labelllaaa
25
end
D' "aaapeed(PrassCClllcttoo)
26
27 end
14

---

Published as a conference paper at ICLR 2023
E MORE CASES
In this section, we present some generated examples of CoG given some specific prefixes. As shown in Figure 4 and 6 it can be observed that the generated continuations are fluent and coherent with the given prefix. However, we also notice some flaws. For example, as shown in Figure 3 CoG copied the phrase 75 mph from the document sustained winds of at least 120 km / h (75 mph), which is incoherent with the previous copied phrase 106 km / h. Moreover, as shown in Figure 6 CoG copied the phrase Rhine and Main from the document (Bi the terms of the Peace of Basel (22 July 1795), the Prussian army was leave the Rhine and Main river valleys ...). However, the complete phrase should be Rhine and Main river valleys, and CoG only copy part of leading to inaccurate generation results (rivers).
Guldea Gilbee
comee
2008
Notinn wie stroumeed as
and
Sepporting
Deceneal
Dicanor
zonisated
Input Prefix
2008. McCoemack udevision misiacroo The Andiomeda Strin
Sanuary Natae playo-mmaagr Laylon Oreen,...
While
Seven Paychopalls
He cael
Matthex Morrioon as
cast Psychoppttss
Decnmer
Decemnt
Dacanvet Of principal east Roan said Custing Manllioe Mormaon
After pecniciin at the Tonatto Film Festival, Seven Psychopothii Farell'ss socoral mocic
He bert in the role. She pottnayed...
Generated Continuation
Figure 4: An example generated by CoG. The dotted squares denote that the content (highlighted in red Jis generated from the token vocabulary, and the solid squares denote that the content (highlighted in blue is copied from other documents.
In the Philippines, officials people. Inbudo stroug typhoos to stilke since Typhoon Zeb five years prior.
106 km/h
000
Docuueal
Deceunat
Docasutt
Cyclone Calasanjy cansed hovvy dastage sestem Madiguscar, with peak of 195 kmch
Input Prefix
the Wilmington international Airpoet while gusts reached d6 mphh
sustriieed winds of Jeast 120 kan / mpli
being measured
on Pagan Island
with pask wind of 106 km m75
Texas coustal plain led extremely high rainfall being measuned Jefiernon
with 185 km 114 Pagan typhoon crused herey crop
Generated Continuation
Figure 5: An example generated by CoG. The dotted squares denote that the content (highlighted in red Jis generated from the token vocabulary, and the solid squares denote that the content (highlighted in blue is copied from other documents.
15

---

Published as a conference paper at ICLR 2023
Delerenzeis known the
isk
woer buttle lought
ltatle canko and Battle Sanko German
Dacaaust
Input Prefix
reteat codenamed Operation Aster I Gemaan:
fooght September
Was battle lought
apainst feecs Ardames preoccupped Hitler mind...
Caiemeheeen Astar
Napolconic Wass daring
Frennch invasian of Rusin
near the junction of the
Rhine and Main
fivers
S October
Lineenchmenn Asterisk battl fought on
near
Dvcumett
Spailied Farm,
the temes of the Peece (sssel (22 July 1795 Prussian Rline
independence in June 1991; huwwever the doclansion csmme effect October (99)
located junction of
Generated Continuation
river valleys
Figure 6: An example generated by CoG. The dotted squares denote that the content (highlighted in red Jis generated from the token vocabulary, and the solid squares denote that the content (highlighted in blue is copied from other documents.
16


Reviews:
[{"review_id": "s_DCrayTGWX", "reviewer": "Reviewer_LpvV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed Copy-Generator architecture is an attempt to depart from the classical probabilistic token-by-token language modelling perspective for language generation. Instead, generation is framed as gluing together phrases that are retrieved from a collection of documents. Searching for phrases at inference time is implemented in a heuristic coarse-to-fine-grained fashion to reduce the computational cost. The issue of computing a partition function over a potentially ginormous set of phrases is side-stepped by using an NCE loss in training, and either using greedy decoding or nucleus sampling (normalization over top-k items) for inference.", "summary_of_the_review": "The idea of tackling neural language generation on the phrase-level is neat, but the practical limitations (latency, sensitivity regarding the size and content of phrase table) are not fleshed out enough in the paper.", "strength_and_weaknesses": "I think that the idea is a reasonable follow-up to the kNN-LM approach of Khandelwal et al. The ability of adding in-domain data without retraining is very appealing. But there are some practical disadvantages compared to vanilla LMs that are brushed over to some extent. First, Copy-Generator does not define a probability distribution over a fixed vocabulary (as the authors admit in Sec. 4.3), and that limits its applicability to pure generation - it does not work in combination with other models (e.g. Bayesian, (re-)scoring...). More importantly, the idea of retrieving phrases from the training data contradicts the trend towards using bigger and bigger training sets. In fact, while latency is reported on Wiki-103 (which has 100M tokens, so rather small by today's standards), it is not reported on the En-Wiki corpus. But even En-Wiki with 3B tokens is not outrageously large nowadays.\n\nA potential way around this latency concern would be a separate treatment of the training set and the document collection used at inference time. A good data point would be the En-Wiki Copy-Generator with an inference-time document collection trimmed down to match the latency of the baselines. More details about the sensitivity regarding the size/domain of the inference-time phrase table would be interesting. I think the note made in Sec. 5.3 \"we can use learn COPY-GENERATOR with a smaller corpus but leverage\nadditional information in a larger corpus in a plug-and-play fashion\" misses the point: we want to train with a large corpus, and plug in a small corpus at for inference. \n\nA better sense for the importance of the phrase table would also be provided by reporting Wiki-103 performance of the Copy-Generator with Law-MT phrase table. Is Copy-Generator more/less prone to catastrophic forgetting?\n\nMinor comments:\n- It is nice to know that the average length of phrases is 3.95, but more details (variance, full histogram..) would be useful.\n- Phrase extraction by segmentation seems limiting as it does not allow any overlap between phrases.\n- Why is the Copy-Generator latency for nucleus smaller than for greedy\n- Abstract: coping -> copying \n- Sec. 3 \"We denote all the e all the phrases\"\n- Sec. 3 \"all phrases are context-sensitive\" clarify that they depend on the source document context, not on the target prefix context at inference time.\n- Sec 4.2: \"does not suit work well\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qjIfgE202ah", "reviewer": "Reviewer_pM7y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a retrieval-based LM that can generate phrases. Their base model is a standard transformer, and the retrieval component uses BERT. Their model is trained using in-batch positives / negatives. Retrieval is actually done at document-level, then phrase extracted using a segmentation algorithm. For evaluation, rather than focus solely on language modeling, they evaluate in a text generation setting. They primarily evaluate with MAUVE but also report other metrics useful for analysis. The results are strong compared to baselines, but text generation is very hard to evaluate and I have some questions / suggestions about this.", "summary_of_the_review": "The work is very interesting and relevant to the community. The main issue is the authors have taken a language modeling task for Wikipedia and converted it into an unconditional text generation task. In unconditional text generation, there are many valid and diverse continuations of text. But in Wikipedia especially I believe that factual information is important to get right, making it hard to interpret some of the evaluation. Human evaluation is one way to immediately address this concern, but perhaps other evaluations besides MAUVE could be useful too. I mentioned perplexity, but maybe BLEU with small sliding window could help understand factual correctness of the model AND leverage the phrase-based aspect.", "strength_and_weaknesses": "# Strengths\n\n- Strong results using MAUVE in all settings (although I find it hard to interpret MAUVE here, and there should be more effort to help readers calibrate their understanding of the reported metrics).\n\n- The method can generate phrases instead of only single words (although I have questions about efficiency).\n\n- The retrieval component uses rich context information.\n\n# Weaknesses\n\n- It is not ideal that COPY uses BERT for retriever and kNN-LM uses vanilla transformer trained from scratch. To me this is a major confounder, and maybe having a strong retrieval component (i.e. BERT) is more important than using phrase-based vocab.\n\n- The rep-N metric is very strict (measures contiguous duplicate n-grams), and perhaps is useful for debugging, but I imagine it does not tell use much about generation quality. It would be more useful to include distribution novel n-grams with respect to ground truth. Can review RAVEN paper for guidance (McCoy et al., 2021. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN).\n\n- The results are very reliant on the MAUVE metric, but for wikipedia text, factual correctness is very important and I am not sure how will this is reflected by MAUVE. Although there is much discussion about perplexity being a poor measure of text generation, here, I think it would have been helpful since it could reflect factual correctness to some degree. Unfortunately, authors mentioned that PPL is not reliable because of the phrase-based vocabulary --- I wonder if it would be possible to still compute PPL in some settings if not all (certainly the baselines support PPL at least). In any case, some human eval may be appropriate here so we can better calibrate the interpretation of MAUVE in this setting.\n\n- It would help considerably to have other baselines. This might alleviate concerns about MAUVE too.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W6mCzW4inK", "reviewer": "Reviewer_8gPK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates text generation as copying from existing text segments. The authors decompose text generation as a series of copy-and-paste operations by seeking text spans from existing articles and copying them as the generation result at each step. Experimental results on WikiText-103 show that the proposed method can achieve higher MAUVE scores and also support effective domain adaption and better inference efficiency.", "summary_of_the_review": "Although generating texts via a series of copy operations is interesting, I’d recommend the authors to solve the concerns about the novelty and experimental settings before making this paper ready for publication.", "strength_and_weaknesses": "Strengths:\n\n1) Reformulating text generation as a series of actions that fully depend on copying is an interesting idea, which may work well in some NLG scenarios.\n\n2) From the experimental results, copy-generator can achieve better generation quality while having lower latency compared with Transformer-based baselines, which shows its effectiveness.\n\n3) This paper is well-organized and easy to follow.\n\nWeaknesses:\n\n1) The idea which combines vanilla token-level generation and phrase copying has been already discussed in the existing works such as [1]. In my view, the main difference only falls into the format of knowledge sources (unstructured documents in this paper and structured knowledge in [1]). Thus, the authors should further clarify the difference between copy-generator and existing works to show their novelty.\n\n2) The metrics used in the experiment are somewhat questionable. First, this paper lacks the fluency metric such as perplexity. From my experience, if copy operations dominate the generation process compared with token-level generation, the fluency of the whole sentence may be degraded. Then, MAUVE as a distribution-based metric may be biased to evaluate copy-generator because copy-generator largely copies source texts. If the distributions of source texts and the test set are similar enough, it’s not surprising that copy-generator can achieve a high MAUVE score.\n\n3) The latest baseline selected in this paper is in 2020. The authors should add state-of-the-art baselines about retrieval-augmented language models such as [2].\n\n4) Since the experiment is mainly conducted on WikiText-103, I wonder whether there is a principle to collect the source texts for a specific downstream task / dataset, such as machine translation, text summarization or dialogue generation. It’s important because the generation quality of copy-generator largely depend on the quality of source texts.\n\n5) Typo: “all the e” in the second paragraph of Section 3 should be removed.\n\n[1] Latent Relation Language Models. AAAI 2020.\n\n[2] GNN-LM: Language Modeling based on Global Contexts via GNN. ICLR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EHkxFkQa83", "reviewer": "Reviewer_siHy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a language model that can autoregressively choose tokens from a fixed vocabulary, or multi-token phrases from a pre-encoded corpus. The paper builds on previous work on phrase encoding for question answering, but extends this work to freeform text generation. The model (Copy-generator) is evaluated on three separate language modeling tasks. Two that use Wikipedia text and one that involves domain transfer to a legal domain. When compared to a vanilla LM and a KNN-LM that retrieves memories of individual tokens, Copy-generator performs better according to the recently proposed MAUVE metric. It does less well, in comparison to KNN-LM, according to diversity and Rep-N metrics on some tasks, though. Traditional perplexity scores are not particularly meaningful here because the model can choose different segmentations of the text than those used by the baselines.\n\nThe phrase encoder is trained along with the language model in the following fashion. First a greedy segmentation algorithm is run to choose a single segmentation of each training example into phrases that exist elsewhere in the training corpus. Then, for each phrase, a piplined retrieval system is run. First a document retrieval system retrieves a small set of documents that are similar to the prefix. Then, a phrase search is applied over all phrases in this retrieved set of documents. Phrases are represented using encodings of their start and end position, following previous work on phrase retrieval. The phrase retrieval model and language model are trained using a combination of InfoNCE loss over phrase candiates, as well as language model loss. The document retrieval model is fixed (as far as I can make out).\n\nAt inference time, phrases or individual tokens are selected from the union of all phrases in a large retrieval corpus and the fixed length vocabulary. Examples show that Copy-generator often chooses to select long phrases instead of individual tokens (I would like to also see statistics illustrating how often different length n-grams occur), and the generated text is deemed significantly better than the baseline systems according to the automatic MAUVE metric, which has been trained to correlate with human judgements. An analysis of performance shows that quality increases with the number of documents in the phrase corpus, and this increase is particularly pronounced when greedy search is used during decoding. \n", "summary_of_the_review": "This paper presents a novel approach to language modeling, that can copy large phrases from a retrieval corpus rather than decoding them one token at a time. The performance on three language modeling datasets are positive, according to the MAUVE metric and in relation to the similar KNN-LM baseline that performs retrieval of single token encodings. It would be nice to see some more analysis of the outputs, and how the copy mechanism is being used during inference time. It also would be very interesting to see how well this model does on a wider variety of text generation tasks. However, the current paper is sufficiently novel and interesting for acceptance at ICLR.\n", "strength_and_weaknesses": "#### Strengths\n- This is a novel and interesting architecture that is significantly different to previous work.\n- Copy generator performs well in comparison to reasonable baselines.\n- While the model is only applied to language modeling tasks, I can imagine extensions to other forms of generation where retrieving and copying text is likely to be a good strategy (e.g. multi-document summarization).\n\n#### Weaknesses\n- The paper relies heavily on the recently proposed MAUVE metric. It would be great to also have some human judgement of quality, no matter how small, to get more intuitions of where Copy-generator does better or worse than the baselines.\n- Copy-generator makes use of context-dependent phrase encodings. However, the examples in Figures 2 and 5-8 seem to show that large phrases are being drawn from very different contexts to the target context. It'd be good to see some analysis of how important the is here.\n- It would be interesting to see a comparison to retrieval augmented language models (e.g. RETRO, ATLAS) which also include document retrieval at inference time (but also have a greater inference-time computational cost than copy-generator).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster