WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus
Authors: Hongjin Qian
Yutao Zhu
Zhicheng Dou
Haoqi Gu
Xinyu Zhang
Zheng Liu
Ruofei Lai
Zhao Cao
Jian-Yun Nie
Ji-Rong Wen
Venue: Submitted to ICLR 2023 (2023)
Keywords: factual generation
retrieval-augmented generation
new large-scale dataset

Abstract: In this paper, we introduce a new NLP task – generating short factual articles for queries by mining supporting evidence from the Web. In this task, called WebBrain, the ultimate goal is to generate a fluent, informative, and factually-correct short article (e.g., Wiki article) for a factual query unseen in Wikipedia. To enable experiments on WebBrain, we construct a large-scale dataset WebBrain-Raw by extracting English Wikipedia articles and their crawlable Wiki references. WebBrain-Raw is ten times larger than the previous biggest peer dataset, which can greatly benefit the research community. Besides, we empirically analyze the performances of the current state-of-the-art NLP techniques on WebBrain and introduce a new framework ReGen, which enhances the generation factualness by improved evidence retrieval and task-specific pre-training for generation. Experiment results show that ReGen outperforms all baselines in both automatic and human evaluations.

Under review as a conference paper at ICLR 2023
WEBBRAIN: LEARNING TO GENERATE FACTUALLY CORRECT ARTICLES FOR QUERIES BY GROUNDING ON LARGE WEB CORPUS
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, we introduce a new NLP task - genneraing short factual articles for queries by mining supporting evidence from the Web. In this task, called WEB- BRAIN, the ultimate goal is to generate fluent, informative, and factually- short article (e.g.. a Wikipedia article) for factual query unseen in Wikipedia. To enable experiments on BRAIN,,,, we construct a large-scale dataset WebBrain- Raw by extracting English Wikipedia articles and their crawlable Wikipedia ref- erences. WebBrain-Raw is ten times larger than the previous biggest peer dataset, which can greatly benefit the research community. Besides, we empirically an- alyze the performances of the current tate-of-the-ee NLP techniques on WEB- BRAIN and introduce a new framework ReGen, which enhances the generation factualness by improved evidence retrieval and task-specific pre-training for gen- eration. Experiment results show that ReGen outperforms all baselines both automatic and human evaluations.
INTRODUCTION
Information acquisition is one of the fundamental daily needs of human beings. Acquiring infor- mation from the Web undoubtedly a convenient and efficient way. However, with the exponen- the Web becomes scattered and evolves quickly, making it
tial growth of the Web, information challenging for users to acquire the expected information quickly. As a result, Wikipedia articles become the best bet for most users when searching answers for factual queries on the Web (Singer) et al. (017). The reason is that Wikipedia articles provide credible content in which most claims can be supported by references from reputable sources. While Wikipedia is a good source of an- swers for factual queries, the need for manual editing (crowd-sourcing and editor checking) curbs its growth coverage on broader range of information needs. What if Wikipedia articles could be automatically generated?
In this paper, we introduce new task, WEBBRAIN, exploring the capacity of generating short fac- tual articles for queries via a large web corpus. Given a factual query, the goal of the task is to enable a system to mine supporting evidence from the Web and generate a short factual article in which the claims are supported by the mined evidence (defined in Section 3.Tp. One of the potential generation targets for WEBBRAI is the first section of a new Wiki page, based on which further explore generating long factual articles (e.g., complete Wiki page). WEBBRAIN can be greatly helpful in various scenarios, including generating Wiki pages for new entities, intel- ligent writing assistance, knowledge-intensiv QA, etc. WEBBRAIN'S goal is considered one of the ultimate goals of the future search engine (Metzler et al. 2021). Figure illustrates a case of our WEBBRAING
To establish the data foundation of WEBBRAIN, we construct a large-scale dataset, WebBrain-Raws from scratch by extracting all English Wikipedia articles and all the corresponding reference arti- cles. To the best of our knowledge, WEBBRAIN-Rav is the biggest dataset sourced from Wikipedia (about 10x larger than the previous biggest peer WikiSum (Liu al. 2018), introduced in Sec- tion 3.2p. Along with WEBBRAIN, we empirically investigate the ability of the current state-of-the- art techniques and conclude that most current models lack the ability to correctly cite references and
the text generation result is obtained via OpanAl's GPT3 API: httph:ttp///cc... http:///ettaccoo comm

---

Under review as a conference paper at ICLR 2023
Input query
Input query
Q Introduce GPT-3
Q Introduce GPT-3
Pre-trained Language Models (e.g., BART, T5, GPT-3)
Retrieval Module
Web Articles
Generative Module
ReGen
Generative Pre-trained Transformer 3 (GPT-3) autoregressive language model, which is created
[1] Shead, Sam (July 23, 2020). "Why everyone is talking about the text generator released by Elon Musk-backed CNBC.
GPT-3 is a machine learning platform that enables developers to train and deploy Al models. It also said to be scalable and efficient with the ability to handle large amounts of data.
by OpenAl, an artificial intelligence research
laboratory. I0I GPT-3 is part of a trend in natural language processing (NLP) systems of pre-trained language representations."' .1
[2] Bussler, Frederik (July 2020). "Will GPT-3 Kill Coding?". Towards Data Science.
(1) Text Generation
(2) WebBrain
Figure Comparison between text generation task and WEBBRAIN.
generate factual descriptions. Based on the best current models (SPLADE=FI++BAR)) (Izacard & Grave 2021 Formal et al. (202T)) we propose a new framework, ReGinn, which enhances the factual correctness of generation by (1) controlling the topic consistency of the retrieved support- ing evidence; (2) introducing the citation mark into the generation process to indicate the source of referenced claim; (3) warming-uu up the generation model by pre-training with fact-oriented self- supervised tasks (described in Section 4
Current tasks that are similar WEBBRAIN include: (1) text generation with generative pre-trained
language models (e.g., GPT-3) (Brown et al. 20200; (2) retrieval-
QA (e.g.. RAG (Lewis)
et al. 2020b), GopherCite (Menick al. 2022), REALM (Guu al. 2020); (3) multi-document
summarization (MDS) [Liu et al. 2018 Liu & Lapata 2019); (4) WebGPT (Nakano et al.) 2021).
WEBBRAIN distinguishes itself from the existing tasks in the following aspects: (1) Existing gen- erative models generate text solely depending the implicit knowledge stored in the model's pa- rameters, which are prone to generating factually incorrect statements, a phenomenon commonly called hallucination. In contrast, the supporting evidence mined from the Web; (2) Retrieval-augment enhance answering specific questions whose answers are usually text span single sentence. Given a factual query, our WEBBRAIN explores capturing all the knowledge available on the Web to generate a comprehensive and accurate short article; (3) WEB BRAIN has a more complex pipeline than the MDS task, which focuses on summarizing documents that are already prepared. WEB- BRAIN requires models mine useful evidence before generating the factual article; (4) WebGPT mimics the behavior of a human by browsing the Web to answer a specific question. This involves many challenges that are difficult solve (e.g., collecting training data by recording human behav- iors). Instead, WEBBRAIN grounds itself on large number of Wikipedia articles that are already created and edited by crowds, which makes the task much more realistic.
WEBBRAIN aims generate factual statements depending QA utilizes the retriever to
In summary, our contributions are threefold: (1) we introduce a new task, WEBBRAIN, dedicated to answering factual queries by generating short factual articles based on evidence retrieved from a large web corpus; (2) we construct a large-scale dataset, WebBrain- Raw, to evaluate the potential of WEBBRAIN. WebBrain- Raw is ten times larger than the previous biggest peer dataset; and (3) empirically analyze the performance of current of-t-oo-thheart techniques WEBBRAIN, and propose our fnctuality-enhaam framework, ReGen, which outperforms all baselines through both automatic and human evaluation.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Pre-trained Language Models Pre-trained language models (PLMs) have been widely applied to various natural language processing (NLP) tasks and have achieved outstanding performance. In general, these models are first pre-trained on a large-scale corpus, and then fine-tuned on down- stream datasets for specific tasks. By pre-training, the model can learn effective language represen- tations, thus improving its performance on downstream tasks. Typical paradigms of PLMs include masked LM (Devlin et al. 22019] Zhang et al., 2019 Sun et al. 2019). left-to-right LM (Radford) et al. 2019 Black et al. 2021), prefix LM (Dong et al. 2019; Bao et al. 2020), and encoder-decoder
LM (Sogg et al., 2019 Raffel et 2020 Lewis et al. 2020a).. Masked LMs are generally most suitable for natural language understanding or analysis tasks, e.g., text classification and natural language inference. The other three types of PLMS can be naturally used for text generation.
Wikipedia-related Tasks Wikipedia is a popular knowledge source for NLP tasks that aim to uti- lize external knowledge. Examples include bhowledge-groundeer conversation (Dinan et al. 2019 fact checking Thorne et al. 2008)), open-domain QA Kwiatkowski et al. (2019), and slot filling EI-
Sahar et al.] (2018) etc. Petroni et al. (202I) propose a benchmark for those knowledge-inteee -nttenieeeeeneiveeee language tasks (KILT) in which all tasks apply snapshot of Wikipedia the knowledge base. Com- pared to the KILT, other Wikipedia-rehe related tasks focus more on Wikipedia itself. Liu et al. (2018) construct dataset WikiSum from Wikipedia to enable multi-document summarization (MDS). The MDS task is then further explored by [Lu & Lapata (2019) with a hierarchical model and Perez- BBllrachini et al. (2019) with topic-guided structure. WikiWrite (Banerjee & Mitra 2016) pro- posed a systematic solution to constructing Wiki pages by assigning retrieved content to different topical sections. Fruit (Iv et al., (022) explores updated information on existing Wiki pages. Re- cently, Piktus et al. (2021) construct a dataset Sphere from CCNet (Wenzek et al. 2020), using which they explore refining the citation quality of Wikipedia pages.
Retrieval-Augmentrr Text Generation Pre-trained language models are prone to generating hal- lucinations (i.e., factually incorrect statements) (Vinyals & Le 2015 Koehn & Knowles] [017]] Rohrbach et al. 2018 Raunak et al. 2021). Retrievallauugmeneed text generation, as new text generation paradigm, can be a possible way to alleviate this problem. Compared with generation- based counterpart, this new paradigm can reduce the reliance on storing enormous knowledge in parameters by providing more references. Retricval-auu -atrivealaugmmeneed text generation has been widely applied to many NLP tasks, such as dialogue generation (Weston et al. 2018 Zhu et al. 2020) machine translation (Gu et al.) 2018) Cai et al.] (022]. and open-domain question answering (Lewis) et al. 20206 Guu et al. 2020). In this work, we formalize
retrieval-augmeente text generation task- WEBBRAIN with two new features: (1) The retrieval base is a large-scale open-domain web corpus; and (2) The target to generate natural, informative, and factual text rather than short span. Both features make the task extremely challenging. Our experiments will show that, though existing models can be applied to this task, there is still a large space for research and exploration.
TASK: WEBBRAIN
3.1 TASK DEFINITION
Formally, given an input query g, our task is to generate a short article Tg for g grounded by prior knowledge O We have:
TH = arg max θ).
(I)
For a pre-trained language model solely, the knowledge prior θ is stored in the PLM's internal parameters. For WEBBRAIN, the knowledge prior θ contains the implicit language knowledge stored in the PLM's s parameters and the explicit knowledge y defined by the supporting evidence V. which is mined from an external large web corpus C. In practice, C can be an off-line processed large web corpus, online search engine API, or combination of both. Intuitively, coupling 0 and y, e = (0, 7) can be characterized by the strong language capacity from the PLM and the factual constraints from the supporting evidence.
Supposing the generated T, contains n sentences (t,,... .....,,,,, tn.) and the retrieved supporting evidences evideees
V = (Vi,"'. ..)) E C, expect each sentence would be supported by V's subset V", denoted
3

---

Under review as a conference paper at ICLR 2023
Table Datasets built on Wikipedia. WEBBRAIN-RAN significantly larger than existing datasets.
Dataset
# Wiki Pages
# Refs
Status
Storage Size
WikiSum (Liu et al.
2018)
2.3M
87M
Need crawling
300GB
WikiCatSum (Perez- beltrachini al. 2019
0.17M
23.5M
Ready
4.8GB
Hiersumm (Liu Lapata 2019)
1.66M
Ready
6.9GB
WebBrain-Raw
14.86M
259.5M
Ready
2.8TB
as g-V More precisely, we have:
T = (ti
- arg max (TT.gq, 0, V==V
Note that when the generated sentence only relies on the PLM's language knowledge 0 (e.g.. com- mon sense generation), the corresponding supporting evidence is V = 0.
3.2 DATA COLLECTION
Wikipedia is one of the most inclusive and widely-used encyclopedias on the Web. By the nature of an encyclopedia, Wikipedia can be regarded as a large collection of factual articles on various topics enty of Wikipedia page). The factual content in Wikipedia is generated by crowd-sourcing, and most claims can be supported by the citations in the References section. Due to such nature, Wikipedia is the most appropriate resource to leverage for WEBBRAIN. The specific reasons are: (1) Wikipedia's s format conforms to the WEBBRAIN task's 'ss definition in Eq. (2 We can view the Wiki page title factual query q, the citations (or parts of them) as the retrieved supporting evidence y, and the Wiki article as the target factual article T (2) Wikipedia has a huge volume of references which are usually reputable sources. Using the references,
can build large web corpus of good quality as the retrieval source; (3) Wikipedia covers wide range of topics of common interest, which provides good knowledge basis for models generalize to open-domain queries.
Many previous studies collected Wikipedia data for different tasks (Liu et al. 2018 Liu & Lapata 2019 Perez-Beltrachiní et al., 2019). However, none of these datasets provides full-size Wikipedia pages and the corresponding references. To lay a good data foundation for WEBBRAIN, we build a large-scale dataset from scratch by extracting all Wikipedia pages and all crawlable references.
Table [ shows the statistics of different datasets. In the Supplementary Material, we attach 500 raw data samples and the corresponding experiment data for demonstration. The full datasets we construct for WEBBRAIN will be released upon acceptance of this paper.
Data Cleaning As shown in Table I the WebBrain-Raw data contains full-size Wiki pages and a huge volume of references, which are open-domain web pages. Regarding data quality, the Wiki pages are good due to their server stability and format consistency. How- ever, the references contain much noise useless for our task. For example, they may be images, tables, or empty content, To guar- antec data quality, we process the WebBrain- RAW dataset in the following steps:
Table 2: Statistics of data for experiments.
WebBrain-R
WehBrain-G
Queries
2.74M
12.32M
Ref. passages
3.20M
12.61M
# Tokens / Query
3.2
2.9
Tokens Passage
237.5
250.0
Tokens Target
108.6
# Training
4.46M
12,30M
# Validation
0.2M
0.5M
# Test
88,935
24,546
(1) Wiki Article Cleaning: for Wiki pages, we remove the Wiki ( special symbols (e.g., ASCII symbols), and other invalid tex 3
(2) Reference Cleaning: for reference, we filter out articles that are shorter than 16 words or non- English token ratio > 0.3. We remove invalid text such those containing only HTML tag. unusu- ally long sentences (e.g., > 256 tokens), text tables and URLs, etc. Furthermore, as the citations
"See details in httptttt///// //mps://en... wann.wike wik pedi a wtwg//wikkk /wik a Projjerr We observe some template text that may not be useful for model training (e.g., You can help Wi k pedi by expanding it.)
lttes
4

---

Under review as a conference paper at ICLR 2023
may be missing (uncrawlable filtered out) or incorrectly cited (Piktus et al. 2021 we define the following metric to calculate the term recall of a reference to choose "reasonable" references:
TT T-87 -
PSt
(3)
where Q is the term set of query g, T is the term set Wiki article sentence (-V7 that cites a reference set VT, V is the term set of Vr, and S is set of stop words. Intuitively, if more terms in the sentence claim t can be found the references V the references would be more useful. Therefore, only keep references that have Pst > p.
Reference Passage Selection The original Wiki articles and reference articles tend to be very long. To adapt the capacity of most pre-trained language models (e.g.. BERT has 51 512-token limit), we use the first section of Wiki articles as the generation target, and we select the passage from the reference article with the highest PSt value as the supporting passage. Specifically, we first split reference article by sentences and concatenate the sentences into passages (stride = 1, max lenghh = 256), then compute the PST value for each passage and keep the highest one.
Dataset Generation As defined in Eq. (2), given a query q. the WEBBRAIN task aims to re- trieve the supporting evidence V from a web corpus C. to generate a factual article Tg. To enable training the in-domain retriever and generator for WEBBRAIN we generate two separate datasets, CeehrriinR(etrieee and (reeeaiinn/eeeeattor) from WebBrain-Raw, For WehBrain-G, we keep five supporting passages for each query and retrospectively assign reference marks to all unmarked sentences in the Wiki article by measuring their P'T value. We append "[O]" mark to the sentence end for these fail match any references. For WebBrain-R, we use the supporting passages as the positive and randomly select four negative passages from the top 30-50 retrieved results via a BM25 engine. Table sables the statistics of the two datasets.
4 FRAMEWORK: REGEN
4.1 RETRIEVER
By the definition of WEBBRAIN in Eq. (2. ReGen starts from a retriever that mines the support- ing evidence V from the large web corpus C. ReGen utilizes parametric sparse lexical model SPLADE (Formal et al. 2021a b) as the retriever, which encodes a sequence s = (w1. wn) into a sparse lexical representation s E RI by predicting token-level importance in WordPiece vocabulary size (e.g., DD 30522). Specifically, is computed based the dense representation (h,,... hn) output by the underlying PLMs:
w. = EWh, b, E [1,
max log(1 ReLUUw::).
xE[1.n]
where W E 7688766 and b E IR are trainable transition matrix and bias, respectively, and E E I 768x||D refers to the PLM's input embedding matrix. In the training phase, given query q and a document d, the ranking score g(g, d) is computed by the dot product of the their representations Sy and Sd-
Previous studies that utilize a retriever in the generation loop usually apply the traditional term-match-based retriever (e.g., BM25), dense passage retriever (e.g., DPR), or hybrid re- triever (Karpukhin et al. 2020 Lewis et al. 2020b Izacard & Grave 2021 Glass et al., 2022) We build ReGen's is retriever based SPLADE because of the following reasons: (1) Factual queries in WEB WEAN are usually open-domain and entity-centric, while recent work observed that DPR tend to perform poorly for entity-centric queries or out-of-domain queries (Sciavolino et al., 2021 Xiong et al., 202T] Thakur et al. 202T). ). SPLADE, on the contrary, inherits the good properties of sparse retrievers such as exact-entity match and generalizability. making suitable for WEBBRAIN (2) SPLADE also inherits the contextualized semantics of the underlying PLMs, enabling the query encoder to perform contextualized query expansion (Formal et al., 2021a), which alleviates the vo- cabulary mismatch problem of BM255. (3) Empirical results also prove the effectiveness of SPLADE WEBBRAIN (see Section 5.3
statistics shows that the first section of 96%+ Wiki pages has equal less than five references.
5

---

Under review as a conference paper at ICLR 2023
Ranking Loss Generally, given query q, a positive document d' and a set of negative documents d the optimization goal of retriever is to maximize the probability:
ellada))
P(g, d D) el(g,
(5)
Es ED (/(,)=))
where D represents the entire corpus. We use the mined negative passages and the in-batch negative samples as d_. We train the model with contrastive LCE loss (Gao et al. 2021).
Asymmetric Sparsity We introduce two sparsity constraints to control SPLADE's sparsity: L1 regularization and FLOPS regularization (Paria et al., (020). For the query encoder and document encoder, we apply asymmetric sparsity regularization. We use LI regularization with small weight Xg for the query encoder and FLOPS regularization with big weight Xa for the document encoder. Thus, final loss SPLADE becomes:
L= Lrank + AyCLn + AaCFIOPS.
We apply the asymmetric sparsity because: (1) small X lowers the sparsity of the query encoder, enabling the query encoder to generate semantically-ridd query representation; (2) the FLOPS reg- ularization is not sensitive to the number of documents in batch so both positive and negative samples; (3) a big Aa increases the sparsity of the document encoder, making the off-line document index efficiently, and producing self-denoising effect more unimportant tokens tend to be ignored.
compute Leices
Topic Consistency and Diversity For document pair (dm, dn) and their Diise sparse repre- sentations (Sm, Sn), we measure their topic distance by:
1s s/ |
dr (dm, dn) di (ssttance(sm, 8.)
3/|s 1 j E
(7)
where 8 = (.1.... 8))) s' is a binary variant of s, and μ is a threshold to eliminate tokens with low importance. For (dm; d,),, when dr - O, the documents tend have the same impor- tant tokens and are likely to be redundant; whereas when dr - 1, the two documents tend to have non-overlapping important tokens and are likely topically irrelevant. In the ReGen frame- work, we propose a strategy to maintain topic consistency and avoid redundancy: Considering k sorted retrieved evidence documents (dl,-.. .dg,, we first filter out the documents d; that have dr (d1 d,) > 0.9. [2, k] (topically-rrreleann to the top-1 document). We then remove docu- ments d, that have (d,, dy) 0.1,j E [1, - (redundant to fore-rank documents).
4.2 GENERATOR
ReGen employs sequence-to-sequenne model with Fusion-in in-Decoder (FiD) structure (Izaaard & Grave 2021) as the generator. The generator takes the query and references as input and generates factual description. More precisely, given query g and set of & references {VI, u2, ; U returned by the retriever, we add special tokens before the query/reference and concatenate the query with each reference
= [query] [ref_i] D4.
(8)
These sequences are processed independently by the encoder as:
h; Encoder(x:), i= 1,2, ....,...k..
Finally, the decoder conducts attention over the concatenation of the representations of all the quences, and outputs a text sequence T with auto-regressive mechanism:
T Decoder((" h) II PT h)
[i h2: he].
(10)
The model can perform evidence fusion in the decoder and generate more factual claims with the references. The generator optimized by minimizing the negative log-likelihood function of the ground-truth text:
T
LNII(I,
Σ
(II)

---

Under review as a conference paper at ICLR 2023
Model Warm-Up for Sentence-Referent Matching In our task, the model is trained generate claims for a query based on several references. This task is difficult as the model should first find clues from various references and then determine which part is the most relevant in the current gen- eration status. To facilitate the model's capability of extracting key information from the reference, design model warm-up stage. Specifically, for target text, we break it into multiple sentences and extract the ones that are marked with references. As a result, we can obtain several (reference, sentence) pairs. Then, we concatenate the query with the reference, and train the model generate the corresponding sentence. This warm-up stage has two advantages: (1) The task is to generate a sentence with a single reference. Compared with the final generation task (i.e., generating multiple sentences with multiple references), this warm-up task is much simpler. Therefore, the overall train- ing follows a curriculum learning paradigm, i.e., learning from easy tasks to hard tasks. This learning paradigm has been shown to be effective in improving performance (Bengio et al. [2009] Hacohen & Weinshall 2019). (2) Existing models often apply general tasks for pre- vre-training. For example, BART (Lewis et al. 2020a) uses several sequence denoising tasks for pre- pre-training. These tasks have different objectives from text generation. Our proposed warm-up task is also a text generation task,
so it can help the model mitigate the gap of tasks between pre-training and fine-tuning
EXPERIMENTS
5.1 SETTINGS
ReGen applies SPLADE the basis of retriever Formal et al. (2021a), improves the vanilla SPLADE with a training strategy of asymmetric sparsity, and uses a ranking strategy consider- ing documents' topic consistency and diversity. For comparison,
report the performance of the vanilla SPLADE, DPR (Karpukhin et al., 2020) and BM25. We use RetroMAF. as the underlying PLM of DPR, which is shown to be a tfate-of-the-art model in many retrieval benchmarks (Liu & Shao 2022). For the generator, we use FiD+BART as the backbone model. As a comparison, we report the performance of BART (Lewis et al. 2020a), GPT-2 (Radford et al. 2019). and vanilla FiD+BART (Izacard & Grave 2021). All models are initialized with the off- the-tfh-shelf checkpoints provided by HuggingFace (Wolf et al. 2019). The implementation details are in Appendix [A]
To evaluate the performance, we employ several automatic metrics as follows and perform a human annotation: BLEU (Papineni et al., 2002). METEOR (Banerjee & Lavie, 2005). ROUGE (Lin)
2004), and CIDEr (Vedantam et a 2015): These are metrics based on n-gram overlapping between generated text and ground-truth text. A higher value indicates the generated text is more similar to the ground truth. We compute the metrics using the alg-eval package (Sharma et al. 2017). QAGS /Wang et al. (020) and Triple Score (Goodrich et al. 2019) are two metrics for evaluating factualness. We also compute them between generated text and ground-truth text For QAGS, several questions are generated from the generated text, which are then answered by both the ground- truth text and the generated text. A higher value reflects that more identical answers are obtained, which further suggests that the generated text is more factual. To compute Triple Score, the relations in both generated text and ground-truth text are first extracted by OpenIE (Angeli et al. 20155). Then, the score is computed by the precision of relations in the generated text. A higher score indicates that more relations are correctly generated. We compute QAGS and triple score by Fact Summ package (Heo, 2021). As for human annotation, we compare the result of our ReGen with that of baselines concerning fluency. informativeness, and faithfulness. detailed description of the annotation criteria is given in Appendix B
5.2 OVERALL RESULTS
The results of various models are shown in Table B and Table As can be seen, our ReGen model (in both sizes) achieves the best results on all metrics. This clearly demonstrates the effectiveness of our framework. Specifically, (1) Compared with BART and GPT-2, FiD and ReGen achieve better performance, especially on factualness metries (QAGS and triple score). This indicates that the knowledge from the Web corpus is critical for WEBBRAIN task. (2) Large models can always perform better than base models, but the main improvement is made on language quality (e.g..
'These scores can also be computed between generated text and reference, but how to combine the scores based on multiple references still under-explored.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Performance of different models on WebBrain-G: The best results are in bold.
Model
BLEU-1
BLEU-4
METEOR
ROUGE-L
CIDEr
QAGS
TripleScore
BART (Base)
8.16
2.98
9.28
23.08
38.30
28.49
10,91
BART (Large)
11.63
4.55
10.49
24.45
45.54
28.61
12.47
GPT-2
9.35
2.85
8.57
20.22
25.09
25.61
7.30
FiD+BART (Base)
11.20
4.72
12.60
27.42
51.32
39.88
15.89
FiD+BART (Large)
15.90
6.89
14.03
28.64
57.06
39.52
17.36
ReGen (Base)
12.11
5.26
12.98
27.81
52.92
40.00
16.19
ReGen (Large)
17.97
8.82
15.31
30.09
62.39
40.32
19.43
Table 4: Retrieval (left) and generation (right) performance of ReGen (Large) with different retriev- ers. Passages for generation are retrieved from the full passage set. The best results are in bold.
Model
R@D
R@5
MAP
BLEU-1
METEOR
ROUGE-L
CIDEr
TripleScore
ReGen
0.399
0.746
0.550
27.81
14.89
22.00
19.40
13.68
BM225
0.290
0.558
0.410
27.16
14.54
21.38
17.32
12.98
DPR
0.228
0.390
0.301
24.57
13.17
19.82
15.21
10.74
SPLADE
0.389
0.724
0.536
25.60
14.98
20.71
14.24
12.35
Oracle
17.97
15.31
30.09
62.39
19.43
BLEU and ROUGE) rather than factualness (e.g., QAGS). This is consistent with our assumption that external knowledge is hard to be "remembered" in model parameters. (3) Compared with FiD, we design a warm-up stage for ReGen model. This strategy is effective regardless of model size but works better for larger model. It suggests that larger models with more capacity are promising for better results, but their potential needs to be activated through well-tailored tasks.
5.3 DISCUSSION
Impact of Retriever To explore how different retrievers impact the end-to-end performance of ReGen, we replace ReGen's under- lying retriever with BM25, DPR (RetroMAE), and SPLADE. To simulate real-world scenarios, for generation, all passages are re- trieved from full passage (without additional selection). From
Len
140
25
20
15
80
the results shown in Table 4 we can make the following observa- tions: (1) Compared to using oracle references, the BLEU-1 score increases while the other metries decrease, indicating that the trieved knowledge enriches the informativeness of the generation but undermines its quality. This implies that the retrieval quality crucial for the factuality of generation, and how to retrieve help- ful knowledge remains challenge the WEBBRAIN task. (2) ReGen surpasses the other three retrievers on most metrics, which implies that ReGen could retrieve more helpful knowledge. With such knowledge, ReGen could generate better answers regarding factuality, confirming the effec- tiveness of the fact-ennhncement strategies applied in ReGen. (3) Regarding retrieval performance, our Regen outperforms all baselines on all metrics (see full retrieval results in Table
ROUSZ:L
Tiglescore
40
Figure 2: Performance of ReGien (Large) with differ- ent numbers of retrieved ref- erences.
We also evaluate ReGen's performance with different numbers of retrieved references. Specifically, we randomly sample 100 queries from the test set and feed them into ReGen's generator together with zero to five references retrieved by ReGen's 's retriever. TigureBiilustree with more references, BLEU score and text length increase, but ROUGE score and triple score fluctuate. This implies that more content generated, but the quality cannot be guaranteed. The ROUGE score achieves the best with three references. Combined with the retriever's performance, we attribute this effect to the noisy nature of retrieved references. Finally, we note that, even with references, generating long and faithful text still challenging problem.
the results. Generally,
8

---

Under review as a conference paper at ICLR 2023
20
18
140-
BART
GPT
BEEWT
20
25
Trideclecee
166
GPT2
BART
120
100
20
FID
1.0
ReGien
80
60
ReGer
ReGen
Figure 3: Performance of different models with different numbers of ground-truth references.
Impact Number of References As references provide evidence for generating factual content, their quantity will impact the quality of generation. To investigate this, we randomly sample 500 queries from the test set with zero five references, respectively (3,000 in total). Different from the previous analysis, we use ground-trus -ruth references in this experiment. Figure shows the results. We observe: (1) For models using no reference (i.e., BART and GPT-2), their performance is decreasing (0 + 5 references). Intuitively, target texts with more references are much more difficult to generate, as they require more knowledge to construct the content. This result validates our assumption that writing human-like descriptions for queries heavily relies on external knowledge, which is difficult for pre-trained models to store as parameters. (2) The performance of FiD and ReGen are improving in terms of BLEU and TripleScore. This is because target texts with more references are usually longer, and the references can provide more information for generation. However, as suggested by ROUGE, more noise also appears in the generated text.
Performance vs. GPT-3 GPT-3 (Brown et al. 2020 large-scale pre-trained language model with 175B parameters. The model demonstrates strong few- or zero-shot performance on many text-based tasks, as vast knowledge has been stored in its parameters. To compare this strong model with ours, we first ran- domly sample 100 queries from the test set. Then, for GPT-3, we use the prompt "Introduce [X] in Wiki Mikipedia-styy
40
GPT Pramatt ReDen (Lange)
35
30
15.
10
OAGS
to generate zero-shot results For our model, we use the query and evidence provided by the retriever as input. The results of automatic metries and human annotation are shown in Figure and the last row of Table Our model outperforms GPT-3 by a large margin in terms of BLEU. METEOR, and ROUGE. This implies that our model can generate texts that are closer to the ground-truth (written by humans), indicating that explicit knowledge in web corpus is essential for generating natural and informative text. Only increasing the size of the lan- guage model would fail to capture such knowledge. We also observe clear improvement in our model on factualness metrics. Even with a large number of parameters, it is evident that GPT-3 can- not guarantee factualness. In contrast, evidence from Web corpus can reduce these hallucinations. Due to the limited space, please refer to Appendix for more experimental results and discussion.
Figure 4: Performance of GPT-3 (Prompt) and our ReGen (Large) 100 test samples.
CONCLUSION
In this paper, introduce new NLPtask, WEBRAIN, which aims generate fluent, informative, and factually correct description for factual queries via mining supporting evidence from a large web corpus. We construct large-scal dataset WebBrain-raw the data foundation of WEBRAIN. WebBrain-raw constructed by crawling all Wikipedia articles and the corresponding reference articles. Through empirical analysis, we found that most current NLP techniques lack the ability to maintain the factualness of the WEBRAIN task. To improve the performance on this task, we propose a new framework ReGen which uses several specifically designed strategies. It outperforms all baselines on automatic and human evaluation. We believe that WEBRAIN would highlight a valuable research pathway in which AI models could acquire knowledge from the Web world by themselves and serve human beings by fulfilling a broader range of frie-triented information needs.
We use the model text-dat -002 provided in wwwphhtts////m /tttpss:/bbta... openai

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pp. 344 354. The Association for Computer Linguistics, 2015. doi: 10.3115/11/p15-10344 URL httt:tttss///..oo/ //tp://o/o.../..////
3115/v1/p15- 1034
Satanjeev Banerjee and Alon Lavie. METEOR: automatic metric for MT evaluation with improved correlation with human judgments. Jade Goldstein, Alon Lavie, Chin- Yew Lin, and Clare R. Voss (eds.). Proceedings of the Workshop on Intrinsic and Extrinsic Evalua- tion Measures for Machine Translation and/or Sunmarization @CL 2005, Ann Arbor, Michi- gan, USA, June 29, 2005, pp. 65-72. Association for Computational Linguistics, 2005. URL https: h/tc://caaaattac antho ogy. org / W05-09097
Siddhartha Banerjee and Prasenjit Mitra. Wikiwrite: Generating wikipedia articles automatically. In Subbarao Kambhampati (ed.), Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. IJCAI 2016, New York, NY. USA, 9-15 July 2016, pp. 2740-2746. IJ-
CAI/AAAI Press, 2016. URL LRL httt:://wwwt i jjaii. org/ wbstttattttrsssaee......ssssscte / 389
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang. Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuee Hon. Unilmv2: Pseudo- language models for unified language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 652.652. PMLR, 2020. URL http:://prooeedngg. html
mlr. setess/v199/aa00
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), Proceedings of the 26th Annual International Conference on Machine Learning, ICMI. 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pp- doi: 00.1145/15533741553333 1553374 1553380
41-48. ACM, 2009.
URL https::/ http://i.iorrgg...1 / 1145/
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Mesh-Tensoriie March 2021. URL //doi. org/ If you use this software, please cite it using these metadata.
1...528/122 2t./1//zeodo..o 5297715
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htttppps//ppooceeiins...ooe neurips. cc/pppper/02///aa// 1457c0d6bfcb> cb4 492D44977422 wtt.6/ff......4............2226. -bssaacttttttttt-ttttttttt..... html
Abst
Deng Cai, Yan Wang. Huayang Li, Wai Lam, and Lemao Liu. Neural machine translation with monolingual translation memory. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- fies and the 1 1ihh International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 7307-7318. Associa- tion for Computational Linguistics, 2021. doi: 1ttgg//0..1653/1/2200000 aoo..//
URL https:
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and
10

---

Under review as a conference paper at ICLR 2023
Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT
2019, Minneapolis, MN, USA. June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational X Linguistics, 2019. doi: 10. (0.18663/11/11191442 URL
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-pone poweered conversational agents. In 7th International Conference on Learning Reprexentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL.https://openeeee
ew net. www.:orrmmmmmmma///mm d=r 731RgKm
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' AlchheeBu,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13042-13054, 2019. URL http:::: ptttps::/proccedinng. ineurrrreinngnneerripp..... neurips. http:igs.nr..pssseppepeeeeeeeeee Aaaieesaaabbtrtt...t..
Hady BlSahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon S. Hare, Frédérique Laforest, and Elena Simperl. T-rex: A large scale alignment of natural language with knowledge base triples. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerek, Sara Goggi. Kôiti Hasida, Hitoshi Isahara, Bente Macgaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceed- ings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018 European Language Resources Association (ELRA), http://////reee////////reeer/e/ summaaiissseeee
2018. URL http://www rrec- conf.
632. html
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval. CoRR, abs/2109,10086, 2021a. URL httpt:://///xxv..coom org. http:/bb//21090 10086
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE: sparse lexical and expansion model for first stage ranking. In Fernando Diaz, Chirag Shah. Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 2288 2292. ACM, 2021b. doi: 00.1145/344433534433099 URL https:
1145134411/34443 3463098
Luyu Gao, Zhuyun Dai, and Jamie Callan. Rethink training of BERT rerankers in multi-stage re- trieval pipeline. In Djoerd Hiemstra, Parie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (eds.). Advances in Information Retrieval 43rd Eu- ropean Conference on IR Research, ECIR 2021, Virtual Event, March 28 April 1, 2021, Proceedings, Part II, volume 12657 of Lecture Notes in Computer Science, pp. 280-286. Springer, 2021 doi: 11110007/778-3-00--222222 26 URL hRttpsts: ot..///.g//01001 000-3-3030-2200000
Michael R. Glass, Gaetano Rossiello, Md. Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. Re2g: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 2701-2715. Association for Computational Linguistics, 2022. doi: 10.18653/w1/2022.mmr httpps://oii 3ttp://d...0000...11666663333333 /0w.116655////222 nwa.lalal---....m maalmmmainn... 194
mww.clmaii.n.aa..mmamm URL
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. Assessing the factual accuracy of generated text. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 166-1666771 175. ACM, 2019. doi: 10. 10.1145/32925003333955. URL https: /oii. orq/10. 1145/3292500 3330955

---

Under review as a conference paper at ICLR 2023
Jiatao Gu, Yong Wang. Kyunghyun Cho, and Victor O. K. Li. Search engine guided neural machine translation. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artifi- cial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5133-5140. AAAI Press, 2018 URL https: https://wwww... /ttp://wwwaaaaa....//o////.....a aaai. org. /it///idxx//x//// php/ Abpp/AAAIAAAAIA
paper/view 1 7282
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wos Wei Chang. REALM: retrieval-
augmented language model pre-training. CoRR, abs/2002.08909, 2020. URL https://
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2535-2544, PMLR, 2019. URL htkLhtttpp/prcceeinng. /egppceeeeinnss.....preeeeeelve hwwp:/9//hhhhhenne 9a html
Hoon Heo. Factsumm: Factual consistency scorer for abstractive summarization.
https:
//github. com/Hufooou http://uuffooffttsuuummum 2021.
Robert Iv, Alexandre Passos, Sameer Singh, and Ming-Wor Wei Chang. FRUIT: Faithfully reflect- ing updated information in text. In Proceedings of the 2022 Conference of the North Amer-
ican Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, pp. 3670-3686, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 1111666////2222.maeee mailimmiinn.99 URL httpttss//allaattoollooorr//////
naacl seacl-main. main
269
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 874-880. Association for Computational Linguistics, 2021. doi: 10.18653/v1/001 eacl- main.7 74. URL
eac wwa.aaaalaaaa....
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. (6611 Association for Computational Linguistics, 2020. doi: 10.18653/v emnlp-main.555
emnlp ma in.ainn55000
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Thang Lu- ong. Alexandra Birch, Graham Neubig, and Andrew M. Finch (eds.), Proceedings of the First Workshop on Neural Machine Translation, NMT@ACL 2017, Vancouver. Canada, August 4, 2017, pp. 28 39. Association for Computational Linguistics, 2017. doi: 10. (0.18653v1/w11-3320 URL httts::///o//t/t http://oo.ooorg//// 1ttpg//0011115///11/11111----
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins. Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452 2019, doi: 10. al 00276. URL htLLttts:// http::/doi...gg.g 1162/tacl_a a_ 00276
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-scquenee pre- training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL 2020, Online. July 5-10, 2020, pp. 7871-7880. Association for Computational Linguistics, 2020a. doi: 1..18653/v1/0220 211-200.cllmai
URL
https://doi http://dooooorg///// wro//00..0.. /653/ /0 acl -ma ww....nnn.... 703
12

---

Under review as a conference paper at ICLR 2023
Patrick S. H. Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kicla. Retrieval-augmented generation for knowledge- intiggeveeteeivve NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020b. URL http::///ppoceeenng..eeeeeee 00013322222377800222222623222200
cc/paperr2220//aatl
10888e02e649522ee... 74
Abstract. oastrsasaaaaeeeaeaaaaaaa.......n html
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: http///aaaaahhcccco ww1.:/y-1r9//04-111111100
Jimmy Lin, Xueguang Ma, Cheng-Chieh Lin, Jheng-Honp Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: An easy-to-us python toolkit to support replicable ir research with sparse and dense representations, 2021.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018. Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL (ttthtts: htttipss///peeeeev.eew openrevie...... net/forure F Hyg
Yang Liu and Mirella Lapata. Hierarchical transformers for multi-document summarization. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers. pp. 5070-5081. Association for Computational Linguisties, 2019. doi: 10.18653/v1/p19- 1500. http:////RR...././//../.././////
1500
Zheng Liu and Yingxia Shao. Retromae: Pre-training retrieval-oriented transformers via masked auto-encoder. CoRR, abs/2205.12033 2022. doi: 10.48550aaxxxv22255120333 URL http:
//doi. 48554 2205. 12035
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. net htt//formmm. -eiiidd-kkeeee wii.dd-ikkkkggk..
OpenReview.net, 2019. URL https:://oppeeeee/////
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, H. Francis Song, Martin Chad- wick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingaan, Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes. CoRR, abs/2203.11147, 2022. doi: 10.48550arxi.2223.1112. URL / otg/1o..
48550/arXiv. 2203.
11147
Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. Rethinking search: making domain experts out of diletrantes. SIGIR Forum, 55(1):13:1-13:21,, 2021. doi: 10.1145/347641534476646 URL
https:://oi.orr//0.0
34764476111. 3476428
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. Xu Jiang, Karl Cobbe. Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question with human feedback. CoRR, abs/21 12.b//2222 2021. URL
https:://axxiy abs 2012.1122122 09332
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002. doi: 1033115/1073003.0773335 URL https: http:::://aaaaahhahaaaa
orq /P0/ 1040/
Biswajit Paria, Chih-Kuan Yeh, Ian En-Hsu Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. Minimizing flops to leam efficient sparse representations. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020. URL http:httss//oeeeeiiiiiie htt.://wwww.....eeooooooooooetna
13

---

Under review as a conference paper at ICLR 2023
Laura Perez-Beltrachin Yang Liu, and Mirella Lapata. Generating summaries with topic tem- plates and structured convolutional decoders. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 5107- 5116. Association for Computational Linguistics, 2019. doi: (0.18653/11/111-15554 URL
1504
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin. Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-IILT 2021, Online, June 6-11, 2021, pp. 2523-2544. Association for Computational Linguistics, 2021. doi: 10.18653/v1/202 www.nnaaaamaannnaa
https: / doi /tt.:/ii.oggg//// (ttp://..11653//1//2/1 maap:///11122222maaaaaaammmmmmm http://w///221..mmmmmaa.... .naacl-main.200
URL
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gau- tier Izacard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-tau Yih, and Sebastian Riedel. The web is your oyster snowledge-intensing NLP against very large web corpus. CoRR, abs/2112.09924, 2021. URL htLhhttps:// //apxii///axxiia/ oww.iiiiwooioooooooooooooooooorr abs /tbs//2112 2212. 09924
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67 2020. URL http:://jamt oolg.org//oooooooooooomooooooolr papers/v21/20 074. html
Vikas Raunak, Arul Menezes, and Marcin in neural machine translation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilck Hakkani-Tür, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-IILT 2021, Online, June 6-11, 2021, pp. 1 1112111811 Association for Computational Linguistics, 2021. doi: 18 .2.
The curious case of hallucinations
naac. naaci pataacc-mannn -main. n. 92
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 1835-4045.05 Association for Computational Linguistics, 2018. doi: 10. (0.18653/11/11--14377 URL httthhtts://dee http://ti..o/////////// 18653/vi/d18 1437
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric ques- tions challenge dense retrievers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Repub- lic, 7-11 November, 2021, pp. 6138 6148. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2011 eenb222.eemlppmain 496. /RI.httpss: /doi wrt/ httgg//..1653//1/333 emn 1 p-mma-mm--m- i n, 496
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised met- rics in task-oriented dialogue for evaluating natural language generation. CoRR, fbss//006.09999 2017. http:tttp//aaaa wwwp:iiv.orggaaaaoooee wwg./bgg//......o.o 1706.6000 C9799
Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulezyn, Markus Strohmaier, and Jure Leskovec. Why we read wikipedia. In Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (eds.), Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017 pp. 1591 1600, ACM, 2017. doi: 10. 145/ 3038912.3052716 URL http:////.//.////////oo/oo/o.//. http://0ooooo////////////// 10. htt..//./....1111133331111111220
14

---

Under review as a conference paper at ICLR 2023
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MASS: masked sequence to quence pre- ave-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5926-5936, PMLR, 2019. URLhttt:: http:://proceeeinng. ml www....r....v..... cong19d.html
Machine Learning, ICML 2019, 9-
Yu Sun, Shuohuan Wang, Yu-Kum Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu, ERNIE: enhanced representation through knowledge integration. CoRR, 0es/1994.09223, 2019. htl.htttp//arxx..oorr////
09223
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663,, 2021. URL
James Thorn,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: large- scale dataset for fact extraction and verification. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pp. 809-811 819. As- sociation for Computational Linguistics, 2018. doi: 10. 10.18653/v1/n18- 1074. URL https:
//doi. wttp:///.....113/////11111111-11
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 4566-4575 IEEE Computer Society, 2015, doi: 10. 1109/CVVPR.015.722900
URL http:: http:://ddooo...g/////// 1119/ 201 7299087)
Oriol Vinyals and Quoc V. Le. A neural conversational model. CoRR, abs/1506.05869, 2015. URL http://arxiv. org. abs htt.b///15560055
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5008-5020, Association for Computational Linguis- tics, 2020. doi: 10. 18653/v wtw.18666112200...oo 2020.ac1-man in. 450.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mar- iani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.). Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pp. 4003-4012. European Language Resources Association, 2020. URL antho www.yoorg/222.. Irec-1. 4947
Jason Weston, Emily Dinan, and Alexander H. Miller. Retrieve and refine: Improved sequence generation models for dialogue. In Aleksandr Chuklin, Jeff Dalton, Julia Kiseleva, Alexey Borisov, and Mikhail S. Burtsev (eds.), Proceedings of the 2nd International Workshop on Search- Oriented Conversational AI, SCAI@ EMNLP 2018, Brussels, Belgium, October 31, 2018, pp. 87-92. Association for Computational Linguistics, 2018. doi: 10. 11865//11/w1855177 https://aoi.org//1 http:/....1163//111
URL
5713
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. CORR, abs/1910.03771. 2019. URI.
http:://axxiv org/ abs/ 1910 03771
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual
15

---

Under review as a conference paper at ICLR 2023
Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpss///pentteveew.neee. Eorum?:0=228222yy2.n
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- mual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 14, 2021, virtual, Pp. oasaaaaeeeeeaaaaaaaaaaaaaaaaaaea
2021. URL pttpp:::pppcceeiiiiii//// www.ips.eripoooooe cC/
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https: / //pp://///peeevveeeeeeeee eww. net
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: en- hanced language representation with informative entities. In Anna Korhonen, David R. Traum, and Lluis Märquez (eds.), Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACI. 20/9, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 1441-1451, Association for Computational Linguisties, 2019. doi: 10. 10.18653//1/11--113 URL http:/ppe//ooo.oogg/...1111/////
Yutao Zhu, Zhicheng Dou, Jian- Yun Nie, and Ji- Ji-Rong Wen. Reboost: vetrieval-boosteer sequence- to-sequence model for neural response generation. Inf. Retr. 23(1) 2020. doi: 10. 1007/ s10791-019-09364-x http::////oao.rrr////..0.///1113
16

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
In our ReGen, the retriever is training on the WebBrain-R dataset for five epochs, initializing from speiiieet-splade-eee -a-rreee We set the batch size to 192, the max query length to 16, and the max passage length to 256. We use the AdamW optimizer with a learning rate of 2e- 5. We set (Ny, (d) 5e-3) for the Eq. 6 For baseline models, train SPLADE and DPR with their official settings on the WebBrain-R dataset for five epochs. The underlying RetroMAE We implement the models with Pytorch and We train the models on 8 Tesla V100 32G GPUs. For the BM25 retriever, we use the Pyserini tool with its default parameter setting (Lin al., 202T). As for the generator, we use BART- based and BART-large checkpoints provided by HuggingFace (Wolf et al. 2019) to initialize the model. For training, we use 32 and 64 Tesla V100 32G GPUs for the base and large models, respectively. Correspondingly, the batch size of each GPU is set 8 and 4. AdamW optimizer (Loshchilov & Hutter 2019) is used for optimization with a learning rate of 5e-5. In the inference stage, we apply beam search (width=5), and the maximum generation length is set as 512. This generation setting is also used for GPT-3 (prompt). We provide our code in an anonymous repository for review
of DPR is initialized from the BEIR checkpoint [
Pytorch-Lightning _
Construction of WebBrain-R For each reference document, we perform reference passage se- lection: the passage with the highest Pst value is kept. The reason for the passage selection is to keep the working dataset at reasonable size (considering efficiency), and meanwhile to contain at least one relevant passage support generation. In this way, as shown in Table we construct million-level corpus for WebBrain-R. In this corpus, the Wiki title and the selected passages form a positive pair. To obtain negative pairs, we index the corpus with BM25 engine (Lin et al.) 202 which mines negative passages for each query by randomly selecting four passages from the top 30-50 results.
Construction of WebBrain-G Similar to the construction of WebBrain-R, we perform reference passage selection for each reference document. Then, for each sample, we concatenate the Wiki title and each reference passage respectively as input. The first section of Wiki page is used as target.
B HUMAN EVALUATION
To further evaluate our ReGen model, we perform human evaluation terms of three metries:
(1) Fluency evaluates the linguistic coherency of the generated text; (2) Informativeness measures whether the generated text gives a response in specific details based on the query and references; and (3) Faithfulness reflects the proportion of the generated text that be supported by the retrieved supporting evidence. To reduce the influence of objective factors the manual annotation, we perform pairwise evaluation method: by giving the results generated by ReGen and other baseline models, three annotators are asked to judge which text is better or if there is a tie. For fairness, the results from different models are randomly shuffled, and the annotators would not know the underlying model in advance. The evaluation results are shown in Table 5
C REFERENCE MARK CORRECTION
reference mark used to indicate the source of a referenced text. Correct reference marks can directly improve the factualness and readability of the generated text. However, as the FiD model simultaneously performs attention on the representations of all tokens in multiple references, it is hard to identify the correct source of generated sentence. We try to alleviate this problem by using a Reference mark- Enhanced Decoding (RED) strategy: in the inference stage, if a reference mark (other than [C]) has the largest generation probability at decoding step, we extract the last gen- erated sentence and match it with each reference. Then, the reference with the largest word overlap
htteeer/wangiinieeooooooeeeeffes parggpp--ee-----------e------e and https://
httess//uuuliitaeeccc//mmmaaaaan BEIR
17

---

Under review as a conference paper at ICLR 2023
Table 5: Human annotation results. The first three pairs are all base models.
Fluency
Informativeness
Faithfulness
Comparison
Win
Tic
Losc
Win
Tic
Lose
Win
Tic
Lose
ReGen vs. BART
39%
52%
9%
29%
63%
8%
42%
46%
12%
ReGen vs, GPT-2
42%
43%
15%
30%
59%
11%
54%
30%
16%
ReGen FiD+BART
22%
63%
15%
17%
74%
9%
37%
51%
12%
ReGen (Large) vs. GPT-3 (Prompt)
48%
22%
30%
34%
51%
15%
72%
4%
24%
Table 6: Performance of ReGen (Large) on samples with different numbers of retrieved references.
#Refs.
BLEU-1
BLEU-4
METEOR
ROUGE-L.
CIDEr
QAGS
TripleScore
Len
4.27
1.48
7.67
21.58
46.99
17.14
8.53
41.71
10.04
3.17
9.00
21.56
26.85
53.46
10.93
61.65
15.42
5.23
10.81
22.40
21.14
49.04
12.21
76.73
23.02
7.34
13.21
22.94
19.08
49.36
11.16
100.32
4
24.70
7.72
13.64
22.24
19.30
49.46
11.48
114.86
28.94
8.74
14.87
21.27
19.30
48.72
12.56
141.55
Table 7: Performance of models on samples with different numbers of ground-truth references.
Model
#Refs.
BLEU-4 BLEU-1
METEOR
ROUGE-L
CIDEr
QAGS
TripleScore
Len
0
12.90
4.87
11.28
25.84
49.31
25.22
12.75
45.60
1
10.92
4.11
10.04
23.50
39.45
29.40
10.43
50.42
2
8.21
3.59
10.11
23.10
40.13
30.70
11.05
51.10
BART
4.16
1.20
7.39
16.67
13.16
27.75
8.72
53.96
2.62
1.08
7.20
14.21
25.03
7.88
58.49
5
1.60
0.41
6.13
11.33
1.96
25.22
7.08
73.29
0
15.02
4.11
10,34
22.70
27.94
23.88
7.24
56.01
1
10.91
2.96
8.75
19.66
18.84
25.32
5.81
56.35
2
9.66
3.80
9.84
21.25
33.77
30.09
8.56
57.57
GPT-2
3
4.72
1.19
6.70
14.91
8.99
25.89
5.30
60.34
3.53
1.29
6.82
13.24
10.36
22.89
6.45
66.95
5
L.88
0.33
5,60
10.40
1.49
23.08
5.23
75.90
0
10.49
4.18
10.84
26.50
49.18
20.68
12.02
40.68
1
11.42
5.16
13.06
30.73
70.52
59.63
17.57
46.17
2
15.06
7.18
15.51
30.68
66.91
57.91
20.05
62.31
FiD
3
10.87
4.04
13.48
24.57
29.23
59.46
18,90
72.64
13.01
5.58
14.96
22.51
17.81
55.26
18.46
96.29
5
13.45
4.97
14.36
19.60
14.30
50.64
18.61
138.41
0 11.33
4.39
11.11
26.66
50.04
12.57
42.02
1
12.06
5.67
13,36
31.54
76.35
59.05
17.93
47.10
2
15.65
7.76
15.78
31.29
67.90
59.39
20.22
63.27
ReGen
3
12.58
4.84
14.09
25.21
36.29
58.84
19.45
76.87
14.30
6.17
6.17
23.20
20.48
55.13
19.47
99.75
5
17.11
7.24
15.95
21.93
19.33
50.60
20.79
149.45
ratio is selected as the source of the sentence (i.e., its reference mark is used for this sentence). To test this strategy, for each generated text, we compute a refer score follows:
P. Σ T Tnv-st
(12)
S
where is the number of sentences with references; T is the set of words in the sentence; V is the set of words in the corresponding reference; and is the set of stopwords. This score is similar to the STP score defined in Eq. but focuses how much of the content of the sentence quoted from the corresponding reference. From the results shown Figure we can see this strategy improves refer scores greatly but has less influence other metrics. This indicates that the reference mark
18

---

Under review as a conference paper at ICLR 2023
Base
Base+RED
35
30
600
Large+RED
25
20
5
BLEU-1
ROUGE-L
QAGS
TripleScore
ReferScore
Figure 5: Influence of reference mark-enhanced decoding.
Table 8: Performance of ReGen (Large) with different retrievers on the full and the selected passage set.
Model
BLEU-1
METEOR
ROUGE-L
CIDEr
TripleScore
ReGen (Full)
27.81
14.89
22.00
19.40
13,68
BM25
27.16
14.54
21.38
17.32
12.98
+ DPR
24.57
13.17
19.82
15.21
10.74
SPLADE
25.60
14.98
20.71
14.24
12.35
ReGen (Selected)
24.71
12.99
21.19
20.12
10.97
BM25
25.76
13.73
19.94
14.85
10.43
DPR
20.42
12.71
16.11
6.44
6,05
SPLADE
23.84
14.24
18.98
10.80
9.48
Oracle
17.97
15.31
30.09
62.39
19.43
is labeled more accurately. We apply this strategy in our demo (described in Appendix H which also improves the demo's 's readability. However, our strategy is just an early exploration reference mark accuracy. It is only involved in the inference stage, but not in model training. Moreover, how evaluate the reference mark's accuracy also an unsolved problem. We leave these problems for our future work.
D MOTIVATION AND IMPACT OF REFERENCE PASSAGE SELECTION
As introduced in Section 3.2 during the dataset construction, we perform reference passage selection build both WebBrain R and WebBrain-G. They have different purposes.
For WebBrain-R, we construct a selected reference passage corpus for retriever training and eval- uation. The reason that WebBrain-RP considered working dataset for training, on which the experiments should take a reasonable amount of time to ensure efficiency. Thus, instead of using the full reference passage set, which requires a long time to index, we keep the selected passage corpus at reasonable size (million level) to evaluate the retrievers in the training phase.
For WebBrain-G, our passage selection strategy can ensure that the provided references are relevant (can be treated oracle references), thus helping the model learn to extract useful information from the references. Such filtering may not be possible in practice, but it is a common approach used to build workable dataset in many previous studies.
To simulate real-world scenarios, we conduct an end-to-end evaluation in Section 5.3 In this exper- iment, ReGen's retriever needs to find evidence from the full reference passage (containing about 204 million passages, which is significantly larger than WebBrain-R) R), and ReGen's generator has generate texts based the retrieved evidence. As comparison, also conduct experiment
19

---

Under review as a conference paper at ICLR 2023
Table Performance of various retrievers on WebBrain-R.
Model
R@!
P@1 1
R@5
R@ 10
R@
MRR
MAP
BM25
0.290
0.292
0.558
0.645
0.716
0.411
0.410
DPR (BERT)
0.192
0.193
0.348
0.403
0.457
0.263
0.263
DPR (RetroMAE)
0.228
0.229
0.390
0,438
0.478
0.301
0.301
SPLADE
0.389
0.392
0.724
0.815
0.877
0.537
0.536
ReGen
0.399
0.401
0.746
0.839
0.896
0.551
0.550
Table 10: Performance of different models WebBrain-G. The models evaluated by semantic similarity metrics. The best results are in bold.
BERTS
BARTScore
Model
F1
(lag-likllihod)
BART (Basc)
96.8
94.4
95.5
-4.755
BART (Large)
97.6
94.8
96.2
-4,88
GPT-2
87.3
85.4
86.3
-4,60
FiD+BART (Base)
96.2
93.9
95.1
-4.911
FiD+BART (Large)
97.6
94.9
96.2
-4.114
ReGen (Base)
96.2
93.9
95.0
44,71
ReGen (Large)
97.7
97.8
97.8
-3,936
ReGen (w/ Retriever)
90.2
84.5
87.3
-4.033
BM25
91.6
91.3
91.2
-4.326
+ DPR
83.3
91.4
87.1
-4.664
+ SPLADE
88.1
85.5
86.8
-4.381
on WebBrain-R. This helps us understand how the size of the passage set impacts the end-to-end performance of ReGen. The results are shown in Table We find that:
(1) When using the selected reference passage set, all retrievers' performance decreases over almost all metrics, indicating that the retrievers can find more useful information that benefits generation in the full passage set.
(2) When using the full passage set, our ReGen's retriever surpasses BM25 regarding BLEU-1 and METEOR, proving that ReGen's retriever generalizes better in the large-scale passage set, and therefore finds passages with better informativeness.
(3) Though the full passage set much larger, the performance of all the retrievers has a similar tendency, which verifies that reference passage selection simple yet effective method for dataset construction. Evaluating models' performance on the selected passage set can well reflect its per- formance the full passage set, which is an important finding that allows researchers to conduct experiments on the WebBrain-R.
(4) Although ReGen's generator is trained with oracle passages, it generalize well to the retrieved passages for generation. This demonstrates that learning on WebBrain-G can help the model learn extract useful information from the references.
We will also release the full passage set to support in-depth research on the WEBBRAIN task.
E EVALUATION OF SEMANTIC SIMILARITY
In Section 5.2 we evaluate models by several n-gram overlapping-based metrics. Recently, there have been some new model-based metrics that can evaluate text similarity the semantic-level. Therefore, we also employ BERTScore (Zhang et al. 2020) and BARTScore (Yuan al. 2021 for evaluation For BERTScore, we report precision (P), recall (R), and Fl. For BARTScore,
We the checkpoint provided in their Github repository. BERTScore: Fiiiger/bon httpbbbrtt__rrre BARTScore: https://gitthbb.co http://ttthhh..cco///e////////// HbAAAl/SSOOAAARAeAereeoo
20

---

Under review as a conference paper at ICLR 2023
Table 11: Performance of GPT-3 (Prompt) and our ReGen (Large) on 100 test samples. The first prompt is "Introduce [x]:", while the second prompt is "Introduce [X] in
wikipedia-style. ". The second prompt has a hint of generating Wikipedia- content.
Model
BLEU-1
BLEU-4
METEOR
ROUGE-L
CIDEr
QAGS
TripleScore
GPT-3 (Prompt I)
17.24
2.19
9.14
13.57
6.70
17.00
0.29
GPT-3 (Prompt 2)
19.72
3.60
9.93
14.89
9.33
18.16
2.08
ReGen (Large)
28.94
8.74
14.87
21.27
19.30
48.72
12.56
it is an average log- bikeliii,hood,, so its value is negative. Higher scores indicate that the generated text is more semantically similar to the ground truth text. The results are shown in Table To] In the upper side of the table, we can observe that our ReGen (large) achieved the best results among all models. This is consistent with the results Table 3 demonstrating the superiority of ReGen in generating semantically correct text. More concretely, the recall score of the ReGen (large) is considerably higher than that of other models. This indicates that it can cover the content of ground- truth texts more effectively. In the lower side, we compare the performance of ReGen (large) with other retrievers. Considering BERTScore, the recall score of DPR (a dense retriever) is higher than precision score, showing the opposite trend to other retrievers. This suggests that the retrieved results returned by DPR can cover more content with the ground-truth target but also contain much noise. Combining references from different retrievers may be promising strategy for improving generation quality.
F ADDITIONAL PERFORMANCE COMPARISON WITH GPT-3
In Section we compare our ReGen model with GPT-3. As GPT-3 is tested by prompt, the selec- tion of the prompt template will directly influence its performance. two different kinds of templates, depending whether or not they explicitly specify to generate text Wikipedia-style. After evaluating and comparing the generation quality, we finally choose "Introduce [X] =" and "Introduce [x] in Wikipedia -sstals...eee..l the prompt tem- plates. The results are shown in Table II From the results of GPT-3 with different prompts, we can confirm that prompt for generating Wikipedia-style text considerably improve the naturalness and factualness text. This also indicates the importance of prompt engineering when employing large-scale pre-trained language models. Moreover, ReGen generate texts that both closer to the ground-trui -tuun (achieving higher BLEU, METEOR, ROUGE, and CIDEr) and more factual (ob- taining higher QAGS and TripleScore). This confirms that our use of explicit knowledge in Web corpus is essential for generating natural, informative, and factual text.
our experiments, we consider
G CASE STUDY
We perform a qualitative analysis of our model based on a case study. Specifically, we first compare the generated results of different models on queries with or without references. Then, we show series of generated results for a specific query with different numbers of references.
Comparison of Different Models We first study the generation of queries without references. As shown in Table 12 where reference given, FiD, BART, and ReGen generate similar results, but ReGen can provide more details. We attribute this to the warm-up strategy, which can effectively mitigate the gap of tasks between BART's pre-training and our fine-tuning. Besides, it interesting to see that GPT-3 can also generate correct information about the election date and post. We relate this the large number of parameters in GPT-3. They are capable of storing large amounts of knowledge. However, GPT-3 also generates some detailed information about the candidates that factually incorrect (i.e., hallucination). In the second case shown Table 13 we use green and red colors to label the correct and incorrect information. We can see with one reference provided, both FiD and ReGen can generate more faithful text than BART. This reflects the importance of external knowledge. Furthermore, reference. The result also demonstrates the effectiveness of our proposed warm-up strategy, as it can facilitate our model's capability of reffrenc matching. However, GPT-3 performs poorly
also observe that ReGen can better capture clues from the
21

---

Under review as a conference paper at ICLR 2023
Table 12: Generated results of a query without reference. The green part of information only correctly generated by ReGen.
Query: 2016 Pateros local elections
Ground-truth reference: None
Ground-truth target: Local elections in pateros were held may 9, 2016 within the philippine general election. The voters elected for the elective local posts in the city: the mayor, vice mayor, district representative, and councilors, Six in each of the city's 's two legislative districts.
BART (Large): Local local elections were held in pateros on may 9, 2016 within the philippine general election. The voters elected for the elective local posts in the city: the mayor, vice mayor, and ten councilors. may 2016 within the philippine general election. The voters elected for the elective local posts in the city: the mayor, vice mayor, and eight councilors.
FiD (Large): Local elections was held in pateros
GPT-3: Pateros held its 2016 local elections on May 9, 2016. Elections were held for the mayor, vice mayor and councilors. incumbent Mayor Jaime Medina was term- limited and could not run for re-election. Medina was first elected in 2004 and was re-elected in 2007 and 2010. incumbent Vice Mayor Nestor Padilla was term-limited and could not for re-election. Councilor Mario Marasigan was the only incumbent councilor who ran for re-election. A total of 15 candidates ran for mayor, while 24 candidates ran for vice mayor. 55 candidates ran for councilor.
ReGen (Large): Local elections was held in pateros on may 9. 2016 within the philippine general election. The voters elected for the elective local posts in the city: the mayor, vice mayor, the two congressmen, and the councilors, in each of the city's 's two legislative districts.
in this case. It can generate a natural and fluent text with lots of information, but most of the facts are incorrect. Due to the fact that all knowledge is stored as parameters in GPT-3, it is challenging to determine which part is relevant to the current query. In contrast, ReGen can leverage the knowledge provided in given references, thus generating more factual text.
Impact of Number of References We also evaluate the generated results by feeding various num- bers of references into ReGen. Here, we show a query selected from the test set used in the section "Impact of Retrievr". All used references are retrieved by ReGen's 's retriever. The input query and retrieved references are presented in Table [4, while the generated results are shown in Table [5 For the generated results with n references, we use the first retrieved references in generation. Generally, we can observe that the generated results become longer when more references are given, reflecting the importance of references in content construction. Then, even without reference, Re- Gen is able to generate text with some factual knowledge. This suggests that ReGinn can also store some knowledge in its parameters. We can also observe that the text factualness greatly enhanced when more references are provided. Nevertheless, we still notice some problems in both the trieved references and the generated results. As for the references, some of them are uninformative (such as the second reference) or less relevant (such the fifth reference). They introduce noise to the generation process. As for the generated results, they contain some hallucinations caused by mixing information from various references. All of these results demonstrate that there is much room for improvement in existing methods.
H SYSTEM DEMONSTRATION
Though there a gap between our study and our ultimate goal of generating Wikipedia articles, we still explore potential application scenarios for our current model. We build a demonstration system as shown in Figure Figure and Figure This system can generate either a short article or full page. For generating short articles, we test both entity-style queries similar to Wikipedia titles and natural language-ste styul queries. We illustrate and discuss them respectively.
As shown in Figure 6 first input entity-style query of "Google Translate". This scenario is the same as model inference. We can see our system can generate content relevant to the query. Besides, with the reference mark-enhanced decoding strategy, we can see the reference mark is generated more accurately, which greatly enhances the readability of the generated text.
In addition, also examine the system with natural language-style queries (shown in Figure Intriguingly, we find that though ReGen is trained on entity query-based dataset, it can generalize well in some natural language queries, which have more specific information-seeking intent. Com-
22

---

Under review as a conference paper at ICLR 2023
Artilo
Google Translate
Google Translate
Google transiate is free and open source machine tranclation service provided by google. It can tranclate words sontences and wrh pages belween any combinelion inn supported languages. [1] The service evailable for windows, macus, ius, android, end web browsers. wes presiously available on the google los android opcrating systems. The service uscs convolutiooal noural networks to automotically translate one language another. EI Injuly 2013, google ennounced google translate would be integrated into google chrome
google translate. In january paid laaun service that ollers full lange froe and peid itanolation services, As ol januery 2312 updale,, the service supports languages. The service was previously known google translete, but was renamed google translate in october 2016, Google tronslate is availablein in english french, gemmer, italian, spanish, bilgarian, catalan, cmaaian, danish, dutch, hungerian, finnish, polish, mimenian, slrvek, turkish, and ukrainian, In addition to its own transletion service, google translate also ofters a number of other products and services, including google translate voice, google trandlaee pirto, google transiator video, and google translate audio.!!
[8] In october 2014, service
google launched google translate pro,
Reference
1. a Pttps// http:////miiiiggggoooooooooobbb cog.loooooocoo/auuuutieee
2. http:s//ssscarucccccm/////////// -transistes app now inst-nnttt-lllllyy. warsslates-print printe d-seatsii-27--enguaaee
3. stanmhree-iteeeeeeeddirooooeeeee
Figure 6: Screen shot of our system: entity query.
pared to the query "Google Translate", the intent of the query "how Google Translate work" has a more specific intent. As a result, the generated text is closer to the intrinsic functionality of Google Translate.
We also try to generate full Wikipedia-style page with a heuristic method. Specifically, we first employ another model to generate some section titles for the input query. Then, we respectively concatenate the imput query and each generated section name as new queries. Finally, ReGen can generate a short article for the original query and each section recursively. Figure demonstrates the result of "Google Translate". The generated section titles "early years of Google Trans- late", "neural translation model", and "translate tools". Through the case, we can find: though the generated text can provide good introduction to the section title, different section titles do share a reasonable taxonomy given the input query. For example, the section title "translate tool" seems be more general than "Google Translate". We will continue exploring how to generate full Wikipedia pages in our future work.
I LIMITATION AND FUTURE WORK
In this work, we explore generating short factual articles (like Wikipedia pages) for queries by collecting evidence from the Web. This is very early step towards our ultimate goal: automatically building Wikipedia pages for unseen topics. We collect new dataset to support future research, evaluate several existing models on the new task, and propose some effective strategies. However, there are still many limitations in our study that need further investigation in the future.
(1) The retriever is far from perfect. As shown in our experiments, ReGen's performance with retrieved evidence is worse than that with oracle references. So, two problems appear: (a) how to improve the retriever's performance; and (b) how to generate a better result with non-perfect references.
(2) As we use FiD as our backbone model, hard to determine the reference mark accurately. Moreover, the reference mark accuracy cannot be measured by existing metrics. We plan to design new attention mechanisms to facilitate the model's judgment on these reference marks.
(3) In this study, try to generate short article for given queries. However, the original Wikipedia page contains more information than a short article. Generating long articles will be another chal- lenge in the future.
23

---

Under review as a conference paper at ICLR 2023
Atticee
how Google Translate work
Q
How Google Translate Work
Google translele is e free translation service provided by google. It can tremslate words, sentences and web pages weiween any combination of the suppartedd languages, (1I It looks for petlerns in hundreds of millions of documents to help decide on the best trarelation far you. By dotecting patterss doruments that have alrearyy been traslated by human trandators, gnogle can make intelligent guesses as what appropriate franslation should be. (ll this process called 'statistical machine translation", IlIl in november 201 i, google announced that it would be improving ils lenguage itersletion ser vice wilh new appaoesh tha interpeels whole seriences lime teller thet phrasess piece- by-piere. The company says it will eventually make similer upgiaress to all of the 103 languages it currently supports. On may 12, 2019, google introdured a new teature that allows transtate to automatirally pop up inside any apps atter you've highlighted text a toreign
language, When you do, a little translate icon will eppear-kind of one of facebooks chat heacs-whiich you can tap to read translation.
Google has steted thet is the most eccurate translation service in world (4) As october 2020 updete, service has been evailable more than 100 languages. On october 30, 2021, google launched new translation service calleri google translate.!!) !!]
|
Reference
1. I att2/nntt////taaaattta google www.orongggggggooooooooaaaaaaaaa compsbauttttt///ttttt
3. > httpp///wwwttwwrggg.oom http://wwwwa..oooo////////////1
n. ALL
905/tsp to translate google transtete inside an y-android-aapp
pogle-transsitte--55 2010/
updato 2016/
Figure 7: Screen shot of our system: natural language query.
Arbde
Google
Google Translate
slution provided
and peao any comtination of supported lanqurges. mmm systere,
metts, anatosst
prenisaaly
gassghell
jejay Jo google annourced that google tanouts world be intagrated the google carome orooser october 2014 the sendoe renamed geople memniiss jaraary 2016, google sunched google framilte pro, palc trandation service that offers fall range of free and paid trams services As jariory uo3ou the condoc support
networes antomatiially mamilate soo from one targ guage
uagge The cordon was prodecsly innon to google tradiats but wat revooeed to google mowe ate encetw 2010. Google tronccae eva kuble orggiiil lenels yorman itulion sourions tolgurian vauuins socaiio, deridl Jutins hungerion frrihh polish, romerian clowd, tunkat, and ulramiors In addiiion ia oven bure latinn service, google transtate who otters a number of otter products and se sentoas incluting google translate voice, google transiate photo, google trmniattor video, and goog thanlldte audio!?
tartee Years Oit Google Trrmsiate
provided by goog is lounshed opril 1, soogle intraduaed new "somner welken mooe" feuture that baw users to hare real- dime cormenationo in additional lanquape.. Goog e hurraaa originali developedd content mes hhow tharsiation rewwarch dustnng the rasld
Google vamilate
lang thanolator
since oxporden supportimre than 100 longueges. jomurry
of dee aeetiv 2071 updee,, wtw.e,//wwweeeeeee thwverdka ooer 100 fin words trevdeed from then 7E1 lertz acces Ax Jeasery 7007 spstee cowel by mose live 500 eilliio www.lcinppppeeeeeeooooooo Be (4)
Neand Teazahsiins Modill
Neural mocrine trandation a mochine Pronsaation sigoriig that uses combirattion srottniical and nourti networes leam woree ds are tranciated. (771)) is nybric appreccn machine
trens sulion that connieans strengths both the stututical
inamslationn resral ratuaric approiaches,
coen somerol
ahunccl
siation systerss.
such as vardestrans ats and deepl I1PP Neurral mach transiationn has been shovn to be more wfficent than datiiiicall model. als beer shown that the necral model to weam moaning of wourti that so oonn los thoquentty in the mnining Hosoovee in etl undhin to laan woserable hrowihiiooss worth that traw vvry foo worde the dats/?
Transtane Tooks
tool
anothes
avst sorking with locallzation format and
tnolch original hally cevetoped monpatooil
by trenliteus oteemunty devrlapen.
trensiating lole uvedd getteel
for fiatiizaiionn en internal change to locus endh yoes pladfo one software, the organisation dedidedd to lovefize fermmaal suite.
requated uniing news trok and neay formo that owee nor rifh gyttert Po file ware then convened siff tiee alongride flles, the noclidss are setmen the puthon proppamming
langagg The tooliit used enchont 25 apo Khecleer. Trans the bo0kit has boen uscd to tronditee opencitinn org internal ne format it ras been acopood to handle cponottic internal for (1 has wlso been wotershed maarage clFF (ive whcr grich pall files, It the suppon for open doxumeen ment Eormat (IQ)
Redir
Shitge petermiinnnnnnn geople iebout fittlin ALL
Aotpoc/ eernencheennnni tngggn tomelstte fonandril
>*fepperiieedd core www.eiieenenneeeeeeeeeoooo
ayps
1,12224 wippliieispspslleeeeee. heodegguhhrrr get
httpe////ooogooooo baceee glgoog www.ttttooeeeeoooooooo ireggrted
meauy!
google
1254087 sepeet/ com hompan htw.pp///.//eeeeeeeeo
1.1224497 hwtp:///ww/ww/enaaeooooooooo/ ogg teaeeettee toolit
Rengenoheee Mag pooglep wwwhhr yorrs of
sts-teppchhhee oow google complaabeeeeee
Figure& 8: Screen shot of our system: entity query for generating full Wikipedia page.
24

---

Under review as a conference paper at ICLR 2023
Generated results of query with reference. The green texts are correct information, while the red ones are incorrect.
Table 13:
Query: Charles Lovell (trade unionist) Ground-truth reference 1: The Guardian Charles Lovell, who has died aged 91, led the plumbers' trade union the ptu (later the eetpu) from the 1960s to the 1980s. a period of great turmoil and change in the world of industrial relations. Charles developed partnerships between his union and employer bodies representing the industries in which his members served. The outcome was enormously beneficial both parties and to the industry. Son of Michael, a warchouseman, and Isabella, Charles was born in Newcastle upon tyne and began work in 1939. He was offered an apprenticeship as a plumber and joined the plumbing trade union, but as the second world war approached, he was drafted to swan hunter's shipyard under essential works order. He was never happy there and returned construction as soon he was able. After the war, he also immersed himself in trade union activities and became chairman the wallsend trades council and labour councillor, as well as learning esperanto and providing for his beloved wife, Irene (nee lamb), whom he married in 1944, and their three daughters and two sons. In 1966 he was elected to the executive of the ptu and in 1968 became its merge with the electrical trade
general secretary. To make his union more secure, one of his first acts union to form the cetpu.
Ground-truth target: Charles Lovell (1923 13 october 2014) was a british trade unionist. Born in Newcastle- upon-Tyne, Lovell left school in 1939 and began an apprenticeship as a plumber, also joining the plumbing trades union. Soon after, he was drafted to work at the swan hunter shipyard, but returned to plumbing imme- diately after world war [i.[I] Lovell joined the labour party and was elected wallsend trades council. HE was elected to his union's executive council in 1966, and two years later won election as its general secretary. Hc immediately led it into merger with the electrical trades union, becom- ing national officer of the new electrical, electronic, telecommunications and plumbing union. In this role, he created national pension scheme for plumbers. and championed more flexible apprentieeships, which could be completed under a variety of employers.[1] Lovell retired in 1987, and was made an officer of the order of the British empire. In his spare time, he learned Esperanto.[1] BART (Large)): Charles Lovell was a British trade unionist. Born in london, Lovell worked as a coal miner, and became active in the miners' federation of great Britain. He was elected as the union's 's general secretary. and also served on the general council of the national union of mineworkers. Lovell also served member of the executive of the trades union congress, and was delegate to the general conference the international labour organization. Hc stood unsuccessfully for the labour party at the 1918 united kingdom general election, but was elected at the 1922 general election. In 1923, he was appointed as general secretary of the union, a post he held until his death.
councillor. and also chaired
FiD (Large): Charles Lovell (1921 - 30 June 2011) was British trade unionist. Born in Newcastle upon Tyne, Lovell began working as a plumber in 1939. He joined the plumbing trade union, and was elected to its executive in 1953. Нe was also elected to the wallsend trades council, and served as its chairman in the 1960s.[1] In 1966, Lovell was elected as the general secretary of the plumbers' trade union, and led the union through a period of turmoil and change in the world of industrial relations.[]] Lovell was also active in the labour party, serving as councillor on wallsend borough council and as member of the national executive committee of the trades union congress. [1] He retired from the union in 1986, and died in 201 I, aged 91-[1] GPT-3: Charles Lovell was a noted trade unionist in the early 20th century. He was born in England in 1868, and began working in the coal mines at the age of 10. He later moved to the United States, where he worked in the coal mines of Pennsylvania. He became active in the United Mine Workers of America (UMW). and was elected to the union's 's international executive board in 1902. He also served as the UMW's vice president from 1909 to 1913. Lovell was a leading figure in the union's 's fight for better working conditions and higher wages for miners. Hc played a key role in the UMW's 's successful strike against the coal companies in 1902. He also helped to organize the union's 's successful strike against the anthracite coal companies in 1905. Lovell was a leading advocate of industrial unionism. and played key role in the formation of the Industrial Workers of the World (IWW) in 1905. He served as the IWW's first general secretary from 1905 to 1908. Lovell's 's' radical views and activities caused him to be blacklisted by the coal companies, and he was unable to find work in the coal mines after 1908. He later found work as journalist and editor for several leftist publications. He died in New York City in 1932.
ReGen (Large): Charles Lovell (1928- 15 July 2016) was a British trade unionist. Born in Newcastle upon Tyne, Lovell began work in 1939. He joined the plumbing trade union, but as the second world war approached, he was drafted to swan hunter's s shipyard under an essential works order. He was never happy there and returned to construction as soon as he was able. [1] After the war, he also immersed himself in trade union activities and became chairman of the wallsend trades council and a labour councillor,
well as learning Esperanto and providing for his beloved wife. Irene (nee lamb), whom he married In 1944, and their three daughters and two sons. In 1966 he was elected to the general council of the trades union congress.[]] He led the ptu from the 1960s to the 1980s, a period of great turmoil and change in the world of industrial relations. Charles developed partnerships between his union and employer bodies representing the industries in which his members served.|1 [1] The outcome was enormously beneficial to both parties and to the industry. [1]
25

---

Under review as a conference paper at ICLR 2023
Table 14: A query with five retrieved references. The ground-truth target and generated results are shown in the next table.
Query: Makerere University School of Law Retrieved reference 1: "MUK Students Challenge Proposal on Tuition Increment" Makerere suggested and approved a 10% tuition increase for all their courses starting with the 2014/2015 intake. A 10% increment would mean students at the school of law that have been paying 1.2 million shillings per semester would pay extra 120.000 shillings. The school of medicine where tuition has been in the range of 1.4 1.6 million shillings per semester would have 140,000 and 160,000 Uganda shillings extra fee. The fees increment has been justified by the university management as a means of raising the internal revenue of the university to cater for the burgconing expenses like lecturers' salaries and research fees. Makerere university relies on three sources of funding; student's 's tuition, government contribution and donor funds to run its 122 billion shillings annual budget. Student's contribution accounts for over 70% of the budget. Ivan Bwowe, The university guild president told journalists that they were concerned that the university is not looking at alteraatie sources of funding and only relying on contributions from students' tuition. He says the students are not prepared to let the 10% tuition increment prevail. The university chancellor professor Mondo Kagonyera on Friday decried the culture of strikes in Makerere. He said he would not negotiate with students if they continue to use violence hold the university ransom. Despitc an earlier agreement with the vice chancellor that food will be left the hands of a private contractor, the students still raised queries over how the process would be handled in their petition.
Retrieved reference 2: "List of Universities in Uganda Accredited To Teach Law" Uganda law council +256 414341673 info @ lawcouncil.co.ug georgian house, Caamplla-Ugandae the law council committee on legal education training accredited the universities listed below to teach the law degree programmes Retrieved reference 3: BBC.com a historian of cast Africa, Derek Peterson, says the fire is a disaster for Uganda and for cast Africa. The main building held student records and the archives of the Makerere University "the building holds student records, and the basement is full of archive files spanning the whole history of the institution," " he tweeted, adding that he had been intending to help organise project catalogue the collection. 1970, African presidents, including Makerere graduate Julius Nyerere of Tanzania, attended a ceremony at the university. Makerere was first established in 1922 as a technical school and has grown into a widely respected university. The construction of the "ivory tower" building began in the 1930s and was completed in 1941 its alumni include independence-era leaders such Julius Nyerere of Tanzania, renowned writers including Kenya's ' Ngugi wa Thiong' o. academics and clergy like John Sentamu. the recently retired Anglican archbishop of York. Makerere university is Africa's ffft-best university, according to the latest rankings by the times higher education related topics more on this story sex for marks scandal: 'my lecturer tried to rape mc' top African university probes degree cheats around the BBC Africa today podcasts related internet links Makerere university the BBC is not responsible for the content of external sitcs.
Retrieved reference 4: "Statement: Makerere Visitation Committee lists responsibilities" statement: : Makerere visitation committee lists responsibilities the visitation committee appointed by president Yoweri Museveni to look into the affairs of Makerere University has listed its responsibilities, stating that work will be done in timely and comprehensive manner. A statement by the committee chairperson, Dr Abel Rwendeire, indicates that work started on 17 November with a review of previous reports on the university, and will expand to in- clude consultations with key stakeholders. Makerere University was declared closed indefinitely by president Museveni on November 2016 following strike by lecturers over unpaid incentive arrears. The university stu- dents subsequently joined the strike, demanding to be taught. The committee was thus instituted to look into the issues affecting the university and report after three months. "the committee commenced its work immediately after its inauguration and has already formulated its work plan in five major areas of the university's 's corporate life namely issues of governance and institutional development; academic affairs; student affairs; staff' affairs and financial management," the statement, released on Tuesday. 22 November 2016, reads in part. read the full statement below. Makerere visitation committee statement Retrieved reference 5: "Refugee Law Project Interactions" refugee law project refugee law project the refugee law project, based at Makerere University in Kampala, works to ensure that all people enjoy their human rights, irrespective of their legal status. It seeks to empower asylum seekers, refugees, deportees, idps and host communities in Uganda to enjoy their human rights and lead dignified lives. The gender & sexuality programme works ensure that all people access, recover and enjoy their sexual and gender being and rights, and to raise global awareness of the close relationship between violations of sexuality and gender. . and patterns of forced migration. It is grounded in the reality that many persons do not access their sexual and gender rights due to social, cultural, political and legal obstacles, while many others lose their rights, as a result of sexual and baseed violence, particularly in times of conflict, and in the search for asylum. An important emphasis of such engagement has been to highlight the invisibility of men's 's and boy's experiences of sexual violence, and the extreme Marginalisation and isolation faced by male survivors.
26

---

Under review as a conference paper at ICLR 2023
Table 15: Generated results of ReGen (Large) for a specific query with various references. The query and retrieved references are shown in the previous table.
Makerere University. Kampala is Uganda's 's largest and oldest institution of higher learn- ing. first established as technical school in 1922. It became independent national university in 1970. Today, Makerere university is composed of nine colleges and one school offering programmes for about 36,000 dergraduates and 4,000 postgraduate. The main administrative block was gutted by fire in September 2020 and the cause of the fire is yet to be established. U.S. News & World Report has ranked Makerere University as the eighth best university in Africa and the 569th best university worldwide. In the 2020 U.S. News & World Report ranking, Makerere is the highest-ranked university in sub-Saharan Africa outside of South Africa The Times Higher Education World University Rankings for 2016 ranked as the fourth best university in Africa. [1] Makerere University was alma mater to many post- -oseppndeeeeeecce African leaders, including Ugandan presi- dent Milton Obote [2] and Tanzanian presidents Julius Nyerere and Benjamin Mkapa. [3] The former president of the Democratic Republic of the Congo, Joseph Kabila, and former Kenyan president Mwai Kibaki are also Makerere alumni. In the years immediately after Uganda's 's independence, Makerere University was a focal point for the literary activity that was central to African nationalist culture. Many prominent writers, including Nuruddin Farah, Ali Mazrui, David Rubadiri, Okello Oculi, Ngūgī wa Thiong's o. John Ruganda, Paul Ther- oux, Nobel Prize laureate V. S. Naipaul, and Peter Nazareth, were at Makerere University at one point in their writing and academic careers. Because of student unrest and faculty disenchantment, the university was closed thrce times between 2006 and 2016. The final time was on I November 2016 when president Yoweri Museveni declared it closed indefinitely. [4] The university was reopened in January 2017. [5]
Ground-truth target:
ReGen without reference: Makere university school of law is the law school of the makerere university in
kampala, the capital of uganda. It is one of the oldest law schools in the country.
ReGen with reference: Makere university school of law is the law school of makerere university in uganda.
[I]
ReGen with 2 references: Makere university school of law is the law school of Makerere university in uganda.
It was established in 2002. [1][2]
ReGen with 3 references: Makere university, also known Makerere university, is public research university uganda. It one of the oldest universities in the east african community and the second oldest in the country, after kampala international university. The university was founded in 1922 by the british as a technical training school. It hecame a full-fledged university in 1962. The annual budget of the university is ug $ 122 million (us $2.2 million). of which the student's 's contribution accounts for 70% of the total. [1] The university is accredited by the ugandan ministry of education, [2] and is a member of the association of commonwealth universities. Its alumni include several heads of state and government east africa, including the first president of tanzania, julius nyerere. [3]
ReGen with 4 references: Makere university, also known as Makereer university, is a public research university in uganda. It is the oldest and largest university in the country. The name "Makerere" is compound of the first letters of the names of the ugandan monarchs, mwami and nyerere. The university is located in the city of kampala, the country's 's capital and largest city. It accredited by the university the west african countries, the association to advance collegiate schools of business, and the western association of schools and colleges. It is also one of the few accredited universities in africa to teach the law. [1] [2] The university has produced many notable alumni, including former president of tanzania, julius nyarere, kenya's ngugi thiong's o. academics and clergy such as john sentamu, the recently retired anglican archbishop of nairobi. [3] In november 2016, the university was closed indefinitely following a strike by lecturers over unpaid incentive arrears. the university students subsequently joined the strike, demanding to be taught. [4] The visitation committee appointed by president yoweri museveni to look into the affairs of the university has listed its responsibilities, stating that
work will he done in timely and comprehensive manner. [4]
ReGen with references: Makere university, also known as makerere university, is a public research univer-
sity in uganda. It is one of the oldest and largest universities in the country. It was founded in 1922 by king george v of the kingdom of kenya as a technical school for training teachers. The university is located in the city of kampala, the nation's 's capital and largest city. It is accredited by the commission on higher education, the ugandan ministry of education, and the international association of universities. [2] The university's 's alumni include heads of state and government in africa, writers ngugi wa thiong' 'o and john sentamu, [3] former an- glican archbishop of nairobi, kenneth nyerere, former president of tanzania juvénal habyarimana and many others. In november 2016, the university was closed indefinitely following strike by lecturers over unpaid incentive arrears. The university students subsequently joined the strike, demanding be taught. [4] The visi- tation committee appointed by president yoweri museveni to look into the affairs of the university has listed its responsibilities, stating that work will be done in a timely and comprehensive manner. [4] Makerere's 's refugee law project, based at the university, works to ensure all people enjoy the human rights irrespective of their legal status. it seeks to empower asylum seekers, refugees, deportees, idps and host communities in Uganda to enjoy their human rights and lead dignified lives. [5] An important emphasis of such engagement has been to highlight the invisibility of men's 's and boys' experiences of sexual violence, and to raise global awareness of the close relationship between violations of sexuality and gender, and patterns of forced migration. Students at the school of law pay an extra 120,000 shillings per semester. [1]
27


Reviews:
[{"review_id": "wnqUuGBqjhQ", "reviewer": "Reviewer_Ej3Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new benchmark for generating the first paragraph of wikipedia articles (with references) from supporting paragraphs. The authors compare a retrieve-and-generate baseline against several non-retrieval baselines (e.g., BART, GPT-2, and FiD + BART). None of these methods (including GPT-3) perform well on the task, suggesting room for future improvement.", "summary_of_the_review": "While I like where this paper is going, I feel like the practical artifact and problem tackled here are a bit too small-scale and narrow. We're at a point in NLP where things actually kind of (?) begin to work, and there have been a variety of impressive results on generating and understanding long documents. I think this work would really shine in pushing this direction, but instead, it compromises itself to fit the limitations of existing models, limiting its usefulness for future work. Thus, I'm not sure that I can recommend acceptance.", "strength_and_weaknesses": "Strengths:\n- This is an interesting and important task, and I think it's one that could be a fruitful combination of lot of existing work on both generation and understanding.\n\nWeaknesses:\n- Despite the grandiose motivations, the authors consider an extremely narrow instantiation of this task:\n  - Only the first paragraph is used as the generation target, rather than entire articles. When I read the abstract and introduction I was excited to see how the authors would handle generation / understanding of long sequences, since this is an area where I think this benchmark would be particularly useful, but was disappointed to see that they largely punted on this problem.\n  - Furthermore, the authors don't use the entirety of the reference articles, instead taking only a single paragraph per article that is specifically chosen to maximize informativeness, which I think gives a bit of an unfair advantage to the retrieval-based systems (since they're much more likely to retrieve something useful).\n\n- In general, I feel like the paragraph \"Reference Passage Selection \" in 3.2 completely misses the mark. In particular:\n\n> The original Wiki articles and reference articles tend to be very long. To adapt the capacity of most pre-trained language models (e.g., BERT has a 512 token limit), we use the first section of Wiki articles as the generation target, and we select the passage from the reference article with the highest PST value as the supporting passage.\n\nI'm not sure that I feel very good about our benchmarks having to be \"adapted\" to be consumable for existing models while trading off the essence of the task (synthesizing information from multiple long documents to generate a long document). We should set the benchmarks as goals, see how current models and techniques perform on them, and allow them to be targets for future work.\n\n- I feel like the baselines are pretty weak, and the ReGen model is a pretty standard retrieve-and-generate baseline. I don't think there's too much modeling novelty here.\n\n- Furthermore, the query distribution seems pretty unnatural. For example, in the appendix, the query used is \"how google translate work\". This is a query with a pretty clear information-seeking / question intent that can be answered by a paragraph (or paragraphs). However, the wikipedia page for \"Google Translate\" answers far more than just this question, it contains a variety of general information about the query \"google translate\" (which itself could have many different question intents). So, I feel like the source of supervision provides a different sort of information than what a query like this would actually want (let alone that in practice, only the first passage is used, which contains even less information because of the way that lead sections in wikipedia are structured to contain general high-level information: https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Lead_section).\n  - The original motivation mentioned \"unseen factual queries\", but it's not really clear that generating an article is what you want for this. If it's factual, you can probably answer it with a paragraph or two. I think this work would benefit from a clearer exposition of what these queries would actually be, and what the right source of supervision / data for effectively addressing these queries would look like.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cMVGNXf-tNq", "reviewer": "Reviewer_dCpa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new task called “WebBrain”. The objective of the task is to learn to generate a fluent, informative and factually correct short article from a query given some search results. The task is essentially a combination of 2 components:\n  1. retrieval of evidence passages given a wikipedia page title,\n  2. multi document summarization of the retrieved evidence with the first paragraph of the wikipedia page as target. \n\nThe authors contribute a dataset \"WebBrain-Raw\", comprised of English wikipedia plus all crawlable references. The dataset is cleaned with an interesting set of heuristics.\n\nThe authors additionally propose a baseline system called \"ReGen\". ReGen combines several modeling choices:\n  * training a SPLADE retriever for this task with hard negatives are mined,\n  * tuning sparsity in document representations for retrieval,\n  * filtering retrieved documents trading off consistency and diversity,\n  * fusion-in-decoder for generation,\n  * making generation more grounded in the references, by pre-training the generation component to generate individual sentences from a reference passage at time.\n\nThe authors finally present analysis showing the usefulness of these modeling choices, compared to more vanilla choices.", "summary_of_the_review": "This looks like a useful dataset+baseline contribution for the important task of multi-document summarization. The techniques used by the authors seem solid.", "strength_and_weaknesses": "Strengths:\n* The task examined in this paper (retrieval + multi-document summarization) is important.\n* The dataset released by this paper is much larger than comparable datasets and could be useful for furthering work on this topic.\n* The authors describe in detail many technical details of ReGen. These details can be useful to practitioners for reproducing ReGen’s results and in their own work.\n\nWeaknesses:\n* Limiting the generation to only the first passage of wikipedia pages is a pretty strong limitation, also making this work very close to existing multi-document summarization works. The authors do acknowledge this similarity though.\n* In “Reference Passage Selection” and “Dataset Generation”, the authors select input passages for training with simple word overlap or BM25. It seems like this could be easily improved by using an entailment model, a dense retriever or even SPLADE.\n* The proposed \"WebBrain-Raw\" dataset would be more interesting if it was multilingual instead of English-only.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5z-M9diMDnw", "reviewer": "Reviewer_TtTt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new dataset and associated task that involves generating the first section of Wikipedia pages from a set of retrieved references. The dataset is similar in form to previous work like WikiSumm, but it is significantly larger and the authors promise to release the references, which have been downloaded from non-Wikipedia sites.\n\nThe dataset generation process involves a number of filtering steps, based on term overlap, that narrow down the reference data to passages that are likely to be related to the target Wikipedia article and associate these passages with sentences in the target. The goal, at test time, is to retrieve reference passages that may be related to a topic, and then\ngenerate the target section of the Wikipedia article.\n\nAlong with the dataset, this paper presents a new model, ReGen. ReGen is composed of a retriever (based on SPLADE) and a reference encoder and target decoder (based on FiD). When trained on the WebBrain data, ReGen outperforms both large language model baselines and also a retrieval augmented model with a dense retriever (FiD + BART) according to a range of automatic metrics, and also annotator judgements of a small set of predictions. I have some questions about some of these comparisions below, but it is clear that ReGen is capable of generating coherent and informative text. It is also capable of providing references for this text, which is a significant advantage over methods that don't include retrieval.\n\nThere are a number of nice ablations that show how the different systems' performances change with different numbers of retrievals and gold references. These comparisons are useful in understanding the tradeoffs between retrieving a lot of supporting evidence, with lower precision, vs fewer high quality references. ", "summary_of_the_review": "This paper presents a significant new dataset and task that could be useful to the community, going forward, in benchmarking methods of retrieval augmented generation. The new model is also quite different from previous work, which has generally relied on dense DPR-style retrievers.\n\nOverall, I think this paper is a nice contribution and I am currently leaning toward acceptance. However, I also have some serious questions below about the data release strategy, and how the authors propose to handle concerns about releasing multiple terrabytes of data which may include copyrighted works or personal information. \n", "strength_and_weaknesses": "#### Strengths\n- This paper promises a massive new dataset for retrieval augmented text generation, that could be very useful to the NLP community. Having said that, I do have questions about the release with respect to copyright and privacy concerns (detailed below in the ethics section).\n- The end system can retrieve references, and link them to generated sentences, which is a very nice end product which could have real utility beyond unconstrained text generation, which cannot provide any form of attribution for its predictions.\n- There are a number of interesting modeling and training choices made, which lead to significant improvements over a very large language model (GPT3) and also similar retrieval agumented approaches with different choices of retriever and training procedure.\n\n#### Weaknesses\n- There are a lot of separate contributions here that are not independently evaluated (data filtering / retrieval filtering / warmup strategy). It would be nice to see evaluations that validate these choices.\n- The reference passage filtering process selects the references included in the retrieval corpus references according to term overlap with the target Wikipedia article. If I'm understanding this process correctly, this means that the references stored in the corpus have been selected according to observations of the test time targets, so there is some leakage of information from the test targets into the model input. The paper should discuss this.\n- The prompt given to GPT3 \"Introduce [Page Title]\" does not mention that the target is Wikipedia-style text, and the example in Figure 1. does not look much like the start of a Wikipedia article. Meanwhile, GPT3 is definitely able to generate Wikipedia style text if prompted to do so. The comparision to GPT3 would be stronger if the prompt was better tied to the actual task.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5--_YvGLGN", "reviewer": "Reviewer_892z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new task called Web-Brain which aims to generate short factual articles for queries by mining supporting evidence from Web. The paper also proposes a new large scale dataset with English Wikipedia. The paper also provides a new framework called ReGen based on SPLADE and FiD. The model is evaluated with both n-gram overlapping metrics and factual correctness metrics. The paper analyze the impact of retrieval, number of references in a quantitative way. The paper also did both human and automatic evaluation. ", "summary_of_the_review": "Overall, the paper proposes a new interesting task with a corresponding large-scale Wikipedia-based dataset. The experiment part is quite comprehensive.", "strength_and_weaknesses": "Strength \n1. The paper introduces WebBrain, which lets the model retrieve supporting evidence and generate factual articles given a factual query. The proposed dataset is somewhat similar to the Wizard of Wikipedia (Dinan et al., 2018). The newly proposed dataset is interesting and large-scale. The authors crawled and cleaned Wikipedia. The proposed task and corresponding dataset are very interesting and worthy of future research. \n\n2. The paper proposes a new retrieval-augmented generation framework based on SPLADE and FiD. The proposed methods achieve the best results over automatic and human evaluation. The experiment section is very comprehensive. The authors conduct an ablation study with different retrieval models and show the impact of the different numbers of retrieved references. The paper also checks the impact of a number of references. Those results are clearly represented in tables or charts with detailed explanations. The paper shows the case study, human evaluation, and reference mark correction strategy in the appendix.\n\nWeaknesses\n1. The paper uses n-gram overlapping metrics for automatic evaluation. The paper needs to include some newer metrics such as BERTscore (Zhang et al., 2019), and BARTScore (Yuan et al., 2021) which can check semantic similarity. \n\n2. Most of the experiment analyses are in quantitative way. I would like to see more qualitative analysis. \n\n\n\nZhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.\n\nYuan, W., Neubig, G., & Liu, P. (2021). Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34, 27263-27277.\n\n\n\nDinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., & Weston, J. (2018). Wizard of Wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject