Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new dataset and associated task that involves generating the first section of Wikipedia pages from a set of retrieved references. The dataset is similar in form to previous work like WikiSumm, but it is significantly larger and the authors promise to release the references, which have been downloaded from non-Wikipedia sites.

The dataset generation process involves a number of filtering steps, based on term overlap, that narrow down the reference data to passages that are likely to be related to the target Wikipedia article and associate these passages with sentences in the target. The goal, at test time, is to retrieve reference passages that may be related to a topic, and then
generate the target section of the Wikipedia article.

Along with the dataset, this paper presents a new model, ReGen. ReGen is composed of a retriever (based on SPLADE) and a reference encoder and target decoder (based on FiD). When trained on the WebBrain data, ReGen outperforms both large language model baselines and also a retrieval augmented model with a dense retriever (FiD + BART) according to a range of automatic metrics, and also annotator judgements of a small set of predictions. I have some questions about some of these comparisions below, but it is clear that ReGen is capable of generating coherent and informative text. It is also capable of providing references for this text, which is a significant advantage over methods that don't include retrieval.

There are a number of nice ablations that show how the different systems' performances change with different numbers of retrievals and gold references. These comparisons are useful in understanding the tradeoffs between retrieving a lot of supporting evidence, with lower precision, vs fewer high quality references. 
strength_and_weaknesses: #### Strengths
- This paper promises a massive new dataset for retrieval augmented text generation, that could be very useful to the NLP community. Having said that, I do have questions about the release with respect to copyright and privacy concerns (detailed below in the ethics section).
- The end system can retrieve references, and link them to generated sentences, which is a very nice end product which could have real utility beyond unconstrained text generation, which cannot provide any form of attribution for its predictions.
- There are a number of interesting modeling and training choices made, which lead to significant improvements over a very large language model (GPT3) and also similar retrieval agumented approaches with different choices of retriever and training procedure.

#### Weaknesses
- There are a lot of separate contributions here that are not independently evaluated (data filtering / retrieval filtering / warmup strategy). It would be nice to see evaluations that validate these choices.
- The reference passage filtering process selects the references included in the retrieval corpus references according to term overlap with the target Wikipedia article. If I'm understanding this process correctly, this means that the references stored in the corpus have been selected according to observations of the test time targets, so there is some leakage of information from the test targets into the model input. The paper should discuss this.
- The prompt given to GPT3 "Introduce [Page Title]" does not mention that the target is Wikipedia-style text, and the example in Figure 1. does not look much like the start of a Wikipedia article. Meanwhile, GPT3 is definitely able to generate Wikipedia style text if prompted to do so. The comparision to GPT3 would be stronger if the prompt was better tied to the actual task.

clarity,_quality,_novelty_and_reproducibility: #### Clarity & Reproducability
The paper is clearly written and all parts are pretty well described. The authors commit to releasing the data, which will allow easy replication.

#### Novelty
The dataset is similar in form to previous work, but extends that previous work to the scenario where references must be retrieved from a very large corpus. Similarly, the model is a small modification of existing approaches but it appears to work better than well chosen baselines for this task.

#### Quality
There are a lot of details in the filtering procedures used to select the contents of the dataset, and assign references to sentences. These are all justified int the text, but are not supported by any sort of analysis. If this is going to become a benchmark dataset, going forward, it would be good to see a discussion of how these choices affect the task. In particular, see my question about test target leakage into the retrieval corpus above (under Weaknesses).

summary_of_the_review: This paper presents a significant new dataset and task that could be useful to the community, going forward, in benchmarking methods of retrieval augmented generation. The new model is also quite different from previous work, which has generally relied on dense DPR-style retrievers.

Overall, I think this paper is a nice contribution and I am currently leaning toward acceptance. However, I also have some serious questions below about the data release strategy, and how the authors propose to handle concerns about releasing multiple terrabytes of data which may include copyrighted works or personal information.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new task called “WebBrain”. The objective of the task is to learn to generate a fluent, informative and factually correct short article from a query given some search results. The task is essentially a combination of 2 components:
  1. retrieval of evidence passages given a wikipedia page title,
  2. multi document summarization of the retrieved evidence with the first paragraph of the wikipedia page as target. 

The authors contribute a dataset "WebBrain-Raw", comprised of English wikipedia plus all crawlable references. The dataset is cleaned with an interesting set of heuristics.

The authors additionally propose a baseline system called "ReGen". ReGen combines several modeling choices:
  * training a SPLADE retriever for this task with hard negatives are mined,
  * tuning sparsity in document representations for retrieval,
  * filtering retrieved documents trading off consistency and diversity,
  * fusion-in-decoder for generation,
  * making generation more grounded in the references, by pre-training the generation component to generate individual sentences from a reference passage at time.

The authors finally present analysis showing the usefulness of these modeling choices, compared to more vanilla choices.
strength_and_weaknesses: Strengths:
* The task examined in this paper (retrieval + multi-document summarization) is important.
* The dataset released by this paper is much larger than comparable datasets and could be useful for furthering work on this topic.
* The authors describe in detail many technical details of ReGen. These details can be useful to practitioners for reproducing ReGen’s results and in their own work.

Weaknesses:
* Limiting the generation to only the first passage of wikipedia pages is a pretty strong limitation, also making this work very close to existing multi-document summarization works. The authors do acknowledge this similarity though.
* In “Reference Passage Selection” and “Dataset Generation”, the authors select input passages for training with simple word overlap or BM25. It seems like this could be easily improved by using an entailment model, a dense retriever or even SPLADE.
* The proposed "WebBrain-Raw" dataset would be more interesting if it was multilingual instead of English-only.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and good quality. The system is described in detail and should be reproducible.

The novelty of the approach is mainly in the size of the released corpus and in including retrieval as part of the task, together with multi-document summarization.

The techniques proposed by the authors are very interesting, but they are only applied to the dataset they created, so it's not totally clear how their modeling choices would stack up against other methods on a more competitive benchmark.
summary_of_the_review: This looks like a useful dataset+baseline contribution for the important task of multi-document summarization. The techniques used by the authors seem solid.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new task called Web-Brain which aims to generate short factual articles for queries by mining supporting evidence from Web. The paper also proposes a new large scale dataset with English Wikipedia. The paper also provides a new framework called ReGen based on SPLADE and FiD. The model is evaluated with both n-gram overlapping metrics and factual correctness metrics. The paper analyze the impact of retrieval, number of references in a quantitative way. The paper also did both human and automatic evaluation. 
strength_and_weaknesses: Strength 
1. The paper introduces WebBrain, which lets the model retrieve supporting evidence and generate factual articles given a factual query. The proposed dataset is somewhat similar to the Wizard of Wikipedia (Dinan et al., 2018). The newly proposed dataset is interesting and large-scale. The authors crawled and cleaned Wikipedia. The proposed task and corresponding dataset are very interesting and worthy of future research. 

2. The paper proposes a new retrieval-augmented generation framework based on SPLADE and FiD. The proposed methods achieve the best results over automatic and human evaluation. The experiment section is very comprehensive. The authors conduct an ablation study with different retrieval models and show the impact of the different numbers of retrieved references. The paper also checks the impact of a number of references. Those results are clearly represented in tables or charts with detailed explanations. The paper shows the case study, human evaluation, and reference mark correction strategy in the appendix.

Weaknesses
1. The paper uses n-gram overlapping metrics for automatic evaluation. The paper needs to include some newer metrics such as BERTscore (Zhang et al., 2019), and BARTScore (Yuan et al., 2021) which can check semantic similarity. 

2. Most of the experiment analyses are in quantitative way. I would like to see more qualitative analysis. 



Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Yuan, W., Neubig, G., & Liu, P. (2021). Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34, 27263-27277.



Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., & Weston, J. (2018). Wizard of Wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241.
clarity,_quality,_novelty_and_reproducibility: 1. Some parts of the paper are not very clear. The steps to create WebBrain-R and WebBrain-G is unclear. 

2. The paper attached the implementation details in the appendix. It also provides examples from the dataset for readers to check. However, it does not provide any code for reproduction. It shows the limitation and system demonstration in the Appendix. 
summary_of_the_review: Overall, the paper proposes a new interesting task with a corresponding large-scale Wikipedia-based dataset. The experiment part is quite comprehensive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new benchmark for generating the first paragraph of wikipedia articles (with references) from supporting paragraphs. The authors compare a retrieve-and-generate baseline against several non-retrieval baselines (e.g., BART, GPT-2, and FiD + BART). None of these methods (including GPT-3) perform well on the task, suggesting room for future improvement.
strength_and_weaknesses: Strengths:
- This is an interesting and important task, and I think it's one that could be a fruitful combination of lot of existing work on both generation and understanding.

Weaknesses:
- Despite the grandiose motivations, the authors consider an extremely narrow instantiation of this task:
  - Only the first paragraph is used as the generation target, rather than entire articles. When I read the abstract and introduction I was excited to see how the authors would handle generation / understanding of long sequences, since this is an area where I think this benchmark would be particularly useful, but was disappointed to see that they largely punted on this problem.
  - Furthermore, the authors don't use the entirety of the reference articles, instead taking only a single paragraph per article that is specifically chosen to maximize informativeness, which I think gives a bit of an unfair advantage to the retrieval-based systems (since they're much more likely to retrieve something useful).

- In general, I feel like the paragraph "Reference Passage Selection " in 3.2 completely misses the mark. In particular:

> The original Wiki articles and reference articles tend to be very long. To adapt the capacity of most pre-trained language models (e.g., BERT has a 512 token limit), we use the first section of Wiki articles as the generation target, and we select the passage from the reference article with the highest PST value as the supporting passage.

I'm not sure that I feel very good about our benchmarks having to be "adapted" to be consumable for existing models while trading off the essence of the task (synthesizing information from multiple long documents to generate a long document). We should set the benchmarks as goals, see how current models and techniques perform on them, and allow them to be targets for future work.

- I feel like the baselines are pretty weak, and the ReGen model is a pretty standard retrieve-and-generate baseline. I don't think there's too much modeling novelty here.

- Furthermore, the query distribution seems pretty unnatural. For example, in the appendix, the query used is "how google translate work". This is a query with a pretty clear information-seeking / question intent that can be answered by a paragraph (or paragraphs). However, the wikipedia page for "Google Translate" answers far more than just this question, it contains a variety of general information about the query "google translate" (which itself could have many different question intents). So, I feel like the source of supervision provides a different sort of information than what a query like this would actually want (let alone that in practice, only the first passage is used, which contains even less information because of the way that lead sections in wikipedia are structured to contain general high-level information: https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Lead_section).
  - The original motivation mentioned "unseen factual queries", but it's not really clear that generating an article is what you want for this. If it's factual, you can probably answer it with a paragraph or two. I think this work would benefit from a clearer exposition of what these queries would actually be, and what the right source of supervision / data for effectively addressing these queries would look like.
clarity,_quality,_novelty_and_reproducibility: This paper was reasonably clear, and I thought that the setup was pretty solid. It's an entry in a long line of work on generating Wikipedia paragraphs from references or other external knowledge.
summary_of_the_review: While I like where this paper is going, I feel like the practical artifact and problem tackled here are a bit too small-scale and narrow. We're at a point in NLP where things actually kind of (?) begin to work, and there have been a variety of impressive results on generating and understanding long documents. I think this work would really shine in pushing this direction, but instead, it compromises itself to fit the limitations of existing models, limiting its usefulness for future work. Thus, I'm not sure that I can recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a large scale dataset of Wikipedia articles and the references contained therein. The task is to generate the first section/paragraph of the article given the query (the title of the Wikipedia page). The authors claim that their work generates factually correct information. The method contains two steps: (1) First retrieve a set of documents that are relevant for the given query, (2) use information in these documents to generate factually correct passages.

Strengths
* well-written paper
* release of dataset
* experiments with a bunch of different eval metrics

Weaknesses
* no human evaluation of factuality
* no modeling innovation to enforce factuality
* no interesting processing of data apart from simple crawling and cleaning
* unclear what are the main differences from retrieval augmented approaches
* generating only the first section is pretty limiting as an application
* unclear why certain decisions are made throughout the paper without further details on these (heuristics for reference selection, data cleaning etc.)
* a lot of queries are simply noun compounds that don't display any special linguistic behavior
* why is BLEU-4 score missing from certain tables?
justification_for_why_not_higher_score: (see weaknesses above)
justification_for_why_not_lower_score: n/a
Decision: Reject