Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes an approach to generating the first section of Wikipedia articles (and potentially entire articles). 
First relavant paragraphs are extracted from reference documents and documents retrieved through search engine queries through a TD-IDF-based ranking. Then abstractive summarization is performed using a modification of Transformer networks (Vasvani et al 2017). A mixture of experts layer further improves performance. 
The proposed transformer decoder defines a distribution over both the input and output sequences using the same self-attention-based network. On its own this modification improves perplexity (on longer sequences) but not the Rouge score; however the architecture enables memory-compressed attention which is more scalable to long input sequences. It is claimed that the transformer decoder makes optimization easier but no complete explanation or justification of this is given. Computing self-attention and softmaxes over entire input sequences will significantly increase the computational cost of training.

In the task setup the information retrieval-based extractive stage is crucial to performance, but this contribution might be less important to the ICLR community. It willl also be hard to reproduce without significant computational resources, even if the URLs of the dataset are made available. The training data is significantly larger than the CNN/DailyMail single-document summarization dataset.

The paper presents strong quantitative results and qualitative examples. Unfortunately it is hard to judge the effectiveness of the abstractive model due to the scale of the experiments, especially with regards to the quality of the generated output in comparison to the output of the extractive stage.
In some of the examples the system output seems to be significantly shorter than the reference, so it would be helpful to quantify this, as well how much the quality degrades when the model is forced to generate outputs of a given minimum length. While the proposed approach is more scalable, it is hard to judge the extend of this.

So while the performance of the overall system is impressive, it is hard to judge the significance of the technical contribution made by the paper.

---
The additional experiments and clarifications in the updated version give substantially more evidence in support of the claims made by the paper, and I would like to see the paper accepted. 

review_title: Impressive application, but hard to judge technical contribution

final_score: rating: 7: Good paper, accept
review_content: This paper considers the task of generating Wikipedia articles as a combination of extractive and abstractive multi-document summarization task where input is the content of reference articles listed in a Wikipedia page along with the content collected from Web search and output is the generated content for a target Wikipedia page. The authors at first reduce the input size by using various extractive strategies and then use the selected content as input to the abstractive stage where they leverage the Transformer architecture with interesting modifications like dropping the encoder and proposing alternate self-attention mechanisms like local and memory compressed attention.   

In general, the paper is well-written and the main ideas are clear. However, my main concern is the evaluation. It would have been nice to see how the proposed methods perform with respect to the existing neural abstractive summarization approaches. Although authors argue in Section 2.1 that existing neural approaches are applied to other kinds of datasets where the input/output size ratios are smaller,  experiments could have been performed to show their impact. Furthermore, I really expected to see a comparison with Sauper & Barzilay (2009)'s non-neural extractive approach of Wikipedia article generation, which could certainly strengthen the technical merit of the paper.

More importantly, it was not clear from the paper if there was a constraint on the output length when each model generated the Wikipedia content. For example, Figure 5-7 show variable sizes of the generated outputs. With a fixed reference/target Wikipedia article, if different models generate variable sizes of output, ROUGE evaluation could easily pose a bias on a longer output as it essentially counts overlaps between the system output and the reference. 

It would have been nice to know if the proposed attention mechanisms account for significantly better results than the T-ED and T-D architectures. Did you run any statistical significance test on the evaluation results? 

Authors claim that the proposed model can generate "fluent, coherent" output, however, no evaluation has been conducted to justify this claim. The human evaluation only compares two alternative models for preference, which is not enough to support this claim. I would suggest to carry out a DUC-style user evaluation (http://www-nlpir.nist.gov/projects/duc/duc2007/quality-questions.txt) methodology to really show that the proposed method works well for abstractive summarization.

Does Figure 8 show an example input after the extractive stage or before? Please clarify.

---------------
I have updated my scores as authors clarified most of my concerns.
review_title: Interesting, but evaluation could have been stronger

final_score: rating: 7: Good paper, accept
review_content: The main significance of this paper is to propose the task of generating the lead section of Wikipedia articles by viewing it as a multi-document summarization problem. Linked articles as well as the results of an external web search query are used as input documents, from which the Wikipedia lead section must be generated. Further preprocessing of the input articles is required, using simple heuristics to extract the most relevant sections to feed to a neural abstractive summarizer. A number of variants of attention mechanisms are compared, including the transofer-decoder, and a variant with memory-compressed attention in order to handle longer sequences. The outputs are evaluated by ROUGE-L and test perplexity. There is also a A-B testing setup by human evaluators to show that ROUGE-L rankings correspond to human preferences of systems, at least for large ROUGE differences.

This paper is quite original and clearly written. The main strength is in the task setup with the dataset and the proposed input sources for generating Wikipedia articles. The main weakness is that I would have liked to see more analysis and comparisons in the evaluation.

Evaluation:
Currently, only neural abstractive methods are compared. I would have liked to see the ROUGE performance of some current unsupervised multi-document extractive summarization methods, as well as some simple multi-document selection algorithms such as SumBasic. Do redundancy cues which work for multi-document news summarization still work for this task?

Extractiveness analysis:
I would also have liked to see more analysis of how extractive the Wikipedia articles actually are, as well as how extractive the system outputs are. Does higher extractiveness correspond to higher or lower system ROUGE scores? This would help us understand the difficulty of the problem, and how much abstractive methods could be expected to help. 

A further analysis which would be nice to do (though I have less clear ideas how to do it), would be to have some way to figure out which article types or which section types are amenable to this setup, and which are not. 

I have some concern that extraction could do very well if you happen to find a related article in another website which contains encyclopedia-like or definition-like entries (e.g., Baidu, Wiktionary) which is not caught by clone detection. In this case, the problem could become less interesting, as no real analysis is required to do well here.

Overall, I quite like this line of work, but I think the paper would be a lot stronger and more convincing with some additional work.

----
After reading the authors' response and the updated submission, I am satisfied that my concerns above have been adequately addressed in the new version of the paper. This is a very nice contribution.

review_title: Interesting task. Would appreciate more analysis.

Meta-review: This paper presents a new multi-document summarization task of trying to write a wikipedia article based on its sources. Reviewers found the paper and the task clear to understand and well-explained. The modeling aspects are clear as well, although lacking justification. Reviewers are split on the originality of the task, saying that it is certainly big, but wondering if that makes it difficult to compare with. The main split was the feeling that "the paper presents strong quantitative results and qualitative examples. " versus a frustration that the experimental results did not take into account extractive baselines or analysis. However the authors provide a significantly updated version of the work targeting many of these concerns, which does alleviate some of the main issues. For these reasons, despite one low review, my recommendation is that this work be accepted as a very interesting contribution.


Decision: Accept (Poster)