Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a two-stage summarization system where a document is provided along with (optionally) keywords or a prompt. This supplemental information helps to guide the summarization and possibly make it more user-specific. The keywords and prompt can also be guessed automatically by a BERT-base model, which seems to improve automatic metrics on CNN/daily mail.

Strengths:
* Moving beyond the conventional 'document/summary' framework of existing summarization approaches is a strength of this paper. This paper studies a few different ways that the summaries can be controlled: through prompts, entities, or one-sentence summaries of summaries (contribution and purpose summarization). These seem novel at least to this reviewer and could be helpful for future work.
* When using oracle guidance, performance increases on several different datasets for slightly different forms of summarization (CNN/DM, arxiv, bigpatent).
* The idea of using a two-stage approach (with a BERT-Base extractor to guess keywords to guide the summary) seems novel to this reviewer, and it seems to enable this approach to perform well even in an unconditional setting.

Weaknesses:
* The main weakness to this reviewer is that the evaluation might not be sufficiently convincing to test the key hypothesis: that these keywords/prompts can enable users to get summaries that are closer to their intent (like Figure 1). To this reviewer, this necessitates a human evaluation. Though testing factual correctness in Table 3 seems like a good start to this reviewer, measuring overall summarization quality (both conditional and unconditional on user intent) through a human evaluation seems necessary.
* (minor) one possible reason why the two-stage approach might perform better on unconditional summarization is because there are more parameters when ensembling BERT-Base and BART. Possibly doing something multitask within a single BART model might be cleaner and could clearly test whether the gains come from more parameters/computation, or the keyword approach.


Overall, to this reviewer, this paper seems like it would be strong if it had human evaluations of summarization quality. I would be willing to raise my score if those were provided.

----

Update: thanks for the additional human evaluation results! These help and the results on excluding unimportant entities seem strong to this reviewer. Perhaps it might be more helpful for the annotators themselves to try to interact with the summarizer in some way, but that's a more minor point.

Anyways, I bumped up my score from 5->7.
review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary:

Builds/extends on Controllable Abstractive Summarization (Fan et al) using keywords and other prompts. There’s two phases, and both phases are independent:

1. Extract keywords, z,  using a BERT classifier/sequence-tagger trained to predict keywords
2. Fine-tune BART (Lewis et al) to learn p(summary | document, z).

One can use automatic keywords using (1) and get uncontrolled generation, for which they present SOTA results on some summarization tasks. Two datasets are collected: (a) intro->contributions from arxiv papers; (b) patent->one-sentence summary, which are used to measure performance of prompts specific to those tasks.

# Pros:
1. Improves state-of-the-art results on some summarization benchmarks.
2. Provides BERTScore results in addition to ROUGE.
3. Results provided across multiple summarization datasets.
4. Interesting new datasets for measuring document+prompt->summary performance
5. Interesting zero-shot/transfer results from summarization to Question-answering.

# Cons: 
1. Contribution in methods over Fan et al + BART (Lewis et al) is minimal. Results in Fan et al were weak because the underlying model was much weaker than BART, so it is unclear how much results here improved from simply using BART and adding control tokens from Fan et al, which would be a useful baseline to have that is omitted.
2. Since the focus of the paper is on controlling generation, more results on this would be informative. It is unclear how well control works when not using oracle words or automatically extracted keywords, i.e. user-controlled. An MTurk experiment evaluating how well control works would be useful in assessing this.
3. Comparing BART/PEGASUS to BART+BERT-based model is a little unfair since BERT is another large model in the system, i.e. the total amount of compute and number of parameters is much greater. A more fair comparison would be to compare using a smaller BART or PEGASUS model such that the model sizes are comparable. It is unclear whether the proposed system would do better than BART/PEGASUS scaled to the same amount of total parameters.
4. For the prompt tests, how well does BART/PEGASUS do if the decoder is prompted? This baseline would be useful to have. That is, it’s unclear how the p(y | x, z) improves over using p(y | x) by simply prompting the decoder.

# Clarifications/questions:
1. For "CONTRIBUTION AND PURPOSE SUMMARIZATION", what is the fine-tuning process? Are the models fine-tuned on (paper/patent, keywords)->abstract task before testing on intro->contribution generation? 
2. How are entities randomly selected in the example decodes in the Appendix?
3. Are any of the prompts used in training or is it zero-shot?
4. What is zero-shot state-of-the-art on the QA tasks? Please add to Table 5. GPT-3 zero-shot results would also be informative.
review_title: Interesting techniques for some summarization tasks, but unclear contribution over Fan et al.

final_score: rating: 7: Good paper, accept
review_content: The authors propose an abstractive document summarization model that can
generate summaries that target a specific set of keywords or prompts. This is
in contrast to generic summarization models that learn to summarize a document
but are difficult to control or direct. The authors propose a straightforward
way of obtaining keywords from an article similar in spirit to Gerhmann et al.
2018. Alternatively, "ground truth" keywords can be found using a reference
summary. In either case, the keywords are prepended to the input document and
a BART model is fine-tuned to generate summaries using both the document and
keyword content.

The authors go on to show how a model trained in such a way, which they refer
to as CTRLsum, can generate entity-focused summaries, by using an entity name
as the keyword prefix. Additionally, providing differing numbers of keywords
can be used to control the length of the generated summary.  The authors also
show that CTRLsum can respond sensibly to prompts, i.e. instead of providing
keywords, a question or initial phrase is provided.  Useful summarization
behavior can be achieved including zero shot question answering, or
enumeration of a research paper's contributions or the purpose of an
invention.

While the paper feels largely like an extension of Keskar et al. 2019, the
evaluation of the proposed methods is very thorough on a variety of settings
and domains. I especially enjoyed the break out of entity targeted summaries
based on whether the entity occurred in the lead and/or reference summary. The
use of prompts to obtain question answering and more focused
contributions/purpose summarization was also very interesting.

I would be happy to see this paper accepted to ICLR. This paper offers a
simple method of obtaining a variety of focused or targeted summarization
behaviors from a BART summarization model. In general, I would like to see
more work like this exploring methods of controlling pretrained language
models.  The evaluation of the correctness of the generated utterances
suggests that this method provides fairly reliable control.

There several areas where the paper could improve. The explanation of how
length control is achieved was not very clear. It would help to have examples
like those shown in the appendix present in the section introducing length
control. 

Comparisons are to the standard BART model or to Fan et al. (2018) which
similarly prepend important control information to the input. It would be
interesting to see an evaluation that compared CTRLsum to BART with a
constrained decoding method, such as dynamic beam allocation [1].

Additionally, the authors should say more about the differences in entity
control of their method and Fan et al. 2018, which seem on their face to be
similar.

Did the authors experiment with pairs of entities as entity controls? It would
be especially interesting to see whether the model preserves the correct
relationship between entities, especially for entities that didn't occur in
the same sentences in the original document, e.g. one important entity and one
unimportant entity.

[1] Matt Post and David Vilar. Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation. ACL. 2018.

review_title: A simple but effective method of focusing abstractive summarization models.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper Summary:
* This paper proposes a framework for controllable summarization, CTRLsum.  It is different from standard summarization models that CTRLsum uses a set of keywords extracted from the source text automatically or descriptive prompts to control the summary.  Experiments with three domains of summarization datasets and five control aspects.

Strengthes:
* The authors investigated the effectiveness of the proposed model through extensive experiments.

Weaknesses:
* The proposed method that uses keywords as an additional input text is almost the same as CIT (Saito et al., 2020), and the scores of CTRLsum on the CNNDM dataset reported in Table 7 does not outperformed those of CIT.  Also, it is not novel to use descriptive prompts to control natural language generation.
   * Saito et al.: Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models. CoRR abs/2003.13028 (2020)
* I think that the author's claim, "keywords and prompts are complementary", is not evaluated fully.

Questions:

* With respect to contribution summarization, did you evaluate CTRLsum(keyword without prompt) and CTRLsum(prompt without keywords)?  The control tokens "the main contributions of this paper are : ( 1 )" is far from the keywords used during training, and so I think that the keywords are not effective for contribution summarization.  In fact, BART that uses prompt worked well for contribution summarization.

* Did you evaluate the ablation tests with respect to the special token "|" and keyword dropout?

* Can CTRLsum control the generation with multiple aspects (length and entity control, length and QA control, etc.) simultaneously?  The length of  summaries generated by CTRLsum is strongly dependent the number of keywords, and so I think it is difficult to simultaneously control multiple aspects including length control.

Update:
Thank you for the answers to my questions and additional experiments. 
review_title: Official Blind Review #1

Meta-review:  The paper attempts at controllable summarization in two dimensions: Length, and content. Authors try to achieve this through training data generation approach, where they provide a standard BART model with additional keywords (extracted using a BERT model) in training.

The paper's main motivation on controllable summarization is important and interesting, and despite simplicity, the results are generally positive on multiple datasets.
However, despite positive results, reviewers raised several critical concerns, some of which remained unresolved after reviewer/author discussion period. Examples include concerns regarding lack of methodological novelty over prior work (R1, R2, R4), unfair/incomplete comparisons with prior work (R2, R4, R5), and not evaluating on a real user controlled setting instead of automatic keywords (R1, R4). Although the authors tried addressing human evaluation in their revision, some reviewers remained unconvinced. 
Some quotes from reviewer discussions:

> I'm not convinced the human eval was done properly.

> My concerns are not completely addressed and the score remains unchanged. For human evaluation, I agreed with Reviewer X.

Decision: Reject