Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper is well written and investigates dialogue summarization that has not received much attention. It proposes a new model called CORDIAL which can generate a summary draft followed by a final summary.  It achieves comparable or better results in term of both automatic evaluation metrics, e.g. compress ratio, rouge score, and human evaluations (Consistent and Informative) about the quality of generated summaries in different settings. 

However, there are still several disadvantages of this paper:
(1) It can generate a summary draft but its quality is almost not presented in the paper except the ablation study of table 1. Even the results within table 1 are still about the quality of the final summary. The paper slightly overclaims its usefulness in the draft summary generation. More results or analysis about draft summary should be presented.
(2) The human evaluation has only 30 examples and the scale is too small. Also, does the score is -1, 0, 1, or other scale? Do you use majority vote or mean plus standard deviation to get results in the table? Why gold in table 3 is so low in Consistent?
(3) The method looks applicable to the generable summarization task. More results of this are also interesting. 

Although these drawbacks, its quality is good overall.

review_title: This paper proposes CORDIAL to improve the abstractive dialogue summarization quality and controllability. It has a coarse-to- fine generation strategy that generates a summary draft followed by a final summary and a simple strategy to control the granularity of the final summary.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes CorDial, a new method for dialog summarization. CorDial firstly constructs a coarse draft by generating intent and key phrases for every dialog turn, and splits the dialog into chunks by inserting special boundary tokens; then the segmented original dialog and the constructed draft are feed as input to generate the final summary. CorDial employs BART-xsum as its backbone model, which is a pre-trained language model finetuned on XSUM summary dataset. Experiment result on SAMsum dataset shows CorDial achieves SotA performance under both automatic evaluation metric and human evaluations.

Overall, the paper presents an interesting, practical recipe for dialog summarization. It requires few additional annotation besides the summary, and the human evaluation results looks promising. However, the method proposed here somewhat lacks in novelty, and some part of the paper is not clearly written. Thus I give this paper a weak reject rating.

Comments:
1. In 2.2, how are the intents annotated? Is it a purely automatic process based on keywords matching? Or the keywords are merely cues for human annotators?
2. In 2.3, the algorithm for finding the cutting points is an incremental one. It can't account for the similarity between last chunk and the last sentence in the summary, since the cutting point of second to last chunk already determines the boundary of the last chunk.
3. How are the output generated exactly? The last sentence in 2.4 states each sentence is generated separately. Does it mean each output sentence have different input? How is it different from standard auto-regressive token-by-token generation?
4. 2.4 should also explicitly refer to Figure. 1 for clarity.
5. CorDial uses the BART-xsum as initialization, which is trained on XSUM dataset. Are other baselines also gone through the same XSUM training?
6. What is the model size of all the models in the experiments? It would be better to have some descriptions on model architectures in the experiment section.


review_title: Review for CorDial

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes CorDial for abstractive dialogue summarization. CorDial extends BART by generating an intermediate "summary draft" which provides weakly-supervised signals and controling the length of the final summary. Results show significant improvements over competitive summarization models such as PEGASUS and BART in multiple different metrics.

Some comments:

1. The paper emphasizes that dialogue summarization is challenging due to its multi-speaker standpoints, casual language, and limited data. Although the use of the proposed summary draft would help solve the first challenge, it is hard to see any correlation between the other problems mentioned and the proposed solutions in the paper. This is especially the case for the controlling of the summary length. Why is this useful specifically for dialogue summarization?

2. The "summary draft" is one kind of a content plan, which is widely used in text generation, including text summarization [1]. The technique of extracting key phrase is similar to how content selection is done in [2]. Please compare the proposed solution to other kinds of content planning.

3. To extract key phrases, the method identifies the longest common sub-sequence (LCS) parameterized by a threshold, however how this threshold is set and used is not discussed in the paper. This is important information in order to understand how these key phrases would look like. For example, in Figure 1, how is "s just one of many boring days at work" extracted when the LCS is only "at work" for turn 2?

[1] https://www.aclweb.org/anthology/C18-1101.pdf

[2] https://arxiv.org/pdf/1808.10792.pdf
review_title: Correlation between the given problem and the proposed solution?

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
<Summary>

This paper addresses the problem of abstractive dialogue summarization. Its key idea is to label an interrogative pronoun category and extract key phrases from each dialogue turn as weak guide for dialogue summarization. It also proposes a length-controllable generation method for final summary. The proposed approach is evaluated on the SAMSum as one of the largest abstractive dialogue summarization benchmarks, on which it shows competitive performance over recent models. 

<Strengths> 

1. It proposes a two-step coarse-to-fine approach for abstractive dialogue summarization; it first extracts category labels and key phrases from each dialogue turn, and then generates final summaries by controlling granularity. This idea itself could be novel.

2. It shows strong performance over other recent methods on the recently released SAMSum dataset. 

3. It tests the proposed approach with four recent pre-trained language models including DialoGPT, UniLM, PEGASUS and BART.

<Weakness>

1. This paper proposes a novel coarse-to-fine approach for abstractive dialogue summarization but its implementation is largely ad-hoc and engineering intensive and thus bears little technical novelty.

(1) The “coarse” part aims at generating drafts using interrogative pronoun category prediction and key phrase  extraction. 
These two are largely based on existing techniques (e.g. Ratner et al 2019 and Kitaev & Klien 2018) and some heuristics (e.g. thresholding for key phrases detection). 

(2) The “fine” part aims at generating target summary with controllability of granularity. 
Its implementation is also based on a series of engineering heuristics (e.g. dialogue splitting by ROUGE score, binary classification for cutpoint detection). 

(3) In summary, it is hard to find methodological novelty in the proposed method.  Given that ICLR is a top premier ML venue, it could be a significant weakness to be a publishable work. 

2. Experimental results are rather weak. 

(1) Although SAMSum dataset may be one of the best benchmarks for the target task, experiments on only a single dataset is limited to show the generality and effectiveness of the proposed method.
Given that the proposed method is ad-hoc, I suspect much additional endeavor may be required to apply to another dataset. 

(2) I am not sure whether the comparison in Table 1 is fair enough. Since the proposed approach relies on the additional components for draft construction, it could require more other types of training data or learned modules that other method may not need. This should be clarified in the draft.

<Conclusion>

My initial decision is ‘reject’ mainly due to lack of technical novelty. Limited experiments could be another issue to be improved. 
review_title: Review #1

Meta-review: The paper proposes a method for the interesting task of dialog summarisation which is slowly getting attention from the research community. In particular, they propose a method which first generates a summary draft and then a final draft.

Pros:
1) The paper is well written
2) Addresses an interesting problem
3) SOTA results

Cons:
1) Lack of novelty
2) No quantitative analysis of the summary draft though it is as an important part of the proposed solution
3) Human evaluations are not adequate (the authors have said they will expand on this but clear details are not provided)
4) The BART model seems to have some advantage as it is pre-trained on XSUM data whereas some of the other models are not (the authors haven't clarified this sufficiently in the rebuttal)

Overall, the reviewers were not completely happy with the work and there was not clear champion. 
Decision: Reject