Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the unfaithfulness problem in abstractive summarization with two proposed techniques: (1) using a question-answering model during training where a multi-task encoder encodes either the source document or questions generated from the document, and a QA-enhanced decoder generates either the summary or the generated answers, and (2) using a max-margin loss defined on the difference between the language model and the summarization model. Experiments show that the proposed model outperforms strong baselines on both summary informativeness and factuality evaluations.
strengths_and_weaknesses: Strengths
* The results are very good both in terms of informativeness (ROUGE and BertScore) and factuality (FactCC and QE). 
* The use of a QA-enhanced model is intuitive given that questions in the faithful summary should be answerable given the source document as the context.

Weaknesses
* I find the methodology section to be very unclear. For one thing, the section could have been easier to understand if it included a problem setup formally defining the multi-task setup. It is not clear what are the differences in this setting from a normal summarization setting, where we are only given two sequences of tokens (one for the source and another for the summary). One example is that it is not clear if the corresponding answers to the given questions are also given during training, or if a QA/Q generation model is assumed to be available. Another thing that is unclear is how entity/question/sentence "nodes" are created given the tokens from the transformer encoder.
* I find the second contribution (the use of max-margin loss between the language model and the summarization model) to be counterintuitive in terms of improving the faithfulness of the summaries. Firstly, the summarization model is basically a fine-tuned version of the language model, how can we assume that the summarization model produces "adequate words" while the language model does not? If such "over-reliance" on the language model exists in LM-finetuned summarization models, shouldn't models not finetuned with LM produce more factual summaries? We know given previous work that this is not the case.
questions: How do we interpret the FactCC and QE results? How high can we achieve here, given a perfect summary? If it is 100, are we still far off from producing abstractive and factual summaries, given XSum results?
limitations: None.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Generating summaries that are faithful is an important problem to be addressed for the summarization task. Faithfulness problem in summarization can be perceived in two types: (a) Intrinsic problem; and (b) Extrinsic problem. This paper tries to address these two types of problems with a novel modeling approach. This modeling approach proposed to use question-answering (QA) to examine whether the encoder efficiently represents the document thereby being able to answer the question on key information in the document. Further, the model is optimized on max-margin loss between language model and summarization model so that the model doesn’t over confidentially rely on the language modeling aspect and producing hallucinations. 

Experimental results on CNNDM and XSum datasets suggest that the proposed model is indeed getting better on the factuality.

strengths_and_weaknesses: Strengths:
1) The proposed approach is very interesting and novel. The paper is well written and easy to follow. 
2) Experimental results are thorough in terms of (a) comparing with many previous works and baselines, (b) showing the importance of each of the proposed modules in the model, and (c) showing the factuality improvements with human evaluations. 
3) All the ablations are very interesting and very useful. 

Weaknesses: 
1) Human evaluations seem pretty close, with most of the weight going to Tie. Also, the sample size seems small given the close gap between the models. Can you increase the sample size and also do some statistical significance testing? 
2) Suggestion: Please provide qualitative or quantitative analysis on where the proposed model makes factuality errors. Table 6 is great, but would be more useful to understand the limitations as well.  


questions: 1) Is the human evaluation gap statistically significant? 
2) The proposed model is not 100% factual right? What are its limitations? 

limitations: Adequately addressed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Modifies state-of-the-art seq2seq summarization models (BART/PEGASUS) to be more faithful when generating summaries. There are a few orthogonal techniques/improvements: (1) adding QA auxiliary task to improve representations; (2) using a fancier encoder that creates representations for semantic nodes using Graph attention networks; (3) modifying decoder attention; (4) adding max margin loss comparing LM vs summarization token probabilities.

QA pairs are created for summarization data using QuestEval (Scialom et al); pairs with answers with high ROUGE overlap with summary work better.

Ablations of enhancements are done on the CNN/DM dataset, while final model is also evaluated on XSum. Automatic metrics involve ROUGE/BERTScore, FactCC, and QE. Human eval is done via pairwise comparison of final model with baseline PEGASUS/BART models, evaluating both informativeness and factuality. In the end the final model does best in both automatic and human eval evals.

strengths_and_weaknesses: # Strengths
- Proposes multiple interesting techniques to improve faithfulness in summaries while also improving traditional evals e.g. ROUGE. 
- Ablations done to show all contributed to improvements in ROUGE.
- Overall final model results are good and improves near-SOTA baseline models.
- Comprehensive evaluation was done using multiple automatic metrics and human eval.
- Faithfulness is a very important topic in summarization, and these improvements could have significant practical impact.

# Weaknesses
- Another task/dataset would be nice, e.g. non-news dataset, with known faithfulness issues, e.g. more abstractive. CNN/DM by its extractive nature doesn't typically exhibit huge faithfulness issues.
- Ablations would be better on XSum which has more faithfulness issues (more abstractive)
- Unclear whether all techniques are necessary to significantly improve faithfulness as judged by humans
- Some details of implementation appear to be missing (see Questions)

questions: # Questions / Clarifications
- For the Max-margin loss, how do you train the LM? I don't think this is described.
- In line 209, you use oracle QA Pairs instead of those from the extraction model, which is used at test time. What is the effect on performance vs using the extraction model at training?
- Using QA as an auxiliary task seems intuitive for improving representations for summarization, but it is unclear in the paper how much of improvement is from using a fancier encoder (with graph attention). It'd be great to simply have a baseline with no architecture changes, but only adding the QA auxiliary task.
- Table 2 shows FES doing better than the baseline, but SimCLS also does better. Have you compared SimCLS and FES directly in a side-by-side?
- When training the semantic nodes, is the Transformer encoder frozen?
- In ablations, how can you remove 'multi' without removing QA attention. Doesn't the latter depend on the former since H_e is necessary for QA attention?


# Suggestions/comments
- Please add reference summaries to examples in Appendix
- It'd be nice to see the same ablations on Xsum that you did for CNN/DM. XSum has more faithfulness issues in general.
- It'd be great to compare the model and human references in a faithfulness side-by-side

limitations: While the ablations are in 4.3 are nice, they are only done with automatic metrics. it is unclear if all techniques are necessary to improve faithfulness in a human eval. e.g. multi-task encoder might be the only significant improvement. Which of the orthogonal techniques improves things by themselves (this is already shown in ROUGE on CNN/DM)?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a multi-task model to enhance the faithfulness of abstractive summarization models. A synthetic QA dataset is created with the input documents as context and nouns/entities as answers. The model consists of a graph encoder that represents both the document and questions at various levels of granularity. The decoder is tuned by minimizing the KL divergence between the entity (QA) attention and the token (summary) attention. An additional max-margin loss is supplemented to the overall loss so that the language model does not overtake the decoding process. The model is evaluated on CNN/DM and XSUM and compared against a few baselines. Ablations are performed by iteratively removing the newly added components/losses. 
strengths_and_weaknesses: In this line of research, it is difficult to exactly identify which change results in the largest performance jump and we see the same pattern here. Although it is common knowledge that attending to entities results in more faithful summaries, the paper presents a modeling-based solution to the same and does a good job of validating the solution. I would like to see the paper accepted, but have a few points of concern. I am willing to increase my rating provided all the questions/comments are addressed.
questions: Questions/comments:
- It's usually the entity spans that are considered important. What about the relationships among them? For example, how will the model choose between "Argentina won against Mexico" versus "Mexico won against Argentina"? This problem may be further reinforced by the way it is implemented (Line 227: To avoid the model from learning the position information of entities or questions, we sort the entities and questions in alphabetical order.)
- An interesting experiment would be to plot the values of m_t only when the decoder generates entities. For most function words, it will likely be ~0 since the pre-trained decoder will be very confident while emitting them.
- Evaluating on additional metrics would make the results more convincing (entailment, FEQA, BertFaithful, etc.)
- Since the main objective of the multi-task setup is to get better entity representations into the decoder, wouldn't it be more natural to use NER as the auxiliary task? What benefits does QA bring over NER?
limitations: Other:
- Missing reference: Focus Attention: Promoting Faithfulness and Diversity in Summarization (https://research.google/pubs/pub50585/)
review_title: null

Meta-review: metareview: All reviewers agree that the method is interesting, the experiment results are thorough and significant. Multiple reviewers mentioned the paper would have been stronger by demonstrating performance on a dataset in another domain (i.e., not newswire), but still gave high scores. Given the consensus among reviewers, this paper should be accepted.
confidence: Certain
Decision: Accept