Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces disentangled paper representation, which tries to generate paper summaries that explicitly differentiate between the contributions and the context of the work.  I like the general direction and the approaches, but I feel that the experiments and analysis need more work.

Strengths:
I generally think decomposing scientific article summaries into two pieces, contextual descriptions vs. contributions, would be helpful.  The two methods evaluated here (one using control codes, the other using a multi-head architecture) seem reasonable.  Also, the informativeness approach (equation 2) based on inbound vs. outbound citations is well-motivated and interesting.  The paper uses a reasonable set of automated metrics on a silver data source, and I found those experiments to be relatively well-done although ultimately the results are somewhat inconclusive, in that there were not strong differences between different techniques.

Weaknesses:
Because automated metrics are always highly imperfect, human evaluation of the summaries is important.  Unfortunately I think the human evaluation in this paper leaves too many questions unanswered.  First, some details are missing -- what was the inter-annotator agreement, and what exactly was the prompt given to the annotators?  Further, because the experiment tests standard output (regular abstract format) against a newer thing (with context and contribution structure), it seems possible that annotators were biased to prefer the disentangled summaries just because they’ve novel and/or suspected to be the target of the study.  Some discussion of this bias is necessary.  Also, more analysis of why the annotators preferred the disentangled summaries would help alleviate the bias concerns (ideally, the paper would perform a task-based evaluation showing that the disentangled summaries help users more rapidly or accurately browse the literature).

Additionally, the truthfulness of the generation seems like a potential concern that is not discussed.  For example, the contextual output in Table 3 includes a lot of general facts about the domain that are different from the ones shown for the given abstract.  Are all of these statements factual, i.e. from the source documents, or does the abstractive model “make things up”?  The evaluation does not directly address this question.

Suggestions/questions:
I’d like to see an analysis of how important it is to have the inbound citations---these will not exist for the most recent papers, which are arguably the most valuable ones to summarize.  While the paper does ablate the informativeness measure based on the citations it does not experiment with removing the inbound citation text from the input, I would like to see that.

The ablation studies (Table 2) suggest that the informativeness objective does not particularly help for the control codes model, even with disentanglement.  But multi-head is helped more by informativeness (or more precisely, multi-head suffers with out it).  The paper notes this effect, but can you explain more why MH and CC differ in this way?
review_title: Interesting task, shows promise, but needs more extensive experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work presents a dataset based on S2ORC with additional annotations on the abstracts in terms of contribution and context. Based on this dataset, this paper also adopts two baseline models from prior work with a training strategy (also defined by prior work) to demonstrate the task of summarizing scientific literature from two aspects. 

Overall, I was confused about the contribution of this work. Based on my understanding it could either be presenting a newly annotated dataset or demonstrate this task using some baseline models. However, neither of them has enough evidence to convince me that this is a solid work. 

About the data, I have a few questions about the annotation procedure:

1) about data curation, if there are more than 20 incoming citations, the proposed method suggests to keep only the top 20 papers based on their citations. However, how do we know these 20 papers are the most relevant to a specific work? 

2) I had trouble to understand the value of "silver" standard references. First of all, the classification accuracy is only 86.3%, without further guarantee, we have no idea about the annotation quality. Besides, if the major contribution of this work is about the dataset, then manually annotating 400 abstracts really did not sound like enough contribution.

My confusion about the contribution of this work mainly came from section 5 and 6. Particularly, in section 6, the line "to better understand the strengths and shortcomings of our models" reminds me that maybe the contribution of this work is about those models. Another confusion is that, if this work is about the dataset, then I really think it is necessary to analyze the annotation quality, instead of the performance of existing models. 
review_title: Confused about the major contribution of this work

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discussed the task of generating scientific paper summaries along two axes: contribution (ie. novelty) and context/background. The point is two disentangle these two, since often different readers would be interested in only one of these. While I like the idea of this paper, I think it has the following issues:

- Evaluation: There is no evaluation against a manually annotated gold standard. The references used in section 5 are based on the output of a classifier trained on manual annotations, which itself was not evaluated against manual annotations either. Thus I am rather concerned about what we can learn from the results presented. Section 6 presents some observations in 6.1. but even though they are useful on can't read much in them. And I am not convinced that the lack of expert annotators means that one cannot assess hallucinations; if one can assess whether the novelty is extracted correctly (in the same section), then hallucinations should be possible to assess. In section 6.3 the human evaluation is essentially comparing the model proposed (unclear which of the variants though) against the output of a summarization model that doesn't separate novelty from context. However we don't know how good this summarization model is, let alone that asking about usefulness only conflates a number of aspects that matter in a summary, such as informativeness, fluency, factual correctness, readability, etc. Overall, I think the evaluation is not adequate for a paper at a top conference.

- Modelling: Why not experiment with extractive summarization? Abstracts often overlap with the main paper, and you have a sentence level classifier anyway. And the hallucination wouldn't be an issue then. Also there is previous work on modelling citation function: https://www.cl.cam.ac.uk/~sht25/papers/emnlp06.pdf which should be acknowledged but also could be useful in creating better data and models. Similarly, there is work on scientific article zoning: http://oro.open.ac.uk/58880/

- I am not sure I follow the second branch in equation 2: why should context surprise the articles citing the one being summarized? They would share the context quite often. Also I don't see the connection to the disentanglement.

review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: Thanks to the authors for the hard work on this paper. It starts with an interesting and promising premise, and the human evaluation suggests the authors have really hit on something. I have given it a rating of 4 as I feel that this work needs more analysis before it is ready for publication, but I certainly hope to see it published in a more thorough form sometime soon.

The single biggest concern I have is the silver data that is described in "Reference Generation". The authors manually annotated 400 abstracts, and classified each sentence in each abstract into either "contribution-related" or "context-related". They trained a classifier, and obtained 86% accuracy on a held-out dataset. This classifier is then both their source of training data *and* ground truth for everything except the human evaluation. Despite being the foundation of the paper, the section about "Reference Generation" is short and lacks the kind of details I believe it needs for the paper to be truly convincing. Here is a list of some concerns and questions related to this section:

* 86.3% accuracy is not a revealing metric because we don't know how unbalanced the test set is. Please report area under the ROC curve instead.
* I assume you chose 0.5 as the threshold cut-off for the final context/contribution classifier. In Section 5.3 you report that "context summaries include article contribution information." This may be related to the kinds of errors made by the original classifier. What happens when you change the operating point?
* 400 samples for training is relatively small. How representative are they of the different scientific fields in Table 4? Abstracts in biomedicine are very different than abstracts in philosophy. What is the held-out AUC per field? 
* What if you add another ~50 samples to the 400 samples in the training data? Does your held-out AUC go up? This would help you assess if 400 documents is enough.
* How much inter-annotator agreement is there for this task? How many annotators did you use per document?
* Let's say the abstract is composed of a series of sentences: ABCDEFG, and the "contributions" are BDFG and "context" are ACE. These are non-contiguous passages, and are likely less fluent than the original abstract. How does this reduction in fluency affect your summarization algorithms? Is there a reduction in fluency in the generated passages as well?

Other questions/concerns:
* In data curation, the number of incoming/outgoing citations was limited to 20. Why?
* If I was to imagine a use for a real system based on this work, it would be applied to academic papers shortly after publication. This means that in a production setting, there will be *no* incoming citations. Does your algorithm still work in this case?
* As far as I understand, you used a distilBART model to summarize the entire full text of the paper, without the abstract, but including the title. Is this correct? If so, please clarify exactly what the input to the model is. How do you deal with long documents using a model that has a much smaller maximum input?
* In Eq (2) - how do you compute $p(.| D)$, $p(.| C_O)$, $p(.| C_I)$?
* It's hard to interpret the numbers in Table 2 without some sort of baseline to compare against. Please consider adding a semi-naive baseline of some sort.
* Please provide a bit of information about how you set the various hyper-parameters in section 5.2. By cross-validation? Hand-tuning? 
* In section 6.1 you state that "the factual correctness ... could not be assessed". For scientific text factual correctness of any summarization system is essentially a requirement. Given that you have access to expert evaluation, I think it's critical that some evaluation of factual correctness is done.
* For the human evaluation - how many annotators did you use per document? What was the inter-annotator agreement?


review_title: Interesting direction, needs more work

Meta-review: The paper attempts at generating two types of summaries for scientific papers: summary of contribution and summary of background context. Most reviewers appreciated the motivation and found this research to be quite interesting and useful, however all reviewers had concerns regarding both execution/presentation of the ideas. While authors try to address some of the concerns, there are many clarifying questions and points raised by reviewers. Addressing all these points requires rather a major revision. Therefore the paper is not quite ready yet and would benefit from another iteration.
Decision: Reject