What's new? Summarizing Contributions in Scientific Literature
Authors: Hiroaki Hayashi
Wojciech Maciej Kryscinski
Bryan McCann
Nazneen Rajani
Caiming Xiong
Venue:  (2021)
Keywords: abstractive summarization
scientific papers

Abstract: With thousands of academic articles shared on a daily basis, it has become increasingly difficult to keep up with the latest scientific findings. To overcome this problem, we introduce a new task of $\textit{disentangled paper summarization}$, which seeks to generate separate summaries for the paper contributions and the context of the work, making it easier to identify the key findings shared in articles. For this purpose, we extend the S2ORC corpus of academic articles, which spans a diverse set of domains ranging from economics to psychology, by adding disentangled "contribution" and "context" reference labels. Together with the dataset, we introduce and analyze three baseline approaches: 1) a unified model controlled by input code prefixes, 2) a model with separate generation heads specialized in generating the disentangled outputs, and 3) a training strategy that guides the model using additional supervision coming from inbound and outbound citations. We also propose a comprehensive automatic evaluation protocol which reports the $\textit{relevance}$, $\textit{novelty}$, and $\textit{disentanglement}$ of generated outputs. Through a human study involving expert annotators, we show that in 79%, of cases our new task is considered more helpful than traditional scientific paper summarization.


Under review as a conference paper at ICLR 2021
WHAT'S NEW? SCIENTIFIC LITERATURE
SUMMARIZING CONTRIBUTIONS IN
Anonymous authors Paper under double-blind review
ABSTRACT
With thousands of academic articles shared on daily basis, it has become in- creasingly difficult to keep up with the latest scientific findings. To overcome this problem, introduce new task of disentangled paper summarization, which seeks to generate separate summaries for the paper contributions and the context of the work, making it easier identify the key findings shared in articles. For this purpose, we extend the S2ORC corpus of academic articles, which spans a di- set of domains ranging from economics to psychology, by adding disentan- gled "contribution" and "context" reference labels. Together with the dataset, we introduce and analyze three baseline approaches: 1) a unified model controlled by input code prefixes, 2) model with separate generation heads specialized in gen- erating the disentangled outputs, and 3) a training strategy that guides the model using additional supervision coming from inbound and outbound citations. We also propose comprehensive automatic evaluation protocol which reports the relevance, novelty, and disentanglement of generated outputs. Through human study involving expert annotators, we show that in 79%, of cases our new task is considered more helpful than traditional scientific paper summarization.
INTRODUCTION
With the growing popularity of open-access academic article repositories, such as arXiv or bioRxiv, disseminating new research findings has become nearly effortless. Through such services, tens of thousands of scientific papers are shared by the research community every month" At the same time, the unreviewed nature of mentioned repositories and the sheer volume of new publications has made it nearly impossible identify relevant work and keep up with the latest findings.
Scientific paper summarization, subtask within automatic text summarization, aims to assist re- searchers their work by automatically condensing articles into a short, human-readable form that contains only the most essential information. In recent years, abstractive summarization, proach where models are trained to generate fluent summaries by paraphrasing the source article, has seen impressive progress. Slate-of-the-ar art methods leverage large, pre-trained models (Raffell et al. 2019. Lewis et al. 2020). define task-specific pre-training strategies (Zhang et al.) 2019) and scale to long input sequences (Zhao et al. 2020 Zaheer et al. 2020). Available large-scale bench- mark datasets, such as arXiv and PubMed (Cohan et al. 2018). were automatically collected from online archives and repurpose paper abstracts reference summaries. However, the current form of scientific paper summarization where models are trained to generate paper abstracts has two caveats: 1) often, abstracts contain information which is not of primary importance. 2) the vast majority of scientific articles come with numan--ritten abstracts, making the generated summaries superfluous.
ap-
To address these shortcomings, we introduce the task of disentangled paper summarization. The new task's ' goal is to generate two summaries simultaneously, one strictly focused on the summarized ar- ticle's novelties and contributions, the other introducing the context of the work and previous efforts. In this form, the generated summaries can target the needs of diverse audiences: senior researchers and field-experts who can benefit from reading the summarized contributions, and newcomers who can quickly get up to speed with the intricacies of the addressed problems by reading the context summary and get a perspective of the latest findings from the contribution summary.
https://arxiv..rg/
ission ons

---

Under review as a conference paper at ICLR 2021
For this task, we introduce new large-scale dataset by extending the S2ORC (o et al., 2020 corpus of scientific papers, which spans multiple scientific domains and offers rich citation citation-related metadata. We organize and process the data, and extend it with automatically generated contribution and context reference summaries, to enable supervised model training. We also introduce three ab- stractive baseline approaches: 1) unified, controllable model manipulated with descriptive control codes (Fan et al.] 2018] Keskar et al. 2019). 2) a one-to-many sequence model with a branched decoder for multi-head generation (Luong et al. 2016 Guo al. 2018), and 3) an information- theoretic training strategy leveraging supervision coming from the citation metadata (Peyrard 2019). To benchmark our models, we design a comprehensive automatic evaluation protocol that measures performance across three axes: relevance, novelty, and disentanglement, We thoroughly evaluate and analyze the baselines models and investigate the effects of the additional training objective on the model's behavior. To motivate the usefulness of the newly introduced task, we conducted human study involving human annotators in a hypothetical paper-reviewing setting. The results find disentangled summaries more helpful in 79% of cases in comparison to abstract-oriented out- puts. Code, model checkpoints, and data preparation scripts introduced in this work are available at funhtps:///gg httb:::://qitbb..b com sa sales olaeeeesoe-o//ddssttaaggll-----d
2 RELATED WORK
Recent trends in abstractive text summarization show shift of focus from designing task-specific architectures trained from scratch (See et al. 2017, Paulus et al. 2018) to leveraging large-scale Transformer- baasdd models pre-trained vast amounts of data (Luu & Lapata 2019 Lewis al. 2020), often in multi-task settings (Raffel et al., 2019). A similar shift can be seen in scientific paper summarization, where state-of-the-ar approaches utilize custom pre-training strategies (Zangg al. 2019) and tackle problems of summarizing long documents (Zhao et al. 2020 Zaheer et al. 2020) Other methods, at a smaller scale, seek utilize the rich metadata associated with scientific articles and combine them with graph-base methods (asunaga et al. 2019). In this work, we combine these two lines of work and propose models that benefit from pre-training procedures, but also take advantage of task-specific metadata.
Popular large-scale benchmark datasets in scientific paper summarization (Cohan et al. 2018) were automatically collected from open-access paper repositories and consider article abstracts as the reference summaries. Other forms of supervision have also been investigated for the task, including author-written highlights (Collins al. 2017), human annotations and citations (Yasunaga et al. 2019). and transcripts from conference presentations of the articles (Lev et al. 2019). In contrast, introduce large-scale automatically collected dataset with more inn-ggrined references than abstracts, which also offers rich citation-related metadata.
Update summarization (Dang & Owezarzak) defines a setting in a collection of documents with partially overlapping information is summarized, some of which are considered prior knowledge. The goal of the task is to focus the generated summaries on the novel information. Work in this line of research mostly focuses on novelty detection in news articles (Bysani 2010 Delort & Alfonseca 2012) and timeline summarization (Martschat & Markert, 2018 Chang et al. 2016) on news and social media domains. Here, we propose a novel task that is analogous to update summarization in that it also requires contrasting the source article with the content of other related articles which are considered pre-existing knowledge.
3 TASK
Given a source article D, the goal of disentangled paper summarization is to simultancously sum- marize the contribution Yeon and context Betz of the source article. Here, contribution refers to the novelties introduced in the article D, such as new methods, theories, or resources, while context represents the background of the work D, such as a description of the problem or previous work on the topic. The task inherently requires relative comparison of the article with other related papers to effectively disentangle its novelties from pre-existing knowledge. Therefore, we also consider two sets of citations: inbound citations C2 and outbound citations Co potential sources of useful information for contrasting the article D with its broader field. Inbound citations refer to the set of papers that cite D, i.e. relevant future papers, while outbound citations are the set of papers that
2

---

Under review as a conference paper at ICLR 2021
Table 1: Token length statisties on the training split our dataset compared existing scientific paper sum-
marization datasets. Contribution summaries tend to be shorter than context summaries.
Avg. #Tokens
Dataset
#Examples
Paper
Inbound Cy
Outbound Co
Contribution yeom
Context yetr
ArXiv (Train)
203037
4938
220 (Total summary)
PubMed (Train)
119924
3016
203 (Total summary)
Ours Train
805152
6351
925
877
136
236
Valid
36129
6374
922
875
135
236
Test
54242
6350
927
892
136
237
D cites, i.e. relevant previous papers. With its unique set of goals, the task of disentangled paper summarization poses novel set of challenges for automatic summarization systems overcome: 1) identifying salient content of D and related papers from C₁ and Co. 2) comparing the content of D with each document from the citations, and 3) summarizing the article along the two axes: contributions and context.
3.1 DATASET
Current benchmark datasets used for the task of scientific paper summarization, such arXiv and PubMed (Cohan & Goharian 2015), are limited in size, the number of domains, and lack of citation metadata. Thus, we construct a new dataset based on the S2ORC (Lo al. 2020 corpus, which offers a large collection of scientific papers spanning multiple domains along with rich citation- related metadata, such as citation links between papers and annotated citation spans. Specifically, we carefully curate the data available in the S2ORC corpus and extend it with new reference labels.
Data Curation Some papers in the S2ORC corpu do not contain a complete set of information required by our summarization task: paper text, abstract, and citation metadata. We remove such instances and construct paper summarization dataset in which each example a) has abstract and body text, and b) has at least or more inbound and outbound citations, Cj and Co respectively. In cases where paper has more than K incoming outgoing citations, first sample K citations for each of incoming and outgoing citations and sort them in descending order by the number of their respective citation from and to the target paper. K is hyperparameter and we choose K 20 in this work.
Citation Span Extraction Each article in the set of inbound and outbound citations can be rep- resented by its full text, abstract, or the span of text associated with the citation. In this study, we follow Qazvinian & Radev (2008) and Cohan & Goharian (2055) in representing citations with the sentences in which the citation occurs Thus, an outbound citation is represented by sentence from the source paper. Usually, such sentences directly refer to the cited paper and place its content in relation to the source paper. Analogously, an inbound citation is represented by sentences from the citing paper and relates its content with the source paper.
Reference Generation Our approach relies the availability of reference summaries for both contributions and contexts. However, such annotations are not provided or easily extractable from the S2ORC corpus, and collecting expert annotations infeasible due to the associated costs. There- corpus,
fore, we apply a data-driven approach to automatically extract contribution and context reference summaries from the available paper abstracts. First, we manually label 400 abstracts sampled from the training set. Annotations are done on a sentence-level with binary labels indicating contribution- and context-related sentences This procedure yields 3341 sentences with associated binary labels, which we refer to as golden standard references. Next, we fine-tune automatic sentence classifier using the golden standard data, As our classifier we use SciBERT (Beltagy et al., 2019), which after fine-tuning achieves 86.3% accuracy and 0.932 for Area under ROC curve in classifying tribution and context sentences on a held-out test set. Finally, we apply the fine-tuned classifier
2 "Reccase ID: 20191928.
If publication is cited multiple times within source article concatenate relevant sentences. Sentences not laheled contribution are considered context, finer-grained lahels work.
3

---

Under review as a conference paper at ICLR 2021
code: Distant supervised relation extraction (RF) has boen an effective way of finding novel relational facts from text without labeled training Typically can formalized multi instance multilabel problem. In this papor, introduce
Distant supervised relation extraction (RE) has hoon an offective way of finding novel relational facts from text without labeled truining data. Typically it he formalized as multi instance multilabel problem. In this paper. we introduce
contri bution:
Context:
Shared Encoder Layers
Shared- parameeer Model
Shared Decoder Layers
Contribution Decoder Layer
Context Decoder Layer
In this paper, we propose an attention- -bseed memory network distant supervised relation extraction (RE), which is capable of modeling the semantic relatedness
Relation extraction (RE) is Fundamental task in NLP, and crucial component building structured Knowlodge Base from free texts. Previous methods including... ...
this puper, we propose an attention based memory network for distant supervised relation extraction (RB), which capable of modeling the semantic relatedness
Relation extraction (RE) is fundamental task in NLP,, and crucial component building structured Knowledge Base from froe texts. Previous methods incloding
Model diagram. Left: CONTROL CoDe model. in which inputs are prefixed with prompt symbol and passed to shared model control the oulput mode. Right: MULTIHEAD madel, which shares all of the model's 's parameters apart from last decoder layer for different output modes, and chooses the final decoder layer accordingly to control the output mode.
Figure 1:
to generate reference labels for all examples in our dataset, which we refer to as silver standard references. The statistics of the resulting dataset are shown in Table 1
MODELS
Our goal to build an abstractive summarization system which has the ability to generate con- tribution and context summaries based on the source article. To achieve the necessary level of controllability, we propose two independent approaches building on encoder-decoder architectures:
common approach controlling model-generated text is by condition- ing the generation procedure on a control code associated with the desired output. Previous work on controllable generation (Fan al. 2018 Keskar et al. 2019) showed that prepending a spe- cial token or descriptive prompt to the model's 's input during training and inference is sufficient to achieve fine-grained control over the generated content. Following this line of work, we modify our training instances by prepending textual control codes, contr rittuuuuu ion or context: :, to the summarized articles. During training, all model parameters are updated for each data instance and the model is expected to learn to associate the provided prompt with the correct output mode. The approach does not require changes in the architecture, making it straightforward to combine with existing large-scale, pre-trained models. The architecture is shown the left of Figure D
CONTROL.CDDE (CC)
MULTIHEAD (MH) An alternative way of controlling generation is by explicitly allocating lay- ers within the model specifically for the desired control aspects. Prior work investigating multi-task models (Luong et al. 2016 Guo et al. 2018) showed the benefits of combining shared and task- specific layers within single, multi-tase architecture. Here, the encoder shares all parameters be- tween the two generation modes, while the decoder shares all parameters, apart from the final layer, which splits into two generation branches. During training, each branch is individually updated with gradients from the associated mode. The model shares the softmax layer weights between the out- put branches under the assumption that token-level vocabulary distributions are similar in the two generation modes due to the common domain. This approach is presented on the right of Figure
4.1 INFORMATIVENESS TRAINING
Peyrard 2019) proposed an information-tee -tteoortttcctrtcccccc perspective on text summarization which decom- poses the criteria of a good summary into redundancy, relevance, and informativeness. Among these criteria, informativeness measures the user's 's degree of surprise after reading summary given their background knowledge, and can be formally defined as:
Inf (D, K) - Σ Po(li) log Px (w:;,
where Wi is primitive semantic unit, Pk is the probability over the unit under the user's 's knowledge, Pp is the probability over the unit with respect to the source document, and i is an index over all semantic units within summary.
4

---

Under review as a conference paper at ICLR 2021
informativeness is in direct correspondence to contribution summa- rization. Paper contributions are novel contents introduced to the community, which causes surprisal given the general knowledge about the state of the field. Therefore, in this work explore utilizing this measure as an auxiliary objective that is optimized during training. We define the semantic unit wi the summary itsel which enables simple interpretation of the corresponding probabilities. We estimate PD as the likelihood of the summary given the paper content, PD (w(w) = p(y D). Since each paper is associated with a unique context and background knowledge, treat the back- ground knowledge
As defined by Peyrard
(211)).
all relevant papers published before the source paper, i.e. outbound cita- tions Co. Therefore, Pk is estimated as the likelihood of the summary given the previous work, Px (w(w))= = p(y Co). We formulate the informativeness function
P(yeon log P(ycon Co) if generating contributions (C]) otherwise
Inf(D,
-p(yett
D) log p(yetz
where the conditioning depends on the generation mode of the model, and aim to maximize it during the training procedure. The estimation each term is done by a forward pass on the model with corresponding input and output pairs: p[yeon Co) is computed by estimating the probability of Yeon when feeding Co as the encoder input.
Combined with cross entropy loss LCB, obtain the final objective which aim to the minimize during training:
L = = - À Inf (D,
where À is scaling hyperparameter determined through cross-validation.
EXPERIMENTS AND RESULTS
In this section, we deseribe the experimental environment and report automatic evaluation results. We consider four model variants:
CC, CC+INF: CONTROLCODE model without and with the informativeness objective,
MH. MH++INF: MULTIHEAD model without and with the informativeness objective.
5.1 EVALUATION
We perform automatic evaluation of the system out- puts (Beon, Sctz) against the silver standard refer- ences (Yeon, Motz). For this purpose, we have de- signed comprehensive evaluation protocol, shown in Figure 2 based on existing metrics that evaluates the performance of models across dimensions:
Scon
S ctx
Relevance
Purity
Disentanglement
Ycon
Yctx
Relevance Generated summaries should closely correspond with the available reference summaries. We measure the lexical overlap and semantic sim- ilarity between (Scon- Veon) and (Setz, Vetz) using
Diagram illustrating the evaluation protocal assessing summaries along 3 axes: rel- evance, purity, and disentanglement.
Figure 2:
ROUGE (R-i) Lin 2004) and BERTSScre (Zhang et al. 2020 BS), respectively.
Purity Generated contribution summary should closely correspond with its respective reference summary, but should not overlap with the context reference summary. We measure the lexical overlap between Scon and (Veon, Yetw) using NouveauROUGE con (Ncon-)) (Conroy et al. 2011 The metric reports an aggregate score defined as a linear combination between the two components:
NouveauROUGE...t a a ROUGE-i(scco Noon) a ROUGE- Netw),
where weights aj were set by the original authors to favor outputs with maximal and minimal overlap with related and unrelated references, accordingly. Analogously, we calculate Neto-i in
For simplicity in madeling, we chose the entire summary. However, this goes aguinst the requirement set by Peyrard 2019 that wz is prinitive semantic unit, because paragraph's meaning decomposed into higher granolar units.
5

---

Under review as a conference paper at ICLR 2021
reverse direction between Set and (Yetax Meon). Purity P-i is defined as the average novelty in both directions:
Purity-i = (Nem-i + Natz-11/2 (P-i).
Disentanglement Generated contribution and context summaries should have minimal overlap. We measure the degree of lexical overlap and semantic similarity between (Scon: Sctx) using ROUGE and BERTScore, respectively. To maintain consistency across metrics (higher is better) we report disentanglement scores complements of the associated metrics:
DisROUGE-i 100 ROUGE-i; (D-i),
DisBERTScore = 100 - BERTScore; (DBS).
5.2 IMPLEMENTATION DETAILS
Our models build upon distilBART (Sanh al. 2019 Wolf al. 2019). a Transformer- based Vaswani et al. 2017), pre-trained scquence-fo-sequencee architecture distilled from BART Lewis et al. 2020). Specifically, we used a model with self-attention layers both the Encoder and Decoder. Weights were initialized from model fine-tuned on news summarization task. For the MULTIHEAD model, the final layer of the decoder was duplicated and initialized with identical weights. We fine-tuned on the training set for 80000 gradient steps with fixed learning rate of 3.0 x 10-5 and choose the best checkpoints in terms of ROUGE-1 scores on the validation set. The loss scaling hyparameter À (Eq. was set 0.05 and 0.01 for the CONTROL.CODS and MULTIHEAD models, accordingly. Input and output lengths were set to 1024 and 200, respectively. At inference time, we decoded using beam search with beam size 5. The evaluation was performed using SummEval toolkit (Fabbri et al. 2020
5.3 RESULTS
In Table 2 we report results from the automatic evaluation protocol described in Subsection 5.1
Relevance Across most models and metries, relevance scores for context generation are higher than those for contribution summarization. Manual inspection revealed that in some cases generated context summaries also include article contribution information, while this effeet not observed in the reverse situation. Considering that silver standard annotations may contain noisy examples with incorrectly separated references, may be caused by noisy predictions coinciding with noisy references. Examples of such summaries shown in the Appendix D We also observe that informativenes--uideed models (+INF) perform on par with their respective base versions, and the additional training objective does not affect the performance on the relevance metric. This insight corroborates with Peyrard (2019) who defines informativeness and relevance orthogonal criteria.
suspect that higher ROUGE scores for context summaries
Purity While the informativeness objective was designed to improve the novelty of generated summaries, results show an opposite effect, where informativeness-guide perform their base counterparts. The true reason for such behavior is unknown, however, it might be an indicator that the outbound citations Co not good approximation of reference context summaries Vat. or the relationship between the two is weak. This effect is more evident in the Medical and Biology domains, which are the two most frequent domains in the dataset.
models slightly under-
Disentanglement Results indicate that CONTROL Cose- models perform better than MUL- TIHEAD approaches in terms of generating disentangled outputs. This comes as surprise given that the CC models share all parameters between the two generation modes, but might indicate that the two tasks contain complementary training signals. We also noticed that, both informativeness- guided models performed better in terms of D-1.
Based on both purity and disentanglement evaluations, we suspect that the informativeness objective does guide the models to output more disentangled summaries (second term in Eq 2 but the signal
"We did not ohserve substantial difference in performance hetween distiBART and BART.
Model weights are available wwt.t://t/ee.........o... //hugg/ ingaaae oo/ashleleeeeeoeoooeooeee le for/student _octaattt___________ee__ _on__________________66_________

---

Under review as a conference paper at ICLR 2021
Table 2: Automatic evaluation results on the test set. For all metrics, higher values indicate better results. Con and Ctx refer to contribution summary and context summary, respectively. Purity and Disentanglement are measaured on the pairs of contribution and context summaries.
Relevance
Purity
Disentanglement
Model
R-1
R-2
R-L
BS
P-1
P-2
D-1
D-2
D-L
DBS
Con
39.16
15.96
24.65
63.22
CC
2.77
3.69
52.95
72.18
69.12
33.62
Ctx
41.84
17.24
24.55
63.78
CC+INF
Con
38.92
15.95
24.65
62.94
2.75
3.68
53.68
71.97
68.46
34.09
Ctx
41.49
17.03
24.50
63.40
MH
Con
39.20
15.98
24.72
63.04
2.73
3.68
50.89
69.51
65.97
32.51
Ctx
41.67
17.23
24.65
63.77
Con
38.74
15,90
24.59
62.70
MH+INF
2.68
3.60
53.35
71.47
67.20
33.86
Cix
40.39
16.31
23.83
62.85
is not strong enough to focus on generating the appropriate content (first term in Eq clear that the MULTIHEAD model benefits more from the additional training objective.
It is also 2
6 ANALYSIS
6.1 QUALITATIVE ANALYSIS
To better understand the strengths and shortcomings of our models, we performed a qualitative study of model outputs. Table + shows example of generated summaries compared with the original abstract of the summarized article. Our model successfully separates the two generation modes and outputs coherent and easy follow summaries. The contribution summary clearly lists the novelties of the work, while the context summary introduces the task at hand and explains its importance. In comparison, the original abstract briefly touches on many aspects: the context, methods used, and contributions, but also offers details that are not of primary importance, such as the detailed about the simulation environment.
More generally, the described trends hold across summaries generated by our models. The model outputs are fluent, abstractive, offer good separation between modes, and are on topic. However, the factual correctness of summaries could not be assessed due to the highly specialized content and language the summarized articles. An artifact noticed a few instances of the inspected out- puts was leakage of contribution information into context summaries. Other examples of generated summaries are included in the Appendix
6.2 PER-DOMAIN PERFORMANCE
Taking advantage of the rich metadata associ- ated with the S2ORC corpus, we analyze the performance of models across the 10 most fre- quent scientific domains. Table shows the re- sults of contribution summarization using the CONTROL
Table 4: Relevance evaluation of contribution summaries for the top 10 domains generated us- ing the CONTROLCODE model. Performance on Medicine domain is paricularly low.
CODTRO CODI model. While ROUGE-1 scores oscillate around 40 points for most aca- demic fields, the results indicate that summa- rizing documents from the Medical domain is particularly difficult, with models scoring about 7 points below average. Manual inspection of instances with low scores (R-1 < 20), exposed that contribution summaries in the Medical do- main are highly quantitative (e.g. "Among these treated .. retinopathy was noted in X%"". While other domains such as Biology also suffer from the same phenomenon, low-scoring quantitative summaries were 1.9 times more frequent in Medicine than in Biology. An investigation into the domain distribution in our dataset (Appendix) revealed
R-2
Biology
40.63
17.01
25. 59
64,23
Medicine
33.97
13.08
21.73
61.75
Mathematics
40.13
15.56
24.42
61.58
Computer
43.54
16.41
63.43
None
40.31
18.14
26.68
64,00
Psychology
39.51
15.56
24.34
62.95
Physics
40.09
15.85
24.89
62.10
Chemissry
40.44
17.77
26.
63.93
Economics
39.56
Materials science
42,52
18.96
27.57
65.25
remaining models exhibit same pattem.

---

Under review as a conference paper at ICLR 2021
Table 3: Generated samples compared with the original and generated abstracts of the associated paper. The second rows shows the output decoded from DistiBBART fine-tuned on our dataset, the third rows shows the outputs from CONTROLCODE model. Our model successfully generates disentangled content, thus making it easier to follow than the abstract.
Original Abstract: Energy optimization buildings by controlling Heeating Ventilation Air Conditioning (IVAC) system is being researched extensively. In this paper, a model-fres actor-critic Reinforcement T Leaming (RL. controller is designed using variant of artificial recurrent neural network called Long-Short-ooe Shoor-Tootm Memory (LSTM) networks. Optimization of thermal comfort alongsido eaergy consumption the goal in tuning this RL controller. The test platform, office space, is designed using SketchUp. Using OpenStudio, the HVAC system is installed in the office. The control schemes (ideal thermal comfort, a traditional control and the RL control) are implemented in MATLAB, Using the Ruilding Control Virtual Test Red (BCVTB), the contrl of the thermosstt schedule during each sample time is implemented for the office in EnergyPlus alongside local weather data. Results from training and validation indicate that the RL controller impoves thermal comfort by an average 15% and energy efficiency by an average 2.5% as compared to other strategies mentioned.
Generated Abstract: Despite the advances in reseurch on IVAC cuntrol algerithms, must lield equipment is controlled using classical methods that include hysteresis/o/offf and Propotional Integral and Derivative (PID) controllers, These classical methods do not perform optimally. The high thermal inertia of buildings induces large time delays in the building dynamics, which cannot be handled efficiently by the simple onfoff controllers. However, duc the high non-lincarity in building dynamics coupled with uncertainties such weather, energy pricing, etc., these PID controllers require extensive retuning or aulo-tuning capabilitiss which incremees the difficully and complexity of the control problem.
work, we introduce novel control algorithmss From controls perspective, reinfervement learning algorithms can be considered direct adoptive optimal control. Like optimal control. reinforcement training al algorithms minimize the cumulative sum of costs over a time horizon. Unlike traditional optimization algorithms can learn optimal control actions
a branch of machine learning called reinforcement learning.
Contribution: In this work, we introduze novel comtrol algorithms from a branch of machine learning called reinffrrement leurning. our current approach, the impetus is thermostat control. Instead of traditional onloff heating and cooling control, reinforcement learning is utilized to set this schedule to obtain improved Prodicted Mcan Vore (PMV)-based thermal comfort at an optimal energy expenditure. Henoe, thermostats schodule computed using RL controller. The results show that the Q-learning algorithm can learn adapt lime-varying and monlinear syslem dynamics without explicit identification of the plant model in both systems and controls.
Context: The Heating. Ventilation and Air Conditionin (HVAC) systems account for up the hupes of moving loward a greener. more energy efficient future, significant improvement in energy efficiency is needed to achieve this goal. Despite the advances research HVAC contmil algorihmms, most field equipment contmolled using classical methods hysteresis/on/off and Proportional Integral and Derivative controllers. However, due to the high nonlincarity in building dynamics coupled with uncertainties such as weather, eaergy pricing, etc.. these PID controllers roquire exteasive retuning or auto-tuning capabilities, which increases The high thermal inertia of buildings induces larg time delays the building dynamics,
of total building energy demand.
include
the difficully and complexity of the control problem. probree. which camnot be handled efficiently by the simple onfolT controllers.
that Biology and Medicine are the two best represented fields in the corpus, with Biology having over twice as many examples. We hypothesize that the poor performance of models stems from the fact that generating such quantitative summaries requires deeper, domain -spccfic understanding of the source document and the available in-domain training data is insufficient to achieve that goal.
6.3 HUMAN EVALUATION OF USEFULNESS
To assess the usefulness of the newly introduced task to the research community, we conducted a human study involving expert annotators. 'he study aimed to compare disentangled papers sum- maries with traditional, abstract- based summaries in hypothetical paper reviewing setting. Judges were shown both types of summaries side by side and asked to pick one which would be more help- ful for conducting the paper review. We show a screen shot of the annotation user interface in Fig 3 Abstract-based summaries were generated by model with a configuration identical to the models previously introduced in this work, trained to generate full abstracts using the same training corpus. Annotators that participated in this study hold graduate degrees in technical fields and are active in the research community, however, they were not involved or familiar with this work prior to this experiment. The study used 100 examples, out of which 50 were decoded on the test split of the adapted S20RC dataset, while the other 50 were generated in a zero-shot fashion from articles in the CORD dataset (Wang et al., 2020), recently introduced collection of papers related to COVID-19. The inter-annotator agreement measured by Fleiss' Kappa was 0.41 and 0.33 for the S20RC and CORD datasets, respectively, Results in Table show the proportion of all examples where the annotators preferred the disentangled summaries over the generated abstracts. The numbers indicate strong preference from the judges for disentangled summaries, in the case of both S2ORC and CORD examples. The values on CORD samples are slightly higher than those S2ORC; we suspect this being due to the fact that the annotators were
Table Usefulness of disentangled summaries in percentage, egg., Annotator 1 (Al) chose the disen- tangled summaries 82% out of all the samples from S2ORC.
Dataset
AVG.
S2ORC CORD
82%
78%
70%
77%
88%
76%
78%
81%
8

---

Under review as a conference paper at ICLR 2021
Carti
Contert
Harlwaus ard tee tanatoo iocul spnc de The
hale ditinr in cirs lined sarac, tha piaee, saric U.
pester and s0 laen bemn tholg leancis
legter.
They
wntr oraut of the non cai ferseoo aie laper
Lycaa kin caniec murooo
zona cas andi
Morel danciee thone
what
thet hast
bntkein afforne speiecs withen clnglt drigg zonie inpion
be bade da onuumaoe giris tayi sostir 'N livk ctid, zecn.
sas teppli uiis oo maluues our of
thainitt Vesportiinn
we ue selalle bee live axsuanilly fu eine and
Veuud eillind letor and unlo (7160)
wederns anvallenee
ZAN
rooe 1o leeces in the limnal Donnattee for
Whish of anakiart tuut you courrrder new hertt eviegn?
Scome
Figure 3: The annotation interface, Summaries indicated with A and are disentangled summaries and generated abstract, respectively.
less familiar with the topics described in Covid-related publications and would require more help to review such articles.
6.4 EVALUATION AGAINST GOLD ANNOTATIONS
As discussed in Section 3.1 contribution and context labels are assigned automatically using a data- driven classifier, which could introduce errors in the process. Therefore, we created gold standard evaluation set by manually annotating 100 samples from the test set and report the evaluation results in Table G A sharp drop in ROUGE scores for the context summaries is caused by some examples receiving zero scores for generating context summaries when the manual annotators judged that there are existent. The overall trend of CONTROLCODE model outpeforming MULTIHEAD model is still observed in the evaluation. More importantly, a reverse tendency is noticeable when the two models are trained with the informativeness objective. Specifically, the MULTIHEAD model showed significant improvement in terms of novelty and disentanglement.
Table 6: Automatic evaluation results on 100 samples from the test set manually annotated for contributions. For all metries, higher values indicate better results.
Relevance
Purity
Disentanglement
Model
R-1
R-2
R-L
BS
P-1
P-2
D-1
D-2
D-L
DBS
Con
39.37
15.86
24.73
63.28
CC
Ctx
30.59
11.22
19.08
55.76
2.30
3.22
52.81
71.52
68.36
33.05
CC+INF
Con
38.38
15.21
23.47
62.59
2.17
3.10
52.49
69.64
66.60
32.76
Ctx
30.14
11.10
19.00
55.55
MH
Con
38.63
15.53
24.68
62.84
2.21
3.13
49.62
67.45
64.43
31.39
Ctx
29.82
10.61
18.51
55.24
MH+INF
Con Ctx
15.75
24.77
2.26
3.13
51.56
68.57
64.97
32.35
29.14
10.25
18.48
54.92
CONCLUSIONS
In this paper. we propose disentangled paper summarization, a new task in scientific paper sum- marizing where models simultaneously generate contribution and context summaries. With the task in mind, we introduced large-scale dataset with fine- fnn-grained reference summaries and rich meta- data. Along with the data, we introduced three abstractive baseline approaches to solving the new task and thoroughly assessed them using a comprehensive evaluation protocol design for the task at hand. Through human studies involving expert annotators with motivated the usefulness of the task in comparison to the current scientific paper summarization setting. Together with this paper, we release the code, trained model checkpoints, and data preprocessing scripts to support future work in this direction. We hope this work will positively contribute to creating Al-based tools for assisting scientists in the research process.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3615 3620, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: (0.18653/v1/D199 1/D11-13711 URL http:///wwwww.....c.c.....c//oo/
www http:://wwwwwwlllwbo/o//////////
Praveen Bysani. Detecting novelty in the context of progressive summarization. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 2, 2010, Los Angeles, California, USA Proceedings of the Stu- dent Research Workshop, pp. 13-18. The Association for Computational Linguistics, 2010. URL 30033
https: www. ac web org/
Yi Chang, Jiliang Tang. Dawei Yin, Makoto Yamada, and Yan Liu. Timeline summarization from social media with life cycle models. In IJCAL, pp. 3698-3704, 2016.
Arman Cohan and Nazli Goharian. Scientific article summarization using citation-contes. and ar- ticle's discourse structure. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 390-400. Association for Computational Linguistics, Septem- ber 2015. doi: (0.18653/11/115-11 1045. URL https: //www/wwwwww.// acwweb..ww.................. org wwt.oorgg/atooggggoooo 1045
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-awan attention model for abstractive summarization of long docu- ments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), pp. 615 621, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 00.18633/v//N18 2097. URL https: www. wwebbbb.... htwpbboogggattaaaeaaeeeeeeeeeeee httg//athoooooyylolooe
Ed Collins, Isabelle Augenstein, and Sebastian Riedel. A supervised approach to extractive sum- marisation of scientific papers. In Proceedings of the 2 Conference on Computational Natural Language Learning (CoNLL 2017), pp. 195-205, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/K17710 Exc//anntooloog//11-11022
URL https: www awwwb....
John M. Conroy, Judith D. Schlesinger, and Dianne P. O'Larry Nouveau-ROUGE A novelty metric for update summarization. Computational Linguistics, 37(1):1-8, 2011. doi: 10.1 162//o/lll.... a.
00033. URL httt::ttt///vitt acl wwwpeeb.rrrgaaathoooogoooooe
Hoa Trang Dang and Karolina Owezarzak. Overview of the tac 2008 update summarization task.
Jean-Yves Delort and Enrique Alfonseca. Dualsum: a topic-model based approach for update sum- marization. In Walter Daelemans, Mirella Lapata, and Lluís Mårquez (eds.), EACL 2012, 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012, pp. 214-223. The Association for Computer Linguistics, 2012. URL
https: www. wwwwww.www........ org/ anthology/EIee EE12-11222222
Alexander R Fabbri, Wojciech Kryšcinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626, 2020.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. In Proceed- ings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 15-54, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10613863/v NW18-2706 URL https: www acl www..oogganthhooooogooooooooo
8-2706
Han Guo, Ramakanth Pasumuru, and Mohit Bansal. Soft layer-specific multi-task summarization with entailment and question generation. In Iryna Gurevych and Yusuke Miyao (eds.), Pro-
ceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 687-697. As- sociation for Computational Linguistics, 2018. doi: 10.18653/v1/P18. 1064. URL hetps: 7 7/www. c web. hwwpg//nthoolloo http://anntooooyol--ooaee 10627
10

---

Under review as a conference paper at ICLR 2021
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL
A Conditional Transformer Language Model for Controllable Generation. arXiv preprint arXiv:1909.000 00858, 2019.
Svetlana Kiritchenko and Saif Mohammad. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 465-470 Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 00.18653/v1/P17-2000 URI.
https: www. ac web. orabboorggatthhoooooooo www.annwhwooooooooooeoeee ogy/ ogy/P17-2074
Guy Lev, Michal Shmueli-Scheuer Jonathan Herzig, Achiya Jerbi, and David Konopnicki. Talk- Summ: A dataset and scalable annotation method for scientific paper summarization based conference talks. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pp. 2125-2131, Florence, Italy, July 2019. Association for Computational Linguisties. URL https: www wwwwwb.ww...........
1204
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sqquence-to-sqquene pre- training for natural language generation, translation, and comprchension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguisties. doi: 10. 2t5533/1/202..cl-maaii URL http:s://ww..alleb..roro/////// http./////..tcooooo///2022220 acl main. 703
acl www.wo..........................
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis-
tics. URL https: : www. aclweb. org/ antholoooooooc 1013
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNE pp. 3730- 3740, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v1/DD
URL https: /tt::://wwwwww.. aclweb. wtt. httg///totoooooo/y-1159999
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The seman- tic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pp. acl-main.4444... URL URLhttps:: /ttp://wwwwwoo wwwwwa.wwwb.oo orq/
Online, July 2020. Association for Computa-
tional Linguistics. doi: m.4..447.74.... Sthhology/22200
Minh-T Thang Luong, Quoe V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Leurning Representutions, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016. http: http:///xxxvv.r.oo/o /tabs//511. 06614
Sebastian Martschat and Katja Markert. A temporally sensitive submodularity framework for time- line summarization. In Anna Korhonen and Ivan Titov (eds.). Proceedings of the 22nd Confer- ence on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 November 1, 2018, pp. 230 240. Association for Computational Linguistics, 2018. doi: 10. 10066//////188.... 1023. URL https: doi wwtoo..org/// 11t.:/.......10....1111111111... wwt..//1//111-11111
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018. Conference Track Proceedings. OpenReview.net, 2018. URL
net. wtw.ee/wwwwwteeeeettett....aamaa fow.ummddc.kc11@0000
Maxime Peyrard. A simple theoretical model of importance for summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1059-1073, Flo- rence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18655////199-110.
URL https:: / www. a e web. www..oogg/nnhooooggeeee.ee 1-100
Vahed Qazvinian and Dragomir R. Radev. Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguisties

---

Under review as a conference paper at ICLR 2021
(Coling 2008), pp. 689 696, Manchester, UK, August 2008. Coling 2008 Organizing Committee. URL https: www aww.eb. www.eaeww....r.aaaaaoaoo///----.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, 0tbs/1910.1000. 2019. htLpttt::/// /ttp::://arxvv arxiv. wwwpziiiv.rr/..oo.ooo abs 1910.10683
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. URL http://
arxiv.org/ abs /1910 0111000111000
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1..18663//1/11-10999 URL https:
7 7/www, ac
www.aa..web.rgg//oooo wwt. htt.a//www..o.rraaaaaoooooooooo.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
URL http:://axxiiii org/ http//aas//1/.//////// 03766
Lucy Lu Wang. Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. Cord-19: The covid-19 open research dataset. arXiv preprint arXiv:2004. 10706, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, 1b0/191, 2011..
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander Fabbri, Irene Li, Dan Friedman, and Dragomir Radev. ScisummNet: A large annotated corpus and content-impact models for sci- entific paper summarization with citation networks. In Proceedings of AAAI 2019, 2019. URL
https: http:://wwwaaaa...... / index.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences, 2020.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference Learning Representations, 2020. URL htttss://pppeeeeeeeeeee
net fortmm? wwwemmmmeedseeeeeeeeeneeee
Yao Zhao, Mohammad Salch, and Peter J Liu. Seal: Segment-wise extractive-abstractive long-form text summarization, arXiv preprint arXiv:2006. 10213, 2020.
12


Reviews:
[{"review_id": "g83rl7dHoct", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a dataset based on S2ORC with additional annotations on the abstracts in terms of contribution and context. Based on this dataset, this paper also adopts two baseline models from prior work with a training strategy (also defined by prior work) to demonstrate the task of summarizing scientific literature from two aspects. \n\nOverall, I was confused about the contribution of this work. Based on my understanding it could either be presenting a newly annotated dataset or demonstrate this task using some baseline models. However, neither of them has enough evidence to convince me that this is a solid work. \n\nAbout the data, I have a few questions about the annotation procedure:\n\n1) about data curation, if there are more than 20 incoming citations, the proposed method suggests to keep only the top 20 papers based on their citations. However, how do we know these 20 papers are the most relevant to a specific work? \n\n2) I had trouble to understand the value of \"silver\" standard references. First of all, the classification accuracy is only 86.3%, without further guarantee, we have no idea about the annotation quality. Besides, if the major contribution of this work is about the dataset, then manually annotating 400 abstracts really did not sound like enough contribution.\n\nMy confusion about the contribution of this work mainly came from section 5 and 6. Particularly, in section 6, the line \"to better understand the strengths and shortcomings of our models\" reminds me that maybe the contribution of this work is about those models. Another confusion is that, if this work is about the dataset, then I really think it is necessary to analyze the annotation quality, instead of the performance of existing models. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "OjLgu4xSwcT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks to the authors for the hard work on this paper. It starts with an interesting and promising premise, and the human evaluation suggests the authors have really hit on something. I have given it a rating of 4 as I feel that this work needs more analysis before it is ready for publication, but I certainly hope to see it published in a more thorough form sometime soon.\n\nThe single biggest concern I have is the silver data that is described in \"Reference Generation\". The authors manually annotated 400 abstracts, and classified each sentence in each abstract into either \"contribution-related\" or \"context-related\". They trained a classifier, and obtained 86% accuracy on a held-out dataset. This classifier is then both their source of training data *and* ground truth for everything except the human evaluation. Despite being the foundation of the paper, the section about \"Reference Generation\" is short and lacks the kind of details I believe it needs for the paper to be truly convincing. Here is a list of some concerns and questions related to this section:\n\n* 86.3% accuracy is not a revealing metric because we don't know how unbalanced the test set is. Please report area under the ROC curve instead.\n* I assume you chose 0.5 as the threshold cut-off for the final context/contribution classifier. In Section 5.3 you report that \"context summaries include article contribution information.\" This may be related to the kinds of errors made by the original classifier. What happens when you change the operating point?\n* 400 samples for training is relatively small. How representative are they of the different scientific fields in Table 4? Abstracts in biomedicine are very different than abstracts in philosophy. What is the held-out AUC per field? \n* What if you add another ~50 samples to the 400 samples in the training data? Does your held-out AUC go up? This would help you assess if 400 documents is enough.\n* How much inter-annotator agreement is there for this task? How many annotators did you use per document?\n* Let's say the abstract is composed of a series of sentences: ABCDEFG, and the \"contributions\" are BDFG and \"context\" are ACE. These are non-contiguous passages, and are likely less fluent than the original abstract. How does this reduction in fluency affect your summarization algorithms? Is there a reduction in fluency in the generated passages as well?\n\nOther questions/concerns:\n* In data curation, the number of incoming/outgoing citations was limited to 20. Why?\n* If I was to imagine a use for a real system based on this work, it would be applied to academic papers shortly after publication. This means that in a production setting, there will be *no* incoming citations. Does your algorithm still work in this case?\n* As far as I understand, you used a distilBART model to summarize the entire full text of the paper, without the abstract, but including the title. Is this correct? If so, please clarify exactly what the input to the model is. How do you deal with long documents using a model that has a much smaller maximum input?\n* In Eq (2) - how do you compute $p(.| D)$, $p(.| C_O)$, $p(.| C_I)$?\n* It's hard to interpret the numbers in Table 2 without some sort of baseline to compare against. Please consider adding a semi-naive baseline of some sort.\n* Please provide a bit of information about how you set the various hyper-parameters in section 5.2. By cross-validation? Hand-tuning? \n* In section 6.1 you state that \"the factual correctness ... could not be assessed\". For scientific text factual correctness of any summarization system is essentially a requirement. Given that you have access to expert evaluation, I think it's critical that some evaluation of factual correctness is done.\n* For the human evaluation - how many annotators did you use per document? What was the inter-annotator agreement?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OAWw-qgpJKf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discussed the task of generating scientific paper summaries along two axes: contribution (ie. novelty) and context/background. The point is two disentangle these two, since often different readers would be interested in only one of these. While I like the idea of this paper, I think it has the following issues:\n\n- Evaluation: There is no evaluation against a manually annotated gold standard. The references used in section 5 are based on the output of a classifier trained on manual annotations, which itself was not evaluated against manual annotations either. Thus I am rather concerned about what we can learn from the results presented. Section 6 presents some observations in 6.1. but even though they are useful on can't read much in them. And I am not convinced that the lack of expert annotators means that one cannot assess hallucinations; if one can assess whether the novelty is extracted correctly (in the same section), then hallucinations should be possible to assess. In section 6.3 the human evaluation is essentially comparing the model proposed (unclear which of the variants though) against the output of a summarization model that doesn't separate novelty from context. However we don't know how good this summarization model is, let alone that asking about usefulness only conflates a number of aspects that matter in a summary, such as informativeness, fluency, factual correctness, readability, etc. Overall, I think the evaluation is not adequate for a paper at a top conference.\n\n- Modelling: Why not experiment with extractive summarization? Abstracts often overlap with the main paper, and you have a sentence level classifier anyway. And the hallucination wouldn't be an issue then. Also there is previous work on modelling citation function: https://www.cl.cam.ac.uk/~sht25/papers/emnlp06.pdf which should be acknowledged but also could be useful in creating better data and models. Similarly, there is work on scientific article zoning: http://oro.open.ac.uk/58880/\n\n- I am not sure I follow the second branch in equation 2: why should context surprise the articles citing the one being summarized? They would share the context quite often. Also I don't see the connection to the disentanglement.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BIgnVXPEQk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces disentangled paper representation, which tries to generate paper summaries that explicitly differentiate between the contributions and the context of the work.  I like the general direction and the approaches, but I feel that the experiments and analysis need more work.\n\nStrengths:\nI generally think decomposing scientific article summaries into two pieces, contextual descriptions vs. contributions, would be helpful.  The two methods evaluated here (one using control codes, the other using a multi-head architecture) seem reasonable.  Also, the informativeness approach (equation 2) based on inbound vs. outbound citations is well-motivated and interesting.  The paper uses a reasonable set of automated metrics on a silver data source, and I found those experiments to be relatively well-done although ultimately the results are somewhat inconclusive, in that there were not strong differences between different techniques.\n\nWeaknesses:\nBecause automated metrics are always highly imperfect, human evaluation of the summaries is important.  Unfortunately I think the human evaluation in this paper leaves too many questions unanswered.  First, some details are missing -- what was the inter-annotator agreement, and what exactly was the prompt given to the annotators?  Further, because the experiment tests standard output (regular abstract format) against a newer thing (with context and contribution structure), it seems possible that annotators were biased to prefer the disentangled summaries just because they’ve novel and/or suspected to be the target of the study.  Some discussion of this bias is necessary.  Also, more analysis of why the annotators preferred the disentangled summaries would help alleviate the bias concerns (ideally, the paper would perform a task-based evaluation showing that the disentangled summaries help users more rapidly or accurately browse the literature).\n\nAdditionally, the truthfulness of the generation seems like a potential concern that is not discussed.  For example, the contextual output in Table 3 includes a lot of general facts about the domain that are different from the ones shown for the given abstract.  Are all of these statements factual, i.e. from the source documents, or does the abstractive model “make things up”?  The evaluation does not directly address this question.\n\nSuggestions/questions:\nI’d like to see an analysis of how important it is to have the inbound citations---these will not exist for the most recent papers, which are arguably the most valuable ones to summarize.  While the paper does ablate the informativeness measure based on the citations it does not experiment with removing the inbound citation text from the input, I would like to see that.\n\nThe ablation studies (Table 2) suggest that the informativeness objective does not particularly help for the control codes model, even with disentanglement.  But multi-head is helped more by informativeness (or more precisely, multi-head suffers with out it).  The paper notes this effect, but can you explain more why MH and CC differ in this way?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attempts at generating two types of summaries for scientific papers: summary of contribution and summary of background context. Most reviewers appreciated the motivation and found this research to be quite interesting and useful, however all reviewers had concerns regarding both execution/presentation of the ideas. While authors try to address some of the concerns, there are many clarifying questions and points raised by reviewers. Addressing all these points requires rather a major revision. Therefore the paper is not quite ready yet and would benefit from another iteration.
Decision: Reject