Citance-Contextualized Summarization of Scientific Papers
Authors: Shahbaz Syed
Ahmad Dawar Hakimi
Khalid Al Khatib
Martin Potthast
Venue: EMNLP 2023 Findings (2023)
Keywords: Summarization
Scholarly Document Processing
Scientific Papers
Large Language Models
TL;DR: We propose contextualized summarization of scientific papers grounded in the citation contexts.

Abstract: Current approaches to automatic summarization of scientific papers generate informative summaries in the form of abstracts. However, abstracts are not intended to show the relationship between a paper and the references cited in it. We propose a new contextualized summarization approach that can generate an informative summary conditioned on a given sentence containing the citation of a reference (a so-called ``citance''). This summary outlines content of the cited paper relevant to the citation location. Thus, our approach extracts and models the citances of a paper, retrieves relevant passages from cited papers, and generates abstractive summaries tailored to each citance. We evaluate our approach using **Webis-Context-SciSumm-2023**, a new dataset containing 540K computer science papers and 4.6M citances therein.

Summarization of Scientific Papers
Shahbaz Syed 1
Ahmad Dawar Hakimi
Khalid Al-Khatib
Martin Potthast 1$
Leippig University
UUniversity of Groningen
ScaDS.AA
mhabbaz.syedeuii-eiiie
Abstract
Paper Excerpt
Contextualized Summarization
Current approaches to automatic summariza- tion scientific papers generate informative summaries in the form of abstracts. However, are supported as lozz functions, in
ports miniumm risk training (MRT)
[-.] Addisonally, Nomatus sup
Chen, C. Cherr
Cted Paper
(Shen 2016) optimize arbitrery, fundtio Varius MT
Context
loval
abstracts are not intended to show the rela- tionship between paper and the references cited in it. We propose new contextualized summarization approach that can generate an informative summary conditioned on given sentence containing the citation of reference (a so-called "citance"). This summary out- lines the content of the cited paper relevant to the citation location. Thus, our approach extracts and models the citances of paper, retrieves relevant passages from cited papers, and generates abstractive summaries tailored each citance. We evaluate our approach us- ing
sluding smoothed deeiteeceee level
ELFU (Chen and Cherry, 2014]
Citation
Pletreval
METEOR (Denkowsll
2011), (Stnnjovic ma ant 2014), and any interpola:
Ciannee
Retrieval model
fon of impemmeteed metiics To stacilzee training Nematuss
Flanking
stopping based
Context
cross entropy
lunction defined by the [-]
Bibliography [-] Bosing Chen Coln 2016, Sylenmit Comparion Smpothing- Techniquss for Setiince Lead R. In Proceeings the Nirth Workshop Staieicall Machine Transationn pages 362 367, Boltmon. Maryland
Summariaaton Summay
Language model
Reference
We compare varius smooth- reghecriqussse ser ventence-le-- el Fl Alhogghtther mebres are bxttre at maniing i tnarsda fion quality SLFU emains the pondaadlollssskkmmmett
USA. Assocation for Computational Linguetic.
new dataset containing 540K computer science pa- pers and 4.6M citances therein."
Contextualized summarization at glance: Given citation, approach uses the citance and its contexts as a queries to retrieve relevant content from the cited paper. A contextualized summary is then gen- erated using large language model.
Figure I:
1 Introduction
The original task of automatic summarization has been the abstracting of scientific papers, one of the first tasks studied in computer science (Luhhn, 1958; Baxendale, 1958). Automatically generated abstracts were used to create "index volumes" for specific scientific areas to help researchers access the growing number of publications. Nowadays, paper authors usually write abstracts themselves. However, author-generated abstracts often provide incomplete or biased coverage of scientific papers (Elkiss et al., 2008). As a result, the purpose of automatic paper summarization has evolved to gen- erate more informative summaries, often using ab- stractive summarization approaches (Cohan et al., 2018; Cachola et al., 2020; Mao et al., 2022).
abstracts provide a concise and general overview of the cited papers, they do not meet the information need of readers trying to understand the relevance of a paper in the context of its citation. The gener- ated abstracts are not adapted to the citation context, leaving the reader to consult the cited work directly. In this paper, investigate the suitability of contextualized summaries that are specifically tai- lored individual citation contexts compared generic abstracts. We propose new approach to generating contextualized summaries by oper- ationalizing citation contexts. Figure 1 illustrates our approach, which consists of three steps (Sec- tion 3): (1) Extraction and modeling of the sentence containing citation (the citance), and its contexts from the citing document, (2) retrieval of relevant content from the citing paper using queries based on these citance contexts, and (3) generation of
A practical application of generating summaries is augment reading papers. For example, CITE- (2022) is part of Allen Al's Semantic Reader (Lo et al., 2023) and shows on demand the abstracts as summaries for the cited papers in paper being read. While these
READ by Rachatasumrit
abstractive, citance-contextualized summaries the citing paper. To address this novel task, we create WEBIS-CONTEXTT-SCSSUMM
Equal contribution.
Code:
(Sec-
Data: http:://cemmbboooggdd/////////22 10031002

---

tion 4), large-scale, high-quality corpus consist- ing of 540K computer science papers and 4.6M ci- tances, In an extensive comparative evaluation us- ing our corpus, we explore different variants of our approach in comparison to the abstracts of the cited papers (Section 5). We find that while abstracts have a slight advantage in terms of coverage and focus, when citance fails to align with the central theme of the cited paper, contextualized summaries prove to be a more favorable alternative to abstracts. Examples of contextualized summaries from our approach along with the abstracts are shown in Appendix D.
and from the authors of papers. Control codes and multitask learning were used generate sum- maries; the model also used the paper titles as an additional training signal.
Comprehensive summarization LongSumm is task, which aims to generate comprehensive sum- maries of about 600 words, providing sufficient information in lieu of reading the target paper and overcoming the limitations of abstracts and TL;DR summaries Chandrasekaran et al. (2020). The LongSumm corpus comprises 2236 papers with abstractive and extractive summaries. Sotudeh et al. (2021) created two corpora from arXiv and PubMed consisting of 11,149 and 88,035 paper- summary pairs, respectively. To guide the genera- tion of the long summaries, Sotudeh and Goharian (2022) expanded abstracts with sentences from in- troduction, overview, and motivation sections.
2 Related Work
In this section, we review the literature on generic and citation-based summarization of scientific doc- uments, including different types of generated sum- maries, and approaches that summarize the target paper-based on its citance (contexts), respectively.
2.2 Citation-based Summarization
In citation- based summarization, citances from the source paper are used as queries to extract relevant content from the target paper, and to generate a summary. Qazvinian and Radev (2008) analyzed the citation network of target papers and collected citances from different sources. These citances were clustered, and the central sentences identi- fied as extractive summaries. Mei and Zhai (2008) focused on impact sentence-based summaries, re- flecting the authority and proximity of the citations in paper collection. The impact of the target paper on related work is determined using citations from the source papers. To improve readability and co- herence, Abu-Jbara and Radev (2011) introduced preprocessing step to filter out non-relevant text segments. Then, an extraction phase is conducted to select important sentences from sections such as background, problem statement, method, results, and limitations. In a post-processing step, the over- all readability of generated summaries is improved, replacing pronouns and resolving co-references. Closely related to our work, Cohan and Gohar- ian (2015) used citance contexts, defined the text passages of the target paper that reflect the citation of the source paper. To summarize the target paper, they first collected multiple citance contexts, con- structing a graph based on their intra- intra-connyttivity based on the cosine similarity of tf-idf vectors.
2.1 Generic Summarization
Generic summarization approaches for scientific papers are based on various ground-truth sum- maries, including abstracts (Luhn, 1958; Cohan al., 2018), author-written highlights (Collins al., 2017), author-written promotional blurbs (Chen et al., 2020), and condensed versions sum- maries from peer reviews (Cachola et al., 2020).
Abstract-based summarization Collins et al. (2017) proposed supervised model for extractive summarization trained on 10,148 computer science papers. The model uses an LSTM-based neural encoder with lexical features classify summary- worthy sentences, where auttor-written highlights and abstracts serve as references. Cohan et al. (2018) presented a discourse-aware attention model for the abstractive summarization of scientific pa- pers from the arXiv and PubMed collections. A hierarchical encoder integrates section information generate coherent summaries. Gupta al. (2021) investigated pre-training and tuning BERT-based models for extractive summarization.
TL;DR summarization Ultra-short indicative TL:DR summaries are concise, typically one or two sentences short, and aim to highlight the key findings of paper. Cachola et al. (2020) developed the SCITLDR corpus, consisting of 3.2K papers with accompanying manually written TL:DR sum- maries (15-25 words) sourced from peer reviews
The sentences in this graph were ranked by their importance (number of connections). The retrieved sentences combined with discourse information

---

from the target paper generate an informative summary. Cohan and Goharian (2017) have further improved this model using word embeddings and domain knowledge to enhance the citation contexts.
3.2 Ciancce-ontesttualiied Retrieval
We use the above three citance contexts as queries for retrieval. In addition, we explore the use of ex- tracted keywords from each citance context to im- prove the queries (Carpineto and Romano, 2012) 2). For retrieval, use both shallow and dense trieval models (Section 4.2). We retrieve relevant content two levels of granularity: sentences and paragraphs. Specifically, we extract the top-5 rel- evant sentences and the top-2 relevant paragraphs from the cited paper. This enables the evaluation of which granularity is more suitable for the con- textualized summarization task.
Our work also focuses on contextualizing ci- tations using citance contexts, but differs signif- icantly: We identify different types of citance con- texts and use them to generate multiple contextu- ally relevant summaries for given citance. Instead of relying solely on the literal citance as a query, which represents only one type of citance context, wc employ multiple contexts for deriving queries.
Our corpus is the largest one with citance context-specific summaries of scientific papers, comprising about 540,000 papers and 4.6 million citances. In comparison, the CITESUM corpus of Mao al. (2022) comprises only 93,000 papers, where citance from the related work section of the source paper serves as an ultra-short summary of the target paper. Our corpus includes citances from all sections of the source paper and contains mul- tiple types of citance contexts, as well as multiple summaries for each context. As result, our corpus provides a comprehensive and diverse resource for studying the summarization of scientific papers. g
The top-5 most relevant sentences provide a broader coverage of the eited paper that includes information relevant to the citance. Conversely, the top-2 most relevant paragraphs provide higher degree of focus, where the summary sentences are interconnected. Therefore, we experiment with both granularities to investigate their effectiveness in our approach. Following the retrieval process, we perform a qualitative evaluation of the retrieved content. This evaluation helps us select the optimal combination of query and retrieval models for the subsequent summarization step (Section 5.1).
3 Contextualized Summarization
3.3 Citance-contextualizee Summarization
Our approach to contextualized summarization in- volves using multiple citance contexts in a source paper. In addition to the citance itself (a single sen- tence containing the citation), we consider several types of surrounding contexts. As illustrated in Figure 1, our approach involves three main steps: (I) extraction of citances, (2) retrieval of relevant content from the cited paper, and (3) generation abstractive summarics that are contextualized based on a citance.
After retrieving the relevant content from the cited paper, we use as input to the summarization model. This ensures that the generated summaries are thus contextualized to the citance and focus ex- clusively on the parts of the cited paper that are rele- vant to it. In our approach, we explore the effective- of large language models (LLMs) due to their strong multi-task capabilities (Bommasani et al., 2021). We use prompt-base based, Instruction-tunce models that can understand and execute natural lan- guage instructions from the user to accomplish a specific task. The flexibility and adaptability to different domains distinguishes our approach from domain-specife supervised methods.
3.1 Extraction of Citance-contexts
First, all citances that literally refer to other papers are extracted from given paper. We then con- sider two additional contexts for a citance. The first includes the sentences that immediately pre- cede and follow the citance. The second contains the two semantically most similar sentences of the citance within the same paragraph. This yields three citance contexts: (1) the citance itself, (2) the citance and its neighbors, and (3) the citance and semantically similar sentences. By considering these contexts, we aim to improve the retrieval of relevant content from the cited paper.
Since we have two granularities of input to the summary model (top-5 sentences and top-2 para- graphs), we design two prompts tailored to both. For the top-5 sentences, we use paraphrasing prompt that aims to transform the sentences into a coherent summary. For the top-2 paragraphs, we use an abstractive summarization prompt to gen- crate a coherent summary from them. For more details on the prompts, see Section 4.4.

---

Corpus
Size
Avg.
Length
Style
Type Multiple
LaySumm (Chandrasckaran al., 2020)
572
84 tokens
Layman-Summary
Abs.
SciSummNet (Yasunaga et al., 2019)
IK
151 words
Abstract
Ext.
X
TalkSumm (Lev al., 2019)
1.7K
965 words
Informative
Ext.
LongSumm (Chandrasekaran et al., 2020)
2.2K
779 words
Blog Post
Ext/Abs.
X
SciTLDR (Cachola et al., 2020)
3.2K
21 words
TL;DR
Abs.
FacetSom (Meng et al., 2021)
60K
words
Facet-oriented, Informative
Ext/Abs.
CiteSum (Mao et al., 2022)
93K 23 words
TL;DR
Abs.
X
CORD-SUM (Qi et al., 2022)
123K 223 words
Abstract
Ext.
PuhMed (Cohan al., 2018)
133K 203 words
Abstract
Abs.
ArXiv (Cohan et al., 2018)
215K 220 words
Abstract
Abs.
X
RSCSum (Chen al., 2020)
308K 29 words
Table of Contents
Abs.
ArXiv-Long (Sotudeh and Goharian, 2022)
11.1K
574 tokens
Extended Abstract
Ext.
X
PubMed-Long (Sotudeh and Goharian, 2022)
88K
403 tokens
Extended Abstract
Ext.
X
SEBIS-CONTEXT-SOOE
540K 117 tokens Ciaance-contextuuliize,,, Informative Abs.
Table I: Comparison of our WEBIS-CONTENT SCLAAKT--SSSUMM-----M
corpus with existing corpora for summarizing
scientific papers. The columns show the size of each corpus, the characteristics of their summaries, such as the average length in tokens or words, the target style, the type of summary (extractive or abstractive), and whether there are multiple summaries per document in the corpus. Our corpus provides unique combination multiple, abstractive,
and informative summaries of a cited paper. The average length of the summaries exceeds the 100 references produced for qualitative evaluation (Section 5).
4 Large-Scale Corpus for Contextualized Summarization of Scientific Papers
A
4.2 Citance-contexts and Retrieval Models
As described Section 3.1, we employ three types of citance contexts as queries to retrieve relevant content from a cited paper. The citance and the neighbors contexts are extracted directly. For the similar context, the contextual embeddings from SciBERT (Beltagy al., 2019) were used to iden- tify the two semantically most similar sentences of a citance using the cosine similarity. We also extracted keywords as queries from the contexts using KeyBERT (Grootendorst, 2020).
Previous datasets for summarizing scientific papers do not consider different types of citance contexts, nor do they evaluate multiple retrieval models for extracting relevant content (Section 2). Therefore, these datasets are not suitable for studying citance- contextualized summarization. To address this gap, we introduce MIBBIS-CONTEXTI UMC a new and large dataset created using our approach as described in Section 3.
As retrieval models, we used BM25 (Robertson et al., 1994y$ and the cosine similarity of SciBERT embeddings between the query (citance context) and the document (sentences paragraphs of the cited paper) contrast both shallow and dense trieval paradigms. The combination of the three types of queries (including keyword variants) and the two retrieval models resulted in a total of re- trieval setups, as shown in Table 3, along with their mean NDCG @ scores from our internal evalu- ation (Section 5.1). We indexed 151 million sen- tences and 40 million paragraphs to retrieve the top-5 sentences and the top-2 paragraphs, respec- tively, for each query. For the keyword queries, we used weighted aggregation fuse the individual
4.1 Data Source and Preprocessing
We used the publicly available Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020).² The corpus consists of 136 million scientific docu- ments, of which 12 million are available in full text. We focused subset of 870,000 documents from the field of computer science (Section 5.2). After documents without citations were removed, about 540,000 documents remained. We then extracted citances by identifying the sentences in each docu- ment containing citations. This resulted in a total of 4.6 million citances. Unlike Mao el al. (2022), who only considered citances from the Related Work section, we considered citances from all paper sec- tions, resulting in more diverse set.
from https//hugginggfuce.dd
allenai/scibert, scivncchh_caahhhcceaaa uncased
Wee used the Mark-BM25 toolkit (Brown, 2020) with default parameters for BM25 (k=1, 6=0.55).
We used the S2ORC dataset relcased on 2020-07-05.

---

Model Variant
Description
Paraphrasing Prompt
Alpaca 7B
LLaMA-7B model finetuned on 52K ples of self-instructed responses (Wang al., 2022).
### Instruction:
A chat between curious human and artificial intelligence assistant. The assistant knows how paraphrase scientific text and user will provide the scientific for the assistant paraphrase.
Vicuna 13B
LLaMA models finetuned on user-shared conversations collected from ShareGPT.'
LLaMA-CoT
LLaMA-30B model finetuned on chain-of- thought and logical deduction examples (Qingyi Si, 2023).
###ll Input:
Generate coherent paraphrased text the follow- ing scientific text: (inpur).
Falcon Instruct
Trained the RefinedWeb corpus (Penedo
### Output:
2023), derived through extensive fil- tering and deduplication of publicly avail- able web data.
Summar ization Prompt
GPT4
The latest version of the GPT model from OpenAI." We use it to bootstrup ground- truth summaries (Section 5.2).
### Instruction:
chat between a curious human and an artificial intelligence assistant. The assistant knows how to summarize scientific text and user will provide the scientific text for the assistant to summarize. Input:
Table 2: List of LLMs used for zero-shot abstractive summarization. See Appendix C for more details.
Generate a coherent summary for the following sci- entific text not more than 5 sentences: (input). ). ### Output:
Each ranking's s weight corresponded to a query's s cosine similarity to its citance; the resulting rankings were combined by weighted sum.
rankings:
Figure 2: Best prompts with instructions for paraphras- ing (the top-5 retrieved sentences) and summarizing (the top-2 retrieved paragraphs). We ensured that the summarics for both granularities were similar length by instructing the model not generate more than five sentences for the top-2 paragraphs.
4.3
Corpus Statisties
The compiled corpus consists of 537,155 computer science papers containing total of 4,619,552 ci- tances. On average, each paper contains 8.6 ci- tances. The mean length of citance is 31 tokens, and the median 27 tokens. In addition, the corpus contains 346,450 papers that have multiple citances to the same target paper, facilitating the study of contextualized summarization approaches. Table compares our corpus to other datasets.
and summarizing. The generated summaries for 10 examples from all models were evaluated manu- ally. Based on this evaluation, the best combination of instructions and prompt formulations for cach model was selected. Figure 2 shows the selected combination, and Appendix A gives more details.
4.4 Abstractive Summarization
Using the retrieved content from the cited paper, we used the prompt-based instruction-tunce LLMs listed in Table 2 for abstractive summarization of each citance. For the granularities of retrieved content (top-5 sentences and top-2 paragraphs), we generated separate summaries using the models in a zero-shot setting. For the top-5 sentences, we paraphrased them into coherent text since they al- ready served as extractive summaries. For the top-2 paragraphs, we performed abstractive summariza- tion. Throughout the tasks, we experimented with different instructions and prompt formulations.
Evaluation
We evaluated the retrieval step of our approach an internal evaluation, and the summarization step in an external one.
5.1 Content Retrieval Evaluation
The 12 retrieval setups shown in Table 3 were eval- uated using manual relevance assessments of the content retrieved from cited papers the corre- sponding citance context (query) of citing papers. Ten queries were used to retrieve the top-5 sen- tences of cited paper for each of the setups, resulting in total of 600 sentences. Sentence relevance was assessed on a graded seale: rele- vant, somewhal relevant, and non-relevant. Table 3 shows the results in terms of NDCG@5 5 (Järvelin and Kekäläinen, 2002). Based on them, we se- lected the similar context as query for BM25 and
4.4.1 Prompt Formulation
To generate text conditioned on specific instruc- tions, the above models require a tailored prompt. We conducted experiments with different instruc- tions and prompt formulations for paraphrasing

---

BM25 (Shallow)
SciBERT (Dense)
Model
BERTScore
ROUGE
Query
Mean 5
Query
Mean nDCG@5
R-1
R-2
R-L
citance
0,943
citance
0.943
top-2 paragraphs
similar
0.958
similar
0.918
similar-BM25
neighbors
0.898
neighbors
0.801
Alpaca
0.343
47.3
25.5
44.9
Falcon
0.401
48.2
27.1
45.0
citance-keywords 0.914
cirdsce-keywords
0.617
LLaMA-COT
0.448
53.0
31,9
50.5
similar-keyword 0.944
similar-keywors 0.650
Vicuna
0.465
58.7
35,4
55.8
neighbors-keywords 0.928
neighbors-keywonds
0.706
citance-SeiBERT
Alpaca
0.390
54.3
32.2
52.0
Table 3: Evaluation of 12 retrieval setups combina- tions of a shallow and dense retrieval model with ci- tance contexts as queries to extract relevant content from cited papers. We report mean nDCG@* @ for 600 rele- vance judgments. The best combination (in bold) has been selected for the summarization step.
Falcon
0.413
52.1
29.6
48.9
LLaMA-CoT
0.497
54.7
32.9
52.5
Vicuna
0.431
56.7
34.2
53.9
top-5 sentences
similar-BM25
Alpaca
0,616
56.2
35,4
54.8
Falcon
0,649
57.5
35.6
55.2
LLaMA-COT
0.707
61.2
38.6
60.0
Vicuna
0.551
57.2
34.3
54.9
the citance context as query for SciBERT as best se- tups for shallow and dense retrieval to evaluate the subsequent summarization step. The former uses the top-2 semantically most similar sentences to citance (along with the eitance itself) as query, while the latter uses the citance only.
ciiance-SeiBERT
Alpaca
0.595
56.6
34.7
55.1
Falcon
0.656
56.8
36,2
55.3
LLaMA-COT
0.748
62.9
40.6
60.9
Vicuna
0.607
58.8
36.0
56.6
Table Automatic evaluation summaries from LLMs grouped by two granularities: top-2 relevant para- graphs and top-5 relevant sentences from the cited pa- per. We report BERTScore (accuracy) and different ROUGE scores compared reference summaries from GPT4. For manual evaluation, we selected the best model from each setup based ROUGE overlap with the references: Vicuna (similar- (MMtt) and LLaMA- CoT (citance-SeiBERT) for the top-2 paragraphs and top-5 paragraphs, respectively.
5.2 Summarization Evaluation
The contextualized summaries of the models listed Section 4.4 were evaluated using both quanti- tative and qualitative methods. For quantitative evaluation, we used the ROUGE (Lin, 2004) and BERTScore (Zhang et ul., 2020) metrics. For qual- itative evaluation, we manually scored the top two models in terms of coverage and focus.
Evaluation Data Fifteen articles were selected from the ACL anthology, published between 2016 and 2020. We extracted 363 citances from these articles and randomly selected 25 of them. Using the full texts of the cited papers and the top two retrieval models from Table 3, the top-5 sentences retrieved, resulting
best in summarizing the top-2 paragraphs, while LLaMA-CoT performs best paraphrasing the top-5 sentences as summaries. Moreover, also achieves the highest BERTScore in the top-2 para- graph setting, Therefore, we evaluated it manually for coverage and focus.
and the t paragraphs in total of 100 texts. To create the ground- truth reference summaries, used GPT4 (Bubeck et al., 2023) in a zero-shot setting to paraphrase/summa- rize these texts using the prompts shown in Fig- ure 2. Each summary was then manually reviewed to ensure accuracy and to rule out hallucinations or factual errors. Our set of references consists of 100 summaries (= 25 citances x 2 retrieval models 2 summary types).
Human Evaluation Three domain experts were recruited, including two students and one post-doc, to evaluate the usefulness of the summaries. The annotators were asked to rate the summaries on the two criteria coverage and focus. Ratings were on 5 5-point Likert-scale with indicating the worst and 5 the best rating. Coverage reflects how well the summary captures the essential information from the cited paper that is relevant to a particu- lar citance, whereas focus refers to the coherence and cohesion of the sentences in the summary. A total of 125 summaries of 25 cited papers were evaluated. Each sample consisted of the citance (and its context) displayed on the left-hand side
The reference summaries were used to automatically evaluate the generated contextualized summaries. Table 4 shows the re- sults. According to ROUGE, Vicuna performs
Automatic Evaluation

---

G-EVal Instructions for Automatic Evaluation of Coverage, Focus, and Relevance
You scientist who is currently reading a paper. While reading the paper, you see citation another puper that you want follow. You also given summary of the corresponding ciled paper. Your task is rate this summary on [overaae/fncuu/rrlevance keep this document open while reviewing, and refer to needed.
Please make sure you read and understund these instructions carefully, Please
Evaluation Criteria:
Coverage (1-S)- the amount key information covered by summary that relevant this citation. The summary should contain information from the cited paper that fits the context of the given citation text and helps you to better understand why this paper was cited here.
Focus (1-5)- collective quality sentences. The summary should be well-structured well-organized. The summary should build from sentence sentence to coherent body of information that is relevant to the given citation text.
Relevance (1-5)- - the relevance of the summary to the citation text. The summary should contain only information from cited paper that fits the current reading context. It should be sufficiently informative so that you not need to actually read the cited paper to understand why it was cited here.
Evaluation Steps (Corrected Chain of Thought):
1. Read given citation text that references another paper and make note its content.
2. Read the summary provided of the cited paper.
3. Coverage:
(a) Check if the summary contains only information that is relevant to the citation text.
(b) Assign a score for coverage on scale to 5, where is the lowest and is the highest based on the Evaluation Criteria.
4. Focus:
(a) Check summary coherent and contains well-organized information that relevant the citation text.
(b) Assign score for focus scale of to 5, where 1 is the lowest and is the highest based on the Evaluation
Criteria.
5. Relevance:
(a) Compare the content of the summary with the citation text. Assess whether the summary includes relevant information from the cited paper that directly relates to the context and purpose of the citation.
(b) Determine the summary provides enough information you to understand why cited paper was referenced without needing read entire paper.
(c) Rate the relevance of the summary based the above assessment using scale of I 5, where I indicates low relevance and 5 indicates high relevance.
Example:
Source Text: { [Citance]
Summary: ([Summary]]
Evaluation Form (scores ONLY):
Coverage:
Focus:
Relevance:
Figure 3: G-Eval (Liu et al., 2023) instructions for automatically evaluating the summary coverage, focus, and relevance using GPT4, Instructions merged into single prompt only for illustration purposes and used separately in the automatic evaluation.

---

Summary
Human Eval.
G-Eval
summaries over abstracts. Table 6 shows examples of self-contained and ambiguous citances and an- notator preferences. To substantiate this result, we Rel.
Cov.
Focus
Cov.
Focus
Abstract
3.67
4,50
3.12
3.80
3.23
plan to extend our evaluation to larger number of citances in future work.
similar-M225, top-2 paragraphs
GPT4 (Reference)
2.92
3.83
2.60
3.46
3,10
Vicuna
3.01
3,56
2.96
3,20
2.80
LLM-basee Evaluation To investigate the reli-
citance-Seen SciBERT, top-5 sentences
ability of evaluating the quality of summaries us- ing LLMs, we used G-Eval (Liu et al., 2023), which uses GPT4 to evaluate summary quality based on certain criteria. We evaluated coverage, coherence, and relevance using prompts to assign scores from to 5. G-Eval first lets the underlying model to gen- erate a chain of thought to ensure it understands the task. Evaluation instructions along with the (manually) corrected chain of thought for each cri- terion are illustrated in Figure 3. Table 5 shows the results, which indicate that G-Eval reflects the human assessments for the top models, with slight deviations for the lower-ranked ones. Notably, the reference summaries from GPT4 scored similarly to abstracts in terms of relevance.
GPT4 (Reference)
2.45
2.99
2.35
3.14
2.56
LLaMA-CoT 2.33
2.33
2.40
3.12
2.63
Table 5: Scores for summary quality criteria averaged over 125 summaries according human evaluation and automatic evaluation with G-Eval. We also automati- cally evaluated the relevance of a summary to a citance. Summary types are grouped by retrieval setup.
and five summaries on the right-hand side: the ab- stract of the cited paper, two reference summaries (top-5 sentences and top-2 paragraphs), and the summaries generated by the two best models for the top-5 sentences and top-2 paragraphs. The or- der of the summaries was randomized mitigate order effects (Mathur al., 2017).
Conclusion
IAA and Results Annotators agreement was cal- culated using Cohen's weighted kappa (Cohen, 1960). We obtained x of 0.42 and 0.40 for cov- erage and focus, respectively. While these results indicate some agreement between annotators, they also hint the inherent subjectivity of this annota- tion task. Assessing the usefulness of a summary is influenced by several contextual factors, such as the annotators' goals in reviewing a citation, their prior knowledge of the cited paper, and the presentation of the summary (Jones, 2007). In future work, we plan to investigate these factors further.
We investigated the use of generic paper abstracts compared to tailored contextualized summaries to improve reader's understanding of the relevance of individual paper citations. For this purpose, developed
summarization approach generate cilance-contextuullzed summaries. With WEBIS SOTBES-COOTETT-SSSSSMMMM 2023, we compiled a large-scale corpus to facilitate research this direction. Experiments with zero-shot summariza- tion using LLMs showed that abstracts are slightly preferred over contextualized summaries in terms of coverage and focus, while summaries generated using our approach preferred when citances do not refer to the main contribution of paper.
As shown in Table 5, abstracts as summaries achieved the highest coverage score (3.67), fol- lowed closely by the summaries generated by Vi- cuna (3.01). Abstracts were also rated the best summary for focus (4.50), while the reference sum- mary of GPT4 was only second (3.83). In terms of granularity of retrieved content, the summary based on the top-2 paragraphs outperformed the summary based on the 5 sentences in terms of both coverage and focus.
7 Limitations
Our proposed approach for contextualized summa- rization of scientific papers is based on standard retrieval models and L.Mss. It should be noted that what is considered relevant to a citance is subjec- tive and depends on the reader's prior knowledge of the cited paper. As result, retrieval models may not always retrieve the most relevant context, which may affect the quality of the subsequent ab- stract. This may explain why abstracts intended be informative to broad audience scored better in human evaluations than contextualized summarics.
However, despite the general preference for ab- stracts over generated contextualized summaries, annotators provided feedback that our summaries were more effective when a citance was ambiguous and did not relate to the overall idea of a paper. In such cases, they preferred our retrieval- based

---

self-contained Citance
The skill implements Hybrid Code Networks (HCNs) described in (Williams al., 2017).
Abstract (
End-to-end learning of recurrent neural networks (RNNs) is attractive solution for dialog systems; however, current techniques are data- intensive and require thousands dialogs to lear simple behaviors. We introduce Hybrid Code Networks (HCNs), which combine RNN with domain-specific knowledge encoded as software and system action templates. Compared existing end-to-end approaches, HCNs considerably reduce amount of training data required, while retaining the key benefit inferring a latent representation of dialog state. In addition. HCNs can be optimized with supervised learning, reinforcement learning, or a mixture of both. HCNs attain state-of oh---e--rtt-t--- art performance on the bAbl dialog dataset, and outperform two commercially deployed customer- facing dialog systems.
Vicuna (Top Relevant Paragraphs Summarized)
This paper presents new model called Hybrid Code Networks. (HCNs) for end-to-end learning- HCNs allow developers to express domain knowledge and achieve better performance with less training data compared to existing techniques. The model is trained using supervised or reinforcement learning and be applied to the bAbl dialog dataset and real customer support domains. The model can also be optimized with reinforcement learning.
Ambiguous Citance
In both cases, we use the BIOES labeling scheme for the output tags, following previous work which showed it outperforms other options (e.g., Ratinov and Roth, 2009).
Abstract
We analyze of the fundamental design challenges and misconceptions that underlie the development of efficient and robust NER system. In particular, address issues such as the representation of text chunks, the inference approach needed to combine local NER decisions, the sources of prior knowledge and how use them within an NER system. In the process of comparing several solutions these challenges reach some surprising conclusions, well as develop NER system that achieves 90.8 Fl score on the CoN1.1-2003 NER shared task, the hest reported result for this dataset.
Vicuna (Top 2 Relevant Paragraphs Summarized) (V)
A simple technique of using word clusters generated from unlabeled text can improve performance dependency parsing, Chinese word segmentation, and NER. The technique is based on word class models and uses a hinary tree to represent words. The approach is related to distributional similarity, but not identical. The system's 's performance is significantly impacted by the choice of encoding scheme, and the less used BILOU formalism outperforms the widely adopted BIO tagging scheme.
Table Examples self-contained and ambiguous citances and human annotators' preference for summaries For the self-contained citance, the annotators found that the paper's 's abstract has higher degree of coverage and better fit to the reading context. However, in the case of an ambiguous citance, the abstract was found insufficient to understand the citance because additional information had be retrieved from the document. In this case, annotators preferred the generated contextualized summary over the abstract.
Our approach relies on LLMs that are constantly improved by the research community. The results our experiments may vary with the introduction of newer LLMs. However, the underlying approach itself is intuitive and can be easily adapted to incor- porate newer LLMss as they become available. is also important to recognize an important but often overlooked limitation of any summarization technology, namely the lack of a clear definition of what constitutes a good summary, given the its purpose. In our case, the purpose of the summary is to help readers understand the relevance of a ci- tation without having to look up the cited paper. While we used abstracts as reference for compar- ison, our evaluation method does not induce true information needs relating to this purpose. This makes fair comparison between abstracts and contextualized summaries difficult. In addition, the availability of expert annotators forced us to focus
on the NLP field, which means that our results may not generalize to scientific papers from other fields. We hope that our work will encourage the research commumity to develop a more robust evaluation methodology for summarization tailored to the spe- cific purposes of different types of summaries.
Acknowledgments
This work was partially supported by the Eu- ropean Commission under grant agreement GA 101070014 (OpenWebSearch.eu)) Computations for this work were done (in part) using resources of the Leipzig University Computing Center.
References
Amjad Abu-Jbara and Dragomir Radev. 2011. Coher- ent citation- summarization of scientific papers. In Proceedings of the 49th Annual Meeting of the

---

Association for Computational Linguistics: Human Language Technologies, pages 500-509, Portland, Oregon, USA. Association for Computational Lin- guistics.
Claudio Carpineto and Giovanni Romano. 2012. A survey of automatic query expansion in information retrieval. ACM Compur. Sure, 44(1) 1::::::111:50.
Muthu Kumar Chandrasckaran, Guy Feigenblat, Dayne Freitag, Tirthankar Ghosal, Eduard Hovy, Philipp Mayr, Michal Shmueli-Scheuee and Anita de Waard. 2020. Overview of the first workshop on scholarly document processing (SDP). In Proceedings of the First Workshop on Schotarly Document Processing. pages 1-6, Online, Association for Computational Linguistics.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Ftienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-4OB: an open large language model with sate-of-thee-ae performance.
Phyllis B. Baxendale. 1958. Machine-ma made index for technical literature- an experiment. 1BM J. Res. Dev., 2(4):354-361.
Yifan Chen, Tamara Polajnar, Colin Batchelor, and Si- mone Teufel. 2020. A corpus of very short scien- tific summaries. In Proceedings of the 24th Confer- ence on Computational Natural Language Learning. pages 153-164, Online. Association for Computa- tional Linguistics.
1z Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP)) pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguisties.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt4 with 90%* chatgpt quality. See https://vicua. Imsys. org (accessed 14 April 2023).
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx. Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shya- mal Buch, Dallas Card, Rodrigo Castellon, Ni- ladri S. Chatterji, Annie S. Chen, Kathleen Creel. Jared Quincy Davis, Doroutya Demszky, Chris Don- ahue, Moussa Doumbouya, Esin Durmus, Stefano John Etchemendy, Kawin Ethayarajh. Li Fei- Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. 2021. On the opportunities and risks of foundation models. CORR, htss/208.0025$
Arman Cohan, Franck Dernoncourt Doo Soon Kim. Trung Bui, Seokhwan Kim. Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume (Short Papers), pages 615-621, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Arman Cohan and Nazli Goharian. 2015. Scientific article summarization using citation and arti- cle's s discourse structure. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 390-400, Lisbon, Portugal. Association for Computational Linguisties.
Arman Cohan and Nazli Goharian. 2017. Contextualiz- ing citations for scientific summarization using word embeddings and domain knowledge. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 1133-1136. ACM.
Dorian Brown 2020. Rank-BM25: A Collection of BM25 Algorithms in Python.
Sébastien Bubeck. Varun Chandrasekaran. Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with gpt4.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20( 1).37-466
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld. 2020. TLDR: extreme summarization of sci-
Ed Collins, Isabelle Augenstein, and Sebastian Riedel. 2017. A supervised approach to extractive sum- marisation of scientific papers. In Proceedings of the 21 Conference on Computational Natural Lan- guage Learning (CoNLL 2017), pages 195-205, Van- couver, Canada. Association for Computational Lin- guistics.
entific documents. In Findingy of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4766-4777 Association for Computational L Linguistics.

---

Aaron Elkiss, Siwei Shen, Anthony Fader, Günes Erkan, David J. States, and Dragomir R. Radev. 2008. Blind men and elephants: What do citation summaries tell us about research article? J. Assoc. Inf. Sci. Tech- nol., 59(1)-11-6..
through ai-powered interactive reading interfaces. CoRR, 0bss2303..13344,
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969-4983 Online. Asso- ciation for Computational Linguistics.
Maarten Grootendorst. 2020. Keybert: Minimal key- word extraction with bert.
Vivek Gupta, Prema Bharti, Pegah Nokhiz, and Harish Karnick. 2021. SumPubMed: Summarization dataset of PubMed scientific articles. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing: Stu- dent Research Workshop, pages 292-303, Online. Association for Computational Linguisties.
Hans Peter Luhn. 1958. The automatic creation of liter- ature abstracts. IBM J. Res. Dev., 152):159-165
Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. 2018. Incorporating glosses into neural word sense disambiguation. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20. 2018, Volume 1: Long Papers, pages 2473-2482, Association for Computational Linguis- tics.
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumu- lated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422-44) 446.
Karen Spärck Jones. 2007.
Automatic summarising: The state of the art. Inf. Process. Manag-, 43(6); 1449- 1481.
Yuning Mao, Ming Zhong, and Jiawei Han. 2022. Cite- Sum: Citation text-guided scientific extreme sum- marization and domain adaptation with limited su- pervision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10922-10935, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Guy Lev, Michal Shmueli-Scheuer Jonathan Herzig, Achiya Jerbi, and David Konopnicki. 2019. Talk- summ: A dataset and scalable annotation method for scientific paper summarization based on confer- ence talks. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol- ume 1: Long Popers, pages 2125-2131 Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2017. Sequence effects in crowdsourced annotations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2860 2860-2865. Association for Computational Linguistics.
Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Qiaozhu Mei and ChengXiang Zhai. 2008. Generat- ing impact-based summaries for scientific literature, In Proceedings of ACL-08; HLT, pages 816-824, Columbus, Ohio. Association for Computational Lin- guistics.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NL.G evaluation using GPT4 with better human align- ment. CoRR, abs/2303.16633
Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021. Bringing structure into summaries: faceted sum- marization dataset for long scientific documents. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1080- 1089, Online. Association for Computational Linguis- tics.
Kyle Lo, Joseph Chee Chang, Andrew Head, Jonathan Bragg, Amy X. Zhang, Cassidy Trier, Chloe Anas- tasiades, Tal August, Russell Authur, Danielle Bragg, Erin Bransom, Isabel Cachola, Stefan Candra, Yo- ganand Chandrasekhar, Yen-Sung Chen, Evie Yu- Yen Cheng. Yvonne Chou, Doug Downey, Rob Evans, Raymond Fok, Fangzhou Hu, Regan Huff, Dongyeop Kang, Tae Soo Kim, Rodney Kinney. Aniket Kittur, Hyeonsu B. Kang, Fggor Klevak, Bai- ley Kuehl, Michael Langan. Matt Latzke, Jaron Lochner, Kelsey MacMillan, Eric Marsh, Tyler Mur- ray, Aakanksha Naik, Ngoe-Uyen Nguyen, Srishti Palani, Soya Park, Caroline Paulic. Napol Rachata- sumrit, Smita Rao, Paul Sayre, Zejiang Shen, Pao Siangliulue, Luca Soldaini, Huy Tran, Madeleine van Zuylen, Lucy Lu Wang, Chris Wilhelm, Caroline Wu, Jiangjiang Yang. Angele Zamarron, Marti A. Hearst, and Daniel S. Weld. 2023. The semantie Augmenting scholarly documents
Sewon Min, Victor Zhong, Richard Socher, and Caim- ing Xiong. 2018 Efficient and robust question an- swering from minimal context over documents. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1725-1735. Association for Com- putational Linguistics.
reader project:

---

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.01 16.
Sajad Sotudeh and Nazli Goharian. 2022. TSTR: Too short to represent, summarize with details! intro- guided extended summary generation. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistic:: Human Language Technologies, pages 325- 335, Seattle, United States. Association for Compu- tational Linguistics.
Vahed Qazvinian and Dragomir R. Radev. 2008. Sci- entific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages (89-696, Manchester, UK. Coling 2008 Organizing Commitiee.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable ceoruction-folloiing model. Stan- ford Center for Research on Foundation Models. https://crfm. stanford. edw/2023/03/1 3/alpaca. html, 3(6).7.
Siya Qi, Lei Li, Yiyang Li, Jin Jiang, Dingxin Hu, Yuze Li, Yingqi Zhu, Yanquan Zhou, Marina L Litvak, and Natalia Vanetik. 2022. SAPGraph: Structure-aware extractive summarization for scientific papers with heterogeneous graph. In Proceedings of the 2nd Con- ference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cessing ( (Volume I: Long Papers), pages 575-586, Online only. Association for Computational Linguis- tics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models. CoRR. abs/2302. 13971.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR, abs/2212.10560.
Zheng Lin Qingyi Si. 2023. Alpaca-cot: An instruction fine-tuning platform with instruction data collection and unified large language models interface. https: //github.com Ceithub.comm/Pee wwwceooo/cccceeee http///ooobuSSSappaaat CoT.
Napol Rachatasumrit, Jonathan Bragg, Amy X. Zhang, and Daniel S. Weld. 2022. Citeread: Integrating lo- calized citation contexts into scientific paper reading. IUI 2022: 27th International Conference on Intel- ligent User Interfaces, Helsinki, Finland, March 22- 25, 2022, pages 707-719. ACM.
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexan- der R. Fabbri, Irene Li, Dan Friedman, and Dragomir R. Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for sci- entific paper summarization with citation networks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innova- tive Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2019, Honobulu, Hawaii, USA, January 27 February I, 2019, pages 7386-7393, AAAI Press.
Stephen E. Robertson, Steve Walker, Susan Jones, and Mike Gatford. 1994, Okapi at TREC-3. In Proceedings The Third Text REtrieval Conference, TREC 1994, Gaithers- burg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication, pages 109- 126. National Institute of Standards and Technology (NIST).
Micheline
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexan- dra Birch, Barry Haddow, Julian Hitschler, Marcin munczys-Dommun Samuel Läubli, Antonio Vale- rio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: toolkit for neural machine trans- lation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Software Demonstrutions, pages 65-68. Association for Computational Linguistics.
A Prompt Instructions
We used the following summarzation instructions to summarize the top-2 most relevant paragraphs:
1. Generate a coherent summary for the follow- ing scientific text in not more than 5 sentences.
Sajad Sotudeh, Arman Cohan, and Nazli Gohar- ian. 2021. On generating extended summaries of long documents. In Proceedings of the Work- shop on Scientific Document Understanding co- located with 35th AAAI Conference on Artificial In- religence, SDU@ AAAI 2021, Virtual Event, February 2021, volume 2831 of CEUR Workshop Proceed- ings. CEUR-WS.org.
2. Generate short summary of the following scientific text. The summary should not be more than 5 sentences long.
3. Summarize the following scientific text in not more than 5 sentences.

---

By manually inspecting the model output of each instruction, we found that the first two instructions resulted in fluent summaries. However, exclud- ing "coherent" from the instruction sometimes re- sulted a bulleted list important sentences in the summary. The third instruction aggressively compressed the content into concise summary, but this resulted in (partial) loss of information.
For GPT4 we used the direct instruction prompt without template. The first template worked well only with the Alpaca and Falcon models, but not with the Vicuna and LLaMA-CoT models. If the last sentence of the input contained ':' or an ab- breviation, these models tried to adapt the output to it. The Vicuna model presented summaries an enumerated list. The LLaMA-COT model did not "understand" this template and therefore did not produce any output. Therefore, we excluded this template from further experiments. Alpaca, LLaMA-CoT, and Falcon generated readable sum- maries based on the second template, but Vicuma did not, which prevented us from using it further. For example, the LLaMA-CoT model generated the following output when paraphrasing: "The text you provided is already paraphrased. It contains several sentences that express the same idea in different ways." The third prompt template yielded readable, coherent, and understandable summaries from all four models and caused artifacts the summaries in only a few individual examples. Therefore, we used this prompt template with its associated in- structions for our experiments.
We used the following instructions to paraphrase the-5 most relevant sentences as summary;
1. Generate coherent paraphrased text for the following scientific text.
2. Generate paraphrased text for the following scientific text.
3. Paraphrase the following scientific text.
4. Combine the following scientific text into a coherent and concise text.
A similar effect to the summarization prompts above was observed when the word "coherent" was excluded from the instructions. The last two prompts performed poorly, and in most cases sim- ply returned the input text.
C Summarization Models
B Prompt Templates
1. Alpaca (Taori et al., 2023) was fine-tuned based on the LLaMA 7B model (Touvron et al., 2023) using 52,000 instruction-folloving examples (Wang et al., 2022).
Generate coherent summary for the follow- ing scientific text in not more than sentences. scientific text: {input]
summary:
2. Vicuna (Chiang et al., 2023) was fine-tuned based on LLaMA using user conversations from ShareGPT.$ It has shown competitive effective- ness when evaluated with GPT4. We used the 13B variant.
A chat between curious user and an artificial intelligence assistant. The assistant knows how summarize scientific text and the user will provide the scientific text for the assistant to summarize.
3. LLaMA-COT was fine-tuned based on datasets inducing chains of thought (CoT) and logical reasoning (Qingyi Si, 2023).
USER: Generate coherent summary for the following scientific text in not more than 5 sentences: "(input)"
4. Falcon (Almazrouei al., 2023) was trained based the RefinedWeb dataset (Penedo al., 2023), which has been obtained by extensive filtering and deduplication of publicly available web data. the time writing, is the state of the art on the open-llm leaderboard. 10 We used the 40B-Instruct variant.
ASSISTANT:
### Instruction: A chat between curious human and an artificial intelligence assistant. The assistant knows how to summarize scien- tific text and the user will provide the scientific text for the assistant to summarize.
### Input: Generate coherent summary for the following scientific text in not more than sentences: "(input)"
"htps://sharegpt.com "htts://hugging face. cofausboss/lama 30b supercot 10 htt.://wgggggg.eooo///////ggggnn leaderboard
### Output:

---

5. GPT4 (Bubeck et al., 2023) is the latest version of OpenAl's popular GPT model class, show- ing the best effectiveness several benchmarks. Therefore, we used it to bootstrap reference sum- maries for our summary evaluation (Section 5.2). Unlike the above open source models, it is ac- cessible only through the OpenAl API.
D Contextualized Summary Examples
Tables 7, 8, 9, and 10 show examples of generated contextualized summaries using alternative models.

---

Nematus: Toolkit for Neural Machine Translation (Sennrich et al., 2017)
Citance Context
Additionally, Nematus supports minimum risk training (MRT) (Shen et al., 2016) to optimize towards an arbitrary. sentence-level loss function.
Various MT metrics supported loss functions, including smoothed sentence-level BLEU (Chen and Cherry. 2014), METEOR (Denkowski and Lavie, 2011), 1), BEER (Stanojevic and Sima' an, 2014), and any interpolation of implemented metrics.
stabilize training. Nematus supports early stopping based on cross entropy or an arbitrary loss function defined by the user.
Abstract
BLEU is the de facto standard machine translation (MT) evaluation metric. However, because BLEU computes geometric mean of n-gram precisions, it often correlates poorly with human judgment the sentence-level. Therefore, several smoothing techniques have been proposed. This paper systematically compures 7 smoothing techniques for sentence-level BLEU. Three of them are first proposed in this paper. and they correlate better with human judgments on the sentence-level than other smoothing techniques. Morcover, we also compare the performance of using the 7 smoothing techniques in statistical machine translation tuning.
Top 5 Relevant Sentences Paraphrased (Query Context: citonce-SeiBERT) GPT4
In this study, we conduct a systematic comparison seven different smoothing methods for sentence-level BLEU. Although recent WMT Evaluation Task reports (Callison- Burch al., 201 Callison Hallison-Burcheee al., 201 2) demonstrated that numerous other metrics have stronger correlation human evaluations of translation quality than BLEU, continues to the widely accepted standard assessment and optimization. Smoothing incorporated into the official NIST evaluation toolkit (mteval-v 3a.pl), and as a result, various smoothing approaches for sentence-level BLEU have been suggested (Lin and Och, 2004; Gao and He, 2013).
LLaMA-COT
In this paper, compare various smoothing techniques for sentence-level BLEU. Although other metrics better assessing translation quality, BLEU remains the standard evaluation metric. Smoothing is implemented in official NIST toolkit. Several smoothing techniques have been proposed.
Top 2 Paragraphs Summarized (Query Contextssimilar-Bht BM25) GPT4
Since its invention, BLBU has been the most widely used metric for machine translation evaluation and tuning. Despite other metrics correlating better with human judgments of translation quality, BLEU remains the facto standard. This is likely because models trained using BLEU obtain the highest scores from humans and even from other metrics, making it the best tuning metric from a quality perspective.
Vicuna
BLEU is the most widely used metric machine translation evaluation and tuning since its invention. Although other metrics correlate better with human judgments translation quality, BLEU remains the de facto standard due to its effectiveness as tuning metric, with models trained using BLEU obtaining the highest scores from both humans and other metrics.
Table 7: Examples of automatically generated contextualized summaries from the best models for both granularities (LLaMa and Vicuma). Summaries from GPT4 are used as ground truth for comparison after post processing for faithfulness and hallucinations.

---

Nematus; a Toolkit for Neural Machine Translation (Sennrich et al., 2017)
Citance Context
Additionally, Nematus supports minimum risk training (MRT) (Shen et al., 2016) to optimize towards an arbitrary, sentence-lewe levell loss function.
Various MT metrics are supported as loss functions, including smoothed sentence-level BLEU (Chen and Cherry. 2014), METEOR (Denkowski and Lavie, 201 I), BEER (Stanojevic and Sima an, 4), and any interpolation of implemented metrics.
To stabilize training, Nematus supports early stopping based on cross entropy or an arbitrary loss function defined by the user.
Abstract
This puper describes Metcor 1.3, submission the 2011 EMNLP Workshop Statistical Machine Translation automatic evaluation metric tasks. New metric features include improved text normalization, higher-precistee paraphrase matching, and discrimination between content and function words. We include Runking and Adequacy versions of the metric shown to have high correlation with human judgments translation quality as well as more balanced Tuning version shown to outperform BLEU in minimum error rate training for phrase-base based Urdu- Aodu-EEglishh system.
Top 5 Relevant Sentences Paraphrased (Query Context: Situnce-SciBERTT GPT4
In ussessment tasks, the designed metrics typically aim to score the top translations from systems that are primarily optimized for BLEU. Meteor measures translation hypotheses by aligning them with reference translations and computing sentence- similarity scores. The extra features in Meteor 1.3 enable more balanced parameters that allocate the responsibility of penalizing different kinds of incorrect translations. Since this task does not offer a devtest set, we choose Meteor version by examining the efficiency of employing multiple metric versions to fine-tune phrase-baset translation systems for the same language pair. We utilize 7-MERT to optimize this system for both baseline metries and the Meteor versions mentioned in carlier sections.
LLaMA-COT
In this study, we evaluate the performance various metrics in translation evaluation tasks. We focus on the BLEU metric, which is commonly used to score I-best translations from systems. We also explore the Metcor metric, which is designed to evaluate translation hypotheses by aligning them to reference translations and calculating sentence-level similarity scores. The new features in Meteor 1.3 allow for more balanced parameters, distributing responsibility for penalizing various types of erroneous translations. Since this task does not provide a devtest set, we select a version of the metric by exploring its effectiveness in tuning phrase-based translation systems for the same language pair. We use Z-MERT to tune this system to baseline metrics as well as the versions of Meteor discussed in previous sections.
Top 2 Paragraphs Summarized (Query Contextsimmila--AN GPT4
A machine translation (MT) system tuning experiment was conducted on Urdu Frdu--Enlish data compare the effectiveness of using multiple versions of Metcor in minimum ervor rate training. Although versions tuned to various human judgments did not perform well as the widely used BLEU metric, balanced tuning version of Meteor consistently outperformed BLEU, The Metcor metric has been shown to have high correlation with human judgments in evaluations. However, previous versions of the metric were limited by lack of punctuation handling, noise in paraphrase matching. and lack of discrimination between word types. The introduction of resources for all WMT languages, including normalizers, filtered paraphrase tables, and function word lists, improved Meteor's 's correlation with human translation rankings and adequacy scores on unseen test data.
Vicuna
A team of researchers conducted an experiment to compare the effectiveness of using multiple versions of the Meteor machine translation system on Urdu-English data. They found that a balanced tuning version of Meteor consistently outperformed the widely used BLEU metric. The Meteor 1 metric, which has been shown to have high correlation with human judgments, was used in the experiment. However, previous versions of the metric had limitations such as lack of punctuation handling and discrimination between word types. The team introduced new resources for all WMT languages, including text normalizers. filtered paraphrase tables, and function word lists, which allowed them to tune versions of the metric that showed higher correlation with human translation rankings and adequacy scores on unseen test data. The evaluation resources were found to be modular and usable with any other evaluation metric or MT software.
Table 8: Examples of automatically generated contextualized summaries from the best models for both granularities (LLaMMa and Vicuna). Summaries from GPT4 used as ground truth for comparison after post processing for faithfulness and hallucinations.

---

Incorporating Glosses into Neural Word Sense Disambiguation (I.uo et al., 2018)
Citance Context
Additionally, Nematus supports minimum risk training (MRT) (Shen et al., 2016) to optimize towards an arbitrary, sentence- loss function.
In this puper, we propose novel model GAS: gloss-augmented WSD neural network which is variant of the memory network (Sukhbaatar al., 2015b; Kumar al., 2016; Xiong al., 2016).
In order measure the inner relationship between glosses and context more accurately, employ multiple passes operation within the memory as the re-reading process and adopt two memory updating mechanisms., " In order model semantic relationship of context and glosses, we propose a glossaugmented neural network (GAS) an improved memory network paradigm.
Abstract
Most tasks in natural language processing can be cast into question answering (QA) problems over language input. We introduce the dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations. These results are then reasoned over in hierarchical recurrent sequence model to generate answers. The DMN can be trained end-to- -o-et-t-dd and obtains state of of-the-art art results on several types of tasks and datasets: question answering (Facebook's 's bAbl dataset). text classification for sentiment analysis (Stanford Sentiment Treebank) and sequence modeling for part-of-speech tagging (WSI-PTB). The training for these different tasks relies exclusively on trained word vector representations and iapul-questoon-answee triplets.
Top 5 Relevant Sentences Paraphrased (Query Context: citonce-SeiBERT) GPT4
tasks, which trained using raw input-questionanee sansoon--nwwerreeee triplets. Recent work by Weston et al. memory networks (Weston al., 2015b) emphasizes incorporating a memory component for natural language question answering. This model employs two extremely large and deep LSTMs to encode a sentence one language and subsequently decode another language. The primary innovation the DMN architecture lies in its episodic memory module. Kim (Kim, 2014) documents the previous state-of-the-art result, which based on a convolutional neural network utilizing multiple word vector representations.
We present the Dynamic Memory Network (DMN), neural network framework designed for general
LLaMA-COT
We present the Dynamic Memory Network (DMN). a neural network-based framework for general question answering tasks that is trained using raw suput-qustioon-nnswe triplets. Recent work Weston al. (Weston al., 2015b) focuses adding memory component for natural language question answering. This model uses extremely large and deep LSTMs encode sentence one language and then decode the sentence another language. The novelty of the DMN architecture its episodic memory module. Kim (Kim, 2014) reports the previous state-of-the-art result based convolutional neural network that uses multiple word vector representations.
Top 2 Paragraphs Summarized (Query Contextsimilar-BMM2 GPT4
The Dynamic Memory Network (DMN) is a neural network- framework designed for general anrestion-answerii tasks, trained using raw input-question-ans centttioonansweee wer triplets. It can solve various tasks such as sequence tagging, classification problems, sequence-to-sequenc tasks, and guestion-answerng tasks requiring transitive reasoning. While deep learning models like recursive neural networks have been applied to many NLP tasks, they lack the memory and question modules needed for more varied tasks and transitive reasoning. Recurrent neural networks have heen successfully used in language modeling, speech recognition, and sentence generation from images. The sequence-to- to-sequence model used for machine translation is a special case of the DMN without a question and episodic memory, mapping an input sequence directly to an answer sequence.
Vicuna
We propose the Dynamic Memory Network (DMN), neural network based framework for general question answering tasks that is trained using raw input-question-ane -ansswion--nswerr triplets. It can solve sequence tagging tasks, classification problems, sequence-to- sequence tasks, and question answering tasks that require transitive reasoning. The DMN has memory module and question module, which are absent in other deep learning models such as recursive neural networks and chain shain-structured recurrent neural networks. The DMN is based on a soquence-to-eqqeeee model used for machine translation, but it has a memory and question module.
Table 9: Examples of automatically generated contextualized summaries from the best models for both granularities (LLaMa and Vicuna). Summaries from GPT4 are used as ground truth for comparison after post processing for faithfulness and hallucinations.

---

Efficient and Robust Question Answering from Minimal Context over Documents (Min et al,, 2018)
Citance Context
In addition, they cover ditferent domains, including Wikipedia (Rajpurkar al., 2016; Joshi et al., 2017), news (Hermann al.,
2015; Trischler et al., 2016), fictional stories (Richardson et al., 2013; Kočiský al., 2017), and textbooks (Lai et al., 2017: Xie et al., 2017) :, 'Recently rescarchers have developed largescale QA datasets, which requires answering the question over large set of documents in closed (Joshi et al., 2017) open-doman (Dunn et al., 2017; Berant et al., 2013; Chen et al., 2017;
Dhingra al., 2017).
Following previous work on distant supervision (DS) for relation extraction (Mintz et al., 2009 ). we use procedure to automatically associate paragraphs to such training examples, and then add these examples to our training set.
We use the following process for each questionanswer pair to build our training set.'. 'If there is no paragraph with non-zero overlap. the example is discarded; otherwise we add each found pair to our DS training dataset.
Abstract
Modern models of relation extraction for tasks like ACE are based supervised learning of relations from small hand-labeled corpora. We investigate an alternative paradigm that does require labeled corpora, avoiding the domain dependence of ACEstyle algorithms, and allowing the use of corpora of any size. Our experiments use Freebase, large semantic database of several thousand relations, to provide distant supervision. For each pair of entities that appears in some Freebase relation. we find all sentences containing those entities a large unlaheled corpus and extract textual features train relation elassifier. Our algorithm combines the advantages of supervised IE (combining 400,000 noisy pattern features in a probabilistic classifier) and unsupervised IE (extracting large numhers of relations from large corpora of any domain). Our model able extract 10,000 instances of 102 relations at precision of 67.6%. We also analyze feature performance, showing that syntactic parse features are particularly helpful for relations that are ambiguous lexically distant their expression.
Top 5 Relevant Sentences Paraphrased (Query Context: Situnce-SciBERTT GPT4
Our method employs Freebase (Bollacker et al., 2008), vast semantic database, to offer distant supervision for extracting relationships. Since a single sentence might provide an inaccurate signal, our algorithm trains a multiclass logistic regression classifier, determining weights for each noisy characteristic. The core idca of our distant supervision technique is to utilize Freebase to supply a training set of relationships and entity pairs involved in those relationships. Each entity pair in every sentence of the test corpus undergoes feature extraction, and the regression classifier predicts a relationship name for cach pair based on the features from all sentences in which it was present. For each entity pair, we consolidate the features from the numerous sentences in which the pair appeared into single feature vector, enahling us to furmish our classifier with more data, leading to more precise labels.
LLaMA-CoT
Our algorithm uses Freehase, large semantic database, to provide distant supervision for relation extraction. trains a multiclass logistic regression classifter, learning weights for each noisy feature. The intuition of our distant supervision approach is to use Freehase to give us a training set of relations and entity pains that participate in those relations. Each entity pair in the test corpus is run through feature extraction, and the regression classifter predicts a relation name for each entity pair based on the features from all of the sentences in which it appeared. For each pair of entities, we aggregate the features from the many different sentences in which that pair appeared into single feature vector, allowing us to provide our classifier with more information. resulting in more accurate labels.
Top 2 Paragraphs Summarized (Query Contextssimila--BNN BM22)) GPT4
Distant supervision is approach that uses Freebase, large semantic datahase, to provide training set of relations and entity pairs for relation extraction. The method identifies entities in sentences using named entity tagger and extracts features from sentences containing instances of Freebase relations. This approach allows for the of much larger amounts of data compared to supervised training paradigms, as can utilize millions of Wikipedia articles and instances of relations connecting numerous entities. Combining vast numbers of features in a large classifier helps overcome problems with bad features. Distant supervision is an extension of the paradigm used for exploiting WordNet extract hypernym relations between entities and is similar to the use of weakly labeled data in bioinformatics.
Vicuna
Our distant supervision approach uses Freebase to provide training set of relations and entity pairs that participate in those relations. We use named entity tagger to identify entities in sentences and extract features from sentences that contain two entities participating in a Frechase relation. Our algorithm is an extension of the paradigm used by Snow et al. (2005) and is similar to the use of weakly labeled data in bioinformatics. We use logistic regression classifier combine the vast numbers of features extracted from the text. Our approach can use much larger amounts of data than the supervised training paradigm, which uses only 17,000 relation instances training data. We used 1.2 million Wikipedia articles and 1.8 million instances of 102 relations connecting 940,000 entities in experiments. This helps to overcome the problem of bad features.
Table 10: Examples of automatically generated contextualized summaries from the best models for both granularities (LLaMa and Vicuna). Summaries from GPT4 are used as ground truth for comparison after post processing for faithfulness and hallucinations.


Reviews:
[{"review_id": "Hp6E1e4J7R", "reviewer": "Reviewer_jGux", "summary": "", "questions": "How do you compare the proposed method with papers like (Cohan and Goharian, 2017)?\nwhat is your result improvement and most highlighted novelty in comparison to the mentioned method? is this significant?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We try to answer your key questions below.\n\n__How do you compare the proposed method with papers like (Cohan and Goharian, 2017)?__\n\nCompared to Cohan and Goharian (2017), our method stands out in two distinct ways, as demonstrated in Figure 1 and elaborated from lines 209 to 221. Unlike their approach, which relies solely on verbatim citances for content retrieval, our investigation spans two additional citance contexts (lines 266 to 270), leading to 12 retrieval scenarios (listed in Table 2) that offer nuanced insights into context suitability for summarization. Moreover, while Cohan et al. consolidate all target paper content into a single summary for multiple citances, our approach crafts contextualized abstractive summaries independently for each citance, thus ensuring multiple, tailored summaries for the target paper. \n\n__What is your result improvement and most highlighted novelty in comparison to the mentioned method? is this significant?__\n\nOur approach's key innovation lies in generating multiple abstractive summaries for a cited paper, rooted in its various citances within a citing paper. To achieve this, we curated a sizable and fitting corpus, established clear baselines, and conducted comprehensive evaluations. Our study then substantiates the abstract's strength as a baseline through both automatic and qualitative assessments, spanning dimensions like \"coverage,\" \"focus,\" and \"relevance.\"\n\nWhile our contextualized summaries didn't surpass the abstract in our results, this insight holds crucial implications for future investigations. It notably prompts an exploration of the interplay between citance specificity and preferred summary type. Our annotators highlighted that when citances were generic, requiring minimal reference to the cited paper, abstracts were preferred; however, for more specific citances, contextualized summaries proved superior.\n\nPrevious work has consolidated all relevant information across citances to generate a single summary. Therefore, direct comparisons to such methods are neither possible nor meaningful.\n\n__“Limited ground truth has been generated by system, not human”__\n\nThe system was used only to bootstrap the summarization process. The resulting summaries were manually reviewed and revised if necessary to ensure there are no hallucinations or factual errors (lines 472 to 475).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposing a large-scale dataset consisting of 540K documents and 4.6M citances.", "reasons_to_reject": "1. Limited improvement,\n2. Subjective idea, and intuition\n3. limited ground truth has been generated by system, not human.", "questions_to_authors": "How do you compare the proposed method with papers like (Cohan and Goharian, 2017)?\nwhat is your result improvement and most highlighted novelty in comparison to the mentioned method? is this significant?", "paper_topic_and_main_contributions": "This paper is a summarization paper focusing on citances to enhance the exploration of relevant information from the cited paper.\nIn other words, this work aims to satisfy the readers who track citations and seek specific connections between the citing and cited papers.\nthey consider two additional contexts for a citance. first is the sentence including citance, and the sentence preceding and succeeding it. second, 2 most semantically similar sentences in the same paragraph.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Ese6YxXMu", "reviewer": "Reviewer_Gvqo", "summary": "", "questions": "1. was there any impact to multiple citations of the form \"three systems do this [1,2,3]\"?\n2. more generally, are there properties of the surface form of the source and target that make this approach more effective for the user?\n3. were the human annotators remunerated, and were they given opportunity to consent etc? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comments and especially for the very interesting reference to the Ball, P. paper about unread citations. We answer your questions below.\n\n__was there any impact to multiple citations of the form \"three systems do this [1,2,3]\"?__\n\nThe impact in such cases depends on how differently the three papers approach a given task. Moreover, our approach uses the citance text (and its various contexts) as the query to fetch relevant information, a citance that is too generic (e.g. “three systems do this…”) does not yield much informative queries. This subsequently results in the retrieval of unrelated or generic sentences that are summarized. In such cases, the abstract was more informative (as observed by our annotators).\n\n__more generally, are there properties of the surface form of the source and target that make this approach more effective for the user?__\n\nContinuing from the previous answer, our approach produces more informative and relevant summaries (compared to the abstract) where the citance is highly specific. So, we conjecture that a citance in the source paper must be rather specific to provide better search queries for finding relevant content from the target paper, consequently yielding superior summaries. Further investigation is warranted to consolidate this hypothesis.\n\n__were the human annotators remunerated, and were they given opportunity to consent etc?__\n\nThe human annotators are full-time research assistants employed by our university's Computer Science faculty. They received proper compensation for their dedicated time towards research activities and have voluntarily given their consent to partake in the annotation process. We will clarify this in sufficient detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The ability to create task-tailored abstractive summarisations of this kind has applications beyond academic analysis. The dataset seems large and well-curated, and the example solution explores the possible solution space without fully answering it. ", "reasons_to_reject": "I think there is more work to be done on the comparative judgment - while initial work comparing this kind of summary to others is useful, a more targeted extrinsic evaluation might yield better performance in future work. ", "questions_to_authors": "1. was there any impact to multiple citations of the form \"three systems do this [1,2,3]\"?\n2. more generally, are there properties of the surface form of the source and target that make this approach more effective for the user?\n3. were the human annotators remunerated, and were they given opportunity to consent etc? ", "paper_topic_and_main_contributions": "The primary contribution of this paper is CONTEXT-SCISUMM a corpus of 4.6 Million citation texts over half a million academic papers in the CS domain. the dataset is introduced alongside the task of citance-focused abstract generation from papers cited in a work. The solution tot he latter task described in the paper compares a variety of retrieval methods and summarisation approaches, and evaluates the results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Minor questions about the human volunteers"}, "extra_text": {"Missing_References": "Not necessarily important, but perhaps as part of the discussion, Ball, P. Paper trail reveals references go unread by citing authors. Nature 420, 594 (2002). https://doi.org/10.1038/420594a is one of many papers that discusses the fact that authors may not fully read the papers they are citing - this tool might help alleviate (or detect!) that?"}}, {"review_id": "vkqSWsrXAs", "reviewer": "Reviewer_mxe2", "summary": "", "questions": "Question A: Regarding the dataset could you clarify if any summary is provided for the citance-cited_paper pair (summary length is 117 on avg for the provided dataset)? What metadata are included in the dataset?\n\nQuestion B: In your evaluation, you focus solely on the use of LLMs. Have you considered including standard summarization models or extractive baselines in your comparison to provide a more comprehensive evaluation?\n\nQuestion C: The abstract of a paper is found to be the most appreciated summary in the evaluation. How do you interpret this result in the context of your proposed citance-based summarization approach?\n\nQuestion D: Could you elaborate on the choice of using GPT-4 generated ground truth summaries and manually verifying them? How do you ensure the reliability and lack of bias in the evaluation process, given that manual generation of ground truth summaries might offer a more robust evaluation?\n\nQuestion E: The number of real evaluation data points for the summarization task seems limited, as only 15 papers and 25 randomly chosen citances are used. How representative do you believe this evaluation is for drawing broader conclusions about the effectiveness of the proposed approach? Have you considered expanding the evaluation dataset to ensure more robust results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comments and questions. We first address the rejection reasons (enumerated as R1, R2 etc., ) followed by answers to specific questions.\n\n### R1. Absence of human-generated summaries\n\nWe acknowledge the desirability of having human written, high-quality summaries for each example in the corpus. However, just like in our case, achieving this ideal isn’t feasible for many third-party large scale summarization datasets either due to resource constraints and the inherent cognitive demands associated with the annotation. Notably, well-known datasets such as CNN DailyMail, XSum, and Arxiv adopt heuristics from the news web pages or already available abstracts instead of (new) human-written summaries for each example to train summarization models.\n\nGiven our corpus's substantial scale with nearly 540K documents and 4.6 million citances, along with the manifold context variants for query-based summaries, manual annotation for all instances becomes prohibitively costly. Thus, as a foundational step, we validated several state-of-the-art large language models (LLMs) for zero-shot summarization, supported by rigorous automatic and manual evaluation of summary quality. Additionally, we introduce a benchmark comprising 100 summaries—typically standard for manual evaluations in summarization studies—assessed for coverage, coherence, and focus.\n\nThe optimal model and query configurations we provide can now be harnessed by the community for dataset expansion, including the creation of domain-specific subsets (e.g., medicine, physics, computer science).\n\n\n### R2. Sole focus on LLMs\nA main reason to investigate modern LLMs is their superior capability at generating high quality summaries in a zero-shot manner in comparison to trained supervised models (see table of automatic metric scores provided below). Notably, Goyal et al. (2022) News Summarization and Evaluation in the Era of GPT-3,  demonstrated the marked preference for GPT-3 summaries among humans, especially those generated using simple prompts. Unlike supervised models, LLMs are not prone to dataset-specific challenges, such as learning simplistic patterns or positional biases—a recurrent issue. Additionally, LLMs offer enhanced control over summary characteristics like length, style, and tone by fine-tuning their prompts.\n\nWe consciously opted against extractive summarization due to its tendency to produce incoherent or excessively verbose outputs—sometimes even retaining numeric details or formulas verbatim from the target paper—which could undermine the intended seamless reading experience. In contrast, abstractive summaries address these concerns and contribute to a more cohesive output.\n\n\n### R3. Abstract as a strong baseline\nOur paper aims to explore a new and relevant task—contextualized summarization, as the review noted. Our initial steps involved constructing a large-scale dataset, investigating SOTA LLMs, and comparing them to the familiar abstract baseline. Although annotators favored abstracts, it's likely because our dataset had more \"general\" citations not needing specific details from the target paper. This finding is important: it guides future choices on when to show abstracts (for general cases) or our summaries (for specific cases).\n\nWe aim to further evaluate with more \"specific\" citations that demand details from the target paper. This will give us a clearer view of how well our contextualized summaries perform compared to abstracts.\n\n### R4. Manual generation of ground truth summaries could potentially lead to more accurate and unbiased evaluation results.\nAs mentioned in response to R2, prior work established that zero-shot summaries from GPT-3 were already better than the ground-truth summaries in multiple news datasets. The process of manually annotating summaries for papers in our case requires: (1) reading a citance, (2) the entire target paper, (3) fetching relevant content, (4) summarizing it within a fixed length budget. Moreover, this would still be an “incomplete set of summaries” as just one human reference might be insufficient (or even erroneous) for a given example. Clearly, this is infeasible from a budget perspective to do at scale. LLMs can offer multiple viable references in a cost-effective manner, even though human summaries are considered unbiased.\n\nContrary to assuming no bias in human summaries, summarization heavily relies on background knowledge (education), topic familiarity, and the intended audience for which they are summarizing. For differences between expert and novice summarizers, see (1) Ruth Garner. Efficient text summarization costs and benefits. (2) Ann L Brown, Jeanne D Day, and Roberta S Jones. The development of plans for summarizing texts. \n\nThus to balance for accuracy as well as feasibility, we bootstrapped the ground-truth generation by leveraging the SOTA GPT-4 model, followed by manual checks to identify potential errors.\n\n### R5. Small sample size for manual evaluation\nOur evaluation encompassed 100 summaries, with each document receiving 3 judgments, yielding a total of 300 data points for result computation (lines 466 to 469).This is double in size compared to the evaluation by the related work CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision which evaluated only 50 examples, each with 2 annotators. While additional examples certainly hold value, our selection of 100 aligns with established norms in the field (see SummEval: Re-evaluating Summarization Evaluation).  \n\nWe now answer the direct questions.\n\n__Question A: Regarding the dataset could you clarify if any summary is provided for the citance-cited_paper pair (summary length is 117 on avg for the provided dataset)? What metadata are included in the dataset?__\n\nFor each paper, the dataset contains (1) the citance text with its two contexts, (2) the extracted relevant content from the cited paper, and (3) three contextualized summaries. However, the summaries are only created for the 100 examples we manually evaluated to determine the best combination of citance context and LLM. The average summary length reported is over those bootstrapped by GPT4 for these 100 examples. Summaries for the remaining examples in the dataset will be provided as part of the final corpus release as the model inference for almost 13 million examples (4.6 million citances -> 3 citance contexts each, including citance as query -> 3 contextualized summaries including citance as query) is time and cost intensive for the computing resources at our disposal. \n\n\nThe following JSON illustrates one complete example from our dataset.\n\n```\n{\n  \"citance_No\": 5,\n  \"citing_paper_id\": 51871042,\n  \"title\": \"DeepPavlov: Open-Source Library for Dialogue Systems\",\n  \"citing_paper_authors\": \"D Jason, Kavosh Williams, Geoffrey Asadi, Zweig\",\n  \"citance_marker\": \"(Williams et al., 2017)\",\n  \"citance_section\": \"Implemented Models and Skills\",\n  \"citance\": \"The skill implements Hybrid Code Networks (HCNs) described in (Williams et al., 2017).\",\n  \"prev_sentence\": [\n    \"Some of them are available for interactive online testing.\"\n  ],\n  \"next_sentence\": [\n    \"The model is configurable: embeddings, slot filling component and intent classifier can be switched on and off on demand.\"\n  ],\n  \"reference_paper_title\": \"Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning\",\n  \"reference_paper_link\": \"13214003\",\n  \"top_2_paragraph_retrieval\": {\n    \"similar-BM25\": \"This paper presents a model for end-to-end learning, called Hybrid Code Networks (HCNs) which addresses these problems...\",\n    \"similar-keywords-BM25\": \"We compare to four past end-to-end approaches (Bordes and Weston, 2016; Liu and Perez, 2016; Eric and Manning, 2017; Seo et al., 2016) . We emphasize that past approaches have applied purely sequence-to-sequence models, or (as a baseline) purely programmed rules (Bordes and Weston, 2016)...\",\n    \"citance-SciBERT\": \"For optimization, we selected a policy gradient approach (Williams, 1992) , which has been successfully applied to dialog systems...\",\n    \"citance-BM25\": \"By contrast, Hybrid Code Networks are a hybrid of hand-coded rules and learned models. This paper presents a model for end-to-end learning, called Hybrid Code Networks (HCNs) which addresses these problems... \"\n  },\n  \"top_5_sentence_retrieval\": {\n    \"similar-BM25\": \"This paper presents a model for end-to-end learning, called Hybrid Code Networks (HCNs) which addresses these problems. By contrast, Hybrid Code Networks are a hybrid of hand-coded rules and learned models...\",\n    \"similar-keywords-BM25\": \"By contrast, Hybrid Code Networks are a hybrid of hand-coded rules and learned models. This paper presents a model for end-to-end learning, called Hybrid Code Networks (HCNs) which addresses these problems...\",\n    \"citance-SciBERT\": \"RL policy updates are made after each dia-log. Both the RNN and the developer code maintain state. HCNs also use an RNN to accumulate dialog state and choose actions. In each of these architectures, the RNN learns to manipulate entity values, for example by saving them in a memory...\",\n    \"citance-BM25\": \"By contrast, Hybrid Code Networks are a hybrid of hand-coded rules and learned models. This paper presents a model for end-to-end learning, called Hybrid Code Networks (HCNs) which addresses these problems...\"\n  },\n  \"contextualized_summaries\": {\n    \"abstract\": \"End-to-end learning of recurrent neural networks (RNNs) is an attractive solution for dialog systems; however, current techniques are data-intensive and require thousands of dialogs to learn simple behaviors. We introduce Hybrid Code Networks (HCNs), which combine an RNN with domain-specific knowledge encoded as software and system action templates...\",\n    \"GPT_citance_SciBert_top5\": \"After every dialogue, RL policy updates occur. State is maintained by both the RNN and the developer code. HCNs employ an RNN to gather dialogue state and select actions as well...\",\n    \"GPT_similar_BM25_top2\": \"This paper introduces Hybrid Code Networks (HCNs), a model for end-to-end learning that combines learning an RNN with domain knowledge expressed via software and action templates...\",\n    \"LLaMA_CoT_citance_SciBert_top5\": \"The RL policy is updated after each dialogue. Both the RNN and the developer code retain state. HCNs also use an RNN to accumulate dialogue state and select actions...\",\n    \"Vicuna_similar_BM25_top2\": \"This paper presents a new model called Hybrid Code Networks (HCNs) for end-to-end learning. HCNs allow developers to express domain knowledge and achieve better performance with less training data compared to existing techniques...\" \n  }\n}\n\n```\n\n__Question B: In your evaluation, you focus solely on the use of LLMs. Have you considered including standard summarization models or extractive baselines in your comparison to provide a more comprehensive evaluation?__\n\nWe did experiment with multiple standard summarization models such as BART, T5, and BERT-Extractive. However, none of these models was close enough to the ground truth summaries as evaluated by automatic metrics (Table 3 in the paper). For completion, we provide the scores of these models for both retrieval contexts (top-2 paragraphs and top-5 sentences), alongside the LLMs. \n\n\nThe tables below show the performance of standard summarization models (BART, T5, BERT-Extractive) compared to the LLMs. In most of the scenarios, standard summarization models are outperformed by the LLMs. Thus in our approach, we investigated LLMs extensively.\n\nBoth tables are simple the Table 2 from the paper, extended by scores for the standard summarization models\n\n**Top-2 paragraphs retrieved as the to-be-summarized content from the cited paper.**\n\n\n| Model                 | BERTScore | R-1  | R-2  | R-L  |\n| ---------------------  | --------- | ---- | ---- | ---- |\n| ***similar-bm25***    |           |      |      |      |\n| __LLMs__              |           |      |      |      |\n| Alpaca                | 0.343     | 47.3 | 25.5 | 44.9 |\n| Falcon                | 0.401     | 48.2 | 27.1 | 45.0 |\n| LLaMA-CoT             | 0.448     | 53.0 | 31.9 | 50.5 |\n| Vicuna                | 0.465     | 58.7 | 35.4 | 55.8 |\n| __Standard Models__   |           |      |      |      |\n| BART                  | 0.249     | 39.9 | 18.7 | 37.5 |\n| Bert-Extractive       | 0.345     | 34.2 | 16.0 | 31.6 |\n| T5                    | 0.331     | 40.0 | 19.5 | 37.9 |\n| ***citance-SciBERT*** |           |      |      |      |\n| __LLMs__              |           |      |      |      |\n| Alpaca                | 0.390     | 54.3 | 32.2 | 52.0 |\n| Falcon                | 0.413     | 52.1 | 29.6 | 48.9 |\n| LLaMA-CoT             | 0.497     | 54.7 | 32.9 | 52.5 |\n| Vicuna                | 0.431     | 56.7 | 34.2 | 53.9 |\n| __Standard Models__   |           |      |      |      |\n| BART                  | 0.246     | 42.8 | 21.3 | 39.5 |\n| Bert-Extractive       | 0.334     | 35.8 | 19.6 | 33.7 |\n| T5                    | 0.378     | 46.8 | 23.4 | 44.2 |\n\n**Top-5 sentences retrieved as the to-be-summarized content from the cited paper.** \n\n| Model                 | BERTScore | R-1  | R-2  | R-L  |\n| --------------------- | --------- | ---- | ---- | ---- |\n| ***similar-bm25***    |           |      |      |      |\n| __LLMs__              |           |      |      |      |\n| Alpaca                | 0.616     | 56.2 | 35.4 | 54.8 |\n| Falcon                | 0.649     | 57.5 | 35.6 | 55.2 |\n| LLaMA-CoT             | 0.707     | 61.2 | 38.6 | 60.0 |\n| Vicuna                | 0.551     | 57.2 | 34.3 | 54.9 |\n| __Standard Models__   |           |      |      |      |\n| BART                  | 0.507     | 54.2 | 28.8 | 52.2 |\n| Bert-Extractive       | 0.571     | 38.7 | 21.0 | 37.4 |\n| T5                    | 0.344     | 36.8 | 16.5 | 35.5 |\n| ***citance-SciBERT*** |           |      |      |      |\n| __LLMs__              |           |      |      |      |\n| Alpaca                | 0.595     | 56.6 | 34.7 | 55.1 |\n| Falcon                | 0.656     | 56.8 | 36.2 | 55.3 |\n| LLaMA-CoT             | 0.748     | 62.9 | 40.6 | 60.9 |\n| Vicuna                | 0.607     | 58.8 | 36.0 | 56.6 |\n| __Standard Models__   |           |      |      |      |\n| BART                  | 0.534     | 57.3 | 31.4 | 55.2 |\n| Bert-Extractive       | 0.551     | 39.4 | 21.8 | 38.1 |\n| T5                    | 0.368     | 38.6 | 15.6 | 36.5 |\n\n\n__Question C: The abstract of a paper is found to be the most appreciated summary in the evaluation. How do you interpret this result in the context of your proposed citance-based summarization approach?__\n\nAs described in the response to R3 of this review, we found the abstract to be a strong baseline in cases where the citance was generic and did not require an extensive reading of the cited paper from the reader. On the other hand, our contextualized summaries gained preference from annotators for specific citances. \n\nWe acknowledge that this could have been better quantified in the paper and will do so. Also, we will perform a fine grained analysis of the citances and sample accordingly to balance generic and specific citances for which the contextualized summaries can be reevaluated together with the abstract.\n\n__Question D: Could you elaborate on the choice of using GPT-4 generated ground truth summaries and manually verifying them? How do you ensure the reliability and lack of bias in the evaluation process, given that manual generation of ground truth summaries might offer a more robust evaluation?__\n\nPlease see the response to R4 above. \n\n__Question E: The number of real evaluation data points for the summarization task seems limited, as only 15 papers and 25 randomly chosen citances are used. How representative do you believe this evaluation is for drawing broader conclusions about the effectiveness of the proposed approach? Have you considered expanding the evaluation dataset to ensure more robust results?__\n\nIn addition to the response provided in R5, it's important to elaborate that our choice to evaluate summaries exclusively from the ACL anthology stems from the availability of expert annotators within this specific domain. We recognize the necessity of a large and diverse annotator pool encompassing multiple domains for a robust evaluation across various subject areas. This is a direction we plan to pursue in the future.\n\nHowever, it's worth noting that our approach doesn't rely on domain-specific features for summarization. As such, it can be readily adapted to other domains without any customization.\n\n \nWe hope that we have answered your questions in sufficient detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new dataset. This dataset can be highly valuable for the NLP community, providing a new resource for research and development in the field of summarization and related areas.\n\n2. The paper addresses an interesting and important task in the field of summarization - contextualized summarization of scientific papers. By focusing on generating summaries that are dependent on the citances in the citing paper, the proposed approach offers a novel perspective on summarization.\n\n3. The evaluation methodology of the proposed approach is robust and considers different settings (within the LLMs scope).", "reasons_to_reject": "1. One of the weaknesses of this paper is that the dataset provided, Context-SciSumm, lacks large-scale human-generated summaries. While the dataset contains a large number of citances and scientific papers, the absence of human-generated summaries can limit its usefulness for training deep learning models and evaluating against standard summarization benchmarks.\n[EDIT: I better understand the difficulty of having annotations at scale.]\n\n2. The authors' exploration in the paper is focused solely on the use of Large Language Models (LLMs) for contextualized summarization. The lack of comparison with standard summarization models or even extractive baselines raises concerns about the thoroughness of the evaluation and the performance of the proposed approach compared to existing methods. \n[EDIT: the results reported in the response about other extractive/abstractive methods should be at least in the Appendix].\n\n3. The paper's claim of providing informative summaries tailored to citances might be weakened by the fact that, in the end, the abstract is found to be the most appreciated summary. This discrepancy could raise questions about the effectiveness and utility of the proposed approach in practice.\n\n4. The use of GPT-4 generated ground truth summaries, albeit manually verified, might raise doubts about the reliability and bias in the evaluation process. Manual generation of ground truth summaries could potentially lead to more accurate and unbiased evaluation results.\n\n5. The paper lacks clarity regarding the number of real evaluation data points for the summarization task. The selection of only 15 papers from the ACL anthology and 25 randomly chosen citances for evaluation might not be representative enough to draw general conclusions about the effectiveness of the proposed approach.\n[EDIT: I acknowledge the difficulty of having a strong set of annotators, but the number of papers seems to limited. Maybe the suggestion is to increase the number of papers while reducing the number of citances?]", "questions_to_authors": "Question A: Regarding the dataset could you clarify if any summary is provided for the citance-cited_paper pair (summary length is 117 on avg for the provided dataset)? What metadata are included in the dataset?\n\nQuestion B: In your evaluation, you focus solely on the use of LLMs. Have you considered including standard summarization models or extractive baselines in your comparison to provide a more comprehensive evaluation?\n\nQuestion C: The abstract of a paper is found to be the most appreciated summary in the evaluation. How do you interpret this result in the context of your proposed citance-based summarization approach?\n\nQuestion D: Could you elaborate on the choice of using GPT-4 generated ground truth summaries and manually verifying them? How do you ensure the reliability and lack of bias in the evaluation process, given that manual generation of ground truth summaries might offer a more robust evaluation?\n\nQuestion E: The number of real evaluation data points for the summarization task seems limited, as only 15 papers and 25 randomly chosen citances are used. How representative do you believe this evaluation is for drawing broader conclusions about the effectiveness of the proposed approach? Have you considered expanding the evaluation dataset to ensure more robust results?", "paper_topic_and_main_contributions": "This paper introduces a novel approach for contextualized summarization of scientific papers, focusing on generating informative summaries that depend on the citances (citation texts) in the citing paper. The main contribution is a method that grounds the summaries in the context of the citances, helping readers quickly find relevant information.\n\nThe paper also presents a large dataset from the computer science domain. The proposed approach utilizes LLMs to tailor the summaries to the citances, offering a potential solution for readers seeking specific connections between citing and cited papers.\n\nThe contributions of the paper are:\n1. Introduction of a novel approach for contextualized summarization of scientific papers.\n2. Generation of informative summaries dependent on the citances (citation texts).\n3. Generation of a large dataset from the computer science domain.\n4. Exploration of the use of Large Language Models (LLMs) to tailor summaries to the citances.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Faceted Summarization seems to be relevant (while not citance based) to the present work:\n\nMeng, Rui, et al. \"Bringing structure into summaries: a faceted summarization dataset for long scientific documents.\" arXiv preprint arXiv:2106.00130 (2021).\n\nLa Quatra, Moreno, Luca Cagliero, and Elena Baralis. \"Exploiting pivot words to classify and summarize discourse facets of scientific papers.\" Scientometrics 125 (2020): 3139-3157.\n\nLi, Lei, et al. \"CIST@ CLSciSumm-19: Automatic Scientific Paper Summarization with Citances and Facets.\" BIRNDL@ SIGIR 54 (2019)."}}]

Author rebuttal:

Meta-review: This paper focuses on scientific paper summarization and investigates the grounding of generated summaries in the context of the citances (i.e. citation contexts), aiming at producing more informative summaries to assist readers in understanding a citation without the need to refer to the cited paper. Reviewers see many merits in the paper (adresses an interesting and timely task, new dataset, extensive experiments), and some of their initial concerns were addressed by the comprehensive authors's responses. After the discussion period, I feel that there are no real major issues with this work but reviewers have remaining concerns mainly about the evaluation methodology (e.g. GPT generated ground truth, limited scope of manual evaluation).
Decision: Accept-Findings