OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization
Authors: Shmuel Amar
Liat Schiff
Ori Ernst
Asi Shefer
Ori Shapira
Ido Dagan
Venue: EMNLP 2023 Main (2023)
Keywords: Aspect-based Summarization
Datasets
Annotation
Multi-document Summarization

Abstract: The performance of automatic summarization models has improved dramatically in recent years. Yet, there is still a gap in meeting specific information needs of users in real-world scenarios, particularly when a targeted summary is sought, such as in the useful aspect-based summarization setting targeted in this paper. Previous datasets and studies for this setting have predominantly concentrated on a limited set of pre-defined aspects, focused solely on single document inputs, or relied on synthetic data. To advance research on more realistic scenarios, we introduce OpenAsp, a benchmark for multi-document open aspect-based summarization. This benchmark is created using a novel and cost-effective annotation protocol, by which an open aspect dataset is derived from existing generic multi-document summarization datasets. We analyze the properties of OpenAsp showcasing its high-quality content. Further, we show that the realistic open-aspect setting realized in OpenAsp poses a challenge for current state-of-the-art summarization models, as well as for large language models.

OPENASP: A Benchmark for Multi-document Open Aspect-based
Summarization
Shmuel Amar', Liat Schiff"", Ori Ernst', Asi Shefer', Ori Shapira", and Ido Dagan'
"One AI Sshmulikamar liatschiff1, oriern]@gmail. com com
Bar--Ila University
3 Ammaonn
asi@oneai. com orishapeamazon.com dagan@cs. biu. ac.
Abstract
Generic Summary:
The performance of automatic summarization models has improved dramatically in recent years. Yet, there still gap meeting spe- cific information needs of users in real-world scenarios, particularly when a targeted sum- mary is sought, such in the useful aspect- based summarization setting targeted in this paper. Previous datasets studies for this set- ting have predominantly concentrated ited set of pre-detined aspects, focused solely on single document inputs, or relied on syn- thetic data. To advance research on more realis- tic scenarios, we introduce OPENASP, a bench- mark for multi-document open aspect-based based summarization. This benchmark is created us- ing novel and cost-effective annotation proto- col, by which open aspect dataset is derived from existing generic multi-document summa- rization datasets. We analyze the properties of OPENASP showcasing its high-quality content. Further, we show that the realistic open-aspect setting realized in OPENASP poses challenge for current state-of- the-thr-rt summarization mod- els, as well as for large language models.
larger Chinaa Indlia combined.
Ninety-elghi percent the continent is under
It is drier than the Sahara desert.
Antarctica has been Important on three fronts geopolitical, future economic development and seientifie research.
In 1959 an international treaty placed hold on the territorial claims seven nations, placed Antarctica olf-limits for millitary opened scientific
lim-
Antarctica believed to hold unknown quantities of strategle metals billions of pallons
Many research scientists and Antarctic acminist belleve that without tse possibillly ecosonile payback, funding for Antaretica's very expensiie may
current levels.
Figure I: A generic summary from DUC on the topic Antarctica with some of the marked sentences for each of the 4 identified aspect labels: Overview of Antarctica, Research in Antarctica, Antarctica's 's importance, and Territorial claims. The concatenated sentences of aspect construct the respective aspect- aspect-based summary.
In QFS, a query is highly flexible and can target specific information within particular text. In con- trast, ABS datasets traditionally predefined small sets of generic subtopics within a common topi- cal category on which aspect-based summaries are generated, such as geography and recovery aspects in any article about an earthquake (Amplayo al., 2021). Open-ABS (OABS; Tan al., 2020), al- lows aspects to differ for each source text, yet still just as subtopics in the text.
1 Introduction
When faced with large body of text, a summary is an effective means to get concise version of the salient content. However, informational needs of users vary, calling for summarizers that can focus summary around given request. The summariza- tion community has addressed this demand mainly through query- query-focused summarization (QFS) and aspect-based summarization (ABS). Accordingly, several datasets and benchmarks have been com- piled over time to enable research on these tasks (see Table 1).
Collecting datasets for these tasks is a major obstacle. This kind of data does not naturally oc- cur in available resources, and manually annotat- ing is highly burdening, particularly for the multi- document setting. Indeed, the very few existing OABS datasets are synthetically gathered. addi- tion, they address single-document setting, even though worl-world scenarios involve information
"Equal contribution.
Part the research was conducted during an internship
at One AI.
Work done in cooperation with Bar-Ilan University (ex- ternal and not related to the author's 's work at Amazon).

---

Dataset
Domain
MDS
Open
Collection
Queries Aspects
Instances
SQuALITY (Wang et 2022)
Scifi
manual
437
625
QMSum (Zhong et al, 2021)
Meeting
manual
1,566
1,808
AQuaMuSe (Kulkamni al., 2020)
General
automatic
5,519
7,168
TD-QFS (Baumel et al., 2016)
Medical
manual
40
120
FacetSum (Meng 2021)
Science
automatic
4
60,532
MA-News (Frermann and Klementiev, 2019)
News
X
automatic
6
286,701
OABS ABS
AspectNews (Ahuja al., 2022)
News
X
X
manual
4
400
SPACE (Angelidis 2021)
Reviews
manual
6
900
WikiAsp (Hayashi et al., 2021)
Wikipedia
X
automatic
200
504,546
AnyAsp (Tan et al, 2020)
News
automatic
363,440
2,204,270
OASUM (Yang et 2022)
Wikipedia
automatic
1,045.895
3,747,569
OPENASP (Ours)
News
manual
1,266
1,310
Table I: Prominent datasets query- query-focused summarization (QFS) and abstract- summarization (ABS).
Our annotation protocol enables efficient collection of high-quality multi-document open-aspect-basee summaries. "# Queries| AAspects is the number unique queries or aspects appearing in the dataset, and "# Instances" is the number of data instances in the dataset (document set query query/aspect Jaspect + summary).
navigation within multiple documents on a topic. In this work, we first propose novel efficient protocol to manually derive high-quality multi- document open aspect-based summaries from standard multi-document summarization (MDS) datasets. Through crowdsourcing. open aspects and their summaries are extracted from generic reference summaries.
for instance, focused summaries around questions targeting distinct matters within the document, e.g. summarizing around "What is the CPA and what does it do?" in certain story.
Early research ABS (Hu and Liu, 2004; Titov and McDonald, 2008) recognized the need for more structured information around subtopics of the source text. Relevant datasets (middle of Ta- ble 1) focus on recurring aspects within a domain, and approach this by pre-defining a fixed list of aspects, e.g., "service" and "price" in all restau- rant reviews. Some datasets expanded to multi- document inputs (Angelidis et al., 2021; Hayashi et al., 2021) which is a more realistic setting when seeking topical information.
Applying our protocol, we present OPENASP, OABS dataset for the multi-document setting. The aspects and respective summaries are based on the prominent DUC (NIST, 2002) and Multi- News (Fabbri al., 2019) datasets. See Figure summaries
for an example of open-aspect-baree extracted from generic summary. The dataset con- tains 1,310 aspect-based summaries, split to train, validation and test sets, enabling methodological modeling for the task. We further implement and analyze several baseline models that demonstrate the challenging nature of the task and dataset, even for recent pegh-performing large language models.
An additional direction stemming from ABS permits open aspects that still concisely target subtopics, but can be unique for the individual in- put text (bottom of Table 1, with more details in Table 12 in the appendix). The existing OABS datasets, namely AnyAspect (Tan et al., 2020) and OASUM (Yang et al., 2022), are synthetically com- piled and only address single document inputs. In Aspesp,, the named entities within the document are considered the aspects, and the corresponding summaries include any source sentence mentioning the respective entity. In OASUM, the aspects and the summaries were extracted from Wikipedia arti- cles, where Wikipedia sub-titles are the aspects and their summaries are automatically extracted from the article's abstract section via a greedy lexical similarity method. Applying such artificial meth- ods yield lower-quality input documents, aspects and expected output summaries.
2 Background
QFS is long-standing task that addresses the need summarize around specified user re- quest (Dang, 2005). query is inherently fluid, allowing great variance in length, specificity and format (Carmel et al., 2006). Corresponding datasets, such as those at the top of Table 1, mainly evolved around queries for specific information needs within the source text(s). Wang al. (2022),
'The OPENASP dataset is available at thh com/liatschiff/Oeaa

---

Collecting datasets for the ABS task poses challenging undertaking that includes reading large input texts and writing out the focused summaries. Unlike in generic summarization, where large sources of summaries can be scraped from the web (Hermann et al., 2015; Grusky et al., 2018; Fabbri et al., 2019), summaries for our setting are not generally available. Meanwhile, manu- ally writing high quality aspect-based summaries from scratch is an expensive labor-intensive task. Wang al. (2022) reported 20-40 minutes just for reading a 5000-6000 word story. Summarizing multi-document sets is even more complex since the total input-length may be much larger (e.g.. tens of thousands of words; see Section 6.2), while information-overlap further requires content con- solidation by the summarizer. For instance, Dang (2005) reported 5 hours of labor for generating one multi-document summary by an expert. Employing crowdsource workers, on the other hand, has been shown to lead to poor extractive multi-document summaries (Lloret et al., 2013). Our protocol ex- ploits existing MDS benchmarks and applies con- trolled crowdsoureing (Roit et al., 2020), and is hence substantially cheaper and more efficient than previous manual collection processes.
et al., 2021; Angelidis al., 2021; Angelidis and Lapata, 2018).
The OABS task definition follows previous work (Tan et al., 2020; Yang et al., 2022), and is extended to the multi multi-document setting as follows: Given set of documents D the same topic and an arbitrary aspect label a, the task is to output short aspect-based summary S The summary should consolidate salient information from the document set that is relevant the aspect.
Annotation Protocol
As emphasized in Section 2. manually collecting aspect-based summaries is very costly. We pro- pose novel and cost-effective protocol for gen- crating aspect-based multi mulli--docmment summaries, executed through controlled crowdsourcing (Roit et al., 2020) and a specially-designed annotation tool (Figure 3 in the Appendix). The key idea of our protocol is the extraction of gold aspect-based sum- maries from generic summaries in existing MDS datasets. Notably, the process is accomplished by reading the generic summary text only, as described below, while saving the strenuous need to read the entire set of source documents and to write the aspect-based summary from scratch.
Our OPENASP dataset addresses all the above issues by supporting open-aspects in the multi- document setting, with manually annotated real- istic summaries. Moreover, the annotation protocol can be applied across any available generic summa- rization dataset produce more like-quality aspect-based summaries.
4.1 Collecting Aspects and Summaries
From an existing MDS dataset, we gather pairs con- sisting of a document set D and respective generic summary G. An annotator reads G and identifies prominent aspects within it, specified by aspect la- bels aj Q2, amn- For each identified aspect label a₁ the annotator selects the relevant sentences from G. The concatenation of these sentences, retaining the original sentence-order from G, produces the corresponding aspect-based summary S Accord- ingly, we establish m new aspect-based summaries for D as instances for the dataset. Notice that summary is abstractive with respect to D, being comprised of sentences from the abstractive generic reference summary.
3 Task Formulation
Following prior work (Ahuja et al., 2022; Yang et al., 2022), given a set of texts about a topic, we define an aspect as central theme within a topic. The aspect can be referred by certain phrases. denoted aspect labels. As an example, Research in Antarctica and Territorial claims aspect labels of the Antarctica topic (see Figure 1).
Similar to previous work on ABS (Hayashi et al., 2021; Angelidis et al., 2021), our aspect label is short and concise. In contrast, our aspect defi- nition is open allowing ad-hoc aspects with free- form labels, contrary to having pre-defined domain- specific aspects. Relative a query in query- focused summarization (QFS; Dang, 2005), which might specify a complex information need, our as- pects are restricted to relevant subtopics. (Hayashi
In our process, we favor extraction of fewer but high quality aspects from a generic summary. Specifically, our protocol instructs annotators to detect the aspects that are central in the generic summary, and avoid circumstantial aspects. Al- though our protocol does not exhaustively extract aspects for the topic, the main sub-topics found in the generic summary establish a reliable and suf- ficient sample of aspects for addressing the multi-

---

Split
# Topics
# Instances
# Docs
cally filtered out samples with invalid source docu- ments, avoid consequential hallucinations in the summaries (see Appendix D.2). The large scale of MultiNews allowed further filtration to capture only instances with summaries of 350-880 words, to increase the potential yield of aspect- sum- maries. For all source data, we excluded document- set instances that discuss topics presented as list of related events (e.g., daily news briefs vari- ous unrelated incidents of the same kind), since the generic summaries of such instances typically contain few subtopics, if any.
Test
192
596
6,536
Valid
82
238
2,168
Train
145
476
4,878
Table 2: The size of the OPENASP dataset splits. "#f Topics" denotes the number document sets, "# In- stances" is the total number of aspect-based summaries, and "# Does" is the total number of source documents.
document open ABS task, for training and evaluat- ing models. The full annotation guidelines appear in Appendix A.
5.2 Dataset Collection
Critically, the described protocol avoids read- ing through the full document set and writing text for the summary. Instead, each aspect summary comprises subset of generic summary sentences. We suggest that summary quality is maintained since the extracted summaries are based on depend- able generic gold summary sentences. The validity of our protocol is based on two assumptions: (1) the aspect-related sentences extracted from generic summaries cover well the prominent information about the aspect within the full source document- set; (2) the aspect-based summaries preserve the coherence borrowed from the source summaries. We show that these assumptions indeed hold by assessing our collected dataset Section 6.1.
We followed the annotation protocol described in Section 4.1. Specifically, used controlled crowd- sourcing (Roit et al., 2020) for selecting 3 annota- tors on Amazon Mechanical Turk that successfully completed an introductory summary annotation task and correctly answered followup questions on the task guidelines."
Our workers annotated 236 generic summaries from MultiNews and 208 from DUC. From a total of 444 generic summaries, annotators ex- tracted 1,455 aspect-based multi-document sum- maries. We (paper authors) then applied the cura- tion procedure (Section 4.2) on 1,173 aspect based summaries detailed in Appendix A.2.5 Out of the reviewed summaries, we modified 152 aspect labels, edited sentence choice of 48 summaries, and completely rejected 94 aspect based summaries (92% pass rate). Overall, we gathered 1,361 sum- maries for OPENASP, averaging 3 aspect-based summaries per topic (document set instance), and costing $0.5 per summary.
4.2 Curation Phase
We propose optional curation phase for cleaning the annotated aspect labels and corresponding sum- maries. The process encompasses a manual review, by an expert, of the aspect label and aspect-basee based summary only. The reviewer can edit the aspect label, remove irrelevant sentences from the sum- mary, completely reject the aspect. Similar to the annotation protocol, the curation phase avoids the expensive task of reading the source documents.
We split OPENASP into train, validation and test sets, keeping the original MultiNews splits and splitting DUC datasets by years (Appendix D. We set aside 51 summaries (from 16 topics) from the test and validation sets, denoted analysis-tent and analysis-val sets, for quality assessment and modeling (Sections 6 and 8). Statistics on the final OPENASP sizes appear in Table 2.
5
The OPENASP Dataset
5.1 Source Data
We exploit 2 prominent MDS datasets that con- tain reference summaries with at least 200 words demonstrate our protocol robustness: DUC,2 a high-quality and
Phttpss//www wwwrkt com/
dataset, and MultiNews (Fabbri et al., 2019), with data scraped from newser. com. For MultiNews, we automati-
"Workers were paid $0.9 and $0.6 bonus per task with an average task completion time about minutes, resulting $15. 00/0r as recommended by Whiting et al., 2019.
"Staring from the test and validation sets, and moving on to the train sets. We eventually excluded the MultiNews train set instances in the curation process as pass rates for the other sets were high enough.
#duc.nist nist. gov; we use DUC 2006-07, 07, and DUC 2001-02 task 2 (that contain 200-word summaries).

---

6 OPENASP Assessment
Coherence 4.7 (0.92)
Consistency
Fluency
Relevance
Aspect Rel.
4.8 (0.55)
5.0 (0.22)
4.6 (0.68)
4.7 (0.57)
We next examine the quality of the collected data, and then analyze its properties.
Table 3: Average (std) human evaluation ratings (1-5 scale) the five quality criteria, determined for 20 instances from the analysis-test set.
6.1 Dataset Quality
We applied manual evaluation process to verify the collected summaries' expected qualities. Fol- lowing Fabbri et al. (2021), summary should be measured for quality criteria: (I) relevance, the selection of important content from the source; (2) coherence, the quality of the collective structure of all sentences; (3) consistency, the factual align- ment between the summary and the summarized source; and (4) fluency, the linguistic quality of individual sentences. In the OABS setting, the aspect-relevance an additional expected quality (Amplayo et al., 2021; Angelidis al., 2021). This criterion inspects whether the summary includes information that relevant to the paired aspect.
Although summaries that are extracted from an- other text can easily suffer from coherence is- sues, this is rarely the case with our protocol. We found that 86% of consecutive sentence pairs in our aspect- espect-based summaries respectively con- secutive in the generic summary. Specifically, 60% of the ispect-based summaries are full continu- ous sentence sets from the generic summary. This phenomenon occurred naturally during annotation, without explicit instructions to follow such prin- ciple. Consequently, coherence is also maintained according to the source generic summaries. Over- all, coherence scored very high as well, validating our second assumption that our protocol generates coherent aspect-based summaries even as they are based on generic summaries.
We assess the 5 quality criteria on 20 aspect- based summaries sampled from the analysis-test set (Section 5.2). A summary scale for each criterion by one expert and reviewed by another. In case of disagreement, the two raters resolved the dispute through reconciliation."
rated 1-5
To empirically corroborate the quality of the aspects, we statistically analyzed the source sen- tences corresponding to each of the 51 aspects in our analysis sets. We found that, average, pect relates 13.5% of all source documents' sen- tences and appears 68% of the source documents, indicating its topical dominance, Furthermore, only 11 % of all aspect-related sentences refer to more than one aspect, indicating the high level of dis- tinctness of the aspects. Finally, on average, 40% of all source sentences in document set (topic) related to at least one of the aspects extracted for that topic, indicating the substantial coverage of the set of aspects with respect to the entire doc- ument set content. Taken together, these findings establish that the collected aspects are central to the topic, covered thoroughly by its documents and collectively cover main portion of its content. Overall, the aspect-based summaries represent- ing OPENASP were determined to exhibit high re- liability for the OABS task, consistent with that of the standard generic MDS datasets from which they were extracted.
The relevance and consistency criteria require comparison of the evaluated summary against the aspect-relevant information across the source document-set. Therefore, for each aspect in the analysis sets, we extracted (via crowdsoureing) all the sentences in the corresponding document-set related to the aspect.? 7
The average ratings can be found in Table 3. The high relevance score of 4.6 supports our first ex- traction protocol assumption (Section 4.1) that the aspect-based summaries cover the most important information about the subtopic, even though they originate from generic reference summaries. Con- sistency is expectedly sturdy as well, since sen- tences are copied from gold generic summaries. Hence, consistency issues that are not already present in the generic summaries should not be introduced. Similarly, the fluency of sentences is adopted from that of the source reference sum- maries, which is almost flawless.
'The reviewer-to-ratte agreement was 0.753 linear weighted Cohen's 's Kappa, and the reviewer-t--reconcleed agreement was 0.847,, indicating "suhstantial" (0.6-0.8) and "almost perfect" (0.8-1.0) agreement respectively.
6.2 Dataset Analysis
We discuss properties of OPENASP that emphasize its underlying diversity from several angles. Details for these analyses are available in Appendix D. The input lengths, measured as the aggregated
"We measured 0.642 Cohen's 's Kappa for inter-rater agree- ment of per sentence aspect label, indicating "substantial" agrocment.

---

document lengths in topic, varies from several hundred tens of thousands of tokens (Figure 11 the Appendix), averaging 7,930 words. The sum- mary length ranges from tens to hundreds tokens, with a median input-to-output (compression) ratio of 69:1.
25
OpenAsp n 1,310
Extractive fragment domsiyy
20
o 69:1
15
A topic contains 1-7 aspects with average of 3.1 aspects per topic (see Figure in the ap- pendix). The aspect labels are almost all lexically unique, repeating on average 1.1 times throughout the dataset. The aspect label is 1-10 words, av- eraging 3.6 words. Some examples of topics and aspects from OPENASP appear in Table 15. Since aspect labels in OPENASP are annotated sub- topics of their corresponding topic, and are flexibly scripted, the aspects naturally vary widely.
0.00 0.25
0.50
0.75
Extractive fragment coverage
Figure 2: OPENASP content diversity (Grusky et al., 2018). n and denote number of examples and me- dian compression ratio of OPENASP, respectively. The large area the plot indicates that the dataset contains aspect- based summaries with balance of extractive and abstractive fragments. (See Appendix D.4 for further explanations.)
Due to our annotation protocol, the summary- abstractiveness in source datasets (DUC and Multi- News) is transferred to the summaries in OPENASP. Accordingly, the aspect-basee summaries exhibit varying extents of abstractivness,
apparent in the diversity plot in Figure (Grusky et al., 2018; Fabbri et al., 2019). Consequently, OPENASP re- quires models to perform well on both extractive and abstractive forms of summarization.
our document-sets.
To cope with this we investigate two common schemas used in the MDS setting: (a) Filter-then- summarize (e.g.. Wang et al., 2022; Baumel et al.. 2016), runs sentence selector extract aspect- related sentences, and passes them marization model (7.1); )); (b) Recursive sunmariza- tion (e.g. Shapira and Levy, 2020) summarises sub- sets of documents separately, and then summarizes these summaries (7.2).
Finally, topic consists of 2-25 documents, with average of 10.4 documents per topic (Figure 8 in the appendix). Following Wolhandler et al. (2022), we find that the aspect-based summaries rely a varying number of corresponding souree docu- ments (Figure in the appendix). In practice, this means some summaries require handpicking in- formation from specific documents, while others require consolidating information from across the input document set.
sum-
7.1 Filter-then-ummmrree
Sentence selection. To slim down the input, one remedy is select sub-texts that are more likely to be included in a summary. In MDS in the news do- main, the conventional technique for this is the Lead method, which extracts a number of sen- tences from the beginning of each document, since these often contain the most crucial information in news report.
Overall, the analyzed properties expressly show the diversity of OPENASP and the ensuing chal- lenges of the task.
7 Baseline Models
this and the subsequent sections we demonstrate the challenges that our dataset lays out for sum- marization models, and suggest initial directions to cope with these challenges. A major hurdle to overcome is the large input length of docu- ment set, averaging ~8K tokens in our data, and stretching <3K (Section 6.2). Even with cur- rent advancements made to support growing input sizes, properly attending to relevant information in large input remains a hurdle. There were no feasible models available to us that would all of
However, in the case of ABS, there no guaran- tee that salient aspect-specific information appears at the beginning of each document. Standard ABS models with a closed set of aspects can train clas- sifier map sentence to aspect (c.g. Hayashi et al., 2021). In contrast, our open-aspect setting demands selector that is robust to any aspect. We leverage the Sentence-T5 model (Ni et al., 2022) as an unsupervised sentence selector. Specifically, a cosine similarity score is computed between the

---

recursively summarized to generate the final sum- mary (see details in the Appendix B.2). We ex- dense vectors of an aspect label and sentence. producing a ranking of aspect-relevant sentences. For the summarization models we employ in the subsequent stage, we restrict the applied sentence selector to provide a set of sentences that fit within the input-size limit (IK or 4K tokens). An ap- proximately equal number of sentences from each document is used within the limit.
1024 Tokens
4096 Tokens
Sentence Selector
R
R
Lead Sentence-T5 31.9
16.8
19.0
17.8
55.9
15.4
24.1
37.3
34.4
71.8
21.0
32.5
Table 4: Fi scores, and corresponding Recall (R) and Precision (P) scores, comparing the two sentence se- lectors against gold sentence alignments from com- bined analysis -alid and analysis fest sets. In both lectors the top sentences that fit within 1024 or 4096 tokens are extracted, and evaluated separately.
Summarization. For the second stage of the de- composed procedure, employ three architec- turally different summarization models. The first two are seneenee-t--eqqeene models trained for generic summarization: BART (Lewis et al., 2020) was pre-trained on the CNN/DailyMail dataset (Hermann et al.. 2015) for single-document sum- marization, while PRIMERA (Xiao et al., 2022) was trained on the NultiNews dataset (Fabbri et al.. 2019) for the multi-document setting. Accordingly, the former has a limit of 1K tokens, and the lat- ter 4K. While PRIMERA is more suitable for our multi-document setting, BART has shown strong performance on single-document ABS (Tan et al., 2020; Meng et al., 2021; Wang et al., 2022) and is a worthy candidate. BART is adjusted for multi- document inputs with separation tokens between documents, following PRIMERA. We further fine- tuned the two models with the OPENASP train set, denoted BART Summ and PRIMERA Summ. respec- tively.
quire recursion for the 4K limit, however only about 10% don't fit within the 16K limit, rendering nearly end-to-end summarization on our test set. For the sake of completeness, we also ex- perimented with BART and PRIMERA in the recursive scheme, denoted Reeureerrrvve and PRIMERA ReuMERAR Recursive, respectively. Note that train- ing for this approach would require gold aspect- based summaries for subsets of the document set, which are not available. We therefore activated, in the recursive technique, the BART and PRIMERA systems that were fine-tuned for the filter-then- summarize approach above (Section 7.1).
Recursive
Baseline Evaluation and Analysis
To assess the overall capabilities of base- line models, we show an overall comparison of the methods ($8.1), an ablation analysis on the BART configurations ($8.2), and end with a human evaluation on our best baselines ($8.3).
In addition, we experiment with Ope- nAl's promising ChatGPT LLM, based on gpt-3 3-tturb-tubb-oooe 0001 (OpenAI, 2023), denoted ChatGPT Such prompt-based models have demonstrated promising results the zero-shot setting for summarization (Zhang et al., 2023), which we apply here as well. See Appendix B.2 for technical details.
8.1 Automatic Evaluation
Sentence selection. We first compare the Lead and Sentence-T5 sentence selectors, measuring the relevance of selected sentences to paired pects. To that end, we utilized the sentence alignments (between aspects and socument-set sentences) from the combined analysis-valid and analysis-test sets (Section 6),, consisting of 1,782 aspect-sentence pairs. For the two sentence selec- tors, we selected sentences up to cap of IK or 4K tokens, and measured the Fi scores between the gold alignment pairs and the resulting selector's pairs, over all aspects. As expected, the results in Table 4 strongly favor Sentence-TS. which directly focuses on a given aspect label. The lead sentences from the documents are much less relevant to any given aspect.
7.2 Recursive Summarization
A different approach handle the large input size is summarizing subsets of the input that fit the model size limit. The subset summaries are then
periment with ChatGPT on 4K or 16K input token-size, denoted ChatGPT Recursive. whose ro- bustness enables the recursive scheme without any fine-tuning. About 75% of our test instances re-
WWe concatenate documents (only with the select sentences from the first stage) with separator, with the aspect at the heginning and end of the input (inspired by Wang et al., 2022).

---

Base Model
Input
Fine
Sentence Selector
R-1
R-2
R-L
Fine Tuned
Aspect Input
Sentence Selector
Size
Tuned
R-1
R-2
R-I.
BARTSoun Sunn
IK
Sent-T5
32.4
8.3
18.7
PRIMERASImm
4K
S5
30.5
8.0
18.3
1
Sent-T5
32,4
8.3
18.7
PRIMERA
1K
Sent-T55
31.2
8.3
18.7
2
Lead
31.3
7.6
17.9
CharGPT ChatGiTsumm
4K
S5
33.7
9.4
19.8
3
Sent-TS
32.0
8.0
18.2
ChatGPT Recursive
4K
X
32.4
9.2
19.1
ChatGP
16K
X
31.8
8.1
18.6
4
Lead
27.5
5.3
15.7
5
Lead
25.4
5.0
15.0
Oracle
53.4
27.8
31.7
6
Oracleşa
40.6
14.5
23.2
Table 5: The ROUGE F1 scores of different summa- rization models on our test set. All variants include the aspect label part of the input.
Table 6: ROUGE F1 scores on different configurations of the BART summ model.
Summary quality. We next assess the aspect- based summaries of our baseline filter-then- summarize and recursive summarizers. We apply the commonly used ROUGE metrics (Lin, 2004), which measure the lexical overlap between the sys- tem and reference sopect-based summaries. Here we only apply the Sentence-TS sentence selector in the
Aspect in input. We first observe what happens if the aspect label is left out from the input to the summarizing model. When using the Sentence-TT selector, which already selects sentences relevant to the aspect, very slight improvement in perfor- mance is achieved with the aspect the input (row vs. row 3). However, there a much larger up- grade when the aspect is input with Lead sentences (row 2 vs. row 4). This indicates that simply provid- ing the requested aspect in the input indeed trains BART Summ to attend to aspect-relevant sentences.
the sumumarien- configuration, better of the two selectors, as observed above. Summ outperforms
Table shows that ChatGPT Summ all other methods, including the recursive Chat- GPT counterparts. This stresses the advantage of preliminary selection of aspect-relevant sen- tences. In addition, appears that shorter in- put lengths tend to yield better results, illus- trated by the two size-differences each of the PRIMERASumm and ChatGPT Recursive models. Fi- mally, the effectiveness of fine-tuning BART Summ and PRIMERA with our train set is apparent, as these models are competitive with the relatively strong ChatGPTSsmm model.
Aspect-aware sentence selection. Inputting Lead sentences without the aspect is akin to generic multi-document summarization. Row 5 represents this setting without any fine-tuning, i.e., with the original BART model pre- t--ttrained for generic sum- marization. The large difference in scores with respect to the aspect-aware configurations suggests that the characteristics of our ABS data is distinct from the generic summarization task.
We also produced "Oracle" extractive summaries for our data, generated greedily maximize the average of ROUGE-1 and ROUGE-2 scores, given the reference summary (Nallapati al., 2017). The large gap between the baseline and Oracle scores, as seen Table 5, leaves much room for improve- ment in future work on our task.
Oracle sentence selection. To estimate an upper bound for the BART Summ summarizer, devise "Oracle" that mimics near optimal sentence selec- tor, denoted Oraclesat (not be confused with the Oracle summary in Section 8. 8. It greedily selects the sentences that maximize ROUGE against the reference aspect-based summary, at the allowed in- put size limit of ~ 1K tokens (Hayashi et al., 2021) (see Appendix B.3 for details). As shown in row 6, using Oraclesal with BART Summ produces substan- tially greater scores than the next best option, where the Sentence-TS selector is applied (row 1). This stresses the potential of a good preliminary sen- tence selector when using the filter ttteenssummriza approach.
8.2 Ablation Analysis
We provide insights regarding the contribution of different components in summarization base- lines, operating the BART Summ-bumbber based models as use-case (being the best of the two fine-tuned mod- els). We refer to rows in Table throughout the analysis.
WWe do not report results of the BART BARTReuniv and PRIMERARaONSS systems here, since they are not directly trained for the task, and therefore not compatibly comparable. See Table 9 and Table 10 in the appendix for full results.

---

Relevance Aspect
Relevance Reference Summ.
data collection protocol can be expanded sup- ply even more data for real-world open-ABS and related information-sseking tasks.
Model
ChatGPT Racursie
Recuriive
3,70 (1,38)
2.45 (0.92)
CharGPT Summ" Sent-T5
3.40 (1.31)
2.80 (1.01)
BART Summ Sent-T5
3,05 (1.83)
2.35 (1.46)
Limitations
Table 7: Human evaluation results of the three mod- els with the highest ROUGE scores. The evaluation was conducted on 20 system summaries each model. 'Overall' values are the mean (std) scores on scale of 1-5 for relevance to the aspect and relevance to the respective aspect-based reference summary.
This study leverages existing generic multi- document summaries to generate aspect-based sum- maries by manually extracting aspect-related sen- tences. While this approach proved effective for the specific news datasets we used, it may not be read- ily applicable to different datasets where aspect- related sentences from the summary may not curately capture all the necessary information for that aspect. We assess this in our analyses, and recommend to do so on other potential datasets on which our protocol is applied.
8.3 Human Evaluation
We conduct a manual evaluation on the sum- maries produced by the three top-scoring models: ChatGPT ChatGPT and BART Summ+ with the latter two using the Sentence-TT T5 sentence selector. For 20 random instances from the OPE- NASP test set, we assessed: (1) Relevance the as- pect, i.e., "is the target aspect adequately discussed in the system summary?"; and (2) Relevance the Reference summary, i.e., "does the system sum- mary refer to the information in the aspect- based reference summary?" (Ernst et al., 2022: Lebanoff et al., 2018). Each criterion was rated on 1-5 scale, 5 being best. The outcomes of the evaluation are presented in Table 7.
Although OPENASP contains a representative sample of aspects from the generic summaries, the overall distribution of aspect labels is sparse with small fraction of repeating labels. This limits further analysis of aspect distribution aspect dis- covery that we leave for future work.
Furthermore, the usage of ChatGPT raises cer- tain concerns despite its popularity. Firstly, the lack of detailed documentation regarding ChatGPT's training procedure makes it challenging to deter- mine the specific vaining-data used. This raises the possibility of contamination, where our test data might have been incorporated somehow into the training of ChatGPT.
The ranking of relevance the reference sum mary is consistent with the automatic score rank- ing (in Table 5). Importantly, the manual scores achieved are quite low (all 2-3). Furthermore, the models demonstrate varying levels of success in extracting aspect-rclevant information, as indicated by the moderate aspect-relevancy scores and high standard deviations. Overall, these observations re-emphasize the challenges posed for models the task.
Finally, for our experiments, we employed spe- cific prompts (detailed in the Appendix) to assess the capabilities of ChatGPT for our task. Although we attempted several prompts, it is important to note that other prompts could yield different out- puts. Consequently, cannot make definitive claims about the model's capabilities.
9 Conclusion
10 Ethics and Broader Impact
Summarizing texts around an open-aspect is a ba- sic necessity when consuming information. Our new OPENASP benchmark serves this demand, as the first open ABS dataset in the mulli-docummet setting, with high-quality summaries collected via an efficient protocol. Our protocol overcomes the major hurdle of manually collecting summaries, by tapping into existing generic summaries in multi- document summarization datasets. Our proposed baselines, based on strong models, reveal the gap towards solving this task, posing a challenge even for the best current models. Overall, our efficient
This paper submitted in the wake of a tragic ter- rorist attack perpetrated by Hamas, which has left our nation profoundly devastated. On October 7, 2023, thousands of Palestinian terrorists infiltrated the Israeli border, launching brutal assault on 22 Israeli villages. They methodically moved from home to home brutally torturing and murdering more than a thousand innocent lives, spanning from infants the elderly. In addition to this horrifying loss of life, hundreds of civilians were abducted and taken to Gaza. The families these abductees have been left in agonizing uncertainty, as no infor-

---

mation, not even the status of their loved ones, has been disclosed by Hamas.
3675-3686, Brussels, Belgium. Association for Com- putational Linguistics.
The heinous acts committed during this attack, which include acts such as shootings, sexual as- saults, burnings, and beheadings, are beyond any justification.
Tal Baumel, Raphael Coben, and Michael Elhadad. 2016. Topic concentration in query focused sum- marization datasets. Proceedings of the AAAI Con- ference on Artificial Intelligence, 30(1).
In addition to the loss we suffered as nation and as human beings due to this violence, many of us feel abandoned and betrayed by members of our research community who did not reach out and were even reluctant to publicly acknowledge the inhumanity and total immorality of these acts.
David Carmel, Elad Yom-Tov, Adam Darlow, and Dan Pelleg. 2006. What Makes a Query Difficult? Proceedings of the 29th Annual International ACM Research and Development
SIGIR Conference in Information Retrieval, SIGIR '06, page 390-397, New York, NY, USA. Association for Computing Machinery.
We fervently call for the immediate release of all those who have been taken hostage and urge the academic community to unite in condemnation of these unspeakable atrocities committed by Hamas, who claim to be acting in the name of the Pales- tinian people. We call all to join us in advocating for the prompt and safe return of the abductees, as we stand together in the pursuit of justice and peace.
Hoa Trang Dang. 2005. Overview of due 2005. In Pro- ceedings of the document understanding conference, volume 2005, pages 1-12.
Ori Ersst, Avi Caciularu, Ori Shapira, Ramakanth Pa- sunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition-level clustering for multi- document summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1765-1779, Seat- tle. United States. Association for Computational Linguistics.
11 Acknowledgements
This work was supported by the Israel Science Foundation (grant no. 2827/21), the Israel Min- istry of Science and Technology, and One Al.
Alexander Fabbri, Irene Li. Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi mouttidoccument summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Asso- ciation for Computational Linguistics.
References
Ojas Ahuja, Jiacheng Xu. Akshay Gupta, Kevin Horecka, and Greg Durrett, 2022. ASPECTNEWS: Aspect-Oriented Summarization of News Documents. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 6494-6506, Dublin, Ireland. As- sociation for Computational Linguistics.
Alexander Fabbri, Wojeiech Kryšciñski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391 1-09.
Lea Frermann and Alexandre Klementiev. 2019. Induc- ing document structure for aspect-based summariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6263-6273 Florence, Italy, Association for Compu- tational Linguistics.
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. csptct-co opinion summa- rization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6578-6593, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Max Grusky, Mor Naaman, and Yoav Artzi 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pu- pers), pages 708-719, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021. Extractive opinion summarization in quantized trans- former spaces. Transactions of the Association for Computational Linguistics, 9:277-293.
Stefanos Angelidis and Mirella Lapata. 2018 Sum- marizing opinions: Aspect extraction meets senti- ment prediction and they are both weakly supervised, In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. WikiAsp: A dataset for multi-domain aspect- based summarization. Transactions of the Associa- tion for Computational Linguistics, 9:211-225, 1-255.

---

Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman. and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Jianmo Ni. Gustavo Hemannez Abrego. Noah Constant, Ji Ma, Keith Hall. Daniel Cer, and Yinfei Yang. 2022. Sentence-15: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguisticx; ACL 2022, pages 1864-1874, Dublin, Ireland. Association for Computational Linguistics.
Minqing Hu and Bing Liu, 2004. Mining and sum-
marizing customer reviews. In Proceedings of the
NIST. 2002. Document Understanding Conferences. https://duc .nsiicnisttt.tt. gov/. Accessed: 2023-06-01.
Tenth ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, KDD '04, page
New York, NY, USA. Association for Computing Machinery.
OpenAl. 2023. Models OpenAl API. openai.com.
Accessed: 2023-06-01.
Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene le. 2020. Aquamuse: Automatically generating datasets for query-hased multi-document summarization.
Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, and Ido Dagan. 2020. Controlled crowdsoureing for high-quality QA-SRL annotation. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 7008- 7013, Online. Association for Computational Lin- guistics.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization.. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131-4141, Brussels, Belgium. Association for Com- putational Linguisties.
Ori Shapira and Ran Levy. 2020. Massive Multi- Document Summarization of Product Reviews with Wcak Supervision.
Bowen Tan, Lianhui Qin, Eric Xing. and Zhiting Hu. 2020. Summarizing text on any aspects: A knowledge-informet
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
weakly-supervised approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 6301-6309, Online, Association for Computa- tional Linguistics.
Ivan Titov and Ryan McDonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In Proceedings of ACL-08: HLT, pages 308-316, Columbus, Ohio. Association for Computational Lin- guistics.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational I. Linguistics.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Ja- son Phang, and Samuel R. Bowman. 2022. SQuAL- ITY: Building a long -ong-document summarization dataset the hard way. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1139- 1156, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Elena Lloret, Laura Plaza, and Ahmet Aker. 2013. An- alyzing the capabilities of crowdsourcing services for text summarization. Lang. Resour. Evaluation, 47(2):337-369.
Rui Meng, Khushboo Thaker, Lei Zhang. Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021. Bringing structure into summaries: faceted sum- marization dataset for long scientific documents. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pages 1080- 1089, Online. Association for Computational Linguis- tics.
Mark E. Whiting. Grant Hugh, and Michael S. Bern- stein. 2019. Fair work: Crowd work minimum wage with one line of code. Proceedings of the AAAI Con- ference on Human Computation and Crowdsourcing, 19726997-
Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Da- gan. 2022. How "multi" is mouti-documentt summa- rization? In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing. pages 5761-5769, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ramesh Nallapati. Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarivation of doc- uments. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI 17, page 3075-3081, AAAI Press.
Wen Xiao, Iz Beltagy. Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: Pyramid-based masked

---

sentence pre-training for multi- nolti-docummntt summariza- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol-
B Models Implementation Details
B.1 Fine-tuned Models
ume I: Long Papers), pages 5245-5263, Dublin, Ireland. Association for Computational Linguistics.
BART. We used variant of the BART-large model with 406M parameters, fine-tuned on the CNN Daily Mail dataser
Xianjun Yang, Kaiqiang Song, Sangwoo Cho, Xi- aoyang Wang, Xiaoman Pan, Linda Petzold, and Dong Yu. 2022. Oasum: Large-scale open do- main aspect-based summarization. arXiv preprint arXiv:22/2.09233.
We utilized a variant of PRIMERA which was fine-tuned on the MultiNews dataset! and includes 447M parameters
PRIMERA.
Haopeng Zhang. Xiao Liu, and Jiawei Zhang. 2023. Summit: lterative text summarization via chatgpt.
We fine-tuned PRIMERA and BART OPF- NASP train 2 V100 GPUs with the following ayper-parameters: learning rate of 10e-5, batch size of 1, gradient accumulation steps of 3, and 3 epochs.
Ming Zhong, Da Yin. Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query- based multi-domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
B.2 ChatGPT
In Section B.2, experimented with three models based on OpenAl's ChaiGPT API
Summ- ChatGPTReuusive based while ChatGPT-
on gpt-3.5-turbo-0301
A OPENASP Collection Details
16kReeursive uses The temperature of all models is set to for reproducibility.
A.1 Annotation Interfaces
We present screenshots of the UI and instructions to extract aspect-based summaries for our dataset in Figures 3 and 4, respectively. The provided instructions assist annotators in accurately identi- fying aspects and extracting relevant information related to each of them.
To determine the appropriate prompt, man- ually evaluated the aspect-based summaries gen- erated during a brief manual tuning of the prompt text. The final prompt used for all ChatGPT models is presented Figure 7.
In this paper, we used ChatGPT for summa- rization in two approaches, ChatGPT and Recusive ChatGPT Summ summarizes duced version of the original documents that fits within the model's input length limit. The re- duced version is created using a sentence selection method (described in Section 7.1). The selected sentences from each document in the document set are then consecutively presented within the same prompt individual entries (document#1: document#X: ""..""). ".).
Furthermore, we show in Figure 5 a screenshot with the instructions provided to extract aspect- related source sentences for the analysis-valid and analysis-test sets. The UI was similar to the ver- sion we used for creating except for the given aspects that couldn't 't be edited selected as in OpenAsp. The annotators only selected the related sentences each predefined aspect from a given source document.
OpenAsp dataset
A.2 Curation Phase Details
For ChatGPT Rccussive and ChatGPT- 6k models, full documents are concatenated (separated by document title document#1:) until reaching the input length limit. Then the model generates a summary for the first portion of sentences. This process is repeated until all documents are summa- rized once. Then, the model gets all summaries with the same prompt, and summarizes the sum- maries produce the final summary.
Four paper authors applied the curation phase as detailed Section 4.2. We first defined and refined the curation guidelines (Figure 6) in a discussion before starting curation. Then, we split the data evenly among curators, by original MTurk anno- tators and by data source splits (DUC vs. Multi- News) avoid personal biases in specific sub-parts of the data. Some curated samples are shown in Table The released OPENASP dataset includes the original aspect labels and summaries, as well as the instances from after curation.
10 hhtp:s//wwuggnnggecccoo///ssh/hs hhtp://iwngginaaacccooooo//////r

---

Tig
aspect
spoects
Iy/e
Frevidiance peojary cases
Sectince
Add new Appedt..
AdS
Original Article:
Definition
1. Aerury no cath. the casemmet inacounte
Under federal lew, kncwledoe and irbent requrrementa: the persen must know the statement 55
3. Ala, the tedinnny nony must be muser to case.
interd make statements
ladge denal Tydte arsite "ter jury, repactiess wettin, is sedous offense
Also,
integityy lnge
Luton Garate Tjottrt arate, bardlry regandious of the setting, is serous offeroen that results
6. nerra wasove Answer recessaity pegjayy
logal
Prevalence of perjury cases
Berury a highly technical crime that requires such en airlight casn that prosecuto tore nanely
5. Parury a highty Night ouchnical crime that requires such an airipht casa that prosscuzare ranely porsue Department about prosecut
narrow or exesive answwer necessarty perjar..
8. In 1987, of 48, E55 casas fiidd by feders proses torm only 87 were partary. Stare courts oger faurnee, with over 4000 partury caran iffrria alone.
A Dece tieett of Justice stat dan says the fedoral govemment undertakes about 130
Sentence
by
only pertury
Sentarom generall ly rampe from homa continmmnt mentre jell. palic figars murder phosded yeart probetion.
courts show hightr figuree, 4000 perjory cases Caltfrnta
sentanoed trree
10. interviows with logal exparts suppest the threat of prosecution to crucial dctemrrt to porjury.
zostal supervaer etout senual reietorrhio priion monthe. recahred delardzo communit service but
11. Sentences genersily range from home confinement to monthe yall.
court sent the case badk fore aother sersencel
12. Ludone who were convicted have been removrd from office.
praiident had to pay ser's logal
14. postal supcrvisor lidd acout scaust resnoossip and prison for montho.
15. Tecas youth went jell for days for lying that his rights were not read.
community
appents
16. Bock for a stifiar acrionccl
17.
acouzers logal
Figure 3: Our interface annotating aspect-based summaries. An annotator can add, remove or edit aspect labels (top), and respectively select sentences in the source reference summary that relevant the aspects. The produced summaries shown on the right side and updated in real time. Sentences with no aspect labels should actively annotated with special "N/A" label (sentence 10) ensure workers reading all the article content.
B.3 Oracle Sentence-Selector Details
(Lead, Sent-T5 and Oraclesa). In Table 10 it is apparent that the 'Lead' selector provides supe- rior results over the other two sentence selectors. 'Lead' sentences are less focused aspect-specific information (as revealed in Section 8.1). We hence assume that this characteristic encourages the model focus more on the aspect during train- ing, and consequently to perform better during in- ference.
Following Hayashi et al. (2021) the oracle sen- tence selector implemented as follows. Sen- tences from the document sets are aggregated to maximize the average of ROUGE-1 and ROUGE-2 F-1 scores against the reference aspect-based sum- maries. Once the score no longer improves, the selected sentences omitted from the source, and the process is repeated again. This continues until the input size limit is reached, e.g., 1K tokens for BART.
C.1 System Summary Examples
Tables 13 and 14 present the aspect-based sum- maries generated by varied models for 2 differ- ent aspects, 'Launch into orbit' and 'Reasons for high unemployment rates", respectively. The corre- sponding reference summaries appear in the button line.
C
Model results
Table presents the complete set of experiments conducted on our baseline models the Filter- Shem-Summriize technique, employing the Lead, Sentence-T5, and Oracle sentence selectors.
Table 10 presents the experiments conducted on baseline models in the Recursive summariza- tion technique. As a reminder from Section 7.2. BART Recursive BARTRecrrsve and PRIMERAR Recursive were fine- tuned with the Fillerthen-Summmrizz approach, which can use three different sentence selectors
D OPENASP Details
D.1 OPENASP Source Splits
We split OPENASP dataset into train, validation and test sets based on the source datasets they orig- inated from (see Table 11). For aspect-based sum-

---

Original
Edited
Is passed
Tople
aspect label
Original summary
aupect label
Reason
Modifßed
Comment
The tople is list of incidents and therofore all its are rejected
9. Two quakes caused great destruction
Destructive earthquakes below 7
Earthquakes magnitudo
though their magnitudes were below the level.
10. Twenly-fiv live thouaand were killed in Armeniaa least another 1,000 Tadzhikistan.
X
Topic
0. Clarence Thomas, black conservtivee republican, was confirmed Supreme Court justice
16 October 1991.
8. He warked
attorne and moved Washington
Clarznce Thomas
Law
9. Thomas was appointed several civil rights and
in 1974.
Summary
is not relevant and should removed.
equal employment opporiunity positions beginning in the early 1980s and as judge the U.S.
10. Circuit Coart of Appeals in 1990.
Supremee brought joy to his mother in Pinpoint, Georgia. rise.
confirmation
justice
19. Aurism affects in 300 1000
The name ambiguous and therefore
People with
Highty percent are bays.
Autism
21. Odds are in 20 that family with one autistic
Autism prevalence
Aspect
child
Brick, has an autism cluster.
is changed
Table 8: Examples of three data instances that were modified or rejected during the curation phase. Bold text represents a problematic sentence in the summary. The passed' column states whether instance included in the final data is rejected. 'Reason' explains why the instance rejected. 'Modified' specifies the type of problem I'Aspect name", "Summary", or 'Topic") in case the instance needs a correction in order to be included in the final dataset.
maries originated from MultiNews, we followed the original splits, for summaries originated from DUC, we separated the test years from the train and validation.
erage and density. Coverage is the percentage of words in the summary that are from the source article, and density is the average length of the ex- tractive fragment to which each summary word be- longs. While OPENASP'S summaries are extracted from source generic summaries, the generic sum- maries themselves are different levels of abstrac- tiveness relative to their corresponding document set. Accordingly, the aspect-based summaries also exhibit varying abstractive extents. Figure 2 illus- trates the distribution of coverage on the x-axis, and that of density on the y-axis. As shown, most extracts short, however there few cases of sentence-level extractions. Overall, there is diver- sified balance of abstractiveness in the data. Figure 10, presents the content diversity graphs of OPE- NASP, separated by the source datasets DUC and MultiNews (a stratified version of Figure 2).
D.2 MultiNews Filtering
During annotation phase, we noticed several faulty source documents from MultiNews, probably due to failed crawling. We manually examine some suspicious short source documents, finding a few common phrases that imply the document retrieval failed. For example, "The seed for this crawl was a list of every host the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links includ- ing their embeds) The WARC files associated with this crawl are not currently available to the general public." and several similar cases.
We created short list of such texts and automat- ically filtered all topics from MultiNews contain- ing one or more source documents matching these strings.
D.5 Multi-document Coverage
Multi-document coverage quantifies the amount of documents in the document-set upon which cor- responding summary depends (Wolhandler et al., 2022). Figure 9 shows the multi-document cover- age of OPENASP, compared to the source datasets. The coverage is derived in terms of propositional alignment (Ernst al., 2022) between the highest- matching subset of k documents (x-axis) and the summary. The dispersion score, a function of the wrea-abovethe-curvee the-curve, is a measure of multi- document coverage, where a higher value means
D.3 Documents and Summaries Lengths
Figure 11, presents the distribution and cumula- tive distribution of token lengths in summaries and document-sets.
D.4 Content Diversity
Content diversity (Grusky et al., 2018; Fabbri et al., 2019) is joint measure for extractiveness of cov-

---

Fine Tuned
Aspect
Input Size
Oracle Selector
Lead Sclector
S-T5 Sclector
Base Model
Inpul
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
IK
BART Summ
40.6
14.5
23.2
31.3
7.6
17.9
32.4
8.3
18.7
4K
PRIMERA Asumm
33.8
10.0
20.0
30.0
7.6
17.9
30.5
8.0
18.3
IK
PRIMER Asum
37.9
13.0
22.6
29.9
7.6
17.9
31.2
8.3
18.7
IK
BART Summ
39.5
13.3
22.5
27.5
5.3
15.7
32.0
8.0
18.2
4K
PRIMERA Asumm
31.4
8.2
18.6
26.4
5.2
15.7
29.1
6.6
17.0
1K
PRIMERASH
36.6
12.4
21.9
26.6
5.2
15.6
30.4
7.5
17.8
X
1K
BART Summ
34.6
10.7
20.5
25.5
5.0
15.3
28.5
6.7
17.0
X
4K
PRIMERA
30.9
7.5
17.3
28.7
6.3
16.0
29.5
6.6
16.4
IK
PRIMERASH
35.8
11.1
20.0
29.0
6.4
16.2
31.3
7.8
17.6
X
4K
ChalGPT's Summ
35.6
10.8
20.7
32.8
8.9
19.2
33.7
9.4
19.8
X
IK
BART Summ
34.4
10.6
20.2
25.4
5.0
15.0
28.2
6.5
16.9
X
4K
PRIMERA
31.1
7.7
17.4
28.5
6.2
16.1
29.6
6.7
16.7
X
IK
PRIMERA ASumm
35.8
11.0
20.0
28.9
6.3
16.0
31.1
7.5
17.4
Table The ROUGE F; scores for all model configurations that use sentence selector. The number of extracted sentences by sentence selector is limited by the maximum input token-length, indicated in 'Input Size'. The best configuration for each sentence selector option is marked in bold.
Model
Input Size
Fine
Sentence Selector
Split
Source
# Topics
Aspects
Tuned
BARTReuuiie
Resuaive
IK
Oraclesa
29.8
6.3
16.8
DUC-2002
157
PRIMERA REMMeRAnssss
4K
Oracless
28.6
6.4
16.9
DUC-2007
148
PRIMERARccnI
IK
Oraclesa
27.3
5.6
16.2
Test
MultiNews-Test
291
BART Roamebe
IK
Seat-T5
29.1
6.0
16.2
Total
192
596
PRIMER Agccanise
4K
Seni-TS
29.1
6.7
17.1
PRIMER PRIMERARAcIMI Agecnnive
Sent-T5
27.9
5.7
DUC-2001-Tes
26
78
BART BARTRoussie Rocussive
IK
Lead
30.1
6.6
17.1
DUC-2006
13
38
PRIMERA RAgecnivee
Lead
29.8
7.3
17.8
Valid
PRIMERA PRIMERABccaiie Aoccpise
Lead
28.1
5.8
MultiNews-Valid
43
122
CharGPT Rezuminc
32.4
9.2
Total
82
238
ChaiGPT Resaninc
31.8
8.1
DUC-2001-Train
Table 10: The ROUGE Fi scores for all model con- figurations in the Recursive summarization technique. All variants include the aspect label as part the input. fine-tuned us-
DUC-2006
115
Train
MultiNews- Train
279
Total
145
476
BART Recusivee and PRIMERAReeusi ing the Filter-then- Simen-Summrrzee approach (on the input sentences extracted by a sentence selector), however executed through the recursive technique (see Section 7.2). The best configuration marked in bold.
Table 11:
The size of the OPENASP dataset splits, bro- ken down to their sources. Topics denotes the number of document sets the split; Aspects is the total number of aspect-based summaries in the split.
there higher document-diveeriity overall.
in the dispersion score means there are still aspect- based summaries that align with larger number of documents. Some summaries require handpicking information from specific documents, while others require consolidating information from across the document set.
The dispersion score of OPENASP is 2.9, with standard deviation of 4.1, while DUC-2001/2 and DUC-2006/7 render higher dispersion scores of 10.2 and 16.5 respectively. This can be explained by the longer summarics in the latter datasets, 2.5 times longer than OPENASP summaries. Moreover, generic summary, as opposed an aspect-based summary, expected to cover several subtopics, and hence align with wider range of source doc- uments. Meanwhile, the high standard deviation
D.6 Adding Topic Names
We add each document-set in our dataset a topic name, which was selected as an additional side task. Specifically, we asked Mturk annotators to identify

---

the topic of each presented generic summary. This process helped us later facilitate and quickly identify irrelevant aspects the curation phase in Section 4.2.
D.7 Dataset Examples
Table 15 shows the aspects labels and the corre- sponding aspect-based summaries, which belong to 3 randomly selected different document sets in our OpenAsp. Each document set is represented by its topic name.
D.8 Details on OABS datasets
Table 12 presents more details regarding the OABS datasets from Table 1.

---

Avg # Tokens in Input Doc/Docset
Avg # Tokens in Output Summary
Avg # Tokens in Input Aspect Label
Avg Aspects per Topic
Dataset
Aspect Label Examples
kentucky,
gray, mr tizzard,
julie bishop, bernabcu, watson, kola, cash,
AnyAsp (Tan al., 2020)
818
23
1.6
7.0
cnn.
australia
History,
Career, Background,
Geography, Life, Reception,
OASUM (Yang al., 2022)
1615
40
2.7
Description, Early life, Demographics. Production
Hurricane impact, Drug treatments, Custody Ruling, British response, Elian's rescuc, Availability of gas, Removal of the guardhouse,
OPENASP (Ours)
7,930
96
3.6
3.1
Opinions Hillary Clinton, Expenses controversy, Alterations of waterways
Table 12: Statistics the OABS datasets from Table 1. In the first datasets, the input topic is single document, while in our dataset it is document set. The example aspect labels in the right-most column are taken from sampled topics from across the respective dataset.

---

Instructions collapse expand)
Goals
You are given short journalistic article that summarises a certain topic and contains few sub-topics that we call aspects. Your goal
to:
aspects (usually 3-4) described in article. 2. For each identified main aspect, select all and only the sentences that describes it.
Identify
Your Contribution:
Creating dub-document from the article for each of your selected main aspects, where sub-document describes the article from different angle.
How to Choose Aspects:
1. Choose only the main aspects only the dominant subtopics in the document, which the author clearly chose to describe In their article.
2. Concrete sentences aspect should be the main focus some sentences the article.
not combine distinct aspects Instead "production and sales", define two different aspects "production" and "sales", both main aspects.
4. Avoid anecdotal/ ceaiottalnoo-ccentee non-central aspects which are mentioned only briefly or by passing. For example, in an article about global warming, "melting ice" should not be an aspect if it is only mentioned in the sentence "Melting ice sheets have caused seas rise worldwide." Suitable aspects
more broadly, such "environmental impact" "sea-level rising". Avold generic aspects for the article tople example, document Chimpanzees Animal/Monneyy/Chimpaaae "Behavior" "Habitat".
be
are inadequate aspects. Suitable aspects
6, Avoid examples or lists as aspects avoid aspects such as "cases of ...", "examples", "types of ...', or lists of items. For example, in an article about organic food, "types of food" should not an aspect.
How to give aspect names?
1. Use aoplicit/concette names prefer "USA government regulations' just "Regulations".
2. Aspect names should not include "and" or "or" see point about choosing aspects.
Important Notes:
1. First, read the document briefly to uncover the main aspects. Each
espect. Choose "N/A" only the sentence does not refer to main aspect.
mapped
3. You cannot map text-span shorter than full sentence.
Verify Yourself:
Each aspect name with selected sentences appears on the right side the screen and creates sub-document. For each aspect's any
sub-document please verify following points
1. You identified main aspect the aspect dominant sub-topic that is clearly described. Grouping its selected sentences creates a sub-document that someone would learn about the article's topic from specific and important angle.
2. You chose an appropriate aspect name a short phrase (usually 2-5 words). The aspect name unambiguously identifies the other aspects. 3. You chose only relevant sentences each selected sentence must relate respective
aspect distinguishes
longer necessary identify the aspect's sub-document.
the sentence focus. 4. You did not miss relevant sentence verify that all the unselected sentences for aspect indeed have different main focus than the aspect under consideration.
bonus will be given to workers who submit high quality HITs.
Please read the example below.
Figure 4: Our annotation guidelines for extracting aspects and respective aspect-based summaries, shown to crowdworkers on the Mechanical Turk platform.

---

Instructions (click to collapse/ expand)
Goals
short journalistic article that summarises certain toplc and fixed, predefined list of sub-topics that call
aspects. Your goal is to:
1. For each predefined aspect, select all and only the sentences that describes it.
Your Contribution:
Highlighting every predefined aspect
Important Notes:
1. Each sentence must be mapped at least one aspect.
Choose "N/A" only if the sentence does not refer to any main aspect.
2. You cannot map text-span shorter than full sentence.
Verify Yourself:
Each aspect with selected
appears right side
For each aspect's sub-document please verify
the following points and correct any issue:
1. You chose only relevant sentences each selected sentence must relate to its respective aspect the main sentence focus.
2. You did not miss any relevant sentence verify that all the unselected sentences for an aspect indeed have different main aspect
bonus will be given to workers who submit high quality HITs.
Please read the example below.
Figure 5: Our annotation guidelines for aligning document-sentences aspect (Section 6. I), as presented crowdworkers on the Mechanical Turk platform.
Curation Phase Guidelines
CSV Columns:
aspect Jabel
nspect summary (numbered bullats)
4. gild le.g. Anta IV atppmenee//02000220000000
Review Columns:
reason aspect Nome Toplo cspect Summary
3. medilied led- N S (any comtination) (Naam (Topii (Sjummary
4. comment free toxt Blak
Reviewer SAISOOOS (frst name ledees))
Review Guidelines:
Reviee for the following issuee:
Toplo:
Chargg tostc ihee poliicclly correct uniure, merk comments
you
sepwet armnwhing is may
neading the
took: OR from world o not ramovo
Fir sepaci lobels ihs unmarooraale abble that unee will withe them as such
3. Popost Bummary:
Check that centencas reluantt the
your
cummares
probeb Ity
seinn information)
1. For avampil, a ag earthouzke topic with on aspect Wifected
Locaitons" onl 1-2 mentions of locationa aftected. the summery hew high potembil micsing othor locations offectoc not sallent.
try
muttple problomatic maries
romove
sellsty) quality more importantt iham quantity
3. Profer to remove a cummary then keeping if there uncertalntty
Figure 6: Curation phase guidelines used for cleaning OPENASP.

---

Role
Content
system:
You are an expert summarizer that writes aspect-based summaries correctly and concisely given user's sets of multi-documents news articles on the same topic.
user:
Below are (X} documents on the same topic, each delimited with triple backticks in different user message.
Please write a short, concise aspect based summary about (Aspect Phrase), only summarize content from the above documents, avoiding any external data sources.
user:
Document #1:
user:
Document #2:
user:
Document #X:
user:
Response only with the aspect based summary about '((Apeet Phrase) as instructed above."
Figure prompt template used ChatGPT models, ChatGPT summ: ChatGPT Recursive and ChatGPT. 16k In the recursive configuration, when summarizing summaries, the summaries "documents".
2
3
>
82
209
101
21
2
5
6-9
10-24
25
46
47
47
44
71
75
89
Figure 8: The distribution of the number of aspects (top) and number of documents (bottom) topic for OPENASP's 419 topics. OPENASP an average 3.1 aspects and 10.4 documents per topic.

---

1.00
0.95
0.90
Coverage
0.85
0.80
0.75
Dataset: AAC avg (AAC std)
OpenAsp: 2.9 (4.1)
0.70
DUC 06-07: 16.5 (9.1)
0.65
DUC
(5.7)
MultiNews: 1.8
0.60
1
2
4
5
6
9
Number of source documents (k)
Figure 9: The multi-document coverage and the dispersion (AAC) of OPENASP compared to other MDS datasets as reported in Wolhandler et al. (2022). A larger area above the curve implies that summaries rely on larger subset their corresponding document-set.
25
OpennAsp I B18
OpenAsg-DUC
8= 138:1
OpenAsg-MMLiNNeue
OperAesp-MullNeess 882
Extraclive Iragirent densiy
15
35:1
0.0
02
0.4
0.8
088
fragment coverage
Figure 10: The content diversity graphs (Grusky al., 2018) of OPENASP, separated by the source datasets DUC and MultiNews (a stratified version of Figure 2). and e denote number of examples and median compression ratio respectively. DUC-based summaries contains extracts that are substantially shorter than those of MultiNew.

---

Model
Summary Text
The launch was scrubbed four minutes before liftoff because bad weather. It was rescheduled for 8:31 a.m. EDT March 8, 1989.
The countdown began Saturday aftermoon, April 27, 1989, and the countdown for the launch of the new shuttle, Discovery, began at 3:37pm.
BART 1k S-T5
Five ustronauts climbed into the crew cubin just as dawn broke over the Kennedy Space Center.
They were ready to go out of the cabin at any time to assist with repairs.
Mission Control ordered space- space-walking astronauts to he ready to help with shuttle's final
The countdown began Saturday afternoon for launch for 8:3 lm EDT Tuesday.
PRIMERASume IK S-T5
The launch was scrubbed four minutes before liftoff.
The launch was delayed for two weeks after a faulty power unit forced the National Aeronautics and Space Administration to halt the first launch four minutes before liftoff on April 10. 1989. The second attempt to launch the Hubble Space Telescope was successfully launched on June 8, 1990.
PRIMERA Ssmmn 4K S-TS
The Hubble Space Telesoope was launched into orbit by the Discovery shuttle. marking significant moment for NASA.
The telescope. which cost $1.5 billion, was releused from the shuttle's 's mechunical arm and expected to provide significant scientific data by satellite in a month two. The launch was delayed by two weeks due to faulty power unit, but the procedure went smoothly and the telescope was released successfully. The arbit was dictated by the need to put the telescope above the Earth's 's distorting atmosphere. The mission was considered to be the most important shuttle mission to date. Aspect Based Summary:
ChatGPT 4K S-T5
The Hubble Space Telescope was successfully launched into orbit by the Discovery shuttle after several delays. The telescope that power its instruments. It had some problems with its high-speed antennas and its science instruments were on hold, but it is expected to provide sigmificant scientific data by satellite in a month or two.
deployed with the help of the shuttle's mechanical arm and two solar panels
CharGTT Recunice 4K
The telescope is considered the most important shuttle mission to date and expected to provide clues the age and size of universe, and may solve the mysteries quasars, pulsars, and black holes and even find stars with plancts that could conceivably support life.
The Hubble Space Telescope was successfully launched into orbil by the space shutile Discovery. The launch was delayed multiple times due to technical problems and the Challenger accident. The te telescope, named after astronomer Edwin P. Hubble, will allow astronomers to study stars and galaxies that are billions of years old and provide insights into the origins and expansion of the universe.
ChatGPT Recasioe 16K
The telescope's 's mission expected to last 15 years and it equipped with advanced instruments cupture high-resolution images and collect scientific data.
Deployment of the Hubble Space Telescope (HST), originally scheduled for 1983, was delayed by technical problems and the Challenger accident in 1986 and did not take place until April 1990.
A successful launch of space shuttle Discovery with the HST aboard took place at 8:34 a.m. April 24, 1990 and Discovery ascended into orbit 380 miles above Earth. April 25 the HST was placed in orbit on its own with Discovery trailing behind to provide astronaut assistance if needed.
Gold
The HST's aluminum aperture door was successfully opened on April 27 exposing the telescope's 94.5- inch mirror to starlight. By April 28 there were still problems with the electronic and communications systems, but procedures were in place to take care of these, the HST was in orbit,
and the Discovery astronauts prepared for return to earth on April 29, 1990.
Table 13: The generated based summaries that refer to the aspect 'Launch into orbit' of the topic 'Hubble Space Telescope'. : The bottom line contains the Gold summary in Bold.

---

Model
Summary Text
Unemployment in France is one of the country's main economic problems.
Unemployment is major problem because of combination of factors including lack of development of job market and lack of solidarity among people.
BART Summ 1k S-T5
The French Observatory of Economic Circumstances expects unemployed
population of 140,000 by the end of the year and 120,000 in the following year. France rumning a near-record 12.4% unemployment rate with jobless population
3.1 million, one-third of whom has been unemployed for more than year.
Unemployment France has long been major problem, with 12.4% unemployment rate.
France's 'ss unemployment rate dropped slightyy to 11.6% in October from 11.7% the previous October, but still high and has not reversed over past years.
PRIMERASumas 4K S-T5
mnemployment is fueled by a combination of factors, including a weak economy, lack of development the job market, and social exclusion.
Also, there is concern over the high cost of unemployment insurance and unemployment compensation.
Unemployment in Europe is one of the most important problems facing the country. The unemployment rate in France is 11. 1.2%% up from 11.7% since the Second World War. In 1982 unemployment peaked at 9.7%, the first full year of Reagan's 's administration, while in 1975 unemployment in Germany more than doubled to 3.4%.
ARIMERAsumm 1K S-T5
France has heen struggling with high unemployment rates, which have been major economic hurdle for the country.
The unemployment rate in France stands at 12.5%, with 3.1 million people unemployed, one-third of which have been unemployed for more than year. The major causes of unemployment in France are attributed to the economic crisis. lack of job opportunities, high labor costs, and structural impediments in the functioning of labor markets.
ChatGPT Sumas
Suma 4K S-T5
The rising crime rate among youths from working-class suburbs that have been hard-hit by unemploymmen is also growing concern.
High unemployment rates in France are attributed various factors such as high labor costs,
low economic growth, lack of job opportunities, insufficient social allowances, and the lack of development of the job market.
ChatGPT Recunivee 4K
The govemment has taken measures such increasing unemployment benefits, providing financial aid to the jobless, and subsidizing employment programs.
The OECD recommends raising wage flexibility, reforming the minimum wage system, and reducing the of the social protection system to address the problem unemployment.
The high unemployment rates in France are attributed to various factors, including economic crisis, lack of job opportunities, rigid labor market regulations, social exclusion, and rising crime rate among youths from working class suburbs.
The government' efforts to address the issue through measures such as increasing social allowances and unemployment benefits have been met with protests and demands further action.
ChatGPT Recunive 16K
The need for wage flcxibility, reforming the minimum wage system, and reducing restrictions on mass layoffs have also heen highlighted as potential solutions. Additionally, concerns have been raised about the high costs of France's social protection
system and the need to make the henefit system less generous.
Critics attributed F France's s unemploymen to its near-zero growth cconomic
environment, to its focus on fairness in income distribution, to rigidity in budget policy,
to declines in social policy, and to layoffs in the public sector.
Gold
Additional unemployment causes cited were high costs for low-skilled workers, high unemploymen insurance and compensation costs, and over-regulation of industry.
Crities suggested using the money paying for social aid programs to be used instead for investment in jobs.
Table 14: The generated aspect- the topic 'Unemployment in France". The bottom line contains the Gold summary in Bold.
basect-based based summaries that refer to the aspect 'Reasons for high unemployment rates' of

---

Tokenizer
0.00010
Tokenizer
0.0100
Words
Words
0.00008
BART
BART
Densiyy
0.0075
ChatGPT
ChatGPT
0,00006
0.0050
0.00004
0.0025
0,00002
0,0000
0,00000
1.0
1.0
0.8
0.8
Density
0.6
0.6
0.4
0.4
1024, 0.0351
L: 4096, P: 0.3221
0.2
0.2
L: 128, P: 0.8046
L: 16384, P: 0.8824
0.0
0.0
200
400
0
15000
30000
Summary Tokens Count
Documents Tokens Count
Figure 11: The distribution (top) and cumulative distribution (bottom) of token-lengths in summaries (left) and document-sets (right) OPENASP. Each plot presented with three different tokenization methods: standard word tokenization (NLTK word tokenize), BART (also used PRIMERA) and the ChatGPT tokenizer (tiktoken). The dashed horizontal lines mark 1024, 4096 and 16384 token lengths. For each horizontal line, P is the percentage of instances the dataset whose summary/document-se lengths are less than equal to the corresponding length limit (L), using the word tokenizer. For example, 32.21% % of the instances have input word-lengths less than 4,096 words. As seen, the differences between the tokenizers in terms of content length are negligible.

---

Topic
Aspect Label
Aspect-basse Summary
2000
Senate
She started raising $15 million for a second term in the Senate.Her 2000 Scnate campaign blamed Republicans for rejecting her patients' rights bill, tohacco regulation, and gun purchase restrictions. She ran on improving health care, preserving Lake Tahoc, restricting gun sales, eliminating the gas additive MTBE, and bringing together opponents in California's 's water wars. Opponent Representative Tom Campbell (R-CA), Stanford Law professor and pro-abortion rights, made her Senate race nationally significant and her toughest since 1990, when she was hooed state Democratic convention for supporting the death penalty.
Campaign
Senator Dianne Feinstein
Political tions
Posi-
In 1999 she authored failed resolution censuring President Clinton, but voted against both articles of impeachment. Her 2000 Senate campaign blamed Republicans for rejecting her patients" rights bill, tobacco regulation, and gun purchase restrictions.She ran on improving health care, preserving Lake Tahoe, restricting gun sales, eliminating the gas additive MTBE, and bringing logether opponents in California's water wars.A centrist, her support cuts across lines of party, ethnicity and gender. supported deficit reduction.
Achievements of Feinstein's 's. Sen- atc Career
By 1998 Senator Diane Feinstein (D-CA) was the senior Demoerat on the Judiciary Commitree's panel on technology, lerrorism and government information Her 1984 ban assault weapons and a Mojave Desert national park were crowning achievements of her Senate carcer.Feinsted amended a trude bill to elimmnnte sunctions against Sub-Sahun Saharan African countries producing copycat US AIDS Irugs.In 2000 Feinstein introduced legislation requiring gun owners to obtain licenses and register.Feimstein Feinstein successfully rebutted opponent's 's erroneous charges that support normalizing China trade relations benefited her husbund and that she had hidden related financial disclosure information. Senators Feinstein and Boxer were the first women co-chairs of national political convention.
Scandals
His college career was threatened by the Vietnam War, but he arranged deferments until induction seemed unlikely, then drew a high lottery number, ayoiding military service. This incident drew criticism during subsequent political campaigns He and his wife, Hillary, invested in a real estate venture, Whitewater, and the related failure of a savings and loan. The scandal simmered for In the presidential cumpaign of 1992 he defended his record in Arkansas and his personal and draft history. William Clinton showed Intelligence and promise from Ah overachiever at school, his home life punctuated with long discussions a variety of subjects including desegregation and social After graduation he went on to Georgetown, Oxford (as Rhodes Scholar) and Yale Law School. His college career was threatened by the Vietnum War, but he arrunged deferments until induction secmed unlikely, then drew high lottery number, avoiding military service.
William Clinton
Early life
Political career
Returning home, he ran saccessfully for governor in 1978 at age 32 but was defeated for reelection. Older and wiser, he ran successfully three more times, making improvements in education, the cconomy and welfare. In the presidential campaign of 1992 he defended his revord in Arkansus and his personal and draft history.A German editorial summarizes Clinton's carly presidency. some blunders, a year later he enjoys 60% popularity rating; the economy is up, prices stable and interest and the deficit down.
Problems with the cast
Clark Gable was under contract Selznick's father-in-law, who finally "loaned" him to Selznick. with plenty of strings attached andGable wasn't I thrilled about it Origina director George Cukor didn't It get along with Gable and was eventually repluced by Fleming, who didn't get along with Vivien Leigh." "Leggh hated Fleming. With a passion. Fleming hated CercClark Gable hated David ... Everybody hated David," an assistant said. Fleming quit before returning.
The tomoltoon making of GGoue with the wind"
David Selznick reluctant
In Entertainment Weekly, Chris Nashawaty tells the story of the film's 's making, which centered on producer David Selznick He at first reluctant make the film, despite a glowing review of the book by one of his employees." "I am absolutely off my nut about this book," Katharine Brown wrote, finally convincing him to take action.
make the film
The premiere
And the trouble was far from over: Racism plagued the various premieres, with black cast members in many cuses banned from atlending, the Los Angeles Times reports. That prompted unger from Sekznick, the AP reports; Gable, meanwhile, had already stood against segregated toilets on set, threatening to bail the film, according to a Life magazine book cited by the Times.
sparked racial
tension
Script writing
Among the challenges: The first writer dropped out alter spending months on the script.fteer number of other writers tried their hand, including Selznick himself, writer Ben Hecht took but there was no time for him read Margaret Mitchell's book.So Seiznick and director Victor Fleming "stayed up all night acting out the story for him."
challunges
Table 15: The aspect labels and corresponding aspect- based summaries that belong 3 random selected document sets from our OPENASP dataset. A topic name is assigned to each document set.


Reviews:
[{"review_id": "abY4peHA5U", "reviewer": "Reviewer_LFH5", "summary": "", "questions": "Question A: In the experiments, fine-tuned models (BART and PRIMERA) were evaluated only with the \"filter-then-summarize\" method, and not with recursive summarization, in contrast to GPT3.5. Is there any reason for that? IMO, unless there is a major difficulty, the experiments and result comparison would be more convincing if BART and PRIMERA are also evaluated in the recursive summarization setting.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## A third assumption\nIn fact, the third assumption stated, claiming that all the aspects that are mentioned in the source set of documents and can be of interest to users in an OABS setting are also mentioned in the generic summaries, is not assumed in our framework. While our methodology yields aspects that can be of interest to the user, as they correspond to prominent sub-topics in the document set, there may be additional aspects within the source documents in which users can be interested which are not covered in the generic summary. However, it should be noted that a dataset for the OABS task, where the aspect for which a summary is sought is given in the input, does not need to cover all possible aspects for each topic, but rather provide a sample of such aspects. (Notice that a comprehensive set of aspects would be required for a dataset for an exhaustive aspect extraction task, which constitutes a different task setting). \n\n## Question A: \nWe have not thought originally of experimenting with BART and PRIMERA in the recursive setting because in this setting the models should be first applied to a single document, or to just a subset of the document set. However, our dataset does not include gold summaries to train on which directly correspond to such a partial input (unlike GPT3.5, which is prompt based and does not require training). Yet, given your remark, we realize that it is worth conducting such experiments with BART and PRIMERA, even if using models that were trained under a somewhat different setting, and evaluate their performance for the recursive method. We will add such experiments to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very well written and structured, and the proposed protocol and dataset are clearly described and well assessed.\n\n- Based on the creation protocol and the analysis presented in the paper, this new dataset is very likely the best that currently exists for OABS in a multi-document setting in terms of quality, and the authors explicitly announced that it will be released.\n\n- The OABS experiments and results with fine-tuned and LLM-based models may also be interesting to the community.", "reasons_to_reject": "- The aspect-based summary annotations are done by reading the generic summaries only. As explicitly noted by the authors, the validity of this protocol is based on two assumptions: (1) The aspect-related sentences in the source generic summaries cover well the information about the aspect in the source set of documents; and (2) The aspect-based summaries preserve the coherence borrowed from the source summaries. Both assumptions are shown to be valid in the paper. However, there is a third assumption that the protocol relies on, which is: (3) All the aspects that are mentioned in the source set of documents and can be of interest to users in an OABS setting, are also mentioned in the generic summaries on which the aspect-based annotations are exclusively performed. Although this assumption does not seem unreasonable, it is not assessed nor discussed in the paper.", "questions_to_authors": "Question A: In the experiments, fine-tuned models (BART and PRIMERA) were evaluated only with the \"filter-then-summarize\" method, and not with recursive summarization, in contrast to GPT3.5. Is there any reason for that? IMO, unless there is a major difficulty, the experiments and result comparison would be more convincing if BART and PRIMERA are also evaluated in the recursive summarization setting.", "paper_topic_and_main_contributions": "The problem addressed in this paper is the lack of high-quality datasets for multi-document, open-aspect-based summarization (OABS) and the high burden and cost of manual creation of such datasets.\n\nTo address this problem, the paper offers the following contributions:\n\n- It proposes an efficient protocol for the manual derivation of multi-document OABS datasets from existing, generic (i.e. not aspect-based) multi-document summarization datasets. The protocol is executed through a specially-designed annotation tool and is cost-effective, in particular because the aspect annotation process is accomplished by reading the generic summary text only (not the full source set of documents).\n\n- Following the proposed protocol, the authors created (and will release) OPENASP, a multi-document OABS dataset that contains 1310 instances for 1266 unique aspects. They also performed a relatively robust quality assessment of this dataset.\n\n- The paper also describes OABS experiments with various baseline models using this new dataset, offering a comparison of fine-tuned, transformer-based models and LLM-based models (based on GPT-3.5) and showing that multi-document OABS is still a challenge for current state-of-the-art methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "    "}, "extra_text": {}}, {"review_id": "ZLG93nAhQm", "reviewer": "Reviewer_zYF8", "summary": "", "questions": "1. What was the kappa coefficient between annotators in the data quality experiment?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Data curation by the authors, and possible author biases:\n\nThank you for bringing up this issue, since the curation phase followed concrete detailed guidelines, which underwent substantial discussions and refinements, we expected that at this point annotation biases would be personal, stemming only from subjective comprehension of the employed expert. Hence, we think that it was reasonable to employ the authors for this task, given that:\n* It was performed by 4 authors as experts\n* Curation strictly followed the guidelines\n* We split the data evenly among curators, by MTurk annotator and by data source splits (DUC vs. Multi-News) to avoid personal biases in specific sub-parts of the data\n* Overall, on the curation phase (lines 269-274) filtered 8% of the dataset and modified less than 15% of the remaining instances, further limiting the scope of possible biases\n\nFinally, we will release the unfiltered dataset and curation decisions together with the curation guidelines. Some random examples will be included in an appendix to illustrate the nature of the curation decisions.\n\n## Sparseness of open aspects\n\nThe essence of the OABS task, for which OpenAsp was designed, is to expect that models would perform well on any novel ad-hoc aspect at test time. Accordingly, models are not expected to learn about particular aspects in the training data, but rather learn to generically map the semantics of an input aspect label to relevant source content fragments for the summary. Accordingly, training data should rely on a diverse (sparse) sample of aspects, to avoid overfitting to particular aspects. Similarly, test data should be diverse and distinct from training, to properly simulate the intended use case of responding to novel aspects in test time. These are exactly the properties of OpenAsp, making it fit for the OABS goal. Finally, it should be pointed out that the distribution of aspects in OpenAsp is “naturally sparse”, in the sense that the aspects were collected systematically from a sample of topics in two multi-document summarization datasets, in the news domain.\n\nThat said, when targeting OABS for a relatively narrow domain, then higher aspect density, with frequent aspects in the domain recurring in the data, may be beneficial, since it is likely that such aspects would be repeated at test time as well. However, such a setting, as well as OABS adaptation across domains, were out of the scope of the current paper. We will add the discussion of such scenarios to the paper, with respect to both future work and the limitations of our current dataset. \n\n## Inter-rater agreement statistics in the data quality experiments\n\nIn response to your comment and question, we measured inter-rater agreements in terms of Cohen's-kappa (weighted linear, averaged across the 5 quality dimensions) between the scores assigned by the rater, the reviewer and after reconciliation (final score). \nThe following kappa values were obtained, indicating a high level of agreement, particularly with the final reconciled assessment:  \n* Reviewer-Rater: 0.753 (Substantial)\n* Rater-Reconciled: 0.93 (Almost Perfect)\n* Reviewer-Reconciled: 0.847 (Almost Perfect)\n\nWe will include these figures in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper introduces a manually constructed open aspect-based multi-document summarization dataset, anticipated to prove valuable for future research in aspect-based or query-focused summarization.\n\n- This paper conducts an in-depth analysis of the constructed dataset.\n\n- This paper validates the benchmark dataset using several multi-document summary models, including chatGPT.", "reasons_to_reject": "The limitations of this paper lie primarily in the verification process and data characteristics.\n- After the data was created by the annotators, the authors of the paper performed a curation process. However, there was no objective verification of this curation process. This raises the question of whether the dataset may be influenced by biases from the authors. It would be necessary to include not just the paper's authors, but also an external expert in the verification step of the curation process.\n\n- As indicated in Table 1, there are 1,266 aspects and 1,310 instances. This implies that the aspects are varied and instances are very sparse on an aspect. The nature of open aspect may naturally lead to a variety of aspects, but it also results in sparseness, making it difficult to conduct experiments based on aspects. Additionally, even when instances share the same aspect, they may come from different domains, limiting their use in domain adaptation studies. This points to limitations in terms of data usability.\n\n- This paper did not provide any inter-rater agreement statistics in the data quality experiments. It was stated that disagreements were resolved, but there was no explanation regarding the extent of discrepancies among the annotators.", "questions_to_authors": "1. What was the kappa coefficient between annotators in the data quality experiment?", "paper_topic_and_main_contributions": "This paper introduces a manually constructed benchmark dataset for open aspect-based multi-document summarization. To alleviate the challenge of reading multiple documents for the generation of aspect-based multi-document summarization data, this paper proposes a novel protocol. This protocol facilitates the creation of an aspect-based multi-document summarization dataset derived from existing multi-document summarization datasets, specifically the DUC and MultiNews datasets. Rather than reading multiple documents, annotators are required to only examine the multi-document target summaries and subsequently assign an aspect label. These aspect labels are not pre-determined; instead, they are identified by the annotators based on the content of the summary and are recorded in text form. This study further offers a comprehensive analysis of the dataset and validates its effectiveness using several multi-document summary models, including chatGPT.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ua3lG8r8QL", "reviewer": "Reviewer_XNMG", "summary": "", "questions": "Question A: How do you produce \"Oracle\" extractive summaries in table 5?\nQuestion B: When a generic summary is written without aspect and you provide it later, it is highly probable that you just label document topics as aspects. Do you think your aspects are similar to real-scenario aspects that can result in summaries different than generic summaries?\nQuestion C: What would happen if you just feed the documents (truncated) to BERT trained on generic summarization? I am not sure if this experiment is included in Table 6.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Aspects quality and alignment with the source documents main topics\n\nThank you for bringing up this issue. To start with, we argue that our annotation and curation methodologies were designed to yield a high quality set of aspects that aligns well with a significant portion of the main topics in the source documents, thus providing a useful sample of aspects for the dataset instances. Further, following your comment, we collected additional statistics from our analysis sets (lines 281-285, 313-316), in which each annotated aspect was aligned with all the source sentences, in all documents, to which it corresponds. Our findings, detailed below, further support our claim regarding the quality of the obtained set of aspects.\n\nWith respect to our methodology, the annotation guidelines instructed annotators to choose only aspects that correspond to dominant subtopics that were clearly described in the generic summary. As authors of generic summaries, namely expert summarizers for DUC and journalists for MultiNews, are originally guided to cover salient subtopics within the set of source documents, it is very reasonable to (transitively) assume that the set of aspects collected from these summaries indeed matches a set of prominent sub-topics from the source documents. Further, during the expert curation phase, aspect quality was further assured, as illustrated in Table 12 (page 17). \n\nThanks to your comment, we now further analyzed the statistics of source sentences corresponding to each of the 51 aspects in our analysis sets. We found that, on average, an aspect relates to 13.5% of all source documents’ sentences and appears in 68% of the source documents, indicating its dominance in the source documents of the topic. Further, only 11% of all aspect-related sentences relate to more than one aspect, indicating the high level of distinctness of the aspects. Finally, on average, 40% of all source sentences in a document set (topic) are related to at least one of the aspects extracted for that topic, indicating the substantial coverage of the set of aspects with respect to the entire document set content. Taken together, these findings assess that the collected aspects are central to the topic, covered thoroughly by its documents and collectively cover a main portion of its content. We will extend the Dataset Quality section with these findings in the final version of the paper.\n\n\n## Missing a comparison to similar datasets:\n\nThe paper currently summarizes the main properties of the datasets mentioned in the review and others in Table 1, and provides a high-level comparative discussion of related datasets in the background section. Following your comment, and leveraging the extra page of the final version, we will extend this discussion, while including the information below.  \n\nWe will specifically extend the comparison to the other two open aspect datasets mentioned in the table and in the review, namely AnyAspect and OASUM, including a comparative table with further details, beyond those in Table 1 (e.g. aspect label word lengths, average summary length, aspect label examples etc.). We will also include some qualitative comparison, touching on points such as: \n* In contrast to our dataset, both AnyAspect and OASUM are synthetically created, addressing solely single document inputs. \n* In AnyAspect, each named entity within the document is considered as an aspect, while the summary includes any source sentence mentioning that entity.\n* In OASUM, the aspects and the summaries were extracted from Wikipedia articles, where Wikipedia sub-titles are considered as aspects and their summaries are automatically extracted from the article’s abstract section via a greedy lexical similarity method. \n\nAs suggested in the review, the additional information and comparisons will provide valuable insights and context for our dataset's effectiveness and uniqueness.\n\n\n## Question A:\n\n“Oracle” extractive summaries in Table 5 are extracted following a common greedy methodology in the literature for this purpose (e.g. for creating gold extractive summaries).  Specifically, sentences from the document sets are greedily accumulated for the Oracle summary, one by one, while maximizing in each step the average of Rouge-1 and Rouge-2 F-1 scores between the accumulated summary and the reference (aspect-based) summary, until the score no longer improves. \n\n\n## Question B: \n\nIndeed, and as indicated in response to your aspect quality issue, the aspect labels in our dataset correspond to prominent sub-topics in the document set. From our data observations, the vast majority of them seem very reasonable for realistic user scenarios (see examples of randomly sampled aspects in Table 12). However, we cannot claim that the distribution of our aspects represents the distribution of aspects expected in arbitrary user sessions. Obtaining a *representative* set of aspects would ideally be based on collecting user sessions from an actual open aspect-based summarization system, which are currently not available, requiring an extensive targeted project (as done for “natural” datasets for search and question answering). From this perspective, our dataset offers a substantial step, relative to existing datasets, for developing effective open aspect-based summarization systems, which would further facilitate the collection of more representative user-driven samples of aspects.\n\n\n## Question C: \n\nTable 6 line 5 indeed reports the performance of a generic summarization system in generating aspect-based summaries on our dataset. This system is a BART finetuned on the generic summarization task, which receives as input the concatenation of the documents truncated to their lead sentences in each document in the set, as common in generic multi-document summarization, and, accordingly, without receiving the aspect label as input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset holds significant value in the field of aspect-based summarization, as it has the potential to foster further research in this domain. The experiments and benchmarks provided in the paper are well-defined and contribute to the robustness of the study.", "reasons_to_reject": "It appears that there is a concern regarding the methodology and evaluation of the aspect-based summarization dataset proposed in the paper. Specifically, the paper lacks clarity on whether writing aspects for available generic summaries truly generates a high-quality aspect-based summarization dataset. The aspects generated could potentially cover the main topics or headlines of the documents, but there is no experiment or analysis to demonstrate the quality of these aspects and their alignment with the main topics of the documents. This is distinct from the aspect relevance reported in Table 3.\n\nFurthermore, the paper misses a comparison to similar datasets like AnyAspect and OASUM, which could provide valuable insights and context for the proposed dataset's effectiveness and uniqueness.", "questions_to_authors": "Question A: How do you produce \"Oracle\" extractive summaries in table 5?\nQuestion B: When a generic summary is written without aspect and you provide it later, it is highly probable that you just label document topics as aspects. Do you think your aspects are similar to real-scenario aspects that can result in summaries different than generic summaries?\nQuestion C: What would happen if you just feed the documents (truncated) to BERT trained on generic summarization? I am not sure if this experiment is included in Table 6.", "paper_topic_and_main_contributions": "This paper provides a dataset for multi-document open aspect-based summarization. This dataset (OPENASP) contains a comparable number of instances (1k) compared to previous manually labeled datasets. The authors propose a protocol to make the manual annotation much easier. They use an available multi-doc summarization dataset and its generic summaries. The authors ask annotators to write aspects for the generic summaries. Additionally, they provide benchmark experiments using different models and oracles.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a dataset for open aspect-based summarization, a multi-document aspect-based summarization dataset with no fixed set of aspects. Authors recruit annotators to construct the resource from generic MDS datasets. Reviewers are satisfied with the quality of the constructed dataset and find it valuable for future research. Dataset analysis is also well presented.
That being said, the extraction and creation of aspect-based summaries out of generic summaries from MDS datasets raised questions among reviewers. Authors addressed much of the concerns in the rebuttal.
Decision: Accept-Main