mRedditSum: A Multimodal Abstractive Summarization Dataset of Reddit Threads with Images
Authors: Keighley Overbay
Jaewoo Ahn
Fatemeh Pesaran zadeh
Joonsuk Park
Gunhee Kim
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Abstractive Summarization Dataset
TL;DR: We present mRedditSum, the first multimodal discussion summarization dataset consisting of 3,033 annotated Reddit threads with images.

Abstract: The growing number of multimodal online discussions necessitates automatic summarization to save time and reduce content overload. 
However, existing summarization datasets are not suitable for this purpose, as they either do not cover discussions, multiple modalities, or both. To this end, we present mRedditSum, the first multimodal discussion summarization dataset. It consists of 3,033 discussion threads where a post solicits advice regarding an issue described with an image and text, and respective comments express diverse opinions. We annotate each thread with a human-written summary that captures both the essential information from the text, as well as the details available only in the image. Experiments show that popular summarization models---GPT-3.5, BART, and T5---consistently improve in performance when visual information is incorporated. We also introduce a novel method, cluster-based multi-stage summarization, that outperforms existing baselines and serves as a competitive baseline for future work.

MREDDITSUM: A Multimodal Abstractive Summarization Dataset of Reddit Threads with Images
Keighley Overbay' Jaewoo Ahn
Fatemeh Pesaran zadeh Joonsuk 2,34 Gunhee Kim
'Scoul National University
University of Richmond
NAVER AI Lab
*NAVER Cloud
(keighley.
weigghley.veebbay.... overbay, jaewoo. ahn, latemeh.pesaran (essaranevisie ieriiisiossnuuu. www.snu........ anu.ac.kr
park@joonsuk. org, gunhee@snu.ac..
http:/ http://viison.ee snu. ac. kr.krrkkkk..rr. httppooecss//reedittsst tsum
Abstract
Post:
Post Summary:
The OP usked for help with finding the right coffee table shape to match their brown leatber sectionl.
The growing number of multimodal online dis- cussions necessitates automatic summarization save time and reduce content overload. How- ever, existing summarization datasets are not suitable for this purpose, as they either do not cover discussions, multiple modalities, both. To this end, we present MREDDITSUS UM, the first multimodal discussion summarization dataset. It consists of 3,033 discussion threads where post solicits advice regarding an issue described with an image and text, and respective ments express diverse opinions. We annotate each thread with a numann-wrttee summary that captures both the essential information from the text, as well as the details available only the image. Experiments show that popu- lar summarization and T5 consiseetlyyy improve in performance when visual information incorporated. We also introduce novel method, cluster-based multi-stage summarization, that outperforms existing baselines and serves competitive baseline for future work.
We this couch for our living and need help finding the perfict coffee table go with it.
Comments:
Comment Summaries:
User I: I would circular table.
C1,C2,C4: Commenters sug- gest difforencly shaped coffee from the square one picture, round or triangular or hesagonal.
User 2: Definitell round! There are shorp angles already ...
User There's is way much f'umi- ture in this space, the ottoman has to
C3: commenter suggests eliminating ottoman takes lso much space.
User 4: I think you should look for natural wood triangular shaped coffee table
CS: A commenter suggests adding mig.
User S: You should pet iag.
Full Summary:
asked for help with finding right coffee table shape to match their brown leather sectional. Commenters suggested differenily shuped coffee table from square one has already,
BART,
round, triangular, or hesagonal. A fow commenters suggested eliminating the ottoman. as it is too big for the small space. Others suggested adding
Table An example from the MREDDITSUM dataset. Both the post, several viewpoints from the comments, and the overall thread are summarized along with im- portant content only available in the image (in green), or in both image and text (in blue).
1 Introduction
With the increased popularity of online discussion forums like Reddit, discussion threads that consist post and various numbers of comments, have quickly accumulated. thus becomes overwhelm- ing for users to sift through the threads to find the information they seek, which in turn has led to the development of automated means for text-only dis- cussion summarization (Bhatia et al., 2014; Fabbri et al., 2021, 2022).
Yet, multimodal summarization has so far been lim- ited to news and instructional domains (Zhu et al., 2018; 8; Sanabria et al., 2018; Palaskar et al., 2019; Liu et al., 2020) that are not casily transferable to online discussions surrounding images.
To fill the gap, we tackle multimodal discussion summarization. In particular, we consider Reddit discussion threads in which the post solicits advice regarding an issue described with an image and text, and commenters offer opinions, as opposed to simple reactions or jokes. Here, the goal is to generate an abstractive summary faithfully captur- ing the information from the post-both image and text-and comments. This task especially chal- lenging because along with the need to effectively process the multimodal input, a quality summary
However, discussion threads are often multi- modal, containing visuals in addition text. This added modality cannot be ignored, as plays a key role in the respective discussions. For example, in Table 1, the image of the couch is essential for discussing which coffee table would go well with it.
"Equal contribution.

---

must provide good coverage of commenters' vary- ing perspectives and opinions without redundancy. To facilitate research in this direction, we con- tribute the Multimodal Reddit Summarization (MREDDITSUM) dataset, consisting of 3,033 Red- dit discussion threads containing posts (text and image) and comments (text-only), each accompa- nied by human-written summary, shown in Table 1. We carefully select subreddits with discus- sions surrounding an image, and collect summaries that not only summarize the text, but also make reference to relevant information present only in the image. See Appendix C for more examples.
different domains including NYT articles, Red- dit, StackExchange, and Email threads. Answer- Summ (Fabbri et al., 2022) another dataset con- sisting of 4,631 crestion-answering discussion threads sourced from StackExchange- Answer- Summ shares the most similarities with our dataset, as they also summarize multi-speaker threads, and their annotation pipeline shares some similarities with ours. They also cluster the comments and sum- marize these groups before going through a final summary editing process, similar to our pipeline. The key differences between this dataset and ours is that AnswerSumm is only text-based with no images and operates in different domain, as they are all question-answering threads from StackEx- change. In contrast, our dataset includes both im- ages and text, and focuses Reddit threads where the images play a key role. Additionally, in our annotation pipeline we summarize the original post and image as well, which to our knowledge has not been done in any other forum summarization dataset. This is useful because often the posts alone have unclear intent that may require context derived from the image or forum domain itself.
We also propose cluster-based multi-stage sum- marization (CMS), novel method to summarize multimodal discussions. It processes discussions in three stages: (i) comments by similarity, (ii) each cluster is summarized in a sentence, and (iii) the gluster-summaries are summarized. Experiments show that CMS con- sistently outperforms popular large language mod- els (LLMs) for summarization 5PT-3355 (Brown et al., 2020), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020). Also, incorporating image information, either as dense vector or in text cap- tion, consistently boosts the performance. Our main contributions are as follows:
first clustered
Other related summarization datasets include multi-turn datasets such SamSUM (Gliwa et al., 2019), consisting of chat-like dialogues and human- annotated summaries, and EmailSum (Zhang al., 2021), containing work-related emails and both long and short reference summaries.
We present MREDDITSUM, the first multi- modal discussion summarization dataset with human--witter summaries with essential in- formation from both the text and the image. Table 2 presents comprehensive comparison with existing summarization datasets.
Overall, though there is a small variety of exist- ing discussion thread summarization datasets, they are all currently only text-based and none of these tackle both original post and thread summarization.
We propose cluster-based multi-stage summa- rization (CMS), novel method to summarize multimodal discussions outperforming com- petitive basclines like GPT-3.5, BART and T5, as well as their vision-guided variations.
Multimodal Sunmarization. Though other multimodal research areas such VQA (Agrawal et al., 2017) and text- cott-image pretraining (Radford et al., 2021: Li et al., 2022a, 2023) have been gain- ing attention in recent years, there only exist small handful of works that address multimodal summa- rization, which aims to generate a summary that includes salient information from inputs with mul- tiple modalities. For example, MSMO (Zhu et al., 2018; Qiu et al., 2023), Multimodal Summarization with Multimodal Outputs, takes inputs of various modalities and outputs both text-based summary and a representative image.
2 Related Work
Discussion Thread Summarization. Despite the prevalence of discussion threads online, it has been understudied for summarization. This is likely due to the fact that it can be difficult target for extrac- tive summarization, though there has been small ex- tractive summarization dataset (Bhatia et al., 2014). More recently, several abstractive summariza- tion datasets have been proposed. ConvoSumm (Fabbri et al., 2021) presented dataset of 2000 summarized forum threads, 500 from each of
However, our task aims generate unimodal output-ther is, a purely textual summary. This is similar to the multimodal summarization done on the How2 Dataset (Sanabria et al., 2018; Palaskar

---

Dataset
Domain
# Docs
Doc Len
Sum Len
# Turns
# Speakers
Modality
StructSum
MREDDITSUMOOUS)
Forum
3,033
691.0
91.0
22.6
15.59
AnswerSumm
Forum
4,631
787.0
47.0
6.4
6.17
Forum
500
641.0
65.0
7.88
SamSUM
Dialog
16,396
124.1
2.39
CNN/DM
News
286,817
766.0
53.0
MSMO MailyMail
News
314,581
722.7
55.0
1
How2
Video
79,114
291.0
33.0
1 2,
: speaker info not provided t: text /:: image /v: video
Table 2: A comparison of MREDDITSUM and other summarization datasets. Among forum-based and multi-turn datasets, MREDDITSUM is the only multimodal dataset, and has the highest summary length, number of turns, and number of speakers. Length is reported in the average number of words, and turns are the average number of each instance of a post, comment, or speaker change. StructSum denotes whether there are structural-level summary annotations, such for comment clusters. Statisties taken from the respective papers for AnswerSumm (Fabbri et al.. 2022), ConvoSumm(Fabbe et al., 2021), SamSUM (Gliwa et al., 2019), CNN/DM (Nallapati et al., 2016). MSMO MailyMail (Zhu et al., 2018), and How2 (Yu al., 2021).
et al., 2019), where textual transcript of the video along with the video frames generated into text summary. (Yu al., 2021) reported that in- corporating the additional modality of the video frames into their summarization models showed improvement compared to text-only based mod- els. Though this multimodal summarization task is the most similar to ours, there are some key differ- ences. The How2 dataset uses short vidco captions useudo-summarte instead of detailed human- annotated summaries like we curate for MREDDIT- SUM. Additionally, text rich multi-speaker discussion, rather than transcript of audio. Finally, MREdDITSUM's threads are specifically selected to include images where their information is nec- essarily included in the summary, whereas there is no such assurance for How2's videos.
summarizing the text is sufficient, and multimodal model is unnecessary.
Criterion The discussion needs be mean- ingfully summarizable. Many Reddit threads that include images are meant to incite reactions from other users, or to be shared in a jocular manner. On the other hand, some threads consist of the post clearly asking for advice or opinions, thereby eliciting diverse responses from number of com- menters. Summarizing these opinions along with the advice would be helpful for readers under- stand the gist of the threads.
Sources. Given the aforementioned criteria, we identify subreddits, e.g. r/fashionadvic (See Appendix A for the complete list), which consist primarily of image-based posts where the original poster is soliciting advice opinions about either clothing or interior design. We collect all threads from these subreddits with over 5 comments from years 2015-2022. Collection done with RedCaps (Desai et al., 2021) modified to collect all com- ments from each thread. We additionally follow similar preprocessing steps (Ahn et al., 2023). re- moving all posts that contain NSFW content or images with faces. We also remove any comments with NSFW content, and comments posted by bots. All responses these removed comments are also removed. We replace URL's with the "[URL]" token.
3 The MREDDITSUM Dataset
3.1 Data Preparation
To construct a meaningful multimodal discussion summarization dataset, we imposed three major criteria when selecting Reddit threads.
Criterion The discussion thread needs to contain an image. Since Reddit does not allow images embedded comments, this means that the post needs to contain an image.
Criterion 2. The discussion needs to be centered around image in such a way that the information only available from the image plays key role in the discussion. some threads, the image does not provide any information, e.g. a favorite char- acter of the original poster. In such cases, simply
3.2 Annotation
We then annotate the data after selecting qualified workers from Amazon Mechanical Turk (MTurk). We limit our workers to those from English-

---

speaking countries with a HIT approval rate over 98%, with greater than 5000 HITs approved. For all tasks, workers are required to pass a qualifica- tion task where the results are manually checked for quality. Any workers who are found to submit low- quality work have their qualification revoked. In total, pool of 40 annotators were qualified to per- form annotation tasks. As each task was performed separately, it possible for multiple annotators contribute a single summary. so Additional de- tail on the annotation interface and instructions are found in Appendix B. The annotation is conducted in a sestep annotation pipeline as follows.
Structure
Document
Summary
1.62 sents
1.07 sents
Original Post
18.87 words
23.14 words
Comment Clusters
6.63 sents
1.34 sents
85.05 words
20.17 words
21.6 comments
Full Thread
37.41 sents
5.32 sents
691 words
91.0 words
Table 3: Average statistics across the original post, com- ment clusters, and full thread structures of our dataset.
notators along with the original post and image, and ask them to summarize within one or two sentences the main opinions present in each group of com- ments. We encourage the annotators to reference objects details from the image when necessary. For consistency, we instruct the annotators refer to the commenters as "Commenters" as opposed to people, users, or other words.
Step 1: Original Post Summarization. In the first step, we present workers with the original post along with the image from that post. We ask the annotators to summarize in a single sentence the intent of the original poster, well the most rel- evant details from the image. We use this method because post that simply reads "blue or black?" may only be comprehensible when paired with the image of blue and black heels next to a blue dress, and a true text-only summary should be comprehen- sible without the image. Our summary may then read "The original poster asked if blue or black heels would match better with strapless, knee- length blue dress." eliminating the need of the im- age to comprehend the question. In this way, all information necessary to understand the question should be self-contained within the summary.
Step 3: Summary Synthesis. Finally, catenate the original post summary well the comment cluster summaries, in descending order of their saliency-scores. We then present these sum- maries once more to annotators and ask them to edit them for fluency and readability. We encour- age annotators to reduce repetitive wording, add connectives between sentences, and to rearrange sentences so that related topics are next to each other and the overall summary reads as more nat- ural. We also ensure all summaries are written entirely in the past-tense.
Step Comment Cluster Summarization. We first cluster the comments to identify groups of comments that share a similar opinion. We follow the method described in AnswerSUMM (Fabbri et al., 2022) to allow for clusters of varying sizes and number. We use RoBERTa- based model fine-tuned for semantic similarity get sentence embeddings of the tep-level comments from each thread. We then use agglomerative clustering with average linkage, cosine distance, and a maximum distance of 0.5 to generate clusters of comments.
3.3 Dataset Analyses
Statisties.
The resulting dataset contains a total of 3,033 posts and summaries. We split these into a train, test, and validation set of sizes 2729, 152, and 152, respectively. We present further statistics in Table 2, where we compare with similar sum- marization datasets from few diffcrent domains.
The average summary length for MREDDITSUM is longer than other datasets; however, this is not surprising given the nature of summarizing varying opinions, of which there could be many. Addi- tionally, we describe the structure-level statisties in Table 3; note that while the average length of the Original Post summary is longer than the document, this is due to the additional image description and context. For the full thread, the summary is 13.2% as long as the input on average, which is compara- ble to SamSUM's 19% and How2's 11.3%.
We then rank the comment clusters according to their size and Reddit score. We take the sum of all Reddit scores of the top-level comments in a single cluster as a saliency score of the cluster. We select the top 5 clusters with the highest saliency scores and use these for annotation. We do this limit the size of the summary and help remove irrelevant comments, while encouraging larger clusters of comments with a similar sentiment.
Summary Quality. In order to ensure the qual- ity summary, we additionally performed an exper-
We then present these groups of comments to an-

---

iment rate the annotated summaries out of metries. Following closely related work, Sam- SUM (Gliwa et al.. 2019), we randomly selected 100 thread- -ummary pairs and had 2 independent judges from MTurk grade them on a quality scale -1, or 1, where 11 means a summary is irrele- vant or does not make sense, O means summary extracts only part of the relevant information or makes some mistakes, and I means summary that is understandable and delivers a brief overview of the relevant information from the thread. We asked annotators to score summaries on overall quality, fluency, and faithfulness, similar to our human an- notation found in Sec 5.2. We found the scores were highly positive, with average scores of 0.95, 0.96, 0.83 for overall quality, fluency, and faithful- found Gwet's
Extractive Oracle ROUGE
Dataset
RI
R2
RL
MREDDITŠUM (ours)
36.52
11.95
31.42
AnswerSumm
40.05
18.45
35.70
ConvoSumm,eddt:
35.74
10.45
30.74
DailyMail
55.23
30.55
51.24
Table A comparison of Extractive Oracle ROUGE scores on MREDDITSUM and related datasets. The lower the score, the more abstractive the summaries are. Results for related works are from the respective papers (Fabbri et al., 2022, 2021).
inal Post: Original Post", with "Image: Image Caption." appended for models that include image captions. We then additionally append the com- ments in the form "User I: Comment 1. User 2: Comment 2. ...", where each username has been anonymized. Comments are listed in the order that they are scraped from Reddit in. The target output is the result of our final summary.
ness respectively. Additionally, ACI agreement scores of .91, 0.89, and 0.53, cor- responding high, high, and moderate agreement, respectively. Note that we used the Gwet's AC1 score for interannotator agreement as it performs well despite class imbalances where agreement is high. (Gwet, 2008; Wongpakaran et al., 2013; Wong et 2021)
4.2 Evaluation Metrics
Following the standard metric for summariza- tion evaluation, we use the ROUGE (Lin, 2004) and BertScore (Zhang* et al., 2020). ROUGE! measures the salience of model-generated sum- maries by comparing n-grams in the generated summary and gold summary. We consider three variants: ROUGE-1/2 (R1/2) measure the uni- gram/bigram overlap, and ROUGE-L (RL) deter- mines the longest common subsequence between summaries. BertScore" computes a soft token sim- ilarity using contextual embeddings output from BERT, instead of word matches. We use the default RoBERTa- large model and rescale with baseline.
Abstractiveness. Extractive-Orace ROUGE scores in Table 4 show that our dataset is similar in abstractiveness other multi-turn datasets, and much more abstractive than Mail Though scores are not available for MSMO, it is expected that the scores would be similar to DailyMail.
Relatedness between Text and Images. We also calculate the CLIPScore (Hessel al., 2021), metric that measures the correlation between text and an image, to determine how grounded our summaries are to the images from each thread. Our summarics have an average CLIPScore of 74.62, the post summaries alone achieve 74.89, and the comment cluster summaries alone score 68.34. These suggest our summaries, especially the post summaries, are well-correlated with the images.
4.3 Baseline Models
We consider four text-only bascline models: GPT- 3.5 (zero-shot), BART, T5, and LongT5 (fine- tuned), as well as their extensions to make use of image information, either as image captions or embeddings.
4 Experiments
4.1 Task Description
Extractive Baselines (Lead-1, Comam-Cmmmet,, Ext-Oracle). Lead-1 uses the first sentence from the document as the summary, and uses the leading top 5 comments from the thread. Ext-Oracle extracts passages from the document to achieve the maximum possible ROUGE score, and
We consider the multimodal summarization task, where the input includes all original text and the image and the output is text-only summary that describes both the document and image. The text includes the post and comments, and the goal is accurately summarize both the original poster's intent and commenters' opinions. For this task, we format the text input as the following: "Orig-
https://github..o///prttttouuuuu hhtnss//gathu..omm//iiighhhhrrr

---

thus is the highest possible performance from an extractive model.
Comments
Biac frac with stond.
Text-only Baselines (GPT-3.3, BART, T5, LongT5). GPT-3.5 (Ouyang et al., 2022) is an LLM that has shown excellent zero-shot perfor- mance in summarization tasks (Goyal et al., 2022; Bhaskar et al., 2023). We use the largest model, text-davinci-003, through the OpenAl API, with the prompt "Summarize what the original post was asking about and the general opinions of the commenters." ,, which is determined empirically to perform well and closely mimic the instructions given to annotators. We also evaluate three fine- tuned models, BART-base (Lewis et al., 2020), and T5- base (Raffel et al., 2020), which are high- performing LLMs with good summarization abil- ities, as well as LongTS-base (Guo et al., 2022) which is an extension of T5 that capable of han- dling longer input sequences. We pretrain them on the CNN/DailyMai (Nallapati et al., 2016) summa- rization dataset before fine-tuning it for our task.
serrect supec duch skak
Comment Clustering
ComClus,
ComClus,
wire
paead
Cluster Summarizzatin
OP
Classum
Classum
Conwitic suggatted setted
Cluster Sum Summarization
Everoniittng fest conserers is gugaroed
ceel-
and viice willpoper to the sael
Final Summary
Figure 1: An illustration of Cluster- Multi-stage Summarization (CMS): (1) comments first clustered by similarity, (2) each cluster is summarized in sen- tence, and (3) the summaris are summarized. (OP: the original post I: the post image / ComClusg: the k-th comment cluster / N: the number of comment clusters ClusSumg: the generated summary the k-th comment cluster)
Extensions with Image Captioning (GPT3.5- ImgCap, BART-Ing T5p LongTT-- ImgCap). We extend the text-only baselines to in- corporate visual information through the use of an image caption, denoted as GPT3.5- Impp.p BART- ImgCap, Im-Imp and LongTT5-ImgCan respec- tively. They take advantage of powerful LLMs without large amounts of multimodal training to understand visual features. For image captions, we use the BLIP2 model (Li et al., 2023) trained COCO image captions (Chen et al., 2015) and generate multiple image captions for each image using nucleus sampling. Since more detailed and grounded image caption that describes concrete ob- jects is best for this task, we use image-grounding model, GLIP (Li et al., 2022b), to score each cap- tion by grounding it with the image, and calculate how many image-text grounded pairs are above threshold of 0.7. We then select the image caption with the highest score and append the caption to the input after the original post. We then fine-tune BART-ImgCap. Ta- and LongT5-ImggCan as described above; for GPT3.5-Imgg we use the caption-appendee prompt.
tion for the How2 dataset. We modify the original models by instead using 768-D ViT-base (Dosovit- skiy et al., 2021) image embeddings as input, as they have shown excellent performance as an image backbone. We use cross-modal dot product atten- tion with forget-gate and image transformer, this version performed best in our experiments. We use the same T5-basc and BART-base pretrained on CNN/DM to initialize the encoder and decoder. For VG-BART, we pretrain the visual layers using the COCO image captions before fine-tuning on our dataset; VG-T5 shows no performance increase from visual pretraining, so we initialize its layers from scratch.
4.4 Cluster-based Multi-stage Summarization
Extensions with Vision-Guidance (VG-BART, VG-T5). Vision-Guided BART and T5 are pre- sented in (Yu et al., 2021) for multimodal summa- rization. They include additional visual layers that receive video embeddings as input, and show state- of-the-art performance in multimodal summariza-
One challenge in summarizing discussions is that they can be very long. To confirm that this is caus- ing an issue, we conduct preliminary experiment on the fine-tuned BART model by comparing the results of two different test subsets: the long subset with more than 22 turns and the short subset with

---

Model
R1
R2
RL
BertS
et al., 2020), we use learning rates 3e-5 fine- tune the pre-trained parts of model weights, and
Extructive
Lead-1
15.23 3.46
13.24
11.89
1.5e-4 to train the newly added visual layers
Lead-Cord Saad-Commmnt
22.86
5,55
20,43
7.16
Ext-Oracle
36.52
11.95
31.42
16.71
VG-BART and VG-T5. The decoding process uses beam-search with a size of 5. The average training time for BART, T5, BART-Cap, and T5-Cap is ap- proximately 5 hours; the average training time for VG-BART and VG-T5 is about 8 hours, with the additional visual layers adding about 100 million
Zero-shot Prompting
GPT-3.5
34.29
9.10
30.39
30.15
GPI-3.5-ImgCap
34.59
9.41
30.39
31.07
Fine-tured
BART
44.33
18.4
41.71
41.61
VG-BART
44.97
18.75
42.29
40.85
BART-IngCap
44.91
18.54
42.12
41.34
CMS-VG-BARE VG-BART (ours)
45.13
18.81
42.56
42.13
extra parameters. We use the same training epochs, size for cluster-based multi-stage summarization. All
CMS-BART- BART-ImgCap (ours)
45.55
19.28
42.87
43.89
batch size, learning rates, and
T5
45.29
18.97
42.4
42.32
VG-T5
45.58
18.94
42.75
42.3
T5-ImgCap
45.61
18.97
42.63
42.59
results shown are an average of two runs.
LongT5
45.98
19.44
43.12
41.95
Long 15-
46.6
19.86
43.7
42.63
CMS-VGG-TS (ours)
45.71
19.21
42.97
42.72
5
Results and Analysis
CMS-TS-Imggap (ours)
47.29
19.86
44.13
44.74
Table shows the results of all models evaluated across the test set. We see that our model, Cluster- based Multi-stage Summarization (CMS), outper- formed baseline models for all metrics across both T5 and BART-based models. We believe this is due to models' ability better handle the long length of input threads, even outperforming LongT5 models; see $ D.I for more detailed analy- sis. general across all model types, models that contain image information through an image cap- tion outperform those that only have access to text- information. This supports that our dataset requires multimodal understanding in order to perform well the summarization task. To confirm this, we additionally computed the CLIPScore between the image and the first sentence fo the generated sum- maries, which corresponds to the post summary and is where most of the image information is found. The results in Table 6 support that our methods incorporate more image information compared to non-visual baseline. Vision-Guided models us- ing text embeddings showed mixed results, with marginal or no improvement els; we believe this to be due a limitation of these models effectively incorporate image in- formation. Though they show strong performance on the How2 summarization task (Yu et al., 2021), )), mRedditSum has longer input and summary length, images, and fewer documents, likely contributing to the performance differences. Additionally, we note that T5 models show the best performance, fol- lowed by BART models and GPT3.5 models. For GPT3.5 models, we note that the low scores are likely due inconsistencies in summary format, length, and detail, due to the zero-shot setting, but still receive relatively reasonable BertScore scores.
Table 5: Results for the summarization task mReddit- Sum. Models with "-ImgCap" in the name incorporate image information via image caption, and "VG-", via image embedding. Others are text-only models. Cluster- based multi-stage summarization (CMS) our proposed method of processing discussions in three stages.
less than or equal to 22 turns. The performance on the long subset is noticeably worse than that on the short subset, lower by 4.95 ROUGE-1 and 6.1 BertScore.
To effectively handle this challenge, we present a novel method named cluster-based multi-stage summarization (CMS), consisting of three stages (See Figure 1):
1. Comment Clustering. Similar comments are clustered using RoBERTa sentence embed- ding and agglomerative clustering.
2. Cluster Summarization. Each cluster sum- marized in about a sentence using an LLM with image captioning, or a vision-guided LLM, such as VG-BART or VG-T5.
text-only mod-
3. Cluster-summary Summarization. The cluster summaries are concatenated and fur- ther reduced into coherent summary using a separate model, which is either an LLM with image captioning or a vision- -uiiddd LLM.
4.5
Implementation Details
The fine-tuned models are trained for 50 epochs on single Titan RTX GPU for the BART and T5 models. We use batch size of 4, and fol- lowing (Yu et al., 2021; Raffel et al., 2020; Lewis

---

Model
CLIPScore
BART
68.29
Ovaral
605
4%
105
CMS- BaSS-aart-mgCap
70.26
T5
69.34
Fathfllnes
305
CMS-TS-ImgCap
70.58
Table 6: The average CLIPScores computed between the first sentence of the generated summary and the threads' corresponding image.
CMM ingCap (Ouns)
Tie
ITS Imgrap
We provide further analyses on the effect of input length and subreddit category on performance in $ D.
Figure Human evaluation results randomly sam- pled summaries of CMS-T5-Imgent and T5- 15-ImmCap models.
5.1 Qualitative Analysis
more naturally written, faithfulness measures how truthful the summary is the document, and the overall quality represents general user preference. We randomly sample 25 datapoints from the test set and receive 3 annotations per sample. We note that this limited number of datapoints is due to the fact that this evaluation task is highly challenging for human annotators, given that the input, including the original post, threads, and image, is long and complex.
In addition to automatic evaluation, we check the test results manually for qualitative analysis. Several results can be found in Table 7. The pri- mary advantage of our method, CMS, that has a greater coverage of relevant opinions compared to the baseline models. It is better able to filter out irrelevant or strange comments, while keeping the important opinions and including ones that are presented late in the thread.
We also find that all models, even those incor- porating image information, are still prone to hal- lucinations of what in the image. These include incorrect deseriptions of object color and style, as well as describing objects that are not present in the image at all. Though our multimodal models are generally better at incorporating visual details than text-only models, their power to interpret the image seems still limited; we believe this be due to potential undertraining of the text-vision fusion layers in the VG models, and the limitations of image caption models.
Figure 2 shows the majority vote results that our summaries are overall more preferable in terms of fluency and overall quality, with similar perfor- mance for faithfulness. We believe this to be an indicator of our model's better ability to effectively summarize the thread, offering more fluent and higher-quality summaries. The similar faithfulness scores likely due to method sharing the same base model and image caption as the baseline, T5- I5-Imp granting a similar ability incorpo- rate correct image and text information.
Conclusion and Future Work
Thus, while our CMS model can overcome one weakness of the baseline multimodal summariza- tion models, still believe there be significant room for improvement in the field of multimodal models, and hope that MREDDITSUM can help fa- cilitate such research.
Online discussions are increasingly becoming mul- timodal, yet there are not sufficient resources for summarizing them. To this end, we presented MREDDITŠUM, the first multimodal discussion summarization dataset containing 3,033 discus- sion threads and images with human-written sum- maries. Threads were carefully chosen so that the images play a key role in the respective threads, and summaries were written to capture this. Ex- periments showed that summarization models mak- ing use of visual information consistently outper- form those that do not. Additionally, we introduced Cluster- based Multi-stage Summarization, which accounted for the structure of discussion thread
5.2 Human Evaluation
We additionally perform human evaluation studies via MTurk to compare the summarics generated from CMS-T5-IngCap (ours) versus the baseline 15-Imp model. Based on similar works such as (Zhang et al., 2021), we use three metrics to measure the summary quality: fluency, faithfulness, and overall quality. Fluency measures which is

---

Image
Thread
15
CSpImgCap
CMS-TS-ImgCap
Original Post: Looking for
The OP asked for ideas on how to style black and white floral necklace. One commenter saggested wear- ing neeklace with ted shirt. Another suggested weating button collired whise with high waisted shorts and flats.
The OP wanted auvice on how to style their necklace. which
The OP asked for ideas on how to style their necklace. which is white nosaur palerm on it One commenter sugpested ing a funky skirt and fic- led shirt with the necklave. Another commenter recom- mended button collared white shict with high waisted shorts and flats. One user shared link OP's nack- lace.
ideas on how style this necklace User This gives me Ms. Frizzle vibes. Maybe embrace
floral pattern on it One commenter suggested wearing the necklace with a fitted shirt. Another sug- gested wearing button up collared white shitt with high waisted shorts and flats.
di-
User Yes! Fanky skirt that poofs out, fitted shirt! User Fitted button up collard white shirt with high waisted shorts and llats OP: Ouh that's great ideal Thank you! User 4: [URL]] Link to necklace if anyone is interested...
Original Post: [Update]
The OP aslsed for advice coffee table and accent chair for their living mam, which has brown leather arn- chair and tan leather couch. commenters suggesteed a wood coffee table with a walnut finish and solid white marble top One commenter recommended round or oval coffee ta- ble. Another commenter sug- gosted brown throw pillows and blankets to match the rug. One user suggested OP get non-shedding dop.
The OP asked for advice on coffee table and ao-
The OP asked for help with picking out a collee table and nccent chair for their blne couch. Most commenters suggested getting warm wood collee table. Others sugeeteed a brown leather amnchair creum colored ac- cent chair. One commenter suggested getting throw pil- and blankets to match the rug. Another commenter asked where the rug, from, and the OP said was from AptZB.
Couch anived and this is the coded up choos- ing. Any coffee table ac- cent chair recommendation?? Something with warm wood color would look nice. Check
cent chair for their bine comch. Most commenters suggested wood coffee ta- ble with walnut and solid white marble top. One commenter ommended round or oval coffee table. Another com- menter suggested getting non shocding dog.
antique shops or local furniture shops to find something that has some chamcter and made of solid wood. User 2: Yes, round oval would look nice....
Table 7: Examples of summaries generated from various models. Across all models, hallucinations regarding the image (highlighted in red) are present; however, these are reduced with multimodal models that incorporate image-only information (highlighted in green). Our CMS models tended to include more relevant details (blue) while removing irrelevant comments (orange).
data and outperformed baseline methods. We hope this dataset will help to facilitate active research on multimodal discussion summarization.
be accommodated within a single Titan RTX GPU.
Ethics Statement
As we propose a novel multimodal dataset, there are ethical considerations about the use of the data, Privacy. All data are sourced from Reddit, which is publicly available. Following Desai et al. (2021); Ahn al. (2023), additional measures have been taken to address privacy considerations. This includes the exclusion of images or discussions with clear identifying information, such as names or faces. Additionally, posts that are removed by their authors from Reddit also render the image unavailable for our dataset, we only provide the links the images. Thus, any users who con- cerned about their post being in the dataset may easily remove it from the dataset by deleting it from Reddit.
Limitations
As with any dataset, there are some limitations to MREDDITSUM. Though it is of comparable size to many other summarization datasets, the relatively small size of the dataset makes it hard utilize without significant pretraining, thus limiting the use of the dataset to those with access to large-scale pretraining datasets or pretrained models.
MREDDITSUM only includes Reddit threads with single images, as opposed to multiple images or videos. There is thus still room for improve- ment for multimodal summarization to additionally consider these threads.
Furthermore, our dataset considers only English- language threads from a single forum, Reddit, and a limited number of subreddits. There thus may be some additional bias due to the relatively small domain and raw nature of the dataset.
Bias. As all data are sourced from real discus- sions on a public forum, there may be biases within the discussions due to the demographics of the Red- dit users. Though we use a NSFW filter to remove inappropriate words, and look at each datapoint by hand to further filter out any harmful or inappro- priate images or discussions, it is possible a few may still be present in the dataset. Less obvious bias such as stereotyping based on gender, etc. may also still be present in the dataset.
For our cluster- based multi-stage summarization method, one limitation is the need to train an ex- tra model in addition the base summarization model. As a result, our method incurs some com- putational overhead. However, it is worth noting that both the training and inference processes can

---

Intended Use. The MREDDITSUM dataset is intended be used for rescarch purposes only, and the use is subject to the the Reddit User Agreement, Privacy Policy, and Content Policy".
acts of individual messages help? In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), papes 2127- 2131, Doha, Qatar. Association for Computational Linguistics.
Annotator Compensation. We ensured that our annotators were paid a fair wage of approximately USD S16/hour, which is greater than the minimum wage in all countries that we recruited annotators from: The United States, Canada, Australia, New Zealand, and Great Britain. The time to complete each task was determined by running multiple tri- als with researchers, and the payment per task was calculated from this time. The cost per datapoint was approximately $3.50, with some longer dat- apoints costing more to compensate for the extra annotation time.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish. Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Acknowledgements
Xinlei Chen. Hao Fang, Tsung-Yi Yi Lin, Ramakr- ishma Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitmick. 2015. Microsoft coco captions: Data collection and evaluation server.
This work was financially supported by SNU- NAVER Hyperscale AI Center, as well as the Insti- tute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2019-0-01082, SW StarLab and NO. 2021-0-01343, Artificial Intelli- gence Graduate School Program (Seoul National University)), and the National Research Founda- tion of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1 Joonsuk Park and Gunhee Kim are the correspond- ing authors.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. 2021. Redcaps: Web-curated image-text data created by the people, for the people. In Pro- ceedings of the Neural Information Processing Sys- tems Track on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, vir- tual.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations.
References
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Mar- garet Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2017. Vqa: Visual question answering. Int. J. Comput. Vision, 123(1):4-31. 31.
Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. ConvoSumm: Conversation summa- rization benchmark and improved abstractive sum- marization with argument mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 6866-6880, Online. Association for Computational Linguistics.
Jaewoo Ahn, Yeda Song, Sangdoo Yun, and Gunhee Kim. 2023. MPCHAT: Towards multimodal persona- grounded conversation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3354- 3377, Toronto, Canada. Association for Computa- tional Linguistics.
Alexander Fabbri, Xiaojian Wu, Srini Tyer, Haoran Li, and Mona Diab. 2022. AnswerSumm: A manually- curated dataset and pipeline for answer summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2508 2520, Seattle, United States. Asso- ciation for Computational Linguistics.
Adithya Bhaskar, Alexander R. Fabbri, and Greg Dur- rett. 2023. Zero-shot opinion summarization with gpt-3. In Findings of the Association for Computa- tional Linguistics: ACL 2023. Association for Com- putational Linguistics.
Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2014. Summarizing online forum discussions- can dialog
Bogdan Gliwa, Iwnna Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human-
http:://wwww.rddiiin.c..////////

---

annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Nayu Liu, Xian Sun, Hongfeng Yu, Wenkai Zhang, and Guangluan Xu. 2020. Multistage fusion with forget gate for multimodal summarization in open-domain videos. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1834-1845, Online. Association for Computational Linguistics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang, 2016. Abstrac- tive text summarization using sequence-to-sequeee RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Mandy Guo. Joshua Ainslie, David Uthus, Santiago On- tanon, Jiammo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. T5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguisties: NAACL 2022, pages 724- 736, Seattle, United States. Association for Compu- tational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Kilem Gwet. 2008, Computing inter-rater reliability and its variance in the presence of high agreement. The British journal of mathematical and statistical psychology, 61:29-48
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan
Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Shruti Palaskar. Jindfich Libovický. Spandana Gella, and Florian Metze. 2019. Multimodal abstractive summarization for how2 videos. In Proceedings of the 57th Annual Meeting of the Association for Com- putational I Linguistics, pages 6587-6596, Florence, Italy. Association for Computational Linguisties.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising saaeenceeto-eqqeneee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online. Association for Computa- tional Linguisties.
Jielin Qiu, Jiacheng Zhu, William Han, Aditesh Kumar, Karthik Mittal, Claire Jin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Bo Li, Ding Zhao, and Lijuan Wang. 2023. Multisum: A dataset for multimodal summarization and thumbnail generation of videos.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishinn, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900. PMLR
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(14)):
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Wei Chang, and Jianfeng Gao. 2022b. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 11975,
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar. Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018 How2: large-scale dataset for multimodal language understanding. In Proceedings of the Workshop on Visually Grounded Interaction and Language (VIGIL.. NeurIPS.
Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ka Wong, Praveen Paritosh, and Lora Aroyo. 2021. Cross-replication reliability empirical approach to interpreting inter-rater reliability. In Proceedings of the 59th Annual Meeting of the Association for

---

Computational Linguisties and the 11th International Joins Conference on Natural Language Processing (Volume I: Long Papers), pages 7053-7065, Online. Association for Computational Linguisties.
Nahathai Wongpakaran, Tinakon Wongpakaran, Danny Wedding, and Kilem Gwet. 2013. A comparison of cohen's & kappa and gwet's ac when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13.
Tiezheng Yu, Wenliang Dai, Zihan Liu, and Pascale Fung, 2021. Vision guided generative pre-trained language models for multimodal abstractive summa- rization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3995-4007, Online and Punta Cana, Domini- Republic. Association for Computational Lin- guistics.
Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, and Mohit Bansal. 2021. EmailSum: Abstractive email thread summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Confer- ence Natural Language Processing (Volume I: Long Papers), pages 6895-6909, 6909, Online. Association for Computational Linguistics.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. International Conference on Learning Representations.
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Ji- ajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal summarization with multimodal output. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4154-4164, Brussels, Belgium. Association for Com- putational Linguistics.

---

A Subreddits Used
BART BART+ImgCap
CMS-BARTHimgCap
52
We list all subreddits used for data collection, along with the number of threads from each present in the final dataset in Table 8.
48
46
44
Subreddit
Category
# Threads
42
r/outfits
Clothes
161
rffashionadvice
Clothes
529
40
<
10
[10,15)
(15,20)
[20,25)
2 25
r'plussizefashion
Clothes
19
Number comments
r/handbags
Clothes
90
Figure The influence the number of comments the thread summarization performance (ROUGE-I) on BART-based models measured on the test set.
s/ptitefashionnadiice
Clothes
112
r/weddingdress
Clothes
108
r/designmyroom
Interior
1098
r/malelivingspace
Interior 642
T5
T5+ImgCap CMS-T5+ImgCap
r/femalelivngspaac
Interior
258
54
52
Table 8: The subreddits used for data collection.
50
48
B Annotation Interface
We listed a total of 3 tasks on Amazon Mechan- ical Turk for our data pipeline. We informed all annotators that this data would be used to help in summarizing Reddit threads, asked them to agree with the Reddit Terms of Use before participating, and notified them that participating in the HIT con- stituted acceptance of these terms of use.
40
<10
[10.15)
[15,20)
(20,25)
Number of comments
Figure 4: The influence of the number of comments in the thread on summarization performance (ROUGE-1) of T5-based models. The results based the test set.
We provided annotators with detailed instruc- tions for the task and several acceptable and umac- ceptable examples to help them perform the task. In Figure 6, we show the instructions provided for Task I: similar instructions were used in the other two tasks. Additionally, we show the annotation interface used for Tasks 2 and in Figure and Figure 8.
This indicates that summarization in- deed becomes more challenging when the input length is longer. However, the performance gap between the baseline models (i.e., BART, BART- ImgCap) and the CMS-BARTEE BART-ImgCap generally increases as the number of comments grows, sup- porting the idea that CMS better handles longer threads. As our model generates cluster summaries in stage reduces the average input length by 82.8%, and thus achieving better performance even on relatively challenging long inputs. We also pro- vide results from T5-based models in Figure 4, showing similar trends; the gap between the base- line models and the CMS-T5-ImgCap is large when the number of comments falls within the range of (15,20) and [20,25).
decreases.
Additional Sample Data
We show few additional data points from the MREDDITSUM dataset in Table 9 and Table 10. Table shows a datapoint from the fashion cate- gory, whereas Table 10 shows a datapoint from the interior design category.
D Further Analyses
D.1 Summarization based on the Length of Input Threads
D.2 Summarization per Subreddit
We further explore the summarization across 9 dif- ferent subreddits, shown in Figure 5.
To better understand whether CMS effectively han- dles long inputs, we run a further analysis using BART-based models (see Figure 3). As the number of comments increases, the R1 score consistently
The results reveal that subreddits within the 'In- terior' category (i.c., the left three subreddits Figure 5) exhibit lower ROUGE scores in compar-

---

Rouge1 Rouge2
RougeL.
ROUGE
20
designmyrcom (1082.6) 6)
fernalaivingppace (938.3)
malelvingspace (678.5)
fashionadvica (298.2)
outfis (247 8)
handbage (5436)
palioffshionadvico (392 7)
pliasizefashion (335 6)
weddingdress (726 3)
Figure 5: ROUGE scores obtained from our CMMS-TS-ImCCAS model on the test set, categorized by different subreddits. The number of input words is indicated in parentheses.
ison to subreddits within the 'Clothes' category (i.e., the right six subreddits in Figure 5). This dis- crepancy be attributed to the difference in the input lengths across each subreddit. Given that the average input length of examples from the 'Inte- rior' category exceeds that of examples from the "Clothes' category, it is more difficult for our model to summarize the former. Additionally, we can also explain this gap by comparing the difference between domains. Specifically, while the model can easily comprehend clothing images by focus- ing only salient objects, comprehending inte- rior images more challenging necessitates a broader range of information (e.g., wall color, spatial relationship between furniture, etc). Conse- quently, summarizing examples from the 'Interior' category proves to be more challenging for the mod- els than summarizing examples from the 'Clothes' category.
INSTRUCTIONS
Summarizing Images:
You are to summarize what kind of advice or opinions a Reddit post. for, along summary should be single sentence that contains all necessary information from able to understand the summary completely even without seeing the image.
related given image. Ideally,
post image.
your sentence with OR....
third-person "you") tense. Summarize your original post.
rather papy-iggting
Iv. Include relevant information and details from the image.
v. Do not include personal opinions, or inrelevnnt details.
Figure An example of instructions given for Task I: Original Post Summarization.

---

Post:
OP: "what would you da to cover the ugly breaker box in my bedroom? abviously still needs to be accessible, but keep my door closed a lot and it's such an eyesore. the wail 58 iiches wide and 10 feet nigh.
Comments:
Group
@writc summary
may write multipic
sentences if necessary for this group.
User 4: you could hang a large fromed painting. something liked stretched canvas over frame. no glass, that way to can move and not werry ebout gless breaking. some will even install hinge on opens
Enter your summary for group
one
cabinet door.
User 6: think painting or frame would be nice- you can hinge it you want to give easier access. feel like a gallery style wall with multiple pieces
CNo summary sentence necessary these comments redundant or irrelevant
your
look off
center and weirdly close the
that makes
OP: makes total sense. the closeness to the edge of the wall
really what's throwing me off about it.
User hang something it. painting some kind of rug.
paint same frames the you could easlly
hang
Figure 7: An example of the Cluster Summarization task presented to workers on Amazon Mechanical Turk.
Post
OP: my room about year while fighh instructing, what can do to improve without being too permanent?
Original Summary
Edited Summary
The OP wanted temporary solutions to improve their sperse bedroom with :ust dressar plant. dressing room
The OP wanted tamporary solutions to Improve their
commented
room Another One commented that the room looks like dressing roam for a rub-n-tug, One user recommended getting rug.
sparse bedroom with just bed, dresser and fake plant.
tug. One user recommended getting user suggested curtains which easy put and add lot character and soften the place floor lamp, Another directly under the wincow and adding a small nightstand to the room. The OP sald recommendation great
bed
not
Another user suggested curtains which easy to put and add
planning. Another suggested adding life-size statue to make the place less lonely.
place
Another user suggested moving the bed so not directly under wincow and adding small nightstand far
room. The OP said the recormeendtiio
Figure 8: An example of the Summary Editing task presented to workers Amazon Mechanical Turk.

---

Image:
Image:
of fresh white flowers Post Caption:
Just moved into my first home and this space bothers me. Need some advice make look cohesive.
Comments:
i'd place of those vintage wooden radio clocks in the middle. that's oddly specific know..
User I: center feels empty. if
Post Caption:
what could you pair these with?
User Change nothing in the center
Comments:
User Dressy black pants, colorful blouse. and blazer....
User 3: Center large art piece and move down. Lean the small art off center behind Candles, use basket as tinket tray on console.
User 2: You can pair this with shorts, slacks, or jeans- basically anything. Just make sure that the color of your top bottom matches.
User 4: really like art. agree that the eneter needs something. maybe plant or a stack of books.
User 3: If you are looking for women's 's wear would say very wide leg pustel high waisted punt with a tight/fitted top same color scheme or white.
User 5: rather like The only thing missing is something sort of tallish in the center fill that space. Like, it the perfect spot for vuse filled with flowers. Some color and floral subscription isn't your future maybe full plant would fit the bill
User This reminds of the kind of shoes in anime with sailor style uniforms thh
User A Goodwill donation
Summary:
Summary:
The OP wanted to know what to wear with a pair of white leather loafers have a thick black sole and low heel. One commenter thought pastel pants and white top match the shoes would work. Another commenter said that OP's shoes would pair with any sort of bottoms, hut cautioned that the top and bottom color should match. One user shured links for use us inspiration. Another user thought that the shoes looked like anime sailor shoes. Two commenters didn't like OP's shoes, and suggested they be thrown away donated.
The OP asked what to do with space in their home that presently has light blue walls and brown sideboard with lamp and candlesticks on it. Most commenters agreed the space looked good as-is, but recommended just adding some- thing in the empty center the table, such a vase of white flowers large plant. Others thought vintage wooden radio clock traditional record player the sume green color us the candles would look perfect, while another suggested stack nice books. Others said to center the large art, and check local thrift
for substantial but short piece be the center accent decor. Others recommended using basket as a trinket tray or just huying marhle tray for trinkets on the tuble. They also suid to lean the smallest art pieces against the wall behind candles, or get rid the candles altogether.
Table 9; Another example from our dataset, from the vffasiioadvvie subreddit.
Table 10: Another example from our dataset, from the r/desigmmyroom subreddit.


Reviews:
[{"review_id": "yyBKmUc1gz", "reviewer": "Reviewer_anng", "summary": "", "questions": "A. Figure 2 shows the majority vote results among three human annotators, as explained in Section 5.2. Please explain the significance of the tie.\n\nB. Figure 2's results show very high fluency for the proposed model compared to the baseline, even though the language model remains the same (T5). However, this result does not correlate with Table 6. On the other hand, the faithfulness score is similar even when the proposed model prunes comments using clustering.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer took the time to understand the paper well, and correctly identify our main contributions of introducing a novel multimodal dataset and a new summarization method that have the potential to contribute to the field of multimodal abstractive summarization. \n\n**\"Lack of detail on the total number of annotators and whether the same annotator generated the comment and full summary for the study.\"**\n\nSummaries were written by a pool of 40 annotators who passed our qualification tests. Since each summarization stage was an independent task on AMT and annotators were randomly assigned a thread to summarize, the annotators differed from one stage to another for a given thread. We felt that as all annotators passed strict qualification tests, this did not lead to any degeneration in annotation quality. We will clarify this in the paper.\n\n**\"…The paper does not provide an inter-annotator agreement for a few samples to understand the variance in quality of the summaries.\"**\n\nUnlike that of categorical labels, the quality of written summaries is difficult to quantify with inter-annotator agreement scores, as the notion of “agreement” is not well defined. Thus, rather than measuring the summary qualities with such a metric, we carefully controlled the quality of annotators through a series of filtering procedures. As described in Sec. 3.2, all annotators came from English-speaking countries, had a HIT approval rate over 98%, more than 5000 HITs approved, and passed a strict qualification test where each result was manually checked for quality.\nHowever, we agree that measuring overall summary quality can be beneficial. \nFollowing a closely related work, SamSUM (Gliwa et al., 2019), we randomly selected 100 thread-summary pairs and had 2 independent judges grade them on a quality scale of -1, 0, or 1, where -1 means a summary is irrelevant or does not make sense, 0 means a summary extracts only part of the relevant information or makes some mistakes, and 1 means a summary that is understandable and delivers a brief overview of the relevant information from the thread. We asked annotators to score summaries on overall quality, fluency, and faithfulness, similar to our human annotation found in Sec. 5.2. We found the scores were highly positive, with average scores of 0.95, 0.96, 0.83 for overall quality, fluency, and faithfulness respectively. Additionally, we found Gwet's AC1 agreement scores of .91, 0.89, and 0.53, corresponding to high, high, and moderate agreement, respectively. Note that we used the Gwet's AC1 score for interannotator agreement as it performs well despite class imbalances where agreement is high. (Gwet 2008, Wongpakaran et al. 2013, Wong et al. 2021).\n\n\n**\"Comparison with a text-only baseline model that supports longer sequence lengths, such as LongFormer/LongT5, is missing.\"**\n\nWe agree that the addition of one of these models could provide an interesting comparison, and strengthen the claim that the proposed model is better able to handle longer inputs. As LongFormer is a BERT-like model incapable of text generation alone, we chose to experiment with the LongT5, which also provides a better comparison with our best model. We thus pretrained a LongT5-base model on CNN/DailyMail, as described in Sec. 4.3, and fine-tuned both a text-only and an image-caption based extension. We found the following results:\n\n|                      | R-1   | R-2   | R-3   | BERTScore |\n|----------------------|-------|-------|-------|-----------|\n| LongT5               | 45.98 | 19.44 | 43.12 | 41.95     |\n| LongT5-ImgCap        | 46.6  | **19.86** | 43.7  | 42.63     |\n| CMS-T5-ImgCap (ours) | **47.29** | **19.86** | **44.13** | **44.74**     |\n\nThese results show that while LongT5 outperforms regular T5 in all metrics except BERTScore, our CMS method still outperforms LongT5 despite LongT5’s additional attention mechanism and longer input handling.\n\n**\"Please explain the significance of the tie in Figure 2.\"**\n\nFor the human evaluation experiments, annotators were also given the option to choose a 'Tie' between the two summaries presented to them. We allowed this to create a more fair comparison between model results, without forcing annotators to choose a better model in case they did not notice a significant difference between the summaries. While we focused on the percentage of Wins/Losses for analysis in the paper, the low percentage of ties for the Overall and Fluency scores can be interpreted as a stronger preference, while the higher percentage of ties for Faithfulness show a lower preference between summaries, which is consistent with the percentage of Wins/Losses.\n\n**\"Figure 2 shows high fluency and similar faithfulness for proposed model compared to baseline, despite using the same language model and pruned comments. This does not correlate with Table 6.\"**\n\nWe believe this to be an expected result, and argue that it does in fact correlate with the example summaries presented in Table 6. Although we understand fluency may have differing definitions, we defined fluency as how naturally written the summary is, described in Sec 5.2. We believe that as the proposed model more naturally captured the varying viewpoints of the commenters, without including irrelevant comments, annotators likely felt these were more naturally written. This can be seen in the second example of Table 6, where the proposed model captures more relevant details on the accent chair suggestions without discussing the irrelevant non-shedding dog. Even though the language generation ability of the model was the same and the readability of each summary was similar, the improved content of the summary likely impacted its fluency score as annotators may feel this is more ‘natural’. \nAdditionally, faithfulness was defined as how truthful the summary is to the document. The faithfulness score remains similar between models as both methods utilize the information from the image caption as well as the comments, and thus both have a similar hallucination rate and hence a similar faithfulness score; this can be seen in Table 6 where both the proposed method and baseline ImgCap method either had a hallucination as in the first example, or no hallucination in the 2nd example. We may see lower faithfulness scores with models that did not have image information available at all.\n\nThank you again for the kind feedback - we appreciate the detailed comments and hope we have addressed most concerns!\n\n**References**\n\n[1] Gliwa et al., \"SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization\", ACL workshop 2019.\n\n[2] Gwet, \"Computing inter-rater reliability and its variance in the presence of high agreement\", British Journal of Mathematical and Statistical Psychology 2008.\n\n[3] Wongpakaran et al., \"A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples\", BMC Medical Research Methodology 2013.\n\n[4] Wong et al., \"Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability\", ACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new multimodal abstractive summarization dataset based on Reddit discussions, consisting of 3033 threads and human-written summaries.\n\nThe paper proposes a new method for multimodal summarization on this new dataset, which outperforms the existing baselines.\n\nThe proposed dataset is novel and has the potential to contribute to the field of multimodal abstractive summarization.", "reasons_to_reject": "Lack of detail on the total number of annotators and whether the same annotator generated the comment and full summary for the study. Additionally, the paper does not provide an inter-annotator agreement for a few samples to understand the variance in the quality of the summaries across the dataset.\n\nComparison with a text-only baseline model that supports longer sequence lengths, such as LongFormer/LongT5, is missing. The input sequence length of the document (post+full thread) exceeds 512 tokens. It will also strengthen the claim of the proposed model's ability to handle longer input lengths as discussed in Section 5.", "questions_to_authors": "A. Figure 2 shows the majority vote results among three human annotators, as explained in Section 5.2. Please explain the significance of the tie.\n\nB. Figure 2's results show very high fluency for the proposed model compared to the baseline, even though the language model remains the same (T5). However, this result does not correlate with Table 6. On the other hand, the faithfulness score is similar even when the proposed model prunes comments using clustering.", "paper_topic_and_main_contributions": "The paper presents a new multimodal abstractive summarization dataset (MRedditSum), which contains 3,033 Reddit threads (post+comments) with text (posts and comments) and images (posts). For each thread, a human-written summary was collected first for the post (text and image) and then for the top-5 comment clusters. The post summary and comments summaries were later merged by a human annotator to generate a full summary. Additionally, the paper proposes a new cluster-based multi-stage summarization method that outperforms baselines (text-only, text with the caption, and text with image) on this new dataset. The experiments also show that the performance of current summarization models improves by including visual information.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is, in general, readable"}, "extra_text": {}}, {"review_id": "gGefpfldhU", "reviewer": "Reviewer_2UFg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer noted the value of the dataset to the community and summarized the key contributions well. We would like to address the reviewer’s concerns below.\n\n**\"Looking at the length of the posts, it is not clear if this is really a summarization task.\"**\n\nWe would first like to note that our summarization results in this work focus on the Full Thread summary, which as described in Sec 3.3., is on average 13.2% as long as the input (comparable to SamSUM’s 19% and How2’s 11.3%). We believe this reduction in word length is a clear indication of a summarization task.\nWe believe that the review may be referring to the Original Post summary described in Sec 3.2. and Table 3, where on average the document is 18.87 words while the summary is 23.14 words. Although the summary is on average longer for this structure, the key thing to note is that this is a multimodal summary - that is, we are also summarizing an image along with the text. When the text is short and the image is crucial for understanding - as in the Original Post section of a thread - it is not unexpected that describing the image would add a small amount of length to the summary. We believe multimodal summarization may not strictly adhere to the idea that the summary must be fewer words than the text component of the document, as it is condensing image information into the summary as well.\n\n**\"The proposed approach’s ability to capture multimodal information is largely anecdotal without any extensive evidence of the same.\"**\n\nAs the main focus of our CMS summarization technique was to better capture multiple, relevant viewpoints from the thread, we highlighted the improvements in the overall ROUGE and BERTScore scores in our paper’s analysis. As can be seen in Table 5 and is discussed in Sec. 5, models that contained image information through an image caption showed improvements across all scores, across all model types. As the only difference in the model input across these models was the multimodal information captured by the image caption, we believe the increase in scores can only be attributed to a better capture of the necessary multimodal information via the automatically generated image caption.\nHowever, we agree that a more thorough comparison focusing solely on image information could be useful. To address this, we additionally computed the average CLIPScore between the image and the first sentence of the generated summaries, which corresponds to the post summary and is where most of the image information is found. The scores are as following:\n\n| Model           | CLIPScore |\n|-----------------|-----------|\n| T5              | 69.34     |\n| CMS-T5-Imgcap   | **70.58**     |\n| BART            | 68.29     |\n| CMS-BART-Imgcap | **70.26**     |\n\n\nThese scores provide further evidence that our proposed method significantly increases the amount of multimodal information provided in the summary through the use of an automatically generated image caption across both T5 and BART based models, as expected. We will include this additional analysis in the paper.\n\nWe thank you again for the kind review, and hope that our feedback provided some clarity on the reviewer’s concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A dataset of this nature will be valuable to the community.", "reasons_to_reject": "1. Looking at the length of the posts, it is not clear if this is really a summarization task. \n2. The proposed approach's ability to capture multimodal information is largely anecdotal without any extensive evaluation of the same.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a multimodal summarization dataset based on Reddit threads along with the comments. The authors have annotated summaries along with a pipeline bench mark approach. Evaluations indicate that the proposed approach is competetive to summarize threads and also incorporates details from the images.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a7Spi4ugdf", "reviewer": "Reviewer_BJQs", "summary": "", "questions": "Is not possible to compare the models across other datasets that are listed in Table 2? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer found the contributions easy to understand and well-described, and appreciate that the reviewer clearly understood the main points of the paper and wrote a thorough summary. We would like to respond to the reviewer’s main concern below.\n\n**\"The only criticism is the lack of comparability of CMS on other datasets…is it not possible to compare it across other datasets listed in Table 2?\"**\n\nOur main research question of this work is the summarization of multimodal posts with a long thread of discussion containing multiple, relevant viewpoints from different users. Thus, the input document must be naturally split into clusters, or groups of relevant comments/topics, and include summaries of these clusters. Unfortunately there are few summarization datasets that meet this criteria - the closest one within Table 2 is AnswerSumm, which does include clustered groups of comments, cluster summaries, and a final summary. Given that AnswerSumm is text-only, we originally felt that CMS may not be appropriate for AnswerSumm due to its lack of multimodal information, which is a key part of our experiments. Additionally, AnswerSumm has a far shorter number of comments (average 6.4 vs mRedditSum’s 22.6, as shown in Table 2), potentially weakening the benefit that CMS provides from filtering out long and irrelevant comments. Finally, though AnswerSumm has cluster and cluster summaries, these are human-selected and do not use the agglomerative clustering algorithm that we did for mRedditSum and CMS, which means the final summarization pipeline cannot be fully automated.\nDespite these limitations, we decided to attempt CMS on this dataset as well, using the human-annotated cluster labels provided by AnswerSumm. We found that this technique significantly improved scores, beating the baseline by a total of 6.75 Rouge-2 points for T5 based models. It should be noted that as this method uses the human-annotated clusters during test time, it is not fully comparable to the baseline results that do not use this information. However, we believe it shows the effectiveness of CMS when clustering information is available.\n\n|                    | R-1   | R-2   | R-3   | BERTScore |\n|--------------------|-------|-------|-------|-----------|\n| T5                 | 26.2  | 9.38  | 23.27 | 25.28     |\n| CMS-T5-GoldCluster | 33.98 | 16.13 | 31.0  | 35.24     |\n\nThank you again for the kind review - we appreciate the reviewer’s suggestion, and hope that our response brought some further clarity to this topic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset and the CMS model are the main contributions of the papers. Both are well-described and written with clear details.", "reasons_to_reject": "The only criticism is the lack of comparability of CMS on other datasets. Showing its performance on a single dataset (the proposed dataset) is somewhat limited. ", "questions_to_authors": "Is not possible to compare the models across other datasets that are listed in Table 2? ", "paper_topic_and_main_contributions": "The paper presents a new dataset for multimodal discussion summarization that is curated from Reddit. The task is to create an abstractive summary that takes into account the original post text, attached image and comments from other users. The dataset creation was done by employing annotators. The next contribution of the paper is the CMS summarization architecture that works by clustering the content in multiple steps and resulting in a single summary. The new dataset has been evaluated on 3 other text models including the proposed CMS model. Since the task is multimodal, 2 different model architectures have been chosen: image captioning output and the task text, fine-tuning textual and visual encoders (VG-BART or VG-T5) on the task. Based on the results presented in Table 5, CMS-T5-ImgCap the model obtains the highest score among compared variants of other models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents mRedditSum, a new multimodal abstractive summarization dataset, and a novel cluster-based multi-stage summarization method. The dataset consists of 3,033 Reddit threads where a post solicits advice regarding an issue described with an image and text, and respective comments express diverse opinions. These threads are annotated with a human-written summary that captures both the text's essential information and the image's details. The paper also shows that the performance of summarization models, specifically the proposed CMS model, improves when visual information is incorporated.

Reviewers acknowledge the novelty and potential contribution of the mRedditSum dataset and the proposed cluster-based multi-stage summarization method to the field of multimodal abstractive summarization. The paper is well-written, with clear details about the proposed dataset and method. However, there are also concerns that may merit attention. These include a lack of detail on the total number of annotators and whether the same annotator generated the comment and full summary for the study. It is also pointed out the absence of an inter-annotator agreement for a few samples and a comparison with a text-only baseline model that supports longer sequence lengths.

Overall, this paper proposes a potentially useful dataset for the research community, as well as a summarization model that surpasses SOTA. Authors are encouraged to address the reviewers' comments, in particular the concerns on the human annotation process, and their rebuttals into the final version if accepted.
Decision: Accept-Main