Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization
Authors: Ridwan Mahbub
Ifrad Towhid Khan
Samiha Shafiq Anuva
Md Shihab Shahriar
Md Tahmid Rahman Laskar
Sabbir Ahmed
Venue: EMNLP 2023 Main (2023)
Keywords: Poem summarization
Creative language summarization
Creative language interpretation
Natural language understanding
Automatic text summarization
Language models

Abstract: While research in natural language processing has progressed significantly in creative language generation, the question of whether language models can interpret the intended meaning of creative language largely remains unanswered. Poetry as a creative art form has existed for generations, and summarization of such content requires deciphering the figurative patterns to find out the actual intent and message of the poet. This task can provide the researchers an opportunity to evaluate the creative language interpretation capacity of the language models. Unlike typical text, summarization of poems is a challenging task as poems carry a deeper meaning, which can be easily lost if only the literal meaning is considered. That being said, we propose a new task in the field of natural language understanding called 'Poem Summarization'. As a starting, we propose the first-ever dataset for this task, named 'PoemSum', consisting of 3011 samples of poetry and its corresponding summarized interpretation in the English language. We have benchmarked the performance of different state-of-the-art summarization models and provided observations on their limitations. The dataset and all relevant code used in this work have been made publicly available.

Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization
Ridwan Mahbub', Ifrad Towhid Khan', Samiha Shafiq Anuva', Md Shihab Shahriar', Md. Tahmid Rahman Laskar and Sabbir Ahmed Islamic University of Technology, York University, *Dialpad Canada Inc. [r (ridwanmahbub, ifradtowhid, sami samihashafiq, shihabshahr iartttrrrrrrrrreeeeeeer edu tahmid20eyorku. ca, wwwarraaaeeettiiie edu
Abstract
make poetry more accessible wider range of audiences and applications (Upadhyay al., 2022).
While research in natural language process- ing has progressed significantly in creative lan- guage generation, the question of whether lan- guage models can interpret the intended mean- ing of creative language largely remains unan- swered. Poetry creative art form has ex- isted for generations, and summarization of such content requires deciphering the figura- tive patterns to find out the actual intent and message the poet. This task can provide researchers opportunity to evaluate the cre- ative language interpretation capacity of the language models. Unlike typical text, sum- marization of poems is challenging task as poems carry a deeper meaning. which can be easily lost if only the literal meaning is consid- ered. That being said, we propose a new task in the field natural language understanding called 'Poem Summarization'. As a starting, we propose the first-ever dataset for this task, named 'PoemSum', consisting of 301 samples of poetry and its corresponding summarized in- terpretation in the English language. We have benchmarked the performance of different state- of-the-art summarization models and provided observations on their limitations. The dataset and all relevant code used in this work have been made publicly available
In recent years, there has been notable research conducted on text summarization in the field of Natural Language Processing (NLP) (Altmami and Menai, 2022; Widyassari et al., 2022; El-Kassas et al., 2021; Laskar et al., 2022). However, to the best of our knowledge, such work has been done in the domain of poem summarization yet. While the summarization process of poems seems quite similar the generic text summarization, there are some major differences between the two. Text summarization, in general, is a technique that re- duces the number of sentences and words in a doc- ument by extracting its important information into concise paragraph without changing its meaning (Erkan and Radev, 2004). Most of the summariza- tion models being proposed in recent years have been trained on news-related datasets (Varab and Schluter, 2021; Hasan et al., 2021; Grusky et al., 2018; Laskar et al., 2020a,b), which mostly contain facts, so they do not have to look for deeper mean- ing while generating summary. The same goes for datasets that are built using financial (Russell- Rose et al., 2002) or medical data (Gupta et al., 2021), I). This process only considers the literal meaning of words, and it does not focus on the deeper mean- ings. Applying this summarization technique di- rectly to poems may not be helpful in capturing the essence. Hence, is necessary to study if bascline NLP models (Raffel et al., 2020) are sufficient for poem summarization tasks.
1 Introduction
Poetry has been used for centuries powerful means of expressing the feelings, thoughts, ideas, and emotions of human beings (Freeman, 2000). It holds a profound significance in human culture and expression, transcending time and boundaries, Poem summarization refers to the process reduc- ing a poem to its core ideas to provide a concise and informative overview of the content (Wolosky, 2008). This not only gives us an idea about the main theme but also describes the style, rhyme, techniques, and metaphors used in the poem. Auto- matic summarization of such creative language can
Summarizing literary work poses lots of chal- lenges. For instance, the artistic style of each writer is different, and the literature can be interpreted in many ways. Moreover, understand the figurative language of poems, the model has to understand the interconnection between sentences and the deeper meaning conveyed by the poem as a whole. In or- der to facilitate research in this domain, we have introduced a new task in the field of natural lan-
chttps://gitther wwwp///wwww//////e//eooomm

---

guage understanding named 'Poem summarization' and proposed novel poem summarization dataset. The main contributions of our paper are as follows:
Type
Size
Number of Poems
3011
Number of Poets
930
Introducing a novel poem summarization dataset named 'Poemsum'.
Max poem length"
6830
Max summary length"
1104
Evaluating how the existing summarization models perform this task and identified their limitations to facilitate future research.
Avg. poem length"
209
Avg, summary length'
141
Avg. no. poems per poct
3.24
2
Related Works
*Unit represented in number of words
The text summarization task can be broadly divided into two categories: Extractive (Rau et al., 1989) and Abstractive (Chopra et al., 2016) methods. The extractive summarization approach highlights the key points in the source text, which are then taken out and compiled into succinct summary. On the other hand, the abstractive approach uses differ- ent collection of words compared the original text to construct a paraphrase of the main ideas of a given text. For poem summarization, abstractive summary generation is required as only direct ex- traction of words from poems would not be enough to convey the actual meaning of the poem.
Table Statistics of the PoemSum Dataset
ied the ability of language models to complete nar- ratives containing idioms and similes. In other related research (Stowe et al., 2021), metaphors are generated based on conceptual metaphor theory (Kövecses, 2016).
While creative language generation has seen the limelight in the last few years (Lewis al., 2021; Van de Cruys, 2020; Watanabe and Goto, 2020; Oliveira, 2017), creative language summarization has been mostly left behind. Recent endeavors in the field include 'CREATIVESUMN' (Agarwal et al., 2022): shared task summarization of creative documents like movie scripts, television scripts, etc. However, poems were not used in any of the sub-tasks. Other types of tasks that work with poems include poem classification (Kumar and Minz, 2014), image-to-poent generation (Xu et al., 2018), and developing better Large Language Models for poem generation through human collab- oration (Chakrabarty al., 2022b). However, work focusing on the summarization of poems is missing in prior literature. To this end, we introduce novel poem summarization dataset.
In the abstractive summarization domain, Narayan et al. (2018) introduced dataset called 'X-Sum' by gathering 226,711 online articles from the British Broadcasting Corporation (BBC). In the same year, Grusky et al. (2018) proposed a summa- rization dataset named 'NEWSROOM' containing 1.3 million articles and summaries authored by writ- ers and editors in the newsrooms of 38 major news publications that combine both abstractive and ex- tractive summarization methods. The 'XL-Sum' dataset proposed by Hasan et al. (2021) covers 44 languages, and it the largest abstractive summa- rization dataset in terms of the number languages represented and the number of samples gathered from a single source. Although summarization has made great strides in recent years (Kumar et al.. 2021), most works have used news articles data source (Fabbri al., 2019), where the focus is mostly on extracting key information, forming shorter passage. Poems, on the one hand, do not benefit from such types of summarization, yet there are no studies aimed at exploring the prospects of poem summarization.
3 The PoemSum Dataset
In order to expedite research in the poem sum- marization task, we have curated a novel dataset named 'PoemSum'. The dataset provides a diverse range of poems along with their summaries. Each sample contains the title, poet name, the poem, the source website of the poem, and finally, its cor- responding summary. Appendix A. shows some sample poems of the dataset along with their corre- sponding summaries. We have provided different statistics of the PoemSum dataset in Table 1.
Apart from poetry. there has been a recent insur- gence of datasets in other types of creative language (Mckeown, 2022; Chakrabarty et al., b)23a,b) with the objective of Natural Language Interpretation (NLI). For instance, Chakrabarty et al. (2022a) stud-

---

3.1 Data Collection
Model
R1
R2
RL
BS
To construct the PoemSum dataset, collected the poems and their corresponding summaries from dif- ferent sources. While the summaries were collected from the 'Poem Analysis' website?, which has the largest collection of high-quality poem summaries, in most cases, this website hosted only the sum- mary but not the poem itself. Therefore, we were required to collect the poem from other sources on the internet For this reason, we collected the summaries via web scrapping, while the cor- responding poems were collected manually from numerous websites.
TS
45.03
25,96
33.87
85.98
Pegasus
42.81
22.16
30.36
85.18
Bart
41.18
21.72
29.66
84.76
mT5
35.01
14.25
21.91
83.34
Table 2: Performance of different Models
The summaries which did not meet these spec- ified standards were discarded. Initially, around 3,500 samples were collected, but after an exten- sive data cleaning and data verification process, we ended up with 3,011 samples.
To ensure the reliability and accuracy of the man- ual data collection process, four undergraduate stu- dents majoring in Computer Science and Engineer- ing from reputable university were hired. While selecting these individuals, it was ensured that they possessed the necessary technical skills to collect the required data (i.e., collecting the required po- ems from different websites).
Models
We evaluated the performance of some of the stan- dard summarization models like T5, BART, Pega- sus, and mT5 PoemSum, as they have demon- strated impressive performance in various summa- rization datasets in recent years (Laskar al., 2022: Ravaut et al., 2022).
3.2 Dataset Cleaning
model (Vaswani et al., 2017) that uses a unified text-to- text transfer learning framework that can be used for diverse tasks like translation, summarization, question answering, etc. (Raffel et al., 2020).
T5: A versatile transformer-
Since the data was collected from various sources, further cleaning was required to ensure uniformity of the data. After collecting the raw data, we han- dled discrepancies like duplicate records, garbage values, unnecessary white spaces, corrupt values, HTML tags, and other inconsistencies. The cleaned records were further manually verified ensure the aforementioned inconsistencies did not persist.
Pegasus: transformer-based model specifi- cally designed for text generation tasks like sum- marization, employing pre-training with extractive supervision (Zhang et al., 2020).
BART: A sequence-to-sequence model based on the transformer architecture pre-trained on docu- ment denoising (Lewis et al., 2019).
3.3 Data Verification
To verify the quality of the collected summaries, we set some standards that every sample had to maintain. The conditions are as follows:
m TS: It is extension of T5 designed for mul- tilingual applications. The variant we have opted use is trained for multilingual news summarization in the XL-Sum Dataset (Hasan et al., 2021).
The summary cannot be limited to explaining only the rhyming scheme of the poem.
The pre-trained models were fine-tuned with a learning rate of 0.0001, and AdamW optimizer (Loshchilov and Hutter, 2017) with batch size of 10. The dataset was splitted into the train, valida- tion, and test sets following the ratio of 80:10:10.
The summary cannot contain only the poet's biography without any insights into the poem. The summary cannot only discuss other works of the poet without context.
The summary must not only copy lines from the poem without any attempts to extract deeper meaning.
Results and Discussions
To evaluate the model-generatee summarics against the gold reference summaries, we have used the Rouge-1 (R1), Rouge-2 (R2), Rouge-L (RL), and BertScore (BS) metrics. The results are shown in Table 2. A brief description of the metrics we have used in our experiments is given below:
The summary must not be limited to rewrit- ing the lines of the poem in simple language without any analysis.
"https:/poemmnnlysis..o sommoss.
List of sources available in our github repository

---

Poem
Actual Summary
Generated Summary TS
The Death of the Ball Turret Gunner
"The Death of the Ball Turret Gunner" by Randall Jarrell is moving poem about the death of soldier during the Second World War. Throughout the lines of this piece, Jar- rell's speaker, deceased soldier, deseribes his hirth and innocence while juxtaposing it with his job as gunner and death. The first lines deseribe how he was born from his mother's dream into "the State" or a new, more violent understanding life. There, he hunkers down inside a new womb, the ball turret. His days are filled with black flak and nightmarish scenes life and death. The poem's 's final line reveals that the speaker is dead, and his remains were washed from the "womh" with a hose.
The Death of the Ball Tur- ret Gunner' by Randall Jarrell describes the death of a ball- turret gunner who was shot head. The poem begins with the speaker stating that fell into the state of Alabama and hunched in its belly until his wet für froze, It was six miles away from earth, freed from its dream of life. When he died, it was washed him out of the tur- ret with hose.
-Randall Jarrell
From my mother's sleep
fell into the State,
And hunched its belly
till my wet fur froze.
Six miles from earth,
loosed from its dream of life,
I woke to black flak and the nightmare fighters. When died they washed me out of the turret with hose.
Table 3: Sample of the Generated Summary by the T5 model. The highlighted parts show that some text is almost directly carried over from the poem, which does not help in expressing the figurative meaning of the sentences.
Rouge (Lin, 2004): It measures the overlap between actual and generated summary consid- cring unigrams, bigrams, and longest common subsequence in the case of Rougel, Rouge2, and RougeL,, respectively.
cases is limited to single words like pronouns and prepositions match the tone of the rest of the generated summary. Additionally, some unrelated words are spawned, like 'State of Alabama' instead of just 'State'. All these reveal that creative lan- guage summarization has a long way to go before it can be on par with general text summarization. Apart from lines being carried over from the po- ems to the summary without plausible explanation, there are number of other figurative language pat- terns like metaphors, symbolism, personification. etc., that language models unable to compre- hend (Syafitri and Marlinton, 2018; Abrams, 1999). Appendix A.2 contains a detailed insight into some of these lackings in the generated summaries.
BERTScore (Zhang et al., 2019); Instead of considering exact matches, Bertscore analyzes to- ken similarity using contextual embeddings (De- vlin et al., 2018).
The results show that the T5 model is the best- performing one, both in terms of the three types of Rouge scores and BERTScore, while the mT5 per- formed the worst. The key takeaway here is that the mT5 model that we use in this paper* was trained on a news summarization dataset. It performing the worst as per our initial hypothesis. News data is mostly void of creative clements, which leads to an inability to express and summarize creative language even after fine-tuning. Appendix A.2 con- tains further explanation on this issue.
Conclusion
In this work, we have introduced a novel dataset for poem summarization and assessed the creative lan- guage understanding capability of the popular sum- marization models. We have also discussed how creative language differs significantly from formal language in its structure and expression. Hence, we require different approaches when dealing with cre- ative language, at least in the domain of summariza- tion. After conducting several baseline experiments with the state-of-the-art summarization models, we observed that even the best dest-performing model was unable to produce summaries that capture the main idea of the poems in certain cases. We also used model trained on a popular news article sum- marization dataset and observed that it performed worse compared to other models we have fine- tuned. Our findings have led us to the conclusion that existing text summarization approaches and
Furthermore, we have critically analyzed the re- sults and found a few pitfalls of the evaluated mod- els. A sample summary generated by the T5 model is mentioned in Table 3. As we can see, even in our best-perforning model, the generated summary is void of the true interpretation of the poem. The actual poem summary tells us the story of dying soldier and his final thoughts during death. The generated summary fails to carry on the ideas pre- sented in the poem. We observe exact lines from the poem being carried over to the summary as highlighted in Table 3. It is observable that the only change in the highlighted sentences in most
'The mT5 model (Hasan et al., 2021) used in this work is a variant of the original mT5 model (Raffel et al., 2020)

---

summarization models are not well-suited for poem summarization. We believe that the proposed Po- emSum dataset can serve as a valuable resource for evaluating the performance of NLP models in un- derstanding, interpreting, and generating figurative language within the context of poetry. Meanwhile, the evaluation of Large Language Models (LLMs) (Touvron et al., 2023; OpenAI, 2023; Anil et al., 2023; Jahan al., 2023; Laskar al., 2023) in this task could be a good direction for future work.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. 2023. Palm 2 technical report. arXiv preprint 230.::2305.10000
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022a. It's not rocket science: Interpreting figurative language in narratives. Transactions of the Associa- tion for Computational Linguistics, 10:589-606.
Tuhin Chakrabarty, Philippe Laban. Divyansh Agar- wal. Smaranda Muresan, and Chien-Sheng Wu. 2023a. Art or artifice? large language models and the false promise of creativity. arXiv preprint arXiv:23099. 14556.
Limitations
The limitations of our work include a lower num- ber of samples compared to other datasets for text summarization in general. The number of sources from where credible poem summaries could be collected was very limited. The poems of only one language have been included in the dataset, and hence, the dataset does not contribute toward creative language interprctation in low-resource languages.
Tuhin Chakrabarty, Vishakh Padmakumar, Facze Brah- man, and Smaranda Muresan. 2023b. Creativity sup- port the age of large language models: An empiri- cal study involving emerging writers. arXiv preprint arXiv:2309.12. 12570.
Tubin Chakrabarty, Vishakh Padmakumar, and He Hc. 2022b. Help me write poem: Instruction tuning as a vehicle for collaborative poetry writing. arXiv preprint arXiv:2210. 13669.
Ethics Statement
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with at- tentive recurrent neural networks. In Proceedings of the 2016 conference of the North American chap- ter of the association for computational linguistics: human language technologies, pages 93-98.
The poems and summaries were collected from public websites. The sources of the collection have been disclosed in our GitHub repository. At the same time, the data collectors involved in the data collection and checking process were compensated monetarily for their time and effort above the mini- mum wage.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1.X.vv8810.04000
Acknowledgement
Wafaa S Fal-Kassas, Cherif Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021. Automatic text sum- marization: A comprehensive survey. Expert systems with applications, 165:11 13679.
We are grateful Islamic University of Technology (IUT) for funding the dataset curation process. We also appreciate the hard work and dedication of the data annotators and data collectors. This work would not have been possible without their support.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality salience text sum- marization, Journal of artificial intelligence research, 22:457-479.
References
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint orXivv1906.011
M. H. Abrams. 1999. A Glossary of Literary Terms. (7th ed.). New York: Comell University.
Divyansh Agarwal, Alexander R Fabbri, Simeng Han, Wojciech Kryscinski, Faisal Ladhak, Bryan Li, Kath- leen McKeown, Dragomir Radev, Tianyi Zhang, and Sam Wiseman. 2022. Creativesumm: Shared task on automatic summarization for creative writing, arXiv preprint arXiv:2211.0588
Margaret H Freeman. 2000. Poetry and the scope of metaphor: Toward a cognitive theory of literature. Topics in English Linguistics, 30:253-282.
Nouf Ibrahim Altmami and Mohamed Fll Bachir Menai. 2022. Automatic summarization of scientific articles: survey. Journal of King Saud Snivereity-Computer and Information Sciences, 1-0288.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv: 11233.

---

Vivek Gupta, Prema Bharti, Pegah Nokhiz, and Harish Karnick. 2021. Sumpubmed: Summarization dataset of pubmed scientific articles. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 1 11th International Joint Conference on Natural Language Processing: Siu- dent Research Workshop, pages 292-303.
Danielle Lewis. Andrea Zugarini, and Eduardo Alonso. 2021. Syllable neural language models for english poem generation. In 12th International Conference on Computational Creativity (ICCC'21).
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising peuuence-to-sqqeeee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 191.13961..3
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. XI-sum: Large- scale multilingual abstractive summarization for 44 languages. arXiv preprint arXiv:2106. 13822.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023. A comprehensive eval- uation of large language models on benchmark biomedical text processing tasks. arXiv preprint arXiv:2310.0227.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.0510/
Zoltán Kövecses. 2016. Conceptual metaphor theory. In The Routledge handbook of metaphor and lan- guage, pages 31 45. Routledge.
Kathleen Mckeown, editor. 2022. Proceedings of The Workshop on Automatic Summarization for Creative Writing, Association for Computational Linguistics. Gyeongju, Republic of Korea.
Vipin Kumar and Sonajharia Minz. 2014. Poem clas- sification using machine learning approach. Pro- ceedings of the Second International Conference on Soft Computing for Problem Solving (SocProS 2012), December 28-30, 2012, pages 675-688 6622. Springer.
Shashi Narayan, Shay B Cohen, and Mirella Lap- ata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural net- works for extreme summarization. arXiv preprint 18Xiv:1808. 08745.
Yogesh Kumar, Komalpreet Kaur, and Sukhpreet Kaur. 2021. Study of automatic text summarization ap- proaches in different languages. Artificial Intelli- gence Review, 54()):5897-5929.
Hugo Gonçalo Oliveira. 2017. A survey on intelligent poetry generation: Languages, features, techniques. reutilisation and evaluation. In Proceedings of the 10th international conference on natural language generation, pages 11-20.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan. Shafiq Joty. and Jimmy Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguistics: ACI. 2023, pages 431-469, Toronto, Canada. Association for Computational Lin- guistics.
R OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303 08774.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5511
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020a. Query focused abstractive summariza- tion via incorporating query relevance and transfer learning with transformer models. In Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian AI 2020, Ouawa, ON, Canada, May 13-15, 2020, Proceedings 33, pages 342-34422252 Springer.
Lisa F Rau, Paul S Jacobs, and Uri Zernik. 1989. In- formation extraction and text summarization using linguistic knowledge acquisition. Information Pro- cessing & Management, 25(4):419 428.
Mathieu Ravaut, Shafiq Joty, and Nancy F Chen. 2022. Summareranker: A multi-task mixture-of-experts re- ranking framework for abstractive summarization. arXiv preprint arXiv:2203.06500 06566.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2020b. Wsl-ds: Weakly su- pervised learning with distant supervision for query focused multi-document abstractive summarization. arXiv preprint 2011:2001
Tony Russell- Mark Stevenson, and Miles White- head. 2002. The reuters corpus volume 1-from yes- terday's 's news to tomorrow's language resources.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain adaptation with pre-trained transformers for query-focused ab- stractive text summarization. Computational Linguis- tics, 48(2))279-320.
Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, and Iryna Gurevych. 2021. Metaphor generation with conceptual mappings. arXiv preprint 1r06v:2106.01228.

---

Dewi Syafitri and Melisa Marlinton. 2018. An analysis of figurative language used in edgar allan poe's po- ems. Linguistic, English Education and Art (LEFA) Journal, 2(1):43-59
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoay Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0967 09675.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09285
A
Appendix
A.1 Dataset Sample
Some sample poems and their corresponding sum- maries have been shown from our dataset in Table Along with the poem and its summarized inter- pretation, the dataset also contains the poem title, poet name, and the web link to the source of the poem for each sample.
Aditya Upadhyay, Nidhir Bhavsar, Aakash Bhatnagar, Muskaan Singh, and Petr Motlicek. 2022. Automatic summarization for creative writing: BART based pipeline method for generating summary of movie scripts. In Proceedings of The Workshop on Auto- matic Sunmarization for Creative Writing, pages 44-50, Gyeongju, Republic of Korea. Association for Computational Linguistics.
A.2 Lackings in the Generated Summaries
We observed a number of shortcomings in the gen- erated summaries by the evaluated models. The main issue we observed here that the language models were having hard time dealing with fig- urative language patterns. Such lackings were ob- served even in the case of our best-performing model, T5. The most common pattern the mod- els seem to struggle with is metaphors, which are words or expressions whose literal usage is very different from their actual use in the poetry (Syafitri and Marlinton, 2018). The model is also seen to struggle with other patterns like personification, which means assigning of human-like qualities any objects, and symbolism, which is an object or event that references or signifies something other than itself (Abrams, 1999).
Tim Van de Cruys. 2020. Automatic poetry generation from prosaic text. Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 2471-2480, Online. Association for Computational Linguisties.
Daniel Varab and Natalie Schluter. 2021. Mas- siveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 10150-10161, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Apart from this, another issue that was limited to the mT5 model was the generation of summaries that seemed to be inspired by new article summa- rization. This is completely in line with hy- pothesis the mT5 model was pretrained on news data. One of the characteristics of these summaries includes unnecessary inclusion of facts about the subject of the poem, which adds no value to the interpretation of the poem. For example, in the summary of the poem "Gold" in Table 5, the model gives us some facts about gold that are not a part of the main theme of the poem. The model also seems to be focusing too much numbers, which would be okay for news article summarization but not in the case of poem summarization.
Kento Watanabe and Masataka Goto. 2020. Lyrics in- Analysis, generation, and ap- plications. In Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA), pages 6-12.
formation processing:
Adhika Pramita Widyassari, Supriadi Rustad, Guruh Fa- jar Shidik, Edi Noersasongko, Abdul Syukur, Af- fandy Affandy, et al. 2022. Review of automatic text summarization techniques & methods. Journal of King Saud University- mniversity-Compueer and Information Sciences, 34(4); 11110002---0 1046.
Shira Wolosky. 2008. The art poetry: How to read poem. Oxford University Press.
Linli Xu, Liang Jiang, Chuan Qin, Zhe Wang, and Dong- fang Du. 2018. How images inspire poems: Gen- erating classical chinese poetry from images with memory networks. Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 32.
Table shows some examples of the differ- ent lackings that we found in the case our best-performing model T5 and also our worst- performing model mT5.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- lernational Conference on Machine Learning, pages 11328-11339. PMLR.

---

Poem
Summary
"Ozymandias" by B. Shelley describes a traveler's reaction the half-buried, worn-out statue the great pharaoh, Ramses In this poem, the speaker describes meeting traveler "from an antique land," The title. "Ozymandias" notifies the reader that this land is most probably Egypt since Ozymandias was what the Greeks called Ramses II. He was a great and terrible pharaoh in ancient Egypt. The traveler tells story to the speaker. In the story, he describes visiting Egypt. There, he saw a large and intimidating statue of Ramses in the desert. He can tell that the sculptor must have known his subject well because it is obvious from the statue's 's face that this man was a great leader, but one who could also be very vicious. He describes his sneer as having a "cold command." Even though the leader was probably very great, it seems that the only thing that survives from his realm is this statue, which is half-huried and somewhat falling apart.
met a traveller from an antique land, Who said- vast and trunkless legs of stone Stand in the desert... ... Near them, on the sand, Half sunk a shattered visage lies, whose frown, And wrinkled lip, and sneer of cold command, Tell that its sculptor well those passions read Which yet survive, stamped on these lifeless things, The hand that mocked them, and the heart that fed;
And on the pedestal, these words appear: My name is Ozymandias, King of Kings: Look on my Works, ye Mighty, and despair! Nothing beside remains. Round the decay Of that colossal Wreek, boundless and bure The lone and level sands stretch far away."
How good to lie little while And look up through the tree!
"Friends" by Abbic Farwell Brown deseribes the way that nature, specifically the Sky. Wind, and Sun provide comfort if one willing let them. The poem begins with the speaker describing how she finds joy and peace equal measure out in nature. She likes go and "lie little while" underneath specific tree. From her spot there she sees the sky ahove her, looking down and protecting her. She is also able enjoy the feeling of the dappled sunlight her skin and take in the lace-like patterns produces. In the second half of the poem, she deseribes the way the "Wind" sncaks up on her and whispers "pretty things" her ear. Every around her there a force of nature ready to comfort her if she needs
The Sky is like a kind big smile
Bent sweetly over me.
The Sunshine flickers through the lace Of leaves above my head.
And kisses me upon the face
Like Mother, hefore bed.
The Wind comes stcaling o' er the grass
To whisper pretty things;
And though I cannot see him pass,
feel his careful wings.
So many gentle Friends are near
Whom one can scarcely see,
A child should never feel a fear. Wherever he may be.
Some day, when trees have shed their leaves And against the morning's 's white The shivering birds heneath the caves
Have sheltered for the night, We'll turn our faces southward, love,
"After the Winter" by Claude McKay is a beautiful and simple piece of poetry about an idcalized fugue on the "summer isle" away from winter. The speaker spends the first stanza initiating his dream and suggesting that when winter is over, they 're going to be able to "turn" their faces toward the sumuer isle and forget about the cold, inhospituble past. When they go towards the summer isle, they 'll find peace. There will be everything that could hope to find in the natural world. This includes rivers. flowers, cotton trees, and more. The poet uses personification to depict these scenes as well as good examples of imagery.
Toward the summer isle Where bamboos spire the shafted grove
And wide-mouthed orchids smile.
And we will seek the quiet hill
Where towers the cotton tree,
And leaps the laughing crystal rill.
And works the droning hec.
And we will build cottage there
Beside open glade, With black-ribbed blue-bells blowing near, And ferns that never fade.
Table 4: Samples of Poem and Summary from the PoemSum Dataset

---

Model
Poem
Generated Summary
Lackings Observed
Bleeding Heart
"Bleeding Heart" by Carmen Gim- nez Smith is a powerful poem that describes how the speaker's 's heart is bleeding. The poem begins with the speaker stating that she has lot of blood in her body. It is filled with salt, and it can be seen from every angle. This is something she feels when she speaks about her own life. She suffers because of everything she sees as irreproducible. In the second half of the poem, the speaker tells the reader that she is bleeding for the sake of her sister. She is also bleeding to the tips of her fingers
Carmen Giménez
The generated summary inter- prets bleeding as actual blood flow but the poet has actually used the word "bleeding" as metaphor to symbolize her emotional pain and sympathy, The model only considers the lit- eral meaning of all the imagery, symbolism and metaphors used in the poem. Moreover, it di- rectly includes some sentences from the poem in the summary.
My heart is bleeding. It bleeds upward and fills my mouth up with salt. It bleeds because of city in ruins.
T5
Even for a bad zoning decision, I'll bleed so much you'll 'll he bleeding,
all of us bleeding in and out like i's breathing.
or kissing. and because it righteous and terrible and red.
Song For a Dark Girl Langston Hughes
The generated summary states that the poem is about the re- lationship of the speaker with her lover. But the poem is actu- ally about the tragic loss young woman had suffer when her lover fell victim racial vio- lence. The model cannot com- prehend the speaker's frustration expressed through the rhetori- cal question "What was the use of prayer". The model also fails to understand the metaphor in the last sentence referring to speaker's 's dead lover.
"Song For Dark Girl" hy Langston Hughes is image-rieu rich poem that depicts the speaker's relationship with his black young lover. The poem begins with the narrator de- scribing how he and his white friend hung their love on trees. They hung him in Dixie, South America. Hc then says that he wants break the heart of his daughter. This not because is going to be true. In the next lines, the poet describes what happened to him as he was alone in the same place.
Way Down South in Dixic
(Break the heart of me)
They hung my black young lover
cross roads tree.
Way Down South in Dixie (Bruised body high in air)
Ts
asked the white Lord Jesus What was the use prayer. Way Down South in Dixie (Break the heart of me) Love is a naked shadow On gnarled and naked tree.
Gold!
-Thomas Hood
Gold! Gold! Gold! Gold!
"Gold" by Thomas Hood is a short poem about gold. the first lines of this poem, the speaker begins by stating that gold is an important part of the world. It is valued at more than 100m and it is used to sell for money. The second stanza focuses on the fact that Gold is one of most valuable things in the country.
The content of the generated summary is similar to news summarization. The generated summary only expresses the im- portance of gold in some factual terms but it fails to understand the contrasting nature of gold mentioned in the poem and its impact on humans.
Bright and yellow, hard and cold Molten, graven, hammered and rolled, Heavy to get and light to hold, Hoarded, hartered, bought and sold,
mT5
To save to ruin- to curse to bless As even its minted coins express: Now stamped with the image of Queen Bess, And now of a bloody Mary.
"i am water" by Rupi Kaur is a short poem that speaks on the nature of water. The poem begins with the speaker stating that water is soft enough to offer life tough sufficient to drown it away. It is hard to pro- vide life for humankind, but it is not so much as it can be done. In the next lines, the poet describes what it has to do and how it will help them to survive.
Here the model misunderstands the personification of water. The speaker is not actually talk- ing about the characteristies of water. Rather, she is using the nature of water to convey that both softness and toughness can coexist within her.
am water Rapi Kaur am water soft enough to offer life tough enough to drown away
mT5
Table 5: Generated summaries by the best-performing (T5) and worst-performing (mTS) models along with criticism on their limitations


Reviews:
[{"review_id": "ueYgcpitzF", "reviewer": "Reviewer_6WjZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe extend our gratitude for the invaluable feedback you provided regarding our manuscript. We hope to address the concerns you raised regarding the submitted manuscript.\n\n### Response to reasons to reject:\n**Regarding adding minor aspects in Table-1:** We thank you for this critical observation. We agree that adding these information will improve the clarity of our manuscript. Hence the following information will be added in Table 1 of the manuscript. \nThe information to be appended are as follows: \n| Metrics                             | Value       |\n|---------------------------------------|-------------|\n| Average number of words per poem | 248 |\n| Average number of words per summary | 162 |\n| Average number of poems per poet | 3.24       |\n\nWe will append this information in the revised manuscripts.\n\n**Regarding time distribution of the poems:** The time distribution of poems/authors per century was not available in the sources from where the data was collected and hence we were unable to include this information.\n\n**Regarding list of webpages:** Since we could not share the dataset repository in the initial manuscript, we added a list of sources in the appendix. While we put it in the appendix to provide more transparency of the data curation process, we can also put it on the Github repo when the dataset will be released.\n\n**Regarding reproducibility:** The code and the dataset will be made publicly available along with the train, validation and test split once accepted, hence we firmly believe reproducibility will not be an issue.\n\nYour feedback has been helpful for improving vital aspects of our work. We hope our final manuscript will address your concerns and demonstrate the value of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's main strength lies in the innovative task it proposes and the valuable contribution through the creation of a novel dataset. The significance of interpreting figurative language in various NLG applications is highlighted effectively. The task and its associated challenges are presented and justified with great clarity. Moreover, the paper is skillfully written, making a compelling argument for the importance of this task. Additionally, it conducts a preliminary evaluation of various language models using the provided dataset.", "reasons_to_reject": "While I recommend accepting the paper, I believe there are a few minor aspects that could enhance it. Despite being a short paper, it could include the suggestions provided below, rather than presenting an extensive and unnecessary list of source web pages.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the challenges of interpreting the intended meaning of creative language in poetry and proposes a new task in natural language understanding called \"Poem Summarization.\" The task aims to summarize poems, taking into account their deeper meanings, which are often lost if only the literal meaning is considered. The authors have created a dataset called \"Poem-Sum,\" consisting of 3011 samples of poetry and their corresponding interpretations, which will be publicly available upon acceptance of their manuscript.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper's corpus/data set characterization requires improvement. Enhancing it with relevant figures, possibly in Table 1, such as the average number of words/tokens per poem and summary, the time distribution of poems/authors (and potentially poems) per century, and the average number of poems per poet, would be beneficial."}}, {"review_id": "4gknIaPDii", "reviewer": "Reviewer_r8W8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank you for the invaluable feedback you provided regarding our manuscript. We hope to address your concerns in the following text.\n\n### Response to reasons to reject:\n\n**Regarding Size of Dataset:** The main issue which dictated the size was the availability of good quality poem summaries. If more summaries were available, we could have increased the dataset size even more. We hope to extend the dataset in the near future, also enriching it with poems from other languages as well.\n\n**Regarding the Obtained Results:** While the results are not impressive, we hope this work will highlight the importance and difficulty of this particular task, at the same time inspire others to come up with better architectural solutions for working with creative language.\n\nYour feedback has been helpful for reinstating the importance of the issues being addressed by our work. We hope our comments are able to properly address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Summarization of literary work, and specially summarizing poems which requires a deep understanding of metaphor, symbolism, and emotion, is not an area that has been addressed enough in literature. This paper highlights this point as well as how this task differs from summarizing non-literary text and presents a good starting point for further work in this area. \n- The authors have stated that the dataset described in the paper will be released to the public upon paper acceptance. Assuming that the authors release their exact splits (training, validation, and testing), the work presented will serve as a good benchmark for other researchers continuing in this direction. \n", "reasons_to_reject": "- The presented dataset is some what small, but that is understandable. \n- The presented results are not that impressive, but again this only highlights the difficulty of the problem being addressed", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces \"PoemSum,\" a novel dataset for poem summarization, aimed at understanding the figurative language of poems. It highlights the distinction between creative and formal language, revealing that current summarization models struggle to capture the main ideas of poems. The dataset is proposed as a valuable resource for evaluating language models' comprehension of figurative language in poetry, offering potential for improved understanding and generation of poetic language while advancing the field of creative language summarization. The paper presents the results of finetuning various models using this dataset vs one trained on news data only, highlighting 1) the finetunes models work better, 2) that there is still room for much improvement in the summarization of literary work. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sudd8CB44P", "reviewer": "Reviewer_dv38", "summary": "", "questions": "A) It is not clear in Table 1 what is the unit of Max poem size and Max summary size, can you clarify this?\nB) What figurative language patterns are present in poems that LLMs didn't capture?\n", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe extend our gratitude for the invaluable feedback you provided regarding our manuscript. We hope to address your concerns comprehensively in the subsequent text.\n\n### Response to the questions:\n\n**Unit in Table 1:** The unit of max poem and max summary size in Table-1 is the **number of words**. This information was unintentionally not included in the submitted manuscript. We will surely add this information in the camera-ready version, along with other information like:\n| Metrics                             | Value       |\n|-------------------------------------|-------------|\n| Average **number of words** per poem    | 248  |\n| Average **number of words** per summary | 162  |\n| Average **number of poems** per poet    | 3.24        |\n\n**Regarding figurative patterns that the LLMs don’t capture:** We thank the reviewer for this critical observation. There are some figurative language patterns that are not captured by the LLMs like T5, Pegasus, Bart, mT5 we used. We agree that the inclusion of this analysis will improve the quality of our manuscript. We promise to add them in the Appendix section of our revised manuscript. Some examples are shown in the following table from our best-performing (T5) and worst-performing (mT5) models:\n\n| Model | Figurative Pattern  | Poem                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Generated Summary                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Lackings Observed                                                                                                                                                                                                                                                                                           |                                                            |\n|-------|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|\n| **mT5**    | Personification     | (i am water by Rupi Kaur) i am water soft enough to offer life tough enough to drown it away                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | 'i am water' by Rupi Kaur is a short poem that speaks on the nature of water. The poem begins with the speaker stating that water is soft enough to offer life tough sufficient to drown it away. It is hard to provide life for humankind, but it is not so much as it can be done. In the next lines, the poet describes what it has to do and how it will help them to survive.                                                                                                                                                                        | Here the model **misunderstands the personification of water**. The speaker is not actually talking about the characteristics of water. Rather she is using water as a metaphor to convey that softness and toughness can coexist within her.                                                                                   |                                                            |\n| **T5**    | Metaphor            | (Ariel by Sylvia Plath) Stasis in darkness. Then the substanceless blue Pour of tor and distances  God's lioness How one we grow Pivot of heels and  knees!--The furrow ........ The dew that flies Suicidal, at one with the drive Into the red  Eye, the cauldron of morning.                                                                                                                                                                                                                                                                                       | 'Ariel' by Sylvia Plath is a beautiful poem that depicts the life of Ariel. The poem begins with the speaker stating that she is trapped in darkness. There is no substance to her body, and there is nothing to distract her from it. It is like a furrow splits and passes through its sister. She can't catch it, but it does cast shadows on her. In the second half of the poem, the speaker talks about how she has become a lioness and Godiva.                                                                                                    | The poem actually talks about the mental and emotional transformation that the rider goes through when she faces death but the prediction summary says it depicts the life of Ariel. It **fails to understand the dark metaphors**. For example- black sweet blood mouthfuls of berries indicate blood and danger in this poem. |                                                            |\n| **T5**    | Rhetorical question | (Song For a Dark Girl  by Langston Hughes)  Way Down South in Dixie (Break the heart of me) They hung my black young lover To a cross road tree.  Way Down South in Dixie (Bruised body high in air) I asked the white Lord Jesus What was the use of prayer.  Way Down South in Dixie (Break the heary of me) Love is naked shadow  On a gnarled and naked tree.                                                                                                                                                                                                    | 'Song For a Dark Girl' by Langston Hughes is an image-rich poem that depicts the speaker's relationship with his black young lover. The poem begins with the narrator describing how he and his white friend hung their love on trees. They hung him in Dixie, South America. He then says that he wants to break the heart of his daughter. This is not because it is going to be true. In the next lines, the poet describes what happened to him as if he was alone in the same place.                                                                 | The line “I asked the white Lord Jesus / What was the use of prayer” **poses a rhetorical question** that expresses the speaker’s despair and doubt about her prayer getting accepted when they face brutality because of their skin color. The model **fails to convey this** in the summary.                                                 |                                                            |\n| **T5**    | Symbolism           | (Bleeding Heart  by Carmen Gimnez Smith)  My heart is bleeding. It bleeds upward and  fills my mouth with salt. It bleeds because of a city in ruins, the chair still warm from sister's body, because it will all be irreproducible. My heart bleeds because of baby bear not finding mama bear bear and it bleeds to tips of my fingers like I painted my nails Crimson.                                                                                                                                                                                           | 'Bleeding Heart' by Carmen Gimnez Smith is a powerful poem that describes how the speaker's heart is bleeding. The poem begins with the speaker stating that she has a lot of blood in her body. It is filled with salt, and it can be seen from every angle. This is something she feels when she speaks about her own life. She suffers because of everything she sees as irreproducible. In the second half of the poem, the speaker tells the reader that she is bleeding for the sake of her sister. She is also bleeding to the tips of her fingers | The generated summary fails to capture the inner meaning of the poem. Here ‘bleeding‘ **symbolizes                                                                                                                                                                                                                            poet's sympathy and emotional pain, not actual blood flow**. |\n| **mT5**   | Contrast            | (Gold by Thomas Hood)  Gold! Gold! Gold! Gold! Bright and yellow, hard and cold Molten, graven, hammered and rolled, Heavy to get and light to hold, Hoarded, bartered, bought and sold, Stolen, borrowed, squandered, doled, Spurned by young, but hung by old To the verge of a church yard mold; Price of many a crime untold. Gold! Gold! Gold! Gold! Good or bad a thousand fold! How widely it agencies vary, To save - to ruin - to curse - to bless - As even its minted coins express : Now stamped with the image of Queen Bess, And now of a bloody Mary. | 'Gold!' by Thomas Hood is a short poem about gold. In the first lines of this poem, the speaker begins by stating that gold is an important part of the world. It is valued at more than   100m and it is used to sell for money. The second stanza focuses on the fact that Gold is one of most valuable things in the country.                                                                                                                                                                                                                          | The content of the generated summary is **similar to news summarization**. The generated summary only expresses the importance of gold but it **fails to understand the contrasting nature of gold** and its impact on humans. This example matches with our analysis in **section 5** of our paper regarding the inability of news article pretrained models when it comes to creative language summarization.                                                                                                     |                                                            |\n\n### Response to reasons to reject:\n\n**Regarding working with a single language:** There is a scarcity of high quality poem summaries for other languages, which led us to choose only English poems as a start. It is possible to extend the work to other low resource languages with more rigorous data collection efforts. We aim to do that in our future works. \n\n**Regarding curation process:** We have discussed the data collection and cleaning process in section **3.1** and **3.2**. Due to the space limitation, we skipped a few details, which we promise to add in the final version, expanding the mentioned sections. The following text gives a detailed look into our curation process:\n1. **Search Methodology:** We first searched for public websites with high quality English Poem Summaries. Although these were available in numerous public websites, most websites had a very small collection. Considering this, we used the website with the highest collection of summaries, as mentioned in our submitted manuscript. The summaries along with the poem name, poet name were collected via automatic web scrapping. This website hosts only the summary but not the poem itself. For the purpose of collecting the poems, four data collectors were hired and tasked to collect the poem and URL of the poem's source website given poem name and poet name. Poems that could not be found in credible websites had to be discarded from the dataset.\n2. **Data Cleaning:** The data was first cleaned using automated techniques to remove unnecessary white spaces, garbage values and duplicate records. Sometimes html tags were included in the data due to differences in webpage structure, such inconsistencies were resolved. After automated cleaning, the data was again verified by human annotators with experience in dataset preparation.\n3. **Data Verification:** For the purpose of data verification, another group of annotators with similar level of expertise were hired. They were tasked with cross checking the collected poems from another website other than the one which the initial collectors had listed. If any mismatch was found, the data was discarded. The overall summaries and poems were checked as well. We set some standards that every verified summary had to maintain:\n* Summary cannot be limited to explaining only the rhyming scheme of the poem\n* Summary cannot contain only the poet's biography without any insights into the poem itself\n* Summary cannot only discuss other works of the poet without context \n* Summary must not only copy lines from the poem without any attempts to extract a deeper meaning\n\n  Summaries that did not meet the specified standards were discarded.\n\n**Regarding limited analysis:** We acknowledge the importance of the identified issue for showing future research directions. In this regard, we discussed the overall performance in Table 2, and a sample use case in Table 3, to point out some of the pitfalls of an existing model. As a part of our result analysis, we observed that LLMs don't perform well in poem summarization. But models that were pretrained on news article summarization, like mT5 in our case, performed even worse. As mentioned in **section 5**, this could mean that existing summarization models cannot comprehend figurative language. Due to space constraints, we couldn't add detailed analysis in the manuscript. More analysis like the table shown above will be added in the appendix section of the manuscript.\n\nYour feedback has been instrumental in helping us identify areas that require further clarification and improvement. We hope our final manuscript will address your concerns and demonstrate the value of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel task formulation: Poem summarization is a new task that requires interpreting creative, figurative language. This expands NLP research into a new domain beyond current tasks like news or scientific text summarization.\n- Useful benchmark dataset: PoemSum is the first dataset for poem summarization. It can spur further research in this area and allow systematic evaluation of new models on this task. Availability of datasets is crucial for progress.", "reasons_to_reject": "- Single language: Only English poems are included. Multilingual creative language interpretation may exhibit different challenges.\n- Curation process not explained: Details of how the dataset was compiled, such as search methodology or data cleaning steps, are omitted.\n- Limited analysis: The paper mostly focuses on introducing the task and dataset. Deeper analysis into model capabilities and errors is left for future work.", "questions_to_authors": "A) It is not clear in Table 1 what is the unit of Max poem size and Max summary size, can you clarify this?\nB) What figurative language patterns are present in poems that LLMs didn't capture?\n", "paper_topic_and_main_contributions": "This paper introduces a new dataset and benchmark for poem summarization in English language.\nContributions:\n- The first dataset for poem summarization called \"PoemSum\". It contains 3011 samples of poems in English paired with human-written summaries.\n- Evaluates standard summarization models like BART, T5, Pegasus on PoemSum. Finds that they do not perform well, indicating poem summarization requires new approaches beyond text summarization.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new task (and a dataset) in the field of natural language understanding called "Poem Summarization". 

The reviewers agree on the soundness of the proposal being good-strong, while the excitement causes more variability (mediocre-strong). Taking into account the novelty of the proposal, and that the reasons to reject have nothing to do with the excitement of the proposal, I take it being rather high. The reviewers raised reasonable and constructive comments and concerns that the authors acknowledge and will include in the final version, resulting in an improved work. The main ones being: 
 
- Further dataset metrics
- Further analyses on figurative patterns that LLMs do not capture
- Details on curation processes

Working on a single language, should, in my opinion, do not preclude the publication of the work. Inasmuch as the code and the dataset will be made publicly available along with the train, validation and test split once accepted, I also believe reproducibility should not be an issue.
Decision: Accept-Main