COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generation
Authors: Raghvendra Kumar, Mohammed Salman S A, Aryan Sahu, Tridib Nandi, Pragathi Y P, Sriparna Saha, Jose G Moreno
Accepted at: acl25

COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generation

Raghvendra Kumar1, S. A. Mohammed Salman2, Aryan Sahu3, Tridib Nandi4, Pragathi Y. P.5, Sriparna Saha1, Jose G. Moreno6

1Department of Computer Science and Engineering, IIT Patna, India
2Department of Metallurgical and Materials Engineering, NIT Tiruchirappalli, India
3Department of Computer Science and Information Systems, BITS Pilani Goa, India
4Department of Computer Science and Engineering, IIIT Vadodara, India
5Department of Computer Science and Engineering, BMSCE Bangalore, India
6Université de Toulouse, IRIT UMR 5505 CNRS, France

For inquiries: {raghvendra.kumar1004, samohammedsalman, aryansahu010103}@gmail.com
For inquiries: {tridib.nandi25062002, pragathiyp2352, sriparna.saha}@gmail.com
For inquiries: jose.moreno@irit.fr

Abstract

Despite progress in comment-aware multimodal and multilingual summarization for English and Chinese, research in Indian languages remains limited. This study addresses this gap by introducing COSMMIC, a pioneering comment-sensitive multimodal, multilingual dataset featuring nine major Indian languages. COSMMIC comprises 4,959 article-image pairs and 24,484 reader comments, with ground-truth summaries available in all included languages. Our approach enhances summaries by integrating reader insights and feedback. We explore summarization and headline generation across four configurations: (1) using article text alone, (2) incorporating user comments, (3) utilizing images, and (4) combining text, comments, and images. To assess the dataset’s effectiveness, we employ state-of-the-art language models such as LLama3 and GPT-4. We conduct a comprehensive study to evaluate different component combinations, including identifying supportive comments, filtering out noise using a dedicated comment classifier using IndicBERT, and extracting valuable insights from images with a multilingual CLIP-based classifier. This helps determine the most effective configurations for natural language generation (NLG) tasks. Unlike many existing datasets that are either text-only or lack user comments in multimodal settings, COSMMIC uniquely integrates text, images, and user feedback. This holistic approach bridges gaps in Indian language resources, advancing NLP research and fostering inclusivity. The resources are available at: https://github.com/AaryanSahu/COSMMIC.

1 Introduction

In today’s digital era, online content seamlessly integrates text and visuals, with user comments enriching discussions and providing diverse perspectives. Reader-Aware Comment-Based Summarization (RACBS) harnesses these interactions to generate more relevant and comprehensive summaries by incorporating user feedback (Kumar et al., 2023). However, the development of multilingual, comment-based, multimodal datasets for Indian languages remains largely unexplored. For instance, a news article on the Indian general elections1 often features images alongside a flood of reader comments in multiple languages. While the text delivers factual information and images offer visual context, the comments reflect public sentiment, debates, and interpretations, making them a valuable yet underutilized resource for enhancing summarization.

Motivation for the dataset: The challenge lies in analyzing such multifaceted content comprehensively. Current datasets often fail to integrate all these elements—text, images, and comments—particularly in the context of Indian languages. A dataset limited to text or images alone cannot capture the full spectrum of user engagement and content richness. Our dataset COSMMIC addresses this shortcoming by combining article text, images, and user comments in nine major Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam, Odia, Tamil, Telugu, and Kannada. This integrated approach offers a more

1https://en.wikipedia.org/wiki/2024_Indian_general_election
complete view of online content. It is essential for developing advanced NLP models that can process and understand the full scope of information in today’s digital media landscape.

**NLG tasks driven by the dataset presented:** Summarization and headline generation are key NLP challenges with applications in news aggregation, content curation, and information retrieval (Tan et al., 2017; Singh et al., 2021; Verma et al., 2023a; Hamborg et al., 2017; Khatter and Ahlawat, 2022; Gaci et al., 2023; Mahalakshmi and Fatima, 2022; Esteva et al., 2021). Summarization condenses text into concise, coherent outputs, while headline generation crafts engaging titles, both leveraging advanced language models (Zhang et al., 2024) and multimodal data (Krubiński and Pecina, 2024b) for improved accuracy and relevance.

Integrating images and user comments enriches these tasks, adding context to summaries and enhancing headline relevance (Li et al., 2017; Gao et al., 2019). Multimodal approaches (He et al., 2023; Qiao et al., 2022) combining text, visuals, and user interactions produce more informative outputs. Building on these advancements, we assess the impact of filtered comments and categorized images on NLG tasks. By integrating only informative inputs, our approach enhances coherence and relevance, with evaluations confirming its effectiveness in multilingual and multimodal RACBS. Our contributions enhance context-aware summaries and headlines, as detailed below.

*(1) We introduce “COSMMIC”, an innovative comment-sensitive multimodal, multilingual dataset having article-image pairs with reader comments covering nine major Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam, Odia, Tamil, Telugu, and Kannada. This dataset represents a pioneering effort in this field.*

*(2) Human-written ground truth summaries are provided for all multilingual news articles within the dataset.*

*(3) We establish benchmarks for our dataset by evaluating two NLG tasks—headline generation and summarization—using state-of-the-art language models such as GPT-4 and LLama3.*

*(4) An extensive study is conducted to explore various configurations: article text alone, article text with comments, article text with relevant comments, article text with images, article text with images and comments, and article text with images and relevant comments.*

*(5) To advance this task, we propose methods for filtering noisy comments and classifying images in the Hindi dataset. Using IndicBERT (Kakwani et al., 2020), we develop a classifier to retain only informative comments and filter out noisy comments. Additionally, a threshold-based multilingual CLIP (Carlsson et al., 2022; Reimers and Gurevych, 2019) classifier categorizes images as supplementary (reinforcing the text) or complementary (providing additional context). To assess linguistic preservation in RACBS, we perform a back-translation study on the Hindi dataset, highlighting quality degradation and the need to retain the original language.*

2 Related Works

We explore multilingual, multimodal datasets with reader comments for NLG tasks, as Table 1 details their linguistic, modal, and structural differences, excluding English from Indian languages.

**Multilingual Datasets:** Scialom et al. (2020) introduced MLSUM, a large-scale dataset containing 1.5 million article-summary pairs in five languages, emphasizing cross-lingual biases. Hasan et al. (2021) developed XL-Sum, featuring 1 million pairs across 44 languages for abstractive summarization. Ladhak et al. (2020) presented WikiLingua, which includes 18 languages and image-based alignments for cross-lingual summarization. Hewapathirana et al. (2024) introduced M2DS, the first multilingual dataset for multi-document summarization (MDS) using BBC articles. Aralikatte et al. (2023) developed Varta, a dataset designed for headline generation in 14 Indic languages and English. Kumar et al. (2022) introduced the Indic-NLG (abbreviated as INLG in Table 1) benchmark for NLG tasks across 11 Indic languages. Readers are also referred to notable works such as Urlana et al. (2023), Cao et al. (2020), Varab and Schluter (2021), Feng et al. (2022), and Gliwa et al. (2019).

**Multimodal Datasets:** In the realm of multimodal datasets, Zhu et al. (2018) introduced MSMO, a dataset for multimodal summarization that includes both text and image outputs and features a multimodal attention model. Verma et al. (2023b) developed M3LS, the largest multilingual multimodal summarization dataset, comprising over a million document-image pairs across 20 languages. Krubiński and Pecina (2024a) presented a unified encoder-decoder model for headline generation, which enhances M3LS by incorporating
<table>
  <tr>
    <th>Datasets:</th>
    <th>MMSUM</th>
    <th>M2DS</th>
    <th>Varta</th>
    <th>XLHT</th>
    <th>INLG</th>
    <th>MSMO</th>
    <th>M3LS</th>
    <th>RAMDS</th>
    <th>RASG</th>
    <th>COSMMIC</th>
  </tr>
  <tr>
    <td>Multilingual</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Multimodal</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Comments</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Summaries</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Headlines</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td># Indian Lang.</td>
    <td>8</td>
    <td>1</td>
    <td>14</td>
    <td>8</td>
    <td>11</td>
    <td>0</td>
    <td>8</td>
    <td>0</td>
    <td>0</td>
    <td>9</td>
  </tr>
</table>

Table 1: Comparison of Datasets Based on Multilingualism, Multimodality, Comment Inclusion, Presence of Summaries and Headlines, and Coverage of Indian Languages.

additional images. Wang et al. (2022) released N24News, a dataset from the New York Times with both text and image information organized into 24 categories. Shohan et al. (2024) developed XL-HeadTags (shortened as XLHT in Table 1), a multilingual and multimodal dataset for headline and tag generation, using images and captions in 20 languages. Gotmare et al. (2024) proposed a machine-learning algorithm designed to generate news articles from geo-tagged images. Liang et al. (2023) introduced MM-Sum, a multilingual multi-modal abstractive summarization dataset enhancing summary quality across 44 languages. Other noteworthy works include Kumar et al. (2024), Mita et al. (2023), Fu et al. (2021), Patil et al. (2024), Qiu et al. (2024), Nguyen et al. (2023), and Ghosh et al. (2024). *Limitations: Despite their significance, these datasets are text-only or multimodal without user comments.*

Reader Comments Inclusive Datasets: Zaidan and Callison-Burch (2011) pioneered the Topic-driven Reader Comments Summarization (Torcs) problem by creating a dataset of 1,005 Yahoo! News articles with over one million comments. Li et al. (2017) extended a variational autoencoder-based framework for reader-aware multi-document summarization (RA-MDS), presenting a new dataset with a detailed collection, annotation, and expert review process, demonstrating improved summarization performance with reader comments. Risch and Krestel (2020) introduced a dataset of journalist-reader dialogues from The Guardian, proposing a ranking task for comments. Gao et al. (2019) developed the Reader-Aware Summary Generator (RASG) using reader comments from Weibo to enhance summary relevance. Other noteworthy works include Zaidan and Callison-Burch (2011), Meng et al. (2024), Fujita et al. (2019), Roha et al. (2022), and Shi and Lam (2019). *Limitations: Despite their significance, these datasets lack focus on Indian languages, highlighting the need for our proposed dataset.*

3 COSMMIC Dataset

Our dataset is pioneering in the Indian context, featuring 4,959 article-image pairs and 24,484 reader comments, with comprehensive ground-truth summaries across all languages. It encompasses 500 to 650 articles for each of the nine major languages: Bengali, Hindi, Gujarati, Marathi, Malayalam, Odia, Tamil, Telugu, and Kannada. While the dataset might seem relatively small for each individual language, it offers a significant volume compared to the RAMDS benchmark dataset (Li et al., 2017), which includes 450 news articles and 10,000 comments across 45 topics. Our dataset provides broader coverage across nine languages, enhancing its utility. It also enabled us to manually verify each article to ensure that, after scraping and processing, the dataset is properly structured and free from any irrelevant information.

Additionally, with the rise of LLMs and their emphasis on zero-shot (Kojima et al., 2022) and few-shot learning (Brown et al., 2020), the size of our dataset is well-suited for fine-tuning these models. *Moreover, our dataset encompasses news exclusively from 2024. Extending further back proved impractical due to the limited number of comments on older articles. Even for articles from 2023 and 2022, there were only a few comments per 100 articles. Consequently, we concentrated on more recent news to ensure a robust dataset.*

3.1 Dataset Curation

**Data Source:** We selected DailyHunt for data scraping due to its diverse, multilingual content and integration of text, images, and user comments, providing a rich multimodal dataset for tasks like summarization and headline generation. Its focus on regional languages and local news fills gaps in predominantly English-centric datasets, while its real-time updates enable applications such as trend analysis and fake news detection. Additionally, the platform’s vast articles and user interactions enable
scalable research and model training, making it invaluable to our study.

Scraping tools: We used Selenium to scrape news articles from DailyHunt by navigating URLs, scrolling to load dynamic content, and collecting unique “Read more” links. For each article with comments, we extracted the headline, related images, main content (excluding disclaimers), and comments, saving the data to a CSV file. The process involved initializing the Chrome WebDriver, dynamically loading content, and using BeautifulSoup for HTML parsing. Figure 1 illustrates the extracted components for a sample article. The language-agnostic code was uniformly applied across nine languages. After scraping, we manually verified and cleaned the data, removing author and editor names (1–2% of cases) for consistency.

3.2 Statistics
We present a quantitative overview of our dataset. Table 2 details key attributes per article, Table 3 lists article and comment counts by language, and Table 4 compares average word and sentence counts for headlines and summaries. Marathi has the highest comment volume and density, while Odia has the lowest. Bengali and Marathi lead in article counts, with Marathi, Bengali, and Hindi showing the most comment activity, reflecting higher user interaction.

3.3 Crafting Summaries
Flexibility in Summary Composition: Substantial variability in article features across different Indian languages is observed, as detailed in Table 2, underscoring the necessity of flexibility in summary composition. Imposing a rigid summary length, such as a fixed word count (Li et al., 2017), would not have accommodated the natural linguistic diversity and structural differences among these languages. A one-size-fits-all approach would have risked compromising the quality and relevance of the summaries, failing to capture the true essence of the content. Therefore, flexibility in summary length and structure was crucial to accurately capture the unique characteristics and information density of each language, a principle central to our approach. The summaries created were inclusive of user comments. Noisy comments were manually removed (in appendix A.8).

Annotator Guidelines: A diligently selected team of ten annotators, each holding a master’s in linguistics for their respective languages, was engaged to ensure the highest standards in creating ground truth summaries. Each annotator was proficient in at least three languages, allowing for a profound understanding of linguistic sharpness and cultural contexts across the nine languages in the dataset. The process involved four annotators per language: two generated the summaries, with each summarizing half of the sample set, such as 50 out of 100 samples, while the remaining two conducted thorough reviews of all summaries. A summary was deemed final only when it received approval from both reviewing annotators, ensuring a robust quality assurance process. The guidelines mandated that summaries be concise yet comprehensive, capturing essential ideas and significant details while accommodating natural variability in length. Annotators were instructed to faithfully reflect the meaning and context of the original text, avoiding biases and preserving the tone and intent of the source material. Importantly, reader comments were excluded from consideration to prevent external influences. Adhering to these standards and employing expert annotators ensured the summaries were of high quality and preserved the dataset’s integrity. We compensated annotators at USD 1.20 per 10 summaries.

4 Summarization and Headline Generation

4.1 Methodology
While prior studies Ahuja et al. (2023, 2024); Kumar et al. (2022); Singh et al. (2024); Bhat et al. (2023) have benchmarked NLG for Indic languages using LLMs, none have explored reader-aware summarization or headline generation, a gap our work addresses.

Configurations Used: Our comprehensive studies explore various configurations to assess the
<table>
  <tr>
    <th rowspan="2">Languages →</th>
    <th colspan="4">Hindi</th>
    <th colspan="4">Bengali</th>
    <th colspan="4">Marathi</th>
  </tr>
  <tr>
    <th>Features ↓</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Token</td>
    <td></td>
    <td>118</td>
    <td>4160</td>
    <td>518</td>
    <td>28</td>
    <td>1962</td>
    <td>339</td>
    <td>36</td>
    <td>2652</td>
    <td>353</td>
  </tr>
  <tr>
    <td>Words</td>
    <td></td>
    <td>100</td>
    <td>3712</td>
    <td>466</td>
    <td>24</td>
    <td>1661</td>
    <td>288</td>
    <td>31</td>
    <td>2229</td>
    <td>303</td>
  </tr>
  <tr>
    <td>Unique Words</td>
    <td></td>
    <td>69</td>
    <td>966</td>
    <td>214</td>
    <td>23</td>
    <td>834</td>
    <td>193</td>
    <td>24</td>
    <td>1079</td>
    <td>202</td>
  </tr>
  <tr>
    <td>Sentences</td>
    <td></td>
    <td>2</td>
    <td>228</td>
    <td>25</td>
    <td>2</td>
    <td>119</td>
    <td>26</td>
    <td>2</td>
    <td>210</td>
    <td>24</td>
  </tr>
  <tr>
    <td>Comments</td>
    <td></td>
    <td>1</td>
    <td>148</td>
    <td>6</td>
    <td>1</td>
    <td>107</td>
    <td>6</td>
    <td>1</td>
    <td>153</td>
    <td>6</td>
  </tr>
  <tr>
    <th rowspan="2">Languages →</th>
    <th colspan="4">Gujarati</th>
    <th colspan="4">Malayalam</th>
    <th colspan="4">Odia</th>
  </tr>
  <tr>
    <th>Features ↓</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Token</td>
    <td></td>
    <td>45</td>
    <td>2254</td>
    <td>410</td>
    <td>31</td>
    <td>2005</td>
    <td>259</td>
    <td>46</td>
    <td>3058</td>
    <td>281</td>
  </tr>
  <tr>
    <td>Words</td>
    <td></td>
    <td>43</td>
    <td>2027</td>
    <td>362</td>
    <td>27</td>
    <td>1666</td>
    <td>218</td>
    <td>41</td>
    <td>2511</td>
    <td>242</td>
  </tr>
  <tr>
    <td>Unique Words</td>
    <td></td>
    <td>40</td>
    <td>1010</td>
    <td>212</td>
    <td>24</td>
    <td>1186</td>
    <td>172</td>
    <td>34</td>
    <td>1156</td>
    <td>161</td>
  </tr>
  <tr>
    <td>Sentences</td>
    <td></td>
    <td>1</td>
    <td>137</td>
    <td>23</td>
    <td>1</td>
    <td>186</td>
    <td>23</td>
    <td>2</td>
    <td>297</td>
    <td>20</td>
  </tr>
  <tr>
    <td>Comments</td>
    <td></td>
    <td>1</td>
    <td>34</td>
    <td>3</td>
    <td>1</td>
    <td>82</td>
    <td>6</td>
    <td>1</td>
    <td>25</td>
    <td>2</td>
  </tr>
  <tr>
    <th rowspan="2">Languages →</th>
    <th colspan="4">Tamil</th>
    <th colspan="4">Telugu</th>
    <th colspan="4">Kannada</th>
  </tr>
  <tr>
    <th>Features ↓</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
    <th>Min</th>
    <th>Max</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Token</td>
    <td></td>
    <td>37</td>
    <td>1882</td>
    <td>289</td>
    <td>48</td>
    <td>1918</td>
    <td>327</td>
    <td>53</td>
    <td>1641</td>
    <td>258</td>
  </tr>
  <tr>
    <td>Words</td>
    <td></td>
    <td>28</td>
    <td>1403</td>
    <td>239</td>
    <td>40</td>
    <td>1651</td>
    <td>268</td>
    <td>43</td>
    <td>1402</td>
    <td>215</td>
  </tr>
  <tr>
    <td>Unique Words</td>
    <td></td>
    <td>24</td>
    <td>866</td>
    <td>171</td>
    <td>38</td>
    <td>1113</td>
    <td>191</td>
    <td>39</td>
    <td>820</td>
    <td>155</td>
  </tr>
  <tr>
    <td>Sentences</td>
    <td></td>
    <td>2</td>
    <td>137</td>
    <td>20</td>
    <td>1</td>
    <td>245</td>
    <td>28</td>
    <td>2</td>
    <td>189</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Comments</td>
    <td></td>
    <td>1</td>
    <td>93</td>
    <td>6</td>
    <td>1</td>
    <td>103</td>
    <td>4</td>
    <td>1</td>
    <td>62</td>
    <td>5</td>
  </tr>
</table>

Table 2: Statistical Summary of Feature Counts across Various Indian Languages in the COSMMIC Dataset. Note that the average values have been rounded to the nearest integer.

<table>
  <tr>
    <th>Language</th>
    <th># Articles</th>
    <th># Comments</th>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>543</td>
    <td>3372</td>
  </tr>
  <tr>
    <td>Bengali</td>
    <td>627</td>
    <td>3742</td>
  </tr>
  <tr>
    <td>Marathi</td>
    <td>650</td>
    <td>3870</td>
  </tr>
  <tr>
    <td>Gujarati</td>
    <td>537</td>
    <td>1790</td>
  </tr>
  <tr>
    <td>Malayalam</td>
    <td>505</td>
    <td>2833</td>
  </tr>
  <tr>
    <td>Odia</td>
    <td>577</td>
    <td>1241</td>
  </tr>
  <tr>
    <td>Tamil</td>
    <td>505</td>
    <td>3204</td>
  </tr>
  <tr>
    <td>Telugu</td>
    <td>505</td>
    <td>1774</td>
  </tr>
  <tr>
    <td>Kannada</td>
    <td>510</td>
    <td>2658</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>4959</th>
    <th>24484</th>
  </tr>
</table>

Table 3: Distribution of Articles and Comments Across Indian Languages in the COSMMIC Dataset.

<table>
  <tr>
    <th>Attribute</th>
    <th colspan="2">Headlines</th>
    <th colspan="2">Summaries</th>
  </tr>
  <tr>
    <th>Language</th>
    <th># Words</th>
    <th># Sent.</th>
    <th># Words</th>
    <th># Sent.</th>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>18</td>
    <td>1</td>
    <td>117</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Bengali</td>
    <td>12</td>
    <td>1</td>
    <td>86</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Marathi</td>
    <td>13</td>
    <td>1</td>
    <td>110</td>
    <td>7</td>
  </tr>
  <tr>
    <td>Gujarati</td>
    <td>13</td>
    <td>1</td>
    <td>90</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Malayalam</td>
    <td>12</td>
    <td>1</td>
    <td>59</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Odia</td>
    <td>10</td>
    <td>1</td>
    <td>83</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Tamil</td>
    <td>11</td>
    <td>2</td>
    <td>77</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Telugu</td>
    <td>8</td>
    <td>1</td>
    <td>90</td>
    <td>9</td>
  </tr>
  <tr>
    <td>Kannada</td>
    <td>10</td>
    <td>1</td>
    <td>66</td>
    <td>5</td>
  </tr>
</table>

Table 4: Average Word and Sentence Counts for Headlines and Summaries (rounded to the nearest integer).

COSMMIC dataset for headline generation and summarization, utilizing both GPT-4 and LLAMA-3 models. The configurations included (1) using only the article text, (2) combining the article text with comments, (3) integrating the article text with relevant comments, (4) incorporating images along with the article text, (5) merging article text with both images and comments, and (6) combining article text, images, and relevant comments. These configurations were rigorously tested to assess how different data combinations influence the models’ performance in generating accurate and contextually rich headlines and summaries across the various languages.

Models Used: We employed GPT-4 (Achiam et al., 2023) and LLAMA-3 (Dubey et al., 2024), state-of-the-art NLP models. GPT-4, a paid multimodal model, excels in processing text and images, making it ideal for our dataset. LLAMA-3, an open-source alternative with 8 billion parameters, offers strong language generation capabilities. Recent studies Katz et al. (2024); Waisberg et al. (2023) highlight GPT-4’s superiority in complex tasks, while our evaluation of LLAMA-3 provides insights into the performance of a lighter, cost-effective model for reader-aware summarization and headline generation.
Evaluation Metrics: We assessed NLG tasks using ROUGE (Lin, 2004) for n-gram overlap and BERTScore (Zhang et al., 2019) for semantic similarity, ensuring precise n-gram segmentation in Indic languages with the Polyglot tokenizer (Al-Rfou’ et al., 2013). While these metrics may appear limited, they are the standard for state-of-the-art datasets (Table 1) and provide a rigorous benchmark. Their widespread adoption reinforces their reliability, making them well-suited for our dataset evaluation.

4.2 Experiments, Results and Analysis

Key Prompting Strategies To enhance headline generation and summarization, we employ structured zero-shot prompts tailored for different multi-modal scenarios. For headline generation, prompts ensure factual, concise, and engaging outputs, whether integrating images with article text or incorporating user comments as supplementary context. Similarly, summarization prompts maintain linguistic consistency while prioritizing core information, ensuring balanced and insightful summaries. Complete prompt structures and rationale are provided in the appendix A.2.

Analysis of Summarization with GPT-4
Table 5 shows that “Article + Images” generally provides the best summarization results across most Indian languages, improving both ROUGE and BERTScore metrics. Images offer valuable contextual support, especially in languages with complex scripts and structures, enhancing the understanding and interpretation of content. However, Telugu is an exception where “Article + Images + Rel. Com.” yields the highest scores, highlighting the importance of integrating relevant comments for improved summarization. This variation underscores the need for tailored summarization strategies that account for each language’s unique linguistic and contextual features. The table further shows that incorporating relevant comments enhances summarization quality, as reflected in improved ROUGE and BERTScore metrics across most languages. This reinforces the effectiveness of reader-aware summarization, where comments provide crucial context and sentiment, leading to more accurate summaries.

Analysis of Headline Generation with GPT-4
While article-only inputs yield strong headlines, incorporating relevant comments enhances performance in certain languages by providing crucial context. Languages like Telugu and Kannada benefit significantly, demonstrating that comments help bridge informational gaps and refine headlines. The minimal drop in BERTScore suggests that while images may not improve exact textual matches, they contribute valuable contextual insights. Overall, GPT-4 effectively optimizes NLG tasks across languages with diverse inputs.

Summarization with LLama-3
Table 6 shows that “Only Article” input generally yields the best summarization scores with LLAMA-3. However, incorporating relevant comments significantly boosts performance for languages like Tamil and Telugu, as seen in improved BERTScore metrics. For Bengali and Kannada, including all comments enhances contextual richness, leading to superior BERTScore results. This demonstrates that while minimal input is effective, relevant comments improve summary quality, even for lightweight models.

Headline Generation with LLama-3
While “Only Article” inputs typically perform best for headline generation, Table 6 shows that adding all comments improves BERTScore across languages. This suggests that while simpler inputs work well, richer contextual data enhances semantic relevance and alignment. Thus, LLama-3 benefits from minimal inputs but achieves better headline quality with comments. Human analysis is included in the appendix A.7 due to space constraints.

5 Classifying Comments & Images
As an initial benchmark, we focus on the Hindi section of our dataset due to annotation costs and feasibility. Tasks below aim to enhance data quality and assess the impact of linguistic transformations.

Comment Classification: To refine our dataset, we propose an automated approach to retain only informative comments while filtering out noisy ones in the Hindi section. Using IndicBERT (Kakwani et al., 2020), we develop a classifier to enhance the quality of reader-aware summarization. For training, our annotator manually labelled 250 comments into three categories: Supporting (aligned with the article), Enriching (adding new insights), and Disconnected (irrelevant to the article). We fine-tuned IndicBERT on this labelled data and then applied the trained model to classify the remaining comments. Once categorized, we conducted further experiments to analyze how comment clas-
<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="4">Only Article</th>
    <th colspan="4">Article + Comments</th>
    <th colspan="4">Article + Rel. Com.</th>
  </tr>
  <tr>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
  </tr>
  <tr><td>Hin.</td><td>44.18</td><td>18.94</td><td>33.47</td><td>76.61</td><td>50.98</td><td>24.82</td><td>38.34</td><td>78.22</td><td>52.13</td><td>25.52</td><td>39.43</td><td>78.71</td></tr>
  <tr><td>Ben.</td><td>29.40</td><td>9.32</td><td>21.52</td><td>74.69</td><td>36.64</td><td>13.38</td><td>26.15</td><td>76.94</td><td>37.19</td><td>13.73</td><td>26.82</td><td>77.34</td></tr>
  <tr><td>Mar.</td><td>27.24</td><td>10.49</td><td>25.79</td><td>74.28</td><td>40.38</td><td>18.56</td><td>37.97</td><td>78.02</td><td>41.50</td><td>19.47</td><td>39.03</td><td>78.56</td></tr>
  <tr><td>Guj.</td><td>26.97</td><td>8.38</td><td>25.13</td><td>75.00</td><td>32.00</td><td>12.00</td><td>29.46</td><td>76.26</td><td>32.89</td><td>12.41</td><td>30.29</td><td>76.63</td></tr>
  <tr><td>Mal.</td><td>17.38</td><td>5.45</td><td>16.55</td><td>72.78</td><td>18.88</td><td>6.16</td><td>17.90</td><td>73.49</td><td>20.18</td><td>6.71</td><td>19.11</td><td>74.01</td></tr>
  <tr><td>Odia.</td><td>25.45</td><td>7.98</td><td>18.90</td><td>92.47</td><td>34.99</td><td>13.73</td><td>26.47</td><td>92.91</td><td>36.36</td><td>14.11</td><td>27.68</td><td>92.83</td></tr>
  <tr><td>Tam.</td><td>25.07</td><td>9.52</td><td>23.60</td><td>74.05</td><td>32.63</td><td>14.32</td><td>30.67</td><td>77.20</td><td>34.94</td><td>15.52</td><td>32.73</td><td>77.83</td></tr>
  <tr><td>Tel.</td><td>21.91</td><td>6.83</td><td>20.79</td><td>73.05</td><td>34.32</td><td>13.89</td><td>32.35</td><td>76.80</td><td>36.19</td><td>15.07</td><td>33.93</td><td>77.75</td></tr>
  <tr><td>Kan.</td><td>23.94</td><td>8.04</td><td>22.70</td><td>74.83</td><td>28.85</td><td>10.68</td><td>27.08</td><td>75.89</td><td>29.10</td><td>10.79</td><td>28.01</td><td>76.01</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="4">Article + Images</th>
    <th colspan="4">Article + Images + Comments</th>
    <th colspan="4">Article + Images + Rel. Com.</th>
  </tr>
  <tr>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
  </tr>
  <tr><td>Hin.</td><td>53.41</td><td>26.10</td><td>40.41</td><td>79.38</td><td>49.63</td><td>23.67</td><td>36.66</td><td>77.69</td><td>50.90</td><td>24.42</td><td>37.64</td><td>78.15</td></tr>
  <tr><td>Ben.</td><td>38.16</td><td>14.16</td><td>27.62</td><td>77.87</td><td>34.90</td><td>12.53</td><td>24.34</td><td>76.49</td><td>36.12</td><td>13.20</td><td>25.63</td><td>76.89</td></tr>
  <tr><td>Mar.</td><td>42.83</td><td>20.11</td><td>40.38</td><td>79.03</td><td>39.50</td><td>18.02</td><td>37.43</td><td>77.85</td><td>40.62</td><td>18.57</td><td>38.27</td><td>78.28</td></tr>
  <tr><td>Guj.</td><td>33.74</td><td>12.60</td><td>31.15</td><td>77.20</td><td>31.06</td><td>11.15</td><td>28.66</td><td>76.17</td><td>31.78</td><td>11.52</td><td>29.38</td><td>76.30</td></tr>
  <tr><td>Mal.</td><td>20.27</td><td>6.53</td><td>19.22</td><td>74.15</td><td>18.47</td><td>5.73</td><td>17.60</td><td>73.64</td><td>19.00</td><td>5.90</td><td>18.04</td><td>73.92</td></tr>
  <tr><td>Odia.</td><td>36.31</td><td>14.22</td><td>27.81</td><td>93.40</td><td>33.06</td><td>12.36</td><td>24.59</td><td>91.86</td><td>34.60</td><td>13.21</td><td>25.82</td><td>92.29</td></tr>
  <tr><td>Tam.</td><td>35.87</td><td>15.94</td><td>33.78</td><td>78.32</td><td>31.85</td><td>13.34</td><td>29.89</td><td>76.83</td><td>32.96</td><td>14.05</td><td>31.01</td><td>77.28</td></tr>
  <tr><td>Tel.</td><td>35.33</td><td>14.03</td><td>33.17</td><td>77.54</td><td>32.61</td><td>12.51</td><td>30.61</td><td>76.52</td><td>33.45</td><td>13.15</td><td>31.56</td><td>76.82</td></tr>
  <tr><td>Kan.</td><td>30.15</td><td>11.12</td><td>28.28</td><td>76.55</td><td>27.81</td><td>9.89</td><td>26.28</td><td>75.43</td><td>28.60</td><td>10.50</td><td>26.88</td><td>75.68</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="4">Only Article</th>
    <th colspan="4">Article + Comments</th>
    <th colspan="4">Article + Rel. Com.</th>
  </tr>
  <tr>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
  </tr>
  <tr><td>Hin.</td><td>32.23</td><td>9.58</td><td>24.54</td><td>72.97</td><td>30.27</td><td>8.99</td><td>23.61</td><td>72.89</td><td>30.49</td><td>9.12</td><td>23.94</td><td>72.87</td></tr>
  <tr><td>Ben.</td><td>22.07</td><td>4.96</td><td>17.27</td><td>72.63</td><td>21.01</td><td>4.80</td><td>17.14</td><td>72.66</td><td>21.27</td><td>5.14</td><td>17.28</td><td>72.72</td></tr>
  <tr><td>Mar.</td><td>21.18</td><td>6.51</td><td>18.33</td><td>72.83</td><td>20.19</td><td>5.46</td><td>17.47</td><td>72.63</td><td>20.01</td><td>5.51</td><td>17.48</td><td>72.63</td></tr>
  <tr><td>Guj.</td><td>20.47</td><td>5.37</td><td>17.49</td><td>73.67</td><td>19.81</td><td>4.86</td><td>17.13</td><td>73.89</td><td>20.10</td><td>5.26</td><td>17.42</td><td>73.84</td></tr>
  <tr><td>Mal.</td><td>20.41</td><td>5.53</td><td>17.48</td><td>72.19</td><td>18.99</td><td>4.87</td><td>16.38</td><td>71.75</td><td>19.19</td><td>4.89</td><td>16.09</td><td>71.87</td></tr>
  <tr><td>Odia.</td><td>18.23</td><td>4.80</td><td>15.55</td><td>90.03</td><td>19.09</td><td>4.83</td><td>16.14</td><td>91.35</td><td>19.38</td><td>4.74</td><td>16.32</td><td>91.27</td></tr>
  <tr><td>Tam.</td><td>24.63</td><td>6.66</td><td>22.13</td><td>72.59</td><td>22.03</td><td>6.45</td><td>19.77</td><td>72.37</td><td>22.71</td><td>6.62</td><td>20.45</td><td>72.55</td></tr>
  <tr><td>Tel.</td><td>18.82</td><td>4.51</td><td>16.33</td><td>70.61</td><td>17.97</td><td>4.56</td><td>15.90</td><td>70.63</td><td>18.11</td><td>4.55</td><td>16.10</td><td>70.86</td></tr>
  <tr><td>Kan.</td><td>20.88</td><td>5.04</td><td>17.97</td><td>73.22</td><td>22.12</td><td>6.32</td><td>19.16</td><td>74.07</td><td>22.87</td><td>6.54</td><td>19.78</td><td>74.12</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="4">Article + Images</th>
    <th colspan="4">Article + Images + Comments</th>
    <th colspan="4">Article + Images + Rel. Com.</th>
  </tr>
  <tr>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
  </tr>
  <tr><td>Hin.</td><td>28.56</td><td>8.31</td><td>22.44</td><td>72.34</td><td>29.14</td><td>8.53</td><td>22.88</td><td>72.29</td><td>28.69</td><td>8.37</td><td>22.64</td><td>72.38</td></tr>
  <tr><td>Ben.</td><td>18.21</td><td>3.66</td><td>14.71</td><td>72.24</td><td>18.67</td><td>3.63</td><td>14.85</td><td>72.11</td><td>19.45</td><td>4.04</td><td>15.81</td><td>72.22</td></tr>
  <tr><td>Mar.</td><td>18.18</td><td>4.49</td><td>15.59</td><td>72.26</td><td>19.03</td><td>5.02</td><td>16.34</td><td>72.25</td><td>19.07</td><td>4.67</td><td>16.45</td><td>72.30</td></tr>
  <tr><td>Guj.</td><td>15.98</td><td>3.52</td><td>13.83</td><td>72.51</td><td>16.42</td><td>3.43</td><td>14.25</td><td>72.47</td><td>16.86</td><td>3.56</td><td>14.51</td><td>72.65</td></tr>
  <tr><td>Mal.</td><td>14.80</td><td>2.78</td><td>12.92</td><td>70.40</td><td>15.18</td><td>3.45</td><td>13.02</td><td>70.37</td><td>15.41</td><td>3.53</td><td>13.25</td><td>70.43</td></tr>
  <tr><td>Odia.</td><td>15.89</td><td>3.43</td><td>13.32</td><td>90.11</td><td>16.76</td><td>3.84</td><td>14.18</td><td>90.23</td><td>16.94</td><td>3.89</td><td>14.53</td><td>90.69</td></tr>
  <tr><td>Tam.</td><td>18.93</td><td>4.57</td><td>17.18</td><td>71.81</td><td>20.03</td><td>4.36</td><td>17.99</td><td>71.56</td><td>20.31</td><td>4.74</td><td>18.42</td><td>71.80</td></tr>
  <tr><td>Tel.</td><td>15.44</td><td>3.31</td><td>14.01</td><td>70.28</td><td>16.96</td><td>3.84</td><td>15.21</td><td>70.46</td><td>17.85</td><td>4.36</td><td>16.15</td><td>70.64</td></tr>
  <tr><td>Kan.</td><td>19.46</td><td>4.35</td><td>16.95</td><td>72.92</td><td>19.27</td><td>4.63</td><td>17.04</td><td>73.08</td><td>19.62</td><td>5.05</td><td>17.37</td><td>73.28</td></tr>
</table>

Table 5: Performance of Summarization and Headline Generation using GPT-4 across different languages and input combinations. The table shows ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL), and BERTScore (BS) metrics. Bold values indicate the highest performance metrics achieved for each language and input type, highlighting the most effective combination.

sification improves summarization performance, as shown in Table 7.

Insight: The results in Table 7 indicate that integrating enriching comments (OA+EC) yields the highest improvement in summarization. Interestingly, supporting comments (OA+SC) also enhance performance but slightly less than enriching ones. Including disconnected comments (OA+DC)
<table>
  <tr>
    <th colspan="12">Reader Aware Summarization with Llama-3</th>
  </tr>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="4">Only Article</th>
    <th colspan="4">Article + Comments</th>
    <th colspan="4">Article + Rel. Com.</th>
  </tr>
  <tr>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
  </tr>
  <tr>
    <td>Hin.</td><td>36.49</td><td>14.88</td><td>34.97</td><td>84.45</td><td>31.27</td><td>11.95</td><td>29.89</td><td>83.63</td><td>34.35</td><td>12.76</td><td>33.13</td><td>84.26</td>
  </tr>
  <tr>
    <td>Ben.</td><td>2.97</td><td>1.04</td><td>2.91</td><td>82.10</td><td>2.98</td><td>1.14</td><td>2.89</td><td>82.12</td><td>2.64</td><td>0.76</td><td>2.64</td><td>82.08</td>
  </tr>
  <tr>
    <td>Mar.</td><td>19.96</td><td>8.36</td><td>19.41</td><td>85.29</td><td>18.13</td><td>4.19</td><td>11.09</td><td>79.38</td><td>20.05</td><td>8.01</td><td>19.39</td><td>85.21</td>
  </tr>
  <tr>
    <td>Guj.</td><td>29.71</td><td>11.54</td><td>28.70</td><td>93.48</td><td>29.44</td><td>11.53</td><td>28.45</td><td>93.69</td><td>27.89</td><td>9.82</td><td>26.92</td><td>93.46</td>
  </tr>
  <tr>
    <td>Mal.</td><td>22.04</td><td>7.60</td><td>21.65</td><td>93.60</td><td>20.60</td><td>6.66</td><td>20.10</td><td>93.38</td><td>20.76</td><td>7.36</td><td>20.28</td><td>93.50</td>
  </tr>
  <tr>
    <td>Odia.</td><td>1.96</td><td>0.96</td><td>1.96</td><td>92.18</td><td>2.03</td><td>0.70</td><td>2.03</td><td>91.77</td><td>0.12</td><td>0.03</td><td>0.12</td><td>91.50</td>
  </tr>
  <tr>
    <td>Tam.</td><td>33.61</td><td>12.35</td><td>32.51</td><td>86.91</td><td>32.41</td><td>13.11</td><td>31.23</td><td>86.93</td><td>34.01</td><td>13.72</td><td>33.19</td><td>87.01</td>
  </tr>
  <tr>
    <td>Tel.</td><td>26.21</td><td>11.37</td><td>25.49</td><td>92.01</td><td>22.11</td><td>10.3</td><td>21.59</td><td>91.89</td><td>21.94</td><td>9.88</td><td>21.43</td><td>92.25</td>
  </tr>
  <tr>
    <td>Kan.</td><td>25.34</td><td>8.81</td><td>24.36</td><td>93.29</td><td>23.98</td><td>8.71</td><td>23.47</td><td>93.46</td><td>23.57</td><td>8.36</td><td>22.86</td><td>93.38</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="12">Reader Aware Headline Generation with Llama-3</th>
  </tr>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="4">Only Article</th>
    <th colspan="4">Article + Comments</th>
    <th colspan="4">Article + Rel. Com.</th>
  </tr>
  <tr>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
    <th>R1</th><th>R2</th><th>RL</th><th>BS</th>
  </tr>
  <tr>
    <td>Hin.</td><td>7.76</td><td>0.86</td><td>7.59</td><td>76.60</td><td>5.29</td><td>0.42</td><td>5.19</td><td>76.35</td><td>4.20</td><td>0.33</td><td>4.08</td><td>76.11</td>
  </tr>
  <tr>
    <td>Ben.</td><td>0.59</td><td>0.23</td><td>0.59</td><td>73.51</td><td>0.58</td><td>0.16</td><td>0.58</td><td>72.87</td><td>0.25</td><td>0.07</td><td>0.25</td><td>72.50</td>
  </tr>
  <tr>
    <td>Mar.</td><td>4.25</td><td>0.41</td><td>4.15</td><td>76.03</td><td>2.97</td><td>0.15</td><td>2.89</td><td>75.47</td><td>2.57</td><td>0.23</td><td>2.57</td><td>75.03</td>
  </tr>
  <tr>
    <td>Guj.</td><td>4.57</td><td>1.08</td><td>4.51</td><td>82.94</td><td>4.56</td><td>0.91</td><td>4.50</td><td>82.95</td><td>4.09</td><td>0.59</td><td>4.09</td><td>82.49</td>
  </tr>
  <tr>
    <td>Mal.</td><td>3.56</td><td>0.43</td><td>3.56</td><td>81.89</td><td>2.79</td><td>0.20</td><td>2.79</td><td>82.34</td><td>2.62</td><td>0.06</td><td>2.52</td><td>76.31</td>
  </tr>
  <tr>
    <td>Odia.</td><td>0.76</td><td>0.23</td><td>0.76</td><td>82.63</td><td>0.65</td><td>0.03</td><td>0.65</td><td>82.91</td><td>0.62</td><td>0.24</td><td>0.62</td><td>81.93</td>
  </tr>
  <tr>
    <td>Tam.</td><td>5.34</td><td>0.99</td><td>5.26</td><td>75.85</td><td>6.36</td><td>0.63</td><td>6.36</td><td>75.88</td><td>4.24</td><td>0.34</td><td>4.14</td><td>75.71</td>
  </tr>
  <tr>
    <td>Tel.</td><td>3.94</td><td>0.25</td><td>3.84</td><td>75.40</td><td>3.73</td><td>0.25</td><td>3.63</td><td>75.84</td><td>2.62</td><td>0.06</td><td>2.52</td><td>76.31</td>
  </tr>
  <tr>
    <td>Kan.</td><td>4.83</td><td>0.76</td><td>4.83</td><td>79.37</td><td>3.36</td><td>0.47</td><td>3.36</td><td>79.96</td><td>0.13</td><td>0.04</td><td>0.13</td><td>79.20</td>
  </tr>
</table>

Table 6: Performance of Summarization and Headline Generation using LLAMA-3 across languages and input types, showing ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL), and BERTScore (BS) metrics. Bold values highlight the best results.

significantly deteriorates performance, highlighting the importance of filtering relevant contextual information. This trend continues for headline generation, where OA+EC outperforms other configurations, reinforcing the benefit of additional insights over mere agreement. A detailed discussion is furthermore provided in the appendix A.1 due to space constraints.

Image Classification: We use a threshold-based classifier with multilingual CLIP (sentence-transformers/clip-ViT-B-32-multilingual-v1) (Carlsson et al., 2022; Reimers and Gurevych, 2019) to assess image relevance in multimodal summarization (Jangra et al., 2021). Since images accompany articles from a reputable media house, classification into unrelated categories is unnecessary. Instead, we classify images as supplementary (reinforcing the text) or complementary (providing additional context). CLIPScore is computed using both the ground truth summary and the image, as well as the headline and the image. The average score determines classification: above 0.25 indicates reinforcement, while lower values suggest added context.

The 0.25 threshold is empirically set based on CLIPScore distributions in our dataset, ensuring balanced classification. Prior work Jangra et al. (2021) shows real-world text-image pairs often have low CLIPScores, making higher thresholds (e.g., 0.5) impractical. This approach effectively differentiates between reinforcing and contextual images. We observed that 36.84% of images reinforced the article (supplementary), while most (63.16%) added extra information (complementary). This suggests that Hindi news articles are often accompanied by images that provide background information rather than directly supporting the text. As a result, multimodal summarization models must focus on interpreting contextual visuals rather than relying solely on direct text-image alignment (further strengthening the findings of Table 5). Likewise, headline generation may gain from complementary images for a broader context beyond the main points.

Remark: For a deeper understanding, additional discussion and a detailed back-translation study focusing on preserving linguistic authenticity are provided in the appendix (see A.9 and A.10). These sections explore the nuances of language preserva-
<table>
  <tr>
    <th>Config</th>
    <th>R1</th>
    <th>R2</th>
    <th>RL</th>
    <th>BS</th>
    <th>Config</th>
    <th>R1</th>
    <th>R2</th>
    <th>RL</th>
    <th>BS</th>
  </tr>
  <tr>
    <td>OA</td>
    <td>44.18</td>
    <td>18.94</td>
    <td>33.47</td>
    <td>76.61</td>
    <td>OA</td>
    <td>32.23</td>
    <td>9.58</td>
    <td>24.54</td>
    <td>72.97</td>
  </tr>
  <tr>
    <td>OA+SC</td>
    <td>56.64</td>
    <td>28.33</td>
    <td>42.59</td>
    <td>82.32</td>
    <td>OA+SC</td>
    <td>31.68</td>
    <td>8.98</td>
    <td>24.02</td>
    <td>71.53</td>
  </tr>
  <tr>
    <td>OA+EC</td>
    <td><b>58.22</b></td>
    <td><b>29.19</b></td>
    <td><b>43.93</b></td>
    <td><b>84.15</b></td>
    <td>OA+EC</td>
    <td>32.06</td>
    <td>9.52</td>
    <td>23.89</td>
    <td><b>73.44</b></td>
  </tr>
  <tr>
    <td>OA+DC</td>
    <td>35.62</td>
    <td>20.44</td>
    <td>29.85</td>
    <td>70.44</td>
    <td>OA+DC</td>
    <td>26.54</td>
    <td>7.11</td>
    <td>20.48</td>
    <td>69.85</td>
  </tr>
  <tr>
    <td>OA+AC</td>
    <td>50.98</td>
    <td>24.82</td>
    <td>38.34</td>
    <td>78.22</td>
    <td>OA+AC</td>
    <td>30.27</td>
    <td>8.99</td>
    <td>23.61</td>
    <td>72.89</td>
  </tr>
  <tr>
    <td><b>OA+RC (GPT)</b></td>
    <td>52.13</td>
    <td>25.52</td>
    <td>39.43</td>
    <td>78.71</td>
    <td><b>OA+RC (GPT)</b></td>
    <td>30.49</td>
    <td>9.12</td>
    <td>23.94</td>
    <td>72.87</td>
  </tr>
</table>

Table 7: Performance of different configurations for Summarization (left) and Headline Generation (right) using GPT-4. Configurations: OA = Only Article, SC = Supporting Comments, EC = Enriching Comments, DC = Disconnected Comments, AC = All Comments, RC (GPT) = Relevant Comments selected by GPT.

tion and offer insights into the effectiveness of our approach.

6 Conclusion

In summary, the COSMMIC dataset is a notable advancement in multimodal, multilingual, and comment-sensitive resources for Indian languages. It integrates text, images, and reader comments across nine major languages, addressing gaps in existing datasets. This dataset enhances headline generation and summarization while offering insights into regional language dynamics and user interactions. Our ablation study demonstrates the effectiveness of different component combinations, making COSMMIC a valuable resource.

Limitations

Although the dataset encompasses nine major Indian languages, it does not represent all regional dialects and languages within India. Also, the dataset’s reliance on publicly available sources from DailyHunt means that the coverage may be skewed towards specific topics or viewpoints prevalent on the platform.

Ethics Statement Our dataset, including URLs to original articles and images, content, ground truth summaries, and headlines, is publicly available on GitHub: https://github.com/AaryanSahu/COSMMIC, along with the scraping and experimental code. The scraping process deliberately excludes any user information to protect privacy.

Acknowledgments

Raghvendra Kumar extends appreciation to the Prime Minister’s Research Fellowship (PMRF) for its assistance and successful completion of this research. Sriparna Saha recognizes the support received from the “Technology Innovation Hub, Vishlesan I-Hub Foundation IIT Patna” (Project No. TIH/CSE/ASMO/05) for the achievement of this research. Aryan Sahu sincerely thanks the India Chapter of ACM SIGKDD for the opportunity to participate in the Uplink Internship program, which played a pivotal role in the successful completion of this research work.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, et al. 2023. Mega: Multilingual evaluation of generative ai. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267.

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, et al. 2024. Megaverse: Benchmarking large language models across languages, modalities, models and tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2598–2637.

Rami Al-Rfou’, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria. Association for Computational Linguistics.

Rahul Aralikatte, Ziling Cheng, Sumanth Doddapaneni, and Jackie Chi Kit Cheung. 2023. Varta: A large-scale headline-generation dataset for indic languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3468–3492.

Savita Bhat, Vasudeva Varma, and Niranjan Pedanekar. 2023. Generative models for Indic languages: Evaluating content generation capabilities. In Proceed-
ings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 187–195, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Yue Cao, Xiaojun Wan, Jinge Yao, and Dian Yu. 2020. Multisumm: towards a unified model for multi-lingual abstractive summarization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11–18.

Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. 2022. Cross-lingual and multilingual clip. In Proceedings of the thirteenth language resources and evaluation conference, pages 6848–6854.

Abhimanyu Dubey, Abhinav Jahuri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, and Richard Socher. 2021. Covid-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization. NPJ digital medicine, 4(1):68.

Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2022. Msamsum: Towards benchmarking multi-lingual dialogue summarization. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 1–12.

Xiyan Fu, Jun Wang, and Zhenglu Yang. 2021. MM-AVS: A full-scale dataset for multi-modal summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5922–5926. Online. Association for Computational Linguistics.

Soichiro Fujita, Hayato Kobayashi, and Manabu Okumura. 2019. Dataset creation for ranking constructive news comments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2619–2626, Florence, Italy. Association for Computational Linguistics.

Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. 2023. Targeting the source: Selective data curation for debiasing nlp models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 276–294. Springer.

Shen Gao, Xiuying Chen, Piji Li, Zhaochun Ren, Lidong Bing, Dongyan Zhao, and Rui Yan. 2019. Abstractive text summarization by incorporating reader comments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6399–6406.

Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha, and Setu Sinha. 2024. Clipsyntel: clip and llm synergy for multimodal question summarization in healthcare. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22031–22039.

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79.

Abhay Gotmare, Gandharva Thite, and Laxmi Bewoor. 2024. A multimodal machine learning approach to generate news articles from geo-tagged images. International Journal of Electrical & Computer Engineering (2088-8708), 14(3).

Felix Hamborg, Norman Meuschke, and Bela Gipp. 2017. Matrix-based news aggregation: exploring different news perspectives. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1–10. IEEE.

Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubashshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. XI-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703.

Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shrivastava, and Zhaowen Wang. 2023. Align and attend: Multimodal summarization with dual contrastive losses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14867–14878.

Kushan Hewapathirana, Nisansa de Silva, and CD Athuraliya. 2024. M2ds: Multilingual dataset for multi-document summarisation. arXiv preprint arXiv:2407.12336.

Anubhav Jangra, Sriparna Saha, Adam Jatowt, and Mohammed Hasanuzzaman. 2021. Multi-modal supplementary-complementary summarization using multi-objective optimization. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 818–828.

Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, NC Gokul, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020. Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian
languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961.

Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254.

Harsh Khatter and Anil Kumar Ahlawat. 2022. Content curation algorithm on blog posts using hybrid computing. Multimedia Tools and Applications, 81(6):7589–7609.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Mateusz Krubiński and Pavel Pecina. 2024a. Towards unified uni- and multi-modal news headline generation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 437–450. St. Julian’s, Malta. Association for Computational Linguistics.

Mateusz Krubiński and Pavel Pecina. 2024b. Towards unified uni-and multi-modal news headline generation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 437–450.

Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar. 2022. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5363–5394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Raghvendra Kumar, Ratul Chakraborty, Abhishek Tiwari, Sriparna Saha, and Naveen Saini. 2023. Diving into a sea of opinions: Multi-modal abstractive summarization with comment sensitivity. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 1117–1126, New York, NY, USA. Association for Computing Machinery.

Ritesh Kumar, Shyam Ratan, Siddharth Singh, Enakshi Nandi, Laishram Niranjana Devi, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Akanksha Bansal. 2024. A multilingual, multimodal dataset of aggression and bias: the comma dataset. Language Resources and Evaluation, 58(2):757–837.

Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen Mckeown. 2020. Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048.

Piji Li, Lidong Bing, and Wai Lam. 2017. Reader-aware multi-document summarization: An enhanced model and the first dataset. In Proceedings of the Workshop on New Frontiers in Summarization, pages 91–99.

Yunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang, Yufeng Chen, and Jie Zhou. 2023. Summary-oriented vision modeling for multimodal abstractive summarization. In The 61st Annual Meeting Of The Association For Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

P Mahalakshmi and N Sabiyath Fatima. 2022. Summarization of text and image captioning in information retrieval using deep learning techniques. IEEE Access, 10:18289–18297.

Zixiang Meng, Qiang Gao, Di Guo, Yunlong Li, Bobo Li, Hao Fei, Shengqiong Wu, Fei Li, Chong Teng, and Donghong Ji. 2024. Mmlscu: A dataset for multi-modal multi-domain live streaming comment understanding. In Proceedings of the ACM on Web Conference 2024, WWW ’24, page 4395–4406, New York, NY, USA. Association for Computing Machinery.

Masato Mita, Soichiro Murakami, Akihiko Kato, and Peinan Zhang. 2023. Camera: A multimodal dataset and benchmark for ad text generation. arXiv preprint arXiv:2309.12030.

Laura Nguyen, Thomas Scialom, Benjamin Piwowarski, and Jacopo Staiano. 2023. Loralay: A multilingual and multimodal dataset for long range and layout-aware summarization. In The 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023).

Vaidehi Patil, Leonardo Ribeiro, Mengwen Liu, Mohit Bansal, and Markus Dreyer. 2024. Refinesumm: Self-refining mllm for generating a multimodal summarization dataset. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13773–13786.

Lingfeng Qiao, Chen Wu, Ye Liu, Haoyuan Peng, Di Yin, and Bo Ren. 2022. Grafting pre-trained models for multimodal headline generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 244–253.

Jielin Qiu, Jiacheng Zhu, William Han, Aditesh Kumar, Karthik Mittal, Claire Jin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Ding Zhao, et al. 2024. Mmsum: A dataset for multimodal summarization and thumbnail generation of videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21909–21921.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Julian Risch and Ralf Krestel. 2020. A dataset of journalists’ interactions with their readership: When should article authors reply to reader comments? In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. CIKM ’20, page 3117–3124, New York, NY, USA. Association for Computing Machinery.

Vishal Singh Roha, Naveen Saini, Sriparna Saha, and Jose G Moreno. 2022. Unsupervised framework for comment-based multi-document extractive summarization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 574–582.

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MI-sum: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067.

Bei Shi and Wai Lam. 2019. Reader comment digest through latent event facets and news specificity. IEEE Transactions on Knowledge and Data Engineering, 31(8):1581–1594.

Faisal Tareque Shohan, Mir Tarfeer Nayem, Samsul Islam, Abu Ubaida Akash, and Shafiq Joty. 2024. Xl-headtags: Leveraging multimodal retrieval augmentation for the multilingual generation of news headlines and tags. arXiv preprint arXiv:2406.03776.

Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. Indicgen-bench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. arXiv preprint arXiv:2404.16816.

Rajeev Kumar Singh, Sonia Khetarpaul, Rohan Gorantla, and Sai Giridhar Allada. 2021. Sheg: summarization and headline generation of news articles using deep learning. Neural Computing and Applications, 33:3251–3265.

Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. From neural sentence summarization to headline generation: A coarse-to-fine approach. In IJCAI, volume 17, pages 4109–4115.

Ashok Uralna, Pinzhen Chen, Zheng Zhao, Shay Cohen, Manish Shrivastava, and Barry Haddow. 2023. PMIndiaSum: Multilingual and cross-lingual headline summarization for languages in India. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11606–11628, Singapore. Association for Computational Linguistics.

Daniel Varab and Natalie Schluter. 2021. Massivesumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150–10161.

Prince Verma, Kiran Jyoti, Varsha Sahni, S Jafar Ali Ibrahim, and NS Kalyan Chakravarthy. 2023a. Synthesized feature learning model on news aggregator for chatbot. Soft Computing, pages 1–14.

Yash Verma, Anubhav Jangra, Raghvendra Kumar, and Sriparna Saha. 2023b. Large scale multi-lingual multi-modal summarization dataset. arXiv preprint arXiv:2302.06560.

Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, Sharif Amit Kamran, Nasif Zaman, Prithul Sarker, Andrew G Lee, and Alireza Tavakkoli. 2023. Gpt-4: a new era of artificial intelligence in medicine. Irish Journal of Medical Science (1971-), 192(6):3197–3200.

Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. 2022. N24News: A new dataset for multimodal news classification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6768–6775, Marseille, France. European Language Resources Association.

Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen Mckeown, and Tatsunori B Hashimoto. 2024. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 11:39–57.

Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Ji-ajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal summarization with multimodal output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4154–4164, Brussels, Belgium. Association for Computational Linguistics.

A Appendix

A.1 Detailed Analysis of Comment Classification Impact

The results in Table 7 provide key insights into how different types of comments influence summariza-
tion and headline generation quality. Below, we analyze these findings in detail.

A.1.1 Impact on Summarization

1. Only Article (OA) serves as the baseline, achieving an R1 score of 44.18 and a BERTScore (BS) of 76.61. This reflects the model’s ability to generate summaries without any external context.

2. Adding Supporting Comments (OA+SC) boosts performance significantly (R1: 56.64, BS: 82.32). These comments align with the article’s content and reinforce key information, leading to improved coherence and factual consistency.

3. Enriching Comments (OA+EC) yield the highest scores, achieving R1: 58.22 and BS: 84.15. Unlike supporting comments, these introduce new insights that enhance the richness of generated summaries.

4. Disconnected Comments (OA+DC) degrade performance dramatically (R1: 35.62, BS: 70.44). The presence of unrelated information confuses the model, resulting in incoherent or misleading summaries.

5. All Comments (OA+AC) moderately improve results (R1: 50.98, BS: 78.22) but underperform compared to selective filtering. This suggests that a mix of useful and irrelevant comments dilutes the benefits of contextual enrichment.

6. Relevant Comments (OA+RC) by GPT provide a slight edge over OA+AC (R1: 52.13, BS: 78.71), indicating that LLM-based filtering can help, but our classifier-based selection performs better in identifying enriching content.

A.1.2 Impact on Headline Generation

The trends observed in summarization largely hold for headline generation:

1. OA achieves R1: 32.23, BS: 72.97, serving as the baseline.

2. OA+EC provides the highest BERTScore (BS: 73.44), showing that adding new insights aids headline generation.

3. OA+SC performs slightly worse than OA+EC (BS: 71.53) but still improves over the baseline.

4. OA+DC leads to a major drop (BS: 69.85), reaffirming the negative impact of irrelevant content.

5. OA+AC and OA+RC show marginal improvements over the baseline, but the classifier-filtered enriching comments outperform all.

A.1.3 Key Takeaways

These findings demonstrate that not all comments are equally beneficial. Enriching comments contribute the most to both summarization and headline generation, while disconnected comments negatively impact performance. Automated classification, as performed in this study, proves essential for optimizing results.

A.2 Detailed Prompting Strategies

Prompts Used: We present the zero-shot prompts used in two headline generation scenarios: (i) integrating images with article text and (ii) incorporating user comments alongside the article, as illustrated in Figure 2. The first prompt ensures that headlines effectively capture key textual and visual content, maintaining clarity, conciseness, and factual accuracy. The second prompt integrates relevant user comments while preserving the article’s primary message, subtly reflecting public sentiment without distorting factual integrity.

For summarization, the prompt shown in Figure 3 ensures concise and coherent summaries by distilling essential information. When incorporating images and comments, the prompt balances factual accuracy with supplementary insights from comments, maintaining a focus on the article’s core message. All prompts follow expert guidelines to optimize headline generation and summarization.

Figures 2 and 3 illustrate these prompt designs in detail.

A.3 Summarization using GPT

Utilization of Comments and Images for summarization:

In the provided example summary as shown in figure 4 generated using GPT-4-turbo, the first paragraph primarily draws information from the article. In contrast, the second paragraph incorporates details about the weather in Florida impacting the Pakistan match, which can be inferred from the accompanying image. This highlights the context and background information effectively. The final paragraph captures user opinions from comments expressed in Hindi, suggesting that teams should be awarded total points if the match is cancelled, thereby adding valuable insights. These additional elements significantly enhance the overall quality of the summary.
<table>
  <tr>
    <th>Article + Images</th>
    <th>Article + Relevant Comments</th>
  </tr>
  <tr>
    <td>
      "role": "system",
      "content": "You are an expert news editor. Your primary task is to create headlines that accurately reflect the main content of news articles and images."
      <br><br>
      "role": "user",
      "content": f"Craft a precise and engaging news headline that primarily captures the main point of the article and image. Follow these guidelines:
      <ol>
        <li>Focus predominantly (about 80-90%) on the key information and main message from the article and image.</li>
        <li>Keep the headline concise, ideally under 15 words.</li>
        <li>Use active voice and clear, impactful language.</li>
        <li>Ensure the headline is factual and directly related to the article's main content and the key elements of the image.</li>
      </ol>"
    </td>
    <td>
      "role": "system",
      "content": "You are an expert news editor. Your primary task is to create headlines that accurately reflect the main content of news articles. User comments should be considered as supplementary information only."
      <br><br>
      "role": "user",
      "content": f"Craft a precise and engaging news headline in {[]} that primarily summarizes the main point of the article. Use insights from user comments only as secondary, supplementary information. Follow these guidelines:
      <ol>
        <li>Focus predominantly (about 80-90%) on the key information and main message from the article.</li>
        <li>If relevant, subtly incorporate a minor element (about 10-20%) from the user comments to reflect public sentiment or add context.</li>
        <li>Keep the headline concise, ideally under 15 words.</li>
        <li>Use active voice and clear, impactful language.</li>
        <li>Ensure the headline is factual and directly related to the article's main content.</li>
        <li>Avoid overemphasizing or sensationalizing elements from user comments.</li>
      </ol>"
    </td>
  </tr>
</table>

Figure 2: Headline generation prompts: (a) integrating article text with images and (b) using relevant comments with the article text.

<table>
  <tr>
    <th>Only Article</th>
    <th>Article + Image + Comments</th>
  </tr>
  <tr>
    <td>
      "role": "system",
      "content": "Your task is to summarize the text in the same language as the text."
      <br><br>
      "role": "user",
      "content": f"Generate a summary using the contents. The summary should be concise and cover all the important key points as conveyed by the original content and should have minimal noise. The generated summary should be in the same language as the original content."
    </td>
    <td>
      "role": "system",
      "content": "You are an expert news editor. Your primary task is to create summaries that accurately reflect the main content of news articles and images. User comments should be considered as supplementary information only."
      <br><br>
      "role": "user",
      "content": f"Craft a precise and engaging news summary in {language} that primarily summarizes the main point of the article and image. Use insights from user comments only as secondary, supplementary information. Follow these guidelines:
      <ol>
        <li>Focus predominantly (about 80-90%) on the key information and main message from the article and image.</li>
        <li>If relevant, subtly incorporate a minor element (about 10-20%) from the user comments to reflect public sentiment or add context.</li>
        <li>Use active voice and clear, impactful language.</li>
        <li>Ensure the summary is factual and directly related to the article's main content and the key elements of the image.</li>
        <li>Avoid overemphasizing or sensationalizing elements from user comments.</li>
        <li>If an image is provided, balance the summary to reflect both textual and visual information when applicable.</li>
      </ol>"
    </td>
  </tr>
</table>

Figure 3: Summarization prompts: (a) only article text (b) using image and comments with the article text.
1. कुदरत के निजाम से नहीं बच पाया पाकिस्तान😂😂
2. बाधित मैच के रद होने पर दोनों टीमों को पूरे मार्क्स दिये जाने चाहिए, आपे आधे नहीं...ताकि सभी टीमों को आखिर तक खेलकर अपनी श्रेष्ठता साबित करने का अवसर मिल सके...
3. फट के हाथ में आ गई निजाम की

Summary
टी20 विश्व कप 2024 में पाकिस्तान की मुश्किलें बढ़ी जा रही हैं। पाकिस्तानी टीम ने मेजबान अमेरिका के खिलाफ अपने अभियान की शुरुआत की, जिसमें उन्हें सुपर ओवर में हार का सामना करना पड़ा। इसके बाद, भारत के खिलाफ भी जीतता हुआ मैच अंतिम समय में गंता दिया। हालांकि, कनाडा के खिलाफ एक जीत ने उनकी स्थिति को ढोड़ा मजबूत किया, लेकिन अब सुपर-8 में पहुँचने की उम्मीदें नाकुश हैं। ①
पाकिस्तान का अगला मैच ऑस्ट्रेलिया के खिलाफ है, लेकिन पहले ही यह तय होगा कि उनकी यात्रा समाप्त होगी या नहीं। यदि अमेरिका और ऑस्ट्रेलिया के बीच आगे रात का मैच बारिश के कारण रद हो जाता है, तो पाकिस्तान ग्रुप रेंज से बाहर हो जाएगा। इस बीच, फ्लोरिडा में बारिश की संभावनाएं, जहां यह मैच होना है, पाकिस्तान को और चिंता में डाल रही है। ②
सोशल मीडिया पर प्रशंसक इस स्थिति पर मजे ले रहे हैं, जबकि कई ने सजावट दिया है कि यदि मैच रद होता है, तो टीमों को पूरे अंक दिए जाने चाहिए। ③

Figure 4: Demonstrates the significance of comments and images in the summary generated by GPT-4-turbo

A.4 Challenges faced during Summarization:

1. During our summarization process, we observed minor inconsistencies in the language used in the generated summaries. Despite instructing the model to produce summaries in the original content language, 1-2% of summaries were still generated in English. This issue was comparatively pronounced with the LLAMA model, where not only did the number of English summaries slightly increase, but very few summaries were also generated in incorrect languages. To address this issue, we focused on iteratively rerunning the summarization process for only those articles with erroneous summaries through which we could correct the errors.

2. Another challenge we encountered with the LLAMA model was the generation of incomplete summaries. About 0.5-1% of the summaries were reduced to a single line, failing to capture the full essence of the original content. To mitigate this, we resorted to rerunning the summarization code for those rows with incomplete summaries.

3. Some LLaMA-generated summaries included words combining English and the original language, rather than using only the original language. This issue appears to stem from the lightweight nature of the LLaMA model, as no such problems were encountered when using GPT. Consequently, these mixed-language words in summaries led to noticeable errors, as highlighted in Figure 5.

The accurate summaries generated by the LLAMA model (Figure 5) can be compared with the GPT-4-turbo outputs (Figure 4) to recognize that the GPT-4-turbo summaries were more consistent and coherent and encountered fewer challenges while developing the summaries. This was expected as the LLAMA model was relatively lightweight compared to GPT-4-turbo.

A.5 Headline Generation using GPT
The hyperparameters used while generating the headline were:

• Max output token: 100
• Model used: gpt-4-turbo
• Temperature: 0.7
Figure 5: Summaries generated by the LLAMA model. Additionally, the good and erroneous summaries are compared

A.5.1 Challenges Faced During Headline Generation
• Several iterations of prompt refinement were done in order to come up with the best prompt.

A.5.2 Pre-processing of the Data Included:
1. Removal of redundant data: During the data collection process, redundant information such as the number of likes, shares, and other extraneous text displayed on the website was inadvertently scraped alongside the actual news articles. To ensure data quality, we meticulously processed the scraped content, filtering out these unnecessary elements to retain only the relevant news articles in the final dataset.

2. Removal of noise characters: During the scraping process, extraneous characters—likely resulting from the dynamic nature of the website—were occasionally captured alongside the actual news article content. To maintain the integrity of the dataset, we carefully removed these irrelevant characters, ensuring that only the authentic article content was retained.

3. Incomplete data: At times, only partial news articles were scraped—such as individual paragraphs or sentences—rather than the complete article. This issue arose due to irregular spacing and indentation on the website, combined with the limitations of Selenium in capturing full content under these conditions. To address this, we manually scraped the complete articles in such cases to ensure the dataset was comprehensive and accurate.

A.6 Headline Comparison
A.6.1 Headline (w/o comments,images) as shown in the Figure 6.

Original Headline: USA vs IRE-पाकिस्तान के खिलाफ कुश्ती का निर्णय, एक दौरे में 3 टीमों का काम नहीं
GPT Headline: टी20 वर्ल्ड कप 2024: पाकिस्तान गुप्त रेज़ में बाहर होने के कारण, बारिश और उत्पीड़न से हुई मुश्किलें |

Figure 6: Headline (w/o comments,images).

• Good Aspects: The GPT headline provides a relevant and clear update on Pakistan’s situation in the T20 World Cup, highlighting the impact of rain and upsets. It accurately reflects the severity of the situation.

• Bad Aspects: The GPT headline does not capture the specific match between the USA and Ireland or the broader dramatic impact on multiple teams as described in the original headline. It focuses more on the overall tournament context rather than the immediate consequences of the rain.

A.6.2 Headline with relevant comments as shown in the Figure 7.
• Good Aspects: The GPT headline provides a clear and relevant summary of the consequences of rain on a cricket match and its impact on Pakistan’s chances. It is accurate
Original Headline: USA vs IRE-पाकिस्तान के बीच टेस्ट क्रिकेट का निर्णय, एक दौर में 3 टीमों का काम तय
GPT Headline: बांग्ला के करण भारत और आयरलैंड का मैच प्रभावित हुआ, जिससे पाकिस्तान के सुपर-8 में पहुँचने की संभावना पर गई।

Figure 7: Headline with relevant comments.

in terms of the consequences but misses the original context.

• Bad Aspects: The GPT headline incorrectly names the teams involved, which is a significant deviation from the original headline. Additionally, it does not capture the broader, dramatic impact on multiple teams as described in the original.

A.6.3 Prompting Detail for Article and all comments inclusive:

After experimenting with several prompts to ensure accurate, robust summaries, the prompt given in Fig 8 below was finalized. The prompt has been crafted so that the model captures all the essential details of the article and the relevant user comments in the summary generated.

![Prompt for Article + Comments](page_154_629_563_246.png)

Figure 8: Prompts for Article + Comments

A.7 Human Evaluation

To complement automated evaluation metrics like ROUGE and BERTScore, we conducted human assessments with 10 participants (postgraduate students) to evaluate the quality of summaries produced using the COSMMIC dataset across different configurations for 10 articles in the Hindi language and their corresponding summaries. The evaluators rated the outputs based on key criteria, including grammatical accuracy (Gr-Co), relevancy, and overall summary quality (SQ), using a 1-to-5 scale. These evaluations encompassed both text-only and text-with-images scenarios, assessing the quality of summaries in two contexts: how effectively they conveyed information when paired with the corresponding article image, and how they performed when presented without any visual support. To minimize evaluator bias and enhance reliability, the scores were averaged. This thorough evaluation underscores the advantages of configurations that leverage richer input modalities, such as user comments and images. The evaluations were involved in summarization and headline generation for both the models as shown in Tables 8, 9, 10, 11.

A.8 Filtering and Considering Relevant Comments

Manual Filtering Process: To ensure that user comments included in the summaries enhanced content quality and relevance, a manual filtering process was employed, carried out by the same team of expert annotators responsible for summary creation. This process required an additional layer of effort and compensation, ensuring that only relevant comments were included while noisy or extraneous comments were excluded.

Defining Noisy Comments: Noisy comments were identified as those that detracted from the article’s central themes or introduced irrelevant, redundant, or biased information. Examples included:

• Spam-like entries: Comments promoting unrelated products, services, or containing external links.

• Generic phrases: Statements such as “Nice article” or “Interesting post,” which lacked substantive value.

• Hostile or inappropriate remarks: Overly aggressive or offensive comments not contributing meaningfully to the discussion.

• Unrelated tangents: Comments entirely disconnected from the topic being discussed.

Steps in Manual Filtering: The team of annotators undertook the following steps to manually filter comments:
<table>
  <tr>
    <th>Configuration</th>
    <th>Gr-Co</th>
    <th colspan="2">Relevance</th>
    <th colspan="2">SQ</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>WO-IM</th>
    <th>W-IM</th>
    <th>WO-IM</th>
    <th>W-IM</th>
  </tr>
  <tr>
    <td>Article Text Only</td>
    <td>3.5</td>
    <td>3.6</td>
    <td>3.4</td>
    <td>3.7</td>
    <td></td>
  </tr>
  <tr>
    <td>Article Text + Comments</td>
    <td>3.7</td>
    <td>3.9</td>
    <td>3.6</td>
    <td>3.9</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Relevant Comments</td>
    <td>3.8</td>
    <td>4.0</td>
    <td>3.8</td>
    <td>4.1</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Images</td>
    <td>3.9</td>
    <td>4.1</td>
    <td>4.0</td>
    <td>4.3</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Images + Comments</td>
    <td><b>4.0</b></td>
    <td><b>4.2</b></td>
    <td><b>4.1</b></td>
    <td><b>4.4</b></td>
    <td></td>
  </tr>
</table>

Table 8: Human evaluation scores for various metrics of GPT-based model using COSMMIC configurations. Gr-Co means grammatical correctness, SQ means summary quality (With Images (W-IM) and Without Images (WO-IM)).

<table>
  <tr>
    <th>Configuration</th>
    <th>Gr-Co</th>
    <th colspan="2">Relevance</th>
    <th colspan="2">HQ</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>WO-IM</th>
    <th>W-IM</th>
    <th>WO-IM</th>
    <th>W-IM</th>
  </tr>
  <tr>
    <td>Article Text Only</td>
    <td>3.3</td>
    <td>3.2</td>
    <td>3.4</td>
    <td>3.7</td>
    <td></td>
  </tr>
  <tr>
    <td>Article Text + Comments</td>
    <td>3.2</td>
    <td>3.4</td>
    <td>3.5</td>
    <td>3.8</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Relevant Comments</td>
    <td>3.4</td>
    <td>3.5</td>
    <td>3.6</td>
    <td>4.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Images</td>
    <td>3.3</td>
    <td>3.4</td>
    <td>3.7</td>
    <td>4.1</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Images + Comments</td>
    <td><b>3.5</b></td>
    <td><b>3.6</b></td>
    <td><b>3.9</b></td>
    <td><b>4.2</b></td>
    <td></td>
  </tr>
</table>

Table 9: Human evaluation scores for various metrics of GPT-based model using COSMMIC configurations for headline generation. Gr-Co means grammatical correctness, HQ means headline quality (With Images (W-IM) and Without Images (WO-IM)).

<table>
  <tr>
    <th>Configuration</th>
    <th>Gr-Co</th>
    <th colspan="2">Relevance</th>
    <th colspan="2">SQ</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>WO-IM</th>
    <th>W-IM</th>
    <th>WO-IM</th>
    <th>W-IM</th>
  </tr>
  <tr>
    <td>Article Text Only</td>
    <td>2.3</td>
    <td>2.4</td>
    <td>2.2</td>
    <td>2.5</td>
    <td></td>
  </tr>
  <tr>
    <td>Article Text + Comments</td>
    <td>2.5</td>
    <td>2.6</td>
    <td>2.4</td>
    <td>2.7</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Relevant Comments</td>
    <td><b>2.6</b></td>
    <td><b>2.8</b></td>
    <td><b>2.9</b></td>
    <td><b>3.1</b></td>
    <td></td>
  </tr>
</table>

Table 10: Human evaluation scores for various metrics of LLaMA 3-based model using COSMMIC configurations. Gr-Co means grammatical correctness, SQ means summary quality.

<table>
  <tr>
    <th>Configuration</th>
    <th>Gr-Co</th>
    <th colspan="2">Relevance</th>
    <th colspan="2">HQ</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>WO-IM</th>
    <th>W-IM</th>
    <th>WO-IM</th>
    <th>W-IM</th>
  </tr>
  <tr>
    <td>Article Text Only</td>
    <td>2.1</td>
    <td>2.2</td>
    <td>2.1</td>
    <td>2.3</td>
    <td></td>
  </tr>
  <tr>
    <td>Article Text + Comments</td>
    <td>2.3</td>
    <td>2.4</td>
    <td>2.2</td>
    <td>2.5</td>
    <td></td>
  </tr>
  <tr>
    <td>Text + Relevant Comments</td>
    <td><b>2.4</b></td>
    <td><b>2.6</b></td>
    <td><b>2.5</b></td>
    <td><b>2.6</b></td>
    <td></td>
  </tr>
</table>

Table 11: Human evaluation scores for various metrics of LLaMA 3-based model using COSMMIC configurations for headline generation. Gr-Co means grammatical correctness, HQ means headline quality.

1. Initial Screening: Annotators carefully read through the entire set of comments associated with each article to assess their relevance.

2. Relevance Assessment: Each comment was evaluated on its alignment with the article’s content and its potential to provide contextual insights. Annotators considered whether the comment added value to the discussion or reinforced key ideas presented in the article.

3. Exclusion of Noisy Comments: Spam, generic, or off-topic comments were removed to ensure that the retained comments maintained the summaries’ coherence and relevance.

4. Final Review: A second pass was conducted by another annotator to confirm that no valuable comments were mistakenly filtered out.

Guidelines for Including Relevant Comments:
The manual filtering process was guided by clear criteria for retaining comments that contributed to the quality of the summaries:

• Content Relevance: Comments directly related to the article’s subject matter were prioritized.

• Contextual Depth: Comments that elaborated
<table>
  <tr>
    <th>Setting</th>
    <th>R1</th>
    <th>R2</th>
    <th>RL</th>
    <th>BS</th>
  </tr>
  <tr>
    <td>Original (Only Article)</td>
    <td>44.18</td>
    <td>18.94</td>
    <td>33.47</td>
    <td>76.61</td>
  </tr>
  <tr>
    <td>Translated (Hindi → English)</td>
    <td>35.52</td>
    <td>12.03</td>
    <td>25.91</td>
    <td>69.82</td>
  </tr>
  <tr>
    <td>Back-Translated (Hindi)</td>
    <td>21.79</td>
    <td>4.51</td>
    <td>15.62</td>
    <td>60.23</td>
  </tr>
  <tr>
    <td>Original (Article + Comments)</td>
    <td>50.98</td>
    <td>24.82</td>
    <td>38.34</td>
    <td>78.22</td>
  </tr>
  <tr>
    <td>Translated (Hindi → English)</td>
    <td>40.21</td>
    <td>15.49</td>
    <td>29.12</td>
    <td>71.31</td>
  </tr>
  <tr>
    <td>Back-Translated (Hindi)</td>
    <td>26.29</td>
    <td>6.81</td>
    <td>18.39</td>
    <td>62.48</td>
  </tr>
  <tr>
    <td>Original (Article + Relevant Comments)</td>
    <td>52.13</td>
    <td>25.52</td>
    <td>39.43</td>
    <td>78.71</td>
  </tr>
  <tr>
    <td>Translated (Hindi → English)</td>
    <td>41.49</td>
    <td>16.01</td>
    <td>30.18</td>
    <td>72.10</td>
  </tr>
  <tr>
    <td>Back-Translated (Hindi)</td>
    <td>27.01</td>
    <td>7.22</td>
    <td>19.51</td>
    <td>63.02</td>
  </tr>
</table>

Table 12: Summarization Performance Across Different Translation Settings

on or clarified points in the article were included.

• Collective Sentiment: Comments reflecting collective user opinions or emphasizing key elements of the article were considered valuable for summarization.

Quality Assurance and Compensation: The annotators’ thorough approach to manual filtering ensured that the summaries accurately represented the articles while benefiting from the inclusion of relevant user comments. This additional task required substantial effort, and annotators were compensated at an additional rate of USD 1.20 per 100 comments for their work, recognizing the meticulous nature of the filtering process.

Outcome: This manual filtering process ensured that the final summaries captured the essence of the original articles while incorporating meaningful user contributions. By excluding noisy comments, the integrity and relevance of the dataset were preserved, resulting in summaries that were both concise and enriched by user perspectives.

A.9 Back-Translation Study on Hindi Dataset for Summarization

To evaluate the impact of machine translation on linguistic fidelity in Reader-Aware Comment-Based Summarization (RACBS), we conducted a back-translation study on the Hindi dataset. This experiment aimed to determine whether translating Indian language content into English for processing and then back into the original language affects the quality of summarization and headline generation.

For this study, we first translated the Hindi articles, their corresponding comments, and the ground-truth summaries and headlines into English using GPT-4o. Once translated, we ran all our summarization and headline-generation experiments on the English dataset. Following this, the outputs were back-translated into Hindi, and the same experimental pipeline was applied to the back-translated dataset.

The results on the back-translated Hindi dataset showed a noticeable drop in performance compared to the original Hindi dataset. This degradation highlights the loss of linguistic nuances and contextual accuracy introduced by machine translation. Such findings reinforce our claim that preserving the original language is crucial for maintaining the quality of Indian language summarization and headline generation.

Moreover, incorporating machine translation workflows would require extensive manual verification across multiple languages, which is resource-intensive and currently beyond our scope. This experiment further justifies our decision to exclude machine translation-based approaches and focus on direct processing in Indian languages for this resource paper.

We evaluated three settings: (i) Original Hindi data, (ii) Translated Hindi-to-English data, and (iii) Back-Translated data where the English outputs were retranslated into Hindi. The summarization performance was measured using ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL), and BERTScore (BS). The results are summarized in Table 12.

A.9.1 Analysis

From Table 12, it is evident that the original Hindi dataset consistently achieves the highest performance across all metrics and settings. The ROUGE and BERTScore values decline significantly when content is translated into English and further degrade when back-translated into
<table>
  <tr>
    <th>Setting</th>
    <th>R1</th>
    <th>R2</th>
    <th>RL</th>
    <th>BS</th>
  </tr>
  <tr>
    <td>Original (Only Article)</td>
    <td>32.23</td>
    <td>9.58</td>
    <td>24.54</td>
    <td>72.97</td>
  </tr>
  <tr>
    <td>Translated (Hindi → English)</td>
    <td>22.01</td>
    <td>4.02</td>
    <td>15.49</td>
    <td>65.31</td>
  </tr>
  <tr>
    <td>Back-Translated (Hindi)</td>
    <td>10.39</td>
    <td>1.21</td>
    <td>7.32</td>
    <td>55.08</td>
  </tr>
  <tr>
    <td>Original (Article + Comments)</td>
    <td>30.27</td>
    <td>8.99</td>
    <td>23.61</td>
    <td>72.89</td>
  </tr>
  <tr>
    <td>Translated (Hindi → English)</td>
    <td>21.02</td>
    <td>3.51</td>
    <td>14.69</td>
    <td>64.52</td>
  </tr>
  <tr>
    <td>Back-Translated (Hindi)</td>
    <td>9.79</td>
    <td>0.91</td>
    <td>6.81</td>
    <td>54.02</td>
  </tr>
  <tr>
    <td>Original (Article + Relevant Comments)</td>
    <td>30.49</td>
    <td>9.12</td>
    <td>23.94</td>
    <td>72.87</td>
  </tr>
  <tr>
    <td>Translated (Hindi → English)</td>
    <td>21.50</td>
    <td>3.81</td>
    <td>14.99</td>
    <td>64.81</td>
  </tr>
  <tr>
    <td>Back-Translated (Hindi)</td>
    <td>10.11</td>
    <td>1.11</td>
    <td>7.12</td>
    <td>54.59</td>
  </tr>
</table>

Table 13: Headline Generation Performance Across Different Translation Settings

Hindi, reinforcing the importance of preserving the original language.

For summarization based only on the article, the original Hindi dataset achieves an R1 score of **44.18**, whereas the translated version drops to **35.52**, and the back-translated version further declines to **21.79**. The degradation in **R2 and RL scores follows a similar pattern**, demonstrating that information loss occurs due to translation.

When incorporating comments into summarization, performance improves across all settings. The original Hindi dataset (Article + Comments) achieves an **R1 of 50.98**, which is significantly higher than both the translated (**40.21**) and back-translated (**26.29**) versions. Similarly, the **Article + Relevant Comments** setting yields the best results, with an **R1 of 52.13** in the original Hindi dataset, compared to **41.49** (translated) and **27.01** (back-translated).

These findings emphasize that **machine translation introduces considerable degradation in summarization quality**, likely due to the loss of linguistic nuances and contextual cues specific to Hindi. The back-translated results, performing the worst in all cases, further confirm that relying on translation workflows can negatively impact summarization tasks.

This experiment strongly supports our decision to focus on direct summarization in Indian languages rather than incorporating machine translation workflows, as doing so would compromise linguistic accuracy and overall performance.

A.10 Back-Translation Study on Hindi Dataset for Headline Generation

To further assess the impact of machine translation, we conducted a similar back-translation study on the **headline generation** task. This experiment followed the same methodology as summarization: (i) processing the **original Hindi dataset**, (ii) translating it into **English for experimentation**, and (iii) **back-translating** the English results into Hindi to examine performance degradation.

Table 13 presents the results across different settings, evaluated using ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL), and BERTScore (BS).

A.10.1 Analysis

The results in Table 13 further reinforce the observations made for summarization. The **original Hindi dataset consistently achieves the best performance**, while both **translation to English and back-translation to Hindi lead to severe performance degradation**.

For headline generation using only the article, the original Hindi dataset achieves an **R1 score of 32.23**, but this drops to **22.01** after translation into English and further declines to **10.39** after back-translation. The same trend is observed in **R2 and RL scores**, confirming a significant loss of key information and linguistic quality during translation.

When comments are included, headline generation performance improves slightly, similar to the summarization task. However, even with additional context, **back-translated Hindi results remain substantially weaker**. For instance, in the **Article + Relevant Comments** setting, the **original Hindi dataset achieves an R1 of 30.49**, which is much higher than both the translated (**21.50**) and back-translated (**10.11**) versions.

These findings indicate that **headline generation is even more sensitive to translation degradation than summarization**. This is expected, as headlines require **high linguistic precision and contextual awareness**, which are often distorted
through translation workflows.

Overall, these results strongly validate our claim that machine translation is not a viable pre-processing step for our task. Instead, our dataset should be leveraged directly in its original language to ensure linguistic integrity and maintain the highest possible performance across summarization and headline generation tasks.

A.11 An Intriguingly Intuitive Question

Question: If the headline and summary are written prior to the publication of an article, the scenario of comment-sensitive headline/summary generation can appear non-intuitive. How can the comment-sensitive headline or summary of an article be generated without relying on user comments?

Answer: It is important to recognize the dynamic nature of modern digital media, where articles, summaries, and headlines are often updated in response to user feedback, emerging developments, or shifts in public sentiment. This dynamic environment introduces challenges for comment-sensitive generation, as summaries and headlines must adapt to an ever-evolving flow of information, including real-time user interactions. Therefore, a forward-looking strategy could involve creating standalone summaries of relevant user comments (if present) for each article and storing them as part of a dynamic feedback repository. These stored summaries could then be utilized when publishing related follow-up articles, enabling the generation of headlines and summaries that reflect updated contexts, user sentiment, or emerging trends. By leveraging previously collected user feedback, this method would facilitate the integration of comment sensitivity without requiring real-time comment data for every article. Such a strategy enhances scalability and allows the summarization process to remain adaptable over time, addressing the complexities of a constantly changing digital media landscape. This highlights the potential for innovative comment-sensitive headline and summary generation that balances real-time relevance with long-term adaptability.

Reviewer paper summaries: This article aims to address is how to conduct comment-sensitive summarization and headline generation in a multilingual and multimodal context, particularly for Indian languages. The paper presents COSMMIC, a comment-sensitive multimodal and multilingual dataset, to tackle the problem of multimodal summarization and headline generation for Indian languages.

This article proposes COSMMIC, a multimodal, multilingual, comment sensitive corpus for Indian languages, covering 9 major Indian languages (such as Hindi, Bengali, etc.), containing 4959 article image pairs and 24484 user comments, supporting abstract and title generation tasks. By integrating text, images, and user comments, the author explored various input configurations (such as text only, text+comments, text+images, etc.) and conducted experiments based on GPT-4 and LLAMA-3 models to verify the potential of multimodal data to improve the performance of generation tasks. Experiments have shown that the addition of images and user comments can significantly enhance the contextual relevance of abstracts and titles, especially in specific languages.
Summary of strengths: 1. COSMMIC consists of 4,959 article-image pairs and 24,484 reader comments, covering nine major Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam, Odia, Tamil, Telugu, and Kannada. The dataset provides summaries for all languages. This will be of great help to the development of the multimodal field. 
2. The experiments thoroughly explore six configuration combinations for generating summaries: using only the article text, combining with comments, combining with relevant comments, incorporating images, combining with images and relevant comments. GPT-4 and LLAMA-3 models are used for evaluation. The experiment is very comprehensive.

S1: This work releases a multimodal dataset of Indian languages, integrating text, images, and user comments for the first time, supporting cross linguistic and cross modal NLG task research.

S2: The paper systematically analyzed the impact of multimodal data on generation tasks through various input configurations (such as text, text+comments, text+images, etc.) and SOTA models (GPT-4, LLAMA-3), providing detailed benchmark results.
Summary of weaknesses: 1. I am highly curious about how the threshold of 0.25 is set in the image classification based on CLIP. What kind of supplementary information is provided for the images that are determined as irrelevant by CLIP (with a threshold lower than 0.25)? More importantly, the article contends that the data samples are obtained from a multimedia house, thus suggesting that there are no unrelated image types. However, I believe it is more precise to state that for the input text, there are no unrelated image types. But for the summary, it is possible that there are unrelated image types.

N/A
Comments, suggestions and typos: Please standardize the format of the references.

I regret that I'm not a native Indian speaker, so I can only evaluate this paper with the writing and analysis of the paper. I can not understand the cases display in the paper. Therefore, I set the confidence to 2.
Overall assessment scores: 4

3.5