XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags (2024)

Abstract: Millions of news articles published online daily can overwhelm readers. Headlines and entity (topic) tags are essential for guiding readers to decide if the content is worth their time. While headline generation has been extensively studied, tag generation remains largely unexplored, yet it offers readers better access to topics of interest. The need for conciseness in capturing readers’ attention necessitates improved content selection strategies for identifying salient and relevant segments within lengthy articles, thereby guiding language models effectively. To address this, we propose to leverage auxiliary information such as images and captions embedded in the articles to retrieve relevant sentences and utilize instruction tuning with variations to generate both headlines and tags for news articles in a multilingual context. To make use of the auxiliary information, we have compiled a dataset named XL-HeadTags, which includes 20 languages across 6 diverse language families. Through extensive evaluation, we demonstrate the effectiveness of our plugand-play multimodal-multilingual retrievers for both tasks. Additionally, we have developed a suite of tools for processing and evaluating multilingual texts, significantly contributing to the research community by enabling more accurate and efficient analysis across languages.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 12991–13024 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
The headline serves as a concise and attentiongrabbing summary of a news article. Articles with compelling headlines are more likely to attract increased views or shares (Gu et al., 2020; Song et al., 2020). Unlike summaries, which provide a broad overview (Nayeem et al., 2018), headlines aim to produce brief and engaging statements that draw
* Equal contribution. † Corresponding author. 1Our code, dataset, model checkpoints, developed tools
are available at XL-HeadTags.
readers into the full article (Xu et al., 2019; Zhang and Yang, 2023; Akash et al., 2023). They are often the most-read part of the article, guiding readers in deciding whether the content merits their time (Bukhtiyarov and Gusev, 2020).
Another important feature of a news article are topic tags or “semantic markers”, which serve as dynamic connectors and navigational aids, enhancing the coherence and accessibility of information across articles. The task of tag generation is related to keyphrase generation (Meng et al., 2017). While keyphrases summarize the main themes succinctly of an article, tags provide a broader overview, guiding readers to related articles and facilitating navigation through connected themes. Despite their significant role, the generation of news article tags has remained unexplored in existing literature.
In this work, we introduce a unified framework for generating headlines and tags for news articles in a multimodal-multilingual context, covering
12991
20 languages across 6 diverse language families. Despite the emergence of open-source Large Language Models (LLMs) like Llama (Touvron et al., 2023) and Mistral (Jiang et al., 2023), multilingual models such as mT5 (Xue et al., 2021) and mT0 (Muennighoff et al., 2022), fine-tuned with task-specific data, remain the preferred solution for multilingual tasks, especially for low-resource languages (Ahuja et al., 2023; Zhao et al., 2024; Aggarwal et al., 2024). News articles often include extensive content, incorporating additional details, author quotes, historical context, and advertisements, among others. This poses a challenge in identifying article segments that are both salient and relevant for the creation of highly concise outputs like headlines and tags. This complication leads to what is known as the “Lost-in-the-Context” problem (Liu et al., 2023), wherein vital information embedded within lengthy documents is frequently overlooked by the models (Ravaut et al., 2024).
Fortunately, it is now very common for both online and printed news media to use multimedia content to enhance visibility, support, and context for articles (Oostdijk et al., 2020). Digital assets, like images, often serve as thumbnails across social media, blogs, and various platforms. Equally important are the captions accompanying these images, which not only clarify and enrich the image but also optimize articles for search engines and make news more accessible to those with vision impairments (Liu et al., 2021). This has motivated us to utilize auxiliary information (images and captions as illustrated in Figure 1) to distill salient and localized information from news articles in a multimodal-multilingual context using the CLIP-ViT-B32 model (Radford et al., 2021). This model aligns text and images within a unified dense vector space. Our method is inspired by the Cognitive Load Theory (Sweller, 2011), reflecting how humans employ visual cues and summaries to understand the essence of lengthy texts without being overwhelmed by information. Building on this, with the contents selected using our multimodal retrievers, we utilize instruction tuning to generate both headlines and tags for news articles in a multilingual context.
Our contributions are summarized as follows:
• We compile the XL-HeadTags dataset for headline and tags generation tasks, expanding it to include 20 languages across 6 diverse language families (§2.1).
• We present a new approach to content selection that utilizes auxiliary information from both textual and visual modalities to identify the most salient content within news articles in a multilingual setting. Our modules are crafted as plug-and-play components, allowing for seamless integration with language models of any size and type (§3).
• We utilize instruction tuning to generate both headline and tag words. Our model is capable of producing tag words in both controlled and unrestricted manners through instructions (§3.2). Furthermore, we introduce novel tag words evaluation metrics designed to evaluate scenarios of both controlled and unrestricted generation effectively (§5.1).
• We also develop tools by accumulating opensource resources for processing and evaluating multilingual texts, making it an easy-touse one-stop destination. These tools include (1) Multilingual ROUGE Scorer, (2) Multilingual Sentence Tokenizer, and (3) Multilingual Stemmer. These resources are invaluable to the research community focusing on multilingual NLP (§4).
2 XL-HeadTags: Dataset & Tasks
2.1 Dataset Our study focuses on exploring techniques for multilingual and multimodal retrieval, aiming to extract salient and localized information from documents. This is intended to support the generation of succinct headlines and tags. Our goal is to utilize auxiliary information, such as images and image captions, as queries to retrieve salient information from the document. To achieve our goals, we explore existing large-scale multilingual and multimodal datasets to identify auxiliary information, such as images and their captions. Table 1 highlights several well-known datasets that serve summarization purposes. We choose M3LS (Verma et al., 2023) and XL-Sum (Hasan et al., 2021) as our primary data sources. Since both datasets share the BBC as their source, we anticipate minimal distributional and structural shifts, which could enhance the coherence and efficiency of our retrieval process. M3LS is a large-scale dataset designed for Multi-
modal Multilingual Summarization, featuring headlines, articles, summaries, images, captions and tag words. In contrast, XL-Sum focuses on Multilingual
Abstractive Summarization, comprising headlines, articles, summaries, and news links. It is noteworthy that although the M3LS dataset includes images, captions, and tag words, these were not leveraged for the development of their models. Instead of recrawling the data points, we specifically utilized the auxiliary information from M3LS, for our retrieval augmentation framework. Furthermore, to expand our dataset to additional language families, we incorporated Arabic, Turkish, and Persian from the XL-Sum dataset. A notable limitation of XL-Sum is its absence of images, image captions, and tag words. To address this gap, we utilized the news URLs provided in the XL-Sum dataset to gather the necessary auxiliary information for our framework. We employ a scrapy2 framework-based web crawler to systematically collect detailed information, including images, captions, and tag words from news articles. Our criteria require the presence of key elements (headlines, articles, images, captions, and tags) in each data point.
Our XL-HeadTags dataset consists of 20 languages across six diverse language families. Detailed statistics of our dataset are presented in Table 2 and described in the Appendix A. Additionally, we have developed a mechanism to classify these languages into high-resource and low-resource categories. Conneau et al. (2020) utilized Common-
2 https://scrapy.org
Crawl (CC) pretraining data, quantified in gigabytes (GB), to highlight the disparities between high- and low-resource languages. This differentiation is determined by examining the volume of pretraining data used in developing pretrained language models. In contrast, we define high-resource and low-resource languages based on the availability of task-specific resources. This includes both datasets and pretrained and/or task-specific finetuned models, as per the Huggingface resource ranking3, which offers real-time and up-to-date information. Languages ranked in the top 10 of this list are classified as high-resource, while the others are low-resource. Figure 2 illustrates the distribution of datasets and models across languages.
3 https://huggingface.co/languages
2.2 Tasks
Our goal is to simultaneously generate headline and tags for news articles. Given that headlines provide a condensed summary and tag words serve as semantic markers, we propose that these two tasks can be effectively learned in a unified learning framework. Formally, Given a news Article A consisting of sequence of words {a1, a2,⋯, an}, our objective is to generate an abstractive Headline, H consisting of sequence of words {h1, h2,⋯, hm} and a set of T Tag Words {{T1}, {T2},⋯, {To}}, where each T can be a word T = {t1} or a sequence of word T = {t1, t2,⋯, tp}. Thus the task is XL-HeadTags(A) ⟹ H, T . Controlled Generation During the tag word generation phase, we introduce an additional setting to control the number of tag word generations. This task is formally defined as Con-XL-HeadTags(A,N ) ⟹ H, T con1∶n . Here, N acts as the control factor determining the number of tag words to be generated, and T con1∶n represents the resulting set of n controlled tag words.
3 MultiRAGen
Retrieval-Augmented Generation (RAG) (Lewis et al., 2020a) involves two key phases: first, retrieving contextually relevant information, and sec-
ond, using this information to guide the generation process (Zhao et al., 2023). RAG has been applied to various tasks such as machine translation (He et al., 2021), dialogue generation (Cai et al., 2019), and abstractive summarization (Peng et al., 2019). Inspired by these applications, we leverage multimodal information like images and captions to select salient content from news articles, introducing MultiRAGen (Multimodal Retrieval Augmented Generation). MultiRAGen comprises two main components: (1) Multimodal Retrievers (§3.1) and (2) Instruction Tuning (§3.2).
3.1 Multimodal Retrievers (MultiRet) Our approach utilizes auxiliary information, such as images and captions, to extract salient and relevant sentences from documents through two modules: ImgRet for images (visual modality) and CapRet for image captions (textual modality). Formally, these modules are described as follows:
MultiRet = ⎧⎪⎪⎪⎪⎨⎪⎪⎪⎪⎩ ImgRet(A1∶n′ , I1∶m′ ,K,L) ⟹ AI1∶k, CapRet(A1∶n′ , C1∶m′ ,K,L) ⟹ AC1∶k
where A1∶n′ represents the article consisting of n′ sentences, I1∶m′ and C1∶m′ denote the set of images and image captions within the document, respectively, with m′ ⩾ 1. K is the number of sentences
Algorithm 1: ImgRet function ImgRet(Article, L, Images, K)
sentences ← SenSeg(Article, L) for sen in sentences do
for img in Images do scr ← SimScr(sen, img) sim ← sim + scr
end end // Sort the sentences on sim article ← sentences[1:K] // Sort the sentences on order return article
to be retrieved, AI1∶k and AC1∶k are the resulting subsets containing k sentences selected based on visual and textual modalities, respectively. Here, k = K if K ⩽ n′, otherwise k = n′. L specifies the language of the article.
We tokenize the documents into sentences using our Multilingual Sentence Tokenizer (introduced later in Section 4) and use images and captions as queries to compute semantic similarity with the sentences. This process employs a multilingual version of the OpenAI CLIP-ViT-B32 model4 (Radford et al., 2021), which maps text (in 50+ languages) and images to a shared dense vector space (Reimers and Gurevych, 2019), aligning images closely with their corresponding texts. We then retrieve the top K sentences from the document based on the similarity scores.
Handling Multiple Images and Captions We observed that a single document often contains multiple images and captions without a proper one-toone mapping between them. Consequently, we treat each image and caption as distinct entities and propose a greedy algorithm for aggregating multiple retrievals. The detailed procedural depiction of one this simple yet effective algorithmic processes is presented in Algorithm 1 and detailed in Appendix B.
3.2 Instruction Tuning
Language models can be fine-tuned with supervised datasets containing natural language prompts and their corresponding target completions (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Min et al., 2022). This process, known as “instruction tuning,” enhances the models’ ability to fol-
4clip-ViT-B-32-multilingual-v1
low instructions accurately. Typically, task-specific prefixes are used to guide the model towards the desired output format. Inspired by the adaptability and success of these approaches in managing diverse tasks via a unified text-to-text framework, we apply this methodology to generate headlines and tags for news articles within a multilingual setting. We introduce two instructional variations: one for unrestricted and another for controlled tag word generation along with headline.
Unrestricted Generation allows the model to independently determine the optimal number of tag words to generate. In the following input instruction, bold indicates the input prefix, while underline signifies the selected content (described below). Conversely, in the output instruction, bold marks the output prefix, and underline denotes the generated output, encompassing both the headline and a variable number of tag words (T1, T2, ...) corresponding to each article. This approach aims to simultaneously address two text generation tasks, utilizing output prefixes to distinguish between the outcomes of each task. The complete instruction format is as follows:
Instruction for Unrestricted Generation
Input ➝ Generate Headline and Tag Words: Selected Content.
Output ➝ Headline is: Headline. Tag words are: T1, T2, ⋯, To.
Controlled Generation To tackle the challenge of determining the correct number of tag words, we have adjusted our unrestricted prefix. This modification allows us to directly specify the desired number of tag words in the prefix. Our revised input format for controlled tag word generation is:
Instruction for Controlled Generation
Input ➝ Generate Headline and N Tag Words: Selected Content.
Here, N refers to the number of tag words to generate. During model training, this number is represented as the count of tag words associated with the original article from the training dataset, verbalized in natural language (such as One, Two, Three, ...). The output format remains unchanged.
Selected Content For the input format of the instruction, we utilize three settings: (1) Article,
where the original article is placed; (2) Top K sentences retrieved by our MultiRet; and (3) Top K sentences from our MultiRet concatenated with the Article. MultiRet comprises two modules: ImgRet, which retrieves relevant sentences using images as queries, and CapRet, which uses image captions as queries to retrieve pertinent sentences from the input article as detailed in Section §3.1.
4 Multilingual Tools
We also develop tools by accumulating open-source resources for processing and evaluating multilingual texts, making it an easy-to-use one-stop destination. These tools include (1) Multilingual ROUGE Scorer, (2) Multilingual Sentence Tokenizer, and (3) Multilingual Stemmer. These resources are invaluable to the research community focusing on multilingual NLP, providing essential support for accurate processing and evaluation5
(also detailed in Appendix C).
Multilingual ROUGE Scorer Hasan et al. (2021); Chronopoulou et al. (2023) identified a significant issue in evaluating multilingual summarization performance: the absence of stemmers for certain low-resource languages hindered the processing of generated summaries, resulting in lower ROUGE scores (Lin, 2004). Facing a similar challenge, Aharoni et al. (2023) calculated ROUGE scores using a multilingual tokenizer. However, the lack of word tokenizers for some languages still presents a challenge for fair ROUGE score assessment across different language families. To address this issue, we developed a Multilingual ROUGE Scorer that leverages Byte-Pair Encoding (BPE) tokenization from BERT-multilingual (Devlin et al., 2019), ensuring more accurate evaluation across 104 languages.
Multilingual Sentence Tokenizer Sentence tokenization aims to divide a given document into individual sentences. To achieve this, we integrated various open-source resources for multiple languages into a unified library. This tool can perform tokenization in 41 different languages (details of the sources in Appendix (Table 6)).
Multilingual Stemmer For evaluating the performance of keyphrase generation, both generated and reference keyphrases are normalized before assessing exact matches (Meng et al., 2017; Chen
5These tools are available here XL-HeadTags, and can be easily installed via ‘pip’.
et al., 2020). Given the need to evaluate generated tag words across various languages, we developed a Multilingual Stemmer that integrates open-source stemmers for 18 distinct languages (as detailed in Table 7 in the Appendix). However, open-source stemmers for Chinese and Telugu are unavailable. Therefore, when evaluating tag words in these languages, we report the scores without normalizing the tag words.
5 Evaluating Tags Generation
The terms “Tag Words” and “Keyphrases,” while sharing a common objective, differ in their application. In the keyphrase generation process, a model predicts a set of distinct keyphrases Ŷ =(ŷ1, . . . , ŷm) from a given source text, with these predictions ŷi being ordered based on their relevance (Yuan et al., 2020). The ground truth keyphrases for the source text are denoted as Y . It’s important to note that in the context of tag words, the order of predictions does not necessarily reflect the quality of each prediction. To measure predictive performance, three standard evaluation metrics—namely macro-averaged precision, recall, and F-measure (F1)—are commonly used (Meng et al., 2017). Formally, the metrics of precision, recall, and F1 score are defined as follows:
P = ∣Ŷ ∩ Y∣∣Ŷ∣ , R = ∣Ŷ ∩ Y∣∣Y∣ , F1 =
2 ∗ P ∗R P +R .
(1)
5.1 Proposed Tag Words Evaluation Metrics
As detailed in Section 3.2, our work spans both controlled and unrestricted tag word generation. Inspired by Yuan et al. (2020), we introduce three metrics for assessing performance.
Unrestricted Generation In unrestricted generation, a varying number of tag words are generated. The evaluation employs the metric F1@M, where∣Ŷ∣ = M. Here, M varies with each article, reflecting the model’s autonomous decision on the number of tag words.
Controlled Generation For controlled generation, the goal is to generate a fixed number of tag words. We evaluate this using F1@K and F1@O. K is predefined as 3 and 5, and O corresponds to∣Y∣, the actual number of tag words. This means we assess controlled generation for producing ex-
actly 3, 5, and ∣Y∣ tag words, as specified by a natural language prefix.
6 Experiments
6.1 Data & Evaluation Metrics
Data To ensure a balanced distribution, our dataset division for training, validation, and test sets across all languages consists of 95% (394,353 samples), 1% (5,187 samples), and 4% (15,577 samples) respectively from our XL-HeadTags dataset (detailed in Appendix (Table 2)). Our goal is to enhance the task’s generalizability by developing a unified model capable of performing both controlled (a fixed number of tag words) and unrestricted (the model decides the number of tag words) tag word generation (also elaborated in Section 3.2). To achieve this, we introduce a prefix mixture strategy during training, using a 70:30 allocation ratio. Here, 70% of the data is formatted for controlled tag word generation, while the remaining 30% is free from such constraints. This approach, applied consistently across training, validation, and test datasets for both baseline and our
models, aims to facilitate a model adept at navigating both task variations. Details on data processing are presented in Appendix A.
Evaluation Metrics For the evaluation of generated headlines, we utilize F1 score of our Multilingual ROUGE Scorer (§4) and the BLEU score (Papineni et al., 2002), F1 BERT score (Zhang* et al., 2020), METEOR score (Banerjee and Lavie, 2005), and Length Ratio (LR). For assessing tag words, we apply the metrics we have proposed in Section 5.1 and normalize the tag words using our Multilingual Stemmer (§4).
6.2 Models
Baselines We use mT5-base (Xue et al., 2021), mT0-base (Muennighoff et al., 2022) and Flan-T5-large (Chung et al., 2022) checkpoints available in the Hugging Face (Wolf et al., 2020). Notably, these models are multilingual and pretrained on the mC46 multilingual corpus. It is essential to highlight that the mT0 model underwent additional fine-tuning within a multitask frame-
6 https://huggingface.co/datasets/mc4
work, utilizing the crosslingual task mixture xP3 (Muennighoff et al., 2022) dataset to enhance crosslingual generalization. We conducted finetuning of the mT5-base, mT0-base and Flan-T5large models using the original article in the instruction. Additional baselines like LEAD-1 and EXT-ORACLE are detailed in Appendix D.
Gemini Pro and Mixtral We employed the Gemini Pro (Team, 2023) and Mixtral (Jiang et al., 2023) models for evaluating their efficacy in XLHeadTags multilingual tasks. This assessment occurred in zero-shot prompting conditions, with instructions specifying input (i.e., article) and output formats (§3.2). This encompassed sampling 50 instances from each language.
MultiRAGen (ours) For visual modality, we use images, and for textual modality, we utilize image captions. The number of sentences to retrieve is determined by the parameter K, with explored values of 5, 10, and 15, corresponding to the retrieval of the top 5, 10, and 15 sentences, respectively. After retrieval, we reorder the top-K sentences to their original sequence in the article, ensuring the narrative flow remains coherent. Our experimentation includes two approaches: (1) inputting only the top-K retrieved sentences and (2) combining these sentences with the original article. We apply the same set of hyperparameters for all the baselines and our models, as detailed in Appendix E.
7 Results and Discussion
The evaluation results for headline generation are detailed in Table 3, and the outcomes for tag word evaluation are shown in Table 5. We also compare the performance of our baseline models with three extractive methods—TF-IDF (Sparck Jones, 1988) and TextRank (Mihalcea and Tarau, 2004) as unsupervised, and KEA (Witten et al., 1999) as supervised—on the English test set. These extractive methods are implemented using the PKE module (Boudin, 2016), with a detailed comparison presented in Table 4.
7.1 Headline
Baselines Table 3 shows LEAD-1’s poor performance, indicating its inability to capture the abstractive essence of headlines. EXT-ORACLE, however, provides a robust baseline with improved ROUGE and BLEU scores, though it lacks headline conciseness. mT5, mT0, and Flan-T5 outperform
extractive methods, with mT0 closely matching the conciseness of reference headlines. Flan-T5 excels in high-resource languages but falls short in low-resource setting (also in Table 12, and 18).
MultiRAGen results show that using captions for retrieval with mT5 outperforms the standard mT5 baseline, a pattern also seen with mT0 and Flan-T5. This indicates that including the most relevant information in the context window enhances headline generation. Similarly, using images for retrieval with mT5, mT0, and Flan-T5 also surpasses their respective baselines, performing nearly as well as caption-based retrieval. This suggests potential benefits from integrating both textual and visual modalities in future research to improve retrieval effectiveness (See Limitations (§8)).
7.2 Tag Words Baselines Table 4 compares tag word generation across baseline models and extractive methods (TFIDF, TextRank, and KEA) on an English test corpus. Extractive methods, limited to identifying tag words already present in the text, demonstrate varied performance, with TextRank lagging due to its tendency to extract longer phrases, misaligned with the brevity of reference tags. In contrast, TF-IDF and KEA perform moderately better but are outshined by abstractive models like mT5, mT0, and Flan-T5, which excel in generating concise and relevant tag words, surpassing extractive approaches that cannot synthesize new tag words. This distinction highlights the superior capability of abstractive methods in handling the tag word generation task.
MultiRAGen Table 5 extends the tag word evaluation results to include baselines mT5, mT0, FlanT5, Gemini Pro, Mixtral and MultiRAGen across a multilingual text corpus. The abstractive baselines exhibit commendable performance, with mT0
demonstrating a noticeably higher gain over mT5 and Flan-T5. This improvement is attributed to the further fine-tuning of mT0 on a multitask framework, enhancing its crosslingual generalization. Although Flan-T5 produces results similar to mT0 and mT5 in English (see Table 4), it falls short of generating any tag words when it comes to certain languages with limited resources. This disparity is particularly noticeable in Table 21.
7.3 Discussion
Gemini Pro and Mixtral exhibit notably low performance, indicating a tendency to generate headlines and tag words in English or romanized rather than the native language of the article. More details with “Prompt for LLMs” and example outputs are presented in Appendix F.
Better content selection approach Our experiments explored two methods: (1) using only top-K retrieved sentences and (2) top-K merging these with the original article. We found combining retrieved sentences with the full article improved headline generation (see Table 3 vs. 8), while using solely retrieved sentences was more effective for
tag generation (see Table 5 vs. 9). This difference arises because tags, being more concise, benefit from focused inputs, whereas headlines may require broader context for optimal generation.
High-Resource vs. Low-Resource Languages Our model, MultiRAGen, achieves balanced performance across high-resource and low-resource languages. Detailed results, grouped by resource availability, are presented in the Appendix (from Table 10 to Table 21).
8 Conclusion
In this paper, we compile the XL-HeadTags dataset for headline and tag generation tasks, including 20 languages across 6 diverse language families. We introduce a novel content selection approach that leverages auxiliary information from both textual and visual modalities to pinpoint the most salient content within news articles. We employ instruction tuning to generate both headlines and tags in controlled and unrestricted manners. Furthermore, we have developed a suite of tools by accumulating open-source resources for processing and evaluating multilingual texts.
Limitations
Potential Bias in Dataset Source Our dataset exclusively comprises articles sourced from the BBC, which may introduce a bias toward specific narratives or ideologies, potentially impacting the dataset’s representativeness and diversity.
Handling Multiple Images and Captions In our analysis, we noted that documents frequently contain several images and captions without a direct one-to-one correspondence. Ideally, a precise mapping between each image and its caption would enhance our retrieval process. However, due to the absence of such mappings, we treat each image and caption as separate entities for independent retrieval processes. Integrating both images and captions for simultaneous retrieval presents an opportunity for future research, potentially refining the information extraction process.
Computational Constraints Given the computational constraints and resource limitations, we opted for the base versions of models for our experimentation. Despite these constraints, we introduce a novel content selection strategy that leverages auxiliary information from both textual and visual modalities. This approach aims to pinpoint the most pertinent content in news articles across multiple languages. Our designed modules are plug-and-play, ensuring they can be effortlessly integrated with language models of varying sizes. This flexibility facilitates broader applicability and enhances the adaptability of our approach in diverse computational environments.
Ethics Statement
Data Crawling We took ethical consideration into account when scraping data. The data we have collected is intended exclusively for noncommercial research purposes. We conducted our web scraping activities at a reasonable rate, with no intention of causing a Distributed Denial of Service (DDoS) attack. Additionaly, we read the instructions listed in robots.txt7 of each website to ensure we were able to crawl the desired content as per the Robots Exclusion Protocol (REP) standards8.
7 https://moz.com/learn/seo/robotstxt
8The robots.txt file is part of the robots exclusion protocol (REP), a group of web standards regulating how robots crawl the web.
Protection of Privacy We intentionally chose not to collect certain information, including the author name, the time when the article was written, and any personal contact details such as email addresses, phone numbers, etc, for the purposes of our experiments. Consequently, our dataset does not contain any Personal Identifying Information (PII). This decision underscores our commitment to placing user privacy as a top priority.
Acknowledgements
We thank all the anonymous reviewers for their valuable feedback and constructive suggestions for improving this work. Mir Tafseer Nayeem is supported by a Huawei PhD Fellowship.
References Divyanshu Aggarwal, Ashutosh Sathe, and Sunayana
Sitaram. 2024. Maple: Multilingual evaluation of parameter efficient finetuning of large language models. Preprint, arXiv:2401.07598.
Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2023. Multilingual summarization with factual consistency evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3562–3591, Toronto, Canada. Association for Computational Linguistics.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.
Abu Ubaida Akash, Mir Tafseer Nayeem, Faisal Tareque Shohan, and Tanvir Islam. 2023. Shironaam: Bengali news headline generation using auxiliary information. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 52–67, Dubrovnik, Croatia. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. Preprint, arXiv:1409.0473.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor,
Michigan. Association for Computational Linguistics.
Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075–8096, Online. Association for Computational Linguistics.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2021. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426.
Florian Boudin. 2016. pke: an open source pythonbased keyphrase extraction toolkit. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 69–73, Osaka, Japan.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Alexey Bukhtiyarov and Ilya Gusev. 2020. Advances of transformer-based models for news headline generation. In Artificial Intelligence and Natural Language, pages 54–61, Cham. Springer International Publishing.
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. 2019. Skeletonto-response: Dialogue generation guided by retrieval memory. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1219–1228, Minneapolis, Minnesota. Association for Computational Linguistics.
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2017. Improving multi-document summarization via text classification. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 3053–3059. AAAI Press.
Yllias Chali, Moin Tanvee, and Mir Tafseer Nayeem. 2017. Towards abstractive multi-document summarization using submodular function-based framework, sentence compression and merging. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 418–424, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019. Neural keyphrase generation via reinforcement learning with adaptive rewards. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2163–2174, Florence, Italy. Association for Computational Linguistics.
Jingqiang Chen and Hai Zhuge. 2018. Abstractive textimage summarization using multi-modal attentional hierarchical RNN. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4046–4056, Brussels, Belgium. Association for Computational Linguistics.
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1095–1105, Online. Association for Computational Linguistics.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. 2022. MuRAG: Multimodal retrieval-augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5558–5570, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Radia Rayan Chowdhury, Mir Tafseer Nayeem, Tahsin Tasnim Mim, Md. Saifur Rahman Chowdhury, and Taufiqul Jannat. 2021. Unsupervised abstractive summarization of Bengali text documents. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2612–2619, Online. Association for Computational Linguistics.
Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. 2023. Language and task arithmetic with parameter-efficient layers for zero-shot summarization. Preprint, arXiv:2311.09344.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. Preprint, arXiv:2210.11416.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1723–1732, Beijing, China. Association for Computational Linguistics.
Tanvir Ahmed Fuad, Mir Tafseer Nayeem, Asif Mahmud, and Yllias Chali. 2019. Neural sentence fusion for diversity driven abstractive multi-document summarization. Computer Speech & Language, 58:216– 230.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequenceto-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631– 1640, Berlin, Germany. Association for Computational Linguistics.
Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating representative headlines for news stories. In Proceedings of The Web Conference 2020, WWW ’20, page 1773–1784, New York, NY, USA. Association for Computing Machinery.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687–697, Melbourne, Australia. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XLsum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.
Qiuxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu. 2021. Fast and accurate neural machine translation with translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3170–3180, Online. Association for Computational Linguistics.
Tatsuru Higurashi, Hayato Kobayashi, Takeshi Masuyama, and Kazuma Murao. 2018. Extractive headline generation based on learning to rank for community question answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1742–1753, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Kango Iwama and Yoshinobu Kano. 2019. Multiple news headlines generation using page metadata. In Proceedings of the 12th International Conference on Natural Language Generation, pages 101–105, Tokyo, Japan. Association for Computational Linguistics.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. Preprint, arXiv:1911.00172.
Mateusz Krubiński and Pavel Pecina. 2023. MLASK: Multimodal summarization of video-based news articles. In Findings of the Association for Computational Linguistics: EACL 2023, pages 910–924, Dubrovnik, Croatia. Association for Computational Linguistics.
Aman Kumar, Himani Shrotriya, Prachi Sahu, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Amogh Mishra, Mitesh M Khapra, and Pratyush Kumar. 2022. Indicnlg suite: Multilingual datasets for diverse nlg tasks in indic languages. arXiv preprint arXiv:2203.05437.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020a. Retrieval-augmented generation for knowledgeintensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
Haoran Li, Junnan Zhu, Tianshang Liu, Jiajun Zhang, and Chengqing Zong. 2018. Multi-modal sentence
summarization with modality attention and image filtering. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4152–4158. International Joint Conferences on Artificial Intelligence Organization.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. 2021. Visual news: Benchmark and challenges in news image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6761–6771, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. Preprint, arXiv:2307.03172.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101.
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582–592, Vancouver, Canada. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States. Association for Computational Linguistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Don’t give me the details, just the summary!
topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
Mir Tafseer Nayeem and Yllias Chali. 2017a. Extract with order for coherent multi-document summarization. In Proceedings of TextGraphs-11: the Workshop on Graph-based Methods for Natural Language Processing, pages 51–56, Vancouver, Canada. Association for Computational Linguistics.
Mir Tafseer Nayeem and Yllias Chali. 2017b. Paraphrastic fusion for abstractive multi-sentence compression generation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, page 2223–2226, New York, NY, USA. Association for Computing Machinery.
Mir Tafseer Nayeem, Tanvir Ahmed Fuad, and Yllias Chali. 2018. Abstractive unsupervised multidocument summarization using paraphrastic sentence fusion. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1191– 1204, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Mir Tafseer Nayeem, Tanvir Ahmed Fuad, and Yllias Chali. 2019. Neural diverse abstractive sentence compression generation. In Advances in Information Retrieval, pages 109–116, Cham. Springer International Publishing.
Nelleke Oostdijk, Hans van Halteren, Erkan Bas, ar, and Martha Larson. 2020. The connection between the text and images of news articles: New insights for multimedia analysis. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4343–4351, Marseille, France. European Language Resources Association.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2555–2565, Minneapolis, Minnesota. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. In Journal of Machine Learning Research.
Mathieu Ravaut, Aixin Sun, Nancy F. Chen, and Shafiq Joty. 2024. On context utilization in summarization with large language models. Preprint, arXiv:2310.10570.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointergenerator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computational Linguistics.
Yun-Zhu Song, Hong-Han Shuai, Sung-Lin Yeh, YiLun Wu, Lun-Wei Ku, and Wen-Chih Peng. 2020. Attractive or faithful? popularity-reinforced learning for inspired headline generation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8910–8917.
Karen Sparck Jones. 1988. A Statistical Interpretation of Term Specificity and Its Application in Retrieval, page 132–142. Taylor Graham Publishing, GBR.
John Sweller. 2011. Chapter two - cognitive load theory. volume 55 of Psychology of Learning and Motivation, pages 37–76. Academic Press.
Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao, and Masaaki Nagata. 2016. Neural headline generation on Abstract Meaning Representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1054– 1059, Austin, Texas. Association for Computational Linguistics.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. From neural sentence summarization to headline generation: A coarse-to-fine approach. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 4109–4115. AAAI Press.
Gemini Team. 2023. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.
Ottokar Tilk and Tanel Alumäe. 2017. Low-resource neural headline generation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 20–26, Copenhagen, Denmark. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu,
Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and finetuned chat models. Preprint, arXiv:2307.09288.
Yash Verma, Anubhav Jangra, Raghvendra Verma, and Sriparna Saha. 2023. Large scale multi-lingual multimodal summarization dataset. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3620– 3632, Dubrovnik, Croatia. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Ian H. Witten, Gordon W. Paynter, Eibe Frank, Carl Gutwin, and Craig G. Nevill-Manning. 1999. Kea: Practical automatic keyphrase extraction. Preprint, arXiv:cs/9902007.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
Peng Xu, Chien-Sheng Wu, Andrea Madotto, and Pascale Fung. 2019. Clickbait? sensational headline generation with auto-tuned reinforcement learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3065– 3075, Hong Kong, China. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation. ACM Comput. Surv., 54(11s).
Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2020. One size does not fit all: Generating and evaluating variable number of keyphrases. In Proceedings
of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7961–7975, Online. Association for Computational Linguistics.
Boning Zhang and Yang Yang. 2023. MediaHG: Rethinking eye-catchy features in social media headline generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5766–5777, Singapore. Association for Computational Linguistics.
Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, Huanhuan Cao, and Xueqi Cheng. 2018. Question headline generation for news articles. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, page 617–626, New York, NY, USA. Association for Computing Machinery.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. Preprint, arXiv:2401.01055.
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Do Long, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. 2023. Retrieving multimodal information for augmented generation: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4736–4756, Singapore. Association for Computational Linguistics.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1095–1104, Vancouver, Canada. Association for Computational Linguistics.
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal summarization with multimodal output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4154–4164, Brussels, Belgium. Association for Computational Linguistics.
Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. NCLS: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3054– 3064, Hong Kong, China. Association for Computational Linguistics.
Supplementary Material: Appendices
A Data Processing & Statistics
A.1 Data Processing
The data processing involved the application of regular expressions Regex9 to eliminate URLs, HTML tags, and emojis from the headline, article, and image captions. Furthermore, Phonenumbers package10 was employed to specifically filter out phone numbers from the textual content.
A.2 Data Statistics
Following data processing, the dataset comprises 415,117 news samples, represented as tuples encompassing headline, article , images, image captions, and tag words. The task of headline generation is herein considered as a subproblem integral to extreme summarization. Given the dual nature of summarization extractive and abstractive, our objective is to craft abstractive headlines. In the pursuit of measuring abstractiveness, Narayan et al. (2018b) proposed evaluating the percentage of ngrams in the summary that do not find occurrence in the input article. Simultaneously, Bommasani and Cardie (2020) introduced compression as a metric for quantifying conciseness, expressed mathematically as:
Compression(H,A) = 1 − ∣H∣∣A∣ (2) In this context, ∣H∣ and ∣A∣ represent the
lengths of the headline and article, measured in words. The ideal headline is aimed to be informative, concise, and have a high ratio of novel n-grams.
The process of generating tag words is in line with Keyphrase generation. Extracting tag words seeks to capture existing words in the input document. However, it’s crucial to note that extractive methods can’t predict absent tag words, an important aspect of comprehensive tag word generation. In contrast, tag word generation aims to produce both present and absent tag words (Meng et al., 2017). The percentage of present tag words highlights the challenge of predicting tag words not present in the document.
In Table 2, we present a comprehensive set of quantitative statistics that includes both the previ-
9 https://docs.python.org/3/library/re.html
10 https://pypi.org/project/phonenumbers/
ously mentioned metrics and additional ones. The average word count in articles is 902, surpassing the Transformer Encoder’s contextual window capacity. Using the mt5 multilingual sentencepiece tokenizer with a vocabulary size of 250,112, the mean token count per article to 1631. This is noteworthy considering the mt5 encoder’s context window is limited to 512 tokens, leading to truncation beyond this limit. This emphasizes the need for methodologies capable of capturing essential information from documents exceeding the encoder context limit.
Further analysis shows that, on average, articles require compression of 98.88% for effective headline representation, highlighting the significant challenge in abstractive headline generation. Additionally, we observe that the percentage of tag words in the document consistently falls below 50%, emphasizing the importance of generative approaches for keywords to overcome limitations posed by extraction methods. These findings enhance our understanding of the intricate dynamics of document structure and the challenges associated with information extraction and summarization.
B Handling Multiple Images and Captions
The primary distinction between our modules lies in the modalities of the auxiliary data they utilize. As discussed in Section 3.1, our goal is to retrieve salient and pertinent information from news articles. To achieve this, we developed two separate modules for retrieving the most relevant sentences, each leveraging auxiliary information from different modalities. Specifically, ImgRet uses images, and CapRet uses image captions to retrieve the most relevant sentences from a document, respectively.
We observed that a single document often contains multiple images and captions without a proper one-to-one mapping between them. Consequently, we treat each image and caption as distinct entities and propose a greedy algorithm for aggregating multiple retrievals. The following discussion predominantly focuses on the operations of the ImgRet module, with selective explication of the points where CapRet differs.
The ImgRet module processes input data con-
sisting of the Article, Language, Images, and a parameter denoted as K (indicating the top-k sentences for retrieval). Subsequently, the SentenceSegmenter is utilized to deconstruct the Article into individual sentences, employing both the Article and Language as inputs. Subsequent steps involve iteratively comparing each sentence with the images, accumulating similarity scores for each sentence. The sentences are then sorted based on these scores, and the top K sentences, representing the most similar ones, are selected. The final sorting is performed based on their original order in the Article, yielding the most relevant sentences in their original sequence.
It is noteworthy that CapRet diverges from this procedure solely in the calculation of embeddings for image captions, as opposed to the image embeddings computed in the ImgRet module.
C Multilingual Tools
The goal of Multilingual Sentence Tokenizer is to split a provided document into separate sentences. The language scope and the open-source resources employed for this tool are detailed in Table 6.
Multilingual Stemmer is used for normalizing tag words. The languages supported and their respective implementation sources are outlined in Table 7.
D Baselines
D.1 Headline Generation Baselines LEAD-1 In the field of generating news headlines, we often use a common benchmark called LEAD-1 to set the minimum standard for the task (Kumar et al., 2022; Narayan et al., 2018b). To calculate LEAD-1 scores, we take the first sentence of the article as the system-generated headline and compare it with the original headline.
EXT-ORACLE On the flip side, there’s another approach called EXT-ORACLE, which represents the best possible outcome when generating headlines using an extractive method (Kumar et al., 2022; Narayan et al., 2018b). In this case, we align a sentence from the input article with the reference headline, using the ROUGE-2 metric to assess how well they match up.
D.2 Tags Generation Baselines TF-IDF approach (Sparck Jones, 1988) entails the ranking of extracted noun phrase candidates
based on their term frequency and inverse document frequency within the provided documents.
TextRank In the case of TextRank (Mihalcea and Tarau, 2004), the methodology involves the conceptualization of words as web pages, followed by the application of the PageRank algorithm to identify keyphrases.
KEA On the other hand, KEA (Witten et al., 1999) employs lexical methods to identify candidate keyphrases, computes feature values for each candidate, and utilizes a machine learning algorithm to predict the suitability of candidates as keyphrases.
E Training & Hyperparameters
Given the computational constraints and resource limitations, we chose to work with the base versions of the models for our experiments. The finetuning process for the mT5-base and mT0-base models was carried out on a single RTX 3090 GPU, while the Flan-T5-large model was fine-tuned using an RTX A6000 GPU. We configured the encoder input token sequence length to 512 and the decoder output token sequence length to 64.
The fine-tuning phase for the mT5-base, mT0base, and Flan-T5-large models lasted for five epochs, employing a batch size of 8. We utilized the AdamW optimizer (Loshchilov and Hutter, 2019) throughout the training process to optimize the models. This setup was chosen to balance the trade-off between computational efficiency and model performance, ensuring effective utilization of available hardware resources.
F Performance of LLMs
Gemini (Team, 2023) and Mixtral (Jiang et al., 2023) are two of large language models used for various tasks. Google provides access to Gemini via an API11 for free, while the Mistral AI Team has made Mixtral’s weights12 publicly available. We utilized Gemini-Pro and Mixtral models to assess their performance in a multilingual task called XLHeadTags. This evaluation was conducted under zero-shot prompting conditions, where only input and output formats were provided as instructions. We examined 50 examples from each language.
11Gemini Pro API 12Mixtral
Prompt for LLMs
Instruction Generate Headline and some defined or undefined number for tag words from a given news article. When ask to generate defined number of tag words generate exact defined number of tag words seperated by commas (,).
A single tag word can be multiple word or single word. When asked to generate undefined number of tag words, you should generate the number of tag word you think appropriate.
While generating headline and tag words generate them in the language the article is given in.
Exemplars For example, you will be given task in these input format "Generate Headline and Three Tag Words": "News article". Output should be "Headline is: Generated Headline. Tag Words are: Tag1, Tag2, Tag3".
Another example "Generate Headline and Tag Words": "News article". Your output should be "Headline is: Generated Headline. Tag Words are: Tag1, Tag2, . . . , TagN".
Input: {...}
However, our analysis revealed that their performance is subpar compared to a supervised approach. Several factors may have contributed to this. Firstly, both models heavily favor English. Even when presented with non-English articles and instructed to generate in the respective languages, they often produce results in English due to the dominance of English in their pre-training corpus.
Additionally, Mistral occasionally generates output in a romanized version of the target language, indicating a lack of effort in adapting to nonEnglish contexts. We also encountered issues where the models inconsistently generated headlines and tags in different languages or failed to follow instructions, resulting in excessive tag words or overly lengthy headlines.
G Detailed Related Work
G.1 Headline Generation
Headline generation, a niche within abstractive summarization (Chali et al., 2017; Nayeem and Chali, 2017a; Fuad et al., 2019; Nayeem and Chali, 2017b; Nayeem et al., 2019), particularly in multilingual contexts (Chowdhury et al., 2021), has undergone notable advancements. In English, Rush et al. (2015) introduced an attention-based neural network model that employs a recurrent neural network (RNN) and attention mechanism for concise
summarization. Takase et al. (2016) developed an Abstract Meaning Representation (AMR) encoder within an encoder-decoder framework for headline creation, while Zhang et al. (2018) proposed a dualattention sequence-to-sequence model tailored for question-driven headline generation. In contexts with limited resources, Tilk and Alumäe (2017) demonstrated the efficacy of pre-training neural models to enhance headline generation capabilities.
Zhou et al. (2017) approach to headline generation encompasses a tri-phase process involving sentence encoding, sentence selection via a gate network, and headline decoding. Similarly, Tan et al. (2017) introduced a method that priori-
tizes important sentences for context-based headline generation. On the multilingual front, Kumar et al. (2022) launched IndicNLG, a dataset aimed at headline generation, though it primarily focuses on headline-article pairs without incorporating auxiliary information.
Emerging methodologies in headline generation also explore specialized applications, such as generating headlines for community question answering (Higurashi et al., 2018) and creating multiple headlines for varied contexts (Iwama and Kano, 2019), broadening the scope of research in this field.
G.2 Keyphrases Generation While "Tag Words" and "Keyphrases" both serve to enhance information retrieval, their application
and research emphasis differ significantly. Tag word generation remains underexplored compared to the more developed field of keyphrase generation. For example, Meng et al. (2017) unveiled the CopyRNN model, which leverages an attentional encoder-decoder architecture (Bahdanau et al., 2014) integrated with a copying mechanism (Gu et al., 2016) to predict keyphrases. This method employs over-generation with an extensive beam search to then select the top N predictions, such as the top five or ten keyphrases.
Yuan et al. (2020) took an alternative approach by designing a Keyphrase Generation model that not only predicts multiple keyphrases but also determines the appropriate number of keyphrases for each document. This innovative training scheme enables the model to adaptively generate a variable number of keyphrases tailored to the specifics of each document. Further advancing the field, Chan et al. (2019) introduced the application of Reinforcement Learning to optimize the keyphrase generation process, showcasing the evolving techniques aimed at improving the precision and adaptability of keyphrase generation.
G.3 Multitask Learning and Instruction Tuning
In the realm of NLP, multitask learning (MTL) is increasingly recognized for its potential to enhance model performance on related tasks by exploiting their shared features and distinctions. For example, Cao et al. (2017) developed a model that simultaneously trains on summary generation and text classification, achieving notable improvements in text summarization. Dong et al. (2015) pioneered a multitask learning approach using a sequenceto-sequence (Seq2Seq) framework for translating a single source language into multiple target languages. Zhu et al. (2019) explored the synergies between monolingual summarization, machine translation, and cross-lingual summarization through joint training. Similarly, Guo et al. (2018) proposed a model that concurrently learns abstractive summarization and question generation. Extending the scope of multitask learning, Sanh et al. (2022) demonstrated the effectiveness of prompted multitask fine-tuning on a pre-trained T5 model (Raffel et al., 2020) for zero-shot task generalization.
Furthermore, language models can undergo finetuning on supervised datasets comprising natural language prompts paired with their respective target outputs. This method, termed “instruction
tuning,” significantly refines the models’ proficiency in adhering to instructions. Employing task-specific prefixes, this technique directs the model to generate outputs in a predetermined format, showcasing the versatility and precision attainable through instruction-based training.
G.4 Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) represents a pivotal advancement in Natural Language Generation (NLG), addressing the issue of neural models’ limited contextual understanding. Traditional neural models often falter when the input lacks comprehensive information for generating accurate outputs, particularly in complex realworld applications (Yu et al., 2022). To bridge this gap, KNNLM (Khandelwal et al., 2020) introduced a technique for augmenting language models with examples retrieved from a training text dataset, enhancing contextual relevance. Building on this, RETRO (Borgeaud et al., 2021) leveraged a vastly expanded text corpus, enabling models with a smaller footprint to achieve performance on par with GPT-3 (Brown et al., 2020).
Models such as REALM (Guu et al., 2020) and RAG (Lewis et al., 2020b) incorporate Wikipedia passages as external knowledge bases, significantly boosting their efficacy in tasks like Question Answering. REALM focuses on encoding information through masked language modeling, whereas RAG employs an encoder-decoder structure for generative language tasks.
Expanding on these concepts, MuRAG (Chen et al., 2022) stands out by integrating multimodal knowledge sources, encompassing both visual and textual data. This innovation extends the capabilities of knowledge-enhanced text generation, catering to the nuanced demands of intricate information landscapes.

heading: Future Work
text: This suggests potential benefits from integrating both textual and visual modalities in future research to improve retrieval effectiveness (See Limitations (§8)). 7. 2 Tag Words Baselines Table 4 compares tag word generation across baseline models and extractive methods (TFIDF, TextRank, and KEA) on an English test corpus. Extractive methods, limited to identifying tag words already present in the text, demonstrate varied performance, with TextRank lagging due to its tendency to extract longer phrases, misaligned with the brevity of reference tags. In contrast, TF-IDF and KEA perform moderately better but are outshined by abstractive models like mT5, mT0, and Flan-T5, which excel in generating concise and relevant tag words, surpassing extractive approaches that cannot synthesize new tag words. This distinction highlights the superior capability of abstractive methods in handling the tag word generation task. MultiRAGen Table 5 extends the tag word evaluation results to include baselines mT5, mT0, FlanT5, Gemini Pro, Mixtral and MultiRAGen across a multilingual text corpus. The abstractive baselines exhibit commendable performance, with mT0
demonstrating a noticeably higher gain over mT5 and Flan-T5. This improvement is attributed to the further fine-tuning of mT0 on a multitask framework, enhancing its crosslingual generalization. Although Flan-T5 produces results similar to mT0 and mT5 in English (see Table 4), it falls short of generating any tag words when it comes to certain languages with limited resources. This disparity is particularly noticeable in Table 21. 7