HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew (2024)

Abstract: While large language models (LLMs) excel in various natural language tasks in English, their performance in lower-resourced languages like Hebrew, especially for generative tasks such as abstractive summarization, remains unclear. The high morphological richness in Hebrew adds further challenges due to the ambiguity in sentence comprehension and the complexities in meaning construction. In this paper, we address this resource and evaluation gap by introducing HeSum, a novel benchmark specifically designed for abstractive text summarization in Modern Hebrew. HeSum consists of 10,000 article-summary pairs sourced from Hebrew news websites written by professionals. Linguistic analysis confirms HeSum’s high abstractness and unique morphological challenges. We show that HeSum presents distinct difficulties for contemporary state-of-theart LLMs, establishing it as a valuable testbed for generative language technology in Hebrew, and MRLs generative challenges in general.1

heading: 1 Introduction
text: Recent advances with large language models (LLMs, Brown et al., 2020; Chowdhery et al., 2023) demonstrate impressive capabilities, encompassing diverse tasks such as natural language (NL) understanding and reasoning, including classification tasks such as commonsense reasoning (Bisk et al., 2020) and sentiment analysis (Liang et al., 2022), as well as generative tasks like summarization and dialogue systems (Cohen et al., 2022). However, these impressive achievements are primarily demonstrated for the English language. Our understanding of how these models perform on low-resource languages is limited, as current evaluations are primarily focused on languages with abundant data (Ahuja et al., 2023; Lai et al., 2023).
∗Equal contribution. 1The dataset, code, and fine-tuned models are publicly
available at https://github.com/OnlpLab/HeSum
This concern is particularly relevant for morphologically rich languages (MRLs) such as Hebrew, which is known for their word complexity and ambiguity, leading to processing difficulty (Tsarfaty et al., 2019, 2020). Despite advances in natural language processing for Hebrew, which so far covered tasks as reading comprehension (Keren and Levy, 2021; Cohen et al., 2023), named entity recognition (Bareket and Tsarfaty, 2021), sentiment analysis (Chriqui and Yahav, 2022), and text-based geolocation (Paz-Argaman et al., 2023); a crucial gap persists in the ability to evaluate novel, human-like generated text, as in abstractive text generation.
Abstractive text-generation requires both natural language understanding and reasoning over the input, and the ability to generate grammatically, and in particular morpho-syntactically, correct text, as well as semantically and morpho-semantically coherent, fluent text that conveys consistent meanings. Notably, text-generation models are also prone to ‘hallucinations’ — generating factually incorrect content. These challenges are further amplified in Hebrew due to its morphological richness which leads to a complex realization of sentence structure and meaning.
In order to enable empirically quantified assessment of these aspects of text generation in MRLs, we present a novel benchmark dataset for Hebrew abstractive text Summarization (HeSum). HeSum consists of 10,000 articles paired with their corresponding summaries, all of which have been sourced from three different Hebrew news websites, all written by professional journalists. This curated collection offers several key advantages: (i) High Abstractness – extensive linguistic analysis validates HeSum’s summaries as demonstrably more abstractive even when compared to English benchmarks. (ii) Unique Hebrew Challenges – meticulous linguistic analysis pinpoints the inherent complexities specific to Hebrew summarization, offering valuable insights into the nuanced
26
characteristics that differentiate it from its English counterpart. And (iii) Thorough LLM Evaluation – we conducted a comprehensive empirical analysis using state-of-the-art LLMs, demonstrating that HeSum presents unique challenges even for these contemporary models. By combining high abstractness, nuanced morphological complexities, and a rigorous LLM evaluation, HeSum establishes itself as a valuable resource for advancing the frontiers of abstractive text summarization in MRL settings.

heading: 2 The Challenge
text: Linguistic Challenges in Hebrew Morphologically rich languages (MRLs) pose distinct challenges for generative tasks, above and beyond morphologically impoverished ones such as English.
In MRLs, each input token can be composed of multiple lexical and functional elements, each contributing to the overall structure and semantic meanings of the generated text. For instance, the Hebrew word ‘ !ונתיבמשכו’ is composed of seven morphemes: ‘ו’ (‘and’), ‘ !שכ’ (‘when’), ‘מ|’ (‘from’), ‘ !ה’ (‘the’), ‘ !תיב’ (‘house’), ‘ !לש’ (‘of’), and ‘ !ונחנא’ (‘us’). This has ramifications for both the understanding of MRL texts, a process that necessitates morphological segmentation, and for generating MRL texts, requiring morphological fusion. At comprehension, Hebrew poses an additional challenge due to its inherent ambiguity, with many tokens admitting multiple valid segmentations, e.g., ‘הפקה’ could be interpreted as ‘ה’+‘!הפק’ (‘the’+‘coffee’); as ‘הפקה’ (‘orbit’); or as ‘!Pקה’ + ‘!לש’ + ‘ !איה’ (‘perimeter’+‘of’+‘her’). During generation, the emergence of unseen morphological compositions, where unfamiliar morphemes combine in familiar ways, poses an additional challenge (Hofmann et al., 2021; Gueta et al., 2023). These challenges, coupled with inherent linguistic features like morphological inflections, construct-state nouns (smixut), and flexible word order, create a multifaceted challenge for LLMs in processing and generating Hebrew texts.
The HeSum Task We aim to unlock the comprehension-and-generation challenge in MRL
settings by first tackling the abstractive text summarization task (Moratanch and Chitrakala, 2016), here focusing on Modern Hebrew.
Given an input document in Hebrew, specifically a news article, our goal is to generate a short, clear, Hebrew summary of the key information in the article. In contrast to extractive summarization, here novel morphosyntactic structures need to be generated to communicate the summary.

heading: 3 Dataset, Statistics and Analysis

heading: 3.1 Data Collection
text: The HeSum dataset consists of article-andsummary pairs. The articles were collected from three Hebrew news websites: “Shakuf”,2 “HaMakom”,3 and “The Seventh Eye”.4 These websites focus on independent journalism, providing articles on topics such as government accountability, corporate influence, and environmental issues. Each article on these websites is accompanied by an extended subheading written by a professional editor, that serves as a summary of the content. To ensure data quality, articles that were not in Hebrew, or ones that had particularly short summaries (i.e., the extended subheading was less than 10 tokens) were excluded from the dataset.

heading: 3.2 Linguistic Analysis
text: We examined the linguistic, morpho-syntactic and semantic, properties of the HeSum dataset. For the extraction of syntactic and semantic features, we used DictaBert (Shmidman et al., 2023). Additionally, AlephBert (Seker et al., 2022), a Hebrew monolingual BERT-based encoder model (Devlin et al., 2018), was employed to compute semantic similarity between articles and their corresponding summaries, leveraging the BertScore method (Zhang* et al., 2020). Notably, semantic similarity was performed only on article-summary pairs within the model’s 512-token limit.
2https://shakuf.co.il 3https://www.ha-makom.co.il 4https://www.the7eye.org.il
Table 1 highlights the Hebrew language’s multifaceted complexities as reflected in this task. The notable disparity in the vocabulary size between token and lemma counts underscores extensive morphological richness, necessitating models adept at handling linguistic diversity. The abundance of morphological anaphoric expressions and numerous Hebrew construct-state constructions necessitate advanced models attuned to entity relations that are expressed via Hebrew’s unique morphological traits. Lattice analysis reveals a high degree of word ambiguity (numerous lattice paths), highlighting natural language understanding challenges and the consequent difficulty of accurate tokenization for downstream processing tasks. The substantial document length, necessitate the use of models adept at long-form text processing. Finally, the relatively high semantic similarity score indicates effective information distillation in the summaries.

heading: 3.3 Summarization Intrinsic Analysis
text: To assess the challenges of the HeSum summaries we used three established metrics: (i) Abstractness: the percentage of summary novel n-grams, unseen in the article (Narayan et al., 2018). (ii) Compression Ratio (CMP): the word count in summary S divided by the word count in article A: CMPw(S,A) = 1− |S||A| . Higher compression ratios indicate greater word-level reduction and, subsequently, potentially a more challenging summarization task (Bommasani and Cardie, 2020). (iii) Redundancy (RED): measures repetitive n-grams within a summary (S) using the form: RED(S) =∑m
i=1(fi−1)∑m i=1 fi
where m is the number of unique ngrams in the summary and fi ≥ 1 is the frequency count of a specific n-gram (Hasan et al., 2021).
Table 2 presents a quantitative analysis of HeSum’s summarization characteristics, underscoring its challenges. HeSum demonstrates a high degree of abstractness, with approximately half of its unique vocabulary and over 73% of bigrams unseen in the original articles. Furthermore, HeSum
presents a significant compression challenge, as summaries average less than 5% of the input article length. Additionally, the analysis reveals minimal redundancy within the summaries, with less than 5% repeated n-grams. These findings underscore HeSum’s efficacy in conveying the central ideas of the articles’ information in a novel, distillate, and non-redundant manner. Comparative analysis with established abstractive summarization benchmarks, CNN/Daily Mail (Nallapati et al., 2016) and XSum (Narayan et al., 2018), confirms HeSum’s high abstractness, compression ratio, and low redundancy, even when compared to these datasets.

heading: 4 Experiments

heading: 4.1 Experimental setup
text: Models To demonstrate the complexity of this task, we conducted an evaluation of two LLMs in a zero-shot setting: the GPT-4 model with 32K context window (version 0613), and GPT-3.5-turbo with 16K context (version 0613). To find the most effective prompt format, we tested on the HeSum validation set various prompting strategies, including translating parts of the prompt to English. Ultimately, we adopted the English-translated approach (Brown et al., 2020), where both the instruction and input were translated. The output summaries are strictly in Hebrew. Additionally, to address the limitations of available generative models for Hebrew, we fine-tuned the multilingual mLongT5 model (Uthus et al., 2023) on the HeSum training set with two versions, base (2.37 GB) and large (4.56 GB). mLongT5 is a sequence-to-sequence model based on mT5 (Xue et al., 2020) specifically designed for handling long sequences. Appendix B includes the GPT models’ prompting strategies experiments, and the mLongT5 training details.
Automatic Evaluation Metrics To evaluate the generated summaries with respect to the original texts, we used two standardly-used automatic metrics: ROUGE and BertScore.
ROUGE (Lin, 2004) is a widely-used metric in summarization that measures n-gram overlap between generated summaries and humanwritten references. We calculated ROUGE-1 (unigrams), ROUGE-2 (bigrams), and ROUGE-L scores (longest common subsequence) to capture different levels of granularity. However, n-gram metrics such as ROUGE can struggle with capturing semantic similarity if paraphrases are used. To address this, we also employed BertScore (Zhang* et al., 2020) with AlephBert (Seker et al., 2022) as its backbone. BertScore leverages the pre-trained language model to provide a more semantically meaningful evaluation of the summary.
Human Evaluation To validate the quality of model-generated summaries for the HeSum task, seven independent expert annotators evaluated a total of 186 summaries (62 per model) based on the same set of 62 reference articles. Annotators evaluated each summary using a 1-5 Likert scale (Likert, 1932) based on two key criteria: coherence, which assessed the summaries’ grammaticality and readability, and completeness, which measured the degree to which they capture the main ideas of the articles. To measure the consistency of the annotators’ scores, we calculated Krippendorff’s α (Krippendorff, 2018) for an interval scale, and received an α score of 0.78 which indicates a good inter-annotator agreement rate.

heading: 4.2 Results
text: Quantitative Analysis Table 3 summarizes the quantitative evaluation results. While mLongT5 consistently achieved higher ROUGE scores, a metric focused on surface-level similarity, GPTbased models exhibited superior performance in BertScore, a semantic similarity metric, and in human evaluation scores that assess coherence and completeness. The consistently higher ROUGE scores of mLongT5 might be partially attributed to limitations in the ROUGE metric itself. ROUGE scores favor summaries that closely mimic the source text, even if they lack originality or fluency. Additionally, n-gram-based metrics like ROUGE may discount grammatically correct sentences that convey the required meaning even with morphological or lexical word variations, or changes in word order, compared to the source text.
Furthermore, when comparing ROUGE to human evaluation we found a negative correlation of human evaluation with ROUGE scores. We computed Pearson correlation coefficients (PCC) and found the coefficients to be around -0.16 with highly statistically significant p-values (less than 2.39×10−5), indicating that higher ROUGE scores do not in actuality correspond to human evaluations of good summaries. Similarly, using Kendall’s τ correlations resulted in negative values. Further research is needed for developing automatic summarization metrics that correlate with human scores.
Qualitative Analysis Following the identification of key error categories, we conducted a comparative analysis by randomly selecting 30 summaries generated by each of the three models for the same set of 30 articles. For each model, we then quantified the occurrences of each identified phenomenon within the sampled summaries. The results in Table 4 reveal disparities between the errors of GPT-based models and those of the finetuned mLongT5 on various linguistic phenomena.
The finetuned mLongT5 exhibits pronounced disruptions like repetition (3.33%) and exact copy of sections from the articles (13.33%), which weren’t observed in the GPT-based results. However, the GPT-based models demonstrate errors in morphological phenomena specific to Hebrew, such as incorrect gender and wrong definiteness marking on smixut, indicating that the morphological richness of the language remains a challenge for these LLMs. Additionally, known phenomena of GPTbased models such as “hallucinations” (Cui et al., 2023; Guerreiro et al., 2023) are also observed in our analysis, as is familiar from other languages.

heading: 5 Conclusion
text: This research seeks to fill a critical gap in the field of LLMs assessment for generative creative tasks in MRLs, by presenting HeSum, a new dataset for Hebrew abstractive summarization, that includes 10K article-summary pairs sourced from professional journalists on Hebrew news websites. HeSum offers three key advantages: high level of abstractness in summarization, distinct challenges specific to the Hebrew language, and a thorough empirical assessment of LLMs using this dataset. By integrating these aspects, HeSum establishes itself as a valuable resource for researchers striving to push the boundaries of generative tasks, and specifically abstractive text summarization in Hebrew.
Limitations
Evaluation Metrics based on n-gram matching, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and Meteor (Denkowski and Lavie, 2014), are commonly used for evaluating summarization quality in English. However, these metrics can be problematic when applied to Hebrew text. Hebrew allows for more flexible word order compared to English. Additionally, its morphological richness entails that the same concept can be expressed in multiple ways due to variations in pre-
fixes, suffixes, and root conjugations. Furthermore, Hebrew has variations in spelling words due to missing vowels (Ktiv haser vs. Ktiv male). These factors can lead to n-gram-based metrics overlooking grammatically correct sentences in the generated summary that convey the full meaning even though they show slight differences. Our findings in Section 4.2, which demonstrate a negative correlation between ROUGE scores and human evaluation scores, highlight the limitations of ROUGE evaluation in the context of Hebrew summarization.
Subset of LLMs Although we aspired to evaluate HeSum on a broad range of large language models (LLMs), our current analysis is limited to only two generative models. This might overlook newer models offering potentially superior performance. Additionally, resource constraints prevented us from investigating the behavior of these models in few-shot settings. Having acknowledged that, the timeliness of this resource is uncompromized, as it can be used with contemporary and future models alike, to track advances on this challenge.
Open Access vs. Domain Focus HeSum predominantly comprises articles from news websites, which may bias models’ success in this task towards news-style writing, and may not fully capture the linguistic diversity across different genres and domains. The reason for selecting these domains specifically stems from our ability to obtain a permissive license for the resource, allowing open and free access by the community. However, the websites we have chosen – “Shakuf”, “HaMakom”, and “The Seventh Eye” – deviate from typical news platforms, offering a diverse range of topics that go beyond the typical content found on many popular news websites in Hebrew. This variety ensures that our dataset reflects a broader spectrum of realworld topics.
Dataset Scale vs. Quality In the realm of abstractive summarization, datasets like CNN/Daily Mail (Nallapati et al., 2016) and XSum (Narayan et al., 2018) are commonly employed. These datasets utilize news articles from websites, treating the article content as the document and a corresponding field (often not explicitly intended as a summary) as the summary. However, this approach has faced criticism due to uncertainty about whether the chosen field truly represents a summary (Tejaswin et al., 2021). An alternative approach involves human
summarization, but this tends to result in smaller datasets (e.g., PriMock57, Papadopoulos Korfiatis et al., 2022). To advance Hebrew NLP (and the study of generation in MRLs in general) beyond traditional classification tasks, there is a need for extensive generative datasets. Given the current lack of viable alternatives within the NLP community, we have adopted a similar approach to XSUM, albeit with longer summaries. Additionally, collecting human-generated summaries in lowresource languages presents challenges, including the scarcity of crowdsourcing platforms that support Hebrew. To ensure quality, we meticulously reviewed 100 articles, their subheadings, and brief introductory sentences. Ultimately, we chose subheadings as our summary source because they provided more informative content, capturing additional details from the articles. Furthermore, we filtered out articles with subheadings containing very few tokens (10 or fewer) to ensure our summaries adequately represent the article content.
Ethics
Following the generous permission of “Shakuf”, “HaMakom”, and “The Seventh Eye” — organizations committed to independent journalism, media scrutiny, and transparency in Israel — we were granted the valuable opportunity not only to access and analyze their published articles but also to publish the data for broader research use. This unique collaboration fosters open access and empowers other researchers to build upon the data extracted from their articles and our findings within Hebrew abstraction summarization, expanding knowledge in this important field. Also, we are guaranteed not to have offensive language or hate speech in our data. It should be borne in mind, however, that the opinions or biases reflected in these data may differ from other sources of information (news websites, social media, non-Hebrew news reports, and the like). So, the deployment of technology trained on this resource should be done with care.

heading: Acknowledgements
text: We wholeheartedly thank Naomi Niddam (Editorin-Chief, “Shakuf”), Nir Ben-Zvi (CEO, “Shakuf” and “The Seventh Eye”), Shuki Tausig (Editor-inChief, “The Seventh Eye”), and Ohad Stone (Creative and digital projects, “Shakuf”) for granting us access to and permission to analyze and share their articles for broader research purposes. We
also thank Yoav Goldberg, Ido Dagan, Amir David Nissan Cohen, and the participants of the NLP seminar at Bar-Ilan University for their valuable feedback. This research has been funded by a grant from the European Research Council, ERC-StG grant number 677352, a grant by the Israeli Ministry of Science and Technology (MOST), and a KAMIN grant from the Israeli Innovation Authority, for which we are grateful.

heading: A The HeSum Dataset
text: Collection Protocol Since the websites we collected (Shakuf, HaMakom, and The Seventh Eye) lack archives or RSS feeds, we developed a crawler to systematically navigate through pages, beginning from the homepage and exploring various article links. Leveraging their shared HTML structure, we could efficiently scrape the sites. We excluded pages without textual content, such as multimedia pages or those not in Hebrew. Additionally, articles with summaries of less than 10 tokens were filtered out, as they often lack sufficient detail to be a summary. In addition, all the articles were cleaned from Unicode characters or unrelated content.
Human Evaluation Details We collected annotations from seven volunteered participants aged 25 and above, all with at least one academic degree. The participants were instructed to rate two parameters – coherence and completeness, based on known criteria, as depicted in Figure 1. While completeness measures the extent to which the summary captures all the essential information from the source text, coherence is a more complex metric. According to Reinhart (1980), coherence encompasses three core aspects: (i) cohesion, (ii) consistency, and (iii) relevance. While metrics like BertScore can also assess completeness, automatic evaluation of coherence remains a challenge (Mai-
mon and Tsarfaty, 2023). Therefore, the measurement of coherence is evaluated in this work solely by humans.
Data Analysis Table 5 provides a quantitative analysis of HeSum’s summarization characteristics, highlighting its challenges. The analysis utilizes two tokenization approaches: word-based (above the dashed line) and morpheme-based (below the dashed line). This distinction allows for a deeper examination of the dataset’s abstractness and the influence of morphological features. As the table demonstrates, the number of unique vocabulary items (novel n-grams) decreases when using morpheme tokenization. However, HeSum still exhibits a higher uni-gram count compared to CNN/Daily Mail. This suggests that the task itself inherently involves a high degree of abstractness, and the morphological nature of the data presents an additional challenge.

heading: B Models
text: Fine-tunning mLongT5 The HeSum corpus exhibits characteristics of long-form text, with an average document length of 2,747 tokens and a 90th percentile reaching 5,276 tokens. This extensive content poses challenges for the vanilla mT5 model, whose capacity for processing such lengths may be limited. Consequently, we have fine-tuned the mLongT5 model (Uthus et al., 2023), which is suitable for handling long inputs. The paper presents results obtained with the base version of mLongT5, which is 2.37 GB. We are also releasing
a larger model (4.56 GB).5 The training regimen employed an 8-GPU A100 cluster for 36 hours for the large model, while the base model leveraged a single A100 GPU with 40 GB of memory. Early stopping, utilizing ROUGE-1 as the metric, was implemented to optimize the training process. Further details regarding model performance and implementation specifics are provided in Table 6.
Prompting GPT-based models Here, we leverage the translate-English approach, suggested by (Shi et al., 2022) and (Ahuja et al., 2023), which translates instances from target languages into English before prompting. We decompose the prompt task into three parts: (i) the input article (ii) the instruction (prefix), and (iii) the output. All three parts could be done in either Hebrew or English for the HeSum task. In our experiment, Google Translate API (2023, API, 2023) handled the translation of prompts (input and/or prefix) from Hebrew to English and the translated outputs back to Hebrew for analysis. Testing GPT-3.5 on different configurations of language prompting in the HeSum validation set, we found that the best prompt-language
5https://huggingface.co/biunlp/ mT5LongHeSum-large
configuration is English-English-English (Table 8). We then applied this prompting strategy to both GPT-3.5 and GPT-4 on the test set. The prompt we used depicted in Figure 2.
C Implementation Details
For the intrinsic evaluation of the dataset, we created a Jupyter notebook which computes the different metrics. For computing the n-grams, we used the NLTK package,6 and for loading and processing the data, we used NumPy7 and Pandas.8 For evaluation of the different models, we used the most common ROUGE package for non-English papers,9 and the HuggingFace implementation of Transformers for BertScore.10
6https://pypi.org/project/nltk/ 7https://pypi.org/project/numpy/ 8https://pypi.org/project/pandas/ 9https://github.com/csebuetnlp/xl-sum/tree/
master/multilingual_rouge_scoring 10https://pypi.org/project/transformers/

heading: D Additional Models Performance Analysis
text: Table 7 presents the intrinsic evaluation results for the models, corresponding to the metrics introduced in Section 3.3. Notably, GPT-based models generate text with greater abstractness, as evidenced by their higher count of novel n-grams compared to the fine-tuned mT5. This finding aligns with mT5’s tendency towards repetitive generation, which is further supported by its high RED score and by the qualitative analysis presented in Table 4.

heading: Future Work
text: Having acknowledged that, the timeliness of this resource is uncompromized, as it can be used with contemporary and future models alike, to track advances on this challenge. Open Access vs. Domain Focus HeSum predominantly comprises articles from news websites, which may bias models’ success in this task towards news-style writing, and may not fully capture the linguistic diversity across different genres and domains. The reason for selecting these domains specifically stems from our ability to obtain a permissive license for the resource, allowing open and free access by the community. However, the websites we have chosen – “Shakuf”, “HaMakom”, and “The Seventh Eye” – deviate from typical news platforms, offering a diverse range of topics that go beyond the typical content found on many popular news websites in Hebrew. This variety ensures that our dataset reflects a broader spectrum of realworld topics. Dataset Scale vs. Quality In the realm of abstractive summarization, datasets like CNN/Daily Mail (Nallapati et al. , 2016) and XSum (Narayan et al. , 2018) are commonly employed. These datasets utilize news articles from websites, treating the article content as the document and a corresponding field (often not explicitly intended as a summary) as the summary. However, this approach has faced criticism due to uncertainty about whether the chosen field truly represents a summary (Tejaswin et al. , 2021). An alternative approach involves human
summarization, but this tends to result in smaller datasets (e. g. , PriMock57, Papadopoulos Korfiatis et al. , 2022). To advance Hebrew NLP (and the study of generation in MRLs in general) beyond traditional classification tasks, there is a need for extensive generative datasets. Given the current lack of viable alternatives within the NLP community, we have adopted a similar approach to XSUM, albeit with longer summaries. Additionally, collecting human-generated summaries in lowresource languages presents challenges, including the scarcity of crowdsourcing platforms that support Hebrew. To ensure quality, we meticulously reviewed 100 articles, their subheadings, and brief introductory sentences. Ultimately, we chose subheadings as our summary source because they provided more informative content, capturing additional details from the articles. Furthermore, we filtered out articles with subheadings containing very few tokens (10 or fewer) to ensure our summaries adequately represent the article content