SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
Authors: Elizabeth Clark
Shruti Rijhwani
Sebastian Gehrmann
Joshua Maynez
Roee Aharoni
Vitaly Nikolaev
Thibault Sellam
Aditya Siddhant
Dipanjan Das
Ankur P Parikh
Venue: EMNLP 2023 Main (2023)
Keywords: summarization
evaluation
multilingual
human evaluation
automatic metrics
NLG

Abstract: Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, covering 6 languages, 9 systems, and 4 datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2022). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.

SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
Elizabeth Clark'
Shruti Rijhwani'
Sebastian Gehrmann2 Joshua Maynez' Thibault Sellam' Aditya Siddhant!
Roee Aharoni
Vitaly Nikolaev
Dipanjan Das
Ankur P. Parikh'
GGoogle DeepMind
Google Research
Contact: eaclark@google com
Abstract
Summary: A silent film about the herring industry will be accompanied by a beatboxing performance as part of a tour of Scotland.
Summary: ЗкоHоMbтe Bоay: Чистите syбbi. Peryпярно чистите и откачайте sоyy.
Reliable automatic evaluation of summariza- tion systems is challenging due to the mul- tifaceted and subjective nature of task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted sum- marization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensi- bility, repetition, grammar, attribution, main ideas, and conciseness. SEAHORSE covers languages, 9 systems (including the reference text), and summarization datasets. As re- serve
[Save water. Brush your teeth. Clean pump out water regularly.
Model: PaLM
Model: mt5_small
Dataset: xsum (EN)
Dataset: wikilingua (RU)
Comprehensible
Comprehensible
Repetition free
Repetition free Grammatical Attributable Main idea(s)
Grammatical
Attributable
Main idea(s)
Concise
Concise
Two summaries from the SEAHORSE dataset paired with human ratings for 6 dimensions of quality. In the second summary. the word in bold has gram- matical error in Russian; it uses the wrong aspect. The rater has noted this error, along with several others.
Figure 1:
sult of its size and scope, SEAHORSE both as a benchmark to evaluate learnt metrics,
well large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance out-of-domain meta-evaluation bench- marks: TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2023). We make SEAHORSE dataset and metrics publicly avail- able for future research on multilingual and multifaceted summarization evaluation.'
for many natural language generation (NLG) tasks such machine translation (Kocmi et al., 2021; Freitag et al., 2021a), summarization (Kryscinski et al., 2020), and dialogue (Dziri et al., 2022). Con- sequently, human evaluation often necessary to reliably evaluate NLG systems. However, design- ing human annotation pipelines and obtaining anno- tations is resource inteensvveeeeiieee time-consuming and not casily reproducible. Developing more reliable automatic evaluation metries would make model development faster and more efficient. With this in mind, much recent work has focused on learnt met- rics, i.e., neural classification or regression models that aim to directly predict scores that evaluate the quality of generated text (Zhang et al., 2020; Sel- lam et al., 2020; Rei et al., 2020; Liu et al., 2023), often trained with human ratings.
1 Introduction
Evaluating the quality of generated text is in- creasingly difficult problem as large language mod- els produce text of rapidly improving quality (Rad- ford et al., 2019; Ouyang et al., 2022; Chowdhery et al., 2022). In spite of the improvements, such models often generate text that includes hallucina- tions and other subtle errors (Wiseman et al., 2017; Maynez et al., 2020; Parikh et al., 2020; Ji et al.. 2023; Borji, 2023), making reliable evaluation es- sential for driving progress.
Common n-gram metrics such as BLEU (Pap- ineni et al., 2002) and ROUGE (Lin, 2004) are often not well correlated with human judgments
As result, large-scale collections of human evaluations serve two critical roles in NLG met- ric development: (1) a source of training data for Icarnt metrics and (2) a meta-evaluation benchmark for the performance of these learnt metrics. The
Data and metrics are available at https://goo gle/ seahorse

---

large potential of such datasets is exemplified by the WMT metries shared task, which has enabled rapid development of learnt metries for machine translation that exhibit considerably higher correla- tion to human judgment than BLEU (Bojar et al., 2016; Freitag ct al., 2021b).
lingual generalization potential. To summarize, the contributions of this paper are:
We conduct comprehensive, large- scale hu- man evaluation for summarization across six languages, six quality facets, nine systems and four datasets, resulting in over 96K human- rated summaries. To the best of our knowl- edge, this is the largest multilingual, multi- faceted summarization evaluation resource.
However, outside of machine translation, the ex- istence of such collections of human judgments is limited. Human annotations collected in NLG eval- uations are rarely released (Gehrmann al., 2022), and even when they are, they tend cover a single language (typically English) and are from a single dataset or task, limiting the robustness models and metries trained on these annotations. More- over, such annotations are often based on the test split of existing datasets (e.g., Fabbri et al., 2021; Aharoni al., 2023), which can be problematic for training learnt metrics. This is because the pri- mary advantage of reliable automatic evaluation is
We train learnt metric for each of the eval- uated quality facets, and show that the met- rics outperform strong baselines across our in-domain test set and previously published out-of-domain benchmarks, highlighting the quality of the human annotations we collect and the broad utility of our learnt metrics.
We release our dataset and metrics to foster future work on multilingual, multifaceted sum- marization.
to help model development, e.g., hyperparameter selection on the validation set; therefore a neural metric trained on test set annotations would, in general, lead to overfitting.
2 The SEAHORSE dataset
In this work, we propose SEAHORSE, large- scale dataset for multilingual summarization eval- uation. Our dataset consists of 96K summaries with ratings along 6 quality dimensions: compre- hensibility, repetition, grammar, attribution, main ideas, and conciseness, in 6 languages, for 9 sys- tems (8 models plus the human- authored reference summaries) across 4 summarization datasets (see examples in Figure 1). The training and validation splits of the dataset come from the validation sets of the original summarization corpora to prevent test contamination when training metrics. This permits us to train learnt metric for each qual- ity dimension that can be used for offline model evaluation.
The SEAHORSE dataset consists of 96,645 sum- maries annotated with human ratings along 6 qual- ity dimensions. In this section, we describe the SEAHORSE dataset, how we generated the sum- maries, and how we collected the annotations.
2.1 The summaries
The examples in SEAHORSE are in 6 languages: German (de), English (en), Spanish (es), Russian (ru), Turkish (tr), and Vietnamese (vi). We chose these languages by considering geographic and ty- pological diversity and the availability of summa- rization datasets in those languages.
The summaries are based on articles from 4 dif- ferent datasets in the GEM benchmark (Gehrmann et al., 2021):
We evaluate the metrics learned from SEA- HORSE on the SEAHORSE test set, as well as other existing meta-evaluation benchmarks, such mFACE (Aharoni et al., 2023) and TRUE (Hon- ovich et al., 2022). Our experiments show that the metrics generalize across datasets, tasks, and languages. For example, we demonstrate that al- though SEAHORSE includes data in languages, the resulting learnt metrics achieve strong perfor- mance on the mFFACE benchmark, which consists of 45 languages, exhibiting their zero-shot multi-
XSum (Narayan al., 2018): An English dataset where the task is to generate one- sentence summary of a BBC News article.
XL-Sum (Hasan al., 2021): Similar XSum, the goal of this dataset is to generate a single-sentence summary of BBC news article, but covers 44 languages excluding English.
MI.Sum (Scialom et al., 2020): A summariza- tion dataset obtained from online newspapers in 5 languages.
Pht tps: /ttt///ttmmmm menc-mmtrricss seeetricstasske gi thub. 1o/ "sEAHORSE stands for SummariFs Apnotated with Human Ratings in Six languagEs.

---

language
dataset
articles
annotations
t5 base: The 220M-parameter version of the T5 model (Raffel et al., 2020). (This model English-only, so we only use generate summaries with our en datasets.)
mlsum
3359
7506
de
wikilingua
2999
7085
xsum
894
6651
t5 __base______ 250: under-trained checkpoint, trained for only 250 steps (en only).
b50:_250:
The t5 t5_base model with an
en
xlsum
2433
7884
wikilingua
2383
7804
xlsum
2231
4890
es
mlsum
2235
4857
xxl: The 11B-parameter version of T3 (en only).
wikilingua
2183
5002
xlsum
3298
7254
mt5_small The 300M-parameter version of mT5 (Xue et al., 2021).
ru
wikilingua
2948
7288
xlsum
2186
10627
tr
mt5 115_smmll___ 250: The same mt5 mt5_small model but using the checkpoint after training 250 steps.
wikilingua
770
4791
xlsum
2497
7522
vi
wikilingua
1951
7484
mt5 xxl: The 13B ----parameter mT5 model.
Table The number unique articles and the number of annotated summaries collected from each dataset to create SEAHORSE. Each article summarized by sev- eral different summarization systems, which were eval- uated by human annotators.
palm 1shot: Baramparreettr PaLM model (Chowdhery al., 2022) prompted with one in-domain example.
-000--paraeetrr PaLM model (Chowdhery et al., 2022) finetuned on training data for the respective dataset.
palm finetuned:
WikiLingua (Ladhak et al., 2020): A dataset in 18 languages where the goal is to summa- rize how-to guides from WikiHow.
Our choice of systems covers a range ex- pected system performances in order to capture a large diversity of system outputs and model er- ror types. For instance, an under-trained small model (mt5_small 250) would likely have dif- ferent errors than 1-shot large language model (palm Ishot). Details about how the summaries are generated from these models are Appendix A.
A breakdown of SEAHORSE. across languages and datasets is in Table 1.
For each dataset, we randomly selected articles from their validation splits to comprise the SEA- HORSE training and validation sets, and articles from the test splits to make up the SEAHORSE test set. This distinction important when using the dataset for training evaluation metrics (discussed in $4), because learnt metrics are typically used for model development, and hyperparameter selec- tion is done on the validation set. Using a metric that was trained on test data would lead to overfit- ting. Our dataset construction ensures that learnt metric can be trained on SEAHORSE data without concerns of test set leakage.
2.2 Annotation methodology
For each summary, we collect annotations along 6 dimensions, also referred to as Q1-6:
Q1 comprehensible:
The summary can read and understood by the rater. (If "No," the rest of the questions will be skipped.)
Next, we generate summaries for each article in the dataset. The summaries come from a subset of 9 different systems, which we will denote as follows:
Q2 repetition: The summary free of unneces- sarily repeated information.
Q3 grammar: correct.
The summary is grammatically
reference: The human-authored summaries associated with each article from the original datasets.
Q4 attribution:
All the information in the sum- mary is fully attributable to the source article, as defined Rashkin et al. (2021).

---

Q5 main ideas: The summary captures the main idea(s) of the source article.
model
length
rouge
20% copy
reference
227
20.26
0.00
Q6 conciseness: The summary concisely repre- sents the information in the source article.
t5 2__bae__250
92
20.95
0.00
t5 5_base
101
22.02
0.02
15 _x_x
115
21.65
0.01
For the first 3 questions, annotators see only the summary. The article is revealed when the raters are answering questions 4-6. They can answer "Yes," "No," or "Unsure" to each question and have the option to leave comments flag any issues they see in the article. The annotation interface is shown in Figure
mt5_smal 250
128
21.33
0.02
mt5_small
171
21.81
0.04
mt5_xx/
194
20.77
0.01
palm_Ishot
254
27.34
0.14
palm_finetuned finetuned
194
20.97
0.01
Table The median number of characters (length), ROUGE-L between the summary and article (rouge), and the proportion of summaries where the first 20% of the summary exactly matches the beginning of the source article (20% copy) for all the summaries gener- ated by each model.
Note that our annotation process is reference- less, i.e., the annotator is never comparing a model- generated summary with the reference summary. They evaluate each summary on its own. Given the subjectivity of summarization, we believe this ap- proach allows to adequately reward models that generate relevant summaries that may be different than the reference. Moreover, this enables us to train reference-less metries in $4, which have an added benefit of being able to be used at inference time for re-ranking.
Model reference
Q1
Q2
Q3
Q4
Q5
Q6
0.97
0.97
0.91
0.54
0.68
0.43
15 base_
0.97
0.79
0.91
0.41
0.42
0.25
15, base
0.98
0.92
0.95
0.59
0.59
15 xxl
0.99
0.97
0.95
0.65
0.67
0.51
mi5 smnall
0.43
0.1
mad small
0.86
0.57
0.73
0.36
0.35
0.19
mi5 mi5_xll polm Ishot
0.96
0.94
0.88
0.55
0.65
0.43
0.88
0.85
0.79
0.71
0.57
0.47
The raters are paid, full-time annotators who were trained for this specific task and worked un- der the supervision of project manager. For the non-English languages, the raters are bilingual, pro- ficient in both the annotation language and English. They received a detailed set of instructions in En- glish deseribing the 6 dimensions of quality and positive and negative examples of each in the target language. We created a set of 109 summaries with gold ratings, which we used to train the raters. Each annotator rated 20-30 summaries from this gold set. If the rater performed well on this subset, they were qualified move forward with the annotation task. Otherwise, the annotator received feedback and were asked to complete another 10-20 ratings. This training process was repeated as needed.
palm finctuned
0.98
0.98
0.9
0.69
0.71
0.56
Table 3: The percent of "Yes" responses, broken down by model and question.
The I-shot PaLM model is particularly likely to copy from the article as its output, obtaining the highest ROUGE-L (Lin, 2004) scores between the summary and the article. In 14% of cases, the be- ginning of the 1-shot summaries (the first 20% of the summary) exactly matched the beginning of the reference article.
Table 3 shows the percent of summaries from each summarization system that received positive (i.e., "Yes"" rating from annotators. While there is variation across models and datasets, most sum- maries are rated positively for questions 1-3 (com- prehensibility, repetition, and grammar). The rate of positive responses drops for questions 4-6 (at- tribution, main ideas, and conciseness), indicating that these areas remain a challenge for summariza- tion models. A more detailed break down of the positive response rates is in Appendix B.
A small number of approved annotators were removed during the annotation process, due to is- sues flagged by the annotation team and the authors. The ratings from the removed annotators are not included in the dataset.
3 Dataset analysis
We first analyze the dataset's composition and the quality of the collected annotations. Table 2 con- tains the median length of summaries produced by each model, along with two measures of the over- lap between the summaries and the source articles.
Note that the reference summaries do not always receive the highest rate of positive responses. The
All ROUGE scores in this paper are calculated with SentencePiece tokens: https://gi thub. wtw.coog/////oee google/ sentencepiece

---

Context
[Dxtsuu__ngglshi eng.lshrvaliidatoonees
Evaluaation Rate the summary
SUMMARY
Evaluste summary
health The process of merging two Surrey hospitals it to be scrutinised by the healt overview and senutiny committec.
summary readable and
ARTICLE
Evaluate the summary by itself
Epsom Hospital merger process to scrutinieed The acquistion Epsor Hospital by Ashfard and St Peter's Hospttals NHS Foundation Trust was halted last year. The hosphtal currently part of the Epeom and St Heler NHS Trust Representatives of
Evaluate summary using the article
No
Unaure
both hoaphals will be among those speaking at Thursday meating. The committe has been saked scrutinse the mer process. 'rees uncerteitty Epsom Hospital has now been incorporated into the Better Services Better Value (9SBV) review programme lead South West London NHS. Epsom and St Heller University Hospitals NHS Trust has been looking options for the future the hospital but seid However,
All ohe information provided aummary fully attributable the source
more information
summary that not supported the source article.
decisions had
have been that will not
close under the councilors NHS chiefs. Councillor George Grawford sionthcant uncertalnty. Proposals
Evell
summary article.
"positive BSBV
idea(s)
Choose summary not mention any the artciess points.
summary concisely represents source more summary contains
(opsimma))
3 of
Artdl 8o0 quality
Figure 2: The annotation interface used collect SEAHORSE, First, Question and the summary are shown the evaluator. Once they confirm that the summary is comprehensible, Questions 2-3 are shown. Finally, the article and Questions 4-6 are displayed (as pictured above).
way in which reference texts are collected may limit their quality along some dimensions. For reference
duce low-quality summaries, i.e., summaries that unintelligible to readers. In Table 3, see that mt5_xx produces fewer incomprehensible (Q1) summaries than mt5 3__sall,,, which produces fewer than mt5_small 1_10.22502__ The same holds true for the T5 models, and palm finetuned produces fewer in- comprehensible summaries than palm _sson, re- flecting the expected relationship in quality be- tween model pairs. While these results are aver- aged over the entire dataset, we the same result when controlling for the source article and only considering items that have summaries generated by all systems (see Appendix B).
example, the text that was collected summary may not have been intended to be read as a standalone replacement for the article, and therefore may not be fully attributable to the article, as Rashkin et al. (2021) point out.
We can use the positive response rates to inspect the quality of the dataset by verifying the presence of 3 patterns we expect to in the data: 1) higher positive response rates for better summarization models, 2) high correlation between the responses to Q4&6 and Q5&6, and 3) annotator agreement across the 6 dimensions.
This pattern generally holds across the other dimensions of quality as well. There one no- table exception: PaLM's perfomance on attribution (Q4). For 4 languages, palm_Ishen is more of- ten rated being faithful to the input article than fiintfffnuudd, which is likely due its tendency to copy the article directly.
Order of model quality Our first expectation is that summaries generated by better summariza- tion models should receive more positive responses from raters. We have 3 types of model pairs where we can expect one model to generate better sum- maries than the other: 1) a larger model should outperform a smaller model (the xxl vs. the small model), 2) fully trained model should outper- form under-trained model (the small vs. the amall_250 model), and 3) finetuned model should outperform a I-shot prompted model (the finetuned vs. I-shot PaLM models).
Generally, however, the SEAHORSE ratings cap- ture the relative differences model quality we expect to see when evaluating two models with known differences.
Correlation between dimensions Conciseness (Q6) is related to two other dimensions in our an- notation: attribution (Q4) and main ideas (Q5). A
We compare how often these model pairs pro-

---

These patterns in the annotators' responses are positive indicators about the overall quality of the SEAHORSE ratings. However, the more important Annotator agreement While most items in the dataset were annotated once, we collected 2 addi- tional ratings for a subset of the data to compare annotators' scores. Out of 8,920 duplicated anno- tations, the overall pairwise agreement between raters was 82%. Table 4 breaks down the pairwise accuracy across all languages and questions. Ques- tions 1-3 have higher agreement, while questions 4-6 (which depend more context and have higher degree of subjectivity) have lower agree- ment. A similar trend is reflected in the Krippen- dorff's a values (Krippendorff, 1980, shown in Ta- ble 5), which correct for the probability of random agreement, except grammar (Q3) scores lowest.
qiestion2
0.6
question3 0.05
0.4
question4
0.3
questionS
0.24
0.043
0.38
0.2
0:68
0.1
question2 question3 questions questionS question6
Figure 3: The Pearson correlation between responses for questions 2-6.
Lang
Avg
Q1
Q2
Q3
Q4
Q5
Q6
de 0.84
0.97
0.98
0.95
0.81
0.67
0.66
es
0.82
0.92
0.97
0.83
0.74
0.7
0.74
en
0.81
0.97
0,94
0.95
0.69
0.61
0.69
ru
0.82
0.86
0.97
0.88
0.71
0.73
0.76
test of the dataset's quality is its usefulness for
0.82
0.93
0.96
0.86
0.74
0.7
0.74
developing evaluation metrics, which discuss
vi 0.81
0.95
0.98
0.88
0.68
0.69
in the next section.
avg
0.82
0.93
0.97
0.89
0.73
0.68
0.72
Table The average pairwise agreement, broken down by language and question.
Learning and evaluating metrics with SEAHORSE
summary cannot be considered "concise repre- sentation of the information in the article" if has information that is not in the article (i.e., a "No" response for Q4) or if does not represent the main points in the article (i.e.. a "No" response for Q5), which detail pointed out evaluators in the task instructions. Therefore, we expect Q6 to be positively correlated with both of these dimensions if the annotators understood the task and the rela- tionship between dimensions of quality.
The SEAHORSE dataset is meant to serve both as source of training data for learnt metrics as well as a meta-evaluation benchmark for these metrics. In this section, we evaluate SEAHORSE on these as- pects by looking at how well metrics finetuned with our collected annotations can predict human ratings of generated summaries, both from the SEAHORSE test set and other existing datasets. When training metrics, we a filtered version of the dataset that removes all duplicates and non-Ye: (88,280 total items). We divide the annotations into train/dev/test splits, where the summaries in the train and dev sets are based on articles from the original datasets' validation sets. The test set of SEAHORSE contains summaries of the articles in the original datasets" test sets.
No ratings
In > 99% of cases when the annotator says a summary is not attributable (Q4) or they say it lacks the main ideas from the article (Q5), they also say it is not concise (Q6). This is also reflected in Figure which shows that the strongest correlation between questions is between questions 4&6 and questions 5&6. These results show the pattern we expect to see in the data given the task definition and instructions, and demonstrates the annotators' ability to understand and execute the annotation task.
4.1 Metrics
One way to train a metric using SEAHORSE is to finetune text-to-text generation model, where the model is trained to take article and summary as its input and to output the string "0" or "1' as prediction of the human rating. We finetune m xxx (Xue et al., 2021) with the SEAHORSE train- ing set to do this task, finetuning a separate metric for each dimension of quality. We call this model
Q1
Q2
Q3
Q4
Q5
Q6
0.49
0.87
0.35
0.47
0.4
0.41
Table 5: Krippendorff's s a by question.

---

mISSEAHORSE". More details are in Appendix A. Note that our goal not to train a state-of-the-art metric but rather to evaluate the utility of SEA- HORSE as a resource to train and evaluate such metrics.
to predict the different dimensions of quality in SEAHORSE. The results are shown in Table 6. As expected, we see that the mLSSEAHORSE model is able to predict SEAHORSE ratings better than the baselines according to both our metrics. The repetition (Q2) metric performs the best out of the 6 dimensions, which is also the dimension with the highest pairwise annotator agreement. Examples of summaries paired with human, SEAHORSE, and ROUGE-L can be found in Appendix C. Reducing the size of the base mT5 model from XXL. (13B parameters) to Large (1.2B) drops the performance of the metric. but shows similar trends and still outperforms all baseline approaches. More mt5 LSEAHORSE results can be found in Ap- pendix D.
We compare the performance of miSSEAHORSE to several baselines:
majority caajorityclasss class A majority class baseline (i.c., picking the most frequent class).
ROUGE-L The ROUGE-L score between the article and the summary.
Specifically for the attribution (Q4) task, we consider a third baseline approach; attribution is elosely related to natural language inference (NLI) (Fyodorov et al., 2000; Dagan et al., 2006), and Honovich et al. (2022) show that models fine- tuned on NLI data perform well as faithfulness metrics. Therefore we consider two variants of an NLI-based baseline:
4.3 Evaluation on the mFACE dataset
In addition achieving good performance on the SEAHORSE test set, we would like evaluate how well models trained on SEAHORSE generalize to other multilingual summarization human evalu- ation datasets without any further tuning. This would give evidence that improving on SEAHORSE would lead to better evaluation metrics in general. For this purpose, we choose the mFACE dataset? (Aharoni al., 2023). mFACE contains human evaluations of the XL-Sum test set, which consists of 45 languages on dimensions: qual- ity, attribution, and informativeness. While their definition of attribution is the same as ours (i.e.. following AIS (Rashkin et al., 2021)), their defini- tions of quality (Is the summary comprehensible?) and informativeness (Is the summary good sum- mary of the article?) do not line up exactly with single one of our questions, a misalignment that we expect to occur practice given the lack of stan- dardization of summarization human evaluation. As a result, for each mFACE dimension, we use the SEAHORSE metric for the question that is most similar; attribution clearly aligns with Q4, and for quality and informativeness, we consider Q1 and Q6 to be the closest fit, respectively.
15NLI: An English NLI model proposed by Honovich et al. (2022). T5_xxl fine- tuned on the following datasets: SNLI (Bow- man et al., 2015), MNLI (Williams et al., 2018), Fever (Thorne et al., 2018), Sci- tail (Khot et al., 2018), PAWS (Zhang et al., 2019), and VitaminC (Schuster et al., 2021).
mt5xNLI: A multilingual version, where m'T5_xxl is finetuned on XNLI (Conneau et al., 2018).
We note that since we are operating in the reference-free free setting, other learnt metrics such as BLEURT (Sellam et al., 2020) or BERTScore (Zhang* et al., 2020) are not appli- cable since they measure the similarity between the prediction and reference.
We evaluate the SEAHORSE and baseline metrics in two ways: the area under the ROC curve and the correlation (Pearson's p) between the metric and human scores. These measures are not sensitive to a thresholding value and are also used in the work we compare with (Honovich et al., 2022: Aharoni et al., 2023)..
We evaluate on both the full mFACE dataset (all languages), as well as the 5-language subset that is common to both mFFACE and SEAHORSE (en, es, ru, tr, vi). In addition to our baseline models, also compare to "upper-bound" mT5 _xl model that has been directly trained on mFACE data (m15MFACE)
4.2 Evaluation on the SEAHORSE test set
We first evaluate MESSEAHORSE on the SEAHORSE test set to confirm that a model able to learn
'There are actually 6 different models, one for each ques- tion, we use the notation mISSEAHORSE for simplicity. "https:/ //tg:://ggggg/ccccceeee ace. co/ gotgle/t5..... exl_true. nii_mixture
We obtained the dataset by contacting the authors.

---

Q1
Q2
Q3
Q5
Q6
Metric
P
TOC
P
TOC
P
roc
P
1OC
P
roc
P
roc
uajority_clase
0.5
0.5
0.5
0,5
0.5
0.5
ROUGE-L
0.04
0.54
0.06
0.54
-0.03
0.43
0.13
0.55
0.03
0.53
0.02
0.54
mL5XNL1
0.43
0.78
mt5 LSEAHORSE
0.44
0.88
0.74
0.97
0.37
0.81
0.55
0.82
0.46
0.78
0.45
0.77
MLSSAAORSE
0.52
0.90
0.86
0.98
0.45
0.84
0.59
0.85
0.50
0.80
0.52
0.81
Table Metrics' ability predict SEAHORSE ratings, measured with Pearson's 's coefficient (p) and the area under the ROC curve (roc). mt5 LSEAHORSE is finetuned version of m'15 large; the other mt5 metrics finetune m'15_xxl
mFACE languages Attribuiton
mFACT languages Attribution
Quality
Informatrvioos
Quality
Informativenas
Merric
P
majority class ROUGE-I
0.5
0.5
0.5
0.5
0.5
0.5
Not tralond on mFACE
0.51
0.14
0.06
0.56
0.06
0.52
0.09
0.52
0.09
0.52
mi5xNLI MISERAHORSB
0.45
0.82
0.34
0.74
00.99
0.73
0.50
0.79
0,50
0.81
0.15
0.70
0.52
9.81
0.40
0.74
Trained on iFACE
mISMFACE
0.25*
0.58
0.51
0.81
0.47
0.79
0.35
0.61
0.52
0.82
0.47
0.80
Table 7: Metrics' ability predict mFACE ratings, measured with Pearson's coefficient (p) and the under the ROC curve (roc). The asterisk indicates that the associated model was trained on the training portion of the mFACE dataset.
datasets and tasks beyond summarization. In ad- dition to comparing to the majority class and also performs well on the languages it not finetuned on. mISSEAHORSE even performs com- parably to mt5 on the language subset on all dimensions, and the attribution dimension on the all-language set. MF5CE performs better on quality and informativeness on the all-language set, one would expect, since it has seen supervised data from those languages and dimensions whereas mI5SEAHORSE is upplied in a zero-shot setting.
Results are shown in Table 7. In all but one col- umn, mISSEAHORSE outperforms the other meth- ods that were not trained on the mFACE data and
ROUGE-L baselines, also compare with 1S Results are shown in Table mI5SEAHORSE achieves the best results across the summariza- tion datasets, which is expected as many of these datasets consist of XSum and CNN/DailyMail (Her- mann et al., 2015), the first of which is also a source of the SEAHORSE summaries and the second is a different news summarization dataset. Interestingly, despite only being trained summarization data, mI5ŞEAHORSE performs competitively 15 on the dialogue datasets (BEGIN, Q² and Dial- Fact), indicating its suitability for evaluating tasks outside summarization. 15 performs best on the Fever, VitaminC, and PAWS tasks, which is expected given that the 15 model was trained on these datasets.
4.4 Evaluation on the TRUE Benchmark
Finally, we focus on the attribution dimension of quality, since issues of faithfulness in generated text are increasingly important (Wiseman et al., 2017; Tian et al., 2019; Zhou et al., 2021; Dziri et al., 2022: Ji et al., 2023). The TRUE bench- mark (Honovich et al., 2022) consists of several English datasets across summarization, dialogue, verification, and paraphrasing: FRANK (Pagnoni al., 2021), SummEval (Fabbri al., 2021), MNBM (Maynez
Related work
We briefly review other large-scale datasets of hu- man evaluations of summaries that have been re- leased and compare them to SEAHORSE, but note that most focus on annotating the test data, which would lead to test data contamination when training metrics.
al., 2020), QAGS (Wang et al., 2020), BEGIN (Dziri et al., 2022), Q² (Hon- ovich et al., 2021), I), DialFact (Gupta et al., 2022), FEVER (Thoree et al., 2018), VitaminC (Schuster et al., 2021), and PAWS (Zhang et al., 2019).
SummEval (Fabbri et al., 2021) and Real- Summ (Bhandari al., 2020) are summariza- tion meta-evaluation benchmarks with 12,800 and 7,742 annotations respectively. These benchmarks focus on a single language and single dataset: the
As in the prior section, we apply m15SEAHORSE without any further finetuning to these datasets to assess its ability to evaluate attribution to other

---

SummEEvl
QAGS-C
BEGIN
Q²
DialfFct
VitaminC
majority class
class
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
ROUGE-L
0.35
0.54
mTSSEAHORSS
0.94
0.87
0.83
0.91
0.87
0.84
0.82
0.87
0.91
0.78
0.82
T5NLI
0,90
0.79
0.76
0.77
0.85
0.85
0.83
0.92
0,95
0.98
0.99*
Table 8: Metrics' performance on the TRUE benchmark, measured with area under the ROC curve. 15
T5-xxl model trained mixture of NLI datasets that includes the FEVER, VitaminC, and PAWS training sets (and thus those numbers are indicated with an asterisk).
CNN/DailyMail English summarization dataset. The RoSE benchmark (Liu et al., 2022) contains 22K summary-level annotations across sum- marization datasets, including a subset from the CNNi validation set, and Stiennon et al. (2020) released 65K summary comparisons on the TL;DR dataset (Völske et al., 2017); however, both only consider English summarization tasks. Rashkin et al. (2021) focus on attribution, releasing ~4.5K annotations from English summarization, table-to-text, and dialogue datasets; Gekhman et al. (2023) also release attribution annotations for 1.4M summaries, but the labels are aechine-generater rather than human-annotated. GENIE (Khashabi et al., 2022) released 17K human evaluations across 5 tasks that includes one English summarization task (XSum).
the public release of SEAHORSE enables further research on these topics.
Limitations
The summaries in this work are in 6 languages, and the selection of these languages was based on the number of datasets and articles available for each language. We would like future work explore the incorporation of low-resource languages, perhaps with the use of crosslingual and fewshot summa- rization systems. While the raters we worked with in this project went through several rounds of in- structions and training, there a degree of subjec- tivity inherent in the 6 text quality evaluation tasks and human ratings noisy, each individual rater may interpret and rate qualities slightly dif- ferently. Finally, the m'T5-based metrics presented in this work primarily serve as demonstration of the potential of the SEAHORSE data for developing summarization metrics; they have not optimized via thorough hyperparameter search, comparing different modeling architectures approaches, etc. We hope the dataset and experimental results will provide starting point for this type of exploration in the future.
The only other multilingual summarization eval- uation dataset, the best of our knowledge, is mFACE (Aharoni et al., 2023), which has annota- tions for 31,500 summaries covering a broader set of languages (45 languages). mFACE focuses on one dataset (XL-Sum) and smaller set of models than SEAHORSE. In $4 we use mFACE as com- prehensive oft-of-domain evaluation set, and view it as complementary to SEAHORSE, which aims to provide large-scale and diverse training data for metrics.
Ethics Statement
This work relies on the efforts of human evalua- tors, who were compensated for their work. The summaries in this work machine-genent generated and should not be treated as truth; they may contain misleading or incorrect information. None of the human ratings capture this dimension of the text, as our quality dimensions focus on the relationship
6 Conclusion
In this work, we present SEAHORSE, large-scale multilingual, multifaceted dataset for summariza- tion consisting of 96K human annotations of sum- maries. Due to its size and scope, SEAHHRSE en- ables the training and evaluation of learnt metrics across several quality dimensions. Our results show that SFAHORSF-tree -r-taied metrics not only achieve strong performance
between the summary and the source article, not broader set of information or perspectives. For example, if an article contains factual error, summary that contains the same error should be rated as "Yes" for Q4 (attribution) because is consistent with the article. We used summarization models of varying quality in this work, but all are imperfect and their output should be treated with caution.
test set but also gen- eralize to other external and out-of-domain bench- marks: mFACE and TRUE. In the future, are interested in exploring how SEAHORSE can be used more directly to improve the quality summariza- tion models and metrics, and hope this paper and

---

Acknowledgements
Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaIM: Scaling language modeling with pathways. arXiv preprint TrxXi:2204.0231.
We would like to thank Ashwin Kakarla and his team for help with the annotations, as well as Slav Petrov, Hannah Rashkin, and our EMNLP review- ers for their feedback on the paper.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel Bowman, Holger Schwenk. and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguisties.
References
Roee Aharoni. Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella La- pata. 2023. Multilingual summarization with fac- tual consistency evaluation. In Findings of the As- sociation for Computational Linguistics; ACI. 2023, pages 2562-3591, Toronto, Canada. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evalu- ating Predictive Uncertainty, Visual Object Classifi- cation, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, pages 177-190. Springer.
Manik Bhandari, Pranav Narayan Gour, Atabak Ash- faq. Pengfei Liu, and Graham Neubig. 2020. Re- evaluating evaluation
text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online, Association for Computa- tional Linguistics.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue systems: The BEGIN benchmark. Transactions of the Association for Computational Linguistics, 10:1066- 1083.
Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 met- rics shared task. In Proceedings of the First Con- ference Machine Translation: Volume 2, Shared Task Papers, pages 199-231, Berlin, Germany, As- sociation for Computational Linguistics.
Alexander R. Fabbri, Wojeiech Kryšcińskii Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Asso- ciation for Computational Linguistics, 9:391-409
Ali Borji. 2023. A categorical archive of chatGPT fail- ures. arXiv preprint wrxiv:2302.0349.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey, 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460-1474.
Samuel Bowman, Gabor Angeli. Christopher Potts. and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
Markus Freitag. Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondrej Bojar 2021b. Results of the WMT21 met- rics shared task: Evaluating metrics with expert- based human evaluations TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733-774, Online. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay
Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. Proceed- ings of the 2nd Workshop on Inference in Computa- tional Semantics (ICoS-2).
Sanjay Ghe- mawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson. Liam Fe- dus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai. Thanumalayan Sankara- narayama Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta,
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das. Kaustubh Dhole. Wanyu Du, Esin Durmus, Ondfej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tat- sunori Hashimoto, Yufang Hou, Yacine Jemitee, Harsh Jhamtani, Yangfeng Ji. Shailza Jolly, Mi- hir Kale, Dhruy Kumar, Faisal Ladhak, Aman

---

Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan- Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo. Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrík Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Nat- ural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online. Association for Computational Linguistics.
q²: Evaluating factual consistency in knowledge- grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7856-7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallu- cination in natural language generation. ACM Com- puting Surveys, 55(12)1-38.
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Ha- jishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022. Prompt wayward- ness: The curious case of discretized interpretation of continuous prompts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 363 3631-3643, Seattle, United States. Association for Computational Lin- guistics.
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the cracked foundation: A survey of obstacles in evaluation practices for gener- ated text. arXiv preprint (xXiv:2202.06935.
Zorik Gekhman, Jonathan Herzig. Roce Aharoni, Chen Elkind, and Idan Sxpektor, 2023. TrueTeacher: Learning factual consistency evaluation with large language models. arXiv preprint 1111
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. ScitaiL: A textual entailment dataset from science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. DialFact: A benchmark for fact-checking in dialogue. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I; Long Papers), pages 3785-3801, Dublin, Ireland. Association for Com- putational Linguistics.
Tom Koemi, Christian Federmann, Roman Grund- kiewiez, Marcin Junczys-Dowmunt, Hitokazu Mat- sushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478 478-494, Online. Association for Computational Linguistics.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL. sum: Lcrge-scale multilingual abstractive summa- rization for 44 languages. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 4693-4703, Online. Association for Computational Linguistics.
Klaus Krippendorff. 1980. Content analysis: An intro- duction to its methodology.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346. Online, Association for Computa- tional Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espcholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in Neural Information Processing Systems, 28.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kath- leen McKeown. 2020. Wikil.igga:: A new bench- mark dataset for cross-lingual abstractive summa- rization. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4034- 4048, Online. Association for Computational Lin- guistics.
Or Honovich, Roee Aharoni, Jonathan Herzig, Ha- gai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re- Re-evaluating fac- tual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document- Document-grouadeed Dialogue and Conversational Question Answering, pages 161-175, Dublin, Ireland. Association for Computational Linguistics.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- rion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-Eval:

---

NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303.16634
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Re- search, 21(14)):1-67.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan. Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2022. Revisiting the gold standard: Grounding summarization evaluation with robust hu- man evaluation. arXiv preprint arXiv:2212.0990
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc. and David Reitter. 2021. Measuring attribution in natu- ral language generation models. arXiv preprint arXi::2112.12870
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 1906-1919 On- line. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Associa- tion for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Adam Roberts, Hyung Won Chung, Anselm Lev- skaya, Gaurav Mishra, James Bradbury. Daniel An- dor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. 2022. Scaling up mod- els and data with t5x and seqio. arXiv preprint ar arXiv:2203.17168 17189.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with con- trastive evidence. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 624-643, Online, Asso- ciation for Computational Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Con- ference of the North American Chupter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 1829, Online. As- sociation for Computational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier. Benjamin Piwowarski, and Jacopo Staiano. 2020. MUMU The multilingual summarization corpus. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 8051-8061 Online, Association for Computa- tional Linguistics
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meet- ing of the Asvociation for Computational Linguistics, pages 7881-7892 Online, Association for Computa- tional Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics. pages 31 1-188, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. In Ad- vances in Neural Information Processing Systems, volume 33, pages 3008-3021, Curran Associates, Inc.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to- text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 1173-1188 1186, On- line. Association for Computational Linguistics.
James Thome, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019, Lan- guage models are unsupervised multitask learners. OpenAl blog, 1(8):9.

---

Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting halluci- nated content in conditional neural sequence gener- ation. In Findings of the Association for Computa- tional Linguistics: ACI-IJCNLP 2021, pages 1393- 1404, Online. Association for Computational Lin- guistics.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh, 2019. Sticking to the facts: Con- fident decoding for faithful data-to-text generation. arXiv preprint arXiv: 1910.08644.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein, 2017. TL:DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen, Denmark. Association for Computational Linguistics.
A Training details
The summarization models were trained on the training split of each summarization dataset, with the exception of palm_ Ishot, which generated a summary given a single example from the dataset and the input article. The checkpoint for each model was selected using performance on the val- idation set of each respective dataset, except for t5 base___ae_______________a_aaa and mt5 1i5_small_2200 __0, which were only trained for 250 steps. The input length for the T5 and mT5 models was set to 1024, and 2048 for PaLM. The target length
Alex Wang. Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 5008-5020, Online.
Proceedings of
Association for Computational Linguisties.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
512.
The SEAHORSE metrics were trained on the SEA- HORSE training split, and the best checkpoint was selected based performance the validation set. A separate metric trained for each of the 6 di- mensions of quality. We used only "Yes" and "No" ratings for training and testing the SEAHORSE met- rics. The input length for the learnt metries model is 2048. The article and summary are separated with "premise:" and "hypothesis:" tags, respec- tively, to be consistent with Honovich et al. (2022), All training and inference was done with the t5x framework (Roberts al., 2022) and run with TPU accelcrators.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 683-498, Online. Association for Computa- tional Linguistics.
B Rate of positive responses
Table shows a detailed breakdown of the propor- tion of responses that were positive (i.e., "Yes"), divided by language, dataset, model, and question. Summaries in languages other than English and produced by smaller models tend to have lower scores, indicating good directions for improving our summarization systems.
Tianyi Zhang* Varsha Kishore*, Felix Wu*, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In Interna- tional Conference on Learning Representations.
While most articles in the dataset were assigned to subset of the summarization models, some articles were summarized by all 9 summarization systems (or 6 systems for the non-en languages that did not use the T5 models). Specifically in the test set, there were 543 articles that were summarized by all summarization systems. Table 10 shows the positive response rate across those summaries.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume 1 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.

---

C SEAHORSE example summaries and scores
Figure 4 shows 3 summaries from the SEAHORSE dataset, along with ratings for the attribution (Q4) dimension from the human raters, MISSEAHORSE. and ROUGE-L.
D Comparison between mTT5 ma5__arge and mT5_xxl
Table 11 compares the results of two versions of mT5 finetuned SEAHORSE data, mT5 lalge and
m'55_ xxl, the SEAHORSE and mFACE test sets. Scores are generally close between the models, but mT5_xxl outperforms the large metric all cases except one.

---

Take deep, slow breaths through your nose you feel yourself getting emotional. This will help you calm down and give you something concrete to focus on. [..] Your facial muscles become tense when you cry, and it's natural for you to frown beforehand. Try to relax your frown and release all the tension from your face. You don't have to smile-you're at funeral, after all-but relaxing your face help keep you from crying. If you feel your facial muscles tensing up, take couple deep breaths and relax your shoulders. Relaxing other parts of your body will help you relax your face well. [...]
Article:
Summary: Make your face feel emotional. Relax your face. Relax your face.
Rater: 0
m 0.15
ROUGE-L: 33.59
Comments: While the second (and repeated third) sentence this summary is supported by the article, the first sentence is not, and both the rater and mt5 rate it accordingly. ROUGE incorrectly rates it highly (in the 80th percentile of its scores).
Article:
UN human rights chief backs Apple in FBI encryption row - The FBI has ordered the tech giant to assist it with unlocking iPhone used by San Bernadino gunman Syed Farook. Prince Al Hussein said the law enforcement agency "deserves everyone's full support" in its investigation. However, encryption was essential in the interests of freedom, he added. "There are many ways to investigate whether or not these killers had accomplices besides forcing Apple create software undermine the security features of their own phones," he said in statement. "It is potentially a gift to authoritarian regimes, as well as to criminal hackers. "Encryption and anonymity are needed as enablers of both freedom of expression and opinion, and the right privacy. Without encryption tools, lives may endangered." [...]
Summary:
The UN human rights chief has backed Apple in row with the over encryption.
Rater:
mt5gEaHORSE 0.98
ROUGE-L: 15.66
Comments:
This summary is rewording of the first line of the article, is attributable the article, which the rater and 5t5 agree with. ROUGE rates it low, however (in the 10th percentile).
Wyoming diplodocus skeleton bought for Denmark museum - [...] Mystery had surrounded the buyer, but the Denmark museum confirmed on Tuesday it had acquired the skeleton. The museum bought the female dinosaur, nicknamed Misty, for $400,000 ($652,000), following a donation from the Obel Family Foundation. [..]] Obel Family Foundation chairman Christen Obel said: "I think it's quite obvious and right that the Natural History Museum of Denmark should own dinosaur. 'So when we suddenly had the opportunity to give the museum this early Christmas present, we jumped at the chance. "Misty is an iconic object that fascinates us, and the dinosaur will certainly create value for the museum for many generations come."
Article:
Summary: diplodocus skeleton has been bought by the Natural History Museum Denmark.
Rater:
mt5 0.85
ROUGE-L: 15.82
Comments:
Though the first line of the article seems to contradict the summary (for the museum vs. by the museum), the article later clarifies that was fact the museum that bought the dinosaur, so the rater and mt5 are correct. Only the ROUGE metric rates it low (in the 10th percentile).
Figure 4: Example summaries and ratings from the human raters, MISSEAHORSE, and ROUGE-L for attribution (Q4).

---

DE "YES" RATB
RU "VES" RATB
Muckel
Q1
Modal
Q4
Q
relenemerr
0.99
0.99
0.88
0.33
0.51
0.35
relereoor
0:99
0.95
0.86
0.48
0.83
2215 suail 250
0.83
0.58
0.59
0.68
0.41
0.29
RI3, sanell 250
0,5
0.21
0.29
0.2
0.25
0.1
mA5 small
0.97
0.85
0.87
0.68
0.47
0.38
MAS smaall
0.73
0.55
0.57
0.27
0.47
0.19
misum
2115 xxl
0.98
0.97
0.5
0,8
0.59
0.5
xtrum
R45
0.95
0.91
0.85
0.44
0.76
0.4
polm Ishor
0.93
0.93
0.
0.53
0.73
0.66
polmn Jshoe
0.89
0.89
0.82
0.78
0.66
0.6
polen Rinetured
0.89
0.82
0.73
Rinetued
0.58
0.81
0.77
0.39
refaresce
0.96
0.65
0.63
0.49
reforesce
0.97
0.9
0.65
0.46
221, somil 250
0.82
0.75
0.75
0.08
0.00
0.01
RIS, jwmllllll 250
0.72
0.22
(106
0.31
0,08
0.01
a1S senell
0.91
0.35
0.84
0.4
0.36
0.16
RAS sunell
0.83
0.26
0.75
0.39
0.17
0.09
wikllingua
2155 xxd
0,557
0.91
0.5
0.69
0.62
0.49
wikilingus
0.96
0.92
0.85
0.54
0.61
0.65
point Jshor
0.76
0.72
0.73
0.63
0,53
0.42
polnt, Jshoc
0.92
0.86
0.86
0.74
0.48
0.36
pulm fineluned
0.98
0.97
0.95
0.74
0.79
0.65
pulm finetineed
0.91
0.91
0.89
0.66
0.59
0.51
0.75
0.31
0.44
0.89
0.33
0.92
0.55
0.45
total
0.86
0.73
0.78
0.5
0.53
0.35
"YES" RATE
TR
"YES' RATI
Dotaact
Model
Q1
Q2
Q3
Q4
Q5
Q6
Dotasct
Model
Q1
Q2
Q3
Q4
Q5
06
reformnue
1.0
1.0
(136
0.54
0.66
0.47
reforenee
1.0
1.0
0.88
0.46
0.82
0.43
15. hase 250
0,96
0.88
0.89
0.32
0.43
0.24
MAS smalll 250
0.59
0.41
0.34
0.23
0.33
0.17
15 Dasa
0.96
0.91
0.1
0.42
0.5
0.32
RA5 Janall
0.85
0.72
0.57
0.35
0.49
0.29
axl
0.99
0.98
097
0.58
0.61
0.57
Xlam
0,99
0.95
0.85
0.54
0.78
0.09
250
0.17
0.09
palia__ Jahoc
0.83
0.75
0.72
0.66
small
0.37
0:21
polma,
0.81
0.57
ad5 xxd
0.46
0.58
0.37
0.57
088
07
0.68
0042
polm. Ishot
0.97
0.96
091
0.48
0.55
0.30
reforence
0.94
0.92
0135
0.3
0.73
0.96
polis finetused
0.99
0.99
0.99
0.6
0.66
0.51
maS small 250
0.9
0.34
0.79
0.38
0:2
0.12
Gntall
0.07
0.87
0.87
0.42
0.5
01.3
MA5 smnall
0.82
0.53
0.57
0.3
0.18
0.05
referemor
1.0
1.0
0197
0.6
0,71
0.51
wikilingu
0.91
0.89
0.77
0.44
0.61
0.35
250
0.39
0.59
0.43
Jahoc
0.84
0.77
0.76
0.7
0.63
0.89
(5 huae
0.96
0.65
0.68
0.52
finetuned
0.94
0.62
t5_xcl
0.60
0.72
0.54
30Gal
0:39
0.72
0.5
0.33
xleum
miS small
miS Jenall 250
0.74
0.89
0.78
0.59
0.79
0.39
0.4
0.44
6.34
0.15
0.29
total
0.88
0.78
0.72
0.47
0.61
0.39
ma xsl
0.90
0.95
099
0,62
0.71
0.52
VI
RATE
polm. Jshor
0.95
0.95
0.2
0.73
0.68
0.58
Dusaset
Model
Q]
Q2
Q3
Q4
Q5
416
polm finclunell
1.0
1.0
1.0
0/62
0,90
0.45
reforeace
0,86
0.85
0.81
0.37
0,66
0.35
total
0.9
0.
0.57
0.54
m45, _ew.lllll 250
0.49
0.33
0.39
0.00
0,17
0.06
115 seaall
0,7
0.57
0.59
0.2
0.41
0.15
(5 Julueee
0.31
0.26
0.09
0.64
0.81
0.36
15. haur
0.98
0.89
0.3
0.67
0.57
0.45
polm. Tshox
0.92
0.98
0.05
092
0.68
0.61
0.51
polm fincturral
0.90
0.90
0.93
0.52
0.67
0.42
wikllingu
215 smaill
mi5 small 250
0,95
0,96
0.27
0.65
0.8
091
0:45
0.52
0.37
0.09
0.19
0.02
total
0.50
0.8
0197
0.75
0175
0104
0.37
0.37
0.21
0.51
0.28
0.S1
xxl
0.96
0.2
0.62
0.64
0.09
mAS, www.llllaaaeeee 250
0.82
0.26
0.78
0.25
0.1
0.06
palnt__shee Jshor
0.8
0,58
0.49
mAS small
0,91
0.28
0.87
0.31
0.25
0.13
pollss finctunca
0.63
0.54
wikllings
0.97
0.95
0.03
0.42
Tocal
0.8
0.59
0.43
polm.
0.16
tonal
0.99
0.86
0.9
0.33
0.53
0.38
polcs, Roctuned
0.99
0.95
0.96
0.73
0.39
0.33
lotal
0.91
0.7
0.86
0.49
0.39
0.21
RATE
Intal
0.83
0.72
0.79
0.43
0.45
0.27
Dosaset
Model
QI
Q2
Q23
Q4
Q5
reforence
0.99
0.99
0.88
0,60
0,49
0.33
ma5 jeall. 250
0.78
0.69
0.63
0.38
0.2
0.11
0.94
088
0.61
0.38
0.25
mlamm
0.51
0.90
polma Johor
0,73
0.72
0.41
0.45
0.32
pwim finetumal
0.99
0.99
002
0.92
0.78
0.75
tokal
0.9
0BT
0.7
0.63
0.57
0.35
0,99
0.90
(15h
0.31
0.49
(1.11
m655 Jenall 250
0.64
0.44
0.55
0.17
0,16
0.07
m45 small
0.63
0.71
0.23
0,28
0.12
skem
0.43
0.23
polm. Jshor
polin Roctaued
0.99
0.98
0.5
0.66
0.41
total
0.88
0L81
083
0.39
0.45
0.28
0.00
0197
(19h
0.3
0.83
01.55
mit Jemall 250
0.75
0.61
0.73
0.16
0.08
0.03
mi5 small
0.95
0.37
0.92
0.42
0.28
0.11
wikliings
0.57
0.64
polm. Jshor
0.62
polin, Roottuee
0.99
0.54
0.84
0.74
lotal
19
0.35
0.5
Ina
0.01
0881
077
0.32
0.48
0.33
Table 9: The percent of "Yes" responses, broken down by language, dataset, model, and question number.

---

Model
Q2
Q3
Q4
Q5
reference
0.97
0.96
0.91
0.46
0.66
0.39
base_250
0.96
0.9
0.9
0.44
0.48
0.31
hase
0.98
0.95
0.94
0.51
0.58
0.38
15 15_xxl
0.99
0.98
0.95
0.67
0.72
0,58
mt5 small 250
0.64
0.45
0.5
0.26
0.24
0.12
mt5_smn small
0.81
0.67
0.68
0.34
0.37
0.2
mt5 xxl
0.95
0.94
0.89
0,5
0.66
0.37
palm Ishot
0.93
0.88
0.86
0.75
0.56
0.44
palm finetuned
0.98
0.97
0,91
0.66
0.73
0.56
Table 10: The percent of "Yes" responses for the set of articles that have summaries generated by all systems, broken down by model and question.
Q1
Q2
Q3
Q4
Q5
Q6
Dataset
Metrie
p
roc
P
roc
P
roc
roc
roc
p
roc
SEAHORSE
mt5
0.44
0.88
0.74
0.97
0.37
0.81
0.55
0.82
0.46
0.78
0.45
0.77
m15 XXL
0.52
0.90
0.86
0.98
0.45
0.84
0.59
0.85
0,50
0.80
0.52
0.81
mFACE
mt5 I.
0.14
0.77
0.48
0.78
0.32
0.70
5 langs
mt5 XXL
0.09
0.73
0.50
0.79
0.50
0.81
mFACE-
mi5 L
0.13
0.68
0.46
0.77
0.36
0.71
all langs
mi5 XXL
0.15
0.70
0.52
0.81
0.40
0.74
Table II: Metrics' ability to predict SEAHORSE and mFACE ratings, measured with Pearson's 's coefficient (p) and the area under the ROC curve (roc). QI maps to "Quality" in the mFACE dataset, Q4 "Attribution." " and Q6 "Informativeness." mt5 Li5_Lih is SEAHORSE-finetuned version of mT5 @T5_large; X15_XXL is SEAHORSE--inctuned version of mT5 xxl.


Reviews:
[{"review_id": "iyzd4r53Mp", "reviewer": "Reviewer_A7j7", "summary": "", "questions": "A. What is the reason behind using binary responses for the questions? Some of the questions can be better described if evaluated on multi-level scale such as Q4, Q5 and Q6. Also, Q6 is highly correlated with Q4 and Q5, why did you decide to keep it?\n\n2. Are the models used to generate schemas finetuned further or are used as is? Please provide more details on different models such as whether/how they are trained, data used for training, etc.\n\n3. Please provide the statistics of the instances you used to generate summaries for such as the number of unique articles you used. What is the percentage of articles from different datasets and what is the distribution of articles from different languages? Do different datasets have similar articles and did you remove the duplicates?\n\n4. The first three questions are almost always positive showing that current LLMs are very good at generating readable, coherent and grammatically correct summaries. Why do you include them in the benchmark and can you think of a way to evaluate these metrics in a more fine-grained fashion? Maybe that can reveal aspects which systems still struggle.\n\n5. Why did you only use one annotator for each question and not multiple annotators and then doing a majority vote? This could further improve the quality of responses.\n\n6. What is the justification for using Pearson’s coefficient and ROC for evaluating the learned metrics?\n\n7. For mFACE informativeness metric, why did you use Q6 as a proxy and not Q5? Based on your description for Q5, that seems to be a better alignment.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments and questions; they have been very helpful for identifying the parts of the paper that require more discussion and motivation. Based on your feedback, we will go back and better motivate the design choices in the dataset, especially around the binary scale and the dimensions of quality. Here are more detailed responses to the issues you raised in the \"Reason to reject\" section:\n- Reasons to reject #1: We used two families of models to generate the summaries in the dataset: (m)T5 and PaLM. We included versions of these model architectures in different sizes, from different training checkpoints, and with/without finetuning to make sure there is enough diversity of summarization quality in the dataset. We experimented with other model architectures such as LSTM-based seq2seq models, but found the quality too low and inconsistent to be worth annotating. We will add more motivation in the paper about our choice of models.\n- RR2: Multi-scale scoring systems provide more nuance, but they also add complexity and noise. Annotators may interpret the scale differently, and defining every point on a Likert scale for each dimension is not a trivial task. E.g., what should be considered a “3” vs. a “4” grammatical error on a 5-point Likert scale? Given our goal of using this dataset as training data for new metrics and to simplify the annotation task, we decided to treat each dimension as a flag, so if a problem is present in a summary, the annotator gives it a negative score.\n- RR3: While we agree in principle that Q1-3 likely have less future relevance, as demonstrated through their high scores, in many settings these dimensions are not trivial for models. While the largest, finetuned models (m/T5 XXL and PaLM-finetuned models) generally do very well on Q1-3, the smaller and 1-shot models do not. Performance also varies by language, as seen in Table 9; English models receive higher positive rates for Q2&3 than languages like Russian, Turkish, and Vietnamese. LLMs are known to be less fluent in many non-English languages and the multilingual nature of our dataset is a key contribution of our work. We consider it important to have benchmarks for these dimensions of quality for building metrics that are subject to computational or resource constraints, multilingual settings, or where it is important to make sure quality along these dimensions hasn’t degraded, e.g., RL models (Paulus et al., 2018).\n- RR4: To clarify, is the “system” here referring to the trained metrics? We will expand Figure 4 to include more examples of false positives and negatives for the trained metrics; thank you for the suggestion, and please let us know if there are other specific analyses you would like to see – we will incorporate them in the final version of the paper.\n\nPlease find answers to the \"Questions for the Authors\" below:\n- Question #1: See RR2 above about the decision to use binary responses. Q6 is the dimension that captures if the text is actually a good summary, i.e., whether the main ideas (and only the main ideas) from the article are accurately represented in the summary. A negative Q4 or Q5 score should result in a negative Q6 score (as described in the paper and in the annotator instructions), but Q6 can be negative even if both Q4 and Q5 are positive.\n- Q2: We are not sure what “schemas” refers to here, but if it is referring to summaries: yes, all the summarization models (other than PaLM-1shot) are finetuned on the training splits of the respective datasets. More details about the finetuning process are in Appendix A.1; we will move this information into the main paper in the next version and elaborate on it, as space allows. \n- Q3: Thank you for the suggestion! There are 32,367 unique articles in Seahorse. Here is a breakdown of the number of articles by dataset:\n| Language | Dataset | Count |\n|------|:---:|:--:|\n| de | wikilingua | 2999 |\n| de | mlsum | 3359 |\n| en-US | wikilingua | 2383 |\n| en-US | xsum | 894 |\n| en-US | xlsum | 2433 |\n| es-ES | wikilingua | 2183 |\n| es-ES | xlsum | 2231 |\n| es-ES | mlsum | 2235 |\n| ru | wikilingua | 2948 |\n| ru | xlsum | 3298 |\n| tr | wikilingua | 770 |\n| tr | xlsum | 2186 |\n| vi | wikilingua | 1951 |\n| vi | xlsum | 2497 |\n\nAll languages consist of summaries from at least 2 datasets: one with news articles (xsum, xlsum, mlsum) and one with how-to articles (wikilingua). These are from two separate sources and are two different tasks; the articles are not similar. We have confirmed there are no duplicates between the news article datasets either. \n- Q4: Please see RR2&3 above.\n- Q5: We used one annotator for each data point to cover a broad, diverse, and large set of summaries given the resources and constraints we had. However, Seahorse also contains a subset of the data with multiple annotations (~9K items, as described in section 3) for measuring dataset quality, which could be used as a high quality subset by using majority voting, as you describe. \n- Q6: These are the key metrics used in the papers we compare with; ROC is used in Honovich et al. (2022) (TRUE benchmark), and Pearson’s coefficient in Aharoni et al. (2023) (mFACE benchmark). Unlike other classification metrics, these metrics are not sensitive to a thresholding value. We will clarify the choice of evaluation metrics, along with appropriate citations, in the next version of the paper.\n- Q7: The definition of “informativeness” in the mFace paper is “Is this summary a good summary of the article?” As discussed in Q1, this is closest to what we capture in Q6: is the generated text actually a good summary? Thank you for pointing this out; we will clarify it in the next version of the paper. \n\nReferences:\n- Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. Multilingual Summarization with Factual Consistency Evaluation. ACL 2023.\n- Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating Factual Consistency Evaluation. NAACL 2022.\n- Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. ICLR 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A large-scale dataset of summaries and human-generated ratings of them on 6 dimensions.\n2. In-domain and out-of-domain analysis on the performance of the models trained show the quality of the annotations in the dataset by outperforming the baselines.\n3. The dataset can be helpful as it can ease the automatic evaluation of qualitative aspects of generation tasks (text summarization in this case) as well as being a test bed for evaluating current metrics.  The experimental results show that the metrics can outperform baselines both on in-domain and out-of-domain sets.", "reasons_to_reject": "The models that are used to generate the summaries are not structurally diverse, 6 of the models are from T5 family and the other two are variations of the palm model. Though the number of parameters is different, the types of errors made by these systems might be similar and therefore not many diverse patterns will be learned for predicting each proposed metric.\n\nThe six questions about the quality of the generated summaries are only measured on a binary scale. For some of these evaluation questions, a more fine-grained multi-scale scoring can lead to better evaluation of the quality. \n\nIt is further shown in the experiments that the first three questions are trivial as LLMs are very good at generating readable, coherent and non-repetitive text. This somehow narrows the contributions of the provided dataset as the annotations for these questions might no longer be useful for future systems. \n\nA further analysis of the error types made by system (failure cases where the wrong label is assigned to an instance) can be insightful.", "questions_to_authors": "A. What is the reason behind using binary responses for the questions? Some of the questions can be better described if evaluated on multi-level scale such as Q4, Q5 and Q6. Also, Q6 is highly correlated with Q4 and Q5, why did you decide to keep it?\n\n2. Are the models used to generate schemas finetuned further or are used as is? Please provide more details on different models such as whether/how they are trained, data used for training, etc.\n\n3. Please provide the statistics of the instances you used to generate summaries for such as the number of unique articles you used. What is the percentage of articles from different datasets and what is the distribution of articles from different languages? Do different datasets have similar articles and did you remove the duplicates?\n\n4. The first three questions are almost always positive showing that current LLMs are very good at generating readable, coherent and grammatically correct summaries. Why do you include them in the benchmark and can you think of a way to evaluate these metrics in a more fine-grained fashion? Maybe that can reveal aspects which systems still struggle.\n\n5. Why did you only use one annotator for each question and not multiple annotators and then doing a majority vote? This could further improve the quality of responses.\n\n6. What is the justification for using Pearson’s coefficient and ROC for evaluating the learned metrics?\n\n7. For mFACE informativeness metric, why did you use Q6 as a proxy and not Q5? Based on your description for Q5, that seems to be a better alignment.", "paper_topic_and_main_contributions": "The paper introduces a dataset of human ratings on 6 qualitative dimensions for multi-lingual summaries coming from 4 datasets and covering 6 languages generated using 9 different systems. The dataset can help evaluate the learnt metrics that are used to evaluate qualitative aspects of summarization systems and also serve as a resource for training such evaluation metrics.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rIWwDwFyzP", "reviewer": "Reviewer_8cDd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re glad you feel the dataset will be useful and a valuable contribution to the research community; thank you for the supportive feedback. Please find answers to each of your questions below:\n- Question #1: Serious grammatical errors that render the text unreadable should also be incomprehensible. But if the annotators were capable of understanding the meaning of the text and the grammar errors were only minor (e.g., a missing comma), we thought it would be better to collect the scores for the other dimensions, as they could always be filtered posthoc by the user.\n- Q2: Yes, conciseness is the dimension that captures if the text is a good summary, not just asking about brevity but whether the main ideas (and *only* the main ideas) are accurately represented in the summary. We will make this definition clearer in the paper and consider an alternative name for the dimension.\n- Q3: Despite our best attempts to clearly define each dimension and its criteria, there was a degree of subjectivity for all dimensions. Attribution seems straightforward, but often there are minor discrepancies between the summary and the article that an annotator might not catch or consider important. For example, one annotator thought that references to a “meltdown” after a software update described in an article were enough to justify the summary mentioning computer system “crash,” while another annotator did not. Grammaticality does have high pairwise agreement (0.89), but a lower Krippendorff’s alpha than Q4-6 because there is a higher likelihood of agreement by chance. As with attribution, ambiguity was a cause of disagreement; annotators had different ideas about what spelling/grammar is correct depending on how they interpreted the summary. For example, for the summary “The Mitchell brothers have been reunited with his father's wife, a soap soap maker has said.” One annotator interpreted “his” as referring to the Mitchell brothers and said it was a grammar mistake and should be plural, while another annotator thought it was referring to the soap maker and was therefore correct.\n- Q4: This is a great idea for follow-up work using Seahorse. We did not look into this as part of this research project, but we agree it would be interesting to see how well translation systems would handle the longer documents and if they would preserve the errors in a way that accurately reflects the ratings from the English text. Thank you for the suggestion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset will undoubtedly be useful for the community to benchmark metrics. Its multilingual and multidataset aspects make it a valuable contribution to the community.", "reasons_to_reject": "No major reasons, but see the questions below.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes SEAHORSE, a summarization evaluation dataset that covers 6 languages, 9 systems, and 4 summarization datasets and can be used for metric evaluation and training. Summaries are annotated across 6 dimensions. Metrics are trained for each dimension.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. You have the question for comprehensibility and filter if that is answered no. I wonder if you considered filtering based on grammar. FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization removed summaries labeled as ungrammatical to not have them influence the annotation of other dimensions.\n2. The dimension of conciseness was a bit confusing for me. Especially when in Table 3, many of the ones that score high for conciseness are very long summaries on average according to Table 2. Maybe it’s a matter of naming, but it seems to capture more of an overall quality (correctly gets information while not being too long)\n3. One could argue that attribution should be objective and have higher pairwise accuracy (as discussed in L371) along with grammaticality, whose low agreement is somewhat surprising. \n4. Did you try translating the English subset of SEAHORSE into the respectively languages and training the metric on that? To better understand the benefit/necessity of annotated in multiple languages."}}, {"review_id": "JJMj6z4I10", "reviewer": "Reviewer_SFUF", "summary": "", "questions": "A: As the paper talks a lot about text quality, you may want to consider introducing the phrase \"text quality\" in the title of the paper (just a thought since many researchers working on text quality may not immediately discover your paper as relevant for their own work). If it's too late for that, you may want to stress \"text quality\" a bit more in the abstract.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- Question #1: Thank you for your positive feedback. We have noted your comments about the wording and presentation in parts of the paper (e.g., “systems” and “text quality”), and we will update the paper to reflect these points.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The presented resource fills a gap in summarisation research. To the best of the authors' knowledge (and also to the best of this reviewer's knowledge) it is the largest multilingual and multi-dimensional summarisation evaluation resource.\n- The dataset has a non-trivial size, covers multiple languages and human annotations across six different dimensions.\n- The paper provides a thorough description of the dataset and its annotation process, it also evaluates the dataset in various regards.\n- The paper is very clear and well written.", "reasons_to_reject": "- This reviewer does not see a single reason to reject this paper.", "questions_to_authors": "A: As the paper talks a lot about text quality, you may want to consider introducing the phrase \"text quality\" in the title of the paper (just a thought since many researchers working on text quality may not immediately discover your paper as relevant for their own work). If it's too late for that, you may want to stress \"text quality\" a bit more in the abstract.", "paper_topic_and_main_contributions": "The paper \"SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation\" presents a new resource, SEAHORSE. This dataset for multilingual, multifacted summarisation evaluation consists of 96,000 summaries with human ratings that relate to six quality dimensions (comprehensibility, repetition, grammar, attribution, main ideas, conciseness). The resource covers six languages and nine \"systems\" (one of which is a human-created summary). Four datasets were used to create SEAHORSE (mlsum, xsum, xlsum, wikilingua). SEAHORSE can be used as a benchmark for existing systems or as a dataset to train systems. An evaluation on TRUE and mFACE shows that systems trained with SEAHORSE achieve strong performance. The resource is made freely available (it was also submitted to EMNLP as supplementary material).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n.a.", "Justification_for_Ethical_Concerns": "n.a."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- You use the term \"systems\" a lot in the abstract and also in the main body of the article. It wasn't immediately clear to me that this also includes human reference summaries. Please consider either changing the word (from \"system\" to something else) or adding something like \"(including human reference summaries and LLM-generated summaries)\" to explain this immediately, ideally already in the abstract.\n- In Table 1, please consider using the full names of the languages instead of the acronyms.\n- In Table 3, please consider adding \"comprehensible\", \"repetition\", \"grammar\" etc. to the Q1, Q2, Q3 etc. column heads.\n- Please spell out all numbers between one and ten (\"nine\" instead of \"9\", \"six\" instead of \"6\" etc.)."}}]

Author rebuttal:

Meta-review: This paper introduces a new dataset for multilingual, multifaceted summarisation evaluation, consisting of 96,000 summaries with human ratings that relate to six quality dimensions (comprehensibility, repetition, grammar, attribution, main ideas, conciseness). The resource covers six languages and nine "systems" (one of which is a human-created summary).

The reviewers were unanimous in that the proposed dataset will be a very valuable contribution to the community, thanks to its size, multilingual coverage and rich annotation. The paper also appears to be clearly written and easy to follow. The reviewers only identified minor issues with this paper, some of which could be addressed in the rebuttal.
Decision: Accept-Main