言語処理学会 第30回年次大会 発表論文集（2024年3月）

Beyond ROUGE: Applying an ELO algorithm to rank model
performances in summarization
Romain Harang1
1 The University of Tokyo
romain-harang@g.ecc.u-tokyo.ac.jp

Abstract
We apply an ELO-based algorithm to evaluate the performances of Language models in text generation by generating one-on-one encounters in which a judge function
determines the outcome. We decided to apply this to thetext summarization task on the CNN-Dailymail Dataset using state-of-the-art models and GPT3 and GPT4 in 0-shot
as the judge function and the diﬀerence in ROUGE scoresand compare with ROUGE.
We found that this approach is weakly correlated toROUGE; a correlation can only be found after assuming
a signiﬁcant random noise and giving a radically diﬀerent
ranking of the models on the task. In particular, GPT4
is preferred as a summarizer. We also found that our results contradict the assumption that the ground truth isbest for the CNN-Dailymail Dataset, comforting previous
ﬁndings. This research opens avenues for a more comprehensive understanding of language model performance intext summarization tasks.

1Introduction

With the advent of Language Models [1], tasks related
to text generation have known tremendous developments.
These models were proposed on Translation and achieved
state-of-the-art performance on the WMT 2014 sharedtask; on abstractive summarization, this new approach offered similar results to RNN [2] on its ﬁrst iteration. Nowadays, LLMs are used and preferred in all cases.
Evaluating the performance of generative algorithms fortext has always been a crucial yet sometimes overlooked
aspect of the process. Usually, metrics are not updated
throughout the lifetime of a Dataset. For text Summarizing and Question Answering, it will be ROUGE [3], andfor Translation, Captions will be BLEU [4]. Many othervariants also incorporating language modeling and wordembeddings have been proposed, showing each time better
correlation to human judgment than ROUGE and BLEU
in a limited scope; those metrics have limited actual use,
however (apart from BERTscore [5], that is sometimes
used along ROUGE). Later re-evaluation eﬀorts [6] haveshown no signiﬁcant diﬀerence between these new metrics
and ROUGE and BLEU in their correlation with humanscoring nor in identifying the best model given a speciﬁctask.
Until recently, evaluation was centered around scoring
the similarity of the generated text to a ground truth. But
LLMs outputs are now preferred by humans in many cases[7]. For that reason, using humans to compare model
outputs is gaining traction [8], utilizing ELO-based algorithms. A high correlation to human judgment was also
found using LLMs as judges. However, no comparisonsto other metrics were made nor to a ground truth whenavailable; this work addresses that.

2Related Works

ELO rating: Up until recently, there have been minimal
attempts at using any ELO algorithm while evaluating theperformances of generated text. Human evaluations basedon pair-wise direct comparison of generated text using human annotators have become popular with, for example,
the release of ChatBot arena that uses an ELO algorithm
to judge the comparative performances of LLMs [8] [9].
LLMs have started to be used; however, when our experiments were conducted, no public data was available on
the usage of LLMs in that context, but it has since been
increasingly used as well [10]. At the time of these ﬁndings, there have been no attempts at comparing an ELO
algorithm to currently used metrics or the ground truth inany datasets, to the best of our knowledge.

― 2799 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Text Summarization evaluation: There has been ample review and evaluations of the diﬀerent metrics and theirinner correlations and correlation to the human Judgment[6]. Coupled with the fact that it has been shown [7] that,

4 Applying the ELO algorithm tobenchmark datasets
4.1

Datasetsfor example, in summarization, Humans will prefer GPT-3

We decided to limit ourselves to the summarization tasksummaries to Ground truth in the CNN Dailymail Dataset.

on the CNN-Dailymail Dataset [2]. We will restrict our-Therefore, using metrics based on the sentence similarity

selves to using models on the test split (11.5k pairs ofto Ground truth has shown its limits. Furthermore, thearticles/summaries are included). An important caveat

usage of ROUGE has also been criticized for longer texts;

with this dataset is that although it is the most popular

it has yielded luster lacking results in [11].

for abstractive text summarization, the summaries werenot constructed as such but instead as a concatenation of

3

Pair-wise comparison method and
ELO algorithm

3.1

highlights from the article. From this, it is reasonable toassume that a human or a generative algorithm can ﬁnd abetter option than the ground truth.

Direct comparison

We are given a pair of texts (𝑡𝑒𝑥𝑡 1, 𝑡𝑒𝑥𝑡 2); we usea judge to decide whichever is more ﬁtting for the task,
in our case, the better summary. We use LLMs (GPT-44.2

ModelsState-of-the-art models for the CNN-Dailymail were selected for our set of experiments:or GPT-3) as judges and ROUGE (by taking the text with• BART [13]higher ROUGE). When using the LLMs, we also use the• DistilBART (obtain using the principles in [14] by S.

article text as a reference for judgment, the reference forShleifer)

ROUGE being the ground truth. This allows us to evaluate• PEGASUS [15]the ground truth as well with the LLMs.

• BRIO [16]3.2

• GPT-4 [17]The ELO algorithm

We used the previous implementation of the ELO algorithm [12] to build our own. The goal of this algorithm
is to, given two arbitrary models 𝑚 𝑖 and 𝑚 𝑗 and the generated output 𝑥 = (𝑡𝑖 , 𝑡 𝑗 ) estimate the probability that 𝑡 𝑗
is preferred to 𝑡 𝑖 by our comparison model. Each modelis attributed a ﬁtness or ELO score; then this probability
1is given by 𝑝 = 𝜎( 𝐸 𝐿𝑂 ( 𝑗 ) 𝑏−𝐸 𝐿𝑂 (𝑖) ) where 𝜎(𝑥) = 1+10𝑥All models apart from GPT-4 have been trained explicitly
and speciﬁcally for this task in the CNN-Dailymail train
split as well as other compilations of news articles or summarization datasets such as X-Sum [18] and GigaWord.
GPT-4 will be used on its 0-shot form, which means theinput will be of that form: <prompt><article>. Belowin Fig. 1 is an evaluation of the models used based onand 𝑏 is an interpretation hyper-parameter; ie, a diﬀerence

ROUGE-1,2 and L.

in ELO of 𝑏 corresponds to a ratio of probabilities of 10

ModelR-1

R-2BART

0.366

0.163 0.344(1)= 1−𝜎𝜎(1)(1) = 10).
( 𝜎𝜎(−1)

R-L

DistilBART 0.384

0.173 0.361

The algorithm will try to estimate it, similar to previous

PEGASUS

0.395

0.181 0.372

method assumptions. Still, contrarily to them, such a score

BRIO

0.438

0.196 0.413

is not computed directly (at least not necessarily). Instead,GPT-4

0.275

0.085 0.251Our working hypothesis is that such a ﬁtness score exists.

such scoring is implicitly inferred by estimating the ELO.
The iterative algorithm to determine the ELO will bebased on the maximization of the Log-likelihood functionof our problem. We use the update function based on
Stochastic gradient descent from [12] (details in Annex)

Figure 1

ROUGE-1, 2 and L scores on the test split of thechosen models

4.3 Using ROUGE as the comparison’smodel
ROUGE provides us with a straightforward way of comparing two generated summaries. We can select the win-― 2800 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

ning party with the highest ROUGE (1,2 or l). For ourModel

ELO-4

ELO-3

R-L

speciﬁc experiment, we decided to use ROUGE-l, take

GPT-4

1617

1204

0.251

𝑏 = 400, 𝜃 0 = (1000, ..., 1000), and 𝛽 = 50 with sched-BART

1037

1024

0.344

uled decrease until 1, below in ﬁg. 2 we can see that theBRIO
Baseline

931
816

966
988

0.413
-PEGASUS

798

850

0.372

DistilBART7989660.361

algorithm converges.

Figure 4Final scores for GPT-4 and 3 named ELO-4 and 3 r.
and Rouge-L

with GPT-4 will get a higher conﬁdence score according toGPT-4, it doesn’t automatically translate to GPT-4 having
a stronger preference for itself in our task, at least in theory.
We see, however, that this is the case. Also, the groundtruth is not always preferred to machine-generated text.
Notably, the ELO value has meaning only in a relative
sense; an essential diﬀerence in ELO means a more sigFigure 2Convergence of the ELO algorithm for ROUGE-l

By taking the ﬁnal values for the ELO and comparing

and BART, translating to a comparative advantage of 32

them with ROUGE scores, we get:
ModelR-1

R-2

R-L

BART

0.366

0.163

0.344 1010

DistilBART 0.384

0.173

0.361 1020

PEGASUS

0.395

0.181

0.372

BRIO

0.438

0.196

0.413 1140

GPT-4

0.275

0.085

0.251

Figure 3times likelihood. But with ELO GPT-3, the diﬀerence isELO

only 200, so it is an advantage of only three times.

4.5 Validation: Analysis of the correlation between GPT-3 and 4 judgment outcomes and ROUGE scores960

We have seen that regardless of the aggregation method,

873

ROUGE and judgment scores based on GPT with 0 shots

Scores for ROUGE compared to ELO R-L

give very diﬀerent results. On the granular level, however,

Apart from PEGASUS, there is a good consistency between the two aggregation methods for the ROUGE score.

4.4

niﬁcant comparative advantage. For example, with ELO
GPT-4, we have an oﬀset of around 600 between GPT-4

Using LLMs as the comparison’smodel

we have not looked if there is any correlation between thesescores.
We decided ﬁrst to calculate the Cohen’s Kappa score

Using the same hyperparameters for the ELO algorithm,

between GPT-3,4 and ROUGE pair-wise. We can’t compare judgment outcomes to ROUGE scores directly, so wewe applied diﬀerent judgment methods. Based on a 0-shot,

use the outcomes based on the ROUGE diﬀerence we intro-

Ground-truth-free approach. Our prompt becomes thenduced in ELO-ROUGE. We use Cohen’s kappa in a 2-label<Instructions><Article><Candidate 0><Candidate 1>

format (classiﬁed as [0,1] for the ﬁrst and second models,then the output will be 𝑦 ∈ [0, 1] depending on therespectively). We aggregate the results for each pair ofpreferred candidate.

models below:

The models used are GPT-3 andGPT-4. We also decide to include the ground truth as aMethod 1

Method 2

Cohen Kappa

possible candidate. To account for positional biases, theGPT-3

ROUGE-0.021

candidate orders are systematically randomized.

GPT-3
GPT-4

GPT-4
ROUGE0.32-0.035

If we compare the ﬁnal ELO scores to the Rouge-l score,
we get Fig. 4.
The ﬁrst observation from this experiment is that GPT-4

Therefore, there is no agreement between GPT-3 or 4

is preferred in this context. Especially in the case where themodel used to judge is GPT-4. Even though text generated

and ROUGE (performs as bad as random). Concerning
GPT-3 and GPT-4, they only have a weak, limited agree-― 2801 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

ment with one another. (Good agreement starts at Cohen’s

As hypothesized, we get a negative slope (the lower, thekappa of 0.6 according to common heuristics). This tendsbetter) consistently apart from one case where the coeﬃ-

to show no correlation between ROUGE and GPTs. How-

cient of determination is very low, concluding that this isever, we didn’t use the full range of the ROUGE score. Itan outliner.

is common practice to evaluate the correlation of 2 metricsof that type by using Spearman’s 𝜌, used, for example, in5Discussion[6]. We can’t here as one of them is not continuous but

Our proposed method shows a higher score for someordinal. The workaround is to observe the distribution ofLLMs in this work against the human-made ground truth.

ROUGE’s score for a given label, either model 1 or 2; weGenerally, it is best practice to always assume the human-

get two distributions to compare. One way is to compare

generated text as the best possible or at least a suﬃcientthe mean of each of the two distributions similarly to Co-baseline for which the model should aim. In the case ofhen’s kappa results. We don’t get any diﬀerent signiﬁcative

the CNN-Dailymail Dataset, the summaries proposed are

results that way.

a concatenation of highlights from a news article; it is not

While manipulating the data, it came to light that if we
looked at the histogram of the proportions of classiﬁed

surprising to ﬁnd here and in previous research that it could
be improved.

model 1 at diﬀerent ROUGE scores, there seemed toInstead of the ELO algorithm, we could have used win

be an inverse correlation between these two quantities.

rates as the aggregation method. Both methods could be

As a proposed modelization of that phenomenon, we as-valid in our particular case, but ELO allows for more mean-

sume that the probability given the rouge score diﬀerence

ingful results at a lower cost (i.e., number of pair-wise

𝑅 = 𝑅2 − 𝑅1 is going to be 1/2 at the point of ROUGE

encounters), as shown in [19].

= 2(𝜌 − 12 ) where 𝜌 is the ratio of labels

We argue that even if the Cohen kappa results for GPT-4

model 2. This probability will then vary linearly until

diﬀerence 𝑚and ROUGE suggest that trying to infer the label given by

the ROUGE diﬀerence hits its boundaries (-1 and 1) orGPT-4 with the sign of the ROUGE score for individual

the probability saturates at 0 or 1. Given that we haveinstances of scoring is going to perform similarly to random

𝑃(𝐿𝑎𝑏𝑒𝑙 = 𝑀𝑜𝑑𝑒𝑙 1 |𝑅) = 𝑚 + 𝑎𝑅, in this deﬁnition, we(in fact, slightly worse); it is still possible at the meso-level

expect the slope (𝑎) to be negative, as we presume thatto identify a link between the diﬀerence in ROUGE score

there is a negative correlation between ROUGE diﬀerence

and the probability of being chosen with GPT-4.

and the probability that model 1 is chosen as a victor. We
compare GPT-4 (ELO) and ROUGE-L only for pairs ofmodels where, in the aggregate, either model’s selection
ratio is not lower than 10% (not enough data points). We

6Conclusion
We were able to show the following by using our newly

approximate this distribution using linear regression in Fig.

proposed ELO-based scoring algorithm:

5:In agreement with past results, this scoring method prefers
GPT-4 0-shots summaries, while the ROUGE score ofmodel 2slope𝑟2

DistilBART PEGASUS
BARTbaseline

-0.96
-0.31

0.78
0.68

chine translations to the Ground truth, further suggestingmodel 1

these instances is meager. It also likes some other mathe inadequacy of using it as the sole base to evaluate, atBRIOPEGASUS

-0.66

0.65

least in the CNN-Dailymail Dataset’s case.

BART

BRIO

-0.63

0.61

By looking at correlations with ROUGE scores, we ﬁndDistilBART

baseline-0.440.46

only limited noisy correlations to the point of having aPEGASUS

baseline

-0.54

0.42

slightly negative Cohen Kappa.

BRIO

baseline

-0.33

0.40

DistilBART

BRIO

-0.38

0.32

Further research is necessary to integrate human annotations to compare them with GPT-4 and use the method on

DistilBART

BART

0.67

0.20

other text generation tasks such as Translation.

Figure 5

Regression’s results― 2802 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

References[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,
A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all
you need,” Advances in neural information processing systems, vol. 30, 2017.
[2] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al., “Abstractive text summarization using sequence-to-sequencernns and beyond,” arXiv preprint arXiv:1602.06023,
2016.
[3] C.-Y. Lin, “Rouge: A package for automatic evaluationof summaries,” in Text summarization branches out,
pp. 74–81, 2004.
[4] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: amethod for automatic evaluation of machine translation,”in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318,
2002.
[5] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, andY. Artzi, “Bertscore: Evaluating text generation with bert,”
arXiv preprint arXiv:1904.09675, 2019.
[6] A. R. Fabbri, W. Kryściński, B. McCann, C. Xiong,R. Socher, and D. Radev, “Summeval: Re-evaluating summarization evaluation,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 391–409, 2021.
[7] T. Goyal, J. J. Li, and G. Durrett, “News summarization and evaluation in the era of gpt-3,” arXiv preprint
arXiv:2209.12356, 2022.
[8] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu,Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al., “Judging
llm-as-a-judge with mt-bench and chatbot arena,” arXiv
preprint arXiv:2306.05685, 2023.
[9] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli,
T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma,
N. Elhage, Z. Hatﬁeld-Dodds, D. Hernandez, J. Kernion,
K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark,
S. McCandlish, C. Olah, and J. Kaplan, “A general language assistant as a laboratory for alignment,” 2021.
[10] M. Boubdir, E. Kim, B. Ermis, S. Hooker, and M. Fadaee,“Elo uncovered: Robustness and best practices in language
model evaluation,” 2023.
[11] K. Krishna, A. Roy, and M. Iyyer, “Hurdles to progress inlong-form question answering,” 2021.
[12] D. G. de Pinho Zanco, L. Szczecinski, E. V. Kuhn, andR. Seara, “A comprehensive analysis of the elo rating algorithm: Stochastic model, convergence characteristics,
design guidelines, and experimental results,” 2022.
[13] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart:
Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension,”
arXiv preprint arXiv:1910.13461, 2019.
[14] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert,a distilled version of bert: smaller, faster, cheaper andlighter,” 2020.
[15] J. Zhang, Y. Zhao, M. Saleh, and P. Liu, “Pegasus: Pretraining with extracted gap-sentences for abstractive sum-

[16]

[17]
[18]

[19]― 2803 ―

marization,” in International Conference on Machine
Learning, pp. 11328–11339, PMLR, 2020.
Y. Liu, P. Liu, D. Radev, and G. Neubig, “Brio: Bringing order to abstractive summarization,” arXiv preprint
arXiv:2203.16804, 2022.
OpenAI, “Gpt-4 technical report,” 2023.
S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me thedetails, just the summary! topic-aware convolutional neural networks for extreme summarization,” arXiv preprint
arXiv:1808.08745, 2018.
C. Olsson, S. Bhupatiraju, T. Brown, A. Odena, andI. Goodfellow, “Skill rating for generative models,” 2018.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

AnnexELO algorithm details
For this we deﬁne 𝑀 = (𝑚 0 , ..., 𝑚 𝑛−1 ) the list of 𝑛models that are used, then 𝑥 𝑖 = (𝑥 𝑖,0 , ..., 𝑥 𝑖,𝑛−1 ) the 𝑖thinput where
1if j is the second member of our pair
𝑥 𝑖, 𝑗 = −1 if it is the ﬁrst



0

else


Figure 7

Convergence of the ELO algorithm for GPT-4• We consider each bin a data point of value x theThe complete input is then 𝑋 = (𝑥0 , ..., 𝑥 𝑁 −1 ) where 𝑁

ROUGE diﬀerence corresponding, and value y theis the total number of samples, and then 𝑌 = (𝑦 0 , ..., 𝑦 𝑁 −1 )proportion of model 1 selected.

where 𝑦 𝑖 = 𝟙second team won . Finally, we can deﬁne our• We compute the linear regression of these data points

estimator as 𝜃 = (𝜃 0 , ..., 𝜃 𝑛−1 ) the estimator for 𝑌 will
become 𝑌˜ = 𝜎(𝑋𝜃 𝑇 /𝑏) (where the division and 𝜎 operator

using the number of entries per bin as weights. We
get the slope 𝑎 and the coeﬃcient of determination 𝑟 2

are applied to each member one by one). The stochastic
Gradient descent iterative algorithm, in this case, will be𝜃 𝑘 = 𝜃 𝑘−1 + 𝛽[𝑦 𝑘−1 − 𝜎(𝑥 𝑘−1 𝜃 𝑇𝑘 /𝑏)]𝑥 𝑘−1

Convergence of the Algorithm

Figure 6

Convergence of the ELO algorithm for GPT-3

Regression process details
The method used to evaluate if this modelization works
is the following:• We convert the distributions into bins, the total number of bins being 𝑁• for each bin, we have the proportion of either model
being selected and the total number of data entries inthis bin; for our case, we chose 40 bins.

― 2804 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).