Accuracy is not enough: Evaluating Personalization in Summarizers
Authors: Rahul Vansh
Darsh Rank
Sourish Dasgupta
Tanmoy Chakraborty
Venue: EMNLP 2023 Findings (2023)
Keywords: Personalized Summarization Evaluation
Meta Evaluation
Automated Accuracy Metrics
TL;DR: In this paper we demonstrate that degree of personalization is an independent characteristic than accuracy and hence, propose a novel measure called $e$-$DINS_{sub}$ for computing the degree of personalization of a summarization model.

Abstract: Text summarization models are evaluated in terms of their accuracy and quality using various measures such as ROUGE, BLEU, METEOR, BERTScore, PYRAMID, readability, and several other recently proposed ones. The central objective of all accuracy measures is to evaluate the model's ability to capture $\textit{saliency}$ accurately. Since saliency is subjective w.r.t the readers' preferences, there cannot be a fit-all summary for a given document. This means that in many use-cases, summarization models need to be personalized w.r.t user-profiles. However, to our knowledge, there is no measure to evaluate the $\textit{degree-of-personalization}$ of a summarization model. In this paper, we first establish that existing accuracy measures cannot evaluate the degree of personalization of any summarization model, and then propose a novel measure, called $EGISES$, for automatically computing the same. Using the PENS dataset released by Microsoft Research, we analyze the degree of personalization of ten different state-of-the-art summarization models (both extractive and abstractive), five of which are explicitly trained for personalized summarization, and the remaining are appropriated to exhibit personalization. We conclude by proposing a generalized accuracy measure, called $P$-$Accuracy$, for designing accuracy measures that should also take personalization into account and demonstrate the robustness and reliability of the measure through meta-evaluation.

Accuracy is Not Enough: Evaluating Personalization in Summarizers
Rahul Vansh f* Darsh Rank |
Sourish Dasgupta 1*
Tanmoy Chakraborty 1+
KDM Lab, Dhirubhai Ambani Institute of Information & Communication Technology, India Indian Institute of Technology, Delhi, India
(202111033 201901247, sourish J @daiict.ac.in, tanchak @ @iitd.ac.in
Abstract
measures have been proposed such the widely adopted ROUGE variants (e.g., ROUGE-w/L/SUU4
Text summarization models are evaluated in terms of accuracy and quality using various measures such as ROUGE, BLEU, METEOR, BERTScore, PYRAMID, readability. and sev- eral recently proposed ones. The central objec- tive of all accuracy measures is to evaluate the model's ability capture saliency accurately. Since saliency is subjective w.r.t the readers' preferences, there cannot be a fit-all summary for given document. This means that in many use cases, summarization models need to be personalized w.r.t user profiles. However, to knowledge, there is measure evaluate the degree of personalization of a summariza- tion model. In this paper, we first establish that existing accuracy measures cannot evaluate the degree of personalization of any summarization model and then propose novel measure, called EGISES, for automatically computing the same. Using the PENS dataset released by Microsoft Research, we analyze the degree of personal- ization of ten different state-of-the-art summa- rization models (both extractive and abstrac- tive), five of which explicitly trained for personalized summarization, and the remain- ing are appropriated exhibit personalization. We conclude by proposing a generalized accu- racy measure, called P-Accuracy, for design- ing accuracy measures that should also take personalization into account and demonstrate the robustness and reliability of the measure through meta-evaluation.
etc.) (Lin, 2004), BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), BERTScore (Zhang et al., 2019), PYRAMID (Nenkova and Passonneau, 2004; Gao et al., 2019) and the more recently proposed ones such as SUPERT (Gao al., 2020), WIDAR (Jain et al., 2022), and InfoLM (Colombo et al., 2022). At the same time, several meta-evaluation studies on the reliability of these measures based on the human judgment correlation have been proposed (Graham, 2015; Chatzikoumi, 2020; Bhandari et al., 2020; Deutsch et al., 2021; Fabbri et al., 2021; Peyrard, 2019; Wei and Jia, 2021).
Within the context of all these developments on novel metrics and meta- meta-evaluattin showing varied results different datasets and summarization sys- tems, the central idea remains the same for all ac- curacy measures, i.e., they all need to measure the ability of summarization models to capture saliency accurately. However, saliency subjective to a reader's 's prior reading history and evolving prefer- ences (aka attention drift). In other words, in many use cases, models have to consider user reading patterns and then generate personalized summaries instead of a fit-all generic summary. This calls for summarizers to be personalized. Although, rela- tively speaking, personalized summarization needs much research attention, there have been a few noteworthy studies this direction (Ghodratnama et al., 2020b; Ao et al., 2021). The evaluation of these models has been mostly based on the accu- racy measures (e.g., ROUGE-L). However, in this paper, we establish that we cannot measure the de- gree of personalization of summarization models using accuracy. We theoretically prove that person- alization an independent characteristic compared to accuracy, and possible that while model performs fairly reasonably w.r.t any accuracy mea- sure, it may have a poor degree of personalization.
1 Introduction
The growing availability of large-scale text data has led to an increasing demand for automated text summarization systems that aim to compress a lengthy document into a short paragraph, which includes salient information about the document. Evaluation of such summarization systems (and the underlying models) performed either in terms of their accuracy or quality. To date, several accuracy
All authors have equal contributions.

---

We, thereby, propose novel measure, called the effective Degree of Insensitivity w.rt Subjectivity (EGISES) that conversely measures how insensitive a model is there is a significant difference in the preferences (or expected summaries) of two readers for the same original document, where an insensi- tive model would not have much change in the gen- erated summaries for the two readers. To the best of our knowledge, this notion of degree of personal- ization and its corresponding measure would be the first of its kind. We evaluate EGISES on ten state- of of-the-art summarization models (both abstractive and extractive), including the best-performing ones proposed by (Ao et al., 2021) using the PENS test dataset (Ao et al., 2021), and show that the leader- board so generated does not have any consistent correlation with those generated by standard accu- racy measures, thereby empirically showing that accuracy is not enough.
that measuring personalization is completely dif- ferent from measuring how close a model's gener- ated summary to that of the expected summary (which accuracy measures do). To elucidate this counter-intuitive aspect with an example, imagine Alice and Bob are interested in news related to the ClusiaaUUkriing war, but Bob is more focused on news related to the war fronts, while Alice is in- terested in news related to civilian distress. Now, for given news article, model may be able generate fairly accurate summary for Bob where the core narrative is about the battles, while at the same time, it is also able to get a good accuracy for Alice because it happens to have sufficient periph- eral content (i.e., off-topic mentions) about, say, civilian displacement. In this situation, Alice has to filter out content that is not salient to her to get to her interest. This becomes even more problematic in the case when models also generate headlines. our example, Alice will not get to see the summary headline aligned with her interest and, therefore, may completely skip the summary. Therefore, the degree of personalization provides an insight into how engaging a model can be in terms of better user experience and not just a clini- cal computation of the overlap of model-generated summaries that of their gold references. In the following section, we will provide mathematical sketch-of-proof that will clearly demonstrate that accuracy measures are not capable measuring the degree of personalization of a model.
A key step for any new accuracy metric is to establish its robustness and reliability. The stan- dard method to do so is human-judgment--ean based meta-evaluation using correlation statistics such Pearson's 's Coefficient, Spearman Rank Correla- tion, and Kendall r Rank Correlation. Since di- rect methods are practically infeasible (we will dis- cuss that Section 6) we adopt an indirect meta- evaluation method that establishes a high human- judgment vs. EGISES correlation. We then pro- pose a generic accuracy measure based on EGISES, called P-Accuracy, for calculating the realistic ac- curacy of models that need to exhibit personaliza- tion, and have empirically shown that the measure is stable w.rt its invariance the original accuracy leaderboard.
2.2 Accuracy is Not Enough Proof Sketch In this section, we provide sketch-of-proof that the accuracy measures are insufficient capture the degree of personalization of summarization models. For a given document d,, two reader- profiles (i.e., expected personalized summaries or reference summaries) uij and uik and the corre- sponding summaries Suy and Sug generated by a model Me,u. let's 's choose an arbitrary distance met- ric defined the metric space M where d, u, s defined In other words, (s,, u) is the abstrac- tion over any arbitrary accuracy measure. In a sim- ilar way, the notion of a(usj.Mt)) can be captured by the ratio this ratio the deviation.
2 Preliminaries
2.1 Degree of Personalization
The degree of personalization of a summarization model the quantitative measure of the extent to which a model can generate summaries that align with the reader's subjective appreciation of saliency. While we provide a formal definition in Section 2.3, informally, if two readers' subjective agree- ment of saliency of given document is low, then the model should generate personalized summaries for the readers such that they should not have high overlap, and vice-versa, without affecting the re- quired accuracy. Although the general intuition might be that accuracy measures should be suffi- cient to measure this subjectivity, we demonstrate
a(Sug) Suge) We term
Theorem 1. The deviation of a model M9 on the metric space M can be changed without any
la satisties positivity, reflexive, maximality, symmetry. and the triangle inequality.

---

Definition 1. Personalized Summarization Model. Given document d, and user profile u, a summarization model Ma. said to be personalized iff Ma," + &u. where &y is the best estimated summary of considering user profile u, 0 being model parameters.
Accuracy
Accuracy
Figure 1: Triangulation: (a) High accuracy vs. low personalization; (b) High (and same) accuracy vs. high personalization; blue denotes sunmary-deviation and green denotes reader-deviation.
Definition 2. Weak Insensitivityto-Suuieccii Given two mutually indistiguishable reader profiles, ui and uj. summarization model Ansensitive-to-Sujjeett f (usist
Mo.a is (weakly) iff V(ug, ug).
S Thoz:
change in (ss, u.)
f (Me,uld, uz), Me,u(d, (o,((d,)))) uz)) > "min' where Jaist indicates user profile distance function, S is summary similarity function, Thas is the maximum limit for two different user profiles be mutually indistinguishable, and - is the minimum limit for two generated summary w.n. two different users to be mutually distinguishable. : f (w,, u.) = 0 faisa (U,, Uz) C [0,
Proof. Let d, us be triangulated as per Figure 1. Keeping d, u fixed, we can perform an arbitrary rotation operation (rot(o, , a); a: angle of rota- tion) on Su,, and Suz, s.t. rol u, a)) is a closure operator in M. Now, 3(p, 9) E M, s.t.
max (rot(ssu uny ap); rot Sun u ag))} (p.g)
minfo confooroote (rol((8u Mij ap), rol Wike aq))) (p.q)
Definition 3. Strong wtwprsittiiittt--Sujjeeeee Given two different reader profiles, ui and Uij, summarizatiin model Mo.,. is (strongly) iff the model sat- isfies the condition of weak insensitivity and also V(u,, ug), Seiss uj) f5 (Ma,a(d, uz), Mo,,, (d, uz)) < Tmmn
In other words, a total ordering of deviations ex- ists. Also, for any arbitrary a, au, rot(s, u, a:)
(s, u) by the property of the rotation operator. Since we have kept u fixed, the deviation, there- fore, can be varied by changing a (and thereby (suu. Sug) )) from minimum to maximum. D
3 EGISES: Measure for Insensitivity
The proof shows that two models may have the same accuracy but different degree-of- personalization. Our findings support the same (Section 5.3). Therefore, accuracy measures are not adequate for measuring the degree-of- personalization. We term this proof framework the triangulation framework and will continue to use this in our further discussions.
3.1 Triangulation Space Representation
To calculate how insensitive (or inversely, sensi- tive) model is to the change in the readers' pro- files when the same document has to be summa- rized for two different readers, well-defined notion of deviation w.r. an algebraic space where document set D, the model-generated summary set and readers' subjective expecta- tion set U can be represented. In this paper, we make the bag-of-word assumption of d E D, 8 C S, and u E U. and represent them on the probability space for a specific document dy the lines of the triangulation framework (see Section 2.2) (Ad, Fa,,
need to have
2.3 Measuring Insensitivity to Subjectivity
One way of looking the degree of personalization is how insensitive a summarization model de- tecting the divergence the subjectivity readers' interests and preferences. If the model is highly insensitive, it means it practically generates almost the same summary irrespective of the divergence in the readers' subjective expectations. Hence, in such case, the degree of personalization will be low and may lead low engagement. In this section, we provide a formal and generic framework for de- signing measures for the degree of personalization in terms of tiapdiiviity-t-uuueeeiiiiiy
U
Od (wi E (d, Ulzr sug V} (1)
1=1
Fa, =2 2dz; P. Rdi + 10, 1]: Σ p(w))
w w.End,
(2)

---

tribution over unique word-phrases wy from the sample space Qd; where wis are lexicons in a vo- cabulary V (i.e., dy E Fa,)) Similarly, we represent u on the same space (i.e., for a given dj, M E Fd,,) where we consider Y to be the expected summary of the reader for d (in the evaluation setup, it will be the gold-reference summary). Suz is also defined the same space, and the extractive version can be considered a subset of dj (i.e., for given Eq. 1 indicates that any dj is probability dis-
D
2455
Ud11
140
Bid
86
A/2
(dj, Uzk); Suze E Fa,). We calculate the distribu- tions as follows:
Figure 2: Summary-level deviation calculation of sq-
p(wsldj) count(w; E dy) Nd, count(w;Esujjt
Definition 4. Summary-level Deviation. Given document d; and a reader-profile uij the sunmary-level deviation of a model Mo.a. (Dev(su,))) (di, uxz))) is defined as the proportional divergence between the summary Sag of de that has been generated by Me. for wij from other reuder-profile specific summaries of d, generated (4)
Nouje p(w;ldg (dy)
p( p(assssuk)
p(weldj) (r()) indicates the ratio of w:)
by Me.a w.r.l corresponding divergence of uij from other the reader-profiles (see Figure? 2).
r(Wal1u3x)
p(w: w E Suyz) Σ r(un) Suya)
We formulate Dev(Su., (dd, Unz)) as follows:
WeE
(Estimated probability distribution of summary)
U
min min(Xejk: Yisk)
(5)
Dev (eausBulll (Suny(dd,, uz)) = U max Kijke Yijk)
Continuing with the triangulation framework. the distance between a document d, the user profiles u; and Ujs and the corresponding user- specific summaries Sui and Suj generated by a model Me.a is calculated using Jenson-Shannon divergence (JSD) (Menéndez et al., 1997)² symmetrized version of the Kullback-Leibler di- vergence, which measures the similarity between distributions. For calculating the divergence for the abstractive version of Say where we might encounter out-ol-vocabulary (OOV) word-phrases (i.c., p(wy/d (dy) = Eq. 4, we propose a RoBERTa (Liu et al., 2019) embedding- smoothing method in Section 3.3.
(6)
Xiyk exp(w(us (sx)) JSD (0DD(uus)) (uis) uaa))
(xp(w(u)ylua))
expl (epp(w((r)) ( (u JSD(Ssus [[suss)
Yije=
C exp( (ep(uu(su (s
(8)
JSD(uz
w(us) ((ur)/ux))= JSD(uij)
(9)
JSD (8 ||
w w(Suiyl JSD( (d:))
(10)
Here w(usllus and w(Suay) Suu) measure the relative divergence of ui and the corresponding profile-specific summary Sus, from the document d, A lower value of Dev(Su,, (di, Wij) indicates that while reader-profiles ated summary Say very similar to other reader- specific summaries (or vice versa 4), and hence, is not personalized at the summary-level.
3.2 Deviation of a Summarization Model
Now that we have defined the triangulation space on which deviation can be computed, we propose a measure calculate summary-level insensitivity of model w.r.t subjectivity, called Deviation-of- Summarization Model (Dev(sw,, (di, Mnj))) given reader-profiles (i.c., expected summaries or gold- references) and the document to be summarized.
different, the gener-
"In this paper, we have chosen but both the triangulation space and distance measure can be chosen to something else as well. "This ensures the strong condition of insensitivity.
Divergence
For two distributions Pan and Q: [SD(PIIQ) = [DkL(P)|M) + Dxz(QIMM) M (P14) where DEL 1 is the KL. divergence.

---

3.3 Handling OOV
section, we provide the framework for evaluating these models.
In Section 3.1, discussed that smoothing is de- sirable for abstractive summaries. This is because there may be word-phrases in an abstractive sum- mary, say Suyy that are not present in the original document de (i.e., Out-of- Vocabulary (OOV) word- phrases). This can seriously affect the divergence computations. To solve this, we propose a smooth- ing algorithm using contextual embeddings gener- ated by RoBERTa (Liu et al., 2019). The central idea of the smoothing algorithm (see Appendix A for details) is to predict whether one or more of the OOV word-phrasee present in Suya could be alternatives or augmentation to the word-phrases in the original document dj. In the case of OOVs, we check how much w; E d, and wOo E Suya are re- lated by applying cosine similarity on their contex- tual word embeddings. A bias is added to capture the possibility of woOV to be unrelated addition in the summary. If the bias is higher than the clos- est match of woo in dj. it indicates that woOV is not an allemaatie//uggmenttio usage, and there- fore, no smoothing is required. Else, smoothing is applied by taking the odds of p( pmopov (ugh)) to p( p(uoov E dz).
4.1 Evaluation Dataset
We use the test data in the PENS dataset released by Microsoft Research to evaluate the models. It consists of news headlines together with news ar- ticles. The headlines could be considered extreme summaries of the corresponding news articles. The test set was created in two phases. In the first phase of data collection, 103 native English speakers were asked to browse through 1,000 news hcadlines and mark at least 50 pieces they were interested in. The headlines were randomly selected and arranged ac- cording to their first exposure time (this ensures the dataset has the reader's 's reading sequence cap- tured as well). In the second phase, participants were asked to write their preferred hcadlines (i.e.. gold references, and in our case, these become the reader-profile set U) for 200 different articles without knowing the original news title. These news articles were excluded from the first stage and were redundantly assigned to ensure that, on average, each news article has four gold-reference summaries (this makes this dataset suitable for testing mnsensttivitt--o-uujjettiii
The partic- ipants' click behaviors and more than 20,000 goal- reference personalized headlines of news articles were also collected, regarded as the expected sum- maries (Ao et al., 2021).
3.4 Effective Degree of Insensitivity
Section 3.2, we defined the summary-level in- sensitivity of a model Mæ..- To determine the degree of insensitivity at a system level, we pro- pose EGISES (Effective InnEesivff w.r. Subjectivity). We formulate EGISES as fol-
4.2 Models Studied
lows:
We studied ten off-the-shelf summarization models. We include five models from the PENS framework
D
U
EGISES- 1
Dev Suzg (di, usy))
D|
(Ao et al., 2021): PENS-NRMS Injection-Typp
(11)
I (or TI) and Injection-Type Type 2 (or T2), PENS- NAML T1, PENS-EBNR T1 & T2. We also in- clude five more state-of-the-at models BRIO (Liu al., 2022), SimCLS (Liu and Liu, 2021), SigBird-Pegasiee (Zaheer et al., 2020), ProphetNet (Qi et al., 2020), and T5-base (Orzhenovskii, 2021). Appendix B provides a brief description of cach model. We select these models since they have been reported to be in the top 5 over the last four years on the CNN/Daily Mail news dataset, which is similar in content the PENS dataset used for our evaluation.
A high value of EGISES indicates that the model implies that the model is insensitive to the reader's subjective preferences. As result, model with a high EGISES score would not be a good selection for use cases where we need personalized sum- maries.
4 Personalization of Existing Models
One of the key objectives of this paper is analyze the degree of personalization of different otate-of- the-art summarization models using EGISES. For this purpose, we chose ten different models, includ- ing abstractive and extractive summarizers. this
http://gha....ooom...oooiinnnnnn Perrrnai-erssonizidd--aa News- Hendlino--Geeerattoon

---

4.3 Compared Accuracy Measures
in the generic models clearly helps them to sig- nificantly outperform the PENS models that were specifically designed for personalization. This also shows their ability to utilize the cue injection. At the same time, it shows that EGISES as a measure is clearly able to capture their ability to detect the injected cue and, hence, discriminate these models from the rest. Finally, we see no change of rank as we randomize our sample selection and average over ten draws for each set. The bias and variance fluctuation across sample size are very low, thereby showing the stability of EGISES.
To compare and correlate the leaderboard gener- ated by EGISES with standard accuracy measures that have been reported to have sufficiently fair human-judgment correlation (and therefore, can be trusted), we select two ROUGE variants (RG- L (Lin and Och, 2004) and RG-SU4 (Lin, 2004)), BLEU-1 (Papineni al., 2002), and METEOR (Banerjee and Lavie, 2005) (for a summary, see Appendix C.I). We use three standard correlation measures- Pearson's 's Correlation Coefficient (r), Spearman's ' Coefficient, and Kendall's r Coeffi- cient (see Appendix C.2).
5.3 Accuracy is Not Enough: Leaderboard Correlation Inconsistency
5 Model Performance w.r.f EGISES
Table 2 shows that leaderboard correlation between that of EGISES and the other four accuracy mea- sures w.r. the three chosen correlation measures is inconsistent and inconclusive". This supports our hypothesis that real- weal-world datasets (such as PENS) have all possibilities of triangulations. Therefore, the theoretical proof that personalization and accu- racy are unrelated (see Section 2.2) also empiri- cally established.
5.1 Experiment Design
There are three objectives regarding the state-of- the-art model evaluation: (i) leaderboard genera- tion w.r. EGISES, (ii) establishing that EGISES is stable measure, and (iii) comparing the leader- board with that of accuracy leaderboards. For the first two objectives, we conduct our experiments on ten random sample sets drawn from 100%, 80%, 60%, 40%, and 20% of the PENS test dataset. This is done to understand the bias and variance in the calculated EGISES scores, thereby measuring the stability of the measure. For the third objective, use the correlation measures introduced in Section 4.3. For the PENS framework models, evaluation is direct since the models were designed take user behavioral input in the way it is provided in the PENS dataset. However, for the other models, we need to appropriate the evaluation setup since they are not explicitly designed to be personalized. For these models, we add the gold reference summary of every reader as title to the document that the models had to summarize, thereby creating mul- tiple versions of the document corresponding to each reader. We then expect the model to take the injected titles as cues during summary generation, thereby inducing a personalization behavior. This injection also serves as a good baseline model.
6 Reliability of EGISES
6.1 Meta Evaluation: Experiment Design As a part of standard meta-evaluation of EGISES, collecting human-judgmments for personalization can be practically infeasible. This is because to assess whether a model is insensitive or not, hu- man evaluator needs to go through 3-step pro- cess: (i) give judgment on the divergence be- tween the expected summaries of the readers, (ii) judgment on the divergence between the corre- sponding (personalized) summaries generated by any model, and then (iii) judge whether that diver- gence proportionally varies with expected summary divergences of the readers. This significantly dif- fers from having the human judge provide quality score on the model-generated summaries (H J) and, therefore, requires significantly more resources and time to create such dataset. One way to resolve this bottleneck is utilize the (conditioned) tran- sitivity property of Pearson's correlation (Lang- ford et al., 2001) to establish a sufficiently high r( H J, EGISES). Given that r(HJ, Acc) is high (> 0.7) in most standard datasets such as CNN/DM and TAC-2008 (for RG-L) (Bhandari et al., 2020), and DUC-2001/2002 (for RG-L/SU-4) (Lin, 2004),
5.2 Model Analysis & EGISES Stability
Our first observation (see Table 1) is that there is significant scope for improvement for person- alized summarization considering the base mod- els the best PENS model is PENS-NAML (TI), which ranks sixth with an EGISES score of 0.8991 as compared to the best model BiigBird-Pegasiee that scores 0.4286. The induced personalization
"Appendix contains accuracy scores and ranking.

---

PENS Test Dataset Sample Set (Random Selection)
Ranking
Models
100%
60%
40%
20%
Bias
Variance
SesHird-Pegasin
0.4286
0.4281
0.4208
0.4317
0.4265
0.0027
1.27F-05
SiMCLS
0,5574
0.5583
0,5599
0.5576
0,5503
0.0025
1.11E-05
BRIO
0,6610
0.6599
0.6612
0.6604
0.6597
0.0005
3.70E-07
ProphetNet
0,8655
0.8650
0.8614
0.8661
0.8624
0.0017
3 3,42E-06
5
T5 (Base)
0,8817
0.8821
0,8844
0,8773
0.8872
0.0026
1,07E-05
6
PENS-NAML T1
0.8991
0.8999
0.8963
0.8998
0,9004
0.0016
2.17E-06
7
PENS-NRMS T1
0,9164
0.9166
0.9167
0.9142
0,9125
0.0011
2.85E-06
8
PENS-EBNR T1
0.9531
0.9536
0.9535
0.9543
0,9500
0.0011
1.79E-06
9
PENS-EBNR T2
0.9951
0.9947
0.9955
0.9957
0.9945
0.0004
1.97E-07
10
PENS-NRMS T2
0.9971
0.9975
0.9976
0.9973
0.9956
0.0005
5.33E-07
Table Degree-of-
Degree-offpersonaliaatiiee personalization (unsensttvity..ossbbbecvvvty benchmark using EGISES on ten SOTA summa-
rization models (lower score is better).
Correlation Measure
RG-L.
RG-SU4
BLEU
METEOR
Pearson
-030955
-0.7879
0.1272
-0.8844
EGISES
Spearman P
0.0666
-0.5555
0.2444
-0.51111
Kendall
0.1757
-0.6848
0.2969
-0.7333
Table 2: Personalization (using EGISES) vs. Accuracy based Leaderboard Rank Agreement Inconsistency.
therefore conclude high r(H if the condition below holds:
Ground Truth
nours (Proposed)
r Acc,) + rDDevAces Devecisrs) ((HH,,
Day, DeviS... Sebig Devis,.. Allot Dwy, Pam!
(12)
r( Aces DEVEGISES) (calculated as per Figure 3) denotes the agreement in the degree-of- personalization ranking if replace the pro- posed divergence measure (DevgGises) introduced in Section 3.2 with that of standard accuracy mea- sures that are used to measure divergence, but be- tween model-generated summaries and reference- summaries. Therefore, according to Eq. 12, r( ADe DevEGISES) > 0.3 would be sufficient to establish required r( J, EGISES).
Figure Correlating deviaton w.r.t chosen accuracy measure Deu and proposed DIN SECISES.
that even when the primary objective of eval- uator to measure the accuracy of a model, one should also take into account the ability of person- alization of the model to get realistic accuracy judgment. We call such an accuracy measure as P Acc. However, we would like to emphasize that the key objective of P Acc is not to improve accuracy ranking but rather to achieve more re- liable accuracy value (or score) so that we benchmark our personalized SOTA summarization models better and understand the scope of further improvement. Therefore, rather desirable that P Ace should have a high correlation with cor- responding Acc. Considering this objective, this section aims to establish that EGISEScan be good plugin for P Acc.
6.2 Meta Evaluation: EGISES Reliability
We generate degrre--f-perrsnnlizziiii leader- board based on deviation calculation using all the selected standard accuracy measures (i.c., RG- L/SU-4, BLEU, METEOR). We can observe from Table 3 that in all cases, the Pearson correlation is 0.4, with RG-L having very high correlation of 0.8954. This result helps to conclude that claim sufficiently high r(H EGISES).
7 Personalized Accuracy
In this paper, primarily focus on proving theo- retically and establishing empirically that person- alization a separate attribute from accuracy (and therefore, the leaderboards of both do not corre- late). Having said that, this section, we show
7.1 Formulation
To design P-Ace w.r.t given accuracy measure Acc and score Score (co,e). we can think of the model (Mo..) performance as a vector

---

Correlation Measure
DEVRO-L 0.8954
DeVRG-SU su4 0,5187
DeCPLEU 0.4209
DeEMETEOR
Devrrrss
Pearson
0,5639
Table 3: Analysis of personalization leaderboard Pearson correlation between Dec and DEVEGISES.
Corr.
P-RG-L, RG-I.
P-RG-SU4 vs. RG-SU4
P-BLEU BLEU P-METHOR METEOR
Pearson
0.9707
0.9957
0.9795
0,9918
Spearman p
0.6888
0,8090
0.9111
0.9555
Kendall 0.8182
0.9118
0.9636
0.9878
Table 4: Effect of P-Accuracy on Accuracy Leaderboard (higher is better)
IR2: Score Now, we can generalize EGISES(MM,.)
perform correlation analysis and find that the Kendall 7 coefficient is highly positive (lowest of 0.8182 for RG-L and maximum of 0.9878 for ME- TEOR) (see Table for details). This means in- corporating personalization into the accuracy mea- sures will not adversely affect a model's expected accuracy. However, P-Acc can give more real- istic understanding of how accurate personalized summarization model is.
Score (M),a) as multiplication of the score with its associated unit-scale (Un).Aec). In stan- dard cases, Unit 1. We propose that to under- stand the accuracy of personalized summarization models, the model's original accuracy score should be penalized. In other words, Unit should be penalized in proportion to the EGISES score as fol- lows.
8 Related Work
Une -1 |a- (ie-((at
( / (E..SE
AcoreAce(MM..))
Personalization in summarization models be broadly categorized into two types - iterative hu- man feedback based models (IHF models), and user-profile based models. In IHIF-based models. the reader keeps giving feedback on the summary iteratively till the reader satisfied with the model- generated summary. Ghodratnama et al. (2020b) proposed personalized summarization approach in which Exdos (Ghodratnama et al., 2020a) is used base model for extractive summarization model rank sentences of news body, and then concepts are extracted and shown to readers for their feedback. The system uses this feedback to iteratively generate a summary that includes the most important concepts till no further negative feedback. However, the evaluation metric used was ROUGE variants as accuracy metrics, and estab- lished in this paper, cannot be used for measuring personalization. They also measured the change in ROUGE the iterations increased, but that still is not a measure of subjective deviation. On the other hand, user-profile- based models such as those that were designed using the PENS framework (Ao et al., 2021), which we studied extensively, need significant improvement in personalization. These models, too, were evaluated using ROUGE vari- ants w.r.t accuracy. In the area of personalized recommendation and search, we can find Jaccard Index-based measures, order edit distance-based measures (Hannak et al., 2013), and the popular nDCG (normalized Discounted Cumulative Gain)-
(13)
Here, E (0, the personalization-coefiiieee that controls how much importance one can ex- pect to give to the personalization dimension of the vector (i.e., the squashing of the correspond- ing personalization basis vector)?. a E [0, 1] is the
and regulates the final penalty that needs to be applied. Saig is a sigmoid function to prevent the original accuracy score from getting overly dampened. As a result, we can now formulate P-Ace as.
P-Acc(Map (o. = AcoreAcc(Mo..)) Unit
(14)
We take 1 understand the extreme effect of EGISES the original accuracy leader- board and a 0.5 so as to over- per-penaliiee a fairly accurate model for poor personalization. is up to the evaluator to decide how much personal- ization is to be emphasized (controlled by 8 value) during accuracy evaluation and what percentage of the overall penalty (due to lack of personaliza- tion) (controlled by a:) should be injected into the original accuracy score value.
7.2 Meta Evaluation: P-Ace Leaderboard Correlation
To analyze how stable P-Ace is when compared changes in the corresponding accuracy measure,
7 0 would result in Anc UnitAca

---

based measures (Matthijs and Radlinski, 2011). However, these are not directly applicable to text summarization and are also extrinsic, relying on human feedback (clicks, likes, etc.), and therefore, cannot be automatic intrinsic measures.
the LLMs' capabilities of discerning what is sub- jectively valued and, more importantly what is not. We would like to highly encourage fellow researchers to do serious review of this work and build on the proposed EGISES framework. We would also like to declare that we used the PENS dataset prepared and released by Microsoft Re- search and did not involve any human subject for any part of the evaluation meta-evaluation.
9 Conclusion
In this paper, we establish theoretically and em- pirically that accuracy measures are unsuitable to evaluate the degree-of pfrrooaiiiprsoonliaaaainnnno of sum marization models, specifically when saliency is fairly subjective. We introduce a measure, called EGISES, and show that EGISES is both sta- ble w.r.t bias and variance and reliable w.r.t human- judgment correlation when we analyzed the degree- of pepersonalization of ten summarization models. As an extension, EGISES needs to be appropriated for capturing the sensitivity of models to the time- variant evolving interests of readers.
A Acknowledgements
This work could not have been completed in time without all the accuracy results generated by Isha Motiyani (002118015@daict.a..... Nidhi Somaiya (202218060 2112188000daicc.c.c..m daiict.ac.in), and Radhika Singhal (202218063) daiict.ac.in) who are re- search assistants at KDM Lab, DA-IICT, India.
References
Limitations
Xiang Ao, Xiting Wang. Ling Luo, Ying Qiao, Qing He, and Xing Xie. 2021. PENS; A dataset and generic framework for personalized news headline genera- tion. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 82-92, Online. Association for Computational Linguistics.
As discussed in Section 6.1, creating a dataset for direct meta-evaluation of EGISES is crucial to have a quantitative understanding of the correlation be- tween human judgment and the proposed measure. At present, we have only been able to establish that there a sufficiently high correlation. We are currently in the process of opening up an online survey specifically for the creation of this dataset. Also, currently, we are not in a position to con- clude what kind of metric space EGISES should be defined (in this paper, we took probability space) and what distance metric for measuring deviation would yield the best human-judgment correlation (we used -aasonnSaanoon Divergence and tried to understand how that correlated if replaced by stan- dard accuracy measures but as deviation measures). Finally, we do not have conclusive results of the ef- fect of different contextual embeddings from other SOTA LLMs (specifically GPT-X models) on the overall leaderboard and human-judgment correla- tion. Thorough ablation studies required.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Pro-
Manik Bhandari, Pranav Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. arXiv preprint arXiv.2010.07100.
Eirini Chatzikoumi. 2020. How to evaluate machine translation: A review of automated and human met- rics. Natural Language Engineering, 26(2):137-161.
Pierre Jean A Colombo, Chloé Clavel, and Pablo Pi- antanida. 2022. Infolm: A new metric to evaluate summarization & data2text generation. In Proceed- ings of the AAAT Conference on Artificial Intelligence, volume 36, pages 10554-10562.
Ethics Statement
We believe there needs to be formal frame- work within which both theoretical and empirical analysis of the personalization capabilities of con- temporary large language models (LLMss) can be evaluated. Personalization is not only an impor- tant aspect of "inteelligence,, but it also elucidates
Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A Statistical Analysis of Summarization Evaluation Metries Using Resampling Methods. Transactions of the Association for Computational Linguistics. 9:1 1:1132-1146

---

Abdul Ghafoor Etemad, Ali Imam Abidi, and Megha Chhabra. 2021. Fine-tuned t5 for abstractive sum- marization. International Journal of Performability Engineering, 17(10).
pages 7871-7880, Online, Association for Computa- tional Linguistics.
Chin-Yer Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- rion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Alexander R Fabbri, Wojeiech Kryšciñíski. Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391
Chin-Yew Lin and Franz Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and skip-bigram statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- 04), pages 605-612.
Yang Gao, Wei Zhao, and Steffen Eger. 2020. Supert: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. arXiv preprint arXiv:2005.0372
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Yanjun Gao, Chen Sun, and Rebecca J Passonneau. 2019. Automated pyramid summarization evaluation. In Proceedings of the 23rd Conference Computa- tional Natural Language Learning (CoNLL.).
Samira Ghodratnama, Amin Beheshti, Mehrdad Zak- ershahrak, and Fariborz Sobhanmanesh. 2020a. Ex- tractive document summarization based on dynamic feature space mapping. IEEE Access, 8:139084- 139095.
Yixin Liu and Pengfei Liu. 2021. Simels: A simple framework for contrastive learning of abstractive summarization. arXiv preprint arXiv:2106. 0.0000.........
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Samira Ghodratnama, Mehrdad Zakershahrak, and Fari- borz Sobhanmanesh. 2020b. Adaptive summaries; personalized concept-based summarization ap- proach by learning from users' feedback. CoRR, abs/2012. 13387.
Yvette Graham. 2015. Re-evaluating automatic sum- marization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128- 137, Lisbon, Portugal, Association for Computational Linguistics.
Nicolaas Matthijs and Filip Radlinski. 2011. Personaliz- ing web search using long term browsing history. In Proceedings of the fourth ACM international confer- ence on Web search and data mining, pages 25- 35-34.
M.L.. Menéndez. J.A. Pardo, L. Pardo, and M.C. Pardo. 1997. The jensen-shannon divergence. Journal of the Franklin Institute, 334(():307-318.
Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki. Balachander Krishnamurthy. David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring personalization of web search. In Proceedings of the 22nd international conference on World Wide Web, pages 527-538.
Ani Nenkova and Rebecca J Passonneau. 2004. Evaluat- ing content selection in summarization: The pyramid method. In Proceedings of the human language tech- nology conference of the north american chapter of the association for computational linguistics: Hit- naacl 2004, pages 145-152.
Raghav Jain, Vaibhav Mavi, Anubhav Jangra, and Sri- parna Saha. 2022. Weightee imput document augmented rouge. In Advances in Information Re- trieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I, pages 304-321. Springer.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding- news recom- mendation for millions of users. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 1933-1942, New York, NY, USA. Association for Computing Machinery.
Eric Langford, Neil Schwertman, and Margaret Owens. 2001. Is the property of being positively correlated transitive? The American Statistician, 55(4):322- 325.
Mikhail Orzhenovskii. 2021. T5-long-extract at fns- 2021 shared task. In Proceedings the 3rd Financial Narrative Processing Workshop, pages 67-69.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics,
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-33 111-18..
BART: Denoising sequence-to-sequees

---

Maxime Peyrard. 2019, Studying summarization eval- uation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5093- 5100, Florence, Italy. Association for Computational Linguistics.
A Out-of-Vocabulary Smoothing
While generating the distribution of the summary, there can be w,00y Sugu that are present in the summary but not in the document itself, in that case rather assigning probability assuming that the word is not part of the document, we proposed an algorithm to handle such cases. We proposed Out-of- Vocabulary Smoothing algorithm that cal- culates the probability of having an alternative or augmentation of the woo word in the document. To explain the algorithm with an example, consider toy document: "The red cat is on the red tall ta- ble." " which after preprocessing would be the bow: [red, cat, red, tall, table). Let's say that the reader's s expected summary (gold-reference) is: "The cat is on table" whose bow is: [cat, table). . Now, given the document and the reader's profile an abstractive model generates a summary "The red cat is on the desk", desk being the wgOV, and whose bow is: (red, cat, desk). As per table 5, we can see that desk can be considered an alternative table, and therefore gets a smoothened probability mass of 0.3805 by doing a sum of all softmax (o) simi- larity score of words in the document. Intuitively, softmax similarly score acts as the probability of that word being alternative or augmentation of the w wov Suzz. It's possible that the model gener- ates completely new word, i.e., unrelated to any of the words in the document, which means the probability of that word in the document should be 0. In our algorithm, bias gets the highest similarity score among all scores in that case because bias is calculated from the highest similarity score of words in the document, and the highest similarity word in the document itself gets low similarity score.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for eqquence-to-sqquenee pre-training. arXiv preprint arXiv:2001.04063.
GS Ramesh, Vamsi Manyam, Vijoosh Mandula, Pavan Myana, Sathvika Macha, and Suprith Reddy. 2022. Abstractive text summarization using 15 architecture. Proceedings of Second International Conference on Advances in Computer Engineering and Commu- nication Systems: ICACECS 2021, pages 535-543. Springer.
T Tawmo, Mrinmoi Bohra, Pankaj Dadure, Partha Pakray, et al. 2022. Comparative analysis of 15 model for abstractive text summarization on different datasets. In Proceedings of the International Con- ference on Innovative Computing Communication (ICICC) 2022. SSRN
Johnny Wei and Robin Jia. 2021. The statistical advan- tage of automatic NLG metrics at the system level. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6840-6854, Online. Association for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang. Yongfeng Huang, and Xing Xie. 2019a. Neu- ral news recommendation with attentive multi-view learning.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019b. Neural news recom- mendation with multi- sclf-attention. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 6389-6394, Hong Kong, China. Association for Computational Linguisties.
B Models Details
We briefly introduce the SOTA summarization mod- els that were analyzed to understand their degree- of personalization below:
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang. and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pages 17283-17297. Curran Associates, Inc.
PENS-NRMS Injection- Type 1: The PENS framework (Ao et al., 2021) takes user em- bedding as input along with the news arti- cle to generate personalized summary for that user. To generate user embedding NRMS
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019, Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:7904.09675
'In our implementation, we used the ail-distilroberta-r model from Huggingface. This model maps sentences para- graphs to layers, 768 dimensions and 12 heads, and 82M parameters.
dense vector space and has 6

---

pov
E d
Soos (edelk: Pur,
a afeor (Edosk: Cw,
OOV dcsk)
red
0.537
0.2
0.405
0.175
desk
tall
0.613
0.216
0.3805
table
0.892
0.285
bias[desk]
0.889 < 0.892
0.124
Table 5: OOV handling example
Algorithm Out-of-Vocabult Smoothing
coder's hidden state of the headline generator, which will influence the summary generation.
Input: dj. Suyk
Output: OOV (Estimated probabilities of
PENS-NRMS Injection-Type To gen- erate a personalized summary, NRMS user embedding injected into attention values (Injection-Typp 2) of PENS that helps to per- sonalize attentive values of words in the news body.
OOV word-phrases as allemativee//ugmenatiin to document phrases)
procedure GETEMBEDDING(d, RoBERTa) for wooov E Suza k 1ov & dj) do
ewo GeIEmbedding(w@00 RRoBERT))
3. PENS-NAML Injection-Type 1: NAML (Neural News Recommendation with Atten- tive Multi Multi-View Learning) (Wu et al., 2019a) incorporates a news encoder that utilizes a multi-view (i.c., titles, bodics, and topic cat- egories) attention model to generate compre- hensive news representations. The user en- coder is designed to learn user representations based on their interactions with browsed news. It also allows the selection of highly infor- mative news during the user representation learning process. This user embedding in- jected into the PENS model using Type-1 for personalization.
maxsim [7]
foox few lu dj})}
bias(i) + scoresiml
if bias(]] [i] scoresim then
00v[1] O wooo is not alterna- tive/augmentation usage (i.e. probability or no smoothing required))
else
sumssmm
Σ (foool(e (ew /ew/lup E dj}))
(o: Soottaa(())
count(w (ov
Y/Naujk (Nauge² total num-
4. PENS-EBNR Injection-Type 1: EBNR (Embedding- based News Recommendation for Millions of Users) (Okura et al., 2017) proposes method for representations by using RNN model that takes browsing histories as input sequences. This user embed- ding is injected using Type into the PENS model for personalization.
O
summsim ber of word-phrase in suju-1
return OOV
(Neural News Recommendation with Multi- Head Self-Attention) (Wu al., 2019b) is used. It includes news encoder that utilizes multi-head self-attentions to understand news titles. The user encoder learns user represen- tations based on their browsing history and uses multi-head self-attention to capture con- nections between news articles. Additive at- tention is added learning the news and user representations more effectively by selecting important words and articles. Here, Injection- Type indicates that NRMS user embedding is injected into PENS by initializing the de-
5. PENS-EBNR Injection- Type 2: This person- alized model injects EBNR user embedding into PENS using type-2.
6. BRIO: Instead of traditional MLE-based training approach, BRIO (Liu al., 2022) sumes a non-deterministic training paradigm that assigns probability mass different can- didate summaries according to their qual- ity, thereby helping it to better distinguish

---

between high-quality and low-quality sum- maries.
2. RG-SU4: In addition capturing uni- gram, bigram, and trigram matches, ROUGE- SU4 (Lin, 2004) also considers skip-bigram matches, which allow for gaps of cer- tain words between the matched n-grams, thereby also considering non-contiguous n- gram matches.
7. SimCLS: SimCLS (A Simple Framework for Contrastive Learning of Abstractive Summa- rization) (Liu and Liu, 2021) uses two- stage training procedure. the first stage. Seq2Seq model (BART (Lewis et al., 2020)) is trained to generate candidate summaries with MLE loss. Next, the evaluation model, initiated with RoBERTa is trained to rank the generated candidates with contrastive learn- ing.
3. BLEU: BLEU (Bilingual Evaluation Under- study) (Papineni al., 2002) is popular evaluation metric that measures the preci- sion of n-gram matches between the model- generated summaries and the reference sum- maries. BLEU computes a modified preci- sion score for various n-gram lengths and then combines them using a geometric mean.
8. BigBird-Pegasus, BigBird (Zaheer et al., 2020) is extension of Transformer based models designed specifically for processing longer sequences. utilizes sparse attention, global attention, and random attention mecha- nisms to approximate full attention. This en- ables BigBird to handle longer contexts more efficiently and, therefore, can be suitable for summarization.
4. METEOR: METEOR (Metric for Evalua- tion of Translation with Explicit ORdering) (Banerjee and Lavie, 2005) matches unigrams based on surface forms, stemmed forms, and meanings and then calculates score using a combination precision, recall, and the order- alignment of the matched words w.r. refer- ence summary.
9. NrophetNet: ProphetNet (Qi et al., 2020) is a sequencce-o--sequene pre-trained model that employs n-gram prediction using the n-stream self-attention mechanism. ProphetNet opti- mizes n-step ahead prediction by simultane- ously predicting the next tokens based on previous context tokens, thus preventing over- fitting on local correlations.
C.2 Correlation Measures
Pearson's Correlation Coefficient (r):
E (1: 336
r
E (2 - x) 2ド 7)
10. T5:
T5 (Text-To-Tie Text Transfer Transformer) is based on the Transformer-based Encoder- Decoder architecture that operates on the principle of the unified text-to-text task for any NLP problem, including summarization. Some recent analysis on the performance of T5 summarization task can be found in (Tawmo et al., 2022; Ramesh et al., 2022; Etemad et al., 2021).
where T, J are the means of the variables x; and yi n the number of samples.
Spearman's p Coefficient:
6 6Σ dB 0
nnn2 - 1)
where d the pairwise distances of the ranks of the variables e; and yi the number of samples.
Accuracy and Performance
C.1 Accuracy Measures Compared
3. Kendall's Coefficient:
1. RG-L: ROUGE-L (Rreall-Oriented Under- study for Gisting Evaluation) (Lin and Och, 2004) calculates the longest common subse- quence between the generated summary and the reference summary and then measures the precision, recall, and Fl score based on this comparison.
c-d d
S 25 n(n 1)
T
+ d =
where, c the number of concordant pairs; d = the number of discordant pairs.

---

Leaderboard (PENS test dataset)
Models
EGISES
RG-L
RG-SU4
BLEU
METEOR
BigBird- Pegasus
1
SimCLS
2
7
7
BRIO
3
8
8
ProphetNet
T5 (Base)
5
PENS-NAML
PENS-NRMS
PENS-EBNR
8
PENS-EBNR
9
PENS-NRMS
Table 6: Personalization vs. Accuracy: Leaderboard rank disagreement
C.3 Accuracy Leaderboard
table 6, each column contains the ranking of the summarization model as that specified measure. The table is sorted by the ranking of EGISES.c The ranking of model is inconsistent across dif- ferent measures due to that the correlation between EGISESand the other four accuracy measures is low.


Reviews:
[{"review_id": "nqCwfb4cYk", "reviewer": "Reviewer_wqBM", "summary": "", "questions": "Please refer to the content of “Reasons To Reject”.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Rationale for Exclusion of BERTScore:}$ There were primarily three reasons to exclude BERTScore: (i) according to findings in Deutsch et. al. [EMNLP, 2022], BERTScore was reported to have a significantly lower human correlation on RealSumm (news dataset based on CNN/DM where relevancy annotation was done using a lightweight version of PYRAMID - a highly rigorous manual summarization evaluation metric) in comparison to the metrics that we have chosen, (ii) the system ranking (i.e., model leaderboard) stability of BERTScore was found to be significantly lower, as was evident from the relatively large 95\\% CI (Confidence Interval), (iii) according to the findings in Fabbri et. al. [TACL, 2021], the inter-correlation of system leaderboard generated by BERTScore with that of RG-based metrics was shown to be very high on SummEval dataset (news dataset based on CNN/DM), as compared to other metrics such as METEOR and BLEU. Based on this result we decided that contrasting accuracy leaderboard w.r.t BERT with that of personalization leaderboard w.r.t EGISES will be redundant. Hence, for lack of strong motivation to include BERTScore, we decided to abandon the experiment that otherwise would have been computationally very expensive.\nAlso, apart from ROUGE variants (RG-L, RG-SU4) we have included other non-ROUGE metrics like BLEU and METEOR.\n\n$\\textbf{Computational Complexity of EGISES:}$ The computational complexity can be a legitimate concern if it has an unreasonable wall-clock runtime. We observed an average runtime of 2:37 minutes over an evaluation dataset of 3840 documents (each document containing 4 user reference summaries and corresponding model generated summaries). So, evaluating all 10 models will not take more than 25 minutes, which is reasonable for an offline procedure. $\\textbf{NOTE: System specifications}:$ Machine architecture: x86\\_64; CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz; CPU Cores: 16; Thread(s) per core: 2. Also, as shown below, it can be proved that EGISES will not be computationally more costly (asymptotically) when compared to any other alternative and more simplistic metric.    \n$\\textbf{Complexity Analysis:}$ The time complexity of each summary-level Deviation $Dev(s_{u_{ij}} |(d_i, u_{ij}))$: $|U| \\times O(4 \\times t_{JSD}) = O(|U| \\cdot t_{JSD})$, where $|U|$ is the total number of users in the evaluation dataset and $t_{JSD}$ is the time-complexity of computing JSD. The worst-case $t_{JSD} = O(l_s+l_u)$, where $l_s, l_u$ are lengths of $s, u$. Since any degree-of-personalization metric will have to comparison-based, therefore the lower-bound will be $\\Omega(|U| \\cdot \\sigma(\\bullet,\\bullet))$. If $\\sigma$ is $ROUGE$ (i.e., we use $ROUGE$ as a deviation measure instead of accuracy measure) then applying suffix-tree based matching we can obtain a worst-case complexity of $O(l_s+l_u)$. So, even with a relatively simple measure the best we could have achieved remains the same asymptotically - i.e., $O(|U| \\cdot (l_s+l_u))$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper theoretically proves that EGISES can model personalization for the summarization task.\n* The correlation analysis with the existing measures proves that P-acc differs from the existing measures.\n* The paper is well written and organized.", "reasons_to_reject": "* The paper only compares with the measures of the ROUGE series and lacks comparisons with other classic measures, such as BERTScore.\n* Equation 9-10: EGISES needs to calculate the JSD divergence multiple times, and the computational complexity is relatively high.\n", "questions_to_authors": "Please refer to the content of “Reasons To Reject”.", "paper_topic_and_main_contributions": "The authors propose a new metric, EGISES, to measure personalization in summary generation. The paper proves the theoretical feasibility of measuring personalization, and then proposes a personalization measurement method based on JSD divergence.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Formulas in Preliminaries should also be numbered."}, "extra_text": {}}, {"review_id": "QwKikIAekm", "reviewer": "Reviewer_QpF4", "summary": "", "questions": "-  Line 176: the definition of degree-of-personalization and Figure caption seems contradictory. Assuming figure 1 is correct, shouldn’t the ratio be inverted?\n\n-  Why didn't you include BERTScore in the evaluation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Clarification on $P\\text{-}Acc$ correlation with $Acc$:}$ The paper primarily focuses on proving theoretically and establishing empirically that personalization is a separate aspect from accuracy (and therefore, the leaderboards of both do $\\textbf{not}$ correlate). Having said that, in section 7, we show that even if the primary objective of an evaluator is to measure the accuracy of a model, one should take into account the ability of personalization of the model to get a $\\underline{realistic}$ accuracy judgment. We call such an accuracy measure as $P-Acc$. However, the key objective of $P-Acc$ is not to improve accuracy ranking but rather to achieve a more reliable accuracy $\\textbf{value}$ so that we can benchmark our personalized SOTA summarization models better and understand the scope of further improvement. Therefore, it is rather desirable that $P-Acc$ should have high correlation with corresponding $Acc$. Considering this objective, we have successfully established that EGISES can be a good plugin for $P-Acc$. It is up to the evaluator to decide how much personalization is to be emphasized (controlled by $\\beta$ value) during accuracy evaluation and what percentage of the overall penalty (due to lack of personalization) (controlled by $\\alpha$ value) should be injected into the original accuracy score. \n\n$\\textbf{Clarification on $f_{sig}$ and hyper-parameter setup:}$ $f_{sig}$ is a sigmoid function for squashing (i.e. reducing) the penalty score for higher EGISES values, thereby preventing the original accuracy score from getting dampened too much. In our experiment $\\beta = 1$ since we wanted to understand the extreme case effect of EGISES on the original accuracy leaderboard, and $\\alpha = 0.5$ (since we did not want to over-penalize a model for poor personalization that otherwise could be performing well in terms of accuracy alone). We will incorporate this detail in the camera-ready if accepted.       \n\n$\\textbf{Rationale for exclusion of BERTScore:}$ There were primarily three reasons to exclude BERTScore: (i) according to findings in Deutsch et al. [EMNLP, 2022], BERTScore was reported to have a significantly lower human correlation on RealSumm (news dataset based on CNN/DM where relevancy annotation was done using a lightweight version of PYRAMID (a highly rigorous manual summarization evaluation metric) in comparison to the metrics that we have chosen, (ii) the system ranking (i.e., model leaderboard) stability of BERTScore was found to be significantly lower, as was evident from the relatively large 95\\% CI (Confidence Interval), (iii) according to the findings in Fabbri et al. [TACL, 2021], the inter-correlation of system leaderboard generated by BERTScore with that of RG-based metrics was shown to be very high on SummEval dataset (news dataset based on CNN/DM), as compared to other metrics such as METEOR and BLEU. Based on these results, we decided that contrasting the accuracy leaderboard w.r.t BERT with that of the personalization leaderboard w.r.t EGISES will be redundant. Hence, for lack of strong motivation to include BERTScore, we decided to abandon the experiment that otherwise would have been computationally very expensive.\n\n$\\textbf{Deviation Ratio inconsistency in the formula:}$ The deviation ratio, described on page 2, is consistent in that the denominator will always be greater than or equal to the numerator because of a max/min normalization (see equation 6 for clarification). However, the total ordering property, as argued in the proof, and therefore the argument that deviation varies while accuracy is constant, is independent of what we keep in the numerator.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Personalization in summarization is an important task and needs more attention regarding evaluation. Therefore, this paper tackles a very important problem.\n\n- The paper provides theoretical and empirical evidence that EGISES can model the personalization of a summarization model.\n\n- The paper is nicely written, it was a pleasure to read.", "reasons_to_reject": "- Section 7.2 is hard to understand for me. What were the alpha and beta values you used for Table 6? I could not find any details on that. Table 6 results will vary based on alpha and beta, and it is important to study that sensitivity.\n\n- Table 6 also shows that P-Acc vs. Acc correlations are very high for Pearson and Kendall for all Acc metrics. I find it a little puzzling. The authors claim that:  \"This means that incorporating personalization into the accuracy measures will not have any adverse effect on the expected accuracy of a model. However, P-Acc will give us a more realistic understanding of how accurate a personalized summarization model is.\" I'm not sure if I see the point here. Why a high correlation is a good thing here? A high correlation between P-Acc vs. Acc means they essentially give us the same conclusion. Then, how is P-Acc helping? Am I missing something here?\n\n- Equation 13 is not clearly described. What is \"fsig\" here? What is the overall justification of this equation?", "questions_to_authors": "-  Line 176: the definition of degree-of-personalization and Figure caption seems contradictory. Assuming figure 1 is correct, shouldn’t the ratio be inverted?\n\n-  Why didn't you include BERTScore in the evaluation?", "paper_topic_and_main_contributions": "The paper presents theoretical and empirical evidence that accuracy measures are inadequate for assessing the degree of personalization in summarization models, especially when saliency is subjective. To address this limitation, the authors propose a new measure called EGISES. Their analysis of ten state-of-the-art summarization models demonstrates that EGISES is stable concerning bias and variance and exhibits a reliable correlation with human judgment in measuring the degree of personalization.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ApHkOY1b1B", "reviewer": "Reviewer_GHAL", "summary": "", "questions": "See above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Need for separate personalization measure:}$\nWe have proven mathematically that deviation varies while keeping accuracy constant. Using the same proof technique (i.e., rotation operator), we can also argue (mathematically) that subjective accuracy can vary (i.e., can have a total ordering) while keeping deviation constant. Since deviation is at the core of personalization evaluation, accuracy variance as a measure, therefore, will not be able to capture the degree of personalisation. Also, all the standard accuracy measures do not weigh in user-specific word preferences. This can result in a high variance in accuracy without any discrimination of core-topic match vs. peripheral-topic match between generated summaries and reference summaries, which is at the heart of personalization. Finally, we observed empirically that the accuracy leaderboard has no conclusive correlation with that of EGISES leaderboard at system-level and also at $\\textit{summary-level}$, thereby supporting the fact that in real-world datasets such as PENS, accuracy variance is not good enough. \n\nIt is noteworthy that if the same accuracy measures (such as ROUGE) are plugged into EGISES or a simplified EGISES (see Table 3) instead of JSD, then the personalization leaderboard correlation is acceptable (with RG-L having a high correlation of 0.89). In other words, the issue is not with the measures themselves (which could be used as alternatives in the proposed EGISES framework) but rather with using these measures for accuracy evaluation and expecting that we can evaluate personalization implicitly.    \n\n$\\textbf{Clarification on the use of PENS dataset only:}$ \nThe concern is valid. However, one must keep in mind that the five SOTA personalized summarization models we evaluated do user behavior modeling by embedding the temporal sequence in which a user interacts with documents, i.e., clicking, reading, and writing a reference summary (i.e., expected summary). For this, we need a dataset that should contain the same document to have multiple users' reference summaries together $\\textbf{with the user-interaction sequence timestamp}$. So far, we have not encountered any dataset that satisfies the criteria except the PENS dataset. We would love to include any similar dataset in our future studies whenever we find one.  \n\n$\\textbf{Clarification on lack of human judgment:}$\nThe primary objective of accepting any newly proposed measure as reliable is to show sufficient positive human judgment correlation. This has already been established in the paper in Section 6.2 using equation 12 (i.e., the correlation transitivity property; see Section 6.1), where already published human-correlation results on the news dataset (CNN/DM) are being used. We are currently understanding, quantitatively, how much this positive correlation is via a survey-based human judgment dataset creation and how much we can increase the positive correlation using different alternative distance measures within the EGISES framework. \n\n$\\textbf{Survey setup and initial results:}$ We have collected 110 participant responses so far (61 female, 49 male). Each participant (undergraduate and graduate students studying computer science, mathematical science, electronic engineering, and humanities) is shown six pairs of summaries from the PENS dataset such that one pair consists of randomly selected users' reference summaries, and the other five are corresponding model-generated summaries from five different models. So, with two consecutive participants who are shown the same pair of users' reference summaries, we are able to cover all the ten models we have evaluated along with 55 documents. The participants are not told that the pairs are summaries to avoid any prior bias. Instead, the pairs are shown as plain texts, and they are asked to provide a similarity rating between 1 (low) and 6 (very high) for every pair. This way, we can collect their unbiased judgment of how close two (subjective) user reference summaries are and how close their corresponding model-generated summaries are. We have so far observed a sufficiently high model-leaderboard correlation between EGISES and human judgment ($\\textbf{0.6197}$ Pearson's $r$; $\\textbf{0.5111}$ Kendall $\\tau$; $\\textbf{0.7333}$ Spearman $\\rho$).       \n\n$\\textbf{Computational complexity of EGISES:}$ The computational complexity can be a legitimate concern if it has an unreasonable wall-clock runtime. We observed an average runtime of 2:37 minutes over an evaluation dataset of 3840 documents (each containing four user reference summaries and corresponding model-generated summaries). So, evaluating all 10 models will not take more than 25 minutes, which is reasonable for an offline procedure. $\\textbf{NOTE: System specifications}:$ Machine architecture: x86\\_64; CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz; CPU Cores: 16; Thread(s) per core: 2. Also, as shown below, it can be proved that EGISES will not be computationally more costly (asymptotically) when compared to any other alternative and more simplistic metric.   \n$\\textbf{Complexity analysis:}$ The time complexity of each summary-level deviation $Dev(s_{u_{ij}} |(d_i, u_{ij}))$: $|U| \\times O(4 \\times t_{JSD}) = O(|U| \\cdot t_{JSD})$, where $|U|$ is the total number of users in the evaluation dataset and $t_{JSD}$ is the time-complexity of computing JSD (there are 4 JSD computations and not 8, please refer Equations 9 and 10). The worst-case $t_{JSD} = O(l_s+l_u)$, where $l_s, l_u$ are lengths of $s, u$. Since any degree-of-personalization metric will have to be comparison-based, the lower bound will be $\\Omega(|U| \\cdot \\sigma(\\bullet,\\bullet))$. If $\\sigma$ is $ROUGE$ (i.e., we use $ROUGE$ as a deviation measure instead of an accuracy measure), then applying suffix-tree-based matching, we can obtain a worst-case complexity of $O(l_s+l_u)$. So, even with a relatively simple measure, the best we could have achieved remains the same asymptotically - i.e., $O(|U| \\cdot (l_s+l_u))$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and structured. \n\n* The paper contains several theoretical proofs.", "reasons_to_reject": "The primary concerns include,\n\n* The necessity of evaluating the degree of personalization is not clear to me. \n\n  - According to this paper, I only found three previous research that did personalized summarizers. And all of them utilize the current common metrics to measure performance. It seems these metrics are enough for this task.\n\n  - Let's assume the new evaluation metric is necessary. When we have the pairs of user profiles (such as user-expected summaries) and generated summaries for each user, why can we not use the average and variance of current metrics (such as Rouge) to show the degree of personalization? The average presents the performance of the summarizer to generate high-quality summaries, and variance can represent the performance of generating summaries close to each user. It is much easier to evaluate based on current metrics than new ones. \n\n* The new proposed metric is only tested on a single dataset.\n\n* There is no human judgment for this new metric. I notice the authors said, in Limitations, they are trying for the human evaluation. I think it is better to accept the next version with human judgment results.\n\n* The metric is high time-cost due to the eight Jenson-Shannon Divergence calculations.\n\nBesides, the details of $rot()$ were missed in Line 184.", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "This paper focuses on the personalization evaluation in the summarization task. The authors propose EGISES to measure the degree of personalization.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 177, it should be the distance between two spaces rather than RATIO."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces EGISES, a novel metric for evaluating personalization in summarization. Reviewers acknowledge the paper's theoretical contributions and its importance in addressing personalization in summarization, praising its clarity and organization. However, concerns include limited comparisons with classic metrics like BERTScore, computational complexity due to multiple JSD calculations, ambiguity in correlations between P-Acc and Acc, questions about the necessity of the metric compared to existing measures like ROUGE, limited dataset testing, absence of human judgment, and missing details. During discussions, there was a strong concern regarding the lack of human judgment and the fact that the metric was not compared with baseline metrics. While the authors have shown new results during the response period, it was not possible to check for soundness.
Decision: Accept-Findings