PerSEval: Assessing Personalization in Text Summarizers
Authors: Sourish Dasgupta
Ankush Chander
Tanmoy Chakraborty
Parth Borad
Isha Motiyani
Venue: Accepted by TMLR ()

Abstract: Personalized summarization models cater to individuals' subjective understanding of saliency, as represented by their reading history and current topics of attention. Existing personalized text summarizers are primarily evaluated based on accuracy measures such as BLEU, ROUGE, and METEOR. However, a recent study argued that accuracy measures are inadequate for evaluating the $\textit{degree of personalization}$ of these models and proposed EGISES, the first metric to evaluate personalized text summaries. It was suggested that accuracy is a separate aspect and should be evaluated standalone. In this paper, we challenge the necessity of an accuracy leaderboard, suggesting that relying on accuracy-based aggregated results might lead to misleading conclusions. To support this, we delve deeper into EGISES, demonstrating both theoretically and empirically that it measures the $\textit{degree of responsiveness}$, a necessary but not sufficient condition for degree-of-personalization. We subsequently propose PerSEval, a novel measure that satisfies the required sufficiency condition. Based on the benchmarking of ten SOTA summarization models on the PENS dataset, we empirically establish that -- (i) PerSEval is reliable w.r.t human-judgment correlation (Pearson's $r$ = 0.73; Spearman's $\rho$ = 0.62; Kendall's $\tau$ = 0.42), (ii) PerSEval has high rank-stability, (iii) PerSEval as a rank-measure is not entailed by EGISES-based ranking, and (iv) PerSEval can be a standalone rank-measure without the need of any aggregated ranking.

PerSEval: Assessing Personalization in Text Summarizers
Sourish Dasgupta',, Ankush Chander', Parth Borad', Isha Motiyani', Tanmoy Chakraborty
Dhirubhai Ambani Institute of Information & Communication Technology, India
Indian Institute of Technology Delhi, India Corresponding authors: suurishh.asguptaeddiittttt in, tanchak@iitd.ac...
Abstract
Personalized summarization models cater to individuals' subjective understanding of saliency, as represented by their reading history and current topics of attention. Existing personalized text summarizers are primarily evaluated based on accuracy measures such as BLEU, ROUGE, and METEOR. However, a recent study argued that accuracy measures are inadequate for evaluating the degree of personalization of these models and proposed EGISES, the first metric evaluate personalized text summaries. It was suggested that accuracy is separate aspect and should be evaluated standalone. In this paper, we challenge the necessity of an accuracy leaderboard, suggesting that relying on accuracy-based aggregated results might lead to misleading conclusions. To support this, we delve deeper into FGISFS, demonstrating both theoretically and empirically that it measures the degree of responsiveness, a necessary but not sufficient condition for a novel measure that satisfies the required sufficiency condition. Based on the benchmarking of ten SOTA summarization models on the PENS dataset, we empirically establish that (i) PerSEval reliable w.r.t human-judgment correlation (Pearson's 's 0.73; Spearman's p 0.62; Kendall's 0.42), (ii) PerSEval has high rank-stability, (iii) PerSEval rank- measure is not entailed by EGISES- ranking, and (iv) PerSEval can be standalone rank-measure without the need of any aggregated ranking.
We subsequently propose PerSEval,
1 Introduction
With the incessant rise of information deluge, it has become even more imperative to develop efficient and accurate models to summarize the salient information in long documents for faster consumption, eliminating the irrelevant and supporting faster decision-making Ter Hoeve et al. (2022). However, the notion of saliency can be highly subjective in many use cases, particularly for documents containing multiple aspects and topics. This calls for summarizers that must be personalized to the users' preferences depicted by their reading behavior and current topic(s) of attention [Ghodratnama et al. 2021 Ao et al. 2021 This calls for robust. and reliable measures for evaluating the degree-of- personttatin in them.
Dearth of Personalization Evaluation. Major studies on evaluating text summaries focus on accuracy measurement and include the proposal of multitude of measures such as the ROUGE variants (e.g., ROUGE-n/L./SU4 etc.) Lin 2004). BLEU (Papineni et al.] 2002), METEOR (Banerjoc Lavic 2005),
BERTScore (Zhang et al.) 2020), PYRAMID (Gao al.) 2019) and the more recently proposed ones such SUPERT (Gao et al. 2020), WIDAR (Jain et al. 2022). and InfoLM (Colombo et all (022b). Other studies
acknowledged the need for more qualitative aspects such as consistency, coherence, and fluency (Yuan et al.
2021 Deng et al. 2021 Tam et al. 2023 Jain et al. 2023 Zhong et al., 2022). Recently, Vansh et al. (2023 established, both theoretically and empirically, that personalization a different. aspect than accuracy. The authors proposed a measure for degree-of personaiiaairooiiiiiatiiiooo for the first time and called it EGISES
The EGISES Paradox. To put succinctly, EGISES measures the average ratio of the (normalized)
deviation between model-generated summaries and their corresponding user-expected summaries, capturing

---

Accuracy
High
Low
Personalizateo aesonallizatin
High
Paradox
Figure 1: EGISES Personalization-Accmme Paradox. The absurd case of high personalization (thereby high user-experience), yet low accuracy.
the strong notion of a model's degree of insensitivity users' subjective expectations. However, we argue that this degree of insensitivity does not truly mcasure personalization but rather a related and necessary aspect, which can be understood as the responsiveness of model. A high degree of personalization must imply a very high-quality user experience (UX). However, one cannot expect high UX while having low accuracy. At the same time, we demonstrate that model can have high accuracy but a poor EGISES score. We resolve this apparent paradox by establishing both theoretically and empirically that EGISES, in reality, accounts for only the degree of responsiveness of models. In other words, we mathematically prove that a model can have a very good EGISES score but may still fall short of being personalized simply because of low accuracy performance.
P-Accuracy is not solution. As an extension to standard accuracy measures, Vansh et al. 20033 proposed measure called P-Accuracy (P-Acc) that penalizes accuracy by factor that is function of the EGISES score. Although P-Acc might apparently seem to address the EGISES paradox, in this paper, we prove that P-Ace fails to address the issue for two reasons. First, it has not been designed to evaluate the degree-of-personalzztiin of a model but rather to get a more realistic, human-evaluator adjustable accuracy score. This makes P-Ace unable to distinguish the two models, where one exhibits low accuracy but high EGISES while the other exhibits high accuracy but low EGISES. Secondly, we prove that P-Ace is unstable for models with relatively low accuracy and may lead to anomalous results if measure personalization using it. This motivates to propose a novel consolidated personalization mcasurement framework for text summarizers, called PerSEval (Personalized Summarizer Evaluator) that builds on the design principles of EGISES and bridges the "UX-gap'.
The underlying design principle of PerSEval entails that higher accuracy performance should not obfuscate the original EGISES score model. Otherwise, a model can be considered highly personalized simply because of its high accuracy, which is misleading, as was proven in (Vansh al., 2023 However, the converse should not be true; hence, lower accuracy score should penalize the original propose penalty factor, called EDP (Effective DEGRESS
PerSEval Design Principle.
EGISES score. Based on these design objectives, Penalty) that can be injected into FGISES to form PerSEval to measure the true degree of personalization EDP incorporates the inconsistency of accuracy of model w.r.t its best accuracy performance and how much that is off from the maximum achievable accuracy (which is 0 for a normalized metric). In the best case, the EDP factor comes to (i.e., the penalty 0), while in the worst case, tends to (i.e., the penalty 1).
Observations and Insights. We first empirically establish that the EGISES- based rank does not simply entail the PerSEval- based rank. In other words, models rank differently when EDP is applied. Therefore, we conclude that the EGISES paradox not just theoretically exists but has real evidence. We then show that PerSEval provides much more reliable ranking of models with significantly higher human-judgment correlation in terms of Pearson's r (0.73), Spearman's 's (0.62), and Kendall's T (0.42). For fair comparisons, we consider the same top ten state-of the-art news summarization models and the same PENS test dataset (Ao et al., (021) that |Vansh et al.] (2023) considered to evaluate EGISES. We also take a step beyond [Vansh] et al.] 2023) and demonstrate that the accuracy leaderboard is not only insufficient but is best redundant and at worst can be misleading for evaluating personalized summarizers. This is established by showing that.
2

---

the Borda-Kendan consensus-basse aggregated ranking (Cook & Seiford 1982) of the models does not have better human-judgemmnt correlation when compared to the correlation of PerSEval alone
2 Background
2.1 EGISES is Not Enough: The Personalization-Accuraay Paradox
As proposed in (aansh et al. 2023 the degree-of-rrssomazzzaiin is a quantitative measure of how much a summarization model
for personalization is adaptive to a user's (i.e. reader's) subjective expectation. This also implies that it measures how accurately a model can capture the user's "evolving" profile reflected through reading history (i.e., a temporal span of the reading and skipping actions of a user a sequence of documents that is interleaved by the actions of generating and reading summaries). This is because the subjective expectation is function of the reading history. A low degree of personalization, by definition, implies poor user experience. If a model does not efficiently capture the user's profile, it may lead to summaries that contain (a) additional irrelevant information, (b) are not coherent with the reading history, and (c) do not cover all the topics of interest within the reading history. In this situation, poor UX would mean that the user would have to spend more time getting to the information he/she is interested in or suffer from information overload and fatigue.
To illustrate this, we borrow the example given by Vansh al. (2023) where have two readers, Alice and Bob, having two very different reading histories over the same lime span. Alice has been primarily reading news articles mostly on the broad topic of "civilian distress' in the Hamas--Isaell conflict, while Bob is following up on articles related to 'war-front battles". In a situation where both Bob and Alice come up with an article covering both topics but rather wish to read the summary of the article instead, their respective expected summaries would be quite different. non-personalized summarization model can be considered 100% accurate w.r.t recall and F-score- based measures (e.g., ROUGE-variants and METEOR, respectively) if it generates a summary that is union of the expected summaries of Alice and Bob. On the other hand, model can be considered 100% accurate w.r.t precision- prased measures (eg., BLEU) if it covers the intersection between the expected summarics of Alice and Bob, which will not be much since they are quite different profiles. However, while the first situation would imply that Alice and Bob experience significantly redundant content in the generated summary that would drop their overall UX, the second case would entail that both Alice and Bob will have poor UX in terms of signifficntt lack of content coverage. Hence, the utility of any summarization model as experienced by any user is directly associated with the subjective UX and cannot be traded in for accuracy, which is a rather clinical scoring mechanism that does not truly reflect the UX of user in terms of redundancy and coverage (among other subjective aspects such as correctness and fluency).
This phenomenon was established theoretically and empirically by Vansh et al. (2023 for the first time, thereby emphasizing the fact that accuracy measures can mislead an evaluator to select a model that has poor UX. To address this, they proposed a novel measure, called EGISES, for personalization evaluation summarizers. However, in this paper we establish both theoretically and empirically that if EGISES is used for personalization evaluation (i.e., a measure to understand model's capacity to engage readers in terms of UX), then personalization (i.e., acceptable EGISES score) but low accuracy, and yet, by definition, that would entail high UX (see Figure D) In other words, although high accuracy can lead to poor UX, the inverse (i.e., low accuracy leading to high UX) is absurd. We term this as the porsonalization-ccuraa paradox and altribute it to the incorrect interprelation or usage of EGISES. In this paper, we propose PerSEval as a corrective measure of EGISES that resolves this paradoy In the following section, we show that EGISES measures responsiveness, a necessary yet distinct attribute personalization.
come to rather paradoxical possibility where model can have high degree of
(odee: https://githhb.ce http://hhhhh.oo//DMMMMAMA Htw--LLAAA///PesallMMMMIMI
2perSEval should not be understoad an alternative "improued" measure, and therefore, is not comparable EGISES.
3

---

2.2 Personalization vs. Responsiveness
In this section, we first distinguish responsitveesss from personalization. Informally, responsiveness is the capacity of a model to discern the differences in the profiees (i.e., reading histories) of two readers quantitatively and accurately predict the dissimilarity in their corresponding expected summarics that is proportionate to their profile difference. However, there can be scenarios where a model exhibits high responsiveness at the cost of losing accuracy. To illustrate this, we continue with the example from the previous section. Suppose we observe an arbitrary model to generate two different summaries for given news article, one focusing on Israeli Prime Minister' and the other "Jewnish protests on war', skipping the article's content civilian distress and war-front battle information. In that case, we have to conclude that the model apparently discerned the difference between Alice and Bob's profiles, thereby predicting the proportionate dissimilarity in the expected summaries but not the expected summaries themselves. Thus, the model is inaccurate and yet responsive. Therefore, interpreting such responsiveness as personalization leads us to the personalizttionaaccrayy paradox. We prove this formally Section 3
We establish that EGISES measures how sensitive (or insensitive) model is to the differences in the readers' subjective expectations (i.e., responsiveneas) but not personalization. Therefore, EGISES can give fairly good score to the model in the example. To elucidate this, we first define an Oracle personalized summarization model as follows:
Definition 1. Personalized Summarization Oracle. A summarization model Mo. (parameterized with 0) is an Oracle if for specific j-th reader profile hy (i.e., reading history) il generales an optimal summary ky) of the document d, (i.e., Ma, : [di, h2) + k where ((d,hh)) Sung = Uayi uey is the j-th reader's 's subjective expected summary of d, and is determined by h2.
We now recall the notion of insemniliiiityy-o-xwjjeciiii
the foundation of EGISES, as in Vansh et al. 2023
A summarization model Mo,n is (weakly) Insensitive- to-Subjectivity w.r.t a given document da and corresponding readers j and k, if V(uas, usk), (a ((k)),)k)) S Thaw) > (a(gu Swa) > I Imaz), where a is an arbilrary distance metric defoned on the metric space M, where d, u and are defined maz is the maximum limit for ue, uj be mutually indistinguishable, and I is the marimum limit for Sus Suz to be mutually indistinguishable.
Definition 2. Weak -ornnstttviityto---beeeetttt
Definition 3. Strong tornssttieeto-Sujecttiiei w.r.t given document d; and corresponding readers and k. if ((uj,, Usk); Ma. salisfies: (i) the condilion of weak insensilivily, and (ii (a(ui,, (o(u) - > (a(Buug Sua) S Tmaz))
A summarization model Mo.k is (strongly)
Based on this notion, Vansh et al. 2023) defined (summary-level) 'deviation' of a model M We generalize this to our notion of summary-leve. Degree-of- Resposssisisssssssss (DEGRESS), the measure for responsiveness, as follows:
Definition 4. Summary-level DEGRESS. Given a document d, and j-th reader's 's expected summary Uij. the summary-level responsiveness of personalized model Ma,k, (denoled by DEGRESS( (dd, usg))), is defined as the propprriionll divergence" between gener-eenee summary Su;; of d, for j-th user from all other user-specific summary versions w.r.t a corresponding divergence of usj from all other user-profiles.
DEGRESS( (dd, uij)) is formulated follows:
Ua,1
min(X Xijk, YiSk) + max max((xijk Yask) te
DEGRESS( (m.,, (di, uiz)) JUa
exp(w(uas) waak))
exp(w((ug) (uua)) (Suuy Saw)
Xisk U Σ exp(w(us5lui)
(uig, usa);
Yisk
(1)
exp(W(Sau (ua))
o awij, Wak) (uij. de)
o (Ban Sum) (Suug y di)
w (Wig (Max)
w( (0(SuylSuse
30(ui, us) a(u, uz) E J0, 1]: satisfies positivity, reflexive, maximality, symmetry, and the triangle inequality.
4

---

8 d
82
$1
02 5
$1
U1
U2
U1
U2
$2
$2
(a)
(b)
Figure 2: Existence of EGISES Personalizaioo-AAccureey Paradox: High (and same; (a)) DEGRESS, yet low accuracy (red line; (b)).
Here, D is the total number of documents in the evaluation dataset, [ is the total mumber of users who created gold-reference summaries that reflect their expected summaries (and thereby, their subjective preferences), and |Ua, (= [S is the number of users who created gold-references for document d, w is the divergence of the model-generater summary Su., (and the corresponding expected summary wsy) from document d; itself in comparison to all the other versions. It helps to determine how much percentage (therefore, the softmax function) of the divergence (i.e., (Buug Su,z) should be considered for the calculation of DEGRESS. Tf Sug is farther than San, w.r.t de then DEGRESS( (Sa,, [d,, uzy)) )) DEGRESS( (8 [d; uax)), implying that Ma. is more responsive to the k-th reader. A lower value of DEGRESS($ (d; ug)) indicates that while reader-profiles are different, the generated summary Suy, is very similar to other reader-see reader-specific summaries (or vice versa), and hence, is not. responsive the summary-level. The system-level DEGRESS and EGISES have been formulated follows:
DEGRESS(#Nng ((dr,
D
10a,1
DEGRESS(Mg,N)
(GI.ES(MM..)) DEGRESS
(2)
[D]
EGISES mcasures the degree of lack personalization (i.e., a lower score is better within the range [0, 1]) instead of assigning an absolute goodness score. As can be noted, the EGISES formalism does not enforce any penalty drop. Here, accuracy would be inverse function of a8 Mij for the same metric distance that DEGRESS uses. Hence, EGISES (and DEGRESS) should be interpreted as measure of responsiveness (i.e., proportionate divergence) and not personalization.
for relative benchmarking of how much models
accuracy
3 EGISES Personalization-Accuaa Paradox: Formal Proof
In this section, we mathematically prove the existence of the condition that, for sufficiently high DEGRESS (and thereby EGISFS), there exists low accuracy.
Theorem 1.
The accuracy (a u)) of model Ma, on the metric space M with distance function can vory keeping DEGRESS s., (d. constant.
Proof. We follow the same triangulation proof technique as in (annh al. 2023 Let d, and be triangulated as per Figure Keeping d and u fixed, we can perform an arbitrary rotation operation with d as center (rot(e,d) d) 8: angle of rotation) on Sug and Sun. s.t. rol( ee, d) is a closure operator in M. Now, 3(p q) M, s.t.
max (rot(s(s di, 8p), Wiy) min (rot usy 5p): uiz): And similarly, max (rot(Sua,) d, 6g); use) > min a (rot(sua Usk. 8g), usk)

---

If p==g q, then DEGRESS($u.2 (d,, uz+)) (and therefore, EGISES) remains unchanged for given d,. However, due to the existence of a total ordering of accuracy (o,, u)), for any arbitrary the accuracy, therefore, can be varied by changing from minima to a maxima (f (a(s, u)) E [0,
The proof establishes the theoretical existence of the personalization-ccerrzs paradox if we interpret EGISES as a mcasure of personalization instead of responsiveness. It sets the motivation to design a corrective measure for EGISES that truly measures personalization. As discussed in the introduction, Vansh et al. (2023) proposed a follow-up measure called P-Accuracy (P-Acc), that might seem solve this problem. However, some major limitations of P-Ace prevent it from being used for evaluating degree--o We elucidate that in the following section.
4 Limitations of P-Accuracy (P-Acc) as a Solution to the EGISES Paradox
In this section, prove that P-Ace, proposed in Vansh et. al. (2023) cannot be used solution to the EGISES paradox. We begin with recap of the formulation of P-Ace as follows:
P-Ace(Mo,n) Score (Mo Ani where: Unitp-Ace [a [Seig( (GISES(Me.N) }]; Score Acc. score w.r.t chosen accuracy mcasure Ace
Here, E (0, 1] calibrates the importance be given to lack of personalization in model while [0, is the control that helps the human-evaluator regulate the final penalty to accuracy due lack of personalization. According to P-Ace, it apparently seems that there cannot be situation where if both accuracy and EGISES scores are low (i.e., DEGRESS is high), the final score will be high, thereby preventing the EGISES Paradox. However, three limiting properties of P-Ace render it unusable for personalization evaluation. We prove their existence as follows:
Theorem 2 (Limitation 1: Lack of Interpretability of Score). There enists infinite possible model collections θ = 1Or O = (Me... where v(k, () pair: P-Aco(Me..c = P-Aco(MM...) when (i) Score (Mo,.. > Score (M) (ti) ScoreAcc(Mo..... < Score (Mo...)) and (itt) ScoreAce(Mo.e.. = ScoreAce (Mo..).
Proof.
P-Aco(Ma..) = Scorescc(Me..) a faig EGISE)(Mo.))), (By definition
V(k, I) pair: P-Aco(Mo,a.h) P-Aco( (Ma
Aco - a (3 (3 EGISES(Me....)) = Score (May.h) - a saig (8 (MI))
Acere AMe (M) Aue (M) a (fogg EGISES(Me;...)) fang EGISES(Me.,..)) (AN
Case 1/2: us assume that a sufficiently large exists s.l. Scoreaed (Mo,e.t) Score Co (Ma
:. fany (B- EGISES( (3 (MIS))( = k 6-1 Scoreece (ooresce(Me.) (Ma..t)) a
(B EGISES(Me)L.)) - faig( EGISES(Me).N)) 22)2224 k-1 (: EMI EGISES 2 k EGISES(Me;L.A) K EGISES(Me,L.)) which a possible condition. Hence, the assumption is valid.
Case us assumne Coore Score (Ma
{k=1) 11
EGISES(Me,,A, EGI which is possible condition. Hence, the assumption is valid.
The proof entails that P-Ace scores cannot discriminate the three cases.

---

Theorem 3 (Limitation 2: Lack of Interpretability of Lower Bound). There crists condition when P-Ace(Mo,.. () < 0 even when ScoreAce (N....)))) > 0.
Proof.
A-.o Score (e (Mo.n) a- fsig(B EGISES(Mo.A)), Let Scoresce (M...) = for sufficiently large h
(By definition]
Let assume that P-Acc((Monn)
Scoresec (Mo.s) - (ax foig (13 EGISES(Mo.n)) <0 ScoreAce (Mo.a)
(B (..)EE((S.))) 6+
where 8+ is posilive quantity
Score Ace (Ad
(6+ +
EGISES(MO..)
3 In
8+ Score (Ma.h)
:. 3ex B, 8.. Scure (Ma,) 2 0
The proof implies that P-Ace can take a negative value even for a reasonably high accuracy model. As an example, for a = 0.5, 8 = (i.e., full importance to lack of EGIEEE 2 0.9 if Score < 0.35 then P-Ace O.
Theorem 4 (Limitation 3: Anomalous Behavior). There erists anomalous condition when P- P- P-Aco[Mez..) even when (GI.ES(MMe..A) < (M))))) (NT: lower is better).
Proof.
(kk, pair: (c M < 0 {From theorem 3
:. Score (M) Sccrrecc(MMeei)) < a (Bosg(B EGISES(Me - Saio EGISES(M
Score (Mo Score Aco (M
(B (8 EGISES(Mou..) fang (B EGISES(MO..A)
EGISES(Mo,,.a) EGISES(MO...)) > Score (Mo Score (Mo...))
1 A(Scorr (Mo b), (Mo which a possible condition.
A AEEGISES(Mo,, ,)), EGISES(MO...))
a
A bounding case of the proof condition when for Score (Mo...)))) = Score (c -1 (worst case), and CGISES(Mo...A) -0.37 (better ease). Hence, P-Ace is not suitable for models that have relatively low accuracy scores. The fact that P-Acc is useful to regulate accuracy (and not measure personalization) is evident from Table g in Appendix C.2 where we observe that, unlike PerSEval-scores outlined in section 8.1 P-Acc scores are heavily dominated by the evaluated PENS models' accuracy scores and therefore, do not diffor significantly even though they lack responsiveness (i.c., low EGISES scores; see Table 7 Appendix C.2 As remedy, we propose Peral (Personalized Summarizer Evaluator) in the next section.
(Mo...)
0, EGISES(Me,,..))
(best case) = P-Acc(Meus.)
(worst case)
P-Acc(Mon.h)
5 PerSEval: Measure for Personalization
The design objective of PerSEval is two-fold: (i) to ensure that model is penalized for poor accuracy performance, but at the same time, (ii) to ensure that the evaluation of responsiveness (i.c., DEGRESS) is not obfuscated by high accuracy (since high accuracy does not entail high responsiveness as proved in Vansh et al. (.23)). In other words, the underlying maxim behind the design of PerSEval is accuracy not a reward, but lack of surely a penalty! We term this penalty as Effective DEGRESS Penalty Factor (EDP). As per the design maxim, if accuracy 100%. then there will be no EDP applied, and the PerSEval score will be the the DEGRESS score. In the subsequent sections, we develop the motivation and formulation of EDP.
7

---

5.1 Accuracy-drop Penalty (ADP)
In this section, introduce the first component of EDP- Accuracy-drop Penalty (ADP). ADP is document-level penalty due to drop in accuracy for the best-case scenario where moode--genraaeed summary of document de (su.,) is closest to the corresponding reader's expected summary Uij. In this case, we denoted Suy as Su.. We define ADP as follows:
Definition 5. Accuracy-drop Penalty. Given document d, and user-generated summaries Ud the document-level ADP of a model Mo.n, denoted by ADP(su,.ldd, u;=)), is the relative deviation of the best performance (a" (oaa::M;e)dd of Me.. Vo(Bua Uia)]ds from the best possible performance (i.e., 0) w.rrt. ils procimity to the worst possible performance (i.e., 1).
Document-level ADP is formulated as follows:
ADP(s ((1 ),+)) + 10 exp (-10 (1-0' (Buie wis Ildi)+e)
o* (Suce wie .dd.
(4)
Ua,
where, o² (Su..) Uio (d
min o( ( uij)lds; {er An infinitesimally small number E (0, j=1
Here, ADP is defined as a shifted sigmoid function where 2 4 helps to bring the minimum penalty to zero, while the factor of 10 in the exponentiation ensures that the maximum penalty reaches when the ratio of the difference the best case to the worst case is around 1.5 before it starts exploding (i.c., over-por pra--enalization) ADP ensures that even if the DEGRESS score acceptable, a penalty due to accuracy drop can still be imposed as a part of EDP. ADP, however, fails to address the scenario where the best-case scenario is acceptable (i.e., accuracy is fairly high) but rather an outlier case i.e., for most of the other -enee--geerated summary versions, there is a considerable accuracy drop. To address this issue, we introduce second penalty component within EDP called A courrcy-inccomisseens Penally (ACP).
5.2
iccuracy-nnconsistency Penalty (ACP)
ACP accounts for outlier conditions of the best performance, as explained previously. It evaluates how a model performs w.r.t accuracy for a specific generated summary compared to its average performance. More formally, it defined as follows:
Definition 6. (ecuracy-inconsisteneyy Penalty. Given document d, and user-generated summaries Ud.- the summary-level ACP of a model Ma.s, denoted by ACP( (di, Wij)), is defined as the relative deviation of the summary performance (8s,, 1 uzz) ld; of Mo. from its best performance o (s u dj as compared to the deviation of average performance 7 (Su.. Ud,]dd, from a* (su,., u;+)
It is to be noted that, unlike ADP, ACP is a summary-level measure. This penalty evaluates if the model consistently performs w.rt accuracy and therefore, conversely, does not inject any additional penalty to DEGRESS when model is consistent. The summary-level ACP formulated as:
ACP(s (dd, uiz)) 1 + 10:24 exp -10
o (eugy May lids -o (Buce (he (alsuc Min) di of (Suce Mes d.).
(5)
U
1
where, ; [ds
o (Bugy Wiz ldj
U
5.3 PerSEval: Formulation
We now lay the design of the PerSEval framework as an extension DEGRESS (i.c., 1-EGISES). A multiplicative injection of EDP E (0, 1] should be such that the best accuracy (i.e., ADP = 0) with no inconsistency (i.c., ACP 0) would lead to EDP value of 1, and thereby, DEGRESS remains unobfuscated (which is the desired
8

---

objective). The following formulation of PerSEval guarantees these properties:
PerSEval [di: unz)) DEGRESS 8u.s Unz)) x EDP (su. (di,, Us;)
where, EDP( Su., [dd,, uiz)) = 1 1+ 10023 exp (110 DGP Suy (d4, Uiz)))
(6)
and, DGP (de, Uiz)) ADP( (d;e u;-)) + (s.,, (dis u1y))
The system-level PerSEval score as follows:
PerSEval (sury (ddry((d (di,))s))
14,1
(M)
(7)
D
We design EDP as an inverse sigmoid function of the overall DEGRESS Penally (DGP), which sums up ADP and ACP, a and 3 are ppram-eareeeters that help to control the shape of EDP. a 2 3 ensures that EDP is (i.e., 1 1 0) when there is no penalty, thereby making PerSEval equivalent DEGRESS. 32 ensures that the function does not drop sharply, thereby over-penalizing (and hence, dampening) an otherwise fairly good DEGRESS score (i.c., responsiveness). Since 3 may significantly affect the overall human-judgment correlation, we did an ablation study (Fig 3 Section 8.1 to find the optimal value (which was observed to be 1.7). The system-level PerSEval E [0, 1] and is bounded by the system-level DEGRESS score.
6 Benchmarking of SOTA Summarization Models w.r.t PerSEval
6.1 Model Benchmarking Dataset
Microsoft PENS Dataset (News Domain). Our study, as in (Vansh al. 2023 assesses models using test data from the PENS dataset provided by Microsoft Research (Ao et al. 2021 This dataset pairs news headlines with articles, serving as concise summaries. The test set creation involved two phases: initially, 103 English speakers selected 50 articles of their interest from a pool of 1000, sorted based on exposure time. In the second phase, participants generated preferred headlines (gold references) for 200 articles without knowledge of the originals. The assignment ensured an average of four gold-reference summaries per article. The PENS dataset was chosen because it is the only one that contains the users' reading history, i.c., the temporal sequence of interactions (clicking, reading, and user-generated gold summariess), making it ideal for evaluating five SOTA personalized summarization models that require user reading history input.
OpenAI CNN/Daln Mail Dataset (News Domain). To understand the applicability of PerSEval on mainstreeam gold-standard news datasets, we design an indirect evaluation methodology with the OpenAl CNN/DM dataset (yalidation and test) released by Stiennon et al. (2020). The test dataset contains 639 articles processed by 19 policies, resulting in 5515 summaries rated by the same annotators on accuracy, coherence, coverage, and overall quality using 7-point Likert scale. However, unlike the PENS dataset, this dataset lacks the temporal span of human evaluator interactions forming the reading history required by the PENS summarization models. Additionally, do not have gold-reference summaries (but evaluated the personalization of 4 (out of 19) OpenAT
only ratings) 3 Hence, to conduct our experiments, policies, focusing on RLHF-tunnd policies, as base-model and SFT-based policies are non-personalized. Since the policy-gencrated summarics are not specific to any human annotator as the policies are not trained on any reading history, we do not observe multiple user-specific sunmaries from the same policy necessary condition for evaluating personalization. To address this, we concatenate all summaries generated by different policies (say, such summaries) that were rated above 6 (out of I'max = 7) by a specific annotator (say, Alice) into one combined rele-reference summary per Alice (MAllce)). After this
We comply with the Microsoft Research License Terms.
It. be noted that human-written gold reference summaries were not always selected rated highest by the human annotators, clearly indicating the subjectivity of saliency.

---

step, the content of each of all the policy-generated summarics (s) that received some rating ([1, (maz)) from Alice (s) = Alscc))))) is then concatenated (k + (Tmex (llccl)) ))-times - to create a poliey-generated surrogate personalized summary (SAlice) for Alice. Ideally, this surrogate should match Alice's combined gold-reference "Alice- If Allee is much lower than Tmar, then the surrogate should get penalized. The injection of redundant, unmatchable content. helps to achieve this. We repeat this process for all annotators and generate the PerSEval scores of each policy across 27 (out of 639) articles. Since i and r(s..) can differ between annotators, the penalty (and thereby length) of the surrogates will subjectively differ as well.
OpenAl TL;DR (Reddit) Dataset (Open Domain). To understand the broader applicability of PerSEval, we also appropriated the OpenAT TL;DR TI, dataset Strennonn et al. (200)) This dataset a collection of 123,169 Reddit posts adopted from the dataset by Völske et al. (2017) covering 29 subreddits (i.e., domains) of varied lypes, their corresponding OpenAl poliey-generated summaries, and the human-written gold-reference summaries. A subset of the validation dataset comprises 1038 posts that were fed into 13 policies to generate 7713 summaries. Like the OpenAl CNN/DM dataset, 32 human annotators rated the summaries in the same format as in the previous OpenAl CNN /DM dataset. As a part of the appropriation, we carry the same method as that with CNN/DM leading to the evaluation of 4 RI.FF-tuned policies (out of 13) and 57 (out of 1038) annotated articles. However, we would like to emphasize that this indirect benchmarking is neither an equivalent evaluation methodology to the PENS dataset based one, nor is simulation of the required setup. There is so far no other datasel comparable to PENS for evaluating personalization, and hence, all benchmarking results in the outlined method should be seen baseline within the scope of the method.
6.2 SOTA Summarization Models Evaluated
Personalized Summarization Models (Microsoft PENS dataset). We study ten SOTA summarization models for comparative benchmarking as in (Vansh et al. 2023 Five of them are specifically trained personalized models and follow the PENS framework (Ao et al. 2021): PENS-NRMS Injection-Type- (T1)/Type-2 (T2), PENS-NAML TI, PENS-EBNR Tl and
T2. The others are generic
SOTA models BRIO (Liu et al. 2022), SimCLS (Liu & Liu 2021), BigBird-Pegasus (Zaheer et al. 2020 ProphetNet (Qi et al. 2020), and ba-b [Orzhenvskii] 2021). The selections are based on their consistent.
top-5 ranking over the preceding four years on the MNN/Daily Mail news datasct Hermann et al. Appendix A contains model descriptions.
2015
Non-personalized Summarization Models (Microsoft PENS dataset). For the non non-personalized models, we follow the evaluation setup used by Vanns al. (2023) by augmenting the documents with the reference summaries of each reader as document. titles (i.e., cues). This results in subjective document. versions corresponding to each reader. The models ideally should pick up the cues and generate them back as an output, thereby inducing an "apparent" sense of personalization. This injection process provides robust baselines for comparative evaluation.
RLHF-tuned OpenAl Summarization Policies (OpenAI CNN/DM & TL;DR (Reddit) datasets). We analyze five OpenAl policies (parameter size 1.3B and 6.7B) that have been RLHF-tuned using TL,DR (Reddit) training datasets. The base pre-trained model architecture similar to that of GPT-3. The base model is then further fine-tuned in a supervised setup on filtered set of the Reddit posts to generate SFT-models (denoted 'sup"). The Reward Model (denoted 'rm") is a linear head that predicts the rating. The Reinforcement Learning of the policy model is done via the PPO algorithm Zheng et. al. (023bb)
6.3 Baseline Distance Metrics and Scores
PerSEval is a generic measurement framework (like EGISES) where the specific metric space M on which o is defined should be appropriately selected such that we achieve the best human-judgment (HJ) correlation. In this paper, we choose seven summarization accuracy metries that are defined on standard algebraic spaces and plug them in PerSEval in isolation as distance metrics (i.e., a) (i) ROUGE (RG)-L, (ii) ROUGE (RG)-SU4,
10

---

Basline-l: Microsoft PENS Test Dataset with Cue Injection
Models
PSE-RG-L
PSE-RG-SU4
PSEMEETER
BLE-
PSE-JSD
PSE-BScore
ISE-InfoL-Mon3
BigBBrddPeggauu
0.205
0.143
0.168
0.156
0.253
0.320
0.195
SimCLS
0.079
0.032
0.016
0.014
0.137
0.286
0.18
ProphetNet
0.054
0.027
0.018
0.016
0.087
0.228
0.135
T5 (Base)
0.035
0.022
0.011
0.016
0.073
0.207
0.124
BRIO
0,030
0.021
0.008
0.013
0.107
0:242
0.154
Personalization Benchmarking: Microsoft PENS Test Dataset with Human Reading History
PENS-NAML Tl
0,018
0.013
0.019
0.010
0.025
0.071
0.051
PENS-NRMS T1
0.016
0.011
0.016
0.008
0.022
0.062
0.045
PENS-EBNR T1
0.009
0.008
0.010
0.005
0.015
0.037
0.029
PENS- T2
0.002
0.005
0.003
0.002
0.006
0.008
0.013
PENS-NRMS
0.002
0.004
0.003
0.002
0,006
0.007
0.013
Basline-II: OpenAI CNN/DM Test and TL,DR (Reddit) Validation Datasets with Human-Feedbaee Feedback (i.e.. Rating)
Models (L.., Policies)
PSE-RG-L
PSE-RG-SU4
PSE-METEOR
PSE-BLEU
PSE-JSD
PSE- BScore
PSE-InfoLM-ag
sup1/rmm (6.7B)
0.943/0.719
0.9-1//0.730
0.906/0.525
00.87/0.395
0.943/0.773
0.998/0.719
0.999/0.889
sup4/rm4-t.7 (6.7B3
NA/0.711
NA/0.648
NA/0.599
NA/0.4665
NA NA/0.737
NA/0.790
NA/0.818
sup4/rm4 (1.3B)
523
502
0.413/0.357
0.612/0. 163
0.617/0.566
0.994/0.796
0.999/0.603 /0.605
sup4/rm4-kl14 (6.7B)
0.783/NA
0.783/NA
0.562/NA
0.502/NA
0.763/NA
0.584/NA
0.753/NA
sup4/rm4-t.7 (1.3B)
0.505/0.481
0.5/0.414
0.267/0.417
0.286/0.323
0.502/0.523
0.92/0.776
0.646/0.602
Table 1: SOTA model-benchharking on Mierosoft PENS and OpenAl CNN/DM & TL;DR (Reddit) datasets w.r.t PerSEval (PSE) (NA: not applied on the dataset); PerSEval hyper-parametens:
3, 3 = 1.7
(optimal 3; see 3 and = 4. Observation 1: Disagreement between keyword-based (RG-L, RG-SU4, ) and non-keyword-har based PSE variants (JSD, BScore, InfoLM) in terms of non-personalizzd
METEOR, BLEU) baseline leaderboard from rank-3 owward; Observation 2: InfoLM-ax has more reliable discriminatory performance with less sharp changes for PENS dataset; Observation 3: Specialized personalized models lack personalization w.r.l all variants of PerSEval; Observation 4: RLHFF-buned LLM-bused policies exhibit much higher personalization baseline (t.e, baseline-11) compared non-pen con-personalized specialized summorization models (i.e., baseline-l) within the limited scope of the surrogate-based indirect evaluation; Obscrvation 5: PSE-scoves are very high for lop-performing OpenAl policy due to most surrogale summaries being highly rited and very similar to the combined gold-references.
(iii) BLEU-L, (iv) METEOR. 0 (v) BertScore (BScore) defined on embedding space, (vi) Jenson-Shannos
Distance (Menéndez et al. 1997 on probability space, and (vii) InfoLM-a3 (a = 1; B = 1) on probability space generated from the embedding space of a masked-LM. RG is chosen because it has a very high HJ (Pearson & Kendall) correlation (> 0.7) in most standard datasets such as CNN ANN/DM and TAC-2008 (for RG-L) as reported in (Bhandari al. 2020 Zhang al. 2024), and DUC-2001/2000 /2002 (for RG-L/SU-4) (Lin) 2004]. For the same reason, the aB variant of InfoLM is chosen. Comprehensive benchmark results w.r.t optimal PerSEval hyperparameters (i.e., 3, B 1.7, 7 = 4; see Figure 3 for ablation results) for each of the variants are given in Table] | We observe that most non-personalized models, such as BiggBrd-Peggauu,, produce significantly stronger baselines across most PerSEval variants. However, keyword- based PSE variants (RG-L, RG-SU4, METEOR, BLEU) disagree with non-keyword-baase based PSE variants (JSD, BScore, InfoLM) in terms of the non-personalized bascline leaderboard from rank-3 onward. One reason for this that JSD, BScore, and InfoLM being embedding- maasedding-based can handle out-of--iittibbution (OOD) situations leading to lesser penalty for keyword-level mismatch that happens in abstraction summarization. We also find that the InfoLM-a,3 variant shows 'smoother discrimination' (i.c., no sharp jump in the performance) when compared to RG-SU4, BLEU, and JSD. At the same time, the discriminatory performance of InfoLM-a.3 and JSD variants is much better than BScore, METEOR, and RG variants, leading to a more reliable leaderboard. The results on the OpenAl datasets should be seen within the context that: (a) the models are essentially policy variants that differs w.r.t size and specific PPO implementation, and (b) the indirect benchmarking method is limited (as outlined in section 6.2
'First four defined as similarity functions the string space while BScore is defined as similarity function (as opposed to distance function) on high dimensional embedding space; see Appendix R.1 for details on each of the seven measures.
11

---

7 Meta-evaluation of PerSEval: Experiment Design
In this section, we lay down the experiment design that forms the foundation to establish - (i) the reliability of PerSEval w.r.t human-judgment correlation, (ii) the stability of PerSEval, (iii) PerSEval as a rank-measure is not entailed by DEGRESS-rank (i.e., the EGISES paradox is empirically existent), and (iv) PerSEval can be a standalone rank-mcasure without the nced of any aggregated ranking.
7.1 Meta-evaluation of PerSEval Reliability: Survey-based Direct Method
Mota-evaluation Objective. As pointed by Vansh ct al. (2023), unlike the meta-ovaluation of accuracy measures, direct mela-evaluation of personalization is not a feasible task since that would require human evaluators to work as a team and to understand how their subjective assessment of the PerSEval scores (i.e., to what extent they agrec with the scores) corresponds with the differences in the model-generated summaries and their own subjective expected summarics. As an alternative, propose survey-based evaluation methodology to simulate this scenario. We argue that the mela-evaluation of PerSEval should have two objectives: (i) whether human evaluators would judge the responsiveness of models in the same "ratiowayy as what DEGRESS docs, and (ii) whether human evaluators would apply the accuracy drop to the responsiveness in the same 'Juclor-way' as what PerSEval does. In other words, the central objective is to validate to what extent. human evaluators agree with the design principles of PerSEval at a cognitive level. Participants. We opened the survey to selected pool of graduate students demonstrating fair English comprehension. The pool consisted of students from five different backgrounds: (i) computer science, (ii) electrical engineering, (iii) humanities & social sciences, (iv) mathematics, and (v) physics. The survey was promoted in ongoing courses, student associations, and social media groups. 169 students (~45% male, ~55 (emale) within the age group of 25-40 completed the survey. No other personal details were asked.
Survey Procedure. Each participant was shown a pair of gold-reference summaries (corresponding to two user profiles) for a spocific news article from the PENS datasct. Along with this, five pairs of model-genorated summaries were shown, each pair corresponding to five of the ten models studied in this paper (i.e., two participant responses covered all the ten models for a given document). To eliminate response bias, the participants were unaware of which of the six pairs was a gold-reference, and the model names were also not revealed. The same set (shuffeed) was shown to two random participants to get an average. Each participant was asked to provide similarity ratings between (low) and 6 (very high) for the summary pairs. A sample snapshot questionnaire is provided in Appendix D (figure
Modeling Human-judgment of Personalization (PerSEval-HJ). We model a 'human version' of DEGRESS (termed DEGRESS-HJ) using the normalized rating as the divergences (i.e., (luj, uss) and (Sugg Sug) for document d; and gold-references Uij and u.x). As mentioned above, if PerSEval needs to be reliable, then the necessary condition is that DEGRESS should have a high correlation with DEGRESS-HJJ, failing which it can be concluded that human evaluators do not interpret divergences in the ratio-way of DEGRESS. We use the standard Pearson's ' coefficient (r), Spearman's p. and Kendall's т rank coefficients for this. As a sufficient part of the reliability test, we model the "human version' of EDP using standard accuracy measures (i.e., RG-L, RG-SU4, METEOR, BLEU, and InfoLM-a3) as surrogates. The motivation behind this is that such measures are known to have high HJ-correlation. The objective was to check whether such surrogate, if used factor (just as in PerSEval) with PerSEval-HJ, shows high correlation with PerSEval, failing which implies that human evaluators do not cognilively resonate with this factor-styled discounting of DEGRESS
7.2 Meta-evaluation w.r.t Stability
PerSEval, being rank measure, is said to be stable if, for any random sample of document (and corresponding gold-references) selected from the evaluation datasct, the rank of the evaluated models does not change. This meta-measure is important because it objectively checks if PerSEval can be relied on any arbitrary personalization evaluation dataset, unlike the PerSEval-HJ based reliability evaluation, which is subjective and indirect in nature. In order to establish stability, we need to define w.r.t a specific sampling method.
Sampling Method for Stability Meta-evaluation. To understand the stability of PerSEval, we create random sample collections Cº s,s. where k- = (80%, 60%, 40%, 20%} of the PENS dataset and N [(,, U)).
12

---

Correlation Coefficient
0.7-
0.6
Kendall
Spearman
0.4
1.0
1.1
8 Hyper-parameter (PeoSMval-infoLMM-a33 2.0
14
1.7
Figure 3: PSE-ILM Ablation: Effect of 3 on HJ-Corr; Optimal performance 3 1.7 across all three standard correlation measures (Pearson r, Spearman p. Kendall r).
Each collection has ten random sample sets n C i= 10] (with replacement). We benchmark the models all 40 sample sets to obtain corresponding leaderboards. We compare that with the rank obtained from the entire dataset. We formalize this notion of stability of rank measure as follows:
Definition 7. Weakly Stable Rank Measure. A rank measure is e-weakly stoble if the marimun rank- correlation (w.r.t stat. T) between the measure-generated model-ranking on each n. and model-ranking on the entire dataset N is e.
Definition 8. Strongly Stable Rank Measure. A rank measure is 6-strongly stable if (i) it is e-weakly stable, (ii) the bias over C(D.S,U) w.r.t the mean score of each C.D.s.), for each evaluated model I 6,, and (tit) the variance over C(D,S,U) w.r.t the expected voriance of the scores of each Ct C.D.S,U) for each model is < Svar; max( 8var).
8 Observations and Insights
In this section, we provide empirical support for the reliability and stability of PerSEval and show that accuracy leaderboards may be mislcading (or, at best, redundant) for personalization analysis.
8.1 Meta-evaluation of PerSEval: Results
Reliability of PerSEval. We compute the HJ-correlation of the seven variants of PerSEval w.r.t to each of the five PerSEval-HJ variants (RG-L, RG-SU4, METEOR, BLEU, and InfoLM-a3; see Table for the results An 11-point hyper-parameter ablation study shows that the optimal correlation is at 3 = 1.7 (Figure has the overall best correlation across all five PerSEval-HI
3 We observe that VerSEval variants. We further observed that: (a) PerSEval--HJlutoLM--o
human-judgment estimate has the best performance of each PerSEval-variants (Pearson's is 0.79; Spearman's p 0.68; Kendall's 0.47 w.r.t PSE-HJJifoLM-oo0)) and (b) PerSEval--lnoLLMaus performs the best across. The high performance of SorrEval@=1.7__nmm 2nn can attributed to InfoLM-a33, as an accuracy (and thereby, distance measure) having a very high system-level HJ-correlation (Pearson's 's 7 zs 0.95: Kendall's 7 zs 0.85) when compared to the other measures on the CNN/DM dataset, as reported in Colombo al. (022bb).
Indirect Evaluation Method is Inadequate for Meta-evaluation. As discussed in section 6.2 the method of evaluating personalization of RLHF-tuned policies is indirect and limited in scope, since these policies are trained annotator profile agnostic ratings and not their reading history. Hence, although the OpenAT dataset may be used for generating baselines in this limited setup, they are not suitable for meta-evaluation of any personalization measure. To validate this point, we design an OpenAl dataset- oriented PerSEval-HJ variant where we model the divergence between pelicy-generated surrogate summarics
PPerSEval-HJ. Human judgment est.; Stat. Significance of Corr. (Strong, Moderate, Low, None): p-value < 0.01.
13

---

Microsoft PENS dltasett/Opent CNN-DM dataset Mntaset/Opent /OpenAT TEDRR (Reddic) dataset
PerSEval-II]*GEe
H.I-Com.
PSE-RG-L
PSP-RG-8L.4
PSE-EETER
PSE-BLEU
PSE-JSD
PSE--BSeore
P8E.Info..Moo
Pearsou's
0.23/ 0.90/-0.89
0.89/-088
0.49/ 0.89/
0.43;
0.67/-0.63
-0.05/
0.90/-0.92
-0.41/0.n2/0..7
0.79/ 0.21/011/-00
Spearman's p
0.80/-008
-0.27/
0.80/-0.80
0.15/ 0.81/-0.80
-0.28/ 0.40/-0.80
-0.26/
1.000-1.0
-0.28/0:20/00
0.69/0.40/-0 0.40/-0.80
Kendall's
-0.20/ 0.67/-77--0
-0.24/ 0.67/-0..7
0.16/ 0.67/-0.67
-0.24/ 0.33//0.67
-0.20/ 0.67/-1.00
-0.24/0/0
0.37/0.33/-0.6
PerSEval-
HJ-Corr.
PSE-RG-L
PSE-RG-SU4
PSE-METEOR
PSE-BLEU
PSE.JSD
PSE- BScore
PSE-InfoLM-o7
Prarson's is
0.91/ 0.90//0.88
0.96/ 0:80/-0.81
0.98/ 0.67/-0.82
0.98:
0.80/-0.62
0.77/ 0.89/-0.92
0.40/0.03/0.07
0.23/0.21/00.70
Spearmen's
0.07/ 0.80//0.80
0.05/ 0.80/-0.50
-0.10/ 0.80/-0.50
-0.02/ 0.40/-0.80
0.16/ 0.50/-1.00
0.15/0.20/0
0.77/ 0.77/0400-0..8
Keudall's
0.13/0.67/-00
0.07/ 0.67/-0.67
-0.07/ 0.33/-0.67
-0.02/ 0.67/-40.6
0.20/ 0.67/-1.00
0.16/0/0
0.67 0.33--0677
PerSEval-H.J11TTER
H.--Cor.
PSE-RG-L
PSE.RG-SU4
PEE-EETEOR
PSE-BLEU
PSE.JSD
PSE-BScore
PSE-Intol.M-o7
Pearson's '
0.06/ 0.900/00.00
0.18/
0.90/-0.51
0.36/ 0.89/-0.86
0.29/
0.67/-0.07
-0.23/
0.90/-0.92
-0.56/0.02/00
0.77/ 0.21//00/
Spewmman's
-0.28/ 0.888/0.80--0.00
-0.33/
0.80/-0.80
0.12/ 0.80/-0.80
-0.27/ 0.40/-0.80
-0.32/ 0.80/-1.00
-0.37/0020000
0.71/ 0..1/00000-0.80
Keudall's
-0.29/ 0.67/-0.67
-0.33/ 0.67/-0.67
0.07/ 0.67/-0.67
-0.24/ 0.33/-0.67
-0.38/
0.67/-1100
-0.42/0/0
0.51/ 0.33//0.67
Per ULJ
HJ--Corr
PSE-RG-L
PSE-RG-SU4
PSE-METEOR
PSE-BLEU
PSE-JSD
PSE- BScore
PS
Pearson's r
0.84/ 0.90/-0.88
0.91/ 0.89/-0.51
0.95/ 0.89/-0.82
0.95;
(.67,-0.00
0.67/
0.59/-0.92
0.36/0.03//0.0
0.63/ 0.21/-0.71
Spearmen's
-0.13/ -0.13/0.80/-0008 0.80/-0.80
-0.12/
0.80/-0.80
-0.08/ 0.80/-0.80
-0.19/ 0.40/-0.80
-0.03/ 0.80/-1.00
-0.02/0:20000
0.65/0.400 0.40//0.80
Keudall's
0.16/ 0.67/-0.67
-0.11/
0.67/-0.67
-0.07/ 0.677-0.67
-0.20/ 0.33//00.7
-0.07/ 0.87/-1.00
-0.02/00/
0.51/ 0.33//0..7
PerSEval-H.j251
H.J-Corr.
PSE-RG-L
PSE-RG-SU4
PSE-METEOR
PSE-BLEU
PSE.JSD
PSE-BScore
MSE-IntoLM--03
Pranson's Y
0.54/ 0.90//0.889
0.59/ 0.90/-0.50
0.64/ 0.80/-0.86
0.62;
(-67,-0.68
0.43/
0.90//0..2
0.12/0.01/ 00.00
0.43/ 0.2//0.///-072
Spearuaa's p
0.02/ 0.80/-0.80
-0.02/
0.80/-0.80
-0.0T/ 0.80/-0.50
-0.11/
0.40/-0.80
0.11/ 0,80/-1.00
0.07 20//0
0.50/ 0,10//0.80
Keudall's
0.01 0.67/-0777
-0.04/ 0.67/-0.67
-0.08/ 0.67/-0.67
-0.15/
0.33//0.67
0.14/ 0.67/-1.00
0.09/0/0
0.38/ 0.33/-0.67
PerSEval-H.jae H.JoS
HJ-Corr.
PSE-RG-L
PSE-RG-SU4
PSE-METEOR.
PSE-BLEU
PSE-JSD
PSE-BScore
PSP-Infol.MMo..
Pearsn''s y
0.65/ 0.93/--66
0.59/0.922/00000 0.92/-0.66
0.47/ 0.89/-0.73
0.51/ 0.70/-0.52
0.81/
0.93/-0.80
0.83/-0.04/-0.11
0.56/ 0.24//0.50
Spearman's p 0.60/0.80/-0.80
0.56/
0-80,-0.80
0.24/ 0.80/40.80
0.47/ 0-40/-0.80
0.69/ 0.80/-1.00
0.64/0.20/0
0.4400.40-0044
Kendall's r
0.72/ 0.67/-0.67
0.71/ 0.67/-0.67
0.25/ 0.67/-0.67
0.66; 0.33/-0.07
0.82/ 0.67/-1.00
0.81/0/0
0.73/ 0.33/-0.33
Per H.j.stoMM---
H.J-Corr.
PSE-RG-L
PSE-RG-SU4
PSE-METEOR
PSE-BLEU
PSE-JSD
PSE-BSeore
Peanson's r
0.49/0.91//44
0.60/ 0.90/-0.71
0.86/ 0.58/-0.61
0.58/ 0.67/-0.35
0.78/ 0.90/-00.7
0.51/-0.04/-0.10 10
0.76/ 0.20//00.7
Spearmma's p
0.21/ 0.80/-00000
0.60/ 0:80,-0040 AD
0.50/ 0.80/-0.40
0.55/ 0-30/-0.40
0.62/ 0.80/-0.80
0.47/ 0.20/-0.60
0.61/ 0.40/-0.40
Kendall's r
0.24/ 0.67/-0.33
0.51/ 0.67/-0.33
0.42/ 0.67/-0.33
0.47, 0.33/-0.33
0.45/ 0.67/-0.67
0.38/ 0//00.33
0.42/ 0.33/-0.33
Table 2: PerSEval (PSE) Reliability: Human-juddgment (HJ) corr. between PSE3-1.7.y and PerSEval-HJ- X on Microsoft PENS/OpenAT CNN-DM//penAT TL;DR (Reddit) datasets. Observation 1: PerSEval- InfoLM-aB has maximum highest HJ correlation across all PerSEval-IIJ variants on PENS dataset; Obser- vation 2: PerSEval-HI-InfoLMMaa and PerSEval-HJJBSSoree are optimal meta-evaluation measures since they are most closely aligned to all the seven PerSEval variants; Observation 3: Difference in humon annotator ratings does not serve as reliable surrogate for estimating DEGRESS-HJ (and thereby, PerSEval-H.) as evident from contrudictory results on structurally equivalent OpenAl CNN/DM and TL;DR (Reddit) datasets; Observation 4: As a follow-up, the indirect evaluation method can serve for baseline generation but not meta- eetaaevaauation.
(Sar, Sun )) with the difference in the ratings given by the annotators (i.c., |r(sug)) r(8 )) and that between combined gold-references (a(ui, ux)) with the difference in the average ratings of all summaries included in the combined gold reference (i.e., IF(ug) - F(ux))). For calculating penalty due to EDP, we take the distance between gold-reference and generated summary to be |r(su,,) 1 r(uy)). We then create seven variants of PerSEval-HJ-X as in the PENS dataset-based evaluation, where X stands for the mcasure used for estimating the DEGRESS normalization factor (i.e., the distance of su., and ui: from the document d;; see equation 2 It evident from Table 2 that the HJ-correlation so computed is unstable since the results are contradictory even though both the OpenAl datascts are structured in exactly the same manner with four common policies. This shows that the difference in rating cannot be an alternative the direct similarity judgment that was collected via the survey.
14

---

Inter-Corr.
EG-RG-L EG-RG-SU4
EG-METEOR
EG-BLEU
EG-JSD
EG-BScore
EG-InfoLMMa.
Spearman's
0.903
0.9758
0.8667
0.9636
0.997
0.8476
0.9879
Kendall's 7
0.8222
0.9111
0.7333
0.9111
0.9888
0.7047
0.9553
Table 3: EGISES (EG-X)
Paradox: PSE-X rank-disagreement (< inter-correlation) due to EDP.
See Table 7 in Appendix C.2
for detailed EGISES scores.
PENS
Test Dataset Sample Set (Random Selection)
Ranking
Models
100%
80%
60%
40%
20%
Bias
Variance
SegBird-Pegssus
0.15
0.153
0.1535
0.1558
0.1564
0.0024
5.81E-06
2
SimCLS
0.133
0.1351
0.1356
0.1357
0.138
0.0018
3.08E-06
3
BRIO
0.112
0.1143
0.1151
0.116
0.1155
0.0013
1.77E-06
ProphetNet.
0.098
0.1003
0.1018
0.1012
0.1031
0.0017
2.90E-06
5
T5 (Base)
0.088
0.0899
0.0905
0.091
0.0912
0.0012
1.33E-06
6
PENS-NAMLT T1
0,036
0.0364
0.0367
0.0376
0.039
0.0012
1.41-066
PENS-NRMS T1
0.0315
0.0826
0.0327
0.0331
0.033
0.0006
3.26F-07
8
PENS-EBNE T1
0.0206
0.0209
0.021
0.0212
0.0228
0.0008
6.00F-07
9
PENS-NRMS T2
0.0103
0.0103
0.0102
0.0107
0.0111
0.0003
1.14E-07
10
PENS-EBNE T2
0.0096
0.0097
0.0097
0.0103
0.0107
0.0004
1.84E-07
Table PerSEval 8=1.7 -nfoLM--33 on Stability: 0. 0.0024-strongyy-slabble w.r.l e-Spearman 1; e-Kendall
HJ-Corr.
PSE-ILM-003 BK(PSE-IL.--0, RG-L) (K(PSE-LM--' ILM-a3)
BK(PSE-ILM-aE BLEU)
Spearman P
0.6849
0.1656
-0.3447
0.632
Kendall 5
0.4667
0.0698
-0.3027
0.46
Table 5: Accuracy-leaderboard may mislead: HJ-Corr. of Borda-Kendall (BK) consensus-based aggregated rank vs.
(ILM)-a3.
Evidence of the EGISES Paradox. We look at the inter-corrclations between EGISES-rank and PerSEval-rank of the selected models on the PENS dataset. The values less than 1 across all the variants (see Table 3 suggest that PerSEval not just an offset EGISES and is not entailed by it (which would have otherwise been the case the EGISES paradox was non-existent in reality).
Stability of PerSEval. We compute c-0-stability of the best performing PerSEval- on PENS over the ten models as per the sampling method and stability definitions in Section 7.2 We observe PerSEval to be sto022/sstoo
w.r.t Spearman-e = and Kendall-e = 1 (see Table 4h. This establishes a very high stability of PerSEval along with its reliability, making it
robust. For detailed results, see Appendix C.1
8.2 Accuracy Leaderboards may Mislead
We demonstrate that accuracy leaderboards, at best, are redundant and PerSEval-rank is sufficient to capture personalization. For this, we generate the BendalKendall consensus-based aggregated rank (Colombo al.j 2022ab and compare the HJ-correlation with that of PerSEval-InfoLLM-o.. We observe that the stand-alone HJ correlation has the same strength w.r.t the aggregated rank for accuracy measures like BLEU, thereby rendering them redundant in the context of personalization evaluation, while is significantly higher (Spearman p: 0.51+1; Kendall 0.40+1) than that of mcasures such as RG-L, InfoLM-a:3 (see Table 5 This indicates that accuracy ranking can also inject noise.
8.3 Computational Complexity of PerSEval.
We now show that the best performing PerSEval-a,3 is lincar time w.r.t to summary lengths i.c., +l,++1,) for a fixed evaluation dataset [ where L,,, and l, are lengths of model-gencrated summary su and nisrrgenerated reference summary u.
8 Maximum of 5-bias and 8-variance all models
15

---

Theorem 5 (PerSEval Time Complexity). The best worst-case time complexity of PerSEval is O(ID) (1s, + (,)).
Proof. Let lox be the time-complexity of computing the distance using measure ox
O(DEGRESS( [(d, uz)))) O(UUa, (xx) O(lxx)
{: [U,, = number of reference summaries per article S constant C};
(DEGRESS)) |D| (Uaa, O(DEGRESS(s., (dd, (;y)))) [D] Offax)
Now, O{EDP Suyy (dd, u:y)) (O(ADP(s [(d,, uiz))) (ACP( (:)))) OUU,, -ox) O(lox)
S(PerSEval) = [D] (toxx) ) = ((D) tox)
For ox E (RG-L, RG-SU4, JSD, InfoLM-a88): O(tox) = Ol.a - +lu;
(Suffix-tree based matching for RG:)
(No. of layers, dimensions, and vocabulary size for a fixed MLM (BERT) used in InfoLM can be taken as constant]
For ox E (BLEU, METEOR, BScore}: O(tex) = Ollo lw); (No. of layers, dimensions, and vocabulary size for fixed BERT model used in BScore be taken as constant]
O(PerSEval) ( (l., +(w) for the best-performing PerSEval-a3
For computing PerSEval, we observe an average runtime 2:37 minutes for each of the evaluated ten models on the PENS dataset (3840 articles), which is reasonable for an offline procedure 10
9 Related Work
Personalization evaluation has been well studied in recommendation systems (reosys) /Zangerle & Bauer 2022), such as metrics based on the Jaccard Index, rank-order edit distance (Hannak et al. 2013), MAE RMSE Hit-Ratio (Li et al. 2024), and nDCG (normalized Discounted Cumulative Gain) (Matthijs Radlinski 2011). A comprehensive compilation all recsys-oriented metrics and their applications can be found in (Zangerle Bauer 2022 Kuanr Mohapatra 2021) While relevant to recsys, these metrics are not pertinent for text summarization since they rely on human feedback (such as clicks and likes) on rank list of potentially preferable "ilems" situation that does not exist. for summarizers. A survey-based qualitative analysis of the usefulness of model-generated summaries proposed by Teer Hoeve et al. (022).. Although this work establishes empirically that model-generated summary utility is subjective (as argued in this paper), yet to date, the only work on the formal quantitative evaluation of personalization in summarization models is EGISES (Vansh et al. 2023 which, however, can only capture responsiveness. There is a growing interest in learned summarization evaluation metrics, as opposed to designed ones Peyrard & Eckle-Kohler (2017): Peyrard & Gurevych (2018): Böhm et al. (2019): Liu et al. (2023). ALM-based (2023a) However,
Evaluation of Personalization
automated summarization evaluation has also been studied Shen al. (2023 Zheng
to the best of our knowledge, these works have not focused on evaluating personalized summarization models where subjectivily in saliency needs to be respected (and not normalized). Any such automated evaluator needs to be fine-tuned on large volumes of user profiles having varied subjective expectations. This sort of dataset hard to design, and such a project only makes sense designed metrics are not able to achieve minimum human-judgment correlation.
"In this paper, X: RG-L, RC-SU4, METEOR, BLEU, JSD, BScore (we assuiee pre-computed embeddings), InfoLM-a which BERT Base (uncased; 110M params) pre-trained Masked Language Model. 10system specifications: Machine architecture: _66_64; CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz; CPU Cores: 16; (hread(s) per core: 2.
16

---

Personalized Summarization Aspect-based. An aspect-based summarization model generates sum- maries that are coherent with the aspects (i.e., themes/Lopicee therein (Narayan et al. 2018 Frermann &
Klementiev 2019 Tan et al.) 2020: Hirsch et al. 2021 Hayashi et al. 2021; Meng et al. 2021 Soleimani
ct al. 2022 While explicit aspects can be restrictive and rather broad (e.g., the MA news training dataset
where six broad aspects are identified: "sport", "health', "travel', "news', "science technology', "Iv showbiz') ) implicit aspect-based summarization implics augmenting the aspect query with concepts that are related to the predefined aspects. Although these models have specific use-cases, they are not trained to adapt to the reader's evolving profile (i.e., reading behavioral pattern) that constitutes discourse- interest drift (and not just topic-level static interests). Also, the evaluation was w.r.t accuracy using standard ROUGF-variants.
One of the earliest interactive interface-based iterative personalized summarization frameworks was proposed by Yan et al. (2011) where users could click on specific sentences in the generated summary that are of their interest (implicit preference), and read the associated context. (i.c., surrounding text) of the selected sentence before sending this preference as feodback to the model for a revised version. A similar interactive interface-based framework was proposed by PPS al. (2018) where users can iteratively select sentences and the phrases therein that they prefer (and do not prefer as well), while the summarizer, an ILP-based model proposed by Boudin et al. (2015). updates the summary based on this feedback until the user is satisfied. On similar lines, Ghodratnama et al. 2021) introduced a personalized summarization method for extractive summarization. Extracted summary concepts are presented to readers for their feedback, which is used to iteratively fine-tune the summary until no further negative feedback is received. Bohn & Ling (2021) proposed a framework where the acceptance or rejection of summary sentences was made dynamic as the summary gets generated on-the-fly. However, all these works have been evaluated based standard accuracy evaluations (such as ROUGE variants).
Interactive Hluman-feedback-beseee
eser-preference Trained Reward Model-based. Another way of inducing personalization in models is to train a base model (pre-trained or supervised fine-tuning) as an agent within a reinforcement learning framework (usually policy-gradient based) using a reward model (RM) as the environment. The RM is trained on human preferences to predict human ralings for specific actions of the agent (i.e., selecting specific words ; thereby providing the rewards (Stiennon et al. 2020 Nguyen et al.) 2022). However, whether these models can explicitly "remember' individual preferences (or even that of user groups with similar interests) is still to be probed and not quite clear. Nevertheless, the evaluations were accuracy only.
diserprreference hhwpeenccchisttt--bbbrrr based. There has been considerable work in user hearrencee-iistr--bbeeeee product review summarization. The user profile is usually modeled in terms of discrete attributes such as rating, user- ID, and product-ID, and history text that represents usee- aser-written historical
Some of the proposed models use user-specific vocabularies to predict user-preference words that turn serve as a guide for generating summaries Ma al. [2018); Li et al. (2019); Chan et al.] (2020). Others have proposed models that learn user preference by jointly training these discrete attributes and the historical review-summaries
Liu & Wan 2011)) Xu et al. (2021 2023 where the historical summaries provide information about the writing-style, purchasing preferences, and aspect-of-interest of the users. These works, however, do not fit into the general setup of personalized summarization because the summaries generated are not aligned to a prospective buyer's (i.e., a consumer of seview-summary or reader in our parlance) preference behavior, but rather tuned to a different set of buyers who are active reviewers and who have provided gold-reference review summaries of their own reviews (i.c., the review-to-sin to-summary is a one-to-one mapping and not subjective). Nevertheless, the evaluation has been done using accuracy measures (ROUGE variants) only. So far, the only pertinent work incorporating the reader's history preference is the proposed models that were designed using the PENS framework (Ao et al.] 2021), which we studied extensively (sce Section 6.2 b. It is clear from our study that they need significant improvement in terms of personalization (as measured by PerSEval).
10 Conclusion
In this paper, we presented PerSEval, a corrective measure for EGISES (proposed by Vansh et al. 2023) ) which, to the best of our knowledge, is the only known personalization mcasure for summarizers. We first introduced the concept of responsiveness, in contrast to personalization, as measure to evaluate the capacity of model to discern the differences in reader profiles (i.e., reading histories) and generate
17

---

reader-specific summaries that maintain this difference proportionately. We then showed that EGISES measures the former. We thereby proved theoretically and empirically on the real-world PENS dataset that measuring responsiveness does not imply measuring personalization since there can be models that generate distinctly different summarics for different reader profiles (i.c., high responsiveness) but are quite off from the expected summmarie (i.e., low accuracy and thereby low user-experience (UX)). We then formulated Per (more specifically, DEGRESS) as a discounted EGISES where the discount factor penalty due to accuracy drop called EDP (Effective DEGRESS Penalty Factor). We analyzed the ten SOTA summarization models using seven variants of PerSEval and observed that the model leaderboard reliability depends the chosen variant. We further observed that the variant PerSEval-Info.Moay performs best regarding rank-stability, a meta-evaluation measure we proposed in this paper. We also proposed a novel survey-based meta-evaluation protocol for human-judgment (HJ) to analyze the extent to which human annotators agree with the design principles of PerSEval at a cognitive level. We found that PerSEval-InfoLM-a has the highest overall HJJcorrelation (Pcarson's r = 0.79; Spearman's p = 0.68; Kendall's t = 0.47). We finally established that separate accuracy leaderboards for personalized summarizers can be misleading and PerSEval can serve as a unified measure, thereby emphasizing that personalization and accuracy are inseparable aspects of UX.
Discussions & Future Directions
User guideline: PerSEval as measure is useful in summarization tasks where the user experience of the utility of the summaries generated by any model will be subjective and will depend upon the user's current preferences (which itself is a reflection of past activities). For example, executive summaries and Minutes-of- meetings (MoMs) need to be personalized (as different participants might have different takeaways/toddo from the moeting summary) without compromising on the accuracy of the procoedings. In the same way, news recommendation platforms are also likely to benefit from personalized summarization as it increases the likelihood of the article being clicked and read in full if the title summary is more personalized per the user profile. In such use cases, PerSEval should be adopted rather than accuracy measures or P- P-Accuracy, It is important to note that we do not rule out eristing conventional accuracy measures and also P-Accuracy (as proposed in Vansh et al. (2023) in favor of PerSEval. These measures have their own important role to play for different use cases. Specifically, summarization tasks that do not need personalization should be better evaluated using standard accuracy measures. An example would be the rescarch paper summarization task where saliency, generally, should imply the core contributions of the paper, and hence, the user experience of such summary utility does not depend on specific user preference history. Likewise, there use cases where P-Accuracy can be more useful, especially when accuracy is the primary objective, yet sunmarization models to be evaluated also claim to be personalized. An example would be the task of multi- docti-d update summarization for very specific like-minded stakeholders (say, members of a government event-monitoring unit engaging in daily tweets) whose topics of preferences are already known. In this case, P-Accuracy will factor the lack of personalization (if any) and provide a more realistic accuracy score (and, thereby, the accuracy leaderboard).
Future directions: In this work, we analyze the effect of seven variants of PerSEval on SOTA summarization models, out of which the ROUGF-variants, BLEU, and METEOR are defined on the string space, JSD is defined on the probability space, BERTScore is defined on the embedding space, and InfoLM is defined on the probability space that is generated from the embedding space using a masked-LM. Although these cover all the most common algebraic spaces on which PerSEval can be defined, it remains to be understood how other alternate measures on the same spaces, such as BaryScore (Colombo et al. 2021 MoversScore (Zhao et al., 2019), DepthScore (Staerman et al., 2022). and other variants of InfoLM using multiple Csiszar f-divergences, will behave w.r.t H.J-correlation. Finally, we have only explored one method of estimating PerSEval-H.J (i.e., mimicking the human way of computing PerSEval using RG-L the distance between model-generated summary and human-reference) for analyzing H.J-crrelations. However, there can be other alternative methods of estimating PerSEval-HJ, including incorporating aele-aanotator-agreeeen (such as Kappa statistic). We also consider it a promising future direction to study the effectiveness of PerSEval as an objective function to steer the finc-tuning of the personalized summarization models under a Reward Model-driven reinforcement learning setup. In general, the current work can potentially open up directions of systematic evaluation studies of personalization capabilities in models, particularly LLMss with
statistics
18

---

their In-context Learning capabilities, that apply to all AI challenges where the quality of the model output is subjective to the user's past preferences. We, therefore, encourage the research community to critically study our work and build on it.
Ethics Statement
We would like to declare that we used the PENS dataset prepared and released by Microsoft Research. Our human-judgment survey was conducted according to the norms set by the Institutional Review Board (IRB) and respects participant anonymity as per guidelines.
References
Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. PENS: A dataset and generic framework for personalized news headline generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 92. Association for Computational Linguistics, August 2021. doi: 1..18633vv12022 acc.oo...--.. 7. URL https://aclantthloog. wwwp:/hwoooo.oogg/..... org/2021. acl ac1-long.7
Satanjoev Banerjce and Alon Lavic. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Ncubig. Ro-evaluating evaluation in lext sunmarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9347-9359. Association for Computational Linguistics, November 2020. doi: 10.18653/ v http//2222eemmmlpmee m51..751. URL
org/2020. emn.p-main......mmmmmmmmmmmmm 751
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Kentaro Tnui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP pp. 3110 3120, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ v11 139- URL httpo:///ccaattbooogyoo/////////
Tanner Bohn and Charles X. Ling. Hone you read: A practical type of interactive summarization, 2021.
URL https://arxiv org/ abaaaaas//22/ 0ttps/211.02200
Florian Boudin, Hugo Mougard, and Benoit Favre. Concept-based summarization using integer linear programming: From concept pruning to multiple optimal solutions. In Conference Empirical Methods in Natural Language Processing (EMNLP) 2015, 2015.
Hou Pong Chan, Wang Chen, and Irwin King. A unified dual-view model for review summarization and sentiment classification with inconsistency loss. In Proccedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, pp. 1191 1200, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781-40038016.. doi: 10. 10.1145//33072711330000
URL htttp:://do..org/// /twp://////1111199//911 11453339777333 3401039
Andrzej Cichocki, Sergio Cruces, and Shun-ichi Amari. Generalized alpha-beta divergences and thoir application to robust nonnegative matrix factorization. Entropy, 13:134 170, 2011. URL httpps://ppi
semanticscholar. http:/o/ar....r//oorpus//////010
Pierre Colombo, Guillaume Staerman, Chlos Clavel, and Pablo Piantanida, Automatic text evaluation through the lens of Wasserstein barycenters. In Proccedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10450-10466, 2021.
19

---

Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stéphan Clémençon. What are the best systems? new perspectives on nlp benchmarking. Advances in Neural Information Processing Systems, 2022a.
Pierre Jean Colombo, Chloé Clavel, and Pablo Piantanida. Infolm: new metric to evaluate summarization data2text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10554-10562, 2022b.
Wade D Cook and Lawrence M Seiford. On the borda-kendan consensus method for priority ranking problems. Management Science, (166)6611-33) 1982.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 7580-7605, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: N0.18653/v1/202 httt////2ee.emmmmmiiii.. emnlp-main.599. URL
www.eaaonoooooooo/oeeeeee wrtp///2...0../. apt.:ooooo..rr2222...mmppamaaaa emnlp-main. 599
Abdul Ghafoor Etemad, Ali Imam Abidi, and Megha Chhabra. Fine-tuned t5 for abstractive summarization. International Journal of Performability Engineering, 17(10). 2021.
Lea Frermann and Alexandre Klementiev. Inducing document structure for aspect-based summarization. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pp. 6263 6273, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10. (118633vv1/P99- 11/119-1630. 1630. URL https://aalaanthologyy http://cc_..nttooooooooro////s1/
Yang Gao, Wei Zhao, and Steffen Eger. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1347-1354. Association for Computational Linguistics, July 2020, doi:
URL https://aclaatthoooo.o////////// wwt.220....mma........m.. 1
Yanjun Gao, Chen Sun, and Rebecca J Passonneau. Automated pyramid summarization evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), 2019.
Samira Ghodratnama, Mehrdad Zakershahrak, and Fariborz Sobhammanesh. Adaptive summaries: A personalized concept-based based summarization approach by learning from users' feedback. In Service- Oriented Computing ICSOC 2020 Workshops, pp. 281 293, Cham, 2021. Springer International Publishing. ISBN 978-3-030-76352-7.
Service-Oriented
Aníko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. Measuring personalization of web search. In Proceedings of the 22nd international conference on World Wide Web, pp. 527-538, 2013.
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. WikiAsp: Dataset for Multi-domain Aspect-based Summarization. Transactions the Association for Computational Linguistics, 9:211 225, 03 2021. ISSN 2307-387X. doi: 10.1162/taci a 00362. URL
https://6oi.orr// 11w.o/aclaatttaaaaeeeoaaa 003622
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Tn Proceedings of the 28th International Conference Neural Information Processing Systems Volume 1, NIPS pp. 1693 1701, Cambridge, MA, USA, 2015. MIT Press.
Eran Hirsch, Alon Eirew, Ori Shapira, Avi Caciularu, Arie Cattan, Ori Ernst, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, and Ido Dagan. iFacetSum: Coreference-basee interactive faceted summarization for multi-document exploration. In Proccedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2021. URL https://accaantoology 2021. emnlp-demo.
20

---

Raghav Jain, Vaibhav Mavi, Anubhav Jangra, and Sriparna Saha. Widar-weighted imput document augmented rouge. In Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022,
Stavanger, Norway, April 10 14, 2022, Proceedings, Part 1, pp. 304 321. Springer, 2022.
Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. Multi-dimensional evaluation of text summarization with in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8487-8495, Toronto, Canada, July 2023. Association for Compulational Linguistics. doi: 10.18653/ v1 / www.:0i....nnnnaaalla..... https: https://aclantholoogy. 2r2/2023. 2tt.:222.....n..n findings-acl.533
URL
Madhusree Kuanr and Puspanjali Mohapatra, Assessment methods for evaluation of recommender systems: A survey. Foundations of Computing and Decision Sciences, 46:393 421, 2021. URL https://api.
semanticscholar. http:/shh....a..rroroopps////s/e httg//ooppuID:2453944444
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequencc-to--sqqunnce pro-training for natural language generation, translation, and compreheusion. In Proceedings of the 58th Annual Meeling of the Association for Computational Linguistics, pp. 7871-7850. Association for Computational Linguistics, July 2020. doi: org/2020 acl- maaln.ainn
10.18653/v1 /ttp:///m/22..mmmii...... 703. URL
Junjie Li, Haoran Li, and Chengqing Zong. Towards personalized review summarization via user-aware sequence network. In Proceedings of the Thirty-Third AAAT Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI119 111////AAA19//AAA/A AAAI Press, 2019. ISBN 978-1- 57735-809-1. doi: 10.1609 /aaii
URL https://doi.
wwt.///aaii..a.. v33101. 3301
Shu Li, Yuan Zhao, Longjiang Guo, Meirui Ren, Jin Li, Lichen Zhang, and Keqin Li. Quantification and prediction of engagement: Applied to personalized course recommendation to reduce dropout in moocs.
Information Processing & Management, 61(1):103536, 2024.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74 81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL
https://acLanthology http://ccannttolooooooorr//-/-a/
org/wo4-1013
Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACT-04), pp.
2004.
Hui Liu and Xiaojun Wan. Neural review summarization leveraging user and product information. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19 pp. 2389 2392, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 334455//333338433333
9781450369763. doi:
URL https: /ttps:///ool: org/ 11155/3573884 3358161
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1-2222,, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/w1/2023.om 153
153. URL https::/ wwwp:/gy...rg.2/..... / emnlp-main,
Yixin Liu and Pengfei Liu. SimCLS: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 1065 1072. Association for Computational Linguistics, August 2021. doi: 10.18653 22011 acl-short.113 URL https://acclantholeg. wrw.o/gg..2222 acl.sacclsshrrr.. 135]
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. BRIO: Bringing order to abstractive sum- marization. In Proceodings of the 60th Annual Meeting of the Association for Computational Linguistics
(Volune 1: Long Papers). pp. 2890- 2900, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 22////v/2222..//// acl lo0. URL htttpss//adlaathollggoo///////// acl- long.
21

---

Shuming Ma, Xu Sun, Junyang Lin, and Xuancheng Ren. A hierarchical end-to-end model for jointly improving text summarization and sentiment classification. In Proceedings of the Twenty-Sevent International Joint, Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2018.
Nicolaas Matthijs and Filip Radlinski. Personalizing web search using long term browsing history. In Proceedings of the fourth ACM international conference Web search and data mining, pp. 25 34, 2011.
Rui Meng, Khushboo Thaker, Lei Zhang, Yuc Dong, Xingdi Yuan, Tong Wang, and Daqing Hc. Bring- ing structure into summaries: a faceted summarization dataset for long scientific documents. In Procoodings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pp. 1080 1089. Association for Computational Linguistics, August 2021. doi: 10.18653/ve 1000555/1//// w1/1021.2200 www.accls.....aaeeeeeee URL https://aalaanthooggy org http://..ooo.............aaaooo.
M.I. Menéndez, J.A. Pardo, L. Pardo, and M.C. Pardo. The jensen-shannon divergence. Journal of the
Franklin Institute, 334(2):307 318, 1997. ISSN 0016-0032. doi: https://doi.orr// 11w.://///./....//0/00000 100000000000000000200000 00322))9)996 00063-4. URL https://wwwt http://wwww.cceeeeeeeeeeteeetttt wtt.ccco///////e
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
URL https://aclanthoo wwwprhhooogg.orrr//1111 / 1206.
Duy-Hung Nguyen, Nguyen Viet Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Tien Le, Shahab Sabahi, Minh- Tien Nguyen, and Hung Le. Make the most of prior data: A solution for interactive text summarization with preference feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1919-1930, Seattle, United States, July 2022. Association for Computational Linguisties. doi: 10.18653/v1/ 2022.finding.. naa.l.11a......... URL https://aclanthoolggy
findings- 147.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. Smbeediigg-assed news recommendation for millions of users. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, KDD '17, pp. 1933 1942, New York, NY, USA, 2017. Association for Computing doi:
Machinery. ISBN 9781450348874.
TRRLhhtps://doi.orrg 1145//0991115//
3098108
Mikhail Orzhenovskii. T5-long-cxtract at fns-2021 shared task. In Procoodings of the 3rd Financial Narrative Processing Workshop, pp. 67-69, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Maxime Peyrard and Judith Eekle-Kohler. Supervised learning of automatic pyramid for optimization- based multi-document summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1084-1094, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-I1 URL https://aclanthologyy org/ P17-1100
Maxime Peyrard and Iryna Gurevych. Objective function learning to match human judgements for optimization-basee summarization. In Proccedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. (554-660) New Orleans, Louisiana, June 2018. Association for Computational Linguis- tics. doi: N..16633vv1/M 11165333///N112103 URL https::/aclanntoooggyy.coo 1twp://ooyyorr/N1/-1119000
Avinosh PVS, Benjamin Hättasch, Orkan Özyurt, Carsten Binnig, and Christian M. Meyer. Sherlock: a system for interactive summarization of large text collections. Proc. VLDB Endow.. 11(12):1902 1905, aug 2018. ISSN 21 2050-0097. doi: 10.14778 10.14778/322003333332222 /322//2993333332200
http://d...orrr//...111////26661 URT. htttps://doi.org///// 1001114777388888 3236220
22

---

Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. Net: Predicting future n-gram for sutrenncetooSSqqueeeerr--r-ae
In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2401 2410. Association for Computational Linguistics, November 2020. doi: 10.18653/vv /006633/v/// /2020 moddings-cmnl.2111 URL https://aclanthoolt org/ 2020. www.diiinnnneenll....
GS Ramesh, Vamsi Manyam, Vijoosh Mandula, Pavan Myana, Sathvika Macha, and Suprith Reddy. Ab- stractive text summarization using t5 architecture. In Proceedings of Second International Conference on Advances in Computer Engineering and Commuicattion Systems: ICACECS 2021, pp. 535-543. Springer, 2022.
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. Large language models not yet human-level evaluators for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4215-4233 Singapore, December 2023. Association for org/
Computational Linguistics. doi: 10.18653/ /v1 /2023 httd//22...nnngssmnmmenme finding--ennpp.278 URL 2023. www.fiiinnnnnmmmmm 278
Amir Soleimani, Vassilina Nikoulina, Benoit Favre, and Salah Ait Mokhtar. Zero-shot aspect-based scientific document. summarization using self-supervised pre-training. In Proceedings the 21st Workshop on Biomedical Language Processing, pp. 49 62, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (0.18653/v1/20222 bionlppp... URL httpss//acclanthooogg.. ortplogg..rr//0000 http//222.bioolp....
Guillaume Staerman, Pavlo Mozharovskyi, Pierre Colombo, Stéphan Clémençon, and Florence d' Alché Buc. A psoudo-metric between probability distributions based on depth-trimmed regions, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 3021, 2020.
Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. Evaluating the factual consistency of large language models through news summarization. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 122005225, Toronto, Canada, July 2023. Association for URL https://aclanthoology/ org/
Computational Linguistics. doi: 10. 10.18853/vll http://....nninnnnaa.lll..l
2023. 3023.findings- findings-acl.335
Bowen Tan, Lianhui Qin, Eric Xing, and Zhiting Hu. Summarizing text on any aspects: A knowledge-infomeed weakly-superviseed approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6301 6309. Association for Computational Linguistics, November 2020. doi: 10.18653/
emnlp-main.510000 510. URL https:://allanthoolgg wrwp://g..ogg//0oooo/ ee0.00mmllplmmi..mmm 510
T Tawmo, Mrinmoi Bohra, Pankaj Dadure, Partha Pakray, et al. Comparative analysis of t5 model for abstractive text summarization on different datascts. In Proceedings of the International Conference on Innovative Computing Communication (ICICC) 2022. SSRN, 2022. URL hntpp::/sssnn.oom com =bstract= cottttraattm 409641 109641orrtttp///0 //dtt:::///x..c. dwi. oww.oorg11..00 2139/ ssrn. 4896413.
Maartje Ter Hoeve, Julia Kiseleva, and Maarten Rijke. What makes good and useful summary? Incorporating users in automatic summarization research. In Marine Carpuat, Marie- Carti-CCtheeinee de Marneffe, and Tvan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 46 75, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. Nt.11853/v1/2000 mancl-main.4. URL https://aclanthology wtw.:orggg//.ae.ee htt.:/oog.....g.......aaaaaaaaa. naa.l-aalnnaaaa
Rahul Vansh, Darsh Rank, Sourish Dasgupta, and Tanmoy Chakraborty. Accuracy is not enough: Evaluating personalization in summarizers. In Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 2582-2595, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ (120033.mninngsemmll 169. URL https::/aaclaatholleg..o//////// httpp//////.///tt//////aaaaaaaaa
findings-emnlp 169
23

---

Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the Workshop on New Frontiers in Summarization, pp. 59 63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 1118653/111W17-4000 URL https://aclanthology org/w17-4508
Chuban Wu, Fangzhao Wu, Mingxiao An, Jiangiang Huang, Yongfeng Huang, and Xing Xie. Neural news rec- ommendation with attentive multi-view learning. In Proceedings of the "wenty-Eighth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2019a.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. Neural news recommen- dation with multi-head self-antention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pp. 6394, Hong Kong, China, November 2019b. Association for Computational Linguistics. doi: 10. N0.18653/v1l 11/1//////111111 URL httpss///cclaathooogg.oog///////
Hongyan Xu, Hongtao Liu, Pengfei Jiao, and Wenjun Wang. Transformer reasoning network for personalized review summarization. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retricval, SIGIR '21, pp. 1452-1461, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450399379. doi: 10. 0..11453444883 3462854
URL hnttps://do .rr//
Hongyan Xu, Hongtao Liu, Zhepeng Lv, Qing Yang. and Wenjun Wang. Pre-trained personalized review summarization with effective salience estimation. Tn Findings of the Association for Computational Linguistics: ACL 2023, pp. 10743 10754, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 http////w//222...nnnn-o-o/o/ 684.
URL http:: htttps://aclanthologyy org. org/ 2023. htt.//22............eeee findings acl.
Rui Yan, Jian- Yun Nie, and Xiaoming Li. Summarize what you are interested in: An optimization framework for interactive personalized summarization. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1351, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguistics. URL
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https::/prooceedings
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, vol- ume 33, pp. 17283 17297, 2020. URL https://ppocceddings inw.r//. www.rpp..ccpppprrr.a.........eee wtt.e/e/aappeee//2///iaa/// c8512d142a2en 1B51221-222266477751313882788323 wwwp:///7.....1aaaaaaa.eere .df
Eva Zangerle and Christine Bauer. E'aluating recommender systems: Survey and framework. ACM Comput. Surv., 58) dec 2022. ISSN 0360-0300. doi: 10.1145//556636. URL htttp:://doi.org/11 111.///////1111/335556666
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https:
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, 12:39 57, 01 2024. ISSN 2307-387X. doi: 10.1162/tad a 00632. URL
https://doi.org/// 1162/tacc_aaaae a__0632
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings
24

---

of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint. Conference on Natural Language Processing (FMNI.-IICNT.P), pp. 563-578 578, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 1118653/11/D1910000 URL https:
//aclanthology.co 00rp////000y.////1--19910
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ton Stoica. Judging LLM-as-a- judge with MT-bench and chatbot arena. In Thirty-sovenhh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a. URL https:///ppnnrevieew. wttp://wwww....oooororooeooooooo
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part i: Ppo, 2023b.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation. In Procoedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023 2038, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1 2022.emmlppp... wwnlp-main.nnnne main. URL https:///allanthology / 2022. mmnlp-mainn.. 131
A Model Details
We brielly introduce the SOTA summarization models that were analyzed to understand their degree-of- personalization below:
1. PENS-NRMS Injection-Type 1: The PENS framework AA et al. 2021 takes user embedding as input along with the news article to generale a personalized summary for that user. To generate user embedding, NRMS (Neural News Recommendation with Multi-Head Self-Attention) (Wu et al, 2019b) is used. It includes a news encoder that utilizes multi-head self-attentions to understand news titles. The user encoder learns user representations based on their browsing history and uses multi-head telt-ontention to capture connections between news articles. Additive attention is added to learning the news and user representations more effectively by selecting important words and articles. Here, Injeetion-Type 1 indicates that NRMS user embedding is injected into PENS by initializing the decoder's hidden state of the headline generator, which will influence the summary generation.
2. PENS-NRMS Injection-Type 2: To generate personalized summary, NRMS user embedding is injected into attontion values [IjeccioonTType 2) of PENS that helps to personalize attentive values of words in the news body.
3. PENS-NAML Injection-Type 1: NAML (Neural News Recommendation with Attentive Multi- View Learning) Wu et al. (2019a) incorporates news encoder that utilizes a multi-view (i.e., titles, bodics, and topic categorics) attention model to generate comprchensive news representations. The user encoder is designed to learn user representations based on their interactions with browsed news. It also allows the selection of highly informative news during the user representation learning process. This user embedding is injected into the PENS model using Type-1 for personalization.
4. PENS-EBNR Typpe 1: EBNR (Embedding-based News Recommendation for Millions of Users) Okura et al. (017) proposes a method for user representations by using an RNN model that takes browsing histories as input sequences. This user embedding is injected using Type into the PENS model for personalization.
PENS-EBNR Injection-Type 2: This personalized model injects EBNR user embedding into PENS using type-2.
25

---

6. BRIO:
Instead of traditional MLE-based training approach, BRIO Liu al (2022) assumes non-deterministic training paradigm that assigns probability mass to different candidate summaries according to their quality, thereby helping it to better distinguish between high-quality and low-quality summarics.
7. SimCLS: SimCLS (A Simple Framework for Contrastive Learning of Abstractive Summarization) Liu & Liu (2021) uses two-stage training procedure. In the first stage, a Seq2Seq model (BART (Lewis et al,, 2020)) ) is trained to generate candidate summarics with MLE loss. Next, the evaluation model, initiated with RoBERTa trained to rank the generated candidates with contrastive learning.
8. BigBirdd-Pegasus: BigBird Zaheer et al. (0200) is an extension of Transformer based models designed specifically for processing longer sequences. It utilizes sparse attention, global attention, and random atlention mechanisms lo approximate full tentionn. This enables BigBird to handle longer contexts more efficiently and, therefore, can be suitable for summarization.
ProphetNet Qi et al. (2020) is a sequence-to-sequeece pre-trained model that employs n-gram prediction using the n-stream self-allention mechanism. ProphetNel optimizes n-slep ahead prediction by simultaneously predicting the next n tokens based on previous context tokens, thus preventing overfitting local correlations.
ProphetNet:
10. T5: T5 (Text- Text Transfer Transformer) is based on the Transformer- bassdd Encoder-Decode architecture that operates the principle of the unified text-to-text. task for any NLP problem, including summarization. Some recent analysos the performance of T5 on summarization tasks
can be found in Tawmo et al. (2022); Ramesh et al. (2022) Etemad et al. 20211)
B Accuracy and Performance
B.1 Accuracy Measures Compared
1. RG-L: ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation) Lin & Och 2004 calculates the longest common subsequence between the generated summary and the reference summary and then measures the precision, recall, and Fl score based on this comparison.
2. RG-SU4: ROUGE-SU4 (Lin 2004 was designed to consider skip-bigram matches well, which allows for non-contiguous a-gram matches.
3. BLEU: BLEU (Bilingual Evaluation Understudy) (Papineni et al. 2002) is popular evaluation metric that measures the precision of n-gram matches between the model-generated sunmaries and the reference summaries. BLEU computes a modified precision score for various n-gram lengths and then combines them using a geometric mean.
4. METEOR: METEOR (Metric for Evaluation of Translation with Explicit ORdering) (Banerjoc & Lavie 2005) matches unigrams based on surface forms, stemmed forms, and meanings and then calculates score using a combination of precision, recall, and the order-alignment of the matched words w.r.t reference summary.
5. Jensen-Shannon Distance: The Jensen-Shannon Distance (JSD) Menéndeez et al. 1997 is a metric used in summarization evaluation to measure the dissimilarity between probability distributions of words a reference summary and a generated summary. It quantifies the information divergence and similarity, providing a nuanced assessment of the semantic content overlap between the two summaries.
BertScore: BertScore (BScore) (Zhang al. 2020 is a metric for evaluating machino-generated sum- maries, emphasizing contextual embeddings from BERT to assess both word overlap and contextual relationships. It. overcomes the limitations of keyphrase-based based measures like ROUGE.
26

---

7. InfoLM-a3: Given user-generated reference summary and model-generated summary Sus InfoLM (Colombo et al. 2022b) recursively masks each token position & of both u (denoted [u]** and Su (denoted [s)]" to obtain individual masked contexts of length lu and 4, respectively. For each masked context, it uses a pre-trained maskod- languugge model to estimate the corresponding probability distribution over the vocabulary (i.e., Pe (- [:]%; Ma,1), resulting two bags of distributions of size by and ls for u and 8g. The bags of distributions (for both masked u and masked 8.) are then averaged out, follows
p(. u; Mo,
-
Th x po (. [u]*; Mo.1)
R=1
p(: 8u; Mo.b)
7h po
[su]* Mo..)
k=1
InfoLM then uses a chosen information measure I to compute the following:
InfoLM(u, s.) _ I [p(- u), sp)]
In our experiments, we chose I to be a.3-divergence (also called AB-Divergence; D et al. 2011 where the divergence is defined as:
Cichocki
AB = 1
log Σ Sta + 3
log
Sta
log Σ pig
(8)
a))
B.2 Model Rank Aggregation & Agreement
Kendall Consensus based Rank Aggregation: The Borda-Kendall (BK) consensus entails aggregating a set of permutations, denoted as n ,/. E n, which represent the rankings of N models across L tasks or instances (in our case, the pair of accuracy rank measure and the PerSEval-variant to be aggregated). This aggregation involves summing the ranks of each model and subsequently ranking the obtained sums. Formally:
1//
sum, Σ
the for every 1 N,
BK (n....... n/) = argsort (sum),.... sump)
2. Pearson's Correlation Coefficient (r):
E (2, Z) D)
E (x: -)2 Σi= (1/ π)²
where H, are the means of the variables x; and zi ; the number of samples.
Spearman's p Coefficient:
6 6Σ
p =
nnn -1
where d the pairwise distances of the ranks of the variables xa and yi ; n the number of samples.
Kendall's Coefficient:
d
S
25
n(n
11
where, c = the number of concordant pairs; = the number of discordant pairs.
11 measures of the importance of the l-th token in and respectively s.t. E/e 7h Eley
computed using the corpus-level inverse document frequency (IDF) scores.
27

---

PENS Test Datasct Sample Set (Random Selection)
3-Dias (in bold) of PSE-variants
Modela
PSE-RG-L
PSE-RG-SU4
PSE-METEOR PSE-BLEU
PSE-JSD
PSE-BScore
PSE-InfoLM-art
BigBird- Pogasus
0.0009
0.0034
0.001
0.0031
0.0011
0.0015
0.0024
SimCLS
0.0034
0.007
0.0049
0.0054
0.0019
0.0017
0.0018
BRIO
0.0011
0.0072
0.0055
0.0051
0.001
0.0017
0.0013
ProphenNet
0.0033
0.0059
0.0052
0.0015
0.0017
(Base)
0,0035
0.0049
0.0047
0.0032
0,0004
0.0016
0.0012
PENS- NAML A.I.TT TI
0.0033
0.0011
0.0027
0.0003
0.0001
0.0016
0.0012
PENS-NRMS
0.0029
0.0003
0.0033
0.0008
0,0003
0,9016
0.0006
JENS-EBND 'T
0.0035
0.0004
U.0039
0.0002
0.0001
0.0016
0.0008
PENS-NRME T2
0.0038
0.0002
0.0042
0.0003
0.0001
0.0016
0.0003
PENS EBNR T2
0.0036
0.0007
0.0042
0.0002
0.0015
0.0004
Variance (in bold) PSE-varients
Models
PSE-RG-L
PSE-RG-SU4
PSE-METEOR
PSE-BLEU
PSE-JSD
PSE-DScore
PSE-InfoLMMqu
BigBird- Pegasus
T.74E-07
1.17E-05
LO5E-06
9.43E-06
111E-06
2.36E-06
5.81E-06
SimCLS
L14E-05
4.86E-05
2.44E-05
2.92E-05
3.64E-06
2.82E-06
3.08E-06
BRIO
1.65-005
5.25E-05
3.07E-05
2.61E-05
9.86E-07
2.75E-06
1,77E-06
PropheuNet
1.09
3.47E-05
1.69E.05
2.71E:05
3.20E. 08
2.39E.06
2.90B 06
T6 (Base)
121E-00
2,43E-06
2.17E-05
9.90E-06
154E-07
2.61E-06
1.33E-06
PENS-NAME TI
II2E--05
1,13E-06
7.51E-06
8.96E
4:00E-09
2.43É-06
1.41B-66
PENS-NRMS TI
8.651E-06
9.84E-08
1099-05
6,14E-07
9-4E-00
2433E066
3.26E-0T
PENS EBNR TI
1.228-05
158E
153E 05
5,20E 08
6.40E-09
2.53E.06
6.00E 07
PENS-NRMS T2
L41E-05
5.44E-08
1.79-005
1.02E-07
1.36-08
2.11E-06
1.14E-07
PENS-EBNR 12
1.303-05
4.94E-07
1.752205
3.76E-08
2.31E-06
1.84E-07
Summary
Models
PSE-RG-L
PSE-RG-SU4
PSE-METEO
PSE-BLEU PSE-JSD
PSE-BScore
PSE-InfoLM-ad
d-stability
0.0041
0.0072
0.0055
0.005-1
0.0019
0.0017
0.0024
Spearman
eKendall
1
Table 6: PerSEval Stability: 0.0072-strongly-stable w.r.t e-Spearman 1; e-Kendall across variants
C Detailed Experimental Results
We provide a detailed analysis of all the seven PerSEval variants in terms of their stability performance in the following section.
C.1 PerSEval Stability Results
In this section, we provide detailed analysis of the stability performance of all the seven PerSEval variants (in Section 8.1 we discussed that of the best performing PerSEval-InfoL.N variant only). We analyze the 8-bias and the d-variance of each variant across all the ten SOTA models that have been studied. We observe that while the best-performing variant w.r.t bias is PerSEval-BertScort and w.r.t variance is PerSEval-RG-L, the worst performances w.r.t both are pretty low with an overall 0.0072 d-stability across all the variants (see Table 6). We also observed a consistent 100% rank-correlation (i.e., e-stability) across all the variants, showing PerSEvalto be extremely stable.
C.2 EGISES and P-Acc Results
28

---

Models
EG-RG-L
EG-RG-SL4
EG-METEOR
EG-BLEU
EG-JSD
GG-BSeere
BG-InfoL.MMog
BigBird- DigiiiddPeggasus
0.324
0.097
0.356
0.128
0.387
0.390
0.33
SimCLS
0.429
0.268
0.501
0.376
0.512
0.472
0.28
BRIO
0.180
0.252
0.581
0.434
0.630
0.572
0.189
Prophet Is (Base)
0.548
0.431
01602
0.501
0.608
0.577
0.289
0.592
0.474
0.642
0.551
0,641
0.619
0.225
PENS-NAME T1
0.569
0.702
0.366
0.798
0.383
0.863
0.575
PENS-NRMS T1
0.891
0.745
0.887
0.832
0.901
0.886
0.654
PENS-EBNR T1
0.935
0.820
0.931
0.892
0.933
0.793
PENS-EBNR T2
0.986
0.958
0.986
PENS-NRMS 0.988
0.908
0.963
0.989
0.824
Table 7: SOTA Responsiveness Benchmarking on Microsoft PENS Dataset w.r.t seven EGISES (EG) variants.
Models
R--c--RG-L P-Ace-RG-SU4
P-Aco-METEOR P-Ace-BLA BLEU
P-Ace-JSD
P-Ace- BScore
P-Ace-InfoLM-no
T5 (Base)
0.192
0.591
0.532
0.591
0.111
0.123
0.392
ProphetNot
0.481
0.588
0.368
BRIO
0.468
0.582
0.520
0.593
0.333
0.113
0.344
SimCLS
0.146
0,538
0.500
0,577
0,310
0,107
0,301
BigBird-Pagasus
0.412
0.517
0.438
0.508
0.304
0.119
0,382
PENS-NAMLE Ti
0.391
0.523
0.404
0.516
0.350
0.106
0.522
PENS-NRMS T1
0.385
0.518
0.398
0.512
0.311
0.101
0.510
PENS-EBNR T1
0.383
0.514
0.398
0.508
0.341
0.103
0.512
PENS-NRMS
0.501
0.492
PENS-EANR T2
0.376
0.506
0.300
0.500
0.336
0.102
0.504
Table 8: SOTA Personalized Accuracy Benchmarking on Microsoft PENS Dataset w.r.t P-Accuracy (P-Acc); P-Acc parameeerm 0.5, B 1 (i.e., complete importance to lack of personalization).
Survey Format: Human-Judgment Meta-evaluation of PerSEval
In this section, we present the screenshot of the questionnaire designed for the survey for computing the human-judgment version of PerSEval (PerSEval-HJJ). Two consecutive respondents evaluated the generated summary pairs of all ten benchmarked models.
29

---

Evaluation Metric Correlation Survey
below
inpiione
Your geodle
Irangeede
Undogaatcceeet O tue stuken O Tearke Depesdoo ost
Sentence necdard trawet tirest bade hole Hidey
Sentencc noddadd bhrned M0leal bedippt tole Indky taber hevebose lead ovst Sumar champ jastnrow eipebble brach.
cary mocloodd bedk
beachs
13
pn 2000
Sendence woedard trined S0 hole
woed
tilka the four venn, nested a Sivcii burr wood and becs et mars apee poia in play prpplabee Delar's
pury mooo le
farmer sheesuuut just
Sentwoor wosetrrd mncte ciderd do
sack thar second ouedder of parar becomes men plaper poia in open bo as beach
bincl
fod over famen
peitle braal four tewe
Sovtoce'' wooola clanedt barii
cap on or and puic twr us loci onoe no champee ein rrte wout bocue po too playye to pooe ih tomm olue poblit
Sentence craiesd 5D
totets under
Yow trve enaccret mene 12 to sice the sttiie rnd four teme mesor ver very molrey and aefican bor americee socce wase and uadde
Sentecce pary noodle craa
ous under woodlrd tiicua abroks mujer
soou
mave jooti sctitdl bouth,
sound eeplccrre
Sentico va- atole
pelblle beach liv ardee
champaon lucte noeat pabble beach, thrnk wtneer inthe noo toun-eth iet lost sear zoen champloaat at halivee sayr bestrnge major are stah- has
the rinck, nantted diven Salveay
marugee given
Nacta man dasiluursily moszer perpererppppo Yoe sntionaly
addedts committerr
Figure 4: Sample Questionnaire: Six pairs of summaries for specific document; five pairs model- generated summarics (each user evaluates five of the ten models) for specific document, while one pair is user-generated gold reference).
30


Reviews:
[{"review_id": "8xMwVmlPS0", "reviewer": "Reviewer_j1ks", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n•\tThe necessary explanations provided as to why  EGISES and P-Acc fail to provide a reliable measure of personalization, justifying the necessity and motivation of this paper.\n\n•\tThe thoroughness of including Accuracy-inconsistency Penalty (ACP) to account for corner cases where the best performance can be an outlier. Well done.\n\n•\tThe evaluation framework that comprehensively assesses the reliability, stability, etc of  PerSEval, also involving human review is commendable.\n\n•\tThe use of surrogate personalized summaries across multiple datasets to compensate for the limitations of dataset similar to PENS.\n\nWeaknesses\n\n•\tOverall the paper is well-written, logical, and clear. I also read authors' answers on reviews and don't see big concerns. One thing that authors may consider adding is the user guide for applying this metric in different contexts, i.e. would there be any differences in optimizing PerSEval during model trainings comparing to traditional metrics?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper starts by calling out neither traditional accuracies metrics nor EGISES proposed by recent study are sufficient enough to evaluate degree of personalization of summarizers. The authors empirically proved EGISES measures only the degree of responsiveness and illustrated with examples that EGISES scores can stay unchanged when accuracies vary through experiments. To measure both responsiveness and accuracies, the authors introduced a new evaluation metric called PerSEval, which addressed the limitations of both EGISES and Accuracy with the help of a penalty factor called EDP. The authors also empirically proves that PerSEval shows high correlations to human judgement comparing to other metrics through multiple robust experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "•\tDiscuss the advantages of using PerSEval over other metrics in various contexts and use cases, and the impact of this new metric in this field. This will help readers understand the significance of this paper.", "broader_impact_concerns": "Broader impact concerns are not identified from this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i47ZQWraPp", "reviewer": "Reviewer_hE9R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. It is reasonable to distinguish responsiveness from personalization and then identify the shortcoming of the existing EGISES measure. This is well motivated that there indeed can be scenarios where a model exhibits high responsiveness at the cost of losing accuracy. Under such conditions, the EGISES measure could be risky.\n\n2. This paper theoretically and empirically show that the propose  PerSEval measures the degree of responsiveness, a necessary but not sufficient condition for degree-of-personalization. It makes sense to conclude that the EGISES-paradox not just theoretically exists but has real evidence. \n\n3. The experiments are generally informative and convincing. Two ten SOTA summarization models were assessed. Based on the results, this paper empirically establishes that PerSEval is reliable w.r.t human-judgment correlation (Pearson’s r = 0.73; Spearman’s ρ = 0.62; Kendall’s τ = 0.42). PerSEval has high rank-stability and PerSEval as a rank measure is not entailed by EGISES-based ranking. In addition, PerSEval can be a standalone rank-measure without the need of any aggregated ranking\n\nWeaknesses:\n\nThis paper is overall comprehensive and well written. I did not find major concerns. While this paper seems to be generally sound, the research scope is somewhat narrow. As the authors have already argued that they have not found any containing user behavior history, such as user-click timestamp records, other than the news dataset, the generality could be limited. On the one hand, it seems hard to collect the data with user behavior history in reality. On the other hand, this work is a correction for a specific metric. It remains unclear how this work could facilitate related studies and inspire future work. Any thoughts on those aspects would be beneficial. \n\nMinor Comments:\n\nThere is a reference mistake in the first paragraph (Ln 6) of Section 1, which shows \"?\". Besides, the authors may double check the format of the references and use \\citet{} and \\citep{} correctly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on assessing personalization in text summarizers. Concretely, this paper proposes PerSEval, a corrective measure for EGISES proposed by Vansh et al. (2023). By introducing the concept of responsiveness as opposed to personalization, this paper finds that EGISES only measures the ability of discerning the differences in reader profiles (i.e., reading histories). However, it is also necessary to generate reader-specific summaries that maintain this difference proportionately. One cannot expect high user experience while having low accuracy. This paper then mathematically proves that a model can have a very good EGISES score but may still fall short of being personalized simply because of low accuracy performance. As a result, this work proposes PerSEval that builds on the design principles of EGISES and bridges the gap of user experience. Analysis results show that  PerSEval provides a much more reliable ranking of models with significantly higher human-judgment correlation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Justify the significance of assessing personalization in general text summarization problems.\n\n2. Fix the reference mistakes.", "broader_impact_concerns": "The broader impact of this paper has not been clearly justified yet.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0D5xSXGHDS", "reviewer": "Reviewer_bX5T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts:\n* The argumentation of requiring an accuracy factor next to an accuracy-independent personalization score (or the defined responsiveness) is sensible\n* The conducted evaluation is significantly aligned to the one for EGISES, comprising the same SOTA baseline summarizers as well as the same sampling tests for robustness.\n* The conducted evaluation comprises a user study with a rather large set of participants, where a sensible approach is chosen to assess the approach's capability to replicate human judement.\n* The results concerning the individual performance of PerSEval seem thorough, providing a sensible ranking and showing sufficient correlation to human judgements.\n\n# Weaknesses:\n* A main concern I have on the paper is the missing mention and comparison to the combined personalization plus accuracy metric the authors of EGISES suggest at the end of their paper, namely P-Acc. It therefore remains unclear from the paper what the main novelty is. \n* The evaluation only consists of one direct comparison to EGISES, where the correlation is investigated. A deeper analysis, also with respect to human judgements is missing. It is unclear which method the participants would prefer.\n* Similarly, a deep analysis of Table 2 results, comprising the human judgement evaluation, is missing. Besides mentioning the best performing PerSEval-InfoLM-αβ, no insights are drawn with respect to the used distance metrics which do not perform as well.\n* The related work touches on works RecSys metric using human feedback which is ruled of for text sumarizers. I understand that available datasets might be generated in a static, offline manner, prohibiting a thorough evaluation. What is not clear to me are the boundaries of the taken approach when defining personalization on the history of summaries. \n* Since human feedback based summarizers can gradually improve a summary, an argumentation is missing why one would use the proposed metric on such approaches within the human judgement meta evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension to existing personalized text summarization benchmark EGISES, enriching it with an additional accuracy criterion. The new method is referred to as PerSEval. The authors explain and provide a proof for a shortcoming of EGISES, where a summary can be ranked high in personalization while having low accuracy, resulting in a paradox. The baseline of the argumentation is that EGISES not measures personalization (which would imply high user experience), but a weaker concept referred to as responsiveness. The novel criterion focusses on penalizing low accuracy on top of EGISES, while making sure that high accuracy does not deacrease the personalization. More specifically, the authors add a accuracy drop penalty to ensure sufficiently high accuracy and accuracy-inconsistency penalty to prevent outliers in accuracy scores. \n\nThe approach is evaluated for the PENS dataset, following the EGISES baseline paper. The ranking is calculated for the same set of SOTA summarization models. For the evaluation, 7 metrics, including ROUGE (RG)-L, Meteor and BertScore, are used as distance metric for the proposed score. The results show that their variant PerSEval-InfoLM-αβ provides the best discriminative performance.\n\nIn addition, a meta evaluation was conducted with 169 students in order to find out if PerSEval comes close to human evaluations. The authors therefore had the participants compare summarized sentences in terms of their similarity. PerSEval-InfoLM-αβ seems to again provide results closest to human judments.\n\nThe authors show also show that their approach is not perfectly correlated with EGISES by comparing their outputs and conduct a final evaluation to assess the stability of resulting ranks by sampling different amounts of summaries from PENS. The authors infer that the proposed approach is stable and robust.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the possible, perceived weaknesses, I would propose the following changes:\n* Please add a distinct discussion of the difference of P-Acc to EGISES. Optimally this should be worked in to the introduction and main argumentation for the paper as well, as this is the main piece of related work here. Does P-Acc potentially not solve the problem and one needs a better metric? Can we still make the triangulation proof on the basis of P-Acc? Is the ranking of P-Acc inferior to PerSEval? Until this is clarified, I would not agree that claims and evidence are clear.\n* Would it be able to infer more information from the human judgement experient you conducted which would allow to compare EGISES / P-Acc / PerSEval on further? If so, please add such information or provide argumentation why this not possible / not needed at all.\n* Why are other variants of your system not as competing as PerSEval-InfoLM-αβ? Please add more discussions on your results, helping the reader to take informed design decisions.\n* Please add a discussion on the boundaies of such a crafted metric to a learned metric, which is not possible to be learned without feedback data. To what extent does the defined personalization property hold?\n* The presentation of the paper does not include sufficient examples for the accuracy measure for the claimed difference to personalization. Maybe this is too basic, but I am missing a better example on what accuracy means for a summary - is it false statements, leaving out central parts of the content or both? Could a user not want to trade in accuracy (in terms of removing some parts of the content) for a better focus on a subtopic of interest? Where are the boundaries here, what are the core assumptions? How were they taken?", "broader_impact_concerns": "Broader impact concerns are available in the paper. Since the concept personalization vs. responsiveness vs. UX vs. accuracy is somewhat subjective and philosophical, it might be beneficial to add such a discussion to the section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YpQO3PojzQ", "reviewer": "Reviewer_muUC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.The authors find a gap in the evaluation of text personalization models. The PerSEval tries to distinguish between responsiveness and personalization in this context. It looks like a good new metric in this field.\n\n2. PerSEval incorporates an **accuracy penalty** into the evaluation. lt offers a more comprehensive measure of how well a model personalizes text.\n\n3. The authors how an thorough empirical analysis with a range of state-of-the-art models. It shows that this new method has a high reliability and rank stability.\n\n4. I believe this paper and metric potentially influences future research in the field.\n\nWeaknesses:\n\n1. Too few dataset: The analysis only uses PENS dataset. I am not convinced that it will also work on other datasets.\n\n2. The paper doesn't explore how alternative measures to those chosen for empirical testing perform regarding the human-judgement correlation for PerSEval.\n\n3. How to train model which can optimize for PerSEval, given its complexity?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces **PerSEval**, a **metric** for evaluating the **personalization of text summarization** models. The authors find that the most popular personalization evaluation metric --- **EGISES** --- only measures the degree of responsiveness instead of the actual personalization. To better eval the personalization, the authors propose PerSEval. It adds a penalty for accuracy drop into EGISES to create a more personalized evaluation. The authors test PerSEval using ten state-of-the-art summarization models and find it to be very reliable, with a high correlation with human judgment and high rank stability. They also demonstrate that relying only on accuracy-based aggregated results can lead to misleading conclusions. The authors argue that the PerSEval offers a more comprehensive evaluation of personalization in text summarizers, potentially improving user experience.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The examples in page 3 (Alice) is not clear and confusing.\n\n2. Other datasets results are needed to prove reliability. I am not convinced by its usefulness.\n\n3. Discuss the computational implications of using PerSEval, such as the additional resources required due to the complexity of the method.", "broader_impact_concerns": "1.  the use of personal user data", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to improve improve evaluation of summarization, especially personalization aspect. The authors introduce a new metric PerSEval which is compared on 10+ state-of-the-art models and show high human-judgment correlation and rank stability, suggesting that the proposed metric is a more comprehensive and reliable measure of personalized summarization. The reviewers were leaning positive but had some concerns about (1) limited comparison with P-Acc, (2) dataset diversity (3) lack of computational considerations. In the revised version post rebuttals, the authors responded to all major concerns raised by the reviewers. They added more datasets to address concerns about the generality of their results and provided additional discussions on the limitations of P-Acc, showing that it cannot fully solve the issues related to personalization. As most of these concerns were resolved, it makes the paper ready for publication.
Decision: Accept (poster)