Generating Summaries with Controllable Readability Levels
Authors: Leonardo F. R. Ribeiro
Mohit Bansal
Markus Dreyer
Venue: EMNLP 2023 Main (2023)
Keywords: readability
abstractive summarization
controllable text generation
TL;DR: We propose techniques to generate summaries at a specified reader’s level using fine-grained control

Abstract: Readability refers to how easily a reader can understand a written text. Several factors affect the readability level, such as the complexity of the text, its subject matter, and the reader's background knowledge. Generating summaries based on different readability levels is critical for enabling knowledge consumption by diverse audiences. However, current text generation approaches lack refined control, resulting in texts that are not customized to readers' proficiency levels. In this work, we bridge this gap and study techniques to generate summaries at specified readability levels. Unlike previous methods that focus on a specific readability level (e.g., lay summarization), we generate summaries with fine-grained control over their readability. We develop three text generation techniques for controlling readability: (1) instruction-based readability control, (2) reinforcement learning to minimize the gap between requested and observed readability and (3) a decoding approach that uses lookahead to estimate the readability of upcoming decoding steps. We show that our generation methods significantly improve readability control on news summarization (CNN/DM dataset), as measured by various readability metrics and human judgement, establishing strong baselines for controllable readability in summarization.

study. Generating Summaries with Controllable Readability Levels
Leonardo F. R. Ribeiro
Mohit Bansal i+
Markus Dreyer'
Amazon UNC Chapel Hill
(leonribe, mobansal, mddreyer)@amazoon. com
Abstract
Document: Children may be drawn to greasy fries and burpers, but fatty foods reduce the speed which their braint work according foomd children who ste diet higher saturated fats and cholestrrll had slower reaction times and a poorer working memory. Children who alc the fatty performed worse
Readability refers how easily reader can understand a written text. Several factors affect the readability level, such as the complexity of the text, its subject matter, and the reader's ' background knowledge. Generating summaries based on different readability levels criti- cal for enabling knowledge consumption by diverse audiences. However, current text gener- ation approaches lack refined control, resulting in texts that are not customized to readers' pro- ficiency levels. In this work, we bridge this gap and study techniques to generate summaries at specified readability levels. Unlike previ- ous methods that focus on a specific readabil- ity level (e.g., lay summarization), we gener- ate summaries with fine-grained control over their readability. We develop three text gen- eration techniques for controlling readability: (I) instruction-based readability control, (2) re- inforcement learning minimize the gap be- tween requested and observed readability and (3) decoding approach that uses lookahead estimate the readability of upcoming decod- ing steps. We show that our generation meth- ods significantly improve readability control news summarization (CNN/DM dataset), as measured by various readability metrics and hu- man judgement, establishing strong baselines for controllable readability in summarization.
they given taok-switching game compleie, the resoarchers said, l...] For the sludy, ackentists at the University of Illinots recruited 150 children aged between
and 10 and gave them game invalved learning a patten between a set shapes colours. The gamee was designed to test cognitive flexbiiity [...]
Summarize with readabiliiy level 90: Falty foods can reduce the speed at which children's is besine work Children who ate diet high in performed worse a game. This is because disrupis way muscles break down sngar in the blood.
Summarrize with readabilitt level had slower reaction times They performed worse when given trek-switching game complete. Researchers at Virginia lech claim that cating food for just five days can changy the body's metsholiem for the werse
who fatty and high
Summarize with reddbilitt children who ate diet higher in saturated fals and cholesterol had slower nuttion tirres and a warking memory. Childnen were given involring learning patter between of shapes colours to complese, which tested their cognitive flexibiiity
University Mllinois reseurchers found
Figure I: Summaries generated with different readabil- ity levels using our lookahead method (Sec. 3.3). We requested summaries with Flesch Reading Ease (FRE) readabilities of 90, 70, 30 (corresponding to the levels of II -yeear-old, middle-school and college, respectively). The readability scores of the generated summaries (87.1, 70.3, and 30.9) are close to the requested targets.
lower readability favors specificity, clarity and ac- curacy (August al., 2023). Therefore, the read- ability of a summary is important to ensure that the information is comprehensible to a wider audience, accommodating varying levels of knowledge and understanding (Pitler and Nenkova, 2008).
1 Introduction
Significant progress has been made in abstrac- tive summarization using large language models (LLMs) (Raffel et al., 2020: Zhang et al., 2020a; Goyal et al., 2022a). This approach involves mak- ing various generation decisions, such as determin- ing which content to paraphrase and how specific summary should be. The goal is to generate high-quality summaries that are cohesive, read- able, and factually consistent. Nevertheless, cur- rent methods provide limited mechanisms spec- ify stylistic preferences such as readability (Goyal et al., 2022b). While readability assessment, which measures the level of difficulty to comprehend a
Summaries convey salient pieces of information and their understanding depends the reader's world and domain knowledge. The readability of a text plays crucial role in how easily it can be understood and consumed for learning and edu- cation. Higher readability lowers reading efforts and increases the speed for any reader, and it is particularly beneficial for those who lack high com- prehension (DuBay, 2004). On the other hand,
'Code/data: https: //gi thub. wtt. http://aannnsccccncccc cnntrollable-rradabil...

---

text, is well-established field within NLP (Feng al., 2010; Vajjala, 2022), the control of read- ability natural language generation (NLG) tasks, such as summarization, has not been extensively ex- plored and current readability control performance is low (Luo et al., 2022; Pu and Demberg, 2023).
readable summaries tend to have lower specificity and informativeness). Our results show consider- able improvements over GPT3.5, a state-of-the-art approach for controlling the readability level (Pu and Demberg, 2023).
2 Task Definition: Summaries with Distinct Readability Levels
While previous work (Goldsack al., 2022; Guo al., 2021; Luo et al.. 2022) focused on binary readability control, such as expert versus lay sum- maries, we instead focus on generating summaries various fine- grreneraind reading grade levels (Todi- rascu et al., 2016; Martinc et al., 2021). Figure 1 presents an example of three summaries with di- verse educational levels of readability generated with our lookahead method (Sec. 3.3), given the same document. While the easier-to-underrtand summary, with the highest Flesch Reading Ease score (Kincaid al., 1975), uses simpler words, shorter sentences, and less specialized knowledge, the summary with the lowest score requires the reader to understand more complex sentence struc- tures and words (e.g., "saturated", "cholesterol", "cognitive") and contains more specific details (e.g.. "University of Illinois researchers"), assuming the readers' familiarity with necessary domain and world knowledge.
2.1 Task Statement
Readability refers to the ease of understanding text, impacting the reader's comprehension and engagement. We aim to generate summaries with specified levels of readability, given an input doc- ument. Let æ (#1..... .nn) denote the input document represented by the sequence of n tokens, and y (y)....,, .mm) denote the summary token sequence of length m, where m < n. Let f de- note the desired summary readability level, which can be represented by a score (Sec. 2.2) or a cat- egory name (e.g., "college level", Sec. 3.1). The following formulation represents this task as an auto-regressive problem:
p f. Il p (1)
We study three methods to control the read- ability of generated summaries: First, we present an instruction-prompiin method that prompts the model to output summaries with particular target readability scores or categories, enabling fine-grained control. Next, develop ap- proach based on reinforcement learning (RL), using Proximal Policy Optimization (PPO; Schul- man et al. (2017)) with a novel Gaussian based re- ward that strongly penalizes significant variations in readability, optimizing for summaries with the desired readability. Finally, inspired by Wan et al. (2023), propose readability-awai lookahead decoding method that selects tokens each decod- ing step based on the readability of expected future token generations.
This task presents challenges due to multiple fac- tors: While the method must determine salient in- formation from the input document and compress it into concise summary y. it must also be able to understand different readability levels and adapt the summarization output match the target level
The approach must strike a balance between providing a suceinet, informative summary and en- suring aligns with the reader's 's literacy skills and background knowledge (Collins- Tholinss,hhooson, 2014; August et al., 2023).
2.2 Readability Metrics
We now discuss metrics to assess text readabil- ity. Generally, readability is affected by lexical and syntactic sophistication, sentence structure, dis- course cohesion, background knowledge and use of technical terms (McNamara et al., 2010; Cross- ley et al., 2023). Readability metrics typically fo- cus syntactic complexity, measuring the pres- ence of qualified syntactic phrases, and vocabulary complexity, aligning words from the text corre- lated with a particular age-related level (Beers and Nagy, 2009; Crosslcy et al., 2017). We explore multiple established metrics to control and evaluate
Our contributions in this paper are: (1) We pro- pose three generation, using instruction- instructio-promptingg RI, and lookahead decoding, and (2) show that readability can be explicitly controlled for generating abstrac- tive summaries with finely adjustable levels. Fi- nally, (3) we explore the relation between summary readability and aspects such as specificity, abstrac- tiveness, factuality and informativeness (e.g., more
methods for text

---

Document: Around 16 million tourists visit Florence every year. Its popularity is not surprising the city contains over sixty per cent of the world's art heritage. [...] But for its population of 350,000, that's an overwhelming figure [...]
Summarize for a high-school student:
Summarize with Flesch score of 60:
Summarize a 11-years-oid:
(a) Instruction
(b) RL
(c) Lookahead
LUM
Lun
AL updale (PPO)
decoding
The city of Florence visited by many people
Prompt-besen Supervised training
60
has ....) 0.8
pi contains ...) 0.5
each year...
FRE Score: 68 60
has more than half of the
contains over sixty cent the world's art hartage...
Florence attracts 16m people every year, loul only 360,000 000 live the city [...]
Every year, approximately 16 milionn tourists visit Floreniee..
world...
Compute reward
Rlesc Reading Ease (FRE) :48 High school level: FRE
FRE score: 84 a 80 V
score: 48
X
FRE Score: 37 <
11-years old level
Figure 2: Overview of the proposed methods. (a) illustrates our approach to control the summary readability via fine-gres grained instructions. (b) shows our RL method where given an input document and the readability level, the policy generates a summary to be scored by our Gaussian-based reward, and (c) shows the our lookahead approach which uses readability score of a future summary to guide the generation.
the readability of the summaries. Specifically, we employ Flesch Reading Ease (FRE, Kincaid et al.. 1975), Gunning fog index (GFI, Gunning. 1952) and Coleman-Liau index (CLI, Coleman and Liau, 1975), among others. 2 Those metries calculate an approximation of the (US) grade level of education expected to understand a written text.
Readability
Instruction
Summarize this II l-year-old student: 60g FRE <80 Summarize this middle school student:
FRE 280
40g FRE <60
Summarize this for high school student:
FRE <40
Summarize this for college student:
Table 1: Category- bassd instructions based on readabil- ity scores (FRE).
FRE and GFI metries are determined by sentence lengths. number of (complex) words, and syllables. Alternatively, unlike syllable-based readability in- dices, CLI does not require that the character con- tent of words be analyzed, only their length in char- acters measures readability. Higher FRE scores de- note higher readability; higher GFI and CLI scores denote lower readability. The metrics formulas are described in Appendix A. Finally, readability formulas may fail to consider significant factors such as cohesiveness and macro-level organization, which affect the overall readability and understand- ing of text (Tanprasert and Kauchak, 202 To account for this, we measure other aspects such as coherence and informativeness.
to generate text with desired attributes, we de- velop instructions that encode the summary read- ability level. During training, the instructions are prepended to the source documents create the in- put for their corresponding summaries. In contrast to recent studies that generate summaries with only two levels (expert and plain language) (Goldsack et al., 2022; Luo et al., 2022), we control the read- ability using finegfnn-ggree frriind instructions based on the desired readability, as shown in Figure 2a.
Category-based Instructions. Drawing on es- tablished guidelines for text complexity lev- els (Fountas and Pinnell. 1999; DuBay, 2004), we define four instructions based distinct reading level categories (see Table 1) aligned with partic- ular FRE scores (Vajjala, 2022). For example, we instruct the model to summarize the input docu- ment "for a high-school student". We perform instruction-based fine-tuning, selecting the one in- struction per training sample that matches the given reading level of the reference summary; that way, the model can learn to associate the observed refer- ence summaries with the categories in the instruc- tion prompts. At inference time, we can select any of the four instructions to request a summary in the style of the specified reading level category. We call this method NATEGOOYINSTRUIC
3 Controllable Methods for Readability
what follows, we present our three approaches that employ readability metrics (Sec. 2.2) for controllable summary generation, based on fine- grained instructions (Sec. 3.1), reinforcement learning (RL, See. 3.2), and lookahead decoding (Sec. 3.3). Figure 2 illustrates the methods.
3.1 Instruction-Aliguing Readability Methods Inspired by previous works (He et al., 2022; Zhang and Song, 2022) that explore prompt guidance AAppendix A presents additional readahility metrics.

---

Score-based Instructions. We define second instruction-basee method, which we call ScoRE- INSTrUcT. Here, we instruct the model to summa- rize with a particular score f, rather than a category. For example, we instruct the model to summarize the imput document "with readability level of 62". In supervised fine-tuning, we use as f in the instruc- tion the exact score of each reference summary; this way, the model can learn to associate the observed summaries with the exact reading levels specified in the instructions. At inference time, we can re- quest any score f. Compared to CATEGORYINSTRUCT, this method adds flexibility by avoiding the hard boundaries presented by readability categories (Ta- ble I); a training sample whose reference summary falls between two reading level categories (e.g.. FRE score of 60 is at the boundary between high- school and middle-school level) does not need to be forced into one or the other category.
ure 2b, we formulate it as a normalized Gaussian centered at é:
f f= - o²) R(fy: -
(2)
(3)
The use of Gaussian function ensures that the reward decreases in a nonlinear fashion: Small readability deviations from the requested readabil- ity result in small reward reductions; larger read- ability deviations result in disproportionally larger reward reductions. This is analogous to the nonlin- ear penalties for deviations from Gaussian prior in L2 weight regularization.
We apply RL using the described reward, aiming for the model to learn improved readability control. At the same time, we wish to preserve the existing high salience and coherence achieved by the summarization models tuned with super- vised fine-tuning. To accomplish this, we initialize the RL policy with ScoreInstruct that has been trained on supervised data. We then employ the popular policy gradient method PPO (Schulman et al., 2017), which has been successfully applied to text generation problems (Liu et al., 2022a) and has been shown to be sample efficient and stable (Wu et al., 2021). To ensure stability, PPO employs clipping mechanism during policy updates, pre- venting drastic changes and avoiding the problem of diverging policy updates. We call the resulting PPO-tuned model SCOREINSTRUCT-RL.
PPO.
3.2 Reinforcement Learning for Readability Control
During supervised instruction fine-tuning, using INTEGORYINSTRUCT or ScoREINSTRUCT, we request certain readability levels in the prompts (i.e., read- ability categories or scores), and the reference sum- maries act as demonstrations of summaries of that readability level for the model to learn. However, that supervised learning phase does not explicitly check if the model indeed tends generate sum- maries in the requested reading levels. It merely uses token-wise gradient updates using teacher forcing based on the reference summaries.
3.3 Lookahead Readability Decoding
Ouyang et al. (2022) have shown that it can be helpful to supplement such an initial instruction fine-tuning approach with subsequent reinforce- ment learning phase, in which the model is fur- ther updated in response to sequence-based rewards from a reward model. contrast to Ouyang et al. (2022), who learned a reward model based on hu- man preferences, we define a reward function based on a readability metric. Intuitively, we want to re- ward the model maximally whenever it generates a summary that has the requested readability level f and decrease the reward steeply for generated summaries whose readability deviates from
As a direct consequence of RL training, the model explicitly learns to generate tokens with the goal of maximizing the readability reward. However, RL requires a initialization with a high-quality super- vised model, which might not always be available. Consequently, we explore decoding approach to dynamically adapt the readability level during inference, as shown in Figure 2c.
Previous work (Lu et al., 2022; Wan et al., 2023) develop lookahead approaches for controlling gen- eration through signals such as logic-based lexical constraints or faithfulness. We extend this con- cept to enhance readability control in abstractive summarization at inference time. We develop a lookahead strategy that directs the model gener- ation process, increasing the likelihood that the chosen tokens will lead to a path with the intended level of readability in the search space. Formally,
We design a reward R(fy, f) that as- signs a maximum reward of 1.0 if the observed readability Fy of a generated summary y is equal to the desired readability f and decreases exponen- tially as Fy deviates from f. As illustrated in Fig-
Reward.

---

each summary token yi is selected by:
FREA
FRE,T
GFI,L
CLIsl
GPT3.5 (TENT-DAVINCI-003)
0.24
-0.28
g3y)) log ply: yıs 1, 1,
Best GP3-5 (TEXT- DAVINC--VIC)-
18.4
0.59
-0.50
-0.45
(4)
CategoRYINStRUCT
15.8
0.39
-0.37
-0.55
+ max (yy::-1tss Ln
SCOREINSTRUCT
SCOREINSTRUCT-R
11.8
0.74
-0.42
-0.72
where h(.) function that assigns readability score to the summary and w controls the weight of the readability in the generation process. Ly is a set of possible future summaries that start with the tokens V1 and contains n additional continua- tion tokens likely to be encountered in the future. Our readability evaluation function h is defined as:
SCOREINSTRUCT+LA
4.5
-0.67
Table 2: Readability results for the proposed methods using different automatic metries. A is the absolute difference between the desired and generated FRE read- ability. P is the correlation with the target metric. All correlations statistically significant (p<0.05).
model without readability control. As an additional baseline, which call Best-GPT3.5, we sample k summarics from GPT3.5 for each instruction and select the summary whose readability level is clos- est the requested level.
hyy, 1 - F1
(5)
where Fy is the observed readability of the (possi- bly incomplete) summary y. h(() can be defined with additional metrics which can be combined to desired scores (see Sec. 5.4). Finally, note that this method is computational costly since it needs to generate future summaries per generation step.
We evaluate the readability with the FRE, GF and CLI metrics (Sec. 2.2). We use BertScore (BS, Zhang et al., 2020b) and the Fl measure of ROUGE-L (RG-L, Lin, 2004) for evaluating the summary quality; and FactCC (Kryscinski et al., 2020) and UniEval (Zhong et al., 2022) for faith- fulness assessment.
4 Experimental Setup
We evaluate on CNN/DM (Hermann et al., 2015), which contains news articles and their bullet-point highlights considered as summaries. In line with previous work (Guo et al., 2023) where sum- maries have similar or easier readability level than the input document, we assess the subset of the CNN/DM test set with documents with a FRE score below 50 (high school and college student levels).
Results and Analysis
5.1 Main Readability Results
Table 2 shows the results on CNN/DM summaries, where we compute the absolute difference between the desired and generated PRE readability (A) and Pearson correlations with readability metrics. In comparison to our methods, GPT3.5 summaries are considerably distant from the intended read- ing levels, while Best-GPT3.5 greatly improves over GPT3.5. SCOREINSTRUCT has better perfor- mance than CATEGORY suggesting that fine-grained readability signals are beneficial dur- ing training. ScoREINSTRUCT+RL enhances readabil- ity control significantly, while ScoRELnStRUCT+LA further improves all readability metrics.
'he readability of the summaries are computed using FRE metric and used during training for con- structing the instructions for CATEGORy INSTRUCT and ScoreInstruct, and as the desired readability in ScoReINSTRUCTRRL and ScoreINSTRUCT+LA For INSTRUT,, we map the FRE score to the instruction for each reference summary as shown in Table 1. In SCOREINSTRUCT++U we randomly sam- ple FRE scores which are used in the instruction to generate the summary and as f in the reward. We define a by drawing inspiration from the readabil- ity levels (see Table 1), with each level covering a range of 20 FRE points. We set the standard devia- tion a to half of that value (i.e., 10), so that more than two thirds of the reward values lie within the requested reading level centered at f. In SCORE- INSTRUCT4LA, the number of tokens considered in future is set 20.
Table 3 presents the detailed results by reading level. 4 To compare ScoReINstRuCt's methods with GPT3.5 and CATEGORYINSTRUCT, we used prompts with the specific values for each readability level (30, 50, 70 and 90). Both instruction methods gen- erates summaries with granular readability mea- sured by the readability metrics, while maintain- ing the summary quality close to the baseline.
Our models are initialized with Flan-T5- Large (Chung et al., 2022) and fine-tuned us- ing a context size of 1024. We include as base- lines GPT3.5 (TEXT-DAVINCI-003) and a Flan-T5
For ScoREINSTRUCT+RL. the largest gains are for the 1l -edar-ood level, improving FRE from 67.6 to
"areee-arammeers and prompts are found Appendix B. *Other readability metrics are presented in Appendix D.

---

100
20
Flasch
2
conat
mdlde scred capietee chap?
53
Categoryinstruct
bi Scorelnstruct
c) Sccorinssruct RL
Scoreinstruct LA
Figure 3: Readability scores for summaries generated from CNN/DM test set. The stars indicate the requested readability level.
Readability (fy)
Quality
mary types significantly less than our methods. Recently, Pu and Demberg (2023) found that the proficiency GPT3.5 in adapting the readabil- ity of summaries is behind human-written texts.
Readability (f)
FRE FREA GFI CLI,
BS
RG-L
No control
50.1
10.9
11.6
0.881
38.73
GPT3.5
(TENT-D-VINCI-00))
I1-year-old
44.2
45.7
13.3
12.3
0.872
31.07
Conversely, while selecting a summary with the
Middle school
41.5
28.4
14.0
12.6
0.872
31.38
High school
39.1
12.3
12,9
31.18
closest readability level from the sampled GPT3.5
College student
37.0
10.3
15.3
13.2
0.870
31.19
summaries improves the performance, it is time-
Best-GPT3.5
(TEXT-DANUNCI-00)
consuming and costly.
11 11-year-old
50.9
39.0
12.3
11.2
0.871
31.41
Middle school
47.8
22.2
13.1
11.7
0.871
31.74
Figure visualizes the variance of our ap- proaches for each readability level, according to FRE. while Figure 4a gives an overview of the re- lation between observed and requested readability for SCOCEIN methods. Generally, while the instruction models distinguish summaries
High school
43.9
14.0
12.2 0.870
31.57
College student
33.4
15.9
13.7
0.868
30.69
CATRGORYINSTRUCT
11-year-old
65.7
24.3
8.5
9.1 0.879
36.87
Middle school
56.4
14.6
10.6
0.882
Hlgh school
48.7
10.9
11,9
0.882
38.67
College
40.9
14.6
11.7
13.3 0.881
37.72
SCOREINSTRUCT
different readability degrees, ScoreInstRuCT+RI. is
(11-yarrood)
67.6
22.5
0.878
35.38
70 (Middle school)
57.7
13.6
9.6
10.4
0.882
38.34
more effective telling apart the defined levels,
50 (High school)
48.4
9.5
10.8
12.0
0.882
38.64
while SCrRELN the approach with less variance and stronger fine-grained control.
(College)
41.1
14.7
11.5
13.3
0.881
37.09
SCOREINSTRECT-RU
90 (1 (11-ycar-old)
77.8
13.3
6.7
6,9 0.856
28.60
70 (Middle school)
64.7
9.0
8.5
9.3
0.868
33.21
5.2
Impact of Other Summary Dimensions
(High school)
53.5
9.7
11.3 0.873
34.97
30 (College)
43.8
15.6
10.6
13.0
0.873
34.76
Specificity. Summaries can differ in the level
ACOREINTTRUCT+LA
details they convey, which can range from being highly specific to more general. While easier sen- tences are less detailed and use simple vocabu- lary (Laban et al., 2021), casual language (Pitcher et al., 2022), and concise sentences (Scarton et al., 2018), specific sentences are more likely contain specific words and details. Following Goyal et al. (2022b), we measure the degree of specificity of the generated summaries using Speciteller tool (Li and Nenkova, 2015). Interestingly, we observe Figure 4b that casy-to-read texts are less specific, while summaries with higher readability are more specific, demonstrating that our methods enable summaries with different specificity degrees.
90 (11- (ey-yerr-odd)
83.2
7.1
6.6
6.3 0.868
30.75
(Middle school)
67.1
50 (llggh school)
49.2
2.9
10.7
11.9
0.878
36.01
30 (College)
31.9
4.0
12.5
14.8 0.874 32.66
Table Detailed results on CNN/DM using Instruction- prompting, RL and Lookahead (LA) methods. Higher FRE (f) denotes higher readability, while lower GFI with decrease on RG-L. SCORFINSTRUCT-LA. is very effective further decreasing A over all reading lev- els. On the other hand, the high computational expense decoding (generating future summaries) is disadvantage of this method. While the readabil- ity of GPT3.5-generated expert-style summaries is lower than ansier-to-undersee summaries, the readability variation between the different sum-
and CLI (L) denote higher readability.
77.8. However, the summary quality is affected
A Abstractiveness. Abstractiveness quantifies the extent of rephrasing in generated text, indicating
Future work will consider to distillate into model desired readability levels generated via lookahead (Wan et al., 2023).
"The summaries are segmented into sentences, and the macro-average of sentence-lovel specificity is calculated.

---

100
0.55
0.45
Olberrad llestabiity
80
0.50
Aostrectiveren
80
0.40
0.35
40-
0.35
0.25
20
60
50
100
11yearold
Widdlesshoo High-scholl
college
11-yaarold
Midde-schhol High-scholl
college
Requested Readabiliy
Reatabiltty Level
Readabillt Level
No control
Categoryhoompt
ScorePrompt
ScoreProngt-HH.
Scorwhompte-A
Figure 4: (a) Relation between observed Fy versus requested f. (b) Specificity and (c) Abstractiveness of the CNN/DM summaries based on four readability levels.
Readability (F)
Readability
Informativeness
0x
Summarry leroah
Reference (No control)
16.10
2.40
28.02
2.11
Summay Sentence
Baseline control)
18.45
30.20
2.11
CATEGORYINSTRUCT
11-year-old
26.97
2.16
25.55
2.16
Middle schonl
23.81
2.32
28.60
2.20
High school
17.94
2.33
29.14
2.10
College student
17.37
30.14
2.09
1ayarres
w3derzzed Higviee
caiig
Luyaardd
Highere
SCOREINSTRUCT
Beadatally Level
Beadsbilly
90 (11-ycar-odd)
33.39
21.19
2.20
No control
Categoryhroptt
ScorePramgt--B
70 (Middle school)
23.30
2.27
29:00
2.29
Roforence
Scorefrompt
ScorePrampt:
50 (High school)
22.83
2.45
31.88
2.22
30 (College student)
24.59
2.21
Figure 5: Summary sentence lengths (left) and summary lengths (right) generated by the proposed approaches for different readability levels, compared to reference summaries of the corresponding levels.
SCOREINSTRUCT--L.
90 (11-ycar-od)
34.71
2.39
07.93
3.47
70 (Middle school)
31.13
2.21
21.75
2.22
50 (High school)
29.46
2.26
23.69
2.18
(College student)
2.28
90 (11-ycar-odd
30.41
16.28
2.38
the level to which the words are not directly taken from the input. We employ MINT (Dreyer et al., 2023) to measure the abstractiveness based laps between the input document and the summary. As shown in Figure 4c, easier summaries are more abstractive than more complex summaries, and college-level summaries are slightly more abstrac- tive. We hypothesize that this because most of CNN/DM documents have readability levels in the range of high school, making more likely that the model copy parts from the source with similar level, producing more extractive summaries.
70 (Middle school)
27.91
2.29
25.03
2.24
50 (High school)
21.30
2.27
27.46
2.11
(College student))
2.23
Table Human evaluation readability (μ+) and in- formativeness (μ²) of CNN/DM summaries. and a; are the corresponding standard deviations.
Case Study. Table 5 shows examples of sum- maries with different readability levels generated by SCOREINSTRUCT+LA Note that the observed FRF score decreases as the target FRE decreases. Summaries with lowest readability have more spe- cific words (e.g., "defeated", "contest") and pro- vide more details (e.g., "El Clasico", "Brazilian"), Table in the Appendix presents examples gener- ated by the other methods
Summary Lengths. FRE sensitive to text lengths (Tanprasert and Kauchak, 2021) and hence we check whether simpler summaries are just shorter while more difficult summaries are longer. Optimizing toward higher FRE scores leads to sum- maries that contain words that are shorter and easier read and sentences that are shorter.? 7 As shown in Figure 5, we do not observe very short or long summaries overall. Finally, most methods generate summaries that are similar in length to the (training set) reference summaries, while the RI. method detaches the model from such length constraints.
5.3 Human Evaluation
We conduct human evaluations determine how readable and informative the CNN/DM generated summaries are, using Amazon Mechanical Turk. Details on setup, annotation instructions and fair compensation are described in Appendix E.
We usk to select the most readable and the least readable summaries among three displayed sum-
'Rcadability scores generated using https: h/pp:://gii thub. com/cdi com/ccdimscciopyy- readaiiitt.aaiiiiity--...i.. metricc.....
AAppendix F shows summaries with distinct levels.

---

FRE: GFT CLT)
Document: Team-mates Neymar and Dani Alves proved their dedication to Barcelona by suppottin the club's basketball side. Neymur Alves headed to watch Clasico Thursday night alongside the Brazilian's sister Rafaella, Burca prevatled with narrow 85-80 victory in the Euro Lague contest. Bruzil stur Neymar takes a selfie with friends and Barcelonu team-mute Dani Alves However Real Madrid remain top their Euro League division over their bitter rivals, just by points difference Neymar's 's sister Rafaella headed to watch El Clasico of baskethall with the Barcelona forward
45.7
11.1
10.5
Requested FRE Real Madrid and Barcelona played basketball on Thursday night. Barca won the game 85-80, but Real are of the Euro League by points. Neymar and his sister Rafaella went watch the game with friends.
77.1
5.9
Requested FRE 70: Baroelona beat Real Madrid 85-80 in their Euro League basketball clash Neymar's is sister Rafaella joined game on Thursday. Real Madrid
65.9
9.3
8.0
him and friends
lop of their division, by points difference.
Requested FRE S0: Barcelona defented Real Madrid 85-80 EI Clasico on Thurdday night. Neymar und his Barcelona teum-mates to watch baskethull with his sister Rafaella. Real remuin top the Furo I Leugue table Rurcelona by just points.
50.2
9.8
7.1
Requested FRE 30: Neymur and his Burcelona team-mates attended Clasico basketball game. Barcelona defeated Real Madrid 85-80 in the Euro League contest, The Brazilion forward's 's sister Rafaella also attended the game.
33.1
12.0
9.3
Table 5: Examples of summaries of different readability levels generated using SCORFINSTRUCTHLA.
Readability (f)
FactCC1
UniEval
the effectiveness of our methods, while informa- tiveness is negatively correlated with readability.
Consistencyy Coherencet
SCOREINSTRUCT
I-year-ood
0.63
0,918
0.919
5.4
Factual Consistency and Coherence
Middle school
0.65
0,923
0,924
High school
0.66
0.929
0.931
Factual consistency and coherence are crucial as- pects in abstractive summarization. However, en- suring the factuality and coherence while control- ling for readability is challenging. Table 6 shows that in most cases easy-to-read summaries, which are more abstractive (Figure 4c), are less factual and coherent. Previous work (Ladhak et al., 2022; Dreyer et al., 2023) show that factuality decays with increasing abstractiveness. Note that we ex- plicitly used readability signals to control the mod- els towards generating summaries with different readability degrees. However, a high-quality sum- mary should also be faithful and contain relevant information, traits that may not be captured by read- ability metries alone. Additionally, a signal based only on readability raises the risk of degenerate solutions, which might result on non-coherent sum- maries. We are interested in exploring whether factuality metrics in combination with readability affects this dynamics. As shown in Table 7, we adapt the RL and LA methods with linear com- bination of BS-Fact (BERTScore precision of summary with respect the source document) and FRE readability scores. Note that when optimizing only for readability, FRE A is lower but factual consistency and RG-L are the worst, and using BS- Fact tends to improve the metrics while reducing the readability control. Finally, Table 8 presents the impact of considered future tokens in LA, showing that looking ahead for more tokens in fact improves factuality while decreasing FRE A.
College student
0.65
0.933
0,931
SCOREINSTRUCT+RL
le-yearoold
0.58
0.913
0.919
Middle school
0.62
0,925
0.936
High school
0.64
0.936
0.941
College student
0.62
0.939
0.940
SCOREINSTRUCT+LA
ll-year-old
0.54
0.851
0,801
Middle school
0.59
0.895
0.870
High school
0.59
0.911
0.896
College student
0.54
0,901
0,864
Table 6: Factuality, consistency and coherence results.
maries. Such relative selection is casy to do, while determining an absolute ordinal readabil- ity level per summary would require expert train- ing (Kiritchenko and Mohammad, 2017). We adopt a rating estimation in which game players' skills are estimated based on a series of multi-player games (Weng and Lin, 2011). 9 This is similar to the ELO score (Elo, 1978) used in chess, but it computes rating mean and standard deviation. Rat- ings start at 25.0 and are adapted based wins and losses. We draw 1,000 three-summary sets, where the summaries in each set are generated from the same input article and are randomly drawn from the pool of 18 summarics (4 methods x 4 target readability levels, plus baseline and reference) per input article. Therefore, estimated ratings can be compared across different settings. For informa- tiveness, we instruct annotators to mark the least informative and the most informative summary. Ta- ble 4 shows the human evaluation results. For all four methods, the readability tends to increase as higher target readability is requested, confirming
Related Work
Readability Control in NLG. Early efforts for readability control in NLG include microplanning
"We use the implementation in https:: http::s///ppessill..r

---

FRE Al BertScoref RG-Li FactCC
Future tokens
FRE A. BertScore] RG-L1 FactCCt
SCOREINSTRUCT-RL
3
6.1
0.872
33.10
0.51
Only FRE
11.9
0.869
33.76
0.64
5
5.5
0.873
33.19
0.51
0.65 FRE 0.35 BS-Fact
13.3
0,877
33,30
0.65
0.35 FRE 0.65 BS-Fact
15.2
0.879
35,42
0.64
10
4.8
0.873
33.53
0.52
Only BS-Fact
15.2
0,879
35.43
0.64
20
4.5
0.874
33.69
0.54
ECOREINSTRUUT+LL.
Only FRE
4.89
0.874
33.81
0.56
Table 8: Impact of the number of future tokens (ri) in
0.65 FRE + 0.35 BS-Fact
6.72
0.877
36.02
0.59
0.35 0.65 BS-Fact
9.33
0.879
37.32
0.63
SCOREINSTRUCT+LA in the CNN/DM validation set.
Only Fact
16.23
0.882
38.72
0.68
extend a single decoder framework to a mixture- of-experts architecture with multiple decoders, al- lowing the model to learn different summary styles. Zhang et al. (2022) produce summaries using de- signed attributes such as length and extractiveness. In contrast, in this work, we focus on readability levels in summary generation and propose training and decoding control strategies for their control.
Table 7: Ablation using Readability (FRE) and BS BS-Fact score combinations the CNN/DM validation set.
steps to tailor the generated text to match differ- ent target reading levels (Moraes et al., 2016) and adapting the text complexity in machine translation outputs (Agrawal and Carpuat, 2019; Marchisio et al., 2019). Recently, Luo et al. (2022) investigate controllable abstractive and extractive approaches for generating layman and expert summaries from biomedical documents. Concurrent with our work, Pu and Demberg (2023) conduct a inspection of the ability of a GPT3.5 model to adapt its output to different target audiences and writing styles (for- mal vs. informal), while Imperial (2022) found that that GPT2 models struggle in preserving the linguistic complexity of the input prompts. Impor- tantly, there has been significant development of models for Plain Language Summarization (PLS) from scientific papers (Devaraj et al., 2021; August et al., 2022; Goldsack et al., 2023; Guo et al., 2023). However, different from our proposed methods, such works do not consider fine-grained readability degrees. Concurrent to our work, Chi et al. (2023) employ weakly supervision and prompt methods to control readability complexity level of sentence paraphrasing.
Text Simplification. Text Simplification aims to improve the readability of sentences through re- ducing linguistic complexity while maintaining the original meaning (Alva-Manchego et al., 2021). Different aspects of the simplified output have been controlled, such adapting to specific level (Scarton and Specia, 2018; Nishihara et al., 2019) or incorporating edit operations MaanhMo et al., 2017; Kumar et al., 2020; Mallinson et al., 2020) or lexical and syntactic constraints (Martin et al., 2020) into text simplifications. Maddela et al. (2021) implement linguistically motivated syntac- tic rules with data-driven neural models to enhance the diversity and controllability of the simplifica- tions. In contrast text simplification, which aims to control the degree to which a sentence para- phrased, our approaches must provide succinct and informative summaries while maintaining different fine-grained levels of desired readability.
Controllable Text Generation. Previous work explore different alternatives to tailor text for di- verse target users (Cao et al., 2020; Kumar al., 2022; Pu and Demberg, 2023), while research has also been conducted on style control in various generation tasks, including paraphrasing and story generation (Wang et al., 2017; Shen et al., 2017; Huang et al., 2019). In particular, for summariza- tion, style control emphasizes factors such as ab- stractiveness (Goyal et al., 2022b; Dreyer et al., 2023) length, content (Fan et al., 2018; He et al., 2022: Shen et al., 2022: Liu et al., 2022b). Con- trary these, Böhm et al. (2019) employ RL with rewards from human preferences for generating different summaries, while our reward function is based on readability signals. Goyal et al. (2022b)
7 Conclusion
In this work, we propose three methods for fine- grained control of the readability level of sum- maries. We showed that instruction-ban based methods can be used to guide LLMS to generate summaries with fine- lineggriied readability degrees. We thus pre- sented a RL approach that uses a Gaussian-based reward and a new decoding method that allows to control the readability during inference. We pro- vided extensive evaluation of our approaches and showed that they significantly improves the control of the summaries' readability. Future work includes adapt the methods different NLG tasks and combine different metrics in order to capture distinct summary aspects that impact readability.

---

Limitations
Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume I: Long Papers), pages 295-305, Taipei, Taiwan. Asian Federation of Natural Language Processing.
this paper, we propose different methods to current summarization approaches to enhance the controllability of readability levels. While this ad- justment is not specific to any particular language, we conducted all of our experiments and analy- sis exclusively on English-language summarization datasets. Additionally, we focused solely on study- ing newswire summaries, given their widespread use in summarization research. Hence, this pa- per does not offer insights into the range of style variations found in non-Einglish and non-newswire datasets, nor does it ascertain the generalizability of our findings to other datasets and domains. Second, although the Lookahead method demonstrates en- hanced readability control, it requires heavy com- putational overhead, especially when is used with larger beams. To alleviate these expenses, one pos- sible solution is to employ distillation to enhance decoding speed (Wan et al., 2023). Finally, the ex- perimental cost of requesting API responses from OpenAl to assess ChatGPT's text generation abili- ties imposes significant constraints on the dataset selection. In this way, we limit our experimentation only one summarization dataset.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitability of automatic evalu- ation metrics for text simplification. Computational Linguistics, 47(4):861-88
Tal August, Lucy Lu Wang. Jonathan Bragg. Marti A Hearst, Andrew Head, and Kyle Lo. 2022. Paper plain: Making medical research papers approachable to healthcare consumers with natural language pro- cessing.
arXiv preprint arXiv:2203.000 200000000...0
Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2023. Paper plain: Making medical research papers approachable to healthcare consumers with natural language pro- cessing. ACM Trans. Comput. -Hum. Interact. Just Accepted.
Scott F Beers and William E Nagy. 2009. Syntactic complexity a predictor of adolescent writing qual- ity: Which measures? which genre? Reading and Writing, 22(2); 185-200.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira. Ido Dagan, and Iryna Gurevych. 2019. Bet- rewards yield better summaries: Learning marise without references. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCCNLP) pages 31 3110-3120, 120, Hong Kong, China. Association for Computational Linguistics.
Ethics Statement
While some of the investigated systems have demonstrated high level of controllability on the CNNDM dataset, this does not imply their use as general controllable summarization models. To en- sure reliability, these models should be thoroughly evaluated before being used in different settings.
Yixin Cao, Ruihao Shui. Liangming Pan, Min- Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communi- cation between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1061-1071, On- line. Association for Computational Linguistics.
In the conducted human experiment, for informa- tiveness, workers took median time of 79 seconds per HIT, and we paid $0.40 plus bonus of $0. 10, which amounts to $22.80 per hour. For readability, workers took a median time of 58 seconds per HIT, and we paid $0.20 plus a bonus of $0.05, amount- ing to pay of $15.50 per hour.
Alison Chi, Li-Kuang Chen, Yi-Chen Chang, Shu-Hui Lee, and Jason S. Chang- 2023. Learning to para- phrase sentences to different complexity levels.
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- cry, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra. Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction- finetuned language models.
References
Sweta Agrawal and Marine Carpuat. 2019. Controlling
text complexity in neural machine translation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 1,549-
guage Processing (EMNLP-IJCN) 1564, Hong Kong, China, Association for Computa- tional Linguistics.
instraetion-finetueed

---

Meri Coleman and Ta Lin Liau, 1975. computer readability formula designed for machine scoring. Journal of Applied Psychology, 60:283-284.
In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 10589-10604, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguisties.
Kevyn Collins- Thoniison 2014. Computational assess- ment of text readability: A survey of current and future research. ITL. - International Journal of Ap-
Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2023. Domain-driven and discourse-guided scientific summarisation. In Ad- vances in Information Retrieval: 45th European Con- ference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part I, pages 361-376. Springer.
plied Linguisties, 165(2)97-135.
Scott Crossley, Aron Heintz, Joon Suh Choi, Jordan Batchelor, Mchrnoush Karimi, and Agnes Malatin- szky, 2023. A large-scaled corpus for assessing text readability, Behavior Research Methods, 55(2):491- 507.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022a. News summarization and evaluation in the era of gpt-3.
Scott A. Crossley, Stephen Skalicky, Mihai Dascalu, Danielle S. McNamara, and Kristopher Kyle. 2017. Predicting text comprehension, processing, and famil- iarity in adult readers: New approaches to readability formulas. Discourse Processex, 94(5-6)330--355)
Tanya Goyal, Nazneen Rajani, Wenhao Liu, and Wo- jeiech Kryscinski. 2022b. HydraSum: Disentan- gling style features in text summarization with multi- decoder models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 479, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Edgar Dale and Jeanne S Chall. 1948, A formula for predicting readability: Instructions, Educational re- search bulletin, pages 37-54.
Ashwin Devaraj, Iain Marshall, Byron C Wallace, and Junyi Jessy Li. 2021. Paragraph- simplifica- tion of medical texts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages -9844
Robert Gunning. 1952. The technique of clear writing. McGraw-Hie Hill, New York.
Yue Guo, Tal August, Gondy Leroy, Trevor Cohen, and Lucy Lu Wang. 2023. Appls: A meta-evaluation testbed for plain language summarization.
Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Co- hen. 2021. Automated lay language summariza- tion of biomedical scientific reviews. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1):160-163
Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2023. Evaluating the tradcoff' between abstractiveness and factuality in abstractive summarization. In Findings of the Association for Computational Linguistics: EACL 2023. pages 2089 2105, Dubrovnik, Croatia. Association for Computa- tional Linguisties.
Junxian He, Wojciech Kryscinski, Bryan McCann. Nazneen Rajani, and Caiming Xiong. 2022. CTRL- sum: Towards generic controllable text summariza- tion. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing. pages 5879-5915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
William H DuBay, 2004. The principles of readability. Online Submission.
Arpad E. Elo. 1978. The Rating of Chessplayers, Past and Present. Arco Pub., New York.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS, pages 1693-1701.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceed- ings of the 2nd Workshop on Neural Machine Transla- tion and Generation, pages 45-54, Melbourne, Aus- tralia. Association for Computational Linguistics.
Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang. and Xiaodong He. 2019. Hier- archically structured reinforcement learning for topi- cally coherent visual story generation. Proceedings of the AAAI Conference on Artificial Intelligence. 33(01))8465-84)
Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment. In Coling 2010: Posters, pages 276 284, Beijing, China. Coling 2010 Organizing Comnittee.
Joseph Marvin Imperial. 2022. Uniform complexity for text generation.
Irene C Fountas and Gay Su Pinnell, 1999. Matching Books to Readers: Using Leveled Books in Graded Reading, K-3, ERIC.
Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new read- ability formulas (automated readability index, fog count and flesch reading ease formula) for navy en- listed personnel.
Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Cor- pora for the lay summarisation of scientific literature.

---

Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In ICLR 2015. San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
in Natural Language Processing, pages 8938-88888 8958. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yizhu Liu, Qi Jia, and Kenny Zhu. 2022b. Length control abstractive summarization by pretraining information selection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6885- 6895, Dublin, Ireland. Association for Computational Linguistics.
Svetlana Kiritchenko and Saif Mohammad. 2017. Best- worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceed- ings the 55th Annual Meeting of the Association for Computational Linguistics Volume 2: Short Papers), pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang. Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lian- hui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2022. NouroLogic a" esque decoding: Constrained text generation with lookahead heuris- tics. In Proceedings of the 2022 Conference of the North Americun Chapter of the Association for Com- putaaiional Linguistics: Human Language Technolo- gies, pages 780-799, Seattle, United States. Associa- tion for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2332-9346 Online. Association for Computa- tional Linguistics.
Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vech- tomova. 2020. Iterative edit-based unsupervised sen- tence simplification. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics. pages 1918-7928. Online, Association for Computational Linguistics.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022. Readability controllable biomedical document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4667-4680, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. 2022. Gradient- based constrained sampling from language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2251-2277, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Mounica Maddela, Fernando Alva- Alva-Manchego, and Wei Xu. 2021. Controllable text simplification with ex- plicit paraphrasing. Ineeennnnrrrraeeeeee Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3553,, Online. As- sociation for Computational Linguistics.
Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. 2021. Keep it simple: Unsupervised simplification of maraa-pa text. arXiv preprint arXiv:2100.03444.
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido, 2020. FELIX: Flexible text edit- ing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1244-1255, Online. Association for Computational Linguistics.
Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022. Faithful or extractive? on mitigating the baihfulless-abssraaiiveness trade- off in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1410-1421, Dublin, Ireland. Association for Computational Linguistics.
Kelly Marchisio, Jialiang Guo. Cheng-I Lai, and Philipp Koehn. 2019. Controlling the reading level ma- chine translation output. In Proceedings of Machine Translation Summit XVII: Research Track, pages 193- 203, Dublin, Ireland. European Association for Ma- chine Translation.
Junyi Jessy Li and Ani Nenkova. 2015, Fast and accu- rate prediction of sentence specificity. In Proceed- ings of the Twenty-Ninth AAAI Conference on Artifi- cial Intelligence, AAAI 15. page 2281-2287, AAAI Press.
Louis Martin, Éric de la Clergerie, Benoit Sagot, and Antoine Bordes. 2020. Controllable sentence sim- plification. Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689- 4698, Marseille, France. European Language Re- sources Association
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- tion Branches Out, pages 74-81. Barcelona, Spain. Association for Computational Linguistics.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He. Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022a. Rainier: Reinforced knowledge introspector for commonsense question answering. Proceed- ings of the 2022 Conference on Empirical Methods
Matej Martine, Senja Pollak, and Marko Robnik- Šikonja. 2021. Supervised and unsupervised neu- ral approaches to text readability. Computational Linguistics, 41419114

---

Danielle S McNamara, Scott A Crossley, and Philip M McCarthy, 2010. Linguistic features of writing qual- ity. Written communication, 27(1):57-86.
Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing. Yang You, and Luo Si. 2022. MReD: A meta-review dataset for structure-controlaable text generation. In Findings of the Association for Computational Lin- guistics; ACL 2022, pages 2521-2535, Dublin, Ire- land. Association for Computational Linguistics.
Priscilla Moraes, Kathleen McCoy, and Sandra Carberry. 2016. Enabling text readability awareness during the micro planning phase of NLG applications. In Pro- ceedings of the 9th International Natural Language Generation conference, pages 121-131, Edinburgh, UK. Association for Computational Linguistics.
Tianxiao Shen. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-paralled text by cross-alignment. In Advances in Neural Informa- rion Processing Systems, volume 30. Curran Asso- ciates, Inc.
Daiki Nishihara, Tomoyuki Kajiwara, and Yuki Arase. 2019. Controllable text simplification with lexical constraint loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics: Student Research Workshop, pages 260- 266, Florence, Italy, Association for Computational Linguistics.
Edgar A Smith and R.J. Senter. 1967. Automated read- ability index. AMRL-TR. Aerospace Medical Re- search Laboratories (US), pages 1-14.
Teerapaun Tanprasert and David Kauchak. 2021. Beesch-kincaid is not text simplification evaluation metric. In Proceedings of the 1st Workshop on Natu- ral Language Generation, Evaluation, and Metrics (GEM 2021), pages 1-14, Online. Association for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages Curran Associates, Inc.
Amalia Todiraseu, Thomas François, Delphine Bern- hard, Núria Gala, and Anne-Laure Ligozat. 2016. Are cohesive features relevant for text readability evaluation? In Proceedings of COLING 2016, the 26th International Conference on Computational Lin- guistics: Technical Papers, pages 987-997, Osaka, Japan. The COLING 2016 Organizing Committee.
Nicole Pitcher, Denise Mitchell, and Carolyn Hughes. 2022. Template and guidance for writing a cochrane plain language summary.
Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment re- search. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366- 5377, Marseille, France. European Language Re- sources Association.
Emily Pitler and Ani Nenkova. 2008 Revisiting read- ability: A unified framework for predicting text qual- ity. In Proceedings of the 2008 Conference Empir- ical Methods in Natural Language Processing, pages 186-195, Honolulu, Hawaii. Association for Compu- tational Linguistics.
David Wan, Mengwen Liu, Kathleen McKeown, Markus Dreyer, and Mohit Bansal. 2023. Faithfulness-awaal decoding strategies for ab- stractive summarization.
Dongqi Pu and Vera Demberg. 2023. Chatgpt vs human- authored text: Insights into controllable text summa- rization and sentence style transfer.
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2864-2880, Dubrovnik, Croatia. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Di Wang, Nebojsa Jojic, Chris Brockett, and Eric Ny- berg. 2017. Steering output style and topic in neural response generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2 2140-2150, Copenhagen, Denmark. Association for Computational Linguis- tics.
Carolina Scarton, Gustavo Paetzold, and Lucia Spe- cia. 2018 Text simplification from professionally produced corpora. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018).
Carolina Scarton and Lucia Specia. 2018. Learning sim- plifications for specific target audiences. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 712-718, Melbourne, Australia. Association for Computational Linguistics.
Ruby C. Wenn and Chih-Jen Lin. 2011. A bayesian approximation method for online ranking. Journal of Machine Learning Research, 1269):267-300
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv e-prints.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms.

---

Qingyang Wu, Lei Li, and Zhou Yu. 2021. Textgail:
Computing Infrastructure Optimizer
32GB NVIDIA VI0O GPU
Generative adversarial imitation learning for text gen- eration.
Adam
Optimizer Params learning rate
9, 0.999), 10-8
le-4
Hanqing Zhang and Dawei Song. 2022. DisCup: Dis- criminator cooperative unlikelihood prompt-tuning for controllable text generation. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3392-3406, Abu Dhabi, United Arab Emirates. Association for Com- putational I. Linguistics.
Learning Rate Decay
Lincar
Weight Decay
Warmup Steps Maximum Gradient Norm batch size beam size
epoch
Lookahead Lookahead
25
20
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020a. Pegasus: Pre-training with ex- tracted gap-sentences for abstractive summarization. Proceedings of the 37th International Conference on Machine Learning. ICML'20. JMLR.org.
Table 9: Hyperparameter settings for Flan-T5 methods.
Dale-Chall readability formula (Dale and Chall. 1948) (DCR) necessitates compilation of 3000 words that are deemed understandable by fourth- grade students in the US. Words not found in this list are classified difficult. The following expres- sion is used in calculation:
Tianyi Zhang, Varsha Kishore, Felix Wu", Kilian Q Weinherger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with bert. In International Conference Learning Representations.
Yusen Zhang. Yang Liu, Ziyi Yang, Yuwei Fang. Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. 2022. Macsum: Controllable summarization with mixed attributes.
diflicurrWonds
1osalWords
0.1679
100)
kotalSennenees
Coleman- index (CLJ) (Coleman and Liau, 1975) relies on characters instead of syllables per word: CLI = 0.0588L 0.2965 15.8,
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
where L is the average number of letters and S is the average number of sentences.
B Hyper-parameter Settings
The experiments were executed using the version 3.3.1 of the transformers library released by Hug- ging Face (Wolf et al., 2019). The fine-tuning pro- cess is halted once the model reaches convergence in terms of ROUGE In Table 9, we report the hyperparameters used train the models on CNN/DM. We use the Adam optimizer (Kingma and Ba, 2015) and employ a linearly decreasing learning rate schedule without warm-up.
Appendix
A Readability Metries
Flesch reading ease (FRE) (Kincaid et al., 1975) metric assigns higher scores to texts that are easier read. It is calculated follows:
the validation set.
WordsWords
poalSyyllbles
206.836 1.015(
IntalSenenness
localWords
The Gunning fog index (GFI) (Gunning, 1952) guantifies the level formal education required for a person to comprehend given text upon initial reading, and is computed using the following formula:
C Experiments with GPT3.5
All of our experiments version of GPT3.5 (TEXT-DAVINCI-003) between 25 May 2023 and Jun 2023 by using the OpenAl's API. 10 We set temperature 1, top_p=1, frequency penalty = O, and presence penalty = 0. For Best-GPT3.5, we select k = 3.
conducted the
GFI 0.4( Words 100
long Words totalSentences'
IntalSentences
where long are words longer than charac- ters. Higher values indicate lower readability.
The Automated Readability Index (ART) (Smith and Senter, 1967) is an alternative readability for- mula that provides values correlating to the number of years of education needed comprehend a given text:
We select the the following prompts that gave the best results in terms of readability metrics:
ARI 4.71 IetalCluracterr.
InnalWirds
0.55
21.43.
Totaa Words
ExaaSeenences

---

Readability level
DCR
ARI
E Mechanical Turk Setup
GPT3.5
11-ycar-old
12.5
We provide additional details Amazon Me- chanical Turk setup. AMT annotators are non- experts, so we use several mitigation strategies obtain high-quality human judgements, including simplified task setups, clear annotation guidelines, task task-specific qualification tests, and time checks to exclude potential spammers. We gave annotators fair compensation.
Middle school
10.1
13.3
High school
10.4
14.1
College student
10.6
Best-GPT3
11-ycar-old
9.5
11.4
Middle school
9.8
12.3
High school
10.1
13.3
College student
10.7
15.5
CATEGORYINSTRUUE
11 11-yar-oodd
9.2
7.1
Middle school
9.7
8.8
We give detailed instructions to the annotators, see Figures and 7. We add number of tasks with known answers (i.e., cases where the most/least readable/informative summaries should be clear), enabling us to estimate in real time the accuracy of workers who work on multiple of these. Work- who complete the tasks too quickly or have low accuracy the tasks with known answers are automatically removed from our worker pool; their answers are replaced with new answers. We also use a bonus incentive structure. Every worker who passes the automatic quality checks receives a bonus at the end. In addition, we use custom qualification tests. For any worker to be accepted as annotator for readability and informa- tiveness evaluations, there three hurdles: (1) We only consider workers from country whose main language is English, who has completed 100 or more HITs so far with an acceptance rate of 95% or higher. (2) In addition, workers must have passed an initial custom qualification test for a re- lated text classification task we have conducted in the past. (3) The workers who have passed (1) and (2) qualify take the custom qualification tests for our readability task and our informativeness task. Only the workers who passed these final tests were accepted to work on the human readability or informativeness evaluations this paper.
High school
10.2
10.2
College student
10.8
11.4
SCOREINSTRUCT
90 (11-year-old)
9.0
6.5
70 (Middle school)
9.6
8,4
(High school)
10.2
10.1
(College student)
10.8
11.2
SCOREINSTRUCT+RL
(11-year-old)
4.7
70 (Middle school)
8.9
6.9
50 (lligh school)
9.7
8.6
30 (College student)
10.5
10.1
SCOREINSTRUCT+LA
90 (11-year-old)
8.2
4.6
(Middle school)
50 (High school)
10.2
9.9
30 (College student)
12.4
14.8
Table 10: Comparison CNN/DM using instruction- based, RL and Lookahead (LA) methods using addi- tional readability metrics. Lower DCR and ARI (4) denote higher readability.
[Document] In Summarize the above article in 3 sentences for sixth-grade student.
[Document] In Summarize the above article in 3 sentences for middle-school student.
[Document] In Summarize the above article in 3 sentences for high-school student.
[Document) In Summarize the above article in 3 sentences for college student.
On our batches of 1,000 HITs, we allowed any worker to complete maximum of 333 HITs, so that worker can dominate the results. We two annotators per HIT.
D Results with Additional Readability Metries
Even though we only display three summaries at a time and only receive an annotated relative ranking of these three as opposed to an abso- lute ordinal readability level - we wish to estimate readability score for cach of the four methods (Sec. and their various target readability scores that these setups can be compared based on human judgements. We interpret each set of three summarics as a three-player game in which the method that generated the most readable summary
Table 10 shows results using additional metrics, Dale-Chall readability and Automated Readabil- ity Index for readability assessment. CATEGORY- INSTRUCT and ScoREINSTRUCT methods perform sim- ilarly with SCOREINSTRUCT's improvements in the 1-year-old level. ScoREINSTRuCTHL. able to bet- ter distinguish between readability levels compared to SCOREINSTRUCT+RL..

---

FRE: GFU CT.))
Noymer Dani prosed their dedication Banztions suppoting club's baskethull sile. Naymar and Alves headed watch Neymar civals, just by point difference. Neynur helped Bruall Chil: a- Emirates Statium Sunday feisty costest and had wilsttnd sevenal brutsl challenges frum the Soith Ammton moals Before the inemattinal bnokk Lais Eanque's Barcelona had won their EI Cheino cuttest to move four points clear ihe upp of La Liga. Neymer and his Barceleta teass mates retuzn to La Lipa on Sunday, taking on Calia Vigo as they continne to compote for weble of srphics, Neymar's sater Rataella (ief)) headed to watch El Clasico of bookethall with the Bereclons fortard Neymar's sistr Rafaclla tright attens Euro Leegue basketall mench between Barcelona and Redl Madiid Neymar's is distrctiel at the haskebull as his sieter Rafaella (caming) watchrs on Neymar, Rioril's cogoin, led his naturaal side to 1-0 win over Chile the Linrates last weekend Barceloza team mate Dani Alves (Sront left jonned Neymar and Enends to warch some live basketball. Gary Medel walks off having appcaed to stamp on Neyonr the Emia ites oz Sunday.
Decument: Thunday night akongide
45.7
10.5
proviled
victury
Leagle
(oeasi)) selte with
Barreiona team muie Dam Alves (rigtd However Real Modral
Leugue divason over
ANTEGORYINTTRUS
Requestel leve:: I-atar-old. Neymar and Dani Alves were to watch buskefeell with their siste Rafacll, Barca bes Resi Madril 85-30 the Faro League contaet. Real remain top the leugue by point difference.
70.8
10.2
6.7
Requested level: middle school. Neymar and Dani Alves watched besketball with Neymar's sisee Ratoella, Barceloux best Real Modrid 85 80 is the Euro Lengue
58.1
10.6
6.0
on Thuralay Real remuin top of the league points difference. Neynur heiged Brisil hest Chile 1-0 the Emirates on Sunday.
Requested level: high school. Neymar Dani Alves watched baskerhull with Neymor's sister Rafaells. Baroelons best Real Madrid 65-50 in the Earo Lengue
52.1
10.8
8.0
difiereace.
Requested edllege. Neymar and Studimm Sinday.
iltended EI Clasico bookethlll match. Besceions beat Real Madrid 85-40 Euro League Neynur
51.7
6.7
helpol Brasil
INTRRINSTRUCT
Requested rax 90, Neymar and Dani Alves watched hasksball with Neymar's vider Rafacila, Barza heat Real Malri 85 1--80 in the Euro Leagne on Thurdlay night
60.7
9.9
8.1
Redl remain their division thair biter rivals points diffienmer.
Requested 70, Neymar Dani Alves litee
baskzthull
Rafaella. Harcelonu Real Madrid 85 the Euro Lengue Thursday
56.0
8.1
night Real tog Bais division
just differessee.
Requested 50, Neymar
baoketalll Naymor's
Bartolon Madrid 65 Furn eagge on Thuraby
45.5
89
night remain top their divisian over Hasceiona by just pounts diffesence.
Requested tur 30, Noymar Dusi Alves wasched basketball Neyna''s sister Rafaclla, Barcelona best Midrid 85-80 in Lengue baskethall coneest Real remain top of their divisoo over Barcclona by poins difference.
42.6
11.5
9.2
SCOREINSTER INSTRUCT+ RI
Emirtees Sunday, Neymur
watched Clascc baskethall Neymur's sister Neymar Eure Leaguz oc Tlursda night. Ney side are fous poines the Requestod Fat 70, Neynee and Alvss witched beskthall with Neymar's
Requested
over Chile the
72.5
6.1
Roal Madrid
Rafaella at Clasica on Thunday night Neynar Beazil to over Chile at the Emirates Stadium on Sunday, Barcelona beat Real Matrid 85 the Eumo Lengue bookethall masch Ney mar's side are four poins clear at the top La Liga.
65.2
6.9
Requested run 50, Neymar and Dani Alves warched beoketlull with Noynur's 's sister Rafaella at Clasico or Thunsday right. Bercelora best Real Mudrid 55-80
54.5
12.0
7.5
the Euns Langue baskeibull moach the Emirates Stathmm Neymar led Brooi in 1-0 win over Chile u the weekerd.
Requested tit 30, Neymar and Dani Alves wasctedd basketbali with Neynar's sister itatellla at Clasico or Thursday night. Barcelona beat Real Madnid 55 80
54.5
12.0
7.5
the Euro League basketball mutch the Emirates Stadium. Neymar led Bruzi to 1-0 wio over Chile at the weekerd.
Table II: Examples of generated summaries for different readability levels.
wins and the method that generated the least read-
able summary loses. 22 workers worked on our readability evaluation, while 28 worked on our in- formativeness evaluation.
F Examples of Generated Summaries
Tables 11 and 12 present summaries of distinct levels generated by the different methods and their readability scores given by Flesch reading ease (FRE), Gunning fog index (GFI), and Coleman-Liau index (CLI) metrics.

---

FRE GFT CLT
Document: A group of U.S. senators has written to football's world governing body FIFA, calling for Russia to be removed as bost the 2018 World Cup because of its role in Ukraine crisis and occupation of Crimea. In letter dated Tuesday and released on Wednesday, the 13 Democratic and Republican U.S, lawmakers said they 'strnngyy encourage' FIFA to move the global competition. Allowing Russia to host the World Cup inappropriately bolsters the prestige of the (Russian President Vladimir) Putin rogime time whea it should be condemned and prosides economic reliet at time when much of the international community is imposing economic sanctions." the senators wrote to FIFA President Sepp Blatter. FIFA president Sepp Blatter announces Russia the bost nation for the 2018 World Cup. Republican John McCain one of 13 US senatoes urging FIFA to think aguin.. The lead signers were Senator Robert Menendez, the top Democrat on the Semate Foreign Relations Committee, and Senator Ron Johnson, the Republican chuirman of the foreign relations panel's Europe subcommittee. Republican Senator John McCain, chairman of the Senate Armed Services Committee, also signed the letter, as did Senator Richard Durbin, the number tournament at home in 1996. Mcanwhile, England are interested in bidding for Euro 2028 but are very unlikely to bid for the 2026 World Cup. FA chairman Greg Dyke has rovealed. FIFA announced last week that the decision on the 2026 tournameut will be made in 2017 - when Blatter is still likely to be FIFA president. The EA has decided not to bid for FIFA tournamennt while Blatter in power following England's 's failed for 2018, but eycing a for European Champiouship 2028.
40.5 11.6
15.3
Democrat the Senate. England are planning bid host Eur 2028 after last hokding the
CATEGORY y INSTRUCT
Requested level: 11 -l-ycars-old. 13 US senators have written to FIFA asking for Russia to be removed as host of the 2018 World 2026 World Cup will he made in 2017. Engllnd are planning hid to Euro 2028 are unlikely to hid for the 2026 World Cup.
86.7
8.3
10.0
Cup. They say the decision
They are calling Russin to he removed us host of the 2018 World Cup. The letter was signed by Republican Senator John McCain. Figllnd are interested in bidding for Euro 2028
Requested level: middle school. 13 Democratic and Republicun US senators have written
8.2
are unlikely to hid for 2026 World Cup.
Requested level: high school. 13 Democratic and Republican US senators have written to PIFA. They 'strongly encourage' FIFA to move the global competition. The senators say Russin's 's role in the Ukraine crisis and occupation Crimea should be condemned.
37.8
12.3
11.5
13 Democratic and Republican U.S. lawmakers "strongly encourage" FIFA to move the global competition. The senators said allowing Russia to host the World Cup 'inpproopriately holsters the prestige of the (Russiun President Vladimir) Putin regime'
Requested level: college.
21.7
13.7
13.8
SCOREINSTRUCT Requested FRE 90. 13 US senators call for Russia be removed as bost of the 2018 World Cup. They say the decision should be made in 2017 when FIFA president Sepp Blatter is still likely to be in power. England are planning a bid to bost Euro 2028
79.1
8.8
11.7
but are unlikely bid for the 2026 World Cup.
Requested FRE 70. 13 Democratic and Repablicun U.S, senators write to FIFA. They say Russia's hosting the World Cup
61.2
11.5
9.8
"bolsterss the prestige of the Putin regime' England planning hid to host Furo 2028 but unlikely to hid forr 2026.
Requested FRF 50, 13 Democratic and Republican U.S. senators write to FIFA. They 'strongly encourage" FIFA to move the global competition, England are planning a host Euro
$9.4
11.5
9.2
tournament home Requested FRE 30. 13 Democratic and Republican U.S. lawmakers urge FIHA to move the global competition. 'Allowing Russia to the World Cup inappropriately bolsters the prestige of the (Russian President Vladimir) Putin regime," they wrote.
last holding
30.3
13.3
10.3
SCOREINSTRUCTE
Requested RE 90, 13 senators have written to FIFA calling for Russia to be removed as host of the 2018 World Cup. The 13 senators say the coumnnmen should moved because of role in the Ukraine crisis. England are planning bid to host Euro 2028 but are unlikely to bid for the 2026 World Cup.
72.8
9.1
10.9
Requested FRE 13 senators have written FIFA calling Russia to removed host
World Cup. 13 Democratic and Republican lawmakers say they 'ttonngly encourage' HIHA to move the toumament. Russia were announced as host nation
62.8
10.2
11.3
tournament Russia. England are planning hid to Requested FRE 50. 13 Democratic and Republican senators have written to FIFA president Sepp Blatter calling for Russia be removed as bost of the 2018 World Cup. The seuators say Russia's 's role in the Ukraine crisis and occupation of Crimea should be condermed. England
Euro 2028 are unlikely hid for 2022..
56.4
10.7
12.6
planning bid to bost Euro 2028 but unlikely to bid for 2026 World Cup.
Requested F RE 30. 13 Democratic and Republican seuators have writien to FIFA president Sepp Blateer calling for Russia to be removed as bost of the 2018 World Cup. The seustoes say Russia's role in the Ukraine crisis and occupation of Crimea should be condermned England are unlikely to bid for the 2026 World Cup after failing to quality for 2018 8 tournament.
50.1
11.1
13.9
SCOREINSTRUCE CURCCTAAA
Requested FRE 90. 13 U.S. seastors call for Russia to be removed as bosts of the 2018 World Cup. They say the decision should be made in 2017. England are planning a bid to host Euro 2028 but are unlikely to bid for the 2026 World Cup. Requested FRE 70. US senators call on FIFA move the 2018 World Cup from Russia, Russia
90.0
8.9
9.5
announced the host
65.9
11.1
9.2
nation for the 2018 tournament. 13 Democratic and Republican U.S. lowmakers signed the letter.
Requested FRE 50. FIFA president Sepp Blatter unnounced Russia host nation for 2018 World Cup. Democratic and Republican U.S. senators have written to FIFA president urging him to move the event to 2018. The lawmakers said Russin's role the Ukraine crisis and occupation Crimen should be condemned.
50.4
13.1
12.2
Requested FRE 30. 13 Democratic and Republican U.S. lawmakers urge football's 's world governing body FIFA to move the global competition. The senators said Russia hosting the 2018 World Cup 'inappropriately Presideut Vladimir) Patin regime' England are interested in Eure 2028 but are very unlikely to bid for the 2026 World Cup. FA chairman Greg Dyke has revealed.
30.2
13.4
17.4
the preatige the (Russian
Table 12: Examples of generated summaries for different readability levels measured using FRE, GFI and CLI metrics.

---

Instructions (Click collapse)
Welcome!
need your help on evaluating three automatically generated summaries.
summaries, select readable the readable.
The most readable summary shorter, easier common words, less technical jargon, shorter sentences, less complex grammar. easier read younger
The least readable summary sentences, more complex grammar. harder to read for younger readers.
longer uncommon words, technical jargon, longer
When you select one as most readable, will turn green. When you select
least readable,
red.
Ties: If summaries tled you may select multiple summaries. You select multiple most readiable summaries or multiple readable summaries. of different summaries are very similar. You do NOT need to select multiple most readability (or multiple least readable) summaries every time. It should happen only in the case of ties. is acceptable (and encouraged) to select most readable, as least readable and leave unchecked.
sense
readability
The three summaries page summarize that they are summarizing, save you time and effort.
text. display the summaries only,
Note that summaries occasionally describe distrresinn events, such violent crimes.
Please evaluate the three summaries shown below.
Select the most readable summary (le., uses shorter, easier more common words; less technical jargon, shorter sentences, less complex grammar. It is easier to read for younger readers.
Select the least readable summary (le., uses some longer or uncommon words; more technical jargon, longer sentences, or more complex grammar. is harder to read for younger readers.
Ties: You may select multiple there are multiple very similar most readable or least readable summaries.
Most Least
readable readable
Summary
S(summayy1)
$(summary2)
$(summary3)
Submit
Figure 6: Screenshot of the human readability annotation.

---

Instructions (Click collapse)
Welcome!
We need your help on evaluating three automatically generated summaries by comparing them to the original article that the summaries try summarize.
For these three summaries, you only need to select one as most informative and one as least Informative.
The most informative summary is best at expressing the main points of the article, its content is the most important and relevant. It's not necessarily the longest summary -- just the that contains most important information.
The least informative summary least important and relevant. It's not necessarily the shortest summary -- just the one with the least important information.
expressing the points the article, content
When you select one as most informative, will turn green.
When you select one as least informative, will turn red.
select multiple most informative summaries or multiple least informative summaries. This makes sense when different summaries have very similar informatiivnees.. You do NOT need to select multiple most informativeness multiple least informative) summaries every time. should happen in the case ties. acceptable (and encouraged) informative and leave one unchecked.
summaries tied you may select multiple summaries.
as informative, one
Note that texts occasionally describe distressing events, such as violent crimes.
Please evaluate the three summaries shown below the article.
First, here is the article:
S(input)
Select one summary as the best, Le., the most informative summary! That summary is best at expressing the main points of the article, its content is the most important and relevant.
Select one summary as the worst, l.e., the least informative summary! That summary is worst at expressing main points of the article, its content is the least important and relevant.
Ties: You may select multiple there are multiple very similar most informative least informative summaries.
Least Informative Informative
Most
Summary
$(summary1)
$(summary2)
$(summary3)
Submit
Figure Screenshot the human informativeness annotation.


Reviews:
[{"review_id": "U9LEz1Bjql", "reviewer": "Reviewer_RnEB", "summary": "", "questions": "1. For score based instruction, what does the score here mean? When you want to prompt a summary, what scores would you use? Would the space be too sparse as good summarizations are limited for each document, but the score is a real-value ranging from 0 to 100, say a large space.  I guess the <summary, score> pair space would be very sparse.\n2. Can author show more cases about the generated summary in different readability level?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for noting that our paper is well-written and well-organized.\n\n**- About PPO being a common technique**\n\nWe would like to address your point about the novelty of our paper. While using PPO to optimize toward a reward function is common, our specific reward function (Line 256 - Line 272) is an innovation over the standard PPO, and we designed it for our task of fine-grained control of reading levels requested at inference time. Generally, the reward is given directly by a reward model [1, 2], but in our paper the reward is computed as the (non-linear Gaussian) distance of the observed readability to the requested readability (Line 263 - Line 264). More details about this setting are described in Section 4 (Line 343 - Line 251). This setup introduces flexibility and controllability at inference time that is absent in typical PPO-RLHF papers [3]. PPO is usually used in the context of human feedback (i.e., RLHF), but our approach does not require learning a reward model from human feedback.\n\n**- About the experimental settings**\n\nReadability levels are not determined solely by age, but by educational level. We followed established guidelines for text complexity levels from previous work (Line 153 - Line 159 and Line 194 - Line 199) using (US) grade levels of education. The summary can be understood by someone at the complexity level and below. Moreover, while our proposed methods (CategoryInstruct, ScoreInstruct) are flexible and can encode different readability definitions via instructions – for example the ones based on Common European Framework of Reference for Languages (CEFR) levels [4], we focus on (US) grade levels which are used to estimate the years of formal education using the different readability metrics described in our submission (Appendix A - Line 964 - Line 995). We will provide additional clarification on the levels of education in the final version.\n\n**- About the score-based instructions**\n\nFor our method with score-based instructions (ScoreInstruct), we employ the Flesch Reading Ease (FRE) metric as the score in the experiments (Line 334 - Line 338). Table 1 shows the mapping between the FRE scores and the intended readability level described in the prompts, based on previous work [5]. For example, a summary with a FRE score of 80 or higher is easy and quick to read. In contrast, a text with a FRE score smaller than 40 is difficult to read and best understood by college graduates. For a given readability level, a prompt instruction with a score comprehending such level (Table 1) can be used. \nYou ask if the score-based instruction is sparse. The specific scores in the prompt are indeed sparse. However, we expect that the Flan-T5-Large model, which has close to 1B parameters and has been extensively pre-trained and fine-tuned on different tasks (including arithmetic reasoning [6]). Previous work [7] found that indeed T5 models can learn meaningful representation for integer numbers. Furthermore, Figure 4a shows that our score-based methods learn to generate summaries with different readability levels based on the distinct numerical scores presented in the instructions. Finally, Table 2 shows that the Pearson correlations between the requested score in the instructions and the observed readability are strong: 0.62, 0.74 for ScoreInstruct and ScoreInstruct+RL;, demonstrating that our approaches are able to follow the instructions with numerical values. We will include further details about ScoreInstruct in the final version.\n\n**- About summary examples with different readability levels**\n\nBelow we present examples of summaries with distinct readability levels generated by the lookahead method:\n\n**Document:**\nAn 18-year-old man has been charged with conspiring to commit terrorist acts following Saturday morning's counter-terror police raids, in which police seized a number of knives and swords. Sevdet Besim … was charged with 'conspiracy to commit acts done in preparation for, or planning, terrorist acts' at the Melbourne Magistrates Court…. \n\n**Requested Flesch Reading Ease score: 90 (easily understood by an 11-year-old student).**\n**Summary:** A total of five teens are being held by police for 'simply evil' plans. One man was charged with terrorism and will be remanded in custody. He is one of two men, both 18, who were arrested in the raids. Police seized a number of knives and swords at some of the homes.\n**Observed readability scores:** Flesch Reading Ease (higher more readable): 81.4, Gunning Fog Index (lower more readable): 9.0, Coleman-Liau Index (lower more readable):  5.5.\n\n**Requested Flesch Reading Ease score: 70 (Plain english, fairly easy to read).**\n**Summary:** A total of five men were arrested by police as part of Operation Rising on Saturday morning. One man has been charged with conspiring to commit terrorist acts and is due to appear in court on April 24. Sevdet Besim, 18, did not apply for bail and was remanded in custody for a filing hearing on April 24. Two of the men were 'associates' of Abdul Numan Haider, the 18-year-old who was shot dead on September 23. Police seized a number of 'edged weapons' including knives and a sword.\n**Observed readability scores:** Flesch Reading Ease: 62.3, Gunning Fog Index: 10.2, Coleman-Liau Index: 7.7.\n\n**Requested Flesch Reading Ease score: 50 (fairly difficult to read).**\n**Summary:** Sevdet Besim, 18, was charged with 'conspiracy to commit acts done in preparation for, or planning, terrorist acts' on Sunday afternoon. He is one of two 18-year-old men arrested following counter-terror police raids in Melbourne. The men were 'associates' of Abdul Numan Haider, who was shot dead after stabbing two officers outside a Melbourne police station on September 23. The second teenager is still being interviewed by police who plan to charge him with a number of offences related to preparing for and planning terrorist acts.\n**Observed readability scores:** Flesch Reading Ease: 44.7, Gunning Fog Index: 14.6, Coleman-Liau Index: 12.4.\n\n**Requested Flesch Reading Ease score: 30 (difficult to read).**\n**Summary:**  Sevdet Besim was charged with 'conspiracy to commit acts done in preparation for, or planning, terrorist acts'. He's one of two 18-year-old men arrested for allegedly planning an 'ISIS inspired' attack on an Anzac Day ceremony. The men were 'associates' of Abdul Numan Haider, the 18-year-old gunman who was shot dead after stabbing two officers outside a Melbourne police station on September 23. Victorian Premier Daniel Andrews dubbed the plans 'simply evil'.\n**Observed readability scores:** Flesch Reading Ease: 34.9, Gunning Fog Index: 15.2, Coleman-Liau Index: 13.1.\n\nWe will include examples of generated summaries of various reading levels for different methods to the Appendix in the paper.\n\n\n[1] Better Rewards Yield Better Summaries: Learning to Summarise Without References (Böhm et al., EMNLP-IJCNLP 2019)\n \n[2] Recursively Summarizing Books with Human Feedback (Stiennon et al., NeurIPS'20) \n\n[3] Learning to summarize from human feedback (Wu et al., NeurIPS'20)\n\n[4] Learning to Paraphrase Sentences to Different Complexity Levels (Chi et al., 2023) \n\n[5] How to Write Plain English (Flesch, Rudolf. University of Canterbury. 1979) \n\n[6] Scaling Instruction-Fine Tuned Language Models (Chung et al. 2022) \n\n[7] Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model (Pal & Baral, EMNLP Findings 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and well-organized, making it easy to follow.", "reasons_to_reject": "1. The proposed method seems not very new. Prompting from LLM to do data augmentation, and applying PPO both are very common techniques.\n2. The experimental settings might have some issues. For readability, a older person should understand all the contents within or lower his/her age range, which seems more reasonable. However, here the authors limited the people of given age can only understand the corresponding summaries. It seems unreasonable for me,", "questions_to_authors": "1. For score based instruction, what does the score here mean? When you want to prompt a summary, what scores would you use? Would the space be too sparse as good summarizations are limited for each document, but the score is a real-value ranging from 0 to 100, say a large space.  I guess the <summary, score> pair space would be very sparse.\n2. Can author show more cases about the generated summary in different readability level?", "paper_topic_and_main_contributions": "This paper proposes to generate text summarization with controlled readability. First the proposed method prompts summaries with different readability from LLM. Then it tunes the policy model with PPO to achieve controllable ability. The method is evaluated on CNN Dailymail, and shows better results than GPT 3.5.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iEykaxHrFZ", "reviewer": "Reviewer_MNBZ", "summary": "", "questions": "See Reasons to Reject", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and positive comments, and for appreciating our automatic and human evaluations and ablation studies, and that controlling aspects beyond abstractiveness and factuality has been an understudied area and interesting contribution.\n\n**- About readability control using FRE**\n\nIn our experiments, we find that optimizing toward higher FRE readability scores leads to summaries that contain words that are shorter and easier to read and sentences that are shorter. We do not observe that it leads to very short summaries overall. Importantly, note that (i) instruction-based methods are trained on reference summaries; (ii) the PPO method incorporates a KL penalty between the initial policy and the updated policy making sure the outputs look like CNN/DM summaries; and (iii) the Lookahead method is controlled by a weight in the score function (Equation 4, Line 311 - Line 313), which mitigate such issues.  In what follows, we present the average length of the generated summaries for different levels:\n\n|      | 11-year-old | middle school | high school| college|\n| ----------- | ----------- |----------- |----------- |----------- |\n| CategoryInstruct | 43.6 | 49.4 |   51.5 |   46.4 \n| ScoreInstruct | 35.2 | 46.9 |  47.5 |  42.6 \n| ScoreInstruct+RL | 55.9|  56.3 | 54.8 | 52.5 \n| ScoreInstruct+LA | 34.5 |  42.5 | 46.7 | 40.0\n\nThere is a slight difference in terms of number of words between the more readable and less readable summaries, although the higher readable summaries are not very short. Below we give an example of summaries with the lengths for different readability levels:\n\n**Document:**\n…which Premier League club is the most popular according to Twitter followers. For the first time ever, fans can see a detailed breakdown of how support for every club varies around the world and in the UK supporters can go as in-depth as seeing the three most popular teams in their local constituency…. Liverpool may be struggling … but in terms of Twitter followers they dominate the UK. Arsenal have strong support … Chelsea and Tottenham. In Manchester, it is a win for Louis van Gaal's …\n\n**Requested Flesch Reading Ease score: 90 (easily understood by an 11-year-old student).**\nSummary: Premier League Twitter followers are the new tool to help fans make up their mind. The map shows which club is most popular in each country in the world. Liverpool has the most followers in the UK, but is less popular in other areas. Arsenal are the most popular team in the world with 5.6m followers.\n**Length:** 56, **Observed readability scores:** Flesch Reading Ease (higher more readable): 76.5, Gunning Fog Index (lower more readable): 10.2, Coleman-Liau Index (lower more readable): 7.0.\n\n**Requested Flesch Reading Ease score: 70 (Plain english, fairly easy to read).**\n**Summary:**  Supporters can view a breakdown of how each club is supported around the world. Liverpool dominates the UK market but has less support in other areas. Chelsea and Manchester United are the most popular in Asia apart from Thailand. Arsenal are the most popular team in the world with 5.6million followers.\n**Length:** 51, **Observed readability scores:** Flesch Reading Ease: 61.2, Gunning Fog Index: 9.5, Coleman-Liau Index: 10.0.\n\n**Requested Flesch Reading Ease score: 50 (fairly difficult to read).**\n**Summary:**  The new Twitter map was created by looking at the official Twitter accounts for every team, using their followers as an indicator of allegiance. Liverpool dominates the UK market but is less popular in other parts of the world. Arsenal is the most popular team with 5.6million followers, followed by Chelsea (5.4m) and Manchester United (4.8m) across the globe, Arsenal is the most popular team with 5.6million followers, followed by Chelsea (5.4m) and Manchester United (4.8m)\n**Length:** 76, **Observed readability scores:** Flesch Reading Ease: 53.6, Gunning Fog Index: 12.6, Coleman-Liau Index: 8.5.\n\n**Requested Flesch Reading Ease score: 30 (difficult to read).**\n**Summary:**  Supporters of the Premier League can view how their club's global popularity differs. Liverpool dominate UK Twitter following, followed by Arsenal, Chelsea and Manchester United. Supporters in the UK can also see their favourite team in their local constituency.\n**Length:** 39, **Observed readability scores:** Flesch Reading Ease: 33.1, Gunning Fog Index: 11.3, Coleman-Liau Index: 14.9.\n\nFinally, we will cite and discuss the missing reference and add an analysis of the summary length size along with example summaries in the final version given the extra page.\n\n\n**- About controllable text simplification**\n\nThank you for your insightful suggestion. We will include and discuss related work on text simplification and point out the differences to our work in the final version. We note that in contrast to controllable simplification approaches (which aim to control the degree to which a sentence is paraphrased), controllable summarization approaches must provide succinct and informative summaries while maintaining the desired reading level (Line 135 - Line 139).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough automatic and human evaluation demonstrates the effectiveness of the proposed RL and LA approach.\n2. Supporting ablation analysis shows how readability impacts the specificity and informativeness of the target summary.\n3. Controlling aspects of summary beyond abstractiveness and factuality is relatively less explored and this paper makes an interesting contribution in that direction.", "reasons_to_reject": "1. Readability control using FRE and other readability metrics while a reasonable approach is also easily tunable (see [1]). No qualitative analysis is provided to support whether the summaries generated for the 4 levels differ in an expected fashion (similar to what is shown in Figure 1). FRE scores are biased towards the shorter text and hence it's unclear whether the summaries are simpler or just shorter.\n2. While readability control is less explored in summaries, controllable text simplification has received a lot of attention from the NLP community where the goal is to modify an input text with a desired U.S. reading grade level or control other aspects of readability/simplicity. No discussion of existing literature on controllable simplification is presented in prior work.\n\nIn the Rebuttal period, the author provided appropriate responses for points 1 and 2. I would encourage the authors to include the example and discussion of Point 1 in the camera ready version.", "questions_to_authors": "See Reasons to Reject", "paper_topic_and_main_contributions": "This paper presents three approaches for generating summaries with readability control. This is done via using language models with instruction finetuning where the target readability levels are either presented in a categorical or in a score-based fashion. The score-based instruction models are further finetuned with reinforcement learning (RL) or are decoded using a look-ahead (LA) mechanism to achieve further control over the target readability. Results demonstrate that both RL and LA improve control over target readability compared to the base ScoreInstruct model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Flesch-Kincaid is Not a Text Simplification Evaluation Metric (Tanprasert & Kauchak, GEM 2021)", "Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1: 70 should correspond to middle-school"}, "extra_text": {}}, {"review_id": "MuBfz2e74y", "reviewer": "Reviewer_NGJK", "summary": "", "questions": "1. What is the motivation for the GPT-3.5 baseline?\n\n2. Language models often struggle with math and numeric heavy tasks and it seems like you are just encoding the requested score as any other text for “Score-based Instructions”. Have you done any validation to see if the model understands the numbers provided?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for appreciating that our investigation of the tradeoff between readability and specificity is useful and for seeing our work as a foundation for future research in this direction.\n\n**- About GPT3.5 as baseline**\n\nOur motivation for adding the GPT3.5 baseline is that it is widely considered to be a strong text generation model that performs well on a multitude of NLP tasks based on instructions and achieves state-of-the-art results on summarization [1, (Line 044 - Line 047)] and controllable summarization [2, (Line 515 - Line 518)]. We believe that it is considered a strong model despite not using beam search. We addressed your comment and in order to mitigate this issue (GPT3.5 not using beam search), we generate multiple (3) summaries from GPT3.5 for each example and instruction and select the summary whose readability level is closest to the requested readability level (see BestGPT3.5 below; we experiment with 50 examples given the costs to run GPT3.5 on 3 * 4 readability levels * # examples):\n\n| | ΔFRE(↓) | FREρ(↑) | GFIρ(↓) | CLIρ(↓)\n| --- | ----------- |----------- | ----------- | ----------- |\n| GPT3.5 | 24.6|  0.23 |  -0.24|  -0.16\n| BestGPT3.5 | 21.1|  0.28 | -0.30 | -0.15\n| CategoryInstruct |  16.9 | 0.61 |  -0.40 |  -0.55\n| ScoreInstruct | 15.7 |  0.61 |  -0.41 |  -0.56\n| ScoreInstruct+RL | 12.2 | 0.74 | -0.42 |  -0.73\n| ScoreInstruct+LA | 5.4|  0.95 |  -0.65 |  -0.86\n\nAs shown above, BestGPT3.5 improves over GPT3.5 but is still behind our methods in the readability control. Upon acceptance, we will include BestGPT3.5 as an additional baseline.\n\n**- About the differences in instructions between GPT3.5 and our methods**\n\nFollowing previous work [1], we used sentence-count length GPT3.5 prompts to adapt to CNN/DM dataset. We have experimented with many different instructions for readability control in preliminary experiments inspired by examples provided in the GPT models documentation and have selected the best-performing ones for each category. For example, for the category with highest readability (Flesch Reading Ease ≥80), we experimented with following instructions on CNN/DM 50 examples:\n\n**Instruction:** Summarize the above article in 3 sentences for a sixth-grade student.\n**Readability metrics:** Flesch Reading Ease: 43.9, Gunning Fog Index: 13.4, Coleman-Liau Index: 12.4\n\n**Instruction:** Summarize the above article in 3 sentences for a five-grade student.\n**Readability metrics:** Flesch Reading Ease: 39.3, Gunning Fog Index: 14.4, Coleman-Liau Index: 13.0\n\n**Instruction:** Summarize the above article in 3 sentences for an average 11-years-old student.\n**Readability metrics:** Flesch Reading Ease: 40.1, Gunning Fog Index: 13.9, Coleman-Liau Index: 13.1\n\nWe selected the first instruction which gave the higher readability average scores for all metrics. We will describe all preliminary prompt variants in Appendix. Finally, our findings (Line 392 - Line 399) that GPT3.5 is not strong in fine-grained readability control are in line with recent findings by Pu and Demberg (2023).\n\n**- About the motivation**\n\nYou suggest that we add better motivation for requesting lower readability levels (e.g., college level), e.g., by stressing earlier in the text that high readability levels correlate with lower specificity and interestingness; we will describe this point further and add it to the Motivation section. Additionally, our experiments are on a subset of CNN/DM articles that have lower readability to begin with (high school and college levels), in order to minimize cases where we request a summary with a lower readability level than the input article (Line 328 - Line 333).\n\n\n**- About the numerical scores in the instructions**\n\nIt is true that LLMs tend to struggle with numerical tasks. However, we expect that the Flan-T5-Large model, which has close to 1B parameters and has been extensively pre-trained and fine-tuned (including arithmetic reasoning tasks [3]) and can learn meaningful representations of simple integer numbers under 100 [4] which is not a heavy mathematical task. Figure 4a presents evidence that our score-based methods are capable of generating summaries with different readability levels based on the distinct numerical scores presented in the instructions in a monotonic way. As shown in Table 2, we find strong Pearson correlations between the requested score in the instructions and the observed readability; for example, 0.62, 0.74 for ScoreInstruct and ScoreInstruct+RL, demonstrating that our ScoreInstruct methods are able to follow instructions based on numbers.\n\n\n**- About the presentation improvements**\n\nThanks for the helpful feedback on the text and image improvements, which we will incorporate into the camera-ready version. We will move fair payment and other details about our Mechanical Turk setup to the Ethics statement. \n\n[1] News Summarization and Evaluation in the Era of GPT-3 (Goyal et at. 2023)\n\n[2] Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization (Yang et al. 2023)\n\n[3] Scaling Instruction-Fine Tuned Language Models (Chung et al. 2022) \n\n[4] Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model (Pal & Baral, EMNLP Findings 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I believe that the tradeoff between readability and specificity offered by this variant of the summarization task would be useful\n2. The proposed methods seem like they would be useful a foundation for future work on this task", "reasons_to_reject": "1. I don’t think the comparison to GPT-3 is fair\nIt is evaluated very differently from other other models, so it is not possible to tell if the difference in performance is due to a difference in evaluation:\n\na. Other models are decoding using some sort of beam decoding (beam search?)\nGPT-3 uses sampling with temperature=0, which should be equivalent to greedy search (greedy search is know to perform worse than beam search)\n\nb. A different prompt is used for GPT-3 than for the Category-based Instructions method, were both prompts tried on both methods and the best performing prompt selected? (the difference would be sense in this case, but this is not clearly indicated in the paper)\n\nI would consider trying to make this baseline more comparable to the proposed methods or excluding it\n\n2. Motivation of work is weak\nYou motivate higher readability (“Higher readability lowers reading efforts and increases the speed for any reader”), but you do not motivate why lower readability would be desired (You do show that higher readability results in lower specificity, I would mention something about this in your motivation)\n\nEdit: In the authors rebuttal, they include a results for GPT-3.5 with sampling which shows an improvement in performance, but still less than the proposed method.  The authors will also include the methods used to create prompts for this summarization task and include these in the paper's appendix. The authors said that they will make the motivation for the control of readability levels (higher readability levels leading to lower specificity and interestingness) more clear in the final version of the paper.", "questions_to_authors": "1. What is the motivation for the GPT-3.5 baseline?\n\n2. Language models often struggle with math and numeric heavy tasks and it seems like you are just encoding the requested score as any other text for “Score-based Instructions”. Have you done any validation to see if the model understands the numbers provided?", "paper_topic_and_main_contributions": "This paper introduces a variant on the summarization task which looks at generating summaries at a given readability level\n\nThe authors propose 3 baseline methods that provide control over the readability of summaries:\n\n- Instruction-prompting\n\n- Reinforcing learning\n\n- Lookahead decoding\n\nThe authors evaluate these methods including some combination of them and show that these methods can control the readability of the generated summaries\n\nFinally, the authors analyze the tradeoff between readability and other aspects of summaries like specificity, abstractions, factuality and informativeness", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. It is unclear why there would be a benefit to having less readable summaries until the last paragraph of Section 1 (I would try to include earlier and maybe even in your abstract)\n2. I think ethical considerations including annotator pay are an important consideration for the paper, I would recommend moving that from Appendix E to an Ethics Statement Section following your Limitations Section\n3. It is a little hard to compare the methods in Figure 3, maybe flipping the axes or switching to a vertical orientation might help\n4. Not consistent between “GPT-3.5” (line 1009) and “GPT3.5” (line 1007)\n\nSee: “Authors will be allowed extra space after the 8th page (4th for short papers) for an optional broader impact statement or other discussion of ethics.” at\nhttps://2023.emnlp.org/calls/main_conference_papers/"}}]

Author rebuttal:

Meta-review: The paper explores methods for generating text summaries with controlled readability. It is well-written and well-organized. The effectiveness of the proposed RL and LA approach is demonstrated through comprehensive automatic and human evaluations. The tradeoff between readability and specificity in this summarization task also holds promise. While promising, reviewers have revised some moderate concerns about the motivation for controlling readability levels, the novelty of the proposed method, and issues with the experimental setup.
Decision: Accept-Main