Controlling Pre-trained Language Models for Grade-Specific Text Simplification
Authors: Sweta Agrawal
Marine Carpuat
Venue: EMNLP 2023 Main (2023)
Keywords: Controllable Text generation
Text Simplification
TL;DR: We improve the controllability of text simplification based on large language models by predicting control tokens that specify the nature of TS operations needed for each input.

Abstract: Text simplification systems rewrite text to make it more readable while preserving its content. However, what makes a text easy to read depends on the intended readers. Recent work has shown that pre-trained language models can simplify text using a wealth of techniques to control output simplicity, ranging from specifying only the desired reading grade level, to directly specifying low-level edit operations. Yet it remains unclear how to set these control parameters in practice. Existing approaches set them at the corpus level, disregarding the complexity of individual inputs and considering only one level of output complexity. In this work, we conduct an empirical study to understand how different control mechanisms impact the adequacy and simplicity of text simplification systems. Based on these insights, we introduce a simple method that predicts the edit operations required for simplifying a text for a specific grade level on an instance-per-instance basis. This approach improves the quality of the simplified outputs over corpus-level search-based heuristics.

Controlling Pre-trained Language Models for Grade-Specific Text Simplification
Sweta Agrawal
Marine Carpuat
Department of Computer Science University of Maryland sweagraw( umd. edu
Department of Computer Science
University of Maryland mar i n umd. edu
Abstract
Paracho, the "guitar capital of Mex- ico," makes nearly 1 million classical guitars a year, many exported to the United States.
Original:
Text simplification (TS) systems rewrite text to make it more readable while preserving its con- tent. However, what makes a text easy to read depends on intended readers. Recent work has shown that pre-trained language models can simplify text using wealth of techniques control output simplicity, ranging from spec- ifying only the desired reading grade level, to directly specifying low-level edit operations. Yet it remains unclear how set these control parameters in practice. Existing approaches set them at the corpus level, disregarding the complexity of individual inputs and consider- ing only one level of output complexity. In this work, we conduct an empirical study to understand how different control mechanisms impact the adequacy and simplicity of text sim- plification systems. Based on these insights, we introduce a simple method that predicts the edit operations required for simplifying a text for specific grade level on an setaace-per-instance basis. This approach improves the quality of the simplified outputs over corpus-level search- based heuristics.
Grade 5: Paracho is known as the "guitar cap- ital of Mexico," The town makes nearly 1 mil- lion classical guitars year, with many exported to the United States.
Word Length Ratio (W) 1.19
Dependency Tree Depth Ratio (DTD): 1.50 Grade 3: Paracho is known as the "guitar capi- tal of Mexico," The town makes many guitars and sells some in the United States.
Word Length Ratio (X) 0.96
Dependency Tree Depth Ratio (DTD): 1.00
Figure 1: Simplified texts can be obtained by either specifying the target audience (via grade level) or by using low-level control tokens define the TS operation to be performed relative to the complex text (W, DTD),
degree of simplicity of the output at different lev- cls. At a high level, one can simply specify the desired reading grade level of the output (Scarton and Specia, 2018; Kew and Ebling, 2022). At a low level, one can control complexity by describ- ing the nature of simplification operations to be performed (Mallinson and Lapata, 2019; Martin et al., 2020). For example (Figure 1). one could ob- tain two distinct simplifications of the same inputs by indicating that they are intended for a grade 6 vs. grade 3 audience, or by specifying values for low-level control tokens such as the word length ratio (W) between the source and the target and the maximum dependency tree depth (DTD) ratio between the source and the target. For original complex text grade when simplifying grade 5, the low-level control values indicate a conserva- tive rewrite, whereas, for grade 3, the properties encoded by the control tokens reflect a relatively more lexical and structural change.
1 Introduction
In the NLP task of text simplification, systems are asked to rewrite, restructure modify original text such that it improves the readability of the original text for a targel audience while preserving its meaning. However, text can be simplified in many different ways and what makes text simple to read depends on the reader. Replacing complex or specialized terms with simpler synonyms might help non-na speakers (Petersen and Ostendorf, 2007; Allen, 2009), restructuring text into short sentences with simple words might better match the literacy skills of children (Watanahe et al., 2009). Acknowledging that text simplification is highly audience-centric (Stajner, 2021), recent work has focused on developing techniques to control the

---

While specifying a reading grade level might be more intuitive for lay users, it provides weaker control over the nature of simplification to be per- formed. On the other hand, controlling the outputs' simplicity by setting several low-level properties, such as the number of words or dependency tree depth, provides finer-grained control but can be cumbersome to set by readers, teachers, or other users. As result, it remains unclear how to op- erationalize the control of text simplification in practice. Prior work sets low-level control values (length, degree of paraphrasing, lexical complex- ity, and syntactic complexity) at the corpus level by searching for control token values on a devel- opment set. This is done via maximizing a utility computed using an automatic evaluation metric, SARI, metric designed measure lexical sim- plicity (Xu al., 2016). While this approach is ap- pealing in its simplicity, it remains unclear whether this approach actually helps control complexity for individual inputs, as the control token values are always set at the corpus level.
the complexity of the output language specific audience (Stajner, 2021). Controllable Text Sim- plification can be framed as a conditional language modeling task, where the source text X is rewritten as an output Y that presents attributes V as scored by model P(Y X, V) (Prabhumoye et al., 2020), In sequence-to-sequuee
models, techniques to con- trol the properties V during generation fall under two categories depending on whether they modify the training process (Sennrich et al., 2016; Holtz- man al., 2018; Dathathri et al., 2019; Li et al., 2022) as described below, or are supplied as con- straints during inference (Hokamp and Liu, 2017; Ghazvininejad et al., 2017; Kumar et al., 2021).
Control Token Mechanisms A straightforward method to capture a target attribute, V. in text gen- cration models is to represent it as a special token appended to the input sequence, [V; X]. which acts as a side constraint Sennrich al. (2016). These constraints can be appended the source or the target sequence. ? The encoder learns a hidden rep- resentation for this token as for any other vocabu- lary token, and the decoder can attend to this rep- resentation to guide the generation of the output sequence. This simple strategy has been used to control second-person pronoun forms when trans- lating into German (Sennrich al., 2016), formal- ity when translating to French (Niu al., 2018), the target language in multilingual scenarios (Johnson et al., 2016) and to control style, content, and task- specific behavior for conditional language models (Keskar et al., 2019).
This work presents a systematic empirical study of the impact of control tokens on the degree and quality of simplifications achieved the instance level as measured by automatic text simplifica- tion metrics. Our empirical study shows that most corpus-level control tokens have an opposite im- pact on adequacy and simplicity when measured by BLEU and SARI respectively. As a result, se- lecting their values based on SARI alone yields simpler text at the cost of misrepresenting the orig- inal source content. To address this problem, we introduce simple models to predict what control tokens are needed for a given input text and a de- sired grade level, based on surface-for form features extracted from the source text and the desired com- plexity level. We show that the predicted low-level control tokens improve text simplification trollable TS task compared to corpus-leve' levell search- based optimization.
We provide overview of the control tokens introduced in prior work for text simplification in Tables I and 2. Coarse-graing control over the degree and the nature of the simplification, e.g. via source and target grade levels is easier to inter- pret by end users (Table 1, [1-6, 12]), whereas controlling multiple low-level attributes (Table 1, [7-11]) that map text simplification operations to specific properties of the input and the output text can provide better control over the generated text. However, it is unclear how those low-level control values should be set during inference as these could vary significantly based on the source
con-
2 Background on Controllable Text Simplification
While text simplification has been primarily framed task that rewrites complex text in simpler lan- guage in the NLP literature (Chandrasekar al., 1996; Coster and Kauchak, 2011; Shardlow, 2014; Saggion, 2017: Zhang and Lapata, 2017), in practi- cal applications, it is not sufficient to know that the output is simpler. Instead, it is necessary to target
Pleease refer to Prabhumoye et al. (2020) for a full review on controllable text generation techniques.
II is important to note that the term "control" used in this paper does not imply strict constraint enforcement on the model's 'ss output. Rather, the control tokens or side constraints merely serve as inputs that influence the model's 's hehavior and encourage the generation of outputs with desired properties.

---

PAPER-ID
PAPER
CONTROL TOKENS
How SET?
[1]
Scarton and Specia (2018)
Target Grade and/or Operations
User-defined or Prodicted
[2]
Nishihara al. (2019)
[3]
Agrawal el al. (2021)
Target Grade (TG)
[4]
Yanamoto et al. (2022)
User-defined
[5]
Zetsu et al. (2022)
[6]
Agrawal and Carpuat (2022)
Source Grade
[7]
Martin et al. (2020)
ACCESS (c,
WR, DTD)
[8]
Sheang and Saggion (2021)
Corpus-level Optimization with SARI
[9]
Martin et al. (2022)
1 RL
[10]
Maddela et al. (2021)
cc
Average over the training dataset
[11]
Qiao al. (2022)
10 Psycho-linguistic Features
Corpus-level Optimization with SARI
[12] Kew and Ebling (2022)
per grade-level
Hyperparameter Tuning with SARI
Table I: Control tokens define the nature and degree of simplifications either at a coarse-grained level such as specifying a target grade or via multiple low-level attributes like ACCESS, The control values are typically provided by the users set apriori during inference.
ID
NAME
DESCRIPTION
C
NbChars
character length ratio between source and target.
L
LevSim
character-level Levenshtein similarity (Levenshtein, 1966) be- tween source and target.
WR
WordRank
ratio of log-ranks (inverse frequency order) between source target.
DTD
DepTreeDepth
maximum depth of the dependency tree of the source divided by that of the target.
W
NbWords
word length ratio between source and target.
RL
Replace-only LevSim
character-leved Levenshtein similarity only considering replace operations between source and target. percentage of copying between source and the target
CC
Copy Control
Table 2: Control Tokens introduced in prior work cover a wide range of TS edit operations.
text and the degree of simplification required. In all prior work (Martin et al., 2020, 2022: Sheang et al., 2022; Qiao et al., 2022), these values are set and evaluated at the corpus level. This is achieved by doing hyperparameter search, optimizing for a single metric SARI on the entire validation set, SARI measures the lexical simplicity based on the n-grams kept, added, and deleted by the sys- tem relative to the source and the target sequence. We identify two key issues with this corpus-level search-based strategy for setting control values as described below:
dependent on all these factors exhibited by a large variation observed in the values of the con- trol tokens both the corpus level (Figure 7) and individual target grade levels (Figure 9).
Costly Hyperparameter Search Searching for control tokens value the corpus-level is an ex- pensive process. Martin al. (2022) use the One- PlusOne optimizer with a budget of 64 evaluations using the NEVERGR AD library to set the 4 AC- CESS hyperparameters (up to 2 hours on single GPU). Sheang et al. (2022) select the values that achieve the best SARI on the validation set with 500 runs. This takes >= 3 days when training the model takes only 10-15 hours. As these values are domain and corpus-specifie, optimizing these val- ues even at the corpus level for multiple datasets is computationally expensive.
Input Agnostic Control Setting these control values the corpus level disregards the nature and complexity of the original source text. It does not account for what can and should be simplified in a given input (Garbacea et al., 2021) and to what extent. We show that the control values are indeed

---

We provide an analysis of the impact of these control values defined the corpus level on the degree and nature of TS performed at the instance level in the next section.
12.5
3 How do Control Tokens Impact TS?
7.5
Predictes
Study Settings We study the impact of setting the low-level control values at the corpus level on the instunce-level simplification observed using au- tomatic text simplification metrics. We conduct our analysis on the Newsela-grade dataset (Agrawal and Carpuat, 2019), which consists of news arti- cles associated with multiple reference simplifi- cations for diverse reading grade levels, and thus lets us analyze the degree and nature of simplifica- tion observed across inputs and target readability levels. This data is collected from Newsela, 3 an instructional content platform meant to help teach- ers prepare a curriculum that matches the language skills required at each grade level and has been used in prior work to benchmark controllable TS models (Scarton and Specia, 2018; Nishihara et al.. 2019). It includes up to 4 text rewrites various complexity levels (defined by U.S. reading grade levels 2-12) for an originally complex text. We use the control tokens defined in Sheang et al. (2022) (see Table 2 [1 added to the source text as a side constraint in the format, w_() C_ L_()
2.5
-25
20
Referinnc ARI
Figure 2: ARI accuracy on the Newsela-grade Devel- opment Set: 12%. Setting corpus-level control values results in over under-simplification.
(Flesch Reading Ease): Most control tokens have an opposite impact on SARI and BLEU, except the character length ratio, (C). This suggests that set- ting their values by optimizing for SARI alone at the corpus level can misleading, as a high SARI score can be achieved at the expense of a lower adequacy score. These findings are consistent with the observation of Schwarzer and Kauchak (2018) who note similar negative correlation between human judgments of simplicity and adequacy and caution that: "improvement in one metric and not the other may be due to this inverse relationship rather than actual system performance". These re- sults lead us to concur with the recommendation Alva-Me waa-Manchego et al. (2021), which advocates for al always augmenting SART with an adequacy metric for text simplification evaluation.
(R DTD_() (Source_text)
Instance-Level Findings Following prior work (Martin et al., 2020), we select the control tokens that maximize SARI on the Newsela-grade devel- opment set. We measure the complexity of the outputs generated and compare them with the com- plexity of the Newsela references by computing the Automatic Readability Index (Senter and Smith, 1967, ARI, Refer Equation 1).
Overall, this analysis highlights important lim- itations of the simplification abilities provided by setting control tokens based optimizing SARI at the corpus level. We propose instead simple method predict these values based on each input instance and the desired output complexity.
As can be seen in Figure 2, control tokens set the corpus level using SARI tend to over or under- simplify individual input instances. When the refer- ence distribution exhibits diversity in the degree of the simplification performed, setting corpus-level values is sub-optimal: outputs are frequently over- or under-simplified as illustrated by the difference in the reference and predicted grade levels.
Grade-Specific Text Simplification with Instance-Level Control
Since the simplification for given instance should depend on the original source text, its complexity, and the desired target complexity, we introducc Control Predictor module (CP) that predicts a vector of control token values V for each input X at inference time. Figure 4 shows the overall inference pipeline for generating the simplified text using the control token values predicted using C.P.
Corpus-Level Findings Figure 3 shows the cor- relation between 100 sets of control values set at the corpus level and automatic TS metrics computed using the outputs generated: SARI, BLEU, and FR
Pnewsela.c com

---

0.6
W
0.35
-0.26
-0.72
0.4
-0.16
-0.33
0.26
40.0
0.2
37.5
WR LevSim
0.0
-0.64
0.63
0.48
35.0
-0.2
32.5
-0.09
0.31
0.23
30.0
-0.4
DTD
0.36
-0.6
20
30
40
BLEU
BLEU
Figure 3: Adequacy-Sa -Sipplicy Tradeoff the Newsela- Grade development when using 100 different control tokens set at the corpus level: Most control tokens have an opposite impact on BLEU and SARI, suggesting that setting their values on SARI alone can be misleading.
Feature Extraction
C
Source Text
Control Predictor
TS Model
Output
Target Grade
WR
Figure 4: At inference time, low-level control tokens are first estimated via the control predictor using the source text and a user-defined target grade level. The low-level tokens are then fed as input to the TS generation model to produce simplified output.
Predicting Control Tokens We thus directly
Number of Characters
train a Control Predictor (CP(O): X + V) to pre- dict the control vector given features extracted from input text and the input and output grade levels. Let {a;, y4 D represent complex-simpl. pair, and the ACCESS controls associated with this pair be V; (W; C₁, Li, W RG, DT We propose both single and multi-output regression solutions for predicting as described below:
3. Maximum Dependency Tree Depth
4. Word Rank
Mean Age of Acquisition (Schumacher al., 2016)
We incorporate the source and target grade levels as attributes to accommodate the differences in control token values resulting from the level of simplification needed.
1. CP-Single: The model is trained to predict the individual control tokens, resulting one model per control value.
TS Model Training Given source text (z) and a control vector v, the controllable TS model P Pyyl,, v). is trained to generate a simplified out- put (1) that conforms to u in a supervised fashion, by setting v to oracle values derived from the refer- ence and optimizing the cross- cross-entropy loss on the training data.
2. CP-Multi: The model is trained to optimize the mean RMSE error over the control dimensions.
We train a simple feature-based Gradient Boost- ing Decision Trees classifier 4 to predict the control values, V using the CatBoost library using several surface-form features extracted from the source text deseribed below:
Experimental Settings
Data We use the Newsela-grade dataset (Agrawal and Carpuat, 2019) with 470k/2k/19k samples for training, development and test sets respectively.
1. Number of Words
Metrics We automatically evaluate the truecased detokenized system outputs using:
"httos://catboon al/en/

---

1. SARI (Xu et al., 2016), which measures the lexical simplicity based on the n-grams kept, added, and deleted by the system relative to the source and the target."
Controllable TS Baselines We compare our approach with the corpus-level hyperparameter search strategy (CORPUS-LEVEL) used in prior work that selects the best low-level control values based on SARI only (Martin et al., 2020).
2. BERTSCORE (Zhang et al.) for assessing the output quality and meaning preservation
Source Grade at Inference While the desired target grade level known during inference, we automatically predict the grade level of each source sentence using the ARI score in all the settings.
3. ART-Accuracy (Heilman al., 2008) that represents the percentage of sentences where the system outputs' ART grade level is within grade of the reference text, where ARI is computed as:
Results
We first discuss the accuracy of the Control pre- dictor in estimating the ACCESS control values the Newsela- Gradee dataset and then show the impact of using the predicted control tokens constraints towards controlling the degree of sim- plification in the generated outputs.
ARI 1.71( chars
words, -21.43.
+0.5 sents
words
(1)
4. % Unchanged Outputs (U) The percentage of outputs that are unchanged from the source (i.e., exact copies).
6.1 Intrinsic Evaluation of Control Predictor
We evaluate the fit of the control predictor in pre- dicting V using RMSE and Pearson Correlation with the gold ACCESS values.
CORRELATION())
CONTROL
RMSE
LevSim
WR DTD
CP-SINGLE 0.405 0.407 0.567 0.398 0.567
0.197
CP-MULTI 0,420 0.422 0.568 0.393 0.570
0.196
Model Configuration We finetune the T5-base model following Sheang et al. (2022) with default parameters from the Transformers library except for batch size of 6, maximum length of 256, learn- ing rate of 3e-4, weight decay of 0.1, Adam epsilon of le-8, 5 warm-up up steps, and 5 epochs. For gen- eration, we use beam size of 8. We train all our models on one GeForee RTX 2080Ti GPUs. Train- ing takes 7-8 hours converge.
Table CP-Multi improves correlation (averaged over 10 runs) with ground truth low-level control token values (M, C) on nowsela-grade development set over CP Si Single.
Table shows the correlation and RMSE of pre- dicted values with gold low-level control tokens. Training the model to jointly predict all values, V improves correlation (+0.015) for W, C over training independent models (CP-Single) for individual control tokens. We show that this can be attributed to the correlation amongst the target control values in Figure 5. Both (W, C) exhibit moderate correlation with DTD. There is drop in correlation for when training joint model which is expected as WR is a proxy for lexical com- plexity and is the most independent control token. The correlation scores for the control tokens (W, C, DTD, LevSim, WR) range from -0.33 ($_w, W) to 0.49 (TG, LevSim). The mod- erate correlation between the source features (prepended with S) and the low-level tokens sug- gests that the source text influences the nature of simplification that can be performed. Additionally, LevSim controls the degree of simplification as suggested by its moderate- high correlation with
We use a learning rate of 0.1 and a tree depth of 6 for training all the control predictor models which takes approximately 5-10 minutes.
Controllable TS Variants We compare the pradiction-based TS models above with two vari-
ants:
GRADE TOKENS: model that high- level control token values, i.e. the source grade (SG) and the target grade (TG) levels when finetuning the generation model (Scar- ton and Specia, 2018).
AVG-GRADE: simple approach that sets con- trol values with the average of the values ob- served for the source-target grade pair.
>https://g; thwb.

---

based models (CP Singli, CP- EP-Multi) that control tokens for each instance based on source features improve simplicity scores com- pared to using Avg-Grade, which only uses grade information to set control values. These models show improvements in SARI (+1.4-1.55 and ARI (+3.6%) scores, highlighting the impor- tance of setting control tokens at the instance level rather than relying solely just the grade in- formation. Furthermore, setting control tokens based on the average values observed for a given source-target grade pair, i.c., Avg Swwg-Graaee GGaad sig- nificantly improves both BERTSCORE and ARI- based metrics across the board compared to the Co corrsleeee approach.
mm
1111
WR
0.00
0.03
DT0
033
0.30
0.09
16
0.17
0.20
49
0:32
0.22
5,0
0.30
0.31
0.19
4.00
4.30
0.11
3,02
4.33
4..
4.:00
4.34
0.06
0.90
19R
000
0.00
0.02
0.29
0.00
0.14
0.05
0.09
5,0TD
0.06
0.06
0.01
-0.00.4.0000
4.00
0.25
0.92
0:30
0.0
S,A0A
0.03
0.03
0.10
0:31
9.00
0.25
0:07
02
0.50
0.22
W
DTD
10
8.6
5.4
5,07D
Figure 5: Correlation between source features and con- trol token values on the Newsela-Gran training set.
Grade-level (high) and operation-specific (low) control tokens exhibit different adequacy and simplicity tradeoffss. Low-level control tokens offer more precise control over the simplicity of outputs, resulting improved SARI scores by at least points compared to Grade Tokens. However, this advantage comes at the cost lower adequacy (BERTSccore) and control over desired complexity (ARI Accuracy). The models trained with low-level control values exhibit lower grade accuracy scores partly due to the limited repre- sentation of the need for text simplification (Gar- bacea al., 2021) during the generation process as suggested by lower percentage of exact copies in the output compared to Grade Tokens (Ex- act copies in references: 12%). On the subset the test set with no exact matches between the source and the reference text, Grade Tokens and CP-Mult receive ARI accuracy of 34.2 and 34.0 respectively. Furthermore, we hypothesize that the models trained with low-level control ex- hibit low meaning preservation because none of the control tokens directly encourage content addition during text simplification. And, while the model learns to perform an appropriate content deletion, it does not generate fitting substitution or addition as required to preserve the meaning of the original source text. We show detailed operiti analysis in the following section.
the target grade level and can be considered the most prominent token for balancing the adequacy- simplicity tradeoff.
6.2 Overall Grade-Specific TS Results
We show how the different control tokens as side constraints influence the degree and the nature of simplification in the generated outputs in Table 4.
CONTROL
BERTSCORE SARI % Acc %U
Low-level
CORPUS-LEVEL
0.922
42.19
3.1
AYG-GRADE
0.945
44.09
32.7
0
CP-SINGLE
0.946
45.54
36.3
2
CP-MULTI
0.946
45.65
36.3
3
High-level
GRADE TOKENS
0.955
43.02
39.8
ORACLE
0.955
53.53
56.7
12
Table 4: Results on the Newsela-gras grade dataset: using source-informed tokens (CP-*) significantly improves SARI over alternative control mechanisms. All dif- ferences are significant except the difference between CP-Single and CP-Multi with p-value of 0.00.
Setting corpus-level control for grade-specific TS is suboptimal. Optimizing SARI alone for selecting the low-level control tokens and setting corpus-level control values is suboptimal for match- ing the desired complexity level. This is indicated by the low ARI accuracy of only 3.1%.
6.3 Impact Control Tokens on TS Edit Operations
Predicting control tokens for individual in- stances improves coverage over the range con- trol values exhibited by the oracle. We show the distribution of control values observed by differ-
Predictor based instance-level control outper- forms grade or corpus-level control. Predictor-

---

Gredk
Grode
Cooaal lood
20
Targnt Crads
Targo Crads
Figure 6: Edit Operations by Target Grade Levels: CP-Single performs correct and diverse edits as suggested by the high Add-FI and Del-P scores all target grade levels
ent ways of setting the low-level control values in Figure 7. The high variance in the range of oracle values confirms that the control tokens vary based the source texts' complexity and desired output complexity. Where Corpus-lewe match the mean distribution, CP-Multi is able to cover wider range of control values. We show that this trend holds across all target grade levels in the Appendix Figure 9.
CP UP-Multi consistently achieves better or com- petitive DEL-P across all target grade levels over alternative control mechanisms, suggesting that set- ting control values informed by both the source and desired complexity level improves the model's ability to appropriately delete redundant informa- tion. The former also generally improves ADD-F1 scores, highlighting that the model also appropri- ately performs lexical substitution or content addi- tion as required across different grade levels (ex- cept grades 2 and 10). Moreover, low-level control tokens (CP (CP-Mult:)
fails
Corpus-level
Aug Grade
Avg-Grade) exhibit more diverse and correct modifications compared to high- level control (Grade Tokens), evident from their better ADD-F1 and DEL-P scores for grade lev- els > 3, where the latter prioritizes meaning preser- vation (high KEEP-F1).
CP-muli
Oracle
Contro
Conclusion
We present a systematic analysis of the impact of control tokens set at the corpus level on the degree and quality of simplification achieved by control- lable text simplification models at the instance level. Our findings show that control tokens exhibit an opposite correlation with adequacy and simplic- ity. Hence, selecting their values at the corpus level based on SARI alone leads to simplifying individual instances. This motivates new approach set low-level control tokens during inference by predicting them given a source text and desired target grade level. We show that this approach is effective at improving the quality and controlling the degree of simplification in gener- ated outputs based on automatic evaluation. Fur- thermore, predicted low-level control tokens yield more diverse edit operations than alternative ways of setting control the Newsela-grade dataset.
W
WR
DTD
Type
Figure 7: Distribution of control values for different control mechanisms: CP CP-multi provides a broader coverage of control values observed in the oracle distribution over Corpus-level and Avg-Grade.
under-
Simplified outputs generated using predicted control tokens exhibit diverse edit operations. Figure 6 shows the distribution of the KEEP-F1, DEL-P, and ADD-F1 scores by target grade level for the models trained with different control types, where ADD-F1 computes the F1 score for the n- grams that are added to the system output relative the source and the reference text. The model's deletion capability is measured by the Fl score for n-grams that are kept (KEEP-FI) and the precision deletion operation (DEL-P) with respect to the source and the reference.
Our proposed simple solutions improve the in- ference capability of the controllable TS model

---

for grade-specific TS and reduce the gap with the oracle over a corpus-level baseline approach. How- ever, more sophisticated techniques can benefit the design and prediction of low-level control values and their usage during inference which we leave to future work.
Acknowledgments
We thank Eleftheria Briakou, Neha Srikanth, the members of the CLIP lab UMD, and the anony- mous EMNLP reviewers for their helpful and con- structive comments. This research is supported in part by the Office of the Director of National Intel- ligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Pro- gram contract 2022-2207220000, by NSF grant 2147292, and by funding from Adobe Research. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, ei- ther expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copy- right annotation therein.
Limitations
We note a few limitations of our work. While our proposed strategies are simple and improve the controllability over the generated simplified texts during inference, the models trained with low- level control tokens struggle to identify when a text needs to be simplified compared to the model that uses high-level weak supervision. These results open space for further research in designing end- to-end controllable TS models that are able to take advantage of both high and low-level control to- kens for controlling both the degree and the nature of simplification.
References
Our work is also limited to one dataset and one language (English) and hence studies the mapping between U.S grade level to low-level edit opera- tions. It remains an open question to study how the control predictor would generalize in other settings, datasets, and language pairs.
Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity neural machine translation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 1549- 1564, Hong Kong, China. Association for Computa- tional Linguisties.
Ethics Statement
Sweta Agrawal and Marine Carpuat. 2022. An imita- tion learning curriculum for text editing with non- autoregressive models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7550- 7563, Dublin, Ireland. Association for Computational Linguistics.
This work is conducted in full awareness of and in line with the ACL Ethics Policy. Models, datasets, and evaluation methodologies used are detailed in Section 5. The Newsela dataset was used with permission and appropriate access rights and li- censes. And, we ground our claims by conducting a thorough evaluation and analysis of the outputs generated by the proposed systems (Section 6).
Sweta Agrawal, Weijia Xu, and Marine Carpuat. 2021. A non-autoregressive edit-based approach to control- lable text simplification. In Findings of the Associa- tion for Computational Linguistics: ACL-IJCNLP 2021, pages 3757-3769, Online. Association for Computational Linguistics.
We note that while text simplification systems are designed with the intention of assisting users in better comprehending complex texts, the potential errors introduced by these systems and ambiguous interpretations of simplified text can cause harm to the reader and other stakeholders. The very nature of simplification involves content removal and rephrasing complex concepts, which can some- times result in oversimplification loss of critical nuances. As consequence, users relying solely on simplified texts may develop an incomplete or inac- curate understanding of the subject matter, leading to potential misconceptions or misinterpretations.
David Allen. 2009. A study of the role of relative clauses in the simplification of news texts for learners of english. System, 37(4)5585-59)
Fernando Alva-Mancheeo, Carolina Scarton, and Lucia Specia. 2021. The (un)suitabiliiy automatic evalu- ation metrics for text simplification. Computational Linguistics, 47(4):861-889)
R. Chandrasekar, Christine Doran, and B. Srinivas. 1996. Motivations and methods for text simplification. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.

---

William Coster and David Kauchak. 2011. Simple En- glish Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 665-669, Portland, Ore- gon, USA. Association for Computational Linguis- tics.
Tannon Kew and Sarah Ebling. 2022. Target-level sen- tence simplification as controlled paraphrasing. In Proceedings of the Workshop Text Simplification, Accessibility, and Readability (TSAR-2022), pages 28-42, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguisties.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yu- lia Tsvetkov. 2021. Controlled text generation as continuous optimization with multiple constraints. Advances in Neural Information Processing Systems. 14414542-14554
Sumanth Dathathri, Andrea Madotto. Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach controlled text generation. In International Conference on Learning Representa- tions.
Vladimir losifovich Levenshtein. 1966. Binary codes capable of correcting deletions. insertions and re- versals. Soviet Physics Doklady, 10(8):707-710. Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.
Cristina Garbacea, Mengtian Guo, Samuel Carton, and Qiaozhu Mei. 2021. Explainable prediction of text complexity: The missing preliminaries for text sim- plification. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Pupers), pages 1086-1097, Online. Association for Computa- tional Linguisties.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022. Diffusion- LM improves controllable text generation. In Ad- vances in Neural Information Processing Systems.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pages 43 43,48, Vancouver, Canada. Association for Computational Linguisties.
Mounica Maddela. Fernando Alva- Manchego, and Wei Xu. 2021. Controllable text simplification with ex- plicit paraphrasing. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguisties: Human Lan- guage Technologies, pages 3536-3553, Online. As- sociation for Computational Linguistics.
Michael Heilman, Kevyn Collins-Thoee Thompson, and Max-
ine Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In Pro- ceedings of the third workshop innovative use of NLP for building educational applications, pages 71 71-79. Association for Computational I. Linguistics.
Jonathan Mallinson and Mirella Lapata. 2019. Con- trollable sentence simplitication: Employing syn- tactic and lexical constraints. arXiv preprint arXiv:1910.04387.
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 1535-1546, Vancouver, Canada. Association for Computational Linguisties.
Louis Martin, Éric Villemonte De La Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sen- tence simplification. In Proceedings of the 12th Lan- guage Resources and Evaluation Conference, pages 4689 4698.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022. MUSS: Multilin- gual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 1651-1664, Marseille, France. European Language Resources Association.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learn ing to write with cooperative discriminators. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages (638-1649 Melbourne, Australia. As- sociation for Computational Linguistics.
Daiki Nishihara, Tomoyuki Kajiwara, and Yuki Arase. 2019. Controllable text simplification with lexical constraint loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guisties: Student Research Workshop, pages 260- 266.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu. Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Multilingual Neural Machine Translation System: Enabling Shoo-Shot Translation. 16Xiv:1611.0453. [cs].
Xing Niu, Sudha Rao, and Marine Carpuat. 2018 Multi- task neural models for translating between styles within and across languages. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 1008-1021, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Nitish Shirish Keskar, Bryan McCann, Lay R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. arXiv preprint 1rXiv:1909.000

---

Sarah E Petersen and Mari Ostendorf. 2007. Text sim- plification for language learners: a corpus analysis. In Workshop on Speech and Language Technology in Education.
14th International Conference on Natural Language Generation, pages 341-352, Aberdeen, Scotland, UK. Association for Computational Linguisties.
Sanja Stajner. 2021. Automatic text simplification for social good: Progress and challenges. In Findings of the Association for Computational Linguistics: ACL- IJCNIP 202), pages 1637-26522 Online. Association for Computational Linguistics.
Shrimai Prabhumoye. Alan W Black. and Ruslan Salakhutdinov. 2020. Exploring controllable text generation techniques. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 1-14, Barcelona, Spain (Online). Interna- tional Committee on Computational Linguistics.
Willian Massami Watanabe, Arnaldo Candido Junior, Vinícius Rodriguez Uzèda, Renata Pontin de Mat- tos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluísio. 2009. Facilita: reading as- sistance for low-literacy readers. In Proceedings of the 27th ACM international conference on Design of communication, pages 29-36.
Yu Qiao, Xiaofei Li, Daniel Wiechmann, and Elma Kerz. 2022.
features meet transformer models for improved explainable and controllable text simplificatinn. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 125-146, Abu Dhabi, United Arab Emirates (Virtual). Association for Computa- tional Linguisties.
Wei Xu, Courtney Napoles. Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
Horacio Saggion. 2017. Automatic text simplification. In Synthesis Lectures on Human Language Technolo- gies.
Daiki Yanamoto, Tomoki Ikawa, Tomoyuki Kajiwara, Takashi Ninomiya, Satoru Uchida, and Yuki Arase. 2022. Controllable text simplification with deep re- inforcement learning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 398 404, Online only. Association for Computational Linguis- ties.
Carolina Scarton and Lucia Specia. 2018 Learning Simplifications for Specific Target Audiences. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), volume 2. pages 712-718.
Elliot Schumacher. Maxine Eskenazi, Gwen Frishkoff, and Kevyn Collins- Thiiiisss-Toooonn... 2016. Predicting the relative difficulty of single sentences with and with- out surrounding context. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1871-1881, Austin, Texas. Association for Computational Linguisties.
Tatsuya Zetsu, Tomoyuki Kajiwara, and Yuki Arase. 2022. Lexically constrained decoding with edit oper- ation prediction for controllable text simplification. In Proceedings of the Workshop on Text Simplifi- cation, Accessibility, and Readability (T'SAR22022) pages 147-153, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
Max Schwarzer and David Kauchak. 2018. Human evaluation for text simplification: The simplicity- adequacy tradeoff. In SoCal NLP Symposium.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling Politeness in Neural Machine Translation via Side Constraints. pages 35-40. Asso- ciation for Computational I Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
RJ Senter and Edgar A Smith. 1967. Automated read- ability index. Technical report, CINCINNATI UNIV OH.
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584- 594, Copenhagen, Denmark. Association for Compu- tational Linguistics.
Matthew Shardlow. 2014. A survey of automated text simplification. International Journal of Advanced Computer Science and Applications, 4(1):58-71) 70.
Kim Cheng Sheang, Daniel Ferrés, and Horacio Sag- gion. 2022. Controllable lexical simplification for En- glish. In Proceedings of the Workshop Text Simpli- fication, Accessibility, and Readability (TSAR-2022)) pages 199-206, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
Kim Cheng Sheang and Horacio Saggion. 2021, Con- trollable sentence simplification with unified text- to-text transfer transformer. In Proceedings of the

---

A Impact of Training Data Size on Control Predictor
0.42
0.10
0.40
$.95
10.36
6:94
0.34
0.26
100000
200000
3C0000
400000
100000
200000
300000
400000
100000
200006
500000
400000
0.56
C30
0.55
0.54
0.66
Type
Type
0.12
mut
mutd
single
diile
Figure 8: Correlation scores for all low-level control tokens with varying training dataset sizes.
We vary the size of the dataset used to train the single and multi-regressor control predictors and show the correlation for all the control values, V. in Figure 8. While correlation for CP-SINGLE saturates with 100 150K instances, CP-MULTI able take advantage of correlation amongst tokens and additional training dataset further improve the prediction of ACCESS control tokens.

---

MI-M Thime Grada
Corpui
Corpan
Chhore
2.5
30-
2.5
2.5
2.--
05-
0.0
2.0
070
010
Grade
Grade
MoMel
Model
Modal
Node
Compun
Oerpa
2.8
355
370
WR
010
boe
Cantol
Figure 9: Distribution of control token values for different model variants by Target Grade level.


Reviews:
[{"review_id": "PSL8hRwu9K", "reviewer": "Reviewer_ydY4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We are pleased that the analysis presented in the paper on comparing various control mechanism and their impact on text quality was acknowledged to be thorough. \n\nWe already provide the definitions of the control tokens in Table 2 and illustrate a subset of them in Figure 1. We can illustrate them further by adding examples to illustrate the impact of different control mechanisms on the output in the camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Analysis in the paper is thorough in comparing various methods and demonstrates impact of the proposed framework on the metrics along with various scenarios and how the proposed framework performs. ", "reasons_to_reject": "It would be helpful to add definitions for critical keywords used in the paper like Grades of the simplification, also adding more details on various control tokens instead of just providing the references in table 1.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper authors analyze impact of various control tokens on creating text simplification at instance level with controlable framework, the work also analyzes the limitations of applying control tokens at corpus level. The proposed framework will give user the flexibility to select grade of the output and input along with the input text to get the desired simplified text.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R1pfHJgMK3", "reviewer": "Reviewer_ctbg", "summary": "", "questions": "1. The authors use T5 model to finetune and study the low level control tokens. I’m curious if the authors experimented with any other models, and if they are able to achieve this level of control using large language model prompting. Note that not using LLMs is not a criticism, I am just curious. \n\n2. Why did you choose not to run a human study? Since the automatic metrics do not provide a full picture and end user usage might be different. \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our paper. We are glad that you found our work focused and potentially beneficial to users. We provide responses to your questions below:\n\nQ1 Controllable TS with LLMs: We did not experiment with other models. Our preliminary experiments with large language models showed limited control over the degree of simplification with prompting. We note that the choice of prompts could also have impacted our findings, as LLMs are shown to be sensitive to the prompt design. We leave the full exploration of LLMs (as well as strategies to design prompts to enable controlled generation) to future work.\n\nQ2 Human Study: We followed automatic evaluation of controllable TS following best practices from prior work (cite). Our automatic evaluation (Section 6.2) and a thorough ablation (Sections 3, 6.3) demonstrate a clear advantage of aligning control tokens with Newsela grade levels over the corpus-level SARI-based optimization (Table 4, Section 6.2). We agree that a human study would be very interesting, however, we left that endeavor to future work, as how to soundly design such an evaluation for audience-specific text simplification is an open question in itself which deserves a separate paper. \n\nDomain Switch: Indeed, we did not explicitly test for the impact of domain switch on output quality. Our training and evaluation dataset are sourced from Newsela which includes texts from a range of topics. Like any other text generation tasks, we expect the output quality to drop in unrelated new domains. However, as our proposed control predictor (Section 4) is trained on surface form features like word and character length, we expect the predictor to be more robust to domain switch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors provide an in depth analysis of the usage of controllable text simplification using simple low level tokens, which can be directly beneficial with a target grade token to help people with varying literacy levels to understand and comprehend text. I believe this work is very focused. \n", "reasons_to_reject": "The authors studied one dataset, however it would be interesting to see how a domain switch would affect the usability of controllable text simplification. Further, they used only automatic metrics to measure quality with no human evaluation - which may or may not correlate with the actual downstream users’ preference. \n", "questions_to_authors": "1. The authors use T5 model to finetune and study the low level control tokens. I’m curious if the authors experimented with any other models, and if they are able to achieve this level of control using large language model prompting. Note that not using LLMs is not a criticism, I am just curious. \n\n2. Why did you choose not to run a human study? Since the automatic metrics do not provide a full picture and end user usage might be different. \n", "paper_topic_and_main_contributions": "The task in this paper is text simplification. The authors present a systematic analysis of the usage of setting control tokens at a corpus level vs at a low level. They show that setting low level control tokens during inference, on an instance level based on the source text and target grade level is better at simplification, with a greater level of control and quality on diverse edits as opposed to corpus level tokens commonly used. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PQKfEpYqjM", "reviewer": "Reviewer_jeYZ", "summary": "", "questions": "a. Have you tested with other Control Predictor features? How relevant is each feature?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and valuable suggestions in improving the presentation of the paper and will incorporate those in the camera ready version of the paper. We also appreciate that the reviewer found our evaluation extensive with previous methods. \n\nSelection of Control Features: The character and word length of the source, the source dependency tree depth and the word ratio are features that directly impact the control token values by the definition (Table 2). These are also the features that have been used to determine the complexity of a text in prior work. We further included the age of acquisition, as this feature was shown to have high correlation with sentence level difficulty [1]. We do not examine the impact of individual features via a leave-one-out ablation, but we support their inclusion in a multi-regression framework via a correlation analysis, as depicted in Figure 5, along with a comprehensive discussion in lines 376-399. \n\n[1] Elliot Schumacher, Maxine Eskenazi, Gwen Frishkoff, and Kevyn Collins-Thompson. 2016. Predicting the Relative Difficulty of Single Sentences With and Without Surrounding Context. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1871–1881, Austin, Texas.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed model is simple and achieves good results.\n- The work provides an extensive evaluation of previous methods.", "reasons_to_reject": "- It would be enriching to have an explanation of how the Control Predictor features were selected.", "questions_to_authors": "a. Have you tested with other Control Predictor features? How relevant is each feature?", "paper_topic_and_main_contributions": "The paper conducts an empirical study to evaluate different control mechanisms for text simplification. Also, it introduces a new text simplification method using a feature-based Gradient Boosting Decision Trees on an instance-per-instance basis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Define the abbreviation TS (text simplification) before start using it.\n- Line 50: grade 6 -> grade 5, as Figure 1 indicates.\n- Line 118: remove \"for\".\n- Line 183: upto -> up to.\n- Lines 340-342: remove the sentence \"The learning rate and [...] models.\", it is repeated in the next paragraph."}}, {"review_id": "QpsuSRa4KW", "reviewer": "Reviewer_Sk1b", "summary": "", "questions": "- The high BERTScore suggests that the outputs are nearly the same as the inputs, right?\n- How do you compute the ARI accuracy and how can we interpret the 30%+ accuracy and the gaps?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading of our paper and your acknowledgment of the motivation and design of our study. We provide our response to your questions below and hope it addresses your concerns.\n\nQ1 High BERTScore: A high BERTScore indicates that the outputs have the same meaning as the inputs. While outputs that barely change from the input can get high BERTScores, Figure 6 (% tokens that are kept the same, deleted, or added by each of these systems) shows that this is not the case here. We see the outputs are notably different from the original text across different systems and grade levels while preserving meaning based on the BERTScore.\n\nQ2 Interpretation of ARI Accuracy: We follow Heilman et al. (2008) and compute the accuracy as the % output instances that are within 1-grade level of the desired target grade level.  The higher the accuracy, the output better matches the desired complexity according to an automatic readability index. We report the ARI Accuracy for all models and comparisons in Table 6 and also study how these scores vary across grade differences (i.e., the difference between the source grade and the intended target grade). As discussed in L442-446 our findings show that models that copy text or are more meaning-preserving (e.g. Grade Tokens) can achieve a high or perfect score when the source matches the reference, i.e. when grade difference is low and no simplification is required. Hence a more fine-grained analysis by bucketing the instances based on target grade levels or desired level of simplification can help interpret the gap. We are happy to include such an analysis in the next version of the paper for all grade differences.\n\nComparison between different systems and human study: We show that our proposed control mechanisms outperform the previous corpus-level approach and other baselines for grade-specific TS via a thorough automatic evaluation in section 6.2.  The numbers in Table 4 while close, are statistically significant with a p-value of 0.00 on a test set of size 20k. While we do not show individual qualitative examples due to the complexity of evaluating fine-grained differences between different systems at the instance level across different target grade levels, we measure the impact of the different control mechanisms for generating the correct TS edit operation (keep/add/delete) for individual target grade levels in Figure 6, where we show that our proposed mechanism, CP-Multi achieves better or competitive precision of deletion operation as well as improved F1 score for addition operation across target grade levels over baselines.  Again, we acknowledge that a human evaluation with experts could be interesting, however, designing an appropriate evaluation framework or user study for audience-specific TS is an open question in itself which deserves a separate paper. \n\nUse of Automatic Metrics: We never claim in the paper that automatic metrics are not good. Instead, we question the way of setting control tokens at the corpus level using just a single evaluation metric, which can result in over or under-simplified outputs (Figure 2). We show that most control tokens show an opposite impact on adequacy-based metric BLEU and simplicity metric SARI in Figure 3, which implies that one can improve SARI by setting low values of these control tokens but significantly compromise the adequacy of the generated outputs. Hence, we argue that these values should be set at the instance level based on the source text and the desired target complexity. We also refer the reader to the examples in Table 4 of the original paper [1] where the authors note that a low character-level Levenshtein similarity can result in hallucinated content in the simplified text for some inputs.\n\nIncremental Innovation: As text simplification efforts focus on tailoring content to specific audiences, adaptable control mechanisms have been developed to address various audience needs. However, there's a gap between how these mechanisms are designed and their practical use during inference. Setting these control values globally via corpus-level optimization can lead to both overly simplistic or complex outputs and is also computationally sub-optimal (Sections 2 & 3). To address this, we propose simple but effective methods that use the source text and desired grade level to improve the quality and control on a controllable grade-specific TS dataset, maintaining interpretability and output variation (Section 6). With a small supervised dataset, the proposed control predictor can be trained and adapted for other contexts and audiences. Our novelty thus lies in identifying and developing a method to enhance controllable TS through interpretable and contextually relevant control mechanisms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I appreciate the authors for the good motivation to better understand the effects of different control mechanisms and design the two staged models to infer control tokens first.", "reasons_to_reject": "- There is a lack of examples to demonstrate the effects of different systems. While claiming that the Automatic metrics are not good or even negatively correlated with the simplification quality (i.e. SARI), I would like to see the human evaluations or some real examples.\n- The innovation of the proposed tasks seems to replace the Control-token values with a new set, which is incremental and the results in Table 4 are super close to each other.", "questions_to_authors": "- The high BERTScore suggests that the outputs are nearly the same as the inputs, right?\n- How do you compute the ARI accuracy and how can we interpret the 30%+ accuracy and the gaps?", "paper_topic_and_main_contributions": "This paper first conducted an empirical survey on the impacts of different control mechanisms on the simplicity of texts. They then provide an approach to first predict the simplification edit operation and then do a second stage rewriting of the texts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method that predicts the edit operations required to simplify a text for a specific grade level on an instance-per-instance basis, which improves the quality of the simplified outputs over existing methods.
There are concerns that the proposed method has only been verified on one dataset and there is no human evaluation.
Decision: Accept-Main