Venue: NAACL 2025

# Aligning Sentence Simplification with ESL Learner's Proficiency for

Language Acquisition

 Guanlin Li1,2, Yuki Arase2, Noel Crespi1

###### Abstract

Text simplification is crucial for improving accessibility and comprehension for English as a Second Language (ESL) learners. This study goes a step further and aims to facilitate ESL learners' language acquisition by simplification. Specifically, we propose simplifying complex sentences to appropriate levels for learners while also increasing vocabulary coverage of the target level in the simplifications. We achieve this without a parallel corpus by conducting reinforcement learning on a large language model. Our method employs token-level and sentence-level rewards, and iteratively trains the model on its self-generated outputs to guide the model to search for simplification hypotheses that satisfy the target attributes. Experiment results on CEFR-SP and TurkCorpus datasets show that the proposed method can effectively increase the frequency and diversity of vocabulary of the target level by more than \(20\%\) compared to baseline models, while maintaining high simplification quality. 1

Footnote 1: Codes available at [https://github.com/JumpyPizza/align-sentence-simplification-with-ESL-learner](https://github.com/JumpyPizza/align-sentence-simplification-with-ESL-learner)

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 492–507

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Controlled text simplification considers audience-targeted attributes when generating simplified texts, so that the generated texts do not only meet the criteria of simplicity, but also preserve desired attributes for the targeted audiences. Recent studies on controlled text simplification aimed to help reading comprehension for language learners and employed school grade levels annotated in the training corpus as the simplification target Scarton and Specia (2018); Sheang and Saggion (2021); Agrawal and Carpuat (2023) or text features (number of words, character-level Levenshtein similarity etc.) between source and target sentences Nishihara et al. (2019); Martin et al. (2020).

Different from these studies, we aim to aid language learning and education for English as a Second Language (ESL) learners by simplifying sentences while preserving desirable attributes for language acquisition. We use the Common European Framework of Reference for Languages (CEFR), the world standard definition of language proficiency. Our method is motivated by two classic L2 learning theories: the input hypothesis Krashen (1981) and frequency effect Ellis (2002). The input hypothesis stated that in order for the language acquisition to happen, the textual input which is either too simple or too complex for learner comprehension will not be useful for acquisition. If a learner's current competence is \(i\), then comprehensible input should contain both \(i\) and \((i+1)\) content Mitchell et al. (2019). Frequency theory holds that the frequency of the words and phrases in the input is a key determinant of acquisition, and words with higher frequency in the usage tend to be easier to acquire Ellis and Ferreira-Junior (2009). The key challenge here is the lack of a parallel corpus for training that should provide complex-simple sentence pairs labelled their levels. Parallel sentences of this kind are naturally scarce, and worse, annotation of difficulty levels, in particular, CEFR, is non-trivial and requires language education experts Arase et al. (2022).

To achieve sentence simplification for aiding language learning without a parallel corpus, we propose reinforcement learning on a pre-trained large language model (LLM). Based on the aforementioned L2 learning theories, the proposed method simplifies the complex sentences to the one corresponding to the learner's proficiency level or one level higher (\(i\) and \(i+1\) levels) and increases the coverage (frequency and diversity) of the corresponding level's vocabulary in the generated simplifications. Specifically, we reformulate the controlled simplification task as a lookahead search problem: in the decoding step \(t\), the model searches for the token that satisfies the target vocabulary constraint while also ensuring that future tokens

increase the target vocabulary coverage as much as possible, and the final hypothesis falls into the desired CEFR level. We combine a simple word-match-based heuristic with the supervised sentence-level signal to guide decoding and train the model iteratively using gradient policy optimization to memorize the search strategy that maximizes the overall reward. Remarkably, we eliminate the need for a parallel corpus by utilizing LLMs' language generation capacity for simplification via reinforcement learning. Experimental results show that the method significantly increases the coverage and diversity of the target vocabulary in the outputs by up to \(20\%\) compared to the baselines, while maintaining high simplification quality.

Our primary contributions are twofold. First, we propose the sentence simplification method that aligns generated simplifications with ESL learners' proficiency level on word, phrase and sentence levels and preserves attributes effective for facilitating language learning. Second, our method is easy to deploy and does not require a parallel corpus that is often expensive to create.

## 2 Related Work

We briefly summarize two lines of simplification methods, controlled simplification and reinforcement learning based simplification.

Controlled Simplificationattaches tokens or prompts to the input to control the simplification-related attributes during generation Yang et al. (2023); Agrawal and Carpuat (2023); Sheang and Saggion (2021); Martin et al. (2020, 2022); Scarton and Specia (2018); Chi et al. (2023). While these methods learn to control levels of simplified sentences using a parallel corpus with annotated difficulties, our method controls attributes useful for language learning without a parallel corpus. As opposed to the training-time controlling, Kew and Ebling (2022) adopted FUDGE Yang and Klein (2021), which adjusts the logits of the text generation model during decoding using a classifier, to directly control the attribute of the simplification in the decoding time.

Reinforcement Learning based Simplification has explored controllability by defining rewards based on simplicity-related criteria Zhang and Lapata (2017); Guo et al. (2018); Nakamachi et al. (2020); Laban et al. (2021). The rewards for the objectives are constructed using supervised or unsupervised evaluation metrics for simplicity, adequacy and fluency. In contrast, we aim to control attributes useful for language learning and education. Furthermore, while RL tends to suffer from unstable training and sensitivity to the choice of hyperparameters, our method achieves training stability by adopting entropy regularization in the model optimization process and introducing a dynamic reward that adjusts based on the data distribution.

## 3 Problem Definition

We aim to facilitate language learning by simplification targeted at ESL learners. In this study, we use CEFR levels as a representative measure for the learners' proficiency and model the target level based on the vocabulary2 (words, phrases, idioms) and sentence CEFR levels3.

Footnote 2: [https://www.englishprofile.org/wordlists/evp](https://www.englishprofile.org/wordlists/evp)

Footnote 3: [https://www.coe.int/en/web/common-european-framework-reference-lanaguages/table-1-cefr-3.3-common-reference-levels-global-scale](https://www.coe.int/en/web/common-european-framework-reference-lanaguages/table-1-cefr-3.3-common-reference-levels-global-scale)

Our problem is thus defined as follows. We assume that learners know their own CEFR level \(i\). Given a sentence above the learner's level \(i\), we generate its simplified version that (a) contains as much vocabulary of the level \(i\) and \(i+1\) as possible, and (b) corresponds to the target (learner's) level \(i\) at the sentence level.

### Constraint Formalization

Generating simplified texts subject to vocabulary constraints can be approached as a lexical-constrained text generation task Zetsu et al. (2022). Traditionally, lexical constraints in text generation involve a _short_ list of required words, which Lu et al. (2021) expressed as a Conjunctive Normal Form (CNF), such as \(\underbrace{(D_{1}\lor D_{2}\vee\cdots)}_{C_{1}}\wedge\dots\wedge\underbrace{ (D_{m-1}\lor D_{m})}_{C_{m}}\) in which \(D_{m}\) stands for a single constraint, and all clauses must be satisfied, imposing hard constraints on the generation process.

In our setting, however, this formulation is no longer applicable because the vocabulary constraint is _as large as the size of the vocabulary of a specific level_. In addition, we aim to satisfy _as many clauses as possible_. Therefore, we formalize constraints as Disjunctive Normal Form (DNF), indicating words and phrases suitable for the target proficiency level: \(D=\underbrace{(D_{1})}_{C_{1}}\vee\underbrace{(D_{2}\wedge D_{3}\land\cdots) }_{C_{2}}\vee\dots\vee\underbrace{(D_{m})}_{C_{m}}\), where the form stands for the word list of the target 

language level, a single \(D_{m}\) represents word and the conjunctive clauses represent several words, namely phrases. Notably, this form of constraints allows for the control of discontinuous phrases, which is difficult in previous methods.

### Optimization Function

Based on the DNF constraints, our task imposes soft constraints that aim to include as many clauses as possible. Given the simplification hypotheses \(\{\text{seq}_{1},\text{seq}_{2},\dots,~{}\text{seq}_{n}\}\), the goal is to maximize:

\[\sum_{j=1}^{m}\sum_{k=1}^{n}\text{count}(C_{j},\text{seq}_{k}), \tag{1}\]

where count\((C_{j},\text{seq}_{k})\) indicates the number of clauses \(C_{j}\) satisfied by seq\({}_{k}\). Consequently, the target during the generation process is to search for the next token that:

* simplifies the original text;
* is contained in \(\exists C_{i}\in D\);
* leads to future tokens that satisfy \(C_{i}\);
* leads to complete phrases or phrases with slots (discontinuity) that satisfy \(C_{i}\).

## 4 Proposed Method

To search for a hypothesis that better satisfies predetermined constraints, some previous methods use rollout in decoding that generates partial future sequences Chaffin et al. (2022); Lu et al. (2022). These methods become infeasible for large models due to the inefficiency of sampling in decoding time and handling the large vocabulary constraints in our task. To effectively and efficiently search for the tokens that satisfy our constraints, we instead consider sampling in the training time and formulate the lookahead search problem using RL search Fickinger et al. (2021) (see Fig. 1).

### RL Search

Consider the text generation process as a Markov Decision Process (MDP), at each timestep \(t\) of the generation, the language model observes a state \(s_{t}\), which is the generated partial sequence seq\({}_{t-1}\), and takes an action \(a_{t}\) to choose the token from its vocabulary. When the EOS token is reached, a reward \(R\) for the generated sequence is calculated and used to update the model. In this setting, the language model is the policy function that searches for a token \(v_{i}\in\mathcal{V}\) where \(\mathcal{V}\) is the vocabulary, and we can use any policy gradient algorithm to guide the language model to search for the generations that maximize the constraint satisfaction. Algorithm 1 indicates our training procedure.

```
0: Complex sentences;
1: Generate simplified texts from the complex sentences using current policy (rollout);
2: Evaluate the current policy and produce rewards to guide the search;
3: Optimize the policy model using the rewards
4: Iteratively perform steps 1-3 till converge.
```

**Algorithm 1** Training Procedure

### Policy Model

The policy model generates a simplified sentence seq given a complex counterpart as a prompt pmt. The policy model is initialized from an instruction tuned language model, which unsupervisedly provides robust text simplifications Kew et al. (2023).

By design, the rewards for the policy model across different proficiency levels are varied. For instance, given the same model response, a positive reward for C level could correspond to a negative reward for A level. Therefore, using the original language model as the backbone, we train separate copies of the policy model for A, B and C levels by adding and updating distinct LoRA parameters to the backbone parameters Hu et al. (2022), while keeping the backbone frozen.

### Reward Models

Inspired by the L2 learning theories, we design two types of rewards at lexical and sentence levels.

#### 4.3.1 Lexical Constraint Reward

We use a simple heuristic to guide the search for generations that satisfy the lexical constraints:

\[H(\text{seq})=\sum_{G_{j}\in D}r(\text{count}(C_{j},\text{seq})), \tag{2}\]

where \(C\) is a clause from \(D\), \(r\) denotes the reward according to the number of satisfactions of \(C\) in seq, and \(H\) denotes the reward score for the generated sequence seq in the current decoding step. To calculate the match counts, we remove basic stop words from the sentence after lemmatization.

As a simple baseline, we define \(r\) as a constant value \(1\) for word and \(1.5\) for phrase to encourage the model to generate more phrases and idioms. However, we found that this simple baseline is easily hacked by the model after a few steps of training, 

i.e., the model only generates a limited set of frequent words that were learnt to produce rewards. To encourage the model to explore more diverse words and improve the overall coverage of target-level words in the generations, the reward should intuitively encourage maximizing the entropy for the clauses in \(D\), so that all the clauses are evenly distributed. Accordingly, we adjust the reward \(r\) for the count of \(C_{j}\) as a **dynamic reward**:

\[r=\begin{cases}1&\text{if $0\leq p_{j}<\frac{1}{m}$},\\ e^{-\alpha p_{j}}&\text{if $\frac{1}{m}\leq p_{j}\leq 1$},\end{cases} \tag{3}\]

where \(p_{j}\) is:

\[p_{j}=\frac{\sum_{k=1}^{n}\text{count}(C_{j},\text{seq}_{k})}{\sum_{k=1}^{n} \sum_{j=1}^{m}\text{count}(C_{j},\text{seq}_{k})}, \tag{4}\]

to discourage the model from exploiting the same clause. Here, \(m\) denotes the total number of clauses in \(D\) and \(\alpha\) is a constant to adjust the penalty degree for too frequent clauses. Eq. 4 is calculated after each epoch and the reward is adjusted accordingly. If matched clauses are above the target level, we give a constant negative score of \(-1\).

#### 4.3.2 Sentence Level Reward

To go beyond words and guide the simplification model's search for a sentence of the target level, we incorporate a sentence-level reward model by simulating human experts' judgment for the sentence's CEFR level. We use pairwise ranking loss to train the reward model, since the class distribution for the CEFR-SP data is imbalanced (Arase et al., 2022). The ranking loss has been shown to be able to encourage the model to only pay attention to the focused class (Henning et al., 2023), thus may mitigate the class imbalance problem.

Consequently, we construct sentence pairs prioritizing the level we focus on generating: for a collection of sentences \(\mathcal{S}=\{s_{1},s_{2},\dots,s_{n}\}\), each sentence \(s_{i}\) is evaluated by human experts and annotated with a language level \(l\). Given the level we want to generate, we select the sentences with the target level \(\mathcal{S}_{\text{tgt}}=\{s_{i}\in\mathcal{S}~{}|~{}l_{i}=\text{level}_{ \text{tgt}}\}\), and randomly sample sentences from other levels to construct a negative set \(\mathcal{S}_{\text{non-tgt}}=\{s_{j}\in\mathcal{S}~{}|~{}l_{j}\neq\text{level }_{\text{tgt}}\}\). Then, we construct sentence pairs \(\mathcal{P}=\{(s_{i},s_{j})~{}|~{}s_{i}\in\mathcal{S}_{\text{tgt}},s_{j}\in \mathcal{S}_{\text{non-tgt}}\}\) randomly selecting from \(\mathcal{S}_{\text{tgt}}\) and \(\mathcal{S}_{\text{non-tgt}}\).

Notably, we do not require the pair to be parallel;

Figure 1: (better viewed in color) The overall framework of the proposed method: the simplification model is initialized from a pretrained large language model which is also used as a frozen (#) reference model to provide entropy regularization (part 0.); top-k sampling is adopted in the decoding process to sample varied simplifications for the complex sentence (part 1.a.); the generated simplifications are evaluated based on the language proficiency level (vocabulary level and sentence level) of the target audience, which is used as rewards to update the simplification model (part 1.b.) to adopt better decoding strategy.

 

they just need to be at different levels. By this design, we disentangle the adequacy requirement for the simplification from the target-level search process. The former is handled by the underlying LLM, and the latter is dealt with by the reward model by level judgment.

With the constructed sentence pairs, we train a sentence-level reward model \(r_{\theta}\). The training objective is to minimize loss:

\[\mathcal{L}(\theta)=-\sum_{(s_{i},s_{j})\in\mathcal{P}}\log\sigma(r_{\theta}( s_{i})-r_{\theta}(s_{j})) \tag{5}\]

where \(\sigma\) is the sigmoid function. After training the reward model, for a generated sentence seq, we take \(r_{l}=\sigma(r_{\theta}(s))\) as the reward, and use a linear combination of the lexical reward and the sentence-level reward as the overall reward:

\[R=\lambda r+\gamma r_{l} \tag{6}\]

### Stabilized RL Training

The original instruct-tuned model is used as a frozen reference model, providing an entropy regularization for the updated policy model to ensure training stability during the search process. Specifically, the simplification seq\({}^{\prime}\) produced by the frozen backbone model \(f^{\prime}\) is added as an entropy regularization to the overall reward:

\[R^{\prime}=R-\log(p_{(f(\text{seq}|pmt)}/p_{(f^{\prime}(\text{seq}^{\prime}| pmt))}). \tag{7}\]

By doing so, we may keep the LLM's strong paraphrasing ability while letting it acquire controllability in CEFR levels.

The policy model \(f\), namely the simplification model is then updated to search for the generations that maximize the reward. In this study, we adopt Proximal Policy Optimization Schulman et al. (2017) to update the policy model, which achieves stable training and faster convergence.

## 5 Experiment Settings

We aim to evaluate the effectiveness of the proposed model in generating high-quality simplifications that align with the target vocabulary and sentence-based CEFR level. This section provides details of the experiment settings.

### Resource and Implementation

Sentence CEFR LevelTo train the sentence-level reward model, we used CEFR-SP Arase et al. (2022), which provides labels of six CEFR levels for a total of \(17k\) sentences annotated by experts. We only used the publicly available subset from the dataset (excluding data based on Newsela Xu et al. (2015)), which resulted in \(10k\) sentences with labels. The statistics of the dataset are described in Table1. We fine-tuned the GPT-2 Radford et al. (using the annotated CEFR levels.

Vocabulary ListFor the lexical constraint reward model, we need vocabulary lists per CEFR level. We downloaded the English Vocabulary Profile (EVP) data4 and used it as a dictionary of words and phrases annotated with their corresponding CEFR levels5. Since our goal is to generate the simplifications in \(i\) and \(i+1\) levels, we always aggregate the vocabulary lists in two levels. For clarity, we consider A1\(+\)A2, B1\(+\)B2, and C1\(+\)C2 levels. In total, we got \(1076\) words for A level, \(3823\) words for B level, \(3612\) words for C level.

Footnote 4: [https://www.englisprofile.org/wordlists/evp](https://www.englisprofile.org/wordlists/evp)

Footnote 5: EVP assigns CEFR levels to each word sense, so the same word can appear at different levels depending on its meaning. For simplicity, we did not conduct word sense disambiguation and assigned the lowest level.

Complex Sentence CollectionWe trained the policy model to iteratively learn to search for a hypothesis that maximizes rewards based on its own generations. The only requirement for our training corpus is a supply of complex sentences that warrant simplification, because sufficiently simple sentences without the need for simplification may disturb the learning. Cegin et al. (2023) showed that large language models are highly capable of paraphrasing. Following this study, we used GPT-46 to synthesize complex sentences from the CEFR-SP training set to create our training corpus. We manually prepared prompts to ensure that the outputs are always at least as complex as the highest C2 level. More details are in Appendix B.

Footnote 6: [https://openai.com/index/gpt-4/](https://openai.com/index/gpt-4/)

We trained separate models for A, B and C levels since different levels require different rewards (see Section 4.2). For computational efficiency, we adopted a relatively small Phi-3-mini-3b model Abdin et al. (2024). M 

can be found in Appendix C.

### Evaluation Datasets

To evaluate the simplification outputs, we need parallel corpora of complex and reference simple paraphrases. Below describes the resources we used for the evaluation.

CEFR-SP-TestAs the formal evaluation dataset, we used CEFR-SP. We expanded its test set to be parallel because CEFR-SP is a non-parallel corpus. Specifically, we generated complex sentences for each sentence in the CEFR-SP test set using the same method described in Section 5.1. These complex sentences were input to models, and outputs were evaluated by comparing them to the original CEFR-SP sentences as references.

TurkCorpusTo assess the applicability of the proposed method for a general simplification task, we also evaluated models on another widely-used dataset, TurkCorpus Xu et al. (2016). We used the test set of the corpus, including \(359\) complex sentences, each has \(8\) human-written simplified sentences as references. It should be noted that TurkCorpus does not provide any level annotations.

### Evaluation Metrics

We evaluated simplification outputs from two perspectives: **simplification quality** and **target audience attributes** by both automatic and human assessments. Simplification quality was assessed across three dimensions: **Simplicity; Fluency; and Adequacy**. As automatic metrics for simplicity, we employed LENS Maddela et al. (2023) and SALSA Heineman et al. (2023), which are two recently proposed model-based evaluation methods. For fluency and adequacy, we employed an instruction-tuned language model as an off-the-shelf evaluation model, which was shown to be effective in automatic translation quality evaluations Kocmi and Federmann (2023). Target audience attributes were measured in terms of **target vocabulary coverage** and **sentence CEFR level**, in which vocabulary coverage includes both frequency and diversity of target vocabulary. For the evaluation of sentence CEFR-level, we used human evaluation. For more details on evaluation metrics, please refer to Appendix A.

### Baselines

Overall, we choose two lines of work as the baselines for comparison.

Controlled SimplificationThere are limited variants in controlled simplification methods which mostly employ control tokens with supervised learning. Based on previous literature, we implemented two baselines for controlling the target level of the simplified texts: a supervised baseline of T5+grade Scarton and Specia (2018) that attaches CEFR levels as control tokens and an unsupervised baseline of FUDGE that uses a discriminator at decoding time Yang and Klein (2021).

Non-controlled SimplificationThe Turk corpus was used to evaluate the effectiveness of the proposed method in general simplification. As opposed to controlled simplification, this task does not consider controlling attributes, such as grade levels, during the simplification. For this line of models, we choose the following methods: DRESS Zhang and Lapata (2017), DMASS Zhao et al. (2018), EditNTS Dong et al. (2019), ACCESS Martin et al. (2020), IterativEdit Kumar et al. (2020). We used outputs of these models shared in the EASSE package Alva-Manchego et al. (2019). In addition, we also compare the vanilla phi3-3b instruction-tuned model as a baseline, under zero-shot setting without fine-tuning on simplification.

\begin{table}
\begin{tabular}{l c c c c c c} \hline
**CEFR-SP** & \multicolumn{1}{c}{A-Frequency} & \multicolumn{1}{c}{A-Diversity} & B-Frequency & B-Diversity & C-Frequency & C-Diversity \\ \hline Reference & \(0.29\) & \(0.527\) & \(0.283\) & \(0.465\) & \(0.080\) & \(0.102\) \\ phi3-3b-vanilla & \(0.252\) & \(0.665\) & \(0.215\) & \(0.435\) & \(0.041\) & \(0.172\) \\ \hline T5+grade-A & \(0.194\) & \(0.438\) & \(0.269\) & \(0.271\) & \(0.072\) & \(0.114\) \\ FUDGE-A & \(0.257\) & \(0.215\) & \(0.207\) & \(0.069\) & \(0.043\) & \(0.018\) \\
**phi3-A** & **0.299** & **0.684** & \(0.196\) & \(0.403\) & \(0.038\) & \(0.141\) \\ \hline T5+grade-B & \(0.204\) & \(0.447\) & **0.275** & \(0.266\) & \(0.069\) & \(0.110\) \\ FUUDE-B & \(0.223\) & \(0.226\) & \(0.231\) & \(0.084\) & \(0.049\) & \(0.027\) \\
**phi3-B** & \(0.151\) & \(0.677\) & \(0.262\) & **0.538** & \(0.064\) & \(0.251\) \\ \hline T5+grade-C & \(0.203\) & \(0.441\) & \(0.276\) & \(0.271\) & \(0.074\) & \(0.114\) \\ FUDGE-C & \(0.239\) & \(0.217\) & \(0.220\) & \(0.077\) & \(0.052\) & \(0.025\) \\
**phi3-C** & \(0.171\) & \(0.658\) & \(0.263\) & \(0.275\) & **0.189** & **0.365** \\ \hline \end{tabular}
\end{table}
Table 2: Results on target attribute controllability on CEFR-SP-Test. For “Reference”, the frequency and diversity metrics were calculated using a subset of each grade level to show distributions in sentences of specific levels.

 



## 6 Experiment Results

This section analyses experiment results of automatic and human evaluations, and ablation study.

### Automatic Evaluation Results

Target AttributesTables 2 and 3 show the evaluation results for the target vocabulary coverage. These results demonstrate that compared to the baseline models, the proposed model significantly increases the frequency of target vocabulary in simplified sentences while also improving vocabulary diversity. Notably, the proposed method successfully increases the frequency and diversity of A and C-level vocabulary, which should be harder than B-level due to the scarcity of level A and C samples (Arase et al., 2022).

Simplification Quality Tables 5 and 6 show the evaluation results for the simplification quality. Overall, these results indicate that our models can produce high-quality simplifications, greatly outperforming the baseline models. Remind that our model does not have reward to encourage the model to follow the adequacy requirement. We attribute these improvements to the benefits of using entropy regularization imposed by the reference model that allows the preservation of the high paraphrasing capability of LLMs. Table 4 shows a randomly picked example of simplification by our method; Appendix E provides more.

### Human Evaluation Results

We perform a human evaluation to assess the simplification quality from human perspectives. We recruited three graduate-level students majoring in linguistics to perform the evaluation. The evaluators were first trained with the background knowledge and then given a guideline to evaluate the following aspects of the samples: fluency, simplicity, adequacy, and CEFR sentence level.

We asked annotators to make binary judgements for fluency, simplicity, and adequacy. For sentence level, because CEFR-level judgements require expertise in language education, we simplified the task to collect reliable decisions. We asked the evaluators to judge if a simplified sentence matches the desired sentence level (denoted as "Level"). We showed a reference with its CEFR level and requested the evaluators to judge if the model output matches the reference's simplicity. In addition, we asked them if a simplification output is preferable in terms of its CEFR level compared to the one generated by a model targeting a different level (denoted as "Prefer"). For example, an evaluator judges if an output of the A-level model is preferable to that of the C-level compared to the

\begin{table}
\begin{tabular}{l l} \hline
**Complex Sentence** & The considerable distance, compounded by Jamie’s current condition of pregnancy, which inexorably engenders a state of increased fatigue, renders the prospect of ambulation to said location prohibitively challenging for her. \\ \hline
**Ref. (level B)** & It is too far for Jamie to walk to, especially because she is pregnant and easily exhausted. \\ \hline
**Simplifications** \\ Level A: Jamie is _too tired to_ walk far because she is pregnant. \\ Level B: Jamie’s pregnancy makes it very hard for her to walk to the location _due to_ the long distance. \\ Level C: Jamie’s pregnancy leads to _fatigue_, making it hard for her to walk to the distant place. \\ \hline \end{tabular}
\end{table}
Table 4: A randomly selected example from the simplification result of the proposed method. The target vocabulary of the corresponding level is marked in _italic_ font.

\begin{table}
\begin{tabular}{l c c c c c c} \hline
**TURK** & A-Frequency & A-Diversity & B-Frequency & B-Diversity & C-Frequency & C-Diversity \\ \hline Reference & 0.176 & 0.229 & 0.227 & 0.132 & 0.056 & 0.046 \\ phi3-byvanilla & 0.166 & 0.180 & 0.177 & 0.083 & 0.034 & 0.023 \\ \hline T5+grade-A & 0.187 & 0.180 & 0.217 & 0.088 & 0.051 & 0.028 \\ FUDGE-A & 0.175 & 0.177 & 0.175 & 0.069 & 0.034 & 0.018 \\
**phi3-A** & **0.216** & **0.208** & 0.153 & 0.063 & 0.031 & 0.018 \\ \hline TS+grade-B & 0.201 & 0.190 & 0.217 & 0.085 & 0.052 & 0.028 \\ FUDGE-B & 0.163 & 0.177 & 0.178 & 0.077 & 0.039 & 0.022 \\
**phi3-B** & 0.126 & 0.201 & **0.330** & **0.112** & 0.066 & 0.035 \\ \hline T5+grade-C & 0.187 & 0.194 & 0.225 & 0.090 & 0.050 & 0.026 \\ FUDGE-C & 0.171 & 0.174 & 0.181 & 0.076 & 0.037 & 0.019 \\
**phi3-C** & 0.151 & 0.178 & 0.193 & 0.092 & **0.091** & **0.041** \\ \hline \end{tabular}
\end{table}
Table 3: Results on target attribute controllability on TurkCorpus

A-level reference.7 For each CEFR level, \(30\) simplifications of the CEFR-SP-Test were randomly sampled and annotated "Level" and "prefer" judgements. We report the ratios of positive judgements as evaluation scores. The details of the annotation guideline and interface are presented in Appendix D.

Footnote 7: Preference score for the reference was judged by comparison with a sentence randomly chosen from another level

Table 6 shows the results; the simplicity score is generally high, close to \(1\), across models. This is expected as the source sentences were generated to be highly complex. The adequacy measurement results are consistent with automatic evaluation; identifying our proposed models as the most adequate. Furthermore, the proposed method achieves the best controllability on sentence levels compared to the baselines as indicated by significantly higher "Level" and "Prefer" scores.

### Ablation Study

In this section, we show how each part of the proposed rewards contributes to the final performance. We compare the following models: vanilla phi3 model, reward using only target vocabulary counts, reward using dynamically adjusted vocabulary coverage rates, and reward using both dynamic vocabulary coverage rate and sentence levels (proposed method). The frequency and diversity evaluation results for A and B level models are presented in Fig. 2. Complete results can be found in Appendix E. It can be seen that changing the simple match count reward to a dynamically adjusted reward indeed encourages the model to increase the entropy inside the target vocabulary and largely improve the vocabulary diversity.

## 7 Conclusion

In this paper, we target ESL learners as audiences for text simplification aiming to facilitate the learning process of the foreign language. Referring to the input hypothesis and frequency effect theory in L2 learning, we propose a reinforcement learning method on LLM to control the simplification model to generate outputs that satisfy the vocabulary and sentence level constraints. Experiment results show that the proposed method can increase the target vocabulary coverage in the generated simplifications, and human evaluation results confirmed that the simplified texts generally preserve the targeted CEFR levels.

In practice, different individuals have varied lev

Figure 2: Reward effects on target vocabulary coverage

\begin{table}
\begin{tabular}{l c c c c} \hline
**TURK** & LENS & SALSA & Fluency & Adequacy \\ \hline Reference & \(35.20\) & \(64.96\) & \(0.732\) & \(0.901\) \\ \hline ACCESS & \(49.90\) & \(62.68\) & \(0.576\) & \(0.780\) \\ DMASS & \(46.52\) & \(58.97\) & \(0.515\) & \(0.665\) \\ DERES & \(59.76\) & \(62.63\) & \(0.807\) & \(0.615\) \\ DRESS-LS & \(60.56\) & \(62.92\) & \(0.838\) & \(0.657\) \\ EditNTS & \(57.71\) & \(64.86\) & \(0.752\) & \(0.710\) \\ IterativEdit & \(37.35\) & \(49.74\) & \(0.409\) & \(0.607\) \\ \hline phi3-3b-vanilla & \(65.08\) & \(71.93\) & \(0.830\) & \(0.807\) \\
**phi3-A** & \(64.92\) & **73.68** & \(0.720\) & \(0.708\) \\
**phi3-B** & **70.25** & \(69.05\) & \(0.855\) & **0.952** \\
**phi3-C** & \(62.24\) & \(70.43\) & **0.869** & \(0.872\) \\ \hline \end{tabular}
\end{table}
Table 6: Simplification quality on TurkCorpus; all models evaluated on the entire sentences as TurkCorpus does not annotate levels.

 

els of knowledge for the language. We plan to extend the method to generate individual learner-targeted personalized simplifications in the future.

### Limitations

This work assumes the target vocabulary for the learner is accessible, which in reality may not be the case as the target vocabulary varies with learner individuals and has to be first estimated. Although it is out of the scope of this paper, this direction constitutes our future work. Besides, currently, we do not control the frequency to a specific number, such as \(95\%~{}i\) level and \(5\%~{}i+1\) level, which is an important aspect to consider according to the L2 learning theory.

The control for target vocabulary and sentence is implemented individually for different levels rather than using one model altogether, causing heavier computational loads. In future, we seek to improve the design of reward model to integrate rewards for different proficiency levels into one model, and explore for a finer control over the frequency of the generated vocabulary.

## References

* A. Abdin, S. J. Jacobs, A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al. (2024)Phi-3 technical report: a highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. Cited by: SS1.
* S. Agrawal and M. Carpuat (2023)Controlling pre-trained language models for grade-specific text simplification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 12807-12819. External Links: Link, Document Cited by: SS1.
* F. Alva-Manchego, L. Martin, C. Scarton, and L. Specia (2019)EASSE: easier automatic sentence simplification evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, Hong Kong, China, pp. 49-54. External Links: ISBN 978-3-642-4651-1, Link, Document Cited by: SS1.
* F. Alva-Manchego, C. Scarton, and L. Specia (2021)The (un)suitability of automatic evaluation metrics for text simplification. Computational Linguistics47 (4), pp. 861-889. Cited by: SS1.
* Y. Arase, S. Uchida, and T. Kajiwara (2022)CEFR-based sentence difficulty annotation and assessment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 6206-6219. External Links: Link, Document Cited by: SS1.
* J. Cegin, J. Simko, and P. Brusilovsky (2023)ChatGPT to replace crowdsourcing of paraphrases for intent classification: higher diversity and comparable model robustness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 1889-1905. External Links: Link, Document Cited by: SS1.
* A. Chaffin, V. Claveau, and E. Kijak (2022)PPL-mCTS: constrained textual generation through discriminator-guided MCTS decoding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 2953-2967. External Links: Link, Document Cited by: SS1.
* A. Chi, L. Chen, Y. Chang, S. Lee, and J. S. Chang (2023)Learning to paraphrase sentences to different complexity levels. Transactions of the Association for Computational Linguistics11, pp. 1332-1354. External Links: Link, Document Cited by: SS1.
* L. Cripwell, J. Legrand, and C. Gardent (2023)Simplicity level estimate (SLE): a learned referenceless metric for sentence simplification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Florence, Italy, pp. 12053-12059. External Links: Link, Document Cited by: SS1.
* Y. Dong, Z. Li, W. Rezagholizadeh, and J. C. Kit Cheung (2019)EditNTS: an neural programmer-interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3393-3402. External Links: Link, Document Cited by: SS1.
* N. C. Ellis (2002)Frequency effects in language processing: a review with implications for theories of implicit and explicit language acquisition. Studies in second language acquisition24 (2), pp. 143-188. Cited by: SS1.
* N. C. Ellis and F. Ferreira-Junior (2009)ConvNet: learning as a function of frequency, frequency distribution, and function. The Modern language journal93 (3), pp. 370-385. External Links: ISSN 0021-905X, Link, Document Cited by: SS1.
* A. Fickinger, H. Hu, B. Amos, S. Russell, and N. Brown (2021)Scalable online planning via reinforcement learning fine-tuning. Advances in Neural Information Processing Systems34, pp. 16951-16963. External Links: Link, Document Cited by: SS1.
* H. Guo, R. Pasunuru, and M. Bansal (2018)Dynamic multi-level multi-task learning for sentence simplification. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, pp. 462-476. External Links: Link, Document Cited by: SS1.

 

David Heineman, Yao Dou, Mounica Maddela, and Wei Xu. 2023. Dancing between success and failure: Edit-level simplification evaluation using SALSA. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 3466-3495, Singapore. Association for Computational Linguistics.
* Henning et al. (2023) Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. 2023. A survey of methods for addressing class imbalance in deep-learning based natural language processing. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 523-540, Dubrovnik, Croatia. Association for Computational Linguistics.
* Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Kew et al. (2023) Tannon Kew, Alison Chi, Laura Vasquez-Rodriguez, Sweta Agrawal, Dennis Aumiller, Fernando Alvandecho, and Matthew Shardlow. 2023. BLESS: Benchmarking large language models on sentence simplification. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13291-13309, Singapore. Association for Computational Linguistics.
* Few and Ebling (2022) Tannon Kew and Sarah Ebling. 2022. Target-level sentence simplification as controlled paraphrasing. In _Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)_, pages 28-42, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
* Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
* Kocmi and Federmann (2023) Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_, pages 193-203, Tampere, Finland. European Association for Machine Translation.
* Krashen (1981) Stephen Krashen. 1981. Second language acquisition. _Second Language Learning_, 3(7):19-39.
* Kumar et al. (2020) Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020. Iterative edit-based unsupervised sentence simplification. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7918-7928, Online. Association for Computational Linguistics.
* Laban et al. (2021) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021. Keep it simple: Unsupervised simplification of multi-paragraph text. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6365-6378, Online. Association for Computational Linguistics.
* Lu et al. (2022) Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2022. NeuroLogic a*seq decoding: Constrained text generation with lookahead heuristics. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 780-799, Seattle, United States. Association for Computational Linguistics.
* Lu et al. (2021) Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4288-4299, Online. Association for Computational Linguistics.
* Maddela et al. (2023) Mouncia Maddela, Yao Dou, David Heineman, and Wei Xu. 2023. LENS: A learnable evaluation metric for text simplification. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 16383-16408, Toronto, Canada. Association for Computational Linguistics.
* Martin et al. (2020) Louis Martin, Eric de la Clergerie, Benoti Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4689-4698, Marseille, France. European Language Resources Association.
* Martin et al. (2022) Louis Martin, Angela Fan, Eric de la Clergerie, Antoine Bordes, and Benoit Sagot. 2022. MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 1651-1664, Marseille, France. European Language Resources Association.
* Mitchell et al. (2019) Rosamond Mitchell, Florence Myles, and Emma Marsden. 2019. _Second language learning theories_. Routledge.
* Nakamachi et al. (2020) Akifumi Nakamachi, Tomoyuki Kaijiwara, and Yuki Arase. 2020. Text simplification with reinforcement learning using supervised rewards on grammaticality, meaning preservation, and simplicity. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop_, pages 153-159, Suzhou, China. Association for Computational Linguistics.
* Nishihara et al. (2019) Daiki Nishihara, Tomoyuki Kaijiwara, and Yuki Arase. 2019. Controllable text simplification with lexical 

constraint loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, Florence, Italy, pp. 260-266. External Links: Link Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311-318. External Links: ISBN 978-1-4503-3213-1, Document, Link Cited by: SS1.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2018)Language models are unsupervised multitask learners. External Links: Link Cited by: SS1.
* C. Scarton and L. Specia (2018)Learning simplifications for specific target audiences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, pp. 712-718. External Links: Link Cited by: SS1.
* J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017)Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: SS1.
* K. Cheng Sheang and H. Saggion (2021)Controllable sentence simplification with a unified text-to-text transfer transformer. In Proceedings of the 14th International Conference on Natural Language Generation, Aberdeen, Scotland, UK, pp. 341-352. External Links: Link Cited by: SS1.
* R. Stodden, O. Momen, and L. Kallmeyer (2023)DEplain: a german parallel corpus with intralingual translations into plain language for sentence and document simplification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 16441-16463. External Links: Link Cited by: SS1.
* E. Sulem, O. Abend, and A. Rappoport (2018)Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 685-696. External Links: Link Cited by: SS1.
* R. Sun, H. Jin, and X. Wan (2021)Document-level text simplification: dataset, criteria and baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 7997-8013. External Links: Link Cited by: SS1.
* W. Xu, C. Callison-Burch, and C. Napoles (2015)Problems in current text simplification research: new data can help. Transactions of the Association for Computational Linguistics3, pp. 283-297. External Links: Link Cited by: SS1.
* W. Xu, C. Napoles, E. Pavlick, Q. Chen, and C. Callison-Burch (2016)Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics4, pp. 401-415. External Links: Link Cited by: SS1.
* K. Yang and D. Klein (2021)FUDGE: controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 3511-3535. External Links: Link Cited by: SS1.
* K. Yang, D. Liu, W. Lei, B. Yang, M. Xue, B. Chen, and J. Xie (2023)Tailor: a soft-prompt-based approach to attribute-based controlled text generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 410-427. External Links: Link Cited by: SS1.
* T. Zetsu, T. Kajiwara, and Y. Arase (2022)Lexically constrained decoding with edit operation prediction for controllable text simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), Abu Dhabi, United Arab Emirates (Virtual), pp. 147-153. External Links: Link Cited by: SS1.
* T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, and Y. Artzi (2020)BertScore: evaluating text generation with bert. In International Conference on Learning Representations, Cited by: SS1.
* X. Zhang and M. Lapata (2017)Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 584-594. External Links: Link Cited by: SS1.
* S. Zhao, R. Meng, D. He, A. Saptono, and B. Parmento (2018)Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 3164-3173. External Links: Link Cited by: SS1.

## Appendix A Evaluation Metrics

In this section, the evaluation metrics are explained in details: fluency, adequacy, target vocabulary frequency, diversity and target sentence level.

Previous studies have introduced various metrics for evaluating simplicity, which we summarize in Table 8. Among these metrics, SARI Xu et al. (2016) is the most commonly employed in the literature. However, recent studies show that SARI may not be an optimal measure for assessing the quality of simplicity Alva-Manchego et al. (2021) 

Maddela et al., 2023; Stodden et al., 2023). Thus, we chose to use LENS (Maddela et al., 2023) and SALSA (Heineman et al., 2023), two recently proposed metrics, to measure simplicity.

Footnote 7: [https://www.lisp.org/](https://www.lisp.org/)

For adequacy and fluency, the ideal approach is human evaluation; however, this is impractical due to the large dataset size. Instead, we employed large language models to assess these two aspects. With a capable language model \(f\), the generated simplification sentence \(s\) is evaluated as:

\[score(s)=\sum_{v\in V_{y}}f(v~{}|~{}(pmt,s) \tag{8}\]

where \(pmt\) is a prompt designed for the model to output "yes" if the model evaluate \(s\) to be adequate or fluent, and \(V_{y}\) is a vocabulary subset for "yes" with \(V_{y}=\{\text{YES},\text{Yes},\text{yes}\}\). We use Llama-3-8b-instruct8 model as the evaluation model in our experiment.

Footnote 8: [https://llama.meta.com/llama3/](https://llama.meta.com/llama3/)

To measure target vocabulary frequency, we took the ratio between the total count of matched target words and the total generated words.

\[\sum_{j=1}^{m}\sum_{k=1}^{n}\text{count}(C_{j},\text{seq}_{k})/\sum_{k=1}^{n} \text{count}(\text{token},\text{seq}_{k}) \tag{9}\]

To measure vocabulary diversity, we took the ratio between the number of matched words and number of words in the word list.

\[\sum_{j\in D}\mathbf{1}\left(\bigvee_{k=1}^{n}\mathbf{1}_{C_{j}}(\text{seq}_ {k})\right)/|D| \tag{10}\]

## Appendix B Complex Sentence Generations

To generate complex sentences, we prompted the GPT-4 model9 to rephrase sentences of varied levels into highly complex sentences. To ensure the diversity of the generated complex data, we initially created a variety of seed prompts manually and instructed GPT-4 to generate additional prompts based on these seed prompts. GPT-4 was then prompted to generate complex sentences based on these diversified prompts. The 5 manually written seed prompts and 10 model generated prompts are presented in Table 9. In total 15 prompts were used to generate complex sentences, for each generation, one of the prompts was randomly selected. We present samples of the generated complex sentences together with simplifications in Table 10 and Table 11.

Footnote 9: [https://openai.com/index/gpt-4/](https://openai.com/index/gpt-4/)

## Appendix C Training Details

Implementation Details of BaselinesWe implemented the baseline models using the transformers library10. T5-s2s models require parallel corpus of complex-simple sentences, for which we used the pseudo-parallel sentences of generated complex sentences and their original sentences, and prepended level tokens for level controlling during training and evaluation. We implemented the FUDGE simplification control model with a Llama-3-8b-instruct model as the generation model, and its logits during the inference were adjusted using the CEFR level classification model released by Arase et al. (2022).

Footnote 10: [https://huggingface.co/docs/transformers/en/index](https://huggingface.co/docs/transformers/en/index)

**Implementation Details of Proposed Method** The PPO training algorithm was implemented using the TRL library11 with a learning rate of \(3e-5\). For the dynamic reward model used in the training, we set the \(\alpha\) to be \(1.2\), as we found a value slightly bigger than \(1\) was shown to have better performance empirically; the reward for phrases is always set to be \(1.5\) times more than words to reward more on the phrase generation. For the overall reward, \(\lambda\) was set to be \(1.5\) to compensate for the vocabulary reward penalty, and \(\gamma\) was set to 1. During training, we used the following prompt for the model to generate simplifications: "Given a complex sentence \(\{\}\), generate a simplified version for it:".

Footnote 11: [https://huggingface.co/docs/trlr/main/en/index](https://huggingface.co/docs/trlr/main/en/index)

Training PerformanceThe performance of the sentence level reward model is shown in Fig. 3. Fig.4 shows mean reward and KL change over the training steps with and without the dynamic reward. Objective/KL indicates the deviation of the simplification model from t

\begin{table}
\begin{tabular}{c|c|c} \hline Metric & Scope & Reference \\ \hline BLEU (Paphemi et al., 2002) & semantic similarity & Y \\ PKGLN (Kimal et al., 1975) & readability & N \\ FKBLUEU (Xue et al., 2016) & readability, similarity & Y \\ SARI (You et al., 2016) & keep, add, delete & Y \\ D-SARI (Sun et al., 2021) & keep, add, delete & Y \\ SAMSA (Salem et al., 2018) & semantic structural similarity & N \\ BERTScore (Zhang et al., 2020) & semantic similarity & Y \\ SLE (Crubysel et al., 2023) & human rating + FKGL & N \\ LINAS (Maddela et al., 2023) & human ratings & Y \\ SALSA (Heinemann et al., 2023) & human rating & N \\ \hline \end{tabular}
\end{table}
Table 8: Metrics used in recent literature. Scope denotes the aspect that the metric aims to evaluate, and reference indicates whether the metric is computed based on references or not.

 

an absurdly high KL indicates model collapse; a burst in mean reward indicates model collapses and only produces a limited set of vocabulary. It can be observed that using the dynamic reward helps stabilize the training, while using the match count alone causes the model to be over-optimized and collapse to a limited vocabulary subset.

## Appendix D Human Evaluation Details

In this section, the annotation guidelines that evaluators used to evaluate the generated simplifications as well as the annotation interface are presented. The annotation guidelines contain the definition of the aspect to be evaluated, the criteria for the evaluation and indications for the annotation interface. The annotation interface is designed to be a binary-choice form, for each aspect to be evaluated, the evaluators chose to tick to indicate that the simplification contains the aspect to be evaluated, and does not satisfy the aspect otherwise. The evaluation results are then used to calculate the binary accuracy of the aspects to be evaluated. The evaluation guidelines are shown in Fig. 5(a) and the evaluation interface is shown in Fig. 5(b).

## Appendix E More Evaluation Results

Fig. 6 shows the ablation study results on all three levels of models.

Tables 10 and 11 present example outputs: the complex sentences, reference sentences, and simplified sentences. For each complex sentences, there are three versions of the simplified sentences, corresponding to A, B and C levels generated by different models targeting the corresponding level, respectively.

Figure 4: Training stability w/wo dynamic reward

Figure 5: Screenshot of annotation guidelines

Figure 3: Sentence level reward model evaluation accuracy



* [15] M. A. Bader, and R. J. Brox. The role of the learning process in a multi-task task. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2378-2390, 2014.
* [16] S. D. Krizhevsky, I. Sutskever, and E. Rabinovich. Learning from tiny images by reducing internal covariate representations. _Journal of Machine Learning Research_**4**: 135-140, 2014.
* [17] Y. Bengio et al. Generative adversarial nets for image recognition. _arXiv preprint arXiv:1405.1002_**, 2014.
* [18] G. Hinton, P. Hariharan, and C. Szegedy. Imagenet classification with deep convolutional neural networks. _CoRR_**406**, 2015.
* [19] L. Bottou, T. Darrell, and V. Vanhoucke. Deep residual learning for large scale visual recognition. In _Advances in Neural Information Processing Systems_, pages 1098-1110.
* [20] N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional network for image recognition. In _Advances in Neural Information Processing Systems_, pages 1098-1110.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l} \hline Complex Sentence \\ Let us endeavor to delve into the intricacies and nuances of the text, striving to comprehend the underlying themes and implications inherent within, as we embark on this journey of intellectual exploration. \\ \hline Reference, level: A \\ Let ’s try to read . \\ \hline Simplifications \\ Let’s try to understand the text well. \\ Let’s try to understand the text’s main ideas and meanings as we read. \\ Let’s understand the text’s themes and meanings. \\ \hline Complex Sentence \\ Are you experiencing any concerns or apprehensions regarding the composition, quality, or safety of the food you consume? \\ \hline Reference, level: A \\ Do you worry about what you eat ? \\ \hline Simplifications \\ Do you have worries about the food you eat? \\ Do you have worries about the food you eat’s quality or safety? \\ Do you have concerns about your food’s composition, quality, or safety? \\ \hline Complex Sentence \\ The considerable distance, compounded by Jamie’s current condition of pregnancy, which inexorably engenders a state of increased fatigue, renders the prospect of ambulation to said location prohibitively challenging for her. \\ \hline Reference, level: B \\ It is too far for Jamie to walk to , especially because she is pregnant and easily exhausted . \\ \hline Simplifications \\ Jamie is too tired to walk far because she is pregnant. \\ Jamie’s pregnancy makes it very hard for her to walk to the location due to the long distance. \\ Jamie’s pregnancy leads to fatigue, making it hard for her to walk to the distant place. \\ \hline Complex Sentence \\ With meticulous care, the young girl positioned the doll on her bed, her actions imbued with a quiet reverence, before softly pressing her lips to its cheek, an act infused with such tenderness that one might easily imagine she believed the doll to be a living boy. \\ \hline Reference, level: B \\ The little girl placed the doll carefully on her bed and kissed its cheek as gently as if it were a real boy . \\ \hline Simplifications \\ The girl put the doll on her bed and kissed it gently. \\ The girl carefully placed the doll on her bed, kissed its cheek gently, and seemed to love it like a boy. \\ The girl gently kissed the doll out of affection. \\ \hline \end{tabular}
\end{table}
Table 10: Complex sentences, references and model simplified sentences in A, B, C levels, respectively

* [150] A. K. M. C., and J. D. L. H., _Theoretical Chemistry of the Physical Properties of Metals: Theory and Experiment_. Oxford University Press, New York, 2003.
* [151] R. W. F., _Theory of the Physical Properties of Metals_. Cambridge University Press, Cambridge, 1974.
* [152] S. G., _Statistical Mechanics of Liquids_. Wiley-Interscience, New York, 1986.

[MISSING_PAGE_POST]