DIMSIM: Distilled Multilingual Critics for Indic Text Simplification (2024)

Abstract: Self-correction techniques have recently emerged as a promising framework to improve the quality of responses generated by large language models (LLMs). Few-shot prompted LLMs act as critics to produce feedback for an input, which is further fed to a refiner (also an LLM) to produce an output. However, these critique-refine steps require multiple expensive LLM calls. To circumvent this large inference cost, we borrow inspiration from prior work on knowledge distillation and propose the use of critique distillation to train critic models. These are smaller sequence-to-sequence models that are trained on input-critique pairs generated by an LLM. We focus on the problem of text simplification for three Indian languages: Hindi, Bengali and Marathi. This task is a good fit for self-correction style techniques. It also has not been systematically explored for Indian languages before. We train two separate critics that focus on lexical and structure complexity, and show that it is surprisingly more effective than using an LLM directly as a critic in both 0-shot and few-shot settings. We also show the benefits of training multilingual critics, as opposed to monolingual critics. Extensive human evaluations show that on average, raters find 80% of DIMSIM’s output to be simple and easy to read.

heading: 1 Introduction
text: Large language models (LLMs) are well-equipped to generate high-quality responses using only a few task-specific examples in its input prompts (Brown et al., 2020; Chowdhery et al., 2023; Wei et al., 2022a; Min et al., 2022). However, these generations are prone to vulnerabilities such as “hallucinations" (Huang et al., 2023b; Mündler et al.,
* Equal contribution
2023), biased content (Kotek et al., 2023; Wan et al., 2023) and incorrect reasoning (Zhang et al., 2023a; Huang and Chang, 2023), to name a few. A promising approach to fix inconsistencies in LLM outputs is self-correction specifically the critiquerefine model (Bai et al., 2022; Nathani et al., 2023; Pan et al., 2023), where the LLM is given feedback or a critique about its outputs – either generated by the LLM itself or by humans – and asked to revise its outputs in accordance with the given feedback.
While self-correction can be effective (Huang et al., 2023a; Tang et al., 2019a), we incur substantial costs when invoking LLMs with over 500B parameters (Chowdhery et al., 2023). Each LLM call requires significant memory and compute utilizing at least 350GB GPU memory to serve a 175B LLM, which is far beyond the scope of what is affordable for most product developers and service providers. To address these computational constraints, knowledge distillation has been used in prior work (Tang et al., 2019a; Hsieh et al., 2023) where smaller models are trained with labels from a larger LLM. However, distillation within the critique-refine paradigm and more so for tasks using multilingual LLMs has not been explored yet.
In this work, we propose the use of distilled trained critics within the critique-refine framework for the problem of text simplification for three Indian languages. We chose the task of text simplification for two main reasons: 1. It is a form of textual style transfer and hence well-suited to the critique-refine paradigm. 2. There is no existing work on text simplification for Indian languages. In fact, multilingual text simplification has not been sufficiently well-explored in prior work (Ryan et al., 2023). Our work is an attempt to bridge this gap. Expanding the task to Indian languages like Hindi, Bengali and Marathi, which are not as well-represented as English in the LLM, also leads us to train multilingual critics for better generalization.
We assume a few-shot setting for Hindi with
16093
access to a small set of expert-written examples of text simplification in Hindi. We adopt the more challenging zero-shot setting for Bengali and Marathi where we assume no access to text simplification instances in these two languages and only make use of the Hindi examples.
We decouple text simplification along two natural axes -- structural and lexical complexity -- and design two separate critic modules that provide critiques specific to each of these two dimensions (§3.1). Each distilled critic is a smaller sequenceto-sequence model (e.g., mT5 (Xue et al., 2021)), trained using synthetic data created via few-shot prompting of the LLM followed by careful data filtering (§3.2).
Apart from critique generation, we also show the merits of both eager and lazy refinement where the LLM (used as a refiner) can either individually refine the outputs based on each trained critic’s feedback or refine the output based on combined feedback from both critic modules, respectively (§3.3).
We evaluate our results using the standard SARI metric (Xu et al., 2016) for text simplification and also undertake detailed human evaluations. We find that trained critics, especially multilingual critics that are trained on data pooled together from Hindi, Bengali and Marathi, offer improved cross-lingual transfer to Bengali and Marathi. The trained critics result in significant performance improvements of up to 2.4 absolute SARI points for Hindi and up to 2 absolute points for Bengali and Marathi in the zero-shot setting, compared to using LLMs at test time. We call our model DIMSIM: Distilled multilingual critics for text simplification.

heading: 2 Related Work

heading: 2.1 Critique Refinement
text: Despite demonstrating impressive task performance (Guo et al., 2023; Suzgun et al., 2022) and reasoning abilities (Wei et al., 2022b; Kojima et al., 2022), LLMs are observed to exhibit undesired behavior such as hallucinations (Lin et al., 2021; Zhang et al., 2023b), unfaithful reasoning (Golovneva et al., 2022; Wu et al., 2023) and generating inappropriate or harmful content (Shaikh et al., 2022; Gehman et al., 2020; Levy et al., 2022). Self correction via critique feedback generated by the LLM (Schick et al., 2022; Madaan et al., 2023) is a very promising direction. (Pan et al., 2023) presents a comprehensive survey of ap-
proaches in this body of work. (Huang et al., 2022) uses chain of thought and self consistency to produce clean task training data that is to further fine tune the LLM. Self-Refine(Madaan et al., 2023) use the LLM iteratively as the generator, refiner and the critic for 7 tasks across dialog generation, coding, reasoning and constrained generation. (Lahoti et al., 2023) argue that instead of sequential iterations, diverse sampling of critiques, refinement followed by voting produces better results. MultiAspect Feedback (Nathani et al., 2023) argues that breaking down critiquing into specialized feedback modules provides better performance than relying on one single generic feedback source. Our approach is similar in that we use specialized critics for various aspects of text simplification. REFINER (Paul et al., 2023) shows that critiquing the intermediate reasoning steps in the chain of thought can improve performance.

heading: 2.2 LLM Distillation
text: One of the main practical limitations of LLMs is the compute and latency cost incurred during inference by virtue of their complex architectures and vast parameter space. Knowledge distillation (Hinton et al., 2015; Tang et al., 2019b; Liang et al., 2020; Fu et al., 2023; West et al., 2021; Li et al., 2022). reduces the serving cost concerns by learning a small “student” model to mimic the behaviour of a larger “teacher" model, an LLM in our case. LLMs have been shown to produce very detailed reasoning and rationale steps for the outputs they produce (Wei et al., 2022b; Kojima et al., 2022). A recent direction of research has started to show that distilling using teacher generated rationales can be very effective (Ho et al., 2022; Wang et al., 2022; Magister et al., 2022; Li et al., 2023). The proposed approach in this paper shares the same spirit. However to the best of our knowledge, we are the first to propose using rationale distillation as a way of training critic models. We show, similar to very recent work (Hsieh et al., 2023), that the distilled critic is able to surpass the performance of the teacher LLM while an order of magnitude smaller.

heading: 2.3 Text Simplification
text: Text simplification is a well-studied NLP problem (Zhu et al., 2010; Xu et al., 2015; AlvaManchego et al., 2020; Al-Thanyyan and Azmi, 2021) that deals with reducing the complexity of text without compromising on its meaning. This task has been commonly modeled either as a super-
vised sequence-to-sequence problem (Nisioi et al., 2017; Zhang and Lapata, 2017; Mallinson et al., 2020) or in an unsupervised setting with no access to parallel complex-simple sentence pairs (Surya et al., 2019; Kumar et al., 2020). Recent work on the BLESS benchmark (Kew et al., 2023) has comprehensively evaluated state-of-the-art LLMs for the task of text simplification and found them to be comparable with existing state-of-the-art baselines. This evaluation, however, was entirely on English datasets. Multilingual text simplification is relatively far less explored. As highlighted in (Ryan et al., 2023), text simplification research has become exceedingly English-centric in the last few years; the authors introduce a multilingual benchmark that covers text simplification for many diverse languages. With our work, we add to this existing multilingual benchmark for text simplification with datasets for three Indian languages, Hindi, Bengali and Marathi.

heading: 3 DIMSIM: Methodology
text: In this work, we present a critique-refine framework for text simplification (TS) using multilingual LLMs. Instead of a single critic that offers overall feedback for simplification, we decompose the simplification problem along two dimensions -- structural complexity and lexical complexity -- and we designate one critic to each dimension. The structure critic offers suggestions on how to alter the structure of a sentence, either by paraphrasing or splitting, to make it simpler. The lexical critic is
more targeted in its critique by identifying archaic or difficult words in the text and offering simpler alternatives. Henceforth, we will collectively refer to these two modules as sub-critics. Finally, a refiner module processes the combined critique from the sub-critics to generate a simplified text output (Nathani et al., 2023).
To prime the sub-critics to offer informative critiques specific to each dimension, we need carefully designed few-shot examples. We create a corpus of 20 manually-curated input-output TS pairs for Hindi. These span a range of simplification operations, and are created by trained language experts. This method is expensive, tedious, and requires access to trained annotators. It does not readily scale to new languages. Therefore, we adapt the critique-refine framework to work in a zero-shot cross-lingual setting for Bengali and Marathi, using Hindi few-shot examples.
Figure 1 offers a schematic illustration of DIMSIM. Section 3.1 describes the key design choices we make to build the sub-critics. Section 3.2 further elaborates on generating synthetic data to train sub-critics. And, Section 3.3 describes how the refiner module processes these critiques to produce the final simplified text.

heading: 3.1 Design of Critic Modules
text: Critique generation is the first step of DIMSIM. As mentioned earlier, we choose two dimensions along which a text might be complex: structural complexity and lexical complexity. Decoupling the problem in this manner has multiple advantages.
Firstly, each sub-critic can now focus exclusively on a specific aspect of the problem and offer targeted critiques. Secondly, sub-critics can now be combined in different ways with the refiner, i.e., either sequentially after each critique is generated (eager refinement); or both the critiques are concatenated and a collective critique is sent to the refiner (lazy refinement). Lastly, decoupling critics allows for the effective use of multilingual data to learn language-agnostic properties of TS and enable better cross-lingual transfer.
The structure critic provides specific suggestions on simplifying the structure of the text, either by paraphrasing, removing repeated information, sentence-splitting, or a combination of the three. The lexical critic provides specific suggestions on simplifying words used in the text. It provides easier alternatives for words that are too difficult, archaic, or uncommon in spoken language. The alternative might be a simpler synonym, a wellunderstood English loanword, or a phrase that explains the meaning of the difficult word. The option of allowing popular English loanwords as replacements yields code-mixed sentences that are conversational, less formal, and typically simpler to comprehend than their monolingual counterparts.
In order to build strong sub-critics, we elaborate on two important design choices below.

heading: 1. Few-shot prompted vs. trained sub-critics.
text: A natural choice for a critic is to prompt an LLM with few-shot examples. We provide examples of difficult sentences in the LLM’s input prompt, along with structural or lexical feedback, and rely on its in-context learning ability to generalize to unseen instances at test time.
Alternatively, we can train models to generate structural or lexical feedback given a complex input sentence. This approach creates the need for a reasonably high-quality corpus consisting of sentencecritique pairs to train the sub-critics. We generate synthetic data via few-shot prompting of the LLM, followed by rigorous data filtering strategies to remove low-quality instances. The synthetic data is used to fine-tune a pre-trained multilingual model like mT5 (Xue et al., 2021) to create a trained subcritic, which can be used at test-time to generate critiques for a test instance. Details about synthetic data creation filtering mechanisms are in Section 3.2. While there is a one-time overhead of training the sub-critics, this additional step is fully justified since we find trained sub-critics to pro-
duce better simplifications, especially in the crosslingual setting. Using smaller trained sub-critics instead of a much larger LLM, also reduces the latency to generate critiques at test-time.

heading: 2. Multilingual vs. Single-language Critics.
text: When training critics, we can either train languagespecific sub-critics or pool all the data for a single dimension and train a multilingual sub-critic. There are trade-offs involved in each option. Languagespecific sub-critics can capture nuances specific to a language, but can also overfit especially when there is limited diversity in the synthetic data. Multilingual sub-critics can learn language-agnostic TS patterns from data pooled across languages that might help cross-lingual generalization, but they are also prone to forgetting due to language interference. We find that multilingual critics outperform their single-language counterparts. This is also a more efficient solution, since we get performance gains from a single model per critic, instead of a model for every critic, language pair.

heading: 3.2 Data Pipeline to Train Critics

heading: 3.2.1 Synthetic Data Generation
text: For all three languages, we start with a monolingual corpus and use a few-shot prompted LLM to generate structural and lexical feedback. The few-shot prompt includes a brief task description, followed by k in-context examples. For Hindi, we set k to 7; for Bengali and Marathi, k is 4. For all three languages, the in-context examples are sampled from our manually curated set of 20 Hindi examples. For Bengali and Marathi, the prompt includes an explicit instruction that Hindi examples are illustrative only, and the LLM should generalize to the target language at inference. Since our critic modules are instructed to provide feedback about text complexity, the prompt also includes examples where the input text is already simple and does not require any modifications.
The few-shot prompt for the lexical critic is shown in Appendix C. The lexical critic itself contains two sub-modules. The first is a classification module that classifies every word in the input text as easy or difficult. The second is a replacement module that suggests a simpler alternative for difficult words identified by the classification module. The output is structured with one suggested replacement per line. While it is possible to merge the two components in a single prompt (i.e. identify difficult words and suggest alternatives), we find that
breaking the task into separate components is specially crucial in the cross-lingual setting.
The few-shot prompt for the structure critic is shown in Appendix C. The prompt contains examples of inputs that require paraphrasing, deletion, or splitting to simplify. Examples contain a chain-of-thought style reasoning, followed by a suggested re-phrasing of the input. To prevent overtriggering, we also provide in-context examples where the input text is simple enough and requires no modifications.

heading: 3.2.2 Data Filtering
text: After generating synthetic data using few-shot prompts as outlined in Section 3.2.1, we adopt the following data filtering steps to maintain the quality of training instances:
• Remove empty or ill-formatted LLM generations.
• Ignore suggestions for lexical replacements when the suggested replacement is nearly identical to the original. This can happen when the LLM suggested replacement is just a different inflectional form of the original word instead of being a simpler substitute.
• Ignore suggestions for lexical replacements when the suggested replacement is for an English word. Since Indian languages have incorporated popular English loanwords as part of colloquial usage, we do not want to replace English words in the original text. We run a token-level language identification pass on the inputs, and remove suggestions where the original word is identified as English.
• Remove structural feedback if incorporating the feedback may lead to meaning loss or incoherent text. Since we do not have a way to directly evaluate the quality of the structure critique, we do it heuristically. The structural feedback is fed to the refiner module to obtain a likely simplified text. If the resulting simplified text has a compression ratio of more than 1.2 or less than 0.7, the corresponding structure critique is deemed to be of poor quality and removed from the training dataset.

heading: 3.3 Refiner Module
text: We re-use the LLM as a refiner, that takes the original complex text along with feedback from the critics as input and produces simplified text as output.
We observe that existing LLMs (such as PALM 2) are proficient at incorporating feedback and simplifying the text, while also ensuring that the generated output is fluent, coherent and semantically similar to the input.
Given the presence of sub-critics in our framework, we have the option of two refinement strategies that we will refer to as eager and lazy refinement. In the eager refinement strategy, the text first undergoes structure simplification, followed by lexical simplification (or vice-versa). The refiner acts on a single critic’s feedback one-at-atime, thereby reducing the complexity of the refinement task. This can be especially important in the cross-lingual setting where the LLM is already burdened with generalizing to a language different from that of the in-context examples. In the lazy refinement strategy, we concatenate feedback from the structure and lexical sub-critics to create a single piece of composite feedback, which is then passed to the refiner. While this increases the complexity of the refinement task, it is more efficient than eager refinement because we incur only a single LLM inference call for refinement.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: Human-annotated test set. To the best of our knowledge, there are no existing TS datasets for Indian languages. As part of this work, we create and publish the first human-annotated TS datasets for Hindi, Bengali, and Marathi. All simplification metrics are reported on these datasets. We hope to be able to contribute these datasets to the multilingual benchmark MULTISIM (Ryan et al., 2023), that currently does not support any of these languages.
For Hindi, our dataset comes from two sources - WMT 2013, and scraped Wikipedia articles. News articles and Wikipedia pages are a natural source of complex and formal sentences, making them suitable for a TS task. We extract sentences and filter them for length between 5 and 30 tokens. We select 500 sentences from the WMT newscrawl corpus, and 541 sentences from Wikipedia articles. Each sentence is simplified and reviewed by trained Hindi language experts. For Bengali and Marathi, our dataset contains 500 sentences each, derived from the WMT 2013 and WMT 2018 newscrawl respectively.
Training dataset for critic modules. To train the lexical and structure critics, we start with 140,000 sentences from the WMT 2020 newscrawl for each language. We filter out sentences that are too long or too short, and retain sentences that contain between 5 and 30 tokens. Starting with this monolingual corpus, the training datasets are created as described in Section 3.2.

heading: 4.2 Metrics
text: To assess how well a model performs on TS, we evaluate it using a set of automatic metrics. Model outputs are evaluated along two dimensions - 1) simplification, and 2) semantic similarity. As is standard in prior work, simplification is measured using SARI (Xu et al., 2016). This is an n-gram based metric that compares model outputs against human references, and explicitly measures the goodness of words that are added, retained, and deleted by the model. Semantic similarity with the input sentence is measured using BERTScore (Zhang* et al., 2020). This metric matches words in the input and output text by computing cosine similarities between their contextual embeddings.
Both SARI and BERTScore are computed using the EASSE package (Alva-Manchego et al., 2019), and a higher score indicates better performance. However, previous work suggests that BERTScore has certain limitations: despite relying on contextual embeddings, it is biased towards rewarding sentences with higher lexical overlap. To contextualize and compare BERTScore between models, we establish a skyline by computing BERTScore between input sentences and human references. Since human references are expected to be largely meaning-preserving, the skyline establishes a fairly high bar. An ideal TS model should be able to simplify the input (=high SARI), without altering the meaning of the original text (=low deviation from BERTScore skyline).
Additionally, we report the percentage of model outputs that are identical to the input. For this metric, a lower score indicates better performance, since we want the model to simplify the input text instead of merely copying it.
It is worth noting that there is an inherent tradeoff between these metrics. For instance, models that tend to copy the input (i.e. higher %age of identical outputs) would naturally do a good job of meaning preservation, resulting in a higher BERTScore. However, copying would simultaneously result in a low SARI because the input text
has not been simplified. In isolation, none of these metrics is a good indicator of model performance; they are meant to be assessed together.
Two prevalent TS metrics, FKGL (Kincaid et al., 1975) (a readability metric) and LENS (Maddela et al., 2023) (a learned metric), are not reported in our work because they are available only for English.

heading: 4.3 Baselines
text: In this work, we focus on comparing our method with existing approaches to self-refinement. Below we list baselines for each language.

heading: 4.3.1 Hindi
text: Few-shot: Input prompt includes a brief task description followed by 10 human-written in-context examples of {input, simplified output} pairs. Chain-of-Thought (CoT): Input prompt includes a brief task description followed by 10 humanwritten in-context examples of {input, analysis, simplified output} tuples. The analysis provides feedback about lexical and structural complexity of the input. For instance, it highlights difficult words or repetitive phrases and offers simpler alternatives. Critique-Revision: We use a "critique request" and "revision request" prompt, both implemented using standard few-shot prompting. The critic request prompt contains human-written examples of {input, analysis} pairs, whereas the revision request prompt contains human-written examples of {input, analysis, simplified output} tuples. Collective-Critiques and Self-Voting (CCSV): This is a self-refinement strategy proposed by (Lahoti et al., 2023). It starts with an initial generation, followed by sampling multiple critiques and revision drafts from the LLM. Once multiple candidate revisions are available, authors prompt the LLM to self-select and vote on the best response. We extend this method for TS and design appropriate prompts for the task. A crucial difference is that while (Lahoti et al., 2023) propose 0-shot CCSV, we find that 0-shot prompting fails to produce meaningful critiques for TS. We instead implement 1-shot CCSV, with a single human-written example per prompt.

heading: 4.3.2 Bengali and Marathi
text: Cross-lingual few-shot: Same as few-shot prompting, except that the human-written incontext {input, simplified output} pairs are in Hindi instead of being in the target language.
.
Cross-lingual CoT: Same as Chain-of-Thought, except that the human-written in-context {input, analysis, simplified output} tuples are in Hindi instead of being in the target language. Translation few-shot: Input prompt includes a brief task-description followed by machinegenerated examples of {input, simplified output} pairs. These in-context examples are derived by translating both input text and simplified output from Hindi to Bengali or Marathi, using the Google Translate API. Translation CoT: Input prompt includes a brief task-description followed by machine-generated examples of {input, analysis, simplified output} tuples. We use translated input and simplified outputs, as above. Hindi phrases and suggested wordreplacements in the analysis string are also translated to Bengali or Marathi.

heading: 4.4 Implementation
text: We use the instruction-tuned PaLM 340 billion params model (Chowdhery et al., 2023; Longpre et al., 2023) as the base LLM for all experiments.
For training the critic models, we use mT5-xxl as the base model (Xue et al., 2021). Inferences for all models, methods and baselines are performed using top-1 decoding at a sampling temperature of 0.7 and 256 decode steps. All numbers are reported over a single run.

heading: 5 Results
text: Tables 1 and 2 summarize our main results for Hindi, Bengali, and Marathi. For Hindi, DIMSIM outperforms all other methods, with a near 3 point improvement in SARI scores (44.31) compared to CCSV, the closest competitor (41.96). It does so while making only 14 the number of LLM calls. Lazy refinement performs slightly better than eager refinement (in both directions). Critique refinement with a single critic is not very effective, and falls behind chain-of-thought prompting by roughly 2 SARI points.
Cross-lingual evaluations for Bengali and Marathi are reported in Table 2. As before, DIMSIM outperforms all other methods, with a gain of close to 2 SARI points over the closest com-
petitor. It is also more efficient, requiring only 14 the number of LLM calls, along with multilingual sub-critics instead of per-language sub-critics.
We observe that with cross-lingual + CoT prompting, the LLM tends to copy inputs more than 50% of the time. This is primarily because it fails to generate a comprehensive chain-of-thought string, and typically regresses to either producing an empty string or irrelevant text in Bengali or Marathi. Using translated examples without CoT slightly boosts SARI scores for both Bengali and Marathi. However, with CoT added, there is a 4 point drop in SARI. Recall that in the translation + CoT approach, Hindi phrases and suggested wordreplacements in the analysis string are also translated to Bengali or Marathi. However, there is no guarantee that words/phrases present in the translated analysis string are actually valid sub-strings of the translated input string. As a consequence, the LLM has to learn from noisy in-context examples, resulting in a performance drop.

heading: 5.1 Human Evaluations
text: We perform both side-by-side, as well as singlesided human evaluations to measure the efficacy of our approach. Rater pool demographics and exact task instructions are reported in the Appendix. Side-by-side: In this eval, human annotators are presented with two texts side by side - the original text, and its model-generated simplification. We ask two questions: 1. Which text is simpler? 2. How similar are they in meaning? Each text pair is evaluated by 3 language experts, and a consensus label is generated via majority vote. Instances without consensus agreement on both questions are removed from analysis.
Table 3 shows results from the side-by-side evaluations of DIMSIM on 200 randomly-sampled sentences from the test set of each language. For Hindi and Bengali, we find that in the majority of instances, raters find the output from DIMSIM to be simpler and also consistent in meaning to the original sentence. Marathi performs a bit worse in comparison, with raters frequently preferring the original input or finding both to be similar. However, DIMSIM outputs are rated simpler than the originals, though not in a majority of cases.
Table 4 compares CCSV (the second-best TS model in terms of SARI) vs. DIMSIM on 200 randomly-sampled Hindi sentences. While the CCSV approach trails slightly in producing simpler outputs, it lags significantly in producing out-
puts that preserve the meaning of the original text. This is corroborated by its BERTScore (in Table 1) which has regressed more than 6 points from the skyline. Single-sided: In this eval, we randomly sample 100 sentences and ask raters to annotate whether it is objectively simple to read and understand. Each sentence is presented to 2 raters, and is deemed to be simple only if both raters consider it simple. Table 5 shows results from a single-sided eval comparing the original and DIMSIM-simplified test corpus for all 3 languages. We see +38 point improvement in the number of simple sentences for Hindi, and more than +20 point improvement for Bengali and Marathi.

heading: 5.2 Impact of Trained Sub-Critics
text: In this section, we attempt to compare the magnitude of lexical and structure changes brought about by different approaches, with the help of a few easy-to-compute metrics. For both kinds of change, we report the trigger rate of few-shot and trained sub-critics (i.e. DIMSIM). A sub-critic is said to trigger for an input if it produces an actionable suggestion to simplify it. To quantify lexical change, we report the total number of word replacements that were suggested by the lexical critic and incorporated in the simplified version. To quantify structure change, we report two metrics: number of sentence-splitting instances, and number of para-
phrasing instances. An instance has undergone splitting if the simplified version contains more sentences than the original. An instance has undergone paraphrasing if the Levenshtein distance between the original and simplified string is greater than 20.
As Table 6 shows, for both lexical and structure sub-critics, the trained variant has a significantly higher trigger rate compared to the few-shot variant. This higher trigger rate, taken together with superior SARI and BERTScores, suggests that the trained critics are making more accurate simplification changes to the input. We further validated this using a small human evaluation exercise. We randomly selected 200 instances where the fewshot sub-critics and DIMSIM produced different refinements of the input text. Human evaluations indicated DIMSIM led to simpler outputs for 32% of the instances, fewshot critics led to simpler outputs in 28% of instances, and raters found both sentences to be at-par for the remaining 40% of instances.
We expect a well-trained critic to trigger less on simpler datasets, and trigger more on a complex dataset. To check whether this holds, we evaluate the trained critic models on 3 different corpora comprising original complex inputs, DIMSIM outputs, and human references. Table 7 shows the trigger rates on all three corpora. As expected, trigger rates are highest for the original inputs, lowest for the human references and trigger rates for DIMSIM outputs are in-between.
These metrics can also help understand differences between DIMSIM and CCSV. Since CCSV does not have explicit sub-critics, it is not straightforward to quantify the magnitude of lexical change. However, it is possible to quantify structure change by comparing the original and simplified version. Recall that CCSV operates by collating multiple sampled critiques from the LLM and revising based on collated critique. Table 6 shows that CCSV is≈2x more likely to trigger a structure change compared to DIMSIM. The large volume of change is because it incorporates diverse feedback from multiple critiques. However, as shown in Section 5.1, CCSV also changes the meaning of the input text. One possible hypothesis is that while trying to incorporate the volume of change brought on by multiple critiques, the LLM is not able to do it in a meaningful way.

heading: 6 Conclusions
text: In this work, we present a new framework for multilingual text simplification using LLMs for selfcorrection. We decouple the simplification problem along two salient dimensions and train individual critics using synthetic data for each critic generated via few-shot prompting of LLMs. We evaluate our framework on text simplification for three Indian languages in a few-shot (for Hindi) and zero-shot (for Bengali and Marathi) settings. While we focus entirely on text simplification in this work, we think our framework is broad enough to be applicable to other generation tasks such as formality transfer, abstractive summarization, etc. and leave this exploration to other tasks as future work.

heading: Acknowledgements
text: The fourth author would like to gratefully acknowledge a faculty grant from Google Research India supporting her research on multilingual models and is also very thankful for Google Cloud credits that helped support all the compute requirements. We are deeply indebted to the Vani and Verse Solutions team (especially Keshav and Deepika Agarwal) for helping us scale and expedite human evaluations on their platform.

heading: 7 Limitations
text: Our contribution in this work is limited to one particular family of languages, for a single task.
We recognize that the principle of using distilled trained critics is more general, and requires rigorous study on a multitude of tasks and languages. Second, methods that use LLMs to generate data may produce patterns of bias or factual inconsistencies, which are likely to propagate to downstream distilled models. Finally, we recognize that while our progress on Hindi and Bengali TS is significant, overall performance on Marathi is relatively low and requires further inspection.

heading: A Human Evaluations
text: We hired a professional data labeling service for all human evaluations. This section provides exact annotation instructions, as well as relevant details about the rater pool.
A.1 Side-by-side Evaluation
In the side-by-side eval, raters were shown a pair of sentences (say, A and B) and asked to evaluate them on simplicity and semantic similarity. Each side-by-side pair was rated by 3 trained humans, followed by majority vote aggregation. When rating a dataset of sentence pairs, the order in which items in the pair are presented (i.e. (A,B) or (B,A)) may bias the raters’ response over time. To mitigate this, items in the pair were randomly ordered so that raters’ do not systematically prefer one side over the other.
A.1.1 Simplicity For this question, raters were asked to select which one of the two texts is more simple, based on the following definition of simplicity: A text is simple if it is easy to understand for an average native speaker. It has simple sentences, everyday vocabulary, and clear structure. It is not verbose or repetitive, and it does not use overtly academic or archaic language.
Raters could select one of 3 possible options: A is more simple, B is more simple, Both are similar.
A.1.2 Semantic Similarity For this question, raters were asked to select how similar are the two texts in their meaning? They could select one of 3 possible options:
Very similar - A and B convey exactly the same information.
Somewhat similar - There is meaning overlap between A and B, but they are not identical. There are some differences in information or context.
Not at all similar - A and B are totally unrelated, there is absolutely no similarity between them.
A.2 Single-sided Evaluation In this task, raters were shown a text and asked a binary question: Is this text simple to read and understand?. The definition of simplicity was identical to that of the side-by-side evaluation.
A.3 Rater Pool Evaluations were performed by bilingual raters from India, who are fluent in English and one of {Hindi, Marathi, Bengali}. Every rater either held or was working towards a Bachelors degree. Raters were paid USD 0.12 for a completed side-by-side task, and USD 0.10 for a completed single-sided task.

heading: B Sample Hindi Outputs
text: In Table 8 we show sample outputs from our best performing model, along with edit operations being performed. The examples show different operations like word replacements, sentence splitting, phrase rewrites, and paraphrasing been done by the model.

heading: C Critic Prompts
text: The lexical critic module contains two sub-modules. The first is a classification module, the second is a replacement module. For brevity, we show the prompts for Bengali in Figure 2 and 3; prompts for Hindi and Marathi are similarly written. As shown in the prompt, the model is tasked to classify each word as easy or difficult in the classification prompt. We provide six set of rules to the model to help it classify the word as easy or difficult. In the subsequent task, the model is provided the replacement prompt to act upon the difficult words earlier identified. Again, we provide three steps in the prompt
to help the model to come up with simpler alternatives. The structure critic prompt is presented in Figure 4. Here, the model is prompted to critique the structure of the text, thereby highlighting a modified structure of the provided text that could make it more readable.

heading: Future Work
text: and leave this exploration to other tasks as future work