# Automatic Simplification of Lithuanian Administrative Texts
## 6 Results
### Qualitative Evaluation  
#### 6.2.1 mT5 and mBART  
To assess the results of the mT5 and mBART text simplification models, two linguists, familiar with Plain Lithuanian, independently evaluated \(n=554\) model-simplified sentences, comparing them to the original ones in the test set, according to three criteria, indicated in Table 4. To assess evaluators' agreement in assigning their scores, inter-rater reliability has been evaluated via [PERSON]'s Kappa scores [90].  
As Table 3 shows, while the criteria of meaning retention and grammaticality demonstrate a fair agreement between the two raters, simplicity, i.e., how well the sentence that has been simplified by mT5 and mBART models correspond to Plain Lithuanian, only shows very slight inter-rater reliability. The raters discussed all the criteria and the approximate significance of each 1 to 5 score before rating the simplified sentences; however, it seems that while it is fairly straightforward to agree on whether the sentence is grammatical, whether the meaning is retained, and whether it is simpler than the original, assigning a uniform rank from 1 to 5 for simplicity generated more discrepancy. Plain Lithuanian is a new language variety even for the experts, who are very familiar with the Plain Language rules.  
Regarding the results of the qualitative assessment, Table 4 shows that mBART scored consistently better on all three criteria. A closer qualitative analysis revealed that both models did comparatively well with shorter sentences to simplify, but differed considerably in their treatment of long sentences, which are characteristic of the Lithuanian formal and administrative language registers [91]. Notably, mT5 at times did not do well with longer passages, as it had a tendency to output nonsensical repetitions of words instead of coherent sentences or deleting whole passages from the original text, instead of transforming them into new shorter sentences. This lowered its scores considerably on all three criteria.  
According to evaluation results, _both models scored better on meaning retention and grammaticality than on simplification_. This resulted in sentences that were only slightly simplified, while some elements were left unaltered that were in need of simplification. Such partly performed simplifications were assigned a simplicity score of 2. These unaltered or mostly unaltered sentences were, naturally, grammatically correct and retained the full meaning of the original sentence, resulting in scores of 5.  
However, the raters noticed that _both models had learned certain linguistic structures, most characteristic of Plain Lithuanian, such as adding subject and possessive pronouns as well as demonstratives and transforming the passive voice into the active voice_. The latter may pose problems with subject identification if the original sentence in passive did not include an oblique agent phrase, but it did not turn out to be a frequent issue, as in most cases, adding a first-person plural pronoun in subject position was sufficient. Finally, _mBART especially managed quite well to transform participial clauses into relative clauses with finite verbs_.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Simplicity** & **Meaning Retention** & **Grammaticality** \\ \hline
**mT5** & 2.67 & 3.16 & 3.33 \\ \hline
**mBART** & 2.80 & 3.72 & 3.88 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Qualitative evaluation scores (averages).  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Criterion** & **Score** \\ \hline Simplicity & 0.1006 \\ \hline Meaning retention & 0.3035 \\ \hline Grammaticality & 0.3338 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Cohen’s Kappa scores.  
On the downside, there were very few sentences that got full scores of 5-5-5. Thus, _the models may need more and/or more diverse training data, before they can be considered reliable._