PromptARA: Improving Deep Representation in Hybrid Automatic Readability Assessment with Prompt and Orthogonal Projection
Authors: Jinshan Zeng
Xianglong Yu
Xianchao Tong
Wenyan Xiao
Venue: EMNLP 2023 Findings (2023)
Keywords: Readability assessment; Deep learning; Prompt learning; Orthogonal projection layer; Linguistic feature
TL;DR: This paper proposes a novel model called PromptARA for the automatic readability assessment through employing prompts and an orthogonal projection layer to improve deep feature representations.

Abstract: Readability assessment aims to automatically classify texts based on readers' reading levels. The hybrid automatic readability assessment (ARA) models using both deep and linguistic features have attracted rising attention in recent years due to their impressive performance. However, deep features are not fully explored due to the scarcity of training data, and the fusion of deep and linguistic features is not very effective in existing hybrid ARA models. In this paper, we propose a novel hybrid ARA model called PromptARA through employing prompts to improve deep feature representations and an orthogonal projection layer to fuse both deep and linguistic features. A series of experiments are conducted over four English and two Chinese corpora to show the effectiveness of the proposed model. Experimental results demonstrate that the proposed model is superior to state-of-the-art models.

PromptARA:
Improving Deep Representation in Hybrid Automatic Readability Assessment with Prompt and Orthogonal Projection
Jinshan Zeng', Xianglong Yu', Xianchao Tong', Wenyan Xiao
School of Computer and Information Science, Jiangxi Normal University Schooo of Digital Industry, Jiangxi Normal University
School of Foreign Languages, JiangXi University of Science and Technology jinshanzeng@jxnu. edu. en, xianglongyu@jxmu. edu. cn xianchaotong@ edu. wy. xiao@jxxstt.tmmmmmm.o.i...u..sst edu. cn
Abstract
developed through empirical pedagogy and psy- chology (Klare, 1963; Davison and Kantor, 1982). Although these formulas are casily interpretable. they rely on surface features and cannot measure the structure or semantic complexity of a text, re- sulting in unsatisfactory performance.
Readability assessment aims to automatically classify texts based on readers' reading levels. The hybrid automatic readability assessment (ARA) models using both deep and linguistic features have attracted rising attention in re- cent years due to their impressive performance. However, deep features are not fully explored due to the scarcity of training data, and the fu- sion deep and linguistic features not very effective in existing hybrid ARA models. In this paper, we propose a novel hybrid ARA model called PromptARA through employing prompts to improve deep feature representa- tions and an orthogonal projection layer to fuse both deep and linguistic features. A series of experiments are conducted over four English and two Chinese corpora to show the effec- tiveness of the proposed model. Experimental results demonstrate that the proposed model is superior to state-of-the-arr models.
Then, the traditional machine learning methods have been applied to train statistical classifiers for ARA. These classifiers exploit large number of features at various levels of a text, including but not limited to vocabulary, semantics and syntax (Hancke al., 2012; Sung al., 2015; Dell' Orletta et al., 201 Francois and Fairon, 2012: Denning et al., 2016; Arfé et al., 2018; Jiang al., 2019). Although they often achieve better performance than readability formulas, the feature engineering and selection for these machine learning methods are generally time-consuming and labor-intensive.
In recent years, deep learning methods have shown impressive performance in natural language processing and their application in ARA has been intensively studied. Pre-trained word embedding models (Mikolov al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) and masked lan- guage models such BERT (Devlin al., 2019) have been exploited by many neural ARA mod- els (Deutsch et al., 2020; Tseng et al., 2019; Zeng et al., 2022). However, the performance of these deep learning methods is limited by the scarcity of training data in many ARA tasks.
1 Introduction
Text readability assessment aims quantify the difficulty of a text, that is, the degree to which can be easily read and understood (McLaughlin, 1969; Klare, 2000). Due to the superiority of an automatic readability assessment (ARA) system on assigning text to a difficulty grade. ARA is useful for identifying texts or books that are suitable for individuals according to their language proficiency, intellectual and psychological development. The studies on ARA can be traced back to the last cen- tury (Lively and Pressey, 1923; Klare, 1963) and have attracted rising attention in recent years, with impressive performance yielded by many neural ap- proaches (Tseng et al., 2019; Schicchi et al., 2020; Azpiazu and Pera, 2019; Deutsch al., 2020; Mar- tine al., 2021; Lee et al., 2021; Vajjala, 2022;
To further improve the performance of neural ap- proaches based on deep learning, the hybrid ARA models utilizing both linguistic and deep features have been recently studied in the literature (Qiu et al., 2018; 88; Deutsch et al., 2020; Lee et al., 2021; Li et al., 2022). The literature (Deutsch et al., 2020; Lee et al., 2021) investigated the joint effect of handcrafted linguistic features and deep features extracted by a deep neural network, where deep features and hand-crafted linguistic features were simultaneously fed into a machine learning model
Tanaka-Ishi et al., 2010; Lee and Vajjala, 2022; Zeng et al., 2022).
In the carly stage, studies in ARA mainly fo- cused on readability formulas, which are typically

---

without fusion to yield readability levels. The re- cent literature (Li et al., 2022) introduced certain difficult--wwart topic features through utilizing the word difficulty knowledge to guide the training of topic model, and fused the deep features and lin- guistic features by a projection scheme.
2 Related Work
Text exhibits an inherent structure, and its diffi- culty is manifested through various linguistic levels, encompassing words, sentences, and entire docu- ments. How to incorporate more effective informa- tion of text into ARA is one of the current research directions of ARA. As the current mainstream mod- els, deep ARA models is mainly based on the hier- archical attention networks (HAN) and pre-trained models (Yang al., 2016; Azpiazu and Pera, 2019; Zeng al., 2022). The basic idea of such kind of models to use multi-level deep representations of texts to yield better representations for predic- tion. In Azpiazu and Pera (2019), a hierarchical attention model called Vec2Read was suggested for the multilingual ARA task, where the word-level information such the grammatical and morpho- logical information was introduced. In the recent literature (Zeng et al., 2022), novel HAN-type model was suggested mainly based on the idea of ordinal regression.
Although the hybrid ARA models have achieved the Mfte-offtteeaa art performance, there are two lim- itations for these models. The first one is that deep features are not fully explored in hybrid ARA mod- els due to the scarcity of training data. In most of existing neural ARA models such as Zeng et al. (2022), deep features are generally extracted by the pre-trained large language models, which com- monly require a large amount of training data. The second one is that the fusion of deep features and linguistic features in existing hybrid ARA models is not paid particular attention.
In this paper, we employ prompts to improve deep feature representations inspired by the great success of prompt learning (Lee and Lee, 2023; Liu et al., 2023; Schick and Schütze, 2021), and an orthogonal projection layer to effectively fuse the improved deep features and linguistic features and particularly remove the redundant information among these features. The major contributions of this paper can be summarized as follows.
Noticing that deep features extracted by the kind of HAN models generally have poor interpretabil- ity and their capacity is limited by the scarcity of training data, the hybrid ARA models by using both linguistic features and deep features have at- tracted rising attention in recent years (Deutsch et al., 2020; Qiu et al., 2021; Lee et al., 2021; ;; Li et al., 2022). In Deutsch et al. (2020), the authors directly used deep and linguistic features to jointly train statistical classifier. In Qiu et al. (2021), the authors firstly yielded an improved linguistic representation by learning the relevance of the used linguistic features and then fused the improved lin- guistic representation with the deep representation by a simple concatenation to yield a new repre- sentation for prediction. The recent literature (Lee et al., 2021) deeply investigated the effectiveness of the linguistic features when incorporated into the deep models for ARA. In Li et al. (2022), the au- thors firstly introduced deep topic feature through introducing the difficulty knowledge, and then sug- gested a projection scheme to fuse the deep and linguistic features by removing redundant informa- tion.
We propose novel hybrid ARA model called PromptARA through exploiting prompts to improve the extraction of deep features, and an orthogonal projection layer to effectively fuse linguistic features and deep features at various levels of text, including word, sen- tence and document levels. The proposed model can focus more on some in-domain in- formation of the text with the help of prompts and reduce the redundant information among different levels of feature representations.
Extensive experiments are conducted over four English benchmark corpora and two Chi- nese corpora to validate the merits of the pro- posed model through comparing with many state-of-the-an models. Experimental results demonstrate that the employed prompts and orthogonal projection layers are effective and that the proposed model is superior to state- of-the art models in most corpora in terms of several important evaluation metrics.
Although existing hybrid ARA models achieve the sate-of-the-art performance, the deep features used in these models are extracted by some pre- trained language models, which generally require a large amount of training data when adapted to downstream tasks. Thus, the performance of these

---

Readability
few Concat Linear Layer & Softmax
Orthogone
Orthagonal Layer
Orthogonal Layer
Orthogonal Layer
Mix-pooling
Linear Layer
Kesi Kaa]
BigBird
Layer Norm
Mix-pooling
rd
Linguistic Feature
Moe cllg
Nean-Pooling
Tell
Traditional Feature Extractor
inpu:
Big3ird
Sentence Segmentation
prompt-tex:
Figure I: Overview of the proposed model.
hybrid ARA models is limited by the scarcity of training data. The recently suggested prompt learn- ing (Schick and Schütze, 2021) provides a potential way to address this issue. In the recent literature (Lee and Lee, 2023), the authors introduced prompt learning to the problem of judging the difficulty for two given texts, which can be roughly regarded binary classification problem, while the ARA task considered in this paper can be generally re- garded multi-classification task. Inspired by the great success of prompt learning, this paper em- ploys prompts to improve the deep representations represented in multiple levels, and an orthogonal projection scheme to effectively fuse the deep and linguistic features for prediction.
orthogonal projection layer. With these refined four types of features, the prediction the readability level for a given text is yiclded by a linear classifier.
3.1 Deep Representations with Prompts
To fully explore deep features from texts, we consider three levels of deep features, where the document- and word-level deep features are ex- tracted from the extended texts with prompts while the sentence-level deep features are yielded from original texts. For the sentence-level deep features, extract them directly from original texts since the prompts designed in this paper tences, which may provide little help for the extrac- tion of sentence-level deep features. Some prompts are presented in Table 1.
3 Proposed Model
A. Document-level deep representations. As depicted in Figure 1, we employ the well-known pre-trained Transformers for Longer Sequences (BigBird) model (Zaheer al.. 2020) extract the document-leve deep features fa from extended texts with prompts, inspired by the advantage BigBird model on dealing with long sequences. B. Word-level deep representations. To yield the word-level deep representations, we firstly feed extended texts with prompts to BigBird ob- tain word-level embeddings and then implement a mix-pooling operation for word-level embeddings. As depicted in Figure I, the mix-pooling opera-
In this section, we describe the proposed model detail. The overview of the proposed model is de- picted in Figure! 1. As shown in Figure four types features including the document-, sentence- and word-level deep features and the traditional lin- guistic features are utilized for prediction. The document- and word-level deep features tracted by texts with prompts, while the sentence- level deep features and traditional linguistic fea- tures are extracted from the original texts. The prompt guided document-level features are utilized to improve the other three types of features via an

---

Prompts
The text readability classification task is currently in progress.
The task is divide the text into difficulty levels.
After reading this article, give the difficulty level of the article.
The readability task is currently underway to determine the difficulty of the text.
Table 1: Some examples of prompts designed for ARA. This paper uses the first prompt as the default one.
tion is the mixing of the well-known max-pooling and mean-poone mrean-pooling operations, aiming to extract the maximum and mean values.
getting rid of the redundant information between these two different levels of representations. In this paper, we take the document-level representation as the main representation and the other three levels of representations as the auxiliary representations, and implement the orthogonal projection operation for the other three levels of representations to keep their orthogonal parts important supplemental information for the main representation.
C. Sentence-level deep representations. To yield the sentence-level deep representations, we firstly segment original texts into single sentences, then feed them into BigBird yield sentence-level embeddings, and later implement the mix-pooling for sentence-level embeddings.
With these representations after orthogonal lay- ers, we concatenate them and feed them into a linear layer followed by the softmax as a statistical classifier to yield the prediction of readability level.
3.2 Linguistic Feature Representations
Many previous studies have shown that linguistic features can provide additional textual information for the deep neural models and thus improve their performance (Sennrich and Haddow, 2016; Qiu al., 2018; Lee al., 2021; Li et al., 2022). As depicted in Figure 1, to yield the linguistic feature representations, we firstly extract a large number of linguistic features at various levels by the traditional feature extractors, where the spe- cific linguistic features for the Chinese texts presented in Tables 8 and 9 in Appendix. For En- glish traditional features, we extract them by im- plementing the lingfeat toolkit developed in Lee et al. (2021) Then, we implement the layer nor- malization for linguistic features for the purpose of training stability. Finally, we yield the linguistic representations by projecting the normalized lin- guistic features into a common space with the same dimension of deep representations.
Experimental Settings
In this section, we describe the experimental settings in detail. We firstly describe the six cor- pora used in the experiments including four English benchmark corpora and two Chinese corpora, and then present the baseline models, and finally de- scribe some implementation details.
4.1 Corpora
We conducted experiments four English cor- pora and two Chinese corpora demonstrate the effectiveness of the proposed model. Some statis- tics of these corpora are presented in Table 2.
Weebit (Vajjala and Meurers, 2012). The Weebit corpus is combined five-level corpus created based WeeklyReader' and BBC-Bitesize², con- taining total of 6388 texts. It often considered as the gold standard for ARA models. For each difficulty category we perform 625 text downsam- pling.
3.3 Fusion by Orthogonal Layers
Noticing that there is redundant information be- tween deep and linguistic representations, we em- ploy the orthogonal projection layer to remove the redundant information, inspired by the literature (Li et al., 2022). As depicted in Figure 1, given main representation and an auxiliary representation, we implement orthogonal projection for these two representations, that is, keep the orthogonal part of the auxiliary representation while eliminating the projection part of the auxiliary representation along the main representation, with the purpose of
Cambridge' (Xia et al., 2016). The levels of the Cambridge English tests (KET, PET, FCE, CAE, CPE) are used categorize articles. For each diffi- culty category, we performed downsampling on 60 texts.
hhtp:://wwwwbbb uk/bitesize http://www.camm http://wwwwwwbbidgniiiisi.ooogo.

---

Properties
Weebit
Cambridge
Newsela English
CLEAR
CMER
CMT
Language category
English
English
English
Chinese
Chinese
Number of classes
5
12
Number of texts
3125
300
9565
4724 171.96
2260
2621
Average length
288
510
747.37
926.94
674.72
Table 2: Statistics for the used English and Chinese corpora.
Newsela* (Xu et al., 2015). Newsela contains 10,786 texts, out which we selected 9565 texts in English. The dataset is a parallel corpus of original and simplified document alignment versions, cor- responding to 11 different unbalanced grade levels (from grade 2 to grade 12).
networks proposed by Yang et al. (2016). It em- ploys static word embeddings and utilizes two hier- archical attention mechanisms at both the word and sentence levels to pay attention to salient words and sentences. This enables the model to assign vary- ing levels of attention to content of differentt impor- tance when constructing text representations. We adopted the same framework used by Martine al. (2021), who replaced Bi-GRU with Bi-LSTM. BigBird (Zaheer et al., 2020) is a model based on Transformer with sparse attention mech- anism, which can handle up to 4096 tokens and get better performance. We use it for ARA for fine-tuning.
CLEAR (Crossley et al., 2022). The corpus comprises 4724 text excerpts and offers a distinct measure of readability for each text, designed for readers in grades 3-12. It also includes metadata such as the year of publication, genre, and other relevant information for the excerpts. The CLEAR corpus represents significant advancement over previous readability corpora in terms of the size and diversity of available excerpts. encompasses more than 250 years of writing across two different genres, and provides a unique criterion for readabil- ity based on teachers' evaluations of text difficulty for student readers. To adapt to the task at hand, we utilized the Lexile Band, resulting in the creation of ten classes.
DTRA (Zeng al., 2022) is model based on BERT word embeddings with HAN-like structure. learns the sequential information of inter-textual difficulty by predicting the relative difficulty of paired texts and using distance-dependenn soft la- bels.
Lee-2021 (Lee et al., 2021) is a hybrid ARA model, where three novel features are proposed in terms of high-level semanties, and the deep and lin- guistic features are directly used to train statistical classifier.
CMT (Cheng et al., 2020). The CMT consists of texts from Chinese textbooks used in mainland China, ranging from the first grade of primary school to the third grade of high school. This cor- pus comprises total of 2.621 texts.
BERT-FP-LBL (Li et al., 2022) is a hybrid ARA model that utilizes the orthogonal projection to fuse the linguistic features and deep features.
CMER (Zeng al., 2022). CMER was col- lected by Zeng al. (2022) from extracurricular books targeted towards children and young adult in the mainland Chinese book markets. The corpus consists of 2,260 texts, which are categorized into 12 categories, aligning with grades through 12.
4.3 Implementation Details
For two Chinese corpora, we extracted 67 lin- guistic features different levels including lexical, semantic, syntactic, and cohesion aspects. As for the four benchmark English corpora, we extracted the linguistic features using the toolkit suggested in Lee et al. (2021) at various levels such as the discourse, syntactic, lcxical and surface levels.
4.2 Baselines
We consider the following state-of-the-art mod- els as the baselines to verify the effectiveness of the proposed model.
BERT (Devlin et al., 2019) represents fine- tuning using the default BERT model.
We evaluated the proposed model in terms of the following four commonly used metrics for classi- fication, i.e., accuracy (Acc), precision (Pre), the macro F1-metric (F1), Quadratic Weighted Kappa (QWK), where the QWK metric is often used
HAN is a model based on hierarchical attention
'https://newslla.c hwwp://.wwwwll.comm
"htps:///ittub.. wtw.aa//ooooooeeeeeCeCAAAAA.....
hwtp:/uuwwooooonnnnannnnnn/n///e Readabllity
"nugss//tuggiigggeeeee/ooooggggu bobertt-bbe whtps://huggingffcc.ooo/l http://hhhgga...oooooooooeiioihe bighird-bhieeee 4096
"htps://hugginggaae co/bert-buse-uncased
"htps://hugiingfac. httpg/gg....ggcttbeeeeeeeeeeeeee co/bert-buse-chinese

---

Dataset
Cambridge
Weebit
CLEAR
Newsela
CVER
CMT
MaxLen
512
2048
2048
2048
Epoch
30
30
30
30
20
20
Learn.
Table 3: Hyperparameter settings for PromptARA.
assess consistency or reliability among multiple assessors, and it provides more accurate insights and decisions in the area of readability assessment. We used the AdamW optimizer (Loshchilov and Hutter, 2017) with a weighting decay parameter 0.01 and a warm-up ratio 0.1 to train the proposed model. All experiments were conducted on RTX 3090 and A40 GPUs, and implemented using the PyTorch framework. Some hyperparameters of the proposed model are presented in Table 3.
(a) HAN
(b) DTRA
(c) BigBird
(d) PromptARA
In particular, since cannot access the repro- ducible source codes, is not easy to reproduce the results in the recent literature in Lee al. (2021) and Li et al. (2022). Thus, we directly took their reported results as the comparison results. For fair comparisons, we followed the similar settings of Lee et al. (2021) and Li et al. (2022) to implement other baselines. Specifically, for each corpus, divided it into the training, validation and testing sets using an 8:1:1 ratio for three times. The experi- mental results were recorded on average by running three trails.
Visualization results of confusion matrices of four deep learning models over Newsela, where the horizontal- and vertical-axis of each figure represents the predicted categories and the true categories of the samples, respectively.
Figure 2:
When compared to these two HAN-type mod- els, i.e., HAN and DTRA, the proposed model significantly outperforms these two deep models with HAN-like structures. It is noteworthy that the multi-level deep representations in HAN and DTRA represented and fused in hierarchical way, whereas the multi-level deep representations in the proposed model are yielded in parallel way and fused by an orthogonal projection scheme, to- gether with the linguistic features. These results show that the proposed model can explore more deep features for ARA.
5
Experimental Results
In this section, we describe and analyze the ex- perimental results in detail.
5.1 Performance Evaluation
Tables 4 and 5 present the experimental results of the proposed model and baselines on the English and Chinese datasets, respectively. Since these two Chinese corpora are not considered in the litera- ture Lee al. (2021) and Li et al. (2022), we do not compare these two baselines over the Chinese corpus as shown in Table
Compared to these two hybrid ARA models Lee et al. (2021) and Li et al. (2022) using both deep and linguistic features, the proposed model also outperforms them over the Cambridge and Weebit corpora in terms of most evaluation metrics. We observe from Table 4 that except the QWK value on Weebit, the proposed model achieves the best results for all other cases. This shows the effectiveness of the proposed models in comparison of existing hybrid ARA models.
A. Evaluation over English Corpora. When re- garding the performance over English corpora, we can observe that the proposed model outperforms these baselines in most of cases. Specifically, when compared to the base model BigBird using only the document-level representations from original texts, the proposed model yields much improvement over all English corpora. This demonstrates that the suggested prompts, word-level and sentence-level deep representations, as well the linguistic features in the proposed model are useful for ARA.
Moreover, can be observed from Table 4 that the performance of BigBird much better than BERT, another pre-trained model used in ARA, mainly due to the superiority of BigBird on dealing with longer sentences.
We further provide some visualization results terms of confusion matrices to demonstrate the ef-

---

BERT-FP
Model
HAN
BERT
DTRA
BigBird
Lee2021*
PromptARA
LBL*
Acc
76.67
75.56
77.78
87.78
76.30
87.78
91.11
Pre
80.49
72.75
79.71
88.29
79.20
89.46
92.24
Cambridge
75.77
72.95
77.07
87.53
75.20
87.73
90.88
QWK
92.64
91.59
92.62
97.04
91.90
97.82
Acc
82.54
91.33
85.29
92.70
90.50
92.70
93.12
Pre
83.73
91.56
85.54
92.73
90.50
92.89
93.19
wcehit
F1
82.76
91.51
85.30
92.70
90.50
92.73
93.09
QWK
94.48
97.10
95.65
97.17
96.80
97.78
97.43
Acc
67.87
76.74
72.09
78,86
82.03
Pre
66.22
76.23
70.75
79.01
81.99
CLEAR
Fl
66.43
76.05
70.81
78.34
81.75
QWK
89.29
92.86
91.85
94.03
94.54
Acc
83.80
77.12
83.07
87.15
88.40
Pre 83.86
78.45
82.96
87.14
88.21
Newsela
F1 83.70
76.59
82.82
87.05
88.24
QWK
98.35
97.67
98.41
98.79
98.88
Table 4:
Comparison results of the proposed PromptARA model and baselines over four English benchmark corpora. Experimental results taken directly from the literature. The best results are marked in bold.
Model
HAN
BERT
DIRA
BigBird
PromptARA
contributing to the observed performance differ- ence.
42.53
38.46
44.42
38.46
43.95
Pre
40.57
38.79
44.24
38.89
43.17
CMT
F1
41.09
37.17
43.87
37.81
41.60
Acc
23.40
22.30
26.50
23.84
26.50
5.2
Ablation Studies
CMER
18:48
13.49
25.16
23.57
23.92
OWK
72.10
65.39
70.53
73.74
68.74
In this subsection, we conducted series of abla- tion studies over two English datasets (Cambridge and CLEAR) and two Chinese datasets (CMT and CMER) to validate the effectiveness and feasibility of our proposed ideas. In view of the deficiencies the Chinese dataset the comparative experiments, wc also used other pre-trained models to carry out relevant experiments. Besides the proposed Promp- ARA model, we considered the following three models:
Table 5: ARA performance on the Chinese datasets. The best and second best results are marked in bold and blue color, respectively.
fectiveness the proposed model over Newsela, as depicted in Figure 2. We can observe from Figure 2 that the confusion matrix yielded by the proposed model is more concentrated to the diagonal. This also shows the effectiveness of the proposed model compared to the baselines.
w/o L denotes the model that does not use the linguistic features PromptARA.
B. Evaluation over Chinese Corpora. When evaluating the performance of the proposed model over these two Chinese corpora, i.e., CMT and CMER, we can observe that the proposed model achieves the competitive results in comparison of baselines. Specifically, the proposed model achieves the best result in terms of the QWK and the second best results terms of the other three evaluation metrics over CMT corpus, while yields the best classification accuracy over CMER.
w/o (L, S, W) denotes the model that does not use the linguistic features, the sentence-level and word-level deep representations in PromptARA.
w/o (L, S, W, P) denotes the model that does not use the linguistic features, the sentence-level and word-level deep representations as well as prompts PromptARA.
Model PromptARA
CLEAR
Cambridge 91.11
CMER
CMT
82.03
26.50
43.56
wlo1
81
91.11
26.71
41.39
The performance behaviour of the proposed model might be attributed to the following two main factors. Firstly, the Chinese variant of the Big- Bird model was not well trained of data, limiting its ability adapt to the complex and diverse nature of Chinese texts. Secondly, the complexity of Chinese texts is influenced by in- tricate semantic and structural differences, further
wio Cl, W)
80.97
58.99
26.35
39001
78.56
35.16
Table 6: Experimental results ablation studies over four corpora in terms of accuracy, where L. S, W and P represent respectively the linguistic features, sentence- level representations, word-level representations and prompts used in the proposed model. The best results are marked in bold.
large amount

---

The experimental results are presented in Table 6. From Table 6, the accuracy gradually decays over most corpora through removing the linguis- tic features, sentence- and word-level features, and prompts from the proposed PromptARA model in sequence. In particular, the introduced prompt information generally yields the biggest improve- ments accuracy shown in the fourth and fifth rows in Table 6. As shown in the third and fourth rows Table 6, the introduced word- and sentence- level representations are useful for ARA. In addi- tion, we can observe from the second and third rows of Table that linguistic features are also useful to refine the deep features in most cases. These results clearly verify the effectiveness and feasibility of the proposed ideas.
improve deep representations and fusing the multi- level deep and linguistic representations with an or- thogonal projection scheme. The proposed model uses informative prompts to provide additional in- formation to enhance the contextual semantic un- derstanding of the model, and combines textual in- formation about the different structures of the text to yield a richer textual representation. Experimen- tal results show that the proposed model achieves excellent performance on both Chinese and English datasets compared with existing approaches. On the English datasets in particular, it shows better generalization performance and robustness. In fu- ture work, we will further explore how to achieve automatic generation of textual prompts. Capturing richer and more accurate information representa- tion at different text levels will also be investigated.
Moreover, we consider the effect of the lan- guage models in the proposed model. Specifi- cally, we compare the performance of the proposed model with its counterparts by replacing the Big- Bird model with other pre-trained models such as CINO (Yang et al., 2022), BERT12, Muiti-BERT 13 Longformer 14 (Iz et al., 2020), over these two Chinese corpora. The comparison results are pre- sented in Table 7. As shown in Table 7, the pro- posed model integrated with the BigBird model achieves the best accuracy, attributed its superi- ority on handling long length texts and outperforms the Longformer model, which also has the ability to handle long texts. For the most other pre-trained language models, they generally have to truncate the texts during the input procedure, resulting in a greater loss of valuable textual information and thus the degradation of the performance.
Limitations
The model poses major challenge due the need to obtain vectors with different structural lev- els, which more demanding on the equipment. Therefore, it is crucial to explore a model that runs on low-demanding devices without compromising the performance of the model. This becomes partic- ularly important when dealing with large amounts of distributed text and large datasets, which must have higher equipment requirements. In addition, we are currently having difficulty deciding which prompt texts are appropriate. Another challenge we face is how to adapt the model to feature se- lection based on the characteristics of the dataset itself. Different datasets exhibit unique features, where too many features introduce too much noise and too few features do not have much impact on the performance of the model.
Dataset
BERT
CINO
Mulli-BEKT
Longformer
BigBird 43,96
CMT
41.23
38.10
38.64
42.86
CMET
24.14
25.13
25.44
23.62
26.50
Table 7: On effect of pre-trained language models used the proposed model. The best results are marked in bold.
Acknowledgements
The work of J. Zeng was supported in part by the National Natural Science Foundation of China [Grant Nos. 62376110, 61977038), and Thou- sand Talents Plan of Jiangxi Province [Grant No. jxsq2019201 124], and the Jiangxi Provincial Nat- ural Science Foundation for Distinguished Young Scholars (20224ACB212004) The work of W. Xiao supported in part by Jiangxi Provincial Educational Science Foundation during the"13tth Five- Plan" (20YB080) and the Humanities and Social Science Foundation of Higher Educa- tion Institutions of Jiangxi Province(YY2299)
6 Conclusion
How to fully explore the deep representations and exploit the linguistic representations of texts is important for ARA. This paper proposed novel hybrid ARA model through employing prompts to
"htps://huggingfaee https//huugginggaeee.ooccccciioc caafl/ccioobbsee-e
"hups://uuggingffee../// co/bert-base-chiee
19 "https:/huugginglaee co/bert- www.t-bbee-mmlllnuulllloooo
4096

---

References
Julia Hancke, Sowmya Vajjala, and Detmar Meurers. 2012. Readability classification for german using lexical, syntactic, and morphological features. In Proceedings of COLING 2012: Technical Papers, page 1063-1080.
Barbara Arfé, Lucia Mason, and Inmaculada Fajardo. 2018. Simplitying informational text structure for stugglingreaders readers. Reading and Writing, 31(9)111 2210.
Beltagy Iz, Peters Matthew E., and Cohan Arman. 2020. Longformer: The long- long-document transformer. abs/2004.55150.
Ion Madrazo Azpiazu and Maria Soledad Pera. 2019, Multiattenttive recurrent neural network architecture for multilingual readability assessment. Transactions of the Association for Computational Linguistics, 7:421-436.
Zhiwei Jiang. Qing Gu, Yafeng Yin, Jianxiang Wang. and Daoxu Chen. 2019. Graw+; A two-view graph propagation method with word coupling for read- ability assessment. Journal of the Association for Information Science and Technology, 7065)433-447.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with
subword information. Transactions of the Associa- fion for Computational Linguisties, 5:135-146.
George R. Klare. 1963. Measurement of readability. Iowa State University Press, lowa, USA.
Yong Cheng. DK Xu, and Jun Dong. 2020. On key fac- tors of text reading difficulty grading and readability formula based on chinese textbook corpus. Applied Linguistics, 1:132-11 143.
George R. Klare. 2000. The measurement of readability: useful information for communicators. ACM Journal of Computer Documentation, 24(3):107-121.
Scott Crossley, Aron Heintz, Joon Suh Choi, Jordan Batchelor, Mehrnoush Karimi, and Agnes Malatin- szky. 2022. A large-scaled corpus for assessing text readability. Behavior Research Methods, pages 1-17.
Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021.
Pushing on text readability assessment: A trans- former meets handcrafted linguistic features. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10669- 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Alice Davison and Robert N. Kantor. 1982. On the fail- ure readability formulas to define readable texts: A case study from adaptations. Reading Research Quarterly, 1722)1177-209.
Bruce W. Lee and Jason Lee. 2023. Prompt- brompt-based ing for text readability assessment. In Findings of the A Association for Computational Linguistics: EACL 2023, pages 1819-1824, Dubrovník, Croatia. Associ- ation for Computational Linguistics.
based learn-
Felice Dell'Orletta. Simonetta Montemagni, and Giu- lia Venturi. 2011 1. Read-it: Assessing readability of italian texts with view to text simplification. Pro- ceedings of the Second Workshop on Speech and Lan- guage Processing for Assistive Technologies, pages 73-83.
Justin Lee and Sowmya Vajjala. 2022. A neural pair- wise ranking model for readability assessment. In Findings of the Association for Computational Lin- guistics; ACL 2022, pages 3802-3813, Dublin, Ire- land. Association for Computational Linguistics.
Joel Denning, Maria Soledad Pera, and Yiu-Kai Ng. 2016. A readability level prediction tool for k-12 books. association for information science and tech- nology, 67(3):550--56
Wenbiao Li, Wang Ziyang, and Yunfang Wu. 2022. A unified neural network model for readability assess- ment with feature projection and length-bul balance loss. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7457.
Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Pro- ceedings of the Fifieenth Workshop on Innovative Use of NI.P for Building Educational Applications, pages 1-17, Seattle, USA. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT 2019: Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171 171-4186.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Bertha Lively and Sidney Pressey. 1923. A
Thomas Francois and Cedrick Fairon. 2012. An "ai readability" formula for french as a foreign language. In Proceedings of the 2012 Joint Conference on Em- pirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 466-477.
method for measuring the vocabulary burden of text- books. Educational administration and supervision. 9(7):399-398
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam.

---

Matej Martine, Senja Pollak, and Marko Robnik- Sikonja. 2021. Supervised and unsupervised neu- ral approaches to text readability. Computational Linguistics, 47(1):141-179
Kumiko Tanaka-Ishi,, Satoshi Tezuka, and Hiroshi Ter- ada. 2010. Sorting texts by readability. Computa- tional linguistics, 36(2):203-27..
Hou-Chine Chiang Tseng, Hsuch- Chen, Kuo-En Chang. Yao-Ting Sung, and Berlin Chen. 2019. An innova- tive bert-based readability model. In International Conference on Innovative Technologies and Learning (ICITL) 2019 Innovative Technologies and Learning. pages 301-308.
G. Harry MeLaughlin. 1969. Smog grading a new readability formula. The Journal of Reading.
Tomas Mikolov. 1lya Sutskever, Kai Chen, Gregory S Corrado, and Jeffrey Dean. 2013. Distributed repre- sentations of words and phrases and their composi- tionality. In NIPS
Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment re- search. In Proceedings of the Thirteenth Language Rexources and Evaluation Conference, pages 5366- 5377, Marseille, France. European Language Re- sources Association.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532- 1543.
Sowmya Vajjala and Detmar Meurers. 2012. On im- proving the accuracy of readability classification us- ing insights from second language acquisition. In Proceedings of the seventh workshop on building ed- ucational applications using NLP, pages 163-173.
Xinying Qiu, Yuan Chen, Hanwu Chen, Jian- Yun Nie,
Yuming Shen, and Dawei Lu. 2021. Learning syn- tactic dense embedding with correlation graph for automatic readability assessment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ihh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3013-3025 Online. Association for Computational Linguistics.
Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second lan- guage learners. In Proceedings of the 11th Workshop Innovative Use of NLP for Building Educational Applications, pages 12-22, San Diego, CA. Associa- tion for Computational Linguistics.
Xinying Qiu, Kebin Deng. Likun Qiu, and Xin Wang. 2018. Exploring the impact of linguistic features for chinese readability assessment. In Natural Language Processing and Chinese Computing: 6th CCF Inter- national Conference, NLPCC 2017, Dalian, China, November 8- 2017, Proceedings pages 771- 783. Springer.
Wei Xu, Chris Callison-Burch and Courtney Napoles. 2015. Problems in current text simplification re- search: New data can help. Transactions the Asso- ciation for Computational Linguistics, 3:283-297.
Zichao Yang, Diyi Yang. Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Associa- tion for Computational Linguistics.
Daniele Schicchi, Giovanni Pilato, and Giosué Lo Bosco. 2020. Deep neural attention-based model for the evaluation of italian sentences complexity. In 2020 IEEE 14th International Conference on Seman- tic Computing (ICSC), pages 253-256 IEEE
Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang. Min Lin, Dayong Wu, and Zhigang Chen. 2022.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
CINO: A Chinese minority pre-trained language model. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 3937- 3949, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
natural language inference.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13297733-77297.
Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Machine Translation: Volume I, Research Papers, pages 83- 91. Berlin, Germany, Association for Computational Linguisties.
Jinshan Zeng Yudong Xie, Xianglong Yu, John Lee. and Ding-Xuan Zhou. 2022. Enhancing automatic readability assessment with pre-training and soft la- bels for ordinal regression. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 4557-4568.
Yao-Ting Sung, Wei-Chun Lin, Scott Benjamin Dyson, Kuo-Tin Chang, and Yu-Chia Chen. 2015. Leveling 12 texts through readability: Combining multilevel linguistic features with the cefr. The Modern Lan- guage Journal, 99(2):311-391.

---

Category
Feature name
Definition
I: Characters
Total number of characters
2: Words
Total number of words
3. Adverhs
Total number of adverbs
4: Verbs
Total number of verbs
5: Nouns
Total numbee of nouns
6: Adjectives
Total number of adjectives
The ratio of the number of adverbs to
Ratio adverbs
the total number of words
The ratio of the number of verbs to the
8: Ratio verbs
total numher of words
The ratio of the number of nouns to the
Ratio nouns
total numher of words
The ratio of the number of adjectives
Ratio of adjectives
the total number of words
Total number of characters with 1-7
11: Low stroke-count characters
strokes
Lexical level
12: Intermediate stroke-count
Total number of characters with 8-15
characters (8~15 strakes)
strokes
13: High stroke-count churacters
Total number of characters with more than 15 strokes
(>15 strokes)
14: Ratio of Low stroke-count
Proportion of ow stroke-count characters
characters
13: Ratio of Intermediate stroke-count characters (8~155 strokes)
Proportion of Intermediate stroke-count characters
16: Ratio of High stroke-count
Proportion of High stroke-count characters
characters (>-15 strokes)
Total number of strokes of each character
17: Average strokes
divided the number of characters
Two-character words
Total number of Two-character words
Three-character words Four-character words
Total number Three-character words
Total number Four-character words
ratio of the number of Two-character
21: Ratio of Two-character words
words to total number words
ratio of the number of Three-character
22: Ratio of Threo-character words
words to the total number words
ratio the numher of Four-character words to the total number words
23: Ratio of Four-character words
Total number of words not in 8,000 Chinese Words
24: Level words
Ivvel words divided by the total number words
25: Level words ratio
Total number of words in level 1.2....?
26-32: Level 1.2....? words
respectively
Level 1.2....7 divided by the total number
33-39: Level 1,2,...7 words ratio
of words respectively
The ratio of level 1,2,3 words together in all words
Level 1,2,3 words ratio
The ratio of level 4,5 words together in all words
Level words ratio
The ratio of level 6,7 words together in all words
42: Level 6,7 words ratio
Calculate the level of all words, calculate the average level
Average level of words
Calculate the level of all words, calculate the mean square level
Mean square level of words
Table Part Chinese Linguistic Features

---

45:
Content words
Total numher of content words
Semantic Level
46:
Frequency content words
Frequency of content words
47.
Sentences
Number sentences
Total number of words divided by the total
48: Average sentence length
numher of sentences
The length of the longest sentence among
49: Maximum sentence length
all sentenees
The total number sentences with length
50: First-level sentences
exceeding 15
The total number sentences with length
51: Second-level sentences
between 16 and 30
The total number sentences between and length
52: Tertiary sentences
The total number sentences between 46
53: Fourth-level sentences
and 60 in length
The total number of sentences with length
54: Fifth-level sentences
greater than 60
Ratio of the total number of sentences with
55: Ratio of First-level sentences
a length not exceeding 15
Ratio of the total number of sentences with
Syntactic level
56: Ratio of Second level-sentences
a length between 16 and 30
Ratio of the total number of sentences between
Ratio of Tertiary sentences
31 and 45 in length
Ratio of The total number of sentences between
Ratio of Fourth-level sentences
46 and 60 in length
Ratio of the total number of sentences with a
Ratio of Fifth-level sentences
length greater than 60
60: Conjunctions
Total number of conjunctions
61: Preposition
Total numher of preposition
62: Pronouns
Total number of pronouns
63: Auxiliary verbs
Total number of auxiliary verbs
The ratio the number of conjunction
64: Ratio of conjunctions
the total number of words
The ratio the number of preposition
65: Ratio of prepositions
Cohesion Level
the total number of words
The ratio the numher pronouns
66: Ratio of pronouns
the total number words
The ratio the number auxiliary
67: Ratio of auxiliary verbs
total number words
Table 9: Part Il of Chinese Linguistic Features


Reviews:
[{"review_id": "AcJg9dCp02", "reviewer": "Reviewer_b6jz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(Q1): Compare proposed method with readability formula-based methods.\n\nA1: Great suggestion. According to Chakraborty et al. (2021), we considered three representative readability formula-based methods, i.e., Automated Readability Index (ARI), Flesch Reading Ease (FE) and Flesch–Kincaid (FK), and compared them with the proposed method over four English corpora during the rebuttal phase. In terms of accuracy, the comparison results are presented as follows: (1) Cambridge: 16.67% (ARI) vs. 34.44% (FE) vs. 20.00% (FK) vs. 91.91% (our); (2) weebit: 19.58% (ARI) vs. 19.67% (FE) vs. 20.00% (FK) vs. 93.12% (our); (3) CLEAR: 9.35% (ARI) vs. 15.43% (FE) vs. 14.16% (FK) vs. 82.03% (our); (4) Newsela: 8.87% (ARI) vs. 7.65% (FE) vs. 8.54% (FK) vs. 88.40% (our). It can be observed that these readability formula-based methods perform worse than both the proposed method and other deep learning-based approaches, as shown in Table 4. Similar claims can be also concluded in terms of other evaluation metrics. We will incorporate these comparison results if the paper is accepted.\n\n(Q2): Compare BERT-with prompts and the proposed method.\n\nA2: We have compared BERT-with prompts (abbreviated as BERTwPrompt) with the proposed method over corpora concerned in the rebuttal phase. For better comparison, we also took the performance of BERT into consideration. In terms of accuracy, the comparison results are presented as follows: (1) Cambridge: 75.56% (BERT) vs. 78.12% (BERTwPrompt) Vs. 91.11% (our); (2) weebit: 91.53% (BERT) vs. 92.38% (BERTwPrompt) Vs. 93.12% (our); (3) CLEAR: 76.74% (BERT) vs. 79.49% (BERTwPrompt) Vs. 82.03% (our); (4) Newsela: 77.12% (BERT) vs. 84.23% (BERTwPrompt) Vs. 88.40% (our); (5) CMT: 38.46% (BERT) vs. 41.23% (BERTwPrompt) Vs. 43.96% (our); (6) CMER: 22.30% (BERT) vs. 24.14% (BERTwPrompt) Vs. 26.50% (our). It can be observed that the performance of the proposed method using the larger pre-trained BigBird model is better than that of BERT with prompts, which is also much better than that of BERT. These results highlight the efficacy of using a larger pre-trained model and the introduced prompts for improved readability assessment. Similar claims can be also concluded in terms of other evaluation metrics. If accepted, we will include these comparison results in the paper.   \n\n(Q3): For the Chinese readability, the authors can utilize existing pre-trained Chinese models to improve the performance.\n\nA3: For the Chinese readability, the BigBird used in experiments is the pre-trained Chinese version of BigBird from https://huggingface.co/Lowin/chinese-bigbird-wwm-base-4096.\nAs suggested, we have tried some other existing pre-trained Chinese models such as CINO and multilingual BERT (called multi-BERT for short) during the rebuttal phase. In terms of accuracy, the comparison results are presented as follows: (1) CMT: 39.19% (CINO) vs. 38.64% (Multi-BERT) vs. 43.96% (BigBird); (2) CMER: 24.37% (CINO) vs. 25.44% (multi-BERT) vs. 26.50% (BigBird). It can be observed that replacing BigBird with CINO and multilingual BERT cannot further improve the performance. We will try more pre-trained Chinese models to improve the performance of the proposed approach for the Chinese readability assessment in the future. \n\n(Q4): Some missing references.\n\nA4: Thanks. We will cite these references in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-The proposed method uses prompt-based readability assessment which overcomes the need of a large dataset for finetuning models\n- Authors evaluate the proposed approach in English and Chinese language\n\n", "reasons_to_reject": "First, I would recommend authors to compare their proposed approach with readability formula-based methods, similar to Chakraborty et al. in [6].\nSecond, to demonstrate the utility of the prompt-based approach, the authors can compare BERT-with prompts [7] and their proposed method: BigBERT-with prompts. Since BigBERT is a larger pre-trained model, the performance improvement over BERt is expected.\n\n\nThird, for the Chinese Readability, the experiment setting is not correct. The authors can utilize the existing pre-trained Chinese models [1, 2], checkpoints for Chinese [3], and multilingual BERT [4] models.\nSpecifically, the results on the English dataset in Table 4 are better than the Chinese dataset in Table 5 by a large margin. \n\nA good example of a multilingual experiment setting is Hollenstein et al. [5]. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Readability assessment is the key step for many NLP tasks, including text simplification and educational applications.\nTraditional feature-based readability formulas don't capture deep semantic meaning. Deep learning-based approaches capture semantic meaning but require a large dataset to train and finetune.\n\nThis paper proposes automatic readability assessment with prompt-based techniques in a large language model, BigBERT to address the two abovementioned challenges.\n\nToward that end, the authors combined prompt-based and traditional linguistic features in their experiment design. The authors conducted experiments with both English and Chinese language datasets.\nTheir empirical results and ablation studies demonstrate superior performance compared to baselines. \n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "References\n1. Yang, Z., Xu, Z., Cui, Y., Wang, B., Lin, M., Wu, D. and Chen, Z., 2022, October. CINO: A Chinese Minority Pre-trained Language Model. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 3937-3949).\n2. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S. and Hu, G., 2020, November. Revisiting Pre-Trained Models for Chinese Natural Language Processing. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 657-668).\n3. Cino Huggingface Checkpoints https://huggingface.co/hfl/cino-large\n4. https://huggingface.co/bert-base-multilingual-cased\n4. PHollenstein, N., Pirovano, F., Zhang, C., Jäger, L.A. and Beinborn, L., 2021, June. Multilingual Language Models Predict Human Reading Behavior. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 106-123). Association for Computational Linguistics.\n5. Chakraborty, S., Nayeem, M.T. and Ahmad, W.U., 2021, May. Simple or complex? learning to predict the readability of Bengali texts. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 14, pp. 12621-12629).\n6. Jiang, T., Jiao, J., Huang, S., Zhang, Z., Wang, D., Zhuang, F., Wei, F., Huang, H., Deng, D. and Zhang, Q., 2022, December. PromptBERT: Improving BERT Sentence Embeddings with Prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 8826-8837)."}}, {"review_id": "xYqGvFbEXk", "reviewer": "Reviewer_JPZk", "summary": "", "questions": "1- For Table 4 (Page 7), it’s better to have the finetuned results for the BERT-FP-LBL for ALL datasets. Is there any particular reason that you choose not to do that?\n\n2- What checkpoints/pretrained weights did you use for all approaches, especially for the Chinese datasets (e.g., the BERT method in Table 5)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(Q1): Limited technical novelties.\n\nA1: The novelties of this paper can be summarized as follows: (1) This paper is the first work to incorporate the idea of prompt learning into the hybrid automatic readability assessment.  Leveraging this, the proposed model outperforms the state-of-the-art models over four English and two Chinese corpora. (2) The architecture of the proposed model significantly differs from those of existing models such as BERT-FP-LBL. Specifically, in the proposed model, we simultaneously use the document-, sentence-, and word-level representations to form the deep representation, while only the document-level deep representation is used in BERT-FP-LBL. The architecture of the linguistic representation module in the proposed model is also very different to BERT-FP-LBL. Notably, our model employs layer normalization to reduce the impact of different feature variations. \n\nMoreover, we have conducted experiments on the Cambridge and weebit corpora to compare the performance of BERT-FP-LBL and the proposed model without prompts (called w/o prompt for short) in the rebuttal phase. In terms of accuracy, the specific comparison results are presented as follows: (1) Cambridge: 87.78% (BERT-FP-LBL) vs. 89.14% (w/o prompt); (2) weebit: 92.70% (BERT-FP-LBL) vs. 92.83% (w/o prompt). It can be observed that the proposed model without prompts also outperforms BERT-FP-LBL. Thus, the proposed model should not be regarded as an extension of BERT-FP-LBL though both models use the hybrid representations and orthogonal projection layers for fusion. \n\nBased on the above discussions, the novelties of this paper should be deemed sufficient.\n\n(Q2): More comparisons between the proposed approach and BERT-FP-LBL are needed, including detailed ablation studies.\n\nA2: Great suggestions. In Table 4, we only present the results of BERT-FP-LBL (reported in the literature) over two English corpora since we cannot access their reproducible codes. As suggested, we have conducted some detailed ablation studies comparing the proposed approach with BERT-FP-LBL in the rebuttal phase. As mentioned in the response to (Q1), we compared the performance of the proposed method without prompts and BERT-FP-LBL. Experimental results show that the proposed method without prompts still outperforms BERT-FP-LBL over Cambridge and weebit. We would like to do more comparisons if we can access their reproducible codes in the future.\n\n(Q3): It would be more helpful if we could have the finetuned results of BERT-FP-LBL.\n\nA3: Totally agree with this. We would like to yield the finetuned results of BERT-FP-LBL if we can access their reproducible codes in the future.\n\n(Q4): For Table 4 (Page 7), it is better to have the fine-tuned results for the BERT-FP-LBL for all datasets. Is there any particular reason that you choose not to do that?\n\nA4: The results of BERT-FP-LBL presented in Table 4 were directly taken from the results reported in the literature (Li et al., 2022) since we cannot access their reproducible codes. \n\nWe would like to yield the fine-tuned results for all datasets if we can access their reproducible codes in the future.\n\n(Q5): What checkpoints/pretrained weights did you use for all approaches, especially for the Chinese datasets (e.g., the BERT method in Table 5)?\n\nA5: For all approaches, we used the pre-trained (Chinese) models from the tool box huggingface.\n\n(Q6): Missing references.\n\nA6: Thanks. We will cite these references in the revised version.\n\n(Q7): It is better to add the “recall” as an evaluation metric in Table 4.\n\nA7: Thanks. We will include the results in terms of recall in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- The paper is well written.\n\n-- The experiments are good.", "reasons_to_reject": "-- Limited technical novelties.\n\n-- More comparisons are needed for the proposed approach and the BERT-FP-LBL, including detailed ablation studies.\n\n-- Instead of directly citing the results from [1] for BERT-FP-LBL, it would be more helpful if we could have the finetuned results. ", "questions_to_authors": "1- For Table 4 (Page 7), it’s better to have the finetuned results for the BERT-FP-LBL for ALL datasets. Is there any particular reason that you choose not to do that?\n\n2- What checkpoints/pretrained weights did you use for all approaches, especially for the Chinese datasets (e.g., the BERT method in Table 5)?", "paper_topic_and_main_contributions": "Summary:\nThis paper extends previous ARA approach (BERT-FP-LBL [1]) by incorporating simple prompting tricks. Multiple experiments are conducted over 4 English and 2 Chinese datasets by comparing to various baselines. \n\n[1] A unified neural network model for readability assessment with feature projection and length-balanced loss, by Li et al., in EMNLP, 2022.\nhttps://aclanthology.org/2022.emnlp-main.504/", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1- In Page 7, Table 4, it’s better to add the recall, similar as the previous work [1].\n"}, "extra_text": {"Missing_References": "Missing references:\n- Comprehensive readability assessment of scientific learning resources, by Arshad et al., in IEEE, 2023\nhttps://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10132466\n\n- Automated assessment of subjective assignments: a hybrid approach, by Birla et al., in 2022\n https://www.sciencedirect.com/science/article/abs/pii/S0957417422006777"}}, {"review_id": "kiMxx5uYc3", "reviewer": "Reviewer_p25R", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(Q1): The ablation study is insufficient to analyze and justify the effectiveness of the prompts or the orthogonal projection layer.\n\nA1: Thanks a lot. As suggested, we have implemented a series of ablation studies over all six corpora during the rebuttal phase. In particular, we considered the following models: 1) the proposed model without prompts (denoted as w/o prompt for short), 2) the proposed model without orthogonal projection layers (denoted as w/o orth-proj for short), i.e., replacing the orthogonal projection layers with the simple concatenation, and 3) the proposed model without both of prompts and orthogonal projection layers (denoted as w/o both). In terms of accuracy, the comparison results together with the proposed model PromptARA are presented as follows: (1) Cambridge: 91.11% (PromptARA) vs. 89.14% (w/o prompt) vs. 90.45% (w/o orth-proj) vs. 88.12% (w/o both); (2) weebit: 93.12% (PromptARA) vs. 92.83% (w/o prompt) vs. 92.88% (w/o orth-proj) vs. 92.72% (w/o both); (3) CLEAR: 82.03% (PromptARA) vs. 79.28% (w/o prompt) vs. 81.62% (w/o orth-proj) vs. 79.05% (w/o both); (4) Newsela: 88.40% (PromptARA) vs. 87.53% (w/o prompt) vs. 88.12% (w/o orth-proj) vs. 87.36% (w/o both); (5) CMT: 43.96% (PromptARA) vs. 40.84% (w/o prompt) vs. 42.58% (w/o orth-proj) vs. 39.72% (w/o both); (6) CMER: 26.50% (PromptARA) vs. 24.84% (w/o prompt) vs. 25.88% (w/o orth-proj) vs. 24.06% (w/o both). \n\nIt can be observed from these ablation results that there are substantial improvements on performance contributed by prompts and orthogonal layers, and the improvements contributed by prompts are generally more significant than those yielded by integrating orthogonal projection layers. These results clearly show the effectiveness of the used prompts and orthogonal projection layers.  \n \n(Q2): The experimentation uses 6 datasets, while ablation studies use 4 datasets.\n\nA2: We have implemented the ablation studies over all six datasets during the rebuttal phase, as pointed out in the response to (Q1). \n\n(Q3): It is unclear from the ablation study how much improvement the prompts or the orthogonal projection layers contribute to the overall performance of the model.\n\nA3: The results of ablation studies from the response to (Q1) indicate that, in terms of accuracy, prompts yield improvements of 1.32% (Cambridge), 0.16% (weebit), 2.57% (CLEAR), 0.76% (Newsela), 2.86%(CMT), 1.82% (CMER), respectively, while the orthogonal projection layers provide improvements of 1.02% (Cambridge), 0.09% (weebit), 0.23% (CLEAR), 0.17% (Newsela), 1.12%(CMT), 0.78% (CMER), respectively. It can be observed that the improvements contributed by the prompts are more significant than those contributed by the orthogonal projection layers.\n\n(Q4): There is insufficient description about what kind of prompts they use, how they use them, and why these designs or choices. \n\nA4: Thanks for pointing out these questions. \n\n(1) What kind of prompts used: In this paper, we employ task prompts that explicitly specify the tasks the model should perform. These prompts usually take the form of questions, commands, or task statements of the target task, ensuring that the model understands the nature of the task. Given that the key goal of prompts is to explicitly inform the model what it is trying to accomplish, it’s essential for task prompts to be clear and concise, ensuring the model understands the task’s essence. Specifically, the task in this paper pertains to readability assessment. According to this design principle, we can design task prompts like \"The text readability classification task is currently in progress.\" as presented in Table 1.\n\n(2) How to use prompts: Prompts are placed at the beginning of the text to ensure that the model explicitly understands the readability task information before reading the text. This helps the model to have a clear starting point when performing tasks. \n\n(3) Why these designs or choices: We determine these designs mainly based on the following principles. The first pertains to task-clarity—using explicit prompts ensures the model’s clear grasp of task requirements, thereby enhancing performance. The second centers on personalization—prompts should be adaptive to tasks to achieve personalized readability optimization. The third emphasizes goal-orientation—prompts should be goal-oriented, such as reducing sentence complexity or improving paragraph structure. In practice, designing and selecting prompts often requires experimentation and feedback to determine which prompts work best for a particular dataset.\n\nWe will incorporate these descriptions to the paper in the future.\n\n\n(Q5): Please double-check your \"Reference\" list. There are many repeated references.\n\nA5: Thanks. We will double-check the reference list to avoid repeated references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is a nice idea to add prompts in the learning process of ARA models.  The paper presents a neural network framework to apply prompts and orthogonal projection layers.  The framework combines learning from the representations of document-level, word-level, sentence-level, and linguistic features of input text.  Experimentation has been conducted on English and Chinese datasets to show the overall improvement of performance.", "reasons_to_reject": "1. The paper states in lines 11 through 15 that their novelty is in the use of prompts and orthogonal projection layers. However, the ablation study is insufficient to analyze and justify the effectiveness of the prompts or the orthogonal projection layer, because:\n\n1.1 The experimentation uses 6 datasets while the ablation study uses 4 datasets.\n\n1.2. It is unclear from the ablation study how much improvement the prompts or the orthogonal projection layers contribute to the overall performance of the model. \n\nFirst, there is no experimentation on the effectiveness of a model with orthogonal projection layers and no-prompt BigBird document representation learning. There's not enough evidential support that adding prompts will effectively improve various \"no-prompt\" model performance. \n\nSecond, there is no experimentation on implementing the orthogonal projection layers with no-prompt BigBird document representation learning. So we can not draw conclusions on the effectiveness of adding prompts in the implementation of the orthogonal projection layers.   \n\nThird, there is no experimentation on how each individual orthogonal projection layer help improve a prompt-enhanced BigBird model.  \n\nSince the paper states that \"prompts\" and \"orthogonal projection\" are their novelty, there needs systematic evidential support to prove the effectiveness of \"prompts\" and/or \"orthogonal projection\".\n\n2. There is insufficient description about what kind of prompts they use, how they use them, and why these designs or choices.  The caption of Table 1 provides \"Some\" examples. There is no discussion how they add the \"default\" and non-default prompt to the documents.  Since \"prompts\" is the paper's major contribution, the paper needs to provide detailed discussion on their prompt engineering process.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to use prompts and orthogonal projection layer to improve the performance of a hybrid automatic readability assessment model that combines the neural embedding features at various granularity levels and the linguistic features.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please double-check your \"Reference\" list. There are many repeated references."}, "extra_text": {}}, {"review_id": "0ECoUOWb1j", "reviewer": "Reviewer_otsg", "summary": "", "questions": "Why can the redundant information of ARA be eliminated through the fusion of orthogonal layers?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(Q1): Why can the redundant information of ARA be eliminated through the fusion of orthogonal layers?\n\nA1: The redundancy between two features (generally represented by two vectors) is roughly embodied by the correlation between them. In linear algebra, the orthogonal projection is a simple way to get rid of the correlation of two vectors. Inspired by this, this paper proposes the utilization of orthogonal projection layers to eliminate the redundant information between two different-level representations. \n\n(Q2): The function of fusion by orthogonal layers is not very clear, which means the authors need to provide examples or experimental results to demonstrate that such processing indeed removes the redundant information.\n\nA2: As presented in the response to (Q1) raised by Reviewer 3, we have conducted a series of ablation studies to demonstrate the effectiveness of prompts and orthogonal projection layers during the rebuttal phase. According to these ablation results, the improvements achieved by the orthogonal projection layers are 1.02% (Cambridge), 0.09% (weebit), 0.23% (CLEAR), 0.17% (Newsela), 1.12%(CMT), 0.78% (CMER), respectively. These results clearly demonstrate that the orthogonal projection layers can indeed remove the redundant information.\n\n(Q3): There are some grammatical issues with this paper, usually missing determiners and the like. \n\nA3: Great thanks. We will carefully proofread the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is demonstrated how deep feature representations and explicit syntactic information contribute to the assessment of the readability level of a document;\n\nIt would be beneficial for those considering how to make use of deep features and other types of features in combination with prompts.", "reasons_to_reject": "I don't think there much risk associated with this paper.", "questions_to_authors": "Why can the redundant information of ARA be eliminated through the fusion of orthogonal layers?", "paper_topic_and_main_contributions": "To explore the deep representations and exploit the linguistic representations of texts for ARA, the paper proposed a hybrid ARA model by employing the prompts to improve deep feature representations and an orthogonal projection layer to fuse both deep and linguistic features. Extensive computational experiments on  four English and two Chinese datasets showed the effectiveness of the model.\n\nStrengths:\n1.It is demonstrated how deep feature representations and explicit syntactic information contribute to the assessment of the readability level of a document;\n2.Sufficient technical details, showing the transformation process;\n3.Several data-sets are included with both Chinese and English;\n4.Extensive computational experiments results showed the effectiveness of the framework.\n\nWeaknesses:\n1.The function of fusion by orthogonal layers is not very clear, which means the authors need to provide examples or experimental results to demonstrate that such processing indeed removes the redundant information.\n\n2.There are some grammatical issues with this paper, usually missing determiners and the like.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I didn’t find any missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "There are some grammatical issues with this paper, usually missing determiners and the like. I would suggest having it proofread."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method for readability assessment. The key concept involves using proper prompting to induce features from a large model. The features are then combined with other features such as linguistic feature. They are combined via orthogonal projection layers. The authors observed improvements over state-of-the-art methods on four corpora. 

All reviewers agree that the paper is clearly written with results seem to be easily reproducible. The evaluation on the four corpora is solid. 
There is certain ambivalence of excitement from the reviewers. 

Given that many NLP tasks have to deal with data scarcity challenge as the readability assessment problem, the presented approach of using prompting to generate feature from LLMs is interesting and can lead to further explorations.
Decision: Accept-Findings