Venue: NAACL 2024

# FPT: Feature Prompt Tuning for Few-shot Readability Assessment

Ziyang Wang\({}^{1,2}\), Sanwoo Lee\({}^{1,3}\), Hsiu-Yuan Huang\({}^{1,3}\) Yunfang Wu\({}^{1,3}\)

\({}^{1}\)National Key Laboratory for Multimedia Information Processing, Peking University

\({}^{2}\)School of Software and Microelectronics, Peking University, Beijing, China

\({}^{3}\)School of Computer Science, Peking University, Beijing, China {wzy232303, huang.hsiuyuan}@stu.pku.edu.cn, {sanwoo, wuyf}@pku.edu.cn,

Corresponding author.

###### Abstract

Prompt-based methods have achieved promising results in most few-shot text classification tasks. However, for readability assessment tasks, traditional prompt methods lack crucial linguistic knowledge, which has already been proven to be essential. Moreover, previous studies on utilizing linguistic features have shown non-robust performance in few-shot settings and may even impair model performance. To address these issues, we propose a novel prompt-based tuning framework that incorporates rich linguistic knowledge, called **F**eature **P**rompt **T**uning (FPT). Specifically, we extract linguistic features from the text and embed them into trainable soft prompts. Further, we devise a new loss function to calibrate the similarity ranking order between categories. Experimental results demonstrate that our proposed method FTP not only exhibits a significant performance improvement over the prior best prompt-based tuning approaches, but also surpasses the previous leading methods that incorporate linguistic features. Also, our proposed model significantly outperforms the large language model gpt-3.5-turbo-16k in most cases. Our proposed method establishes a new architecture for prompt tuning that sheds light on how linguistic features can be easily adapted to linguistic-related tasks.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 280–295

June 16-21, 2024 62024 Association for Computational Linguistics

## 1 Introduction

Readability assessment (RA) is the task of evaluating the reading difficulty of a given piece of text Vajjala (2022). It has wide applications, such as choosing appropriate reading materials for language teaching Collins-Thompson and Callan (2004), supporting readers with learning disabilities Rello et al. (2012), and ranking search results by their reading levels Kim et al. (2012).

Early works in RA mainly focused on designing handcrafted linguistic features such as word length (in characters/syllables), sentence length, and usage of different difficulty-level words. In recent years, RA has been dominated by neural network-based architectures Meng et al. (2021); Azpiazu and Pera (2019). The key challenge of these methods is to learn a better text representation that can capture deep semantic features. Current research has also explored different ways of combining linguistic features with pretrained language models (PLMs), achieving state-of-the-art results on numerous RA datasets Li et al. (2022); Lee et al. (2021). However, these studies have mainly focused on fine-tuning with a large amount of labelled data, while only a few studies have explored few-shot settings.

Prompt-based tuning, shown to be a powerful method for the classification task in the few-shot setting, makes full use of the information in PLMs by reformulating classification tasks as cloze questions. Different prompt-based tuning strategies are illustrated in Figure 1. The hard prompt tuning

Figure 1: Comparison of previous prompt tuning frameworks and our proposed Feature Prompt Tuning (FPT). \(T(\cdot)\) and \(verbalizer(\cdot)\) denote the template and verbalizer function, respectively. FPT utilizes both hard and soft tokens which are projected from the linguistic features extracted from the input \(x\).



applies a template with [MASK] token to the original input and maps the predicted label word to the corresponding class Han et al. (2022); Shin et al. (2020). The performance is sensitive to the quality of template, which introduces time-consuming and labor-intensive prompt design and optimization. To address this problem, researchers propose soft prompt strategies, where continuous embeddings of trainable tokens replace the hard template and are optimized by training Liu et al. (2021); Lester et al. (2021).

Despite the success in a range of text classification tasks, existing prompt-based tuning methods still suffer from inferior performance in RA. This might be attributed to the lack of linguistic knowledge which has been demonstrated to play a crucial role in RA Vajjala (2022); Qiu et al. (2021); Li et al. (2022). Meanwhile, RA differs from general classification tasks in that there exists a notion of ranking order between classes. Our intuition behind the utilization of linguistic knowledge is that the learned representations of different levels should preserve the similarity relationship analogous to that of original linguistic features of different levels.

Motivated by the above insights, in this paper, we propose a novel prompt-based tuning method that incorporates rich linguistic knowledge, called **F**eature **P**rompt **T**uning (FPT), as shown in the bottom of Figure 1. Specifically, our methodology begins with extracting linguistic features from the text. These extracted features are subsequently embedded into feature prompts, functioning as trainable soft prompts. Contrary to the conventional prompt tuning frameworks, our model can explicitly benefit from linguistic knowledge. Furthermore, we devise a new loss function to calibrate the similarity relationships between the embedded features across different categories. Our approach is straightforward and effective, offering wide applicability to other tasks where the importance of handcrafted features is emphasized.

To verify the effectiveness of our proposed methods, we conduct extensive experiments on three RA datasets, including one Chinese data Li et al. (2022) and two English datasets, WeeBit Vajjala and Meurers (2012) and Cambridge Xia et al. (2019). By incorporating linguistic knowledge, our proposed model FPT improves significantly over other prompt-based methods. For instance, in the 2-shot setting, FPT brings a relative performance gain of 43.9% over the traditional soft prompt method on the Chinese dataset and 5.50% on English Swebit. Moreover, compared to other feature fusion methods, FPT outperforms the previous best method Projecting Feature (PF) Li et al. (2022) by 43.19% on Chinese data and 11.55% on English Swebit data. Also, we experiment on the Large Language Model (LLM), demonstrating the superiority of our approach on RA. We will make our code public available 1. We summarize our contributions as follows:

Footnote 1: [https://github.com/Wzy232303/FPT](https://github.com/Wzy232303/FPT)

* We propose a novel prompt-based tuning framework, Feature Prompt Tuning (FPT), which incorporates rich linguistic knowledge for RA.
* We design a new calibration loss to ensure the linguistic features retain their original similarity information during optimization.
* Our experimental results show that our method outperforms other prompt-based tuning methods and effectively leverages linguistic features, leading to better and more stable performance improvements than previous approaches.

## 2 Related Works

### Readability Assessment

Early works have explored a wide range of linguistic features as measurements for readability. Flesch (1948) performed regression over features such as average word length in syllable; Schwarm and Ostendorf (2005) trained an SVM over features including LM perplexity and syntactic tree height; Pitler and Nenkova (2008) illustrated that discourse relations can be a good predictor of readability.

Recent works largely employ deep learning approaches for RA. Several deep architectures, including BERT Devlin et al. (2018), HAN Yang et al. (2016), and multi-attentive RNN were applied to achieve strong performance without feature engineering Martinc et al. (2021); Azpiazu and Pera (2019). However, the performance of neural models tends to fluctuate a lot across different RA datasets Deutsch et al. (2020), suggesting that relying only on neural networks might not be a robust solution for RA. Meanwhile, later works have shown that a hybrid approach combining transformer-based encoders with linguistic features can achieve even higher performance Lee et al. (2021); Lee and Vijjala (2022); Li et al. (2022). Lee and Lee (2023) 

applied a prompt-based learning based on seq2seq model such as T5 and BART, treating RA as a text-to-text generative task. Despite the novelty of their method, it was not included in our baselines since it is hard for this method to draw a meaningful comparison against our approach. In addition to the fundamental discrepancy in the task definition, their method focuses on optimizing hard prompts and combining multiple datasets during training, whereas our method focuses on incorporating linguistic knowledge without leveraging multiple datasets.

### Prompt-based Tuning

Fine-tuning PLMs have shown their prevalence in various NLP tasks. PLMs, such as BERT Devlin et al. (2018), GPT Radford et al. (2018), XLNet Xia et al. (2019), RoBERTa Liu et al. (2019) and T5 Raffel et al. (2020), have been proposed with varied self-supervised learning architectures. It has been demonstrated that larger models tend to perform better in many learning scenarios Brown et al. (2020), which stimulated PLMs with billions of parameters to emerge.

Fine-tuning large PLMs may be prohibitive, and there exist a significant gap between pretraining tasks and downstream tasks. Prompt tuning addresses this challenge by reformulating downstream tasks as a language modeling problem and optimizing the prompt. Prompts are used to probe PLM's intrinsic knowledge to perform a task Min et al. (2022), and various techniques of prompting have been explored to aid PLM better: hard prompt Shin et al. (2020); Schick and Schutze (2021), soft prompt Lester et al. (2021); Li and Liang (2021), verbalizer Cui et al. (2022) and pretrained prompt tuning Gu et al. (2021).

The effectiveness of prompt tuning has been validated in various NLP tasks, including sentiment analysis Wu and Shi (2022), named entity recognition Ma et al. (2022), relation extraction Chen et al. (2022) and semantic parsing Schucher et al. (2021). However, the potential of prompt tuning is less explored in RA. In this work, we focus on the effectiveness of linguistic features for modeling readability, and utilize linguistic features to guide prompt tuning.

## 3 Background

We model RA as a text classification task. Formally, a RA dataset can be denoted as \(\mathcal{D}=\{\mathcal{X},\mathcal{Y}\}\), where \(\mathcal{X}\) is the text set and \(\mathcal{Y}\) is the class set. Each instance \(x\in\mathcal{X}\) consists of several tokens, \(x=\{w_{1},w_{2},...,w_{|x|}\}\), and is annotated with a label \(y\in\mathcal{Y}\), indicating the reading difficulty.

### Fine-tuning PLMs for RA

Given a PLM \(\mathcal{M}\) for RA, fine-tuning methods first convert a text \(x=(w_{1},w_{2},...,w_{|x|})\) into an input sequence \(([\textrm{CLS}],w_{1},w_{2},...,w_{|x|},[\textrm{SEP}])\). The PLM encodes this sequence into the hidden vectors \(h=(h_{[\textrm{CLS}]},h_{1},h_{2},...,h_{|x|},h_{[\textrm{SEP}]})\).

In the conventional fine-tuning, an additional classifier \(FC\) is trained on top of the [CLS] embedding \(h_{[\textrm{CLS}]}\). This classifier produces a probability distribution over the class set \(\mathcal{Y}\) through a softmax function, which can be formulated as:

\[P(\cdot|x)=\textrm{Softmax}(FC(h_{[\textrm{CLS}]})),\]

The objective of fine-tuning is to minimize the cross-entropy loss between the predicted probability distribution \(P(\cdot|x)\) and the ground-truth label \(y\):

\[\mathcal{L}_{classification}=-\frac{1}{|\mathcal{X}|}\sum_{x\in\mathcal{X}}\log P (y|x).\]

### Prompt-based Tuning

Prompt-based tuning aims to bridge the gap between pretraining tasks and downstream tasks, as illustrated in Figure 1.

Hard Prompt.It typically consists of a template \(T(\cdot)\), which transforms the input \(x\) into a prompt input \(x_{prompt}\), and a set of label words \(V\) that are connected to the label space through a mapping function \(\Phi:V\rightarrow\mathcal{Y}\), often referred to as the verbalizer. The prompt input contains at least one [MASK] token for the model to fill with label words.

Taking an example in RA, \(x_{prompt}\) could take the form of

\[x_{prompt}=T(x)=\mbox{``It is [MASK] to read: $x^{*}$}.\]

In this case, the input embedding sequence of \(x_{prompt}\) is denoted as

\[(e(\mbox{``It is''}),e(\mbox{[MASK]}),e(\mbox{``to read: "}),e(x)).\]

Soft Prompt.It replaces hard tokens in the template with trainable soft tokens \([h_{1},...,h_{l}]\), yielding an input embedding sequence of

\[(h_{1},...,h_{l},e(\mbox{[MASK]}),e(x)).\] 

Hybrid Prompt.It combines soft tokens with hard prompt tokens \(T\) to form the input embedding sequence:

\[(h_{1},...,h_{l},e(T),e(\text{[MASK]}),e(x)).\]

By feeding the input embedding sequence of \(x_{prompt}\) into \(\mathcal{M}\), the probability distribution over the class set \(\mathcal{Y}\) is modeled by:

\[P_{\mathcal{M}}(y|x)=P_{\mathcal{M}}(\text{[MASK]}=\Phi(y)|x_{prompt})\]

The learning objective of prompt-based tuning is to minimize the cross entropy loss:

\[\mathcal{L}_{classification}=-\frac{1}{|\mathcal{X}|}\sum_{x\in\mathcal{X}} \log P_{\mathcal{M}}(y|x)\]

## 4 Feature Prompt Tuning

In this section, we propose a novel method for RA with prompt-based tuning, named Feature Prompt Tuning (FPT). The architecture of our model is illustrated in Figure 2. Specifically, we extract linguistic features from the texts and embed them into soft prompts. Then, we employ a loss function to calibrate the similarity relationship between embedded features of different classes. We adopt an alternating procedure to optimize the model with respect to the classification loss and calibration loss.

### Feature Prompt Construction

Feature ExtractionOur approach for extracting linguistic features from text is consistent with previous works Li et al. (2022); Lee et al. (2021). For English texts, the linguistic features are extracted using the _lingfeat_ toolkit Lee et al. (2021), which includes discourse, syntactic, lexical, and shallow features. In terms of Chinese linguistic features, we directly utilize the _zhfeat_ toolkit Li et al. (2022) to extract character, word, sentence, and paragraph-level features. Specific details are provided in Appendix A. For an input text \(x\), we denote the extracted features as \(f_{x}\), which is a \(\alpha\)-dimensional vector with \(\alpha\) representing the number of extracted features.

Feature EmbeddingTo incorporate linguistic knowledge into prompt-based tuning, we transform linguistic feature \(f_{x}\) into \(l\) distinct vectors \(\{v_{1},...,v_{l}\}\) which function as the embeddings of soft tokens, as follows:

\[\{v_{1},...,v_{l}\}=\text{MultiHeadMLP}(f_{x}).\]

Here, MultiHeadMLP is a multi-head MLP with \(l\) output heads. Each head consists of a series of fully connected layers followed by non-linear activation functions.

The purpose of using a multi-head MLP is to allow the model to map \(f_{x}\) into separate vector spaces and learn multiple aspects of the linguistic features. This enables the model to better capture the relationships between different features and their contribution to RA.

Ultimately, we formulate the input embedding

Figure 2: The architecture of the proposed Feature Prompt Tuning. Column-wise ranking orders of similarity matrices are denoted with numbers.

 

sequence of \(x_{prompt}\) as follows:

\[(v_{1},...,v_{l},e(T),e(\text{[MASK]}),e(x)).\]

This input sequence is passed through the PLM \(\mathcal{M}\) to calculate \(\mathcal{L}_{classification}\) loss as described in Section 3.2.

### Inter-class Similarity Calibration

We denote \(\mathcal{F}=\left\{F_{c_{1}},\cdots,F_{c_{n}}\right\}\) as the collection of linguistic features for the dataset \(\mathcal{D}\), which consists of \(n\) classes. Here, \(F_{c_{i}}=\left\{f_{x_{i1}},\cdots,f_{x_{ik}}\right\}\) signifies the extracted features of \(k\) samples which belong to \(i\)-th class. We apply average pooling to the feature embeddings of each sample in \(\mathcal{F}\), resulting in a set of embedded linguistic features, denoted as \(\mathcal{F^{\prime}}=\left\{F^{\prime}_{c_{1}},\cdots,F^{\prime}_{c_{n}}\right\}\). To gauge the similarity between any two classes \(F_{c_{m}}\) and \(F_{c_{n}}\), we employ a pairwise approach based on cosine similarity, expressed as:

\[s_{mn}=\frac{1}{k^{2}}\sum_{i=1}^{k}\sum_{j=1}^{k}\cos(f_{x_{mi}},f_{x_{nj}})\]

With the feature representation and similarity function in place, we can define our calibration objective. The fundamental intuition is that the distribution of extracted linguistic features should be preserved as much as possible. Namely, if the similarity between \(F_{c_{m}}\) and \(F_{c_{n}}\) is relatively low, the similarity between \(F^{\prime}_{c_{m}}\) and \(F^{\prime}_{c_{n}}\) should also be proportionately low, and vice versa. Therefore, during the training process, we devise an objective function based on a list-wise ranking loss function ListMLE Xia et al. (2008), to maintain this initial ranking relationship.

More specifically, we compute the similarity between each pair of classes within \(\mathcal{F}\) to generate the similarity matrix:

\[M=\left[\begin{array}{cccc}s_{11}&s_{12}&\cdots&s_{1n}\\ s_{21}&s_{22}&\cdots&s_{2n}\\ \vdots&\vdots&\ddots&\vdots\\ s_{n1}&s_{n2}&\cdots&s_{nn}\end{array}\right]\]

Likewise, we can derive the similarity matrix \(M^{\prime}\) for \(\mathcal{F^{\prime}}\).

We then use \(\Pi=\left\{\pi_{1},\pi_{2},\cdots,\pi_{n}\right\}\) to denote the ranking order of the columns in \(M\), where \(\pi_{i}\) represents the ranking order of the \(i\)-th column. We obtain \(\hat{M^{\prime}}\) by rearranging the columns of \(M^{\prime}\) according to \(\Pi\):

\[\hat{M^{\prime}}=\left[\begin{array}{cccc}s^{\prime}_{\pi_{11}}&s^{\prime}_ {\pi_{12}}&\cdots&s^{\prime}_{\pi_{1n}}\\ s^{\prime}_{\pi_{21}}&s^{\prime}_{\pi_{22}}&\cdots&s^{\prime}_{\pi_{2n}}\\ \vdots&\vdots&\ddots&\vdots\\ s^{\prime}_{\pi_{n1}}&s^{\prime}_{\pi_{n2}}&\cdots&s^{\prime}_{\pi_{nn}}\end{array}\right]\]

Finally, we aim to minimize the following loss function for similarity calibration:

\[L_{calibration}=-\sum_{k=1}^{n}log\prod_{i=1}^{n}\frac{\exp(s^{\prime}_{\pi_{ ik}})}{\sum_{j=i}^{n}\exp(s^{\prime}_{\pi_{jk}})}\]

### Training Procedure

Training ObjectivesGiven the dataset \(\mathcal{D}\) and the linguistic feature set \(\mathcal{F}\), we establish two training objectives. The primary objective is to minimize the classification loss \(L_{classification}\), which is computed based on the difference between the predicted and actual class labels. The secondary objective is to calibrate the inter-class similarity of the mapped features by minimizing the loss function \(L_{calibration}\) defined in Section 4.2.

```
1:Initialize model parameters \(M\) and feature embeddings \(f\)
2:for each epochdo
3: Shuffle dataset \(D\)
4:for each batch \(b\) in \(D\)do
5: Compute \(L_{classification}\) for \(b\) using \(M\) and \(f\)
6: Update \(M\) and \(f\) by minimizing \(L_{classification}\)
7:endfor
8: Compute \(L_{calibration}\) for \(D\) using \(f\)
9: Update \(f\) by minimizing \(L_{calibration}\)
10:endfor
```

**Algorithm 1** Alternating Training Procedure for Feature Prompt Learning

Alternating Training ProcedureFor training both loss functions, we adopt an alternating training procedure, as encapsulated in Algorithm 1. This procedure iteratively updates the model parameters and feature embeddings by minimizing the classification loss \(L_{classification}\) and the similarity calibration loss \(L_{calibration}\), respectively.

In each epoch, the dataset \(D\) is shuffled to ensure the model is not biased towards any particular ordering of the data. For each batch \(b\) in \(D\), the classification loss \(L_{classification}\) is computed using 

the current model parameters \(M\) and feature embeddings \(f\). The model parameters \(M\) and feature embeddings \(f\) are then updated by minimizing this loss. Subsequently, the similarity calibration loss \(L_{calibration}\) is computed using the updated feature embeddings \(f\) for the epoch, and the feature embeddings are updated by minimizing this loss . This process is repeated for each epoch. The alternating training procedure ensures that the model learns to classify the data accurately while maintaining the inter-class similarity structure of the feature space.

## 5 Experimental Setting

### Datasets

To evaluate the effectiveness of our proposed method, we conduct experiments on one Chinese dataset and two English datasets, following the same data split as Li et al. (2022). The statistics of the datasets can be found in Table 1.

**ChineseLR**Li et al. (2022) is a Chinese dataset collected from textbooks of the middle and primary schools of more than ten publishers. Following the standards specified in the _Chinese Curriculum Standards for Compulsory Education_, all texts are divided into five difficulty levels.

**WeeBit**Vajjala and Meurers (2012) is often considered as the benchmark data for English RA. It was initially created as an extension of the well-known Weekly Reader corpus.

**Cambridge**Xia et al. (2019) consists of reading passages from the five main suite Cambridge English Exams (KET, PET, FCE, CAE, CPE).

### Baselines 1: Prompt-based Methods

For prompt-based methods, we compare with hard, soft, and hybrid prompts. To avoid the influence of verbalizers on experimental results, we adopt a soft verbalizer Hambardzumyan et al. (2021) that employs a linear layer classifier across all prompt-based methods.

**Hard Prompt (HP)**: We implement four manually defined templates for prompt tuning and select a template with the best performance on the development set. As for FPT in Table 2, we report the test set performance averaged over the four templates. This setting poses a challenge to FPT, as the averaged performance of FPT should outperform the best performance of HP to demonstrate its effectiveness. Details of the templates can be found in Appendix B.

**Soft Prompt (SP)**: It replaces manually defined prompts with trainable continuous prompts. We follow the same implementation as Lester et al. (2021) and use randomly sampled vocabulary to initialize the prompts.

**Hybrid Prompt (HBP)**: It concatenates trainable continuous prompts to the wrapped input embeddings. We adopt the implementation from Gu et al. (2022).

**P-tuning**: A hybrid prompt method, which replaces some tokens in manually designed prompts with soft prompts and only retains task-relevant anchor words. The soft prompts are embedded with a bidirectional LSTM and a MLP Liu et al. (2021).

### Baselines 2: Fusion Methods

We also compare with the methods fusing linguistic features and PLMs from previous studies.

**SVM**: Use the single numerical output of a neural model (BERT) as a feature itself, joined with linguistic features, and then fed them into SVM Lee et al. (2021); Deutsch et al. (2020).

**FT**: Standard fine-tuning method without linguistic features, where the hidden representation of [CLS] token is used for classification. This baseline validates whether the linguistic features indeed have a positive effect.

**Concatenation (Con)**: Fine-tune with linguistic features, in which the linguistic features are directly concatenated to the hidden representation of the [CLS] token Meng et al. (2021); Qiu et al. (2021).

**PF**: Fuse linguistic features with hidden representations of [CLS] through projection filtering Li et al. (2022).

### Implementation Details

Under the few-shot setting, we randomly sample \(k=1,2,4,8,16\) instances in each class from the training and development set. For each \(k\)-shot experiment, we sample 4 different training and dev sets and repeat experiments on each training set for 4 times. We select the best model checkpoint based on the performance of the development set

\begin{table}
\begin{tabular}{c c c c c c c} \hline
**Dataset** & \multicolumn{3}{l}{**WeeBit**} & \multicolumn{3}{l}{**Cambridge**} & \multicolumn{1}{r}{**ChineseLR**} \\ \cline{2-7}
**Level** & \# & Avg Len & \# & Avg Len & \# & Avg Len \\ \hline
1 & 625 & 152 & 60 & 141 & 814 & 266 \\
2 & 625 & 189 & 60 & 271 & 1063 & 679 \\
3 & 625 & 295 & 60 & 617 & 1104 & 1140 \\
4 & 625 & 242 & 60 & 763 & 762 & 2165 \\
5 & 625 & 347 & 60 & 751 & 417 & 3299 \\ All & 3125 & 245 & 300 & 509 & 4160 & 1255 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of RA datasets. #: number of the passages. Avg Len: average tokens numbers per passage.

 

and evaluate the models on the entire test set. As for the evaluation metric, we use \(accuracy\) in all experiments and take the mean values as the final results.

All our models and baselines are implemented with the PyTorch Paszke et al. (2019) framework and Huggingface transformers Wolf et al. (2020). We use BERT Devlin et al. (2018) as our Pretrained Language Model (PLM) backbone. We use "bert-base-uncased" for English datasets and "bert-base-chinese" for the Chinese dataset. During training, we employ the AdamW optimizer Loshchilov and Hutter (2019) with a weight decay of 0.01 and a warm-up ratio of 0.05. We tune the model with the batch size of 8 for 30 epochs, and the learning rate is 1e-5. All experiments are conducted with four NVIDIA GeForce RTX 3090s.

## 6 Results and Analysis

### Comparison with Prompt-based Methods

Table 2 shows the results of our proposed method FPT and prompt-based baselines under the few-shot setting. (1) Our method FPT significantly outperforms nearly all baseline methods across all three datasets under different shots, demonstrating that our method exhibits greater robustness and adaptability to variations in data sizes and languages. (2) FTP particularly excels on the ChineaLR dataset, and it outperforms the soft prompt (SP) method by 8.41, 14.1, 10.15, 9.94 and 7.9 points under 1, 2, 4, 8, 16 shots, respectively. (3) In the task of RA, the soft prompt method generally outperforms the hard prompt. Interestingly, the hybrid prompt, a combination of both, does not always yield better results than the standalone soft prompt. This could be attributed to the inherent challenge in designing and selecting effective hard prompts for RA. Nevertheless, as a hybrid prompt approach that integrates linguistic knowledge, our proposed method continues to exhibit robust performance, demonstrating its adaptability and effectiveness.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline \(k\) & **Methods** & **ChineseLR** & **Weebit** & **Cambridge** \\ \hline  & HP & 29.49(5.21) & 41.83(4.72) & 36.25(8.49) \\  & SP & 31.22(4.70) & **46.61**(3.63) & 41.73(8.45) \\
1 & HBP & 33.51(5.19) & 44.46(5.02) & 42.04(9.12) \\  & P-tuning & 33.36(4.12) & 41.23(4.11) & 40.36(7.15) \\ \cline{2-5}  & FPT(ours) & **39.63**(6.38) & 43.61(4.50) & **44.17**(7.12) \\ \hline  & HP & 28.38(8.14) & 49.23(2.85) & 46.88(9.31) \\
2 & SP & 32.14(5.54) & 52.22(4.35) & 43.13(8.88) \\
2 & PHP & 33.38(7.02) & 52.52(2.66) & 49.56(7.12) \\  & P-tuning & 35.12(4.20) & 50.71(3.87) & 48.97(8.47) \\ \cline{2-5}  & FPT(ours) & **46.24**(5.62) & **55.10**(4.04) & **59.79**(10.2) \\ \hline  & HP & 36.56(5.18) & 53.41(4.50) & 48.75(8.49) \\  & SP & 38.78(2.83) & 54.96(3.89) & 49.36(9.14) \\
4 & Hipp & 39.81(2.67) & 56.88(3.52) & 50.13(8.77) \\  & P-tuning & 38.45(3.09) & 54.35(3.21) & 48.85(9.64) \\ \hline  & FPT(ours) & **48.93**(3.21) & **57.70**(4.63) & **53.54**(7.21) \\ \hline  & HP & 41.21(4.83) & 61.31(3.13) & 55.42(6.86) \\
8 & SP & 42.72(2.82) & 62.02(2.67) & 56.75(6.89) \\
8 & HBP & 41.93(4.12) & 63.37(2.02) & 57.34(9.28) \\  & P-tuning & 42.81(4.04) & 61.81(3.28) & 56.90(7.23) \\ \cline{2-5}  & FPT(ours) & **52.66**(5.00) & **64.92**(2.75) & **59.38**(6.58) \\ \hline  & HP & 47.35(3.69) & 63.75(5.41) & 61.67(8.98) \\  & SP & 47.44(2.09) & 67.54(4.56) & 63.77(7.43) \\
16 & HIP & 47.08(3.11) & 67.30(4.69) & 63.98(7.34) \\  & P-tuning & 46.26(3.19) & 65.52(3.84) & 62.03(9.62) \\ \cline{2-5}  & FPT(ours) & **55.25**(2.93) & **68.19**(4.21) & **65.00**(4.25) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental results comparing with prompt-based methods. We report the mean performance and the standard deviation in brackets. The best results are in bold, and the best results of previous prompt-based methods are underlined.

\begin{table}
\begin{tabular}{c l c c c} \hline \hline \(k\) & **Methods** & **ChineseLR** & **Weebit** & **Cambridge** \\ \hline  & FT & 28.59(4.88) & 45.90(2.94) & 34.17(4.33) \\  & SVM & 25.34(3.87) & 44.82(3.14) & 35.31(5.23) \\
1 & Con & 28.53(4.68) & 43.81(3.88) & 33.33(1.01) \\  & PF & 30.13(3.99) & 44.01(2.91) & 35.11(10.12) \\ \cline{2-5}  & FPT(ours) & **33.29**(4.80) & **46.67**(3.50) & **43.96**(7.09) \\ \hline  & FT & 22.87(7.19) & 48.79(3.49) & 44.17(10.4) \\  & SVM & 23.95(9.28) & 49.55(3.78) & 43.99(11.0) \\
2 & Con & 25.61(8.21) & 49.29(2.88) & 41.67(8.16) \\  & PF & 26.12(7.21) & 50.23(2.81) & 41.52(7.34) \\ \cline{2-5}  & FPT(ours) & **37.40**(4.77) & **56.03**(3.48) & **55.83**(6.72) \\ \hline  & FT & 36.64(5.37) & 52.46(4.28) & 47.50(6.29) \\  & SVM & 37.11(6.88) & 53.03(6.65) & 47.58(6.67) \\
4 & Con & 36.64(5.37) & 52.46(4.28) & 47.50(6.29) \\  & PF & 37.13(5.11) & 53.18(2.99) & 48.46(4.79) \\ \cline{2-5}  & FPT(ours) & **44.88**(3.27) & **56.17**(3.84) & **55.00**(4.86) \\ \hline  & FT & 40.45(2.91) & 61.13(3.15) & 61.46(7.81) \\  & SVM & 40.52(3.67) & 60.98(5.78) & 61.55(9.10) \\
8 & Con & 41.65(2.98) & 58.41(3.31) & 58.96(7.43) \\  & PF & 44.40(2.86) & 59.32(2.97) & 55.62(10.9) \\ \cline{2-5}  & FPT(ours) & **47.60**(3.66) & **62.40**(3.30) & **64.17**(5.95) \\ \hline  & FT & 45.73(4.11) & 65.93(5.50) & 71.04(7.97) \\  & SVM & 46.85(3.72) & 63.72(4.98) & 71.22(8.15) \\
16 & Con & 48.33(3.99) & 64.52(4.73) & **71.46**(6.12) \\  & PF & 48.66(3.20) & 65.08(4.60) & 69.38(6.79) \\ \cline{2-5}  & FPT(ours) & **53.94**(3.16) & **68.10**(3.25) & 69.17(7.77) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results comparing with feature fusion methods. Con means Con concatenation. For a fair comparison, here FPT concatenates the embedded linguistic features to the embeddings of the input sequence (without hard prompt template) and outputs the classification logits over \([\text{CLS}]\) token embedding instead of \([\text{MASK}]\) 



### Comparison with Fusion Methods

Table 3 reports the experimental results comparing with fusion methods under the few-shot setting. (1) Our proposed method FPT shows a stable and significant improvement compared to the previous feature fusion methods. For instance, in the 2-shot setting, FPT outperforms the best previous fusion methods by 11.28, 5.8 and 11.66 points on ChineseLR, Weebit and Cambridge, respectively. This demonstrates our method's effectiveness in integrating linguistic features for RA. (2) Methods with linguistic features perform better than standard fine-tuning on Chinese datasets. However, it may not necessarily lead to improvement on English datasets, especially when \(k\) is increased to a sufficient amount, which indicates that simply applying linguistic features to aid in English RA is not consistently effective.

### Ablation Study

To validate the effectiveness of each component in our proposed model, we conduct ablation experiments on both English Weebit and ChineseLR datastes. Table 4 lists the results. Notably, our similarity calibration (SC) is built on the feature prompt (FP), with the aim to maintain consistent inter-class similarity of linguistic features. Therefore, removing FP also detaches SC, explaining why our ablation study is performed incrementally.

Our full model yields the best performance on both datasets. When removing the SC module, the performance is markedly decreased, demonstrating the necessity of retaining the linguistic features' original similarity information during optimization. We have also investigated the impact of SC by visualising the similarity difference matrix before and after applying SC, the results of which are presented in Appendix C. Moreover, further removal of the FP shows a steep drop in performance (12.37 points on ChieseLR and 4.29 points on Weebit when \(k=4\)), validating the effectiveness of incorporating linguistic features as soft prompts. We note that the improvement of SC and FP is more significant on the Chinese dataset compared to the English dataset, indicating that the Chinese RA task is more dependent on linguistic features.

### The Significance of Linguistic Features

To further analyze whether linguistic features improve performance, in our model structure, we replace the linguistic feature vectors with randomly initialized vectors. On the other hand, we re-implement the Hybrid Prompt Tuning by utilizing pseudo tokens as soft prompts. We conduct experiments on WeeBit and ChineseLR datasets, and the comparison results are shown in Figure 3.

The performance on both datasets significantly decreases when the linguistic features are replaced with random vectors, especially on the ChineseLR dataset, where the decrease is up to 16.27%. The fewer the samples, the more severe the decline caused by the replacement, further indicating the beneficial role of linguistic features when data is insufficient. Moreover, compared to pseudo tokens, using vector-form embeddings as soft prompts requires the integration of linguistic knowledge to achieve better performance.

### Comparison with the LLM

The large language model (LLM) excels at various downstream tasks without the need for parameter adjustment. We carry out experiments on LLM, utilizing the gpt-3.5-turbo-16k API. We sample the same examples as in other experiments, and the prompt is generated by GPT-4. Specifically, we provide GPT-4 with the task instructions to generate the system prompt and user input for gpt-3.5-turbo-16k, as shown in Figure 4. Table 5 shows that our model with 110M parameters significantly outperforms the LLM on the English dataset (except for one sample on Cambridge). Moreover,

Figure 3: The comparison results of linguistic features, randomly initialized vectors and pseudo tokens.



gpt-3.5-turbo-16k is unable to perform 1-shot or 2-shot experiments on ChineseLR due to its limited context length. This underscores the necessity for research on handling long texts in RA.

## 7 Conclusion

Inspired by the solid performance of prompt tuning on classification tasks and the importance of linguistic features in the RA task, we empirically investigated the effectiveness of incorporating linguistic features into prompt tuning for RA. We convert linguistic features of the input into soft tokens and utilize the similarity calibration loss to preserve the similarity relationship between classes before and after the transformation. The results show noticeable improvements over previous fusion methods and prompt-based approaches in the few-shot learning setting. The ablation study further illustrated that the proposed model benefits from linguistic features and additional similarity calibration. Our proposed method, FPT, has demonstrated a new possibility of prompt tuning in an era dominated by LLMs, showcasing its undeniable significance and value in linguistic-related tasks.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China (62076008) and the Key Project of Natural Science Foundation of China (61936012).

## Limitations

Our proposed method, which leverages the masked language model (MLM) backbone such as BERT, has demonstrated its efficacy across a variety of natural language processing tasks. Despite its strengths, we acknowledge several limitations that warrant further investigation.

Firstly, our approach exhibits constraints in processing long texts, a scenario frequently encountered in Chinese readability evaluation datasets. The inherent architecture of MLMs like BERT is optimized for shorter sequences, leading to potential performance degradation when dealing with extensive text inputs.

Secondly, while MLM-based methods are proficient in classification tasks, they often fall short in terms of interpretability of the classification outcomes. The black-box nature of these models makes it challenging to trace and understand the decision-making process, which is crucial for applications where justification of results is required.

Lastly, the success of our method is significantly contingent upon the quality of linguistic features extracted from the text. However, the extraction of high-quality linguistic features is not always guaranteed, especially in languages with rich morphology or poor data resources.

In conclusion, while our method stands as a robust approach for several NLP tasks, addressing these limitations is imperative for advancing the field and extending the applicability of MLM-based models to a broader spectrum of text analysis challenges. It is also worth noting that only one Chinese dataset is included in this work, as it appears to be the only Chinese RA dataset available to the best of

Figure 4: The system prompt and the user input for prompting LLM.

\begin{table}
\begin{tabular}{|r|c c c|} \hline \hline \(k\) & Dataset & FPT & LLM \\ \hline  & \multicolumn{1}{l}{Weebit} & - & 30.79 \\
0 & Cambridge & - & 43.33 \\  & ChineseLR & - & 21.67 \\ \hline  & \multicolumn{1}{l}{Weebit} & **43.61** & 31.75 \\
1 & Cambridge & 44.17 & **48.33** \\  & ChineseLR & 39.63 & - \\ \hline  & \multicolumn{1}{l}{Weebit} & **55.10** & 33.17 \\
2 & Cambridge & **59.79** & 54.16 \\  & ChineseLR & 46.24 & - \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison (accuracy) between our model and LLM (gpt-3.5-turbo-16k) on three datasets. \(k\) represents the number of in-context examples. Due to the limitation of context length, the experiments on Chinese dataset cannot be carried out.

 

our knowledge. We urge that more attention should be paid to this field of work and further experiments will be conducted if new datasets are released.

### Ethics Statement

Potential RisksFirstly, as a neural network-based method, the predictive outcomes of our approach should not be applied in practical applications without the involvement of human experts. This is a responsible practice for the actual beneficiaries, the learners. Secondly, as mentioned earlier, low-quality or even incorrect linguistic features can negatively impact our method. Therefore, evaluating the quality of linguistic features is essential for the efficacy of our approach.

About Computational BudgetFor each k-shot experiment, we conducted a total of 16 repetitions (refer to Section 5.4) for all baselines and FPT. The duration of a single experiment varies according to the size of k (approximately 20 seconds to 200 seconds), but the time consumed by different methods is almost identical.

Use of Scientific ArtifactsWe utilize the _lingfeat_ toolkit Lee et al. (2021) to extract linguistic features from English texts; this toolkit is publicly accessible under the CC-BY-SA-4.0 license. For extracting Chinese linguistic features, we employ the _zhfeat_ toolkit Li et al. (2022).

Use of AI AssistantsWe have employed ChatGPT as a writing assistant, primarily for polishing the text after the initial composition.

## References

* L. Madrazo Azpiazu and M. Solead Pera (2019)Multiattentive recurrent neural network architecture for multilingual readability assessment. Transactions of the Association for Computational Linguistics7, pp. 421-436. Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1573-461X, Link, Document Cited by: SS1.
* X. Chen, L. Li, N. Zhang, C. Tan, F. Huang, L. Si, and H. Chen (2022)Relation extraction as open-book examination: retrieval-enhanced prompt tuning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2443-2448. Cited by: SS1.
* K. Collins-Thompson and J. Callan (2004)Information retrieval for language tutoring: an overview of the reap project. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '04, New York, NY, USA, pp. 544-545. External Links: ISBN 978-1-4503-5461-4, Link, Document Cited by: SS1.
* G. Cui, S. Hu, N. Ding, L. Huang, and Z. Liu (2022)Prototypical verbalizer for prompt-based few-shot tuning. arXiv preprint arXiv:2203.09770. Cited by: SS1.
* T. Deutsch, M. Jasbi, and S. Shieber (2020)Linguistic features for readability assessment. arXiv preprint arXiv:2006.00377. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2018)Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: SS1.
* R. Flesch (1948)A new readability yardstick. Journal of applied psychology32 (3), pp. 221. Cited by: SS1.
* Y. Gu, X. Han, Z. Liu, and M. Huang (2021)Ppt: pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. Cited by: SS1.
* Y. Gu, X. Han, Z. Liu, and M. Huang (2022)PPT: pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 8410-8423. External Links: ISBN 978-1-4503-5461-4, Link, Document Cited by: SS1.
* K. Hambardzumyan, H. Khachatrian, and J. May (2021)WARP: word-level adversarial Reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 4921-4933. External Links: ISBN 978-1-4503-5461-4, Link, Document Cited by: SS1.
* X. Han, W. Zhao, N. Ding, Z. Liu, and M. Sun (2022)Ptr: prompt tuning with rules for text classification. AI Open3, pp. 182-192. Cited by: SS1.
* J. Klim, K. Collins-Thompson, P. N. Bennett, and S. T. Dumais (2012)Characterizing web content, user interests, and search behavior by reading level and topic. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM '12, New York, NY, USA, pp. 213-222. External Links: ISBN 978-1-4503-5461-4, Link, Document Cited by: SS1.
* B. W. Lee, Y. Sung Jang, and J. Lee (2021)Pushing on text readability assessment: a transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Vol. 10669-10686.

 

* Lee and Lee (2023) Bruce W. Lee and Jason Lee. 2023. Prompt-based learning for text readability assessment. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1819-1824, Dubrovnik, Croatia. Association for Computational Linguistics.
* Lee and Vajjala (2022) Justin Lee and Sowmya Vajjala. 2022. A neural pairwise ranking model for readability assessment. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 3802-3813, Dublin, Ireland. Association for Computational Linguistics.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Li et al. (2022) Wenbiao Li, Wang Ziyang, and Yunfang Wu. 2022. A unified neural network model for readability assessment with feature projection and length-balanced loss. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 7446-7457, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. _arXiv preprint arXiv:2101.00190_.
* Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Namam Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Ma et al. (2022) Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022. Template-free prompt tuning for few-shot NER. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5721-5732, Seattle, United States. Association for Computational Linguistics.
* Martinc et al. (2021) Matej Martinez, Senja Pollak, and Marko Robnik-Sikonja. 2021. Supervised and unsupervised neural approaches to text readability. _Computational Linguistics_, 47(1):141-179.
* Meng et al. (2021) Changping Meng, Muhao Chen, Jie Mao, and Jennifer Neville. 2021. Readnet: A hierarchical transformer framework for web article readability analysis. _CoRR_, abs/2103.04083.
* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanen Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? _arXiv preprint arXiv:2202.12837_.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. _Advances in neural information processing systems_, 32.
* Pitler and Nenkova (2008) Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In _Proceedings of the 2008 conference on empirical methods in natural language processing_, pages 186-195.
* Qiu et al. (2021) Xinying Qiu, Yuan Chen, Hanwu Chen, Jian-Yun Nie, Yuming Shen, and Dawei Lu. 2021. Learning syntactic dense embedding with correlation graph for automatic readability assessment. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3013-3025, Online. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Rello et al. (2012) Luz Rello, Horacio Saggion, Ricardo Baeza-Yates, and Eduardo Graells. 2012. Graphical schemes may improve readability but not understandability for people with dyslexia. In _Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations_, pages 25-32, Montreal, Canada. Association for Computational Linguistics.
* Schick and Schutze (2021) Timo Schick and Hinrich Schutze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 255-269.
* Schucher et al. (2021) Nathan Schucher, Siva Reddy, and Harn de Vries. 2021. The power of prompt tuning for low-resource semantic parsing. _arXiv preprint arXiv:2110.08525_.
* Schwarm and Ostendorf (2005) Sarah E Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In _Proceedings of the 43rd annual meeting of the Association for Computational Linguistics (ACL'05)_, pages 523-530 

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4222-4235, Online. Association for Computational Linguistics.
* Vajjala [2022] Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 5366-5377, Marseille, France. European Language Resources Association.
* Vajjala and Meurers [2012] Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In _Proceedings of the seventh workshop on building educational applications using NLP_, pages 163-173.
* Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu and Shi [2022] Hui Wu and Xiaodong Shi. 2022. Adversarial soft prompt tuning for cross-domain sentiment analysis. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2438-2447.
* Xia et al. [2008] Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. _Liswise Approach to Learning to Rank: Theory and Algorithm_, page 1192-1199. Association for Computing Machinery, New York, NY, USA.
* Xia et al. [2019] Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2019. Text readability assessment for second language learners. _arXiv preprint arXiv:1906.07580_.
* Yang et al. [2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_, pages 1480-1489.

## Appendix A Details of Linguistic Features

### Chinese Linguistic Features

\begin{table}
\begin{tabular}{l|p{227.6pt}|p{227.6pt}} \hline \multicolumn{1}{c|}{**Idx**} & \multicolumn{1}{c}{**Dim**} & **Feature description** \\ \hline
2 & 1 & Number of character types \\ \hline
3 & 1 & Type Token Ratio (TTR) \\ \hline
4 & 1 & Average number of strokes \\ \hline
5 & 1 & Weighted average number of strokes \\ \hline
6 & 25 & Number of characters with different strokes \\ \hline
7 & 25 & Proportion of characters with different strokes \\ \hline
8 & 1 & Average character frequency \\ \hline
9 & 1 & Weighted average character frequency \\ \hline
10 & 1 & Number of single characters \\ \hline
11 & 1 & Proportion of single characters \\ \hline
12 & 1 & Number of common characters \\ \hline
13 & 1 & Proportion of common characters \\ \hline
14 & 1 & Number of unregistered characters \\ \hline
15 & 1 & Proportion of unregistered characters \\ \hline
16 & 1 & Number of first-level characters \\ \hline
17 & 1 & Proportion of first-level characters \\ \hline
18 & 1 & Number of second-level characters \\ \hline
19 & 1 & Proportion of second-level characters \\ \hline
20 & 1 & Number of third-level characters \\ \hline
21 & 1 & Proportion of third-level characters \\ \hline
22 & 1 & Number of fourth-level characters \\ \hline
23 & 1 & Proportion of fourth-level characters \\ \hline
24 & 1 & Average character level \\ \hline \end{tabular}
\end{table}
Table 6: Character features description.

\begin{table}
\begin{tabular}{l|p{227.6pt}|p{227.6pt}} \hline \hline
**Idx** & **Dim** & **Feature description** \\ \hline
1 & 1 & Total number of words \\ \hline
2 & 1 & Number of word types \\ \hline
3 & 1 & Type Token Ratio (TTR) \\ \hline
4 & 1 & Average word length \\ \hline
5 & 1 & Weighted average word length \\ \hline
6 & 1 & Average word frequency \\ \hline
7 & 1 & Weighted average word frequency \\ \hline
8 & 1 & Number of single-character words \\ \hline
9 & 1 & Proportion of single-character words \\ \hline
10 & 1 & Number of two-character words \\ \hline
11 & 1 & Proportion of two-character words \\ \hline
12 & 1 & Number of three-character words \\ \hline
13 & 1 & Proportion of three-character words \\ \hline
14 & 1 & Number of four-character words \\ \hline
15 & 1 & Proportion of four-character words \\ \hline
16 & 1 & Number of multi-character words \\ \hline
17 & 1 & Proportion of multi-character words \\ \hline
18 & 1 & Number of idioms \\ \hline
19 & 1 & Number of single words \\ \hline
20 & 1 & Proportion of single words \\ \hline
21 & 1 & Number of unregistered words \\ \hline
22 & 1 & Proportion of unregistered words \\ \hline
23 & 1 & Number of first-level words \\ \hline
24 & 1 & Proportion of first-level words \\ \hline
25 & 1 & Number of second-level words \\ \hline
26 & 1 & Proportion of second-level words \\ \hline
27 & 1 & Number of third-level words \\ \hline
28 & 1 & Proportion of third-level words \\ \hline
29 & 1 & Number of fourth-level words \\ \hline
30 & 1 & Proportion of fourth-level words \\ \hline
31 & 1 & Average word level \\ \hline
32 & 57 & Number of words with different POS \\ \hline
33 & 57 & Proportion of words with different POS \\ \hline \hline  & \multicolumn{1}{p{227.6pt}}{} & \multicolumn{1}{p{227.6pt}}{} \\ \end{tabular}
\end{table}
Table 7: Word features description.

 



### English Linguistic Features

\begin{table}
\begin{tabular}{l|p{28.5pt}|p{143.9pt}} \hline \hline
**Idx** & **Dim** & **Feature description** \\ \hline
1 & 1 & Total count of Noun phrases \\ \hline
2 & 1 & Average count of Noun phrases per sentence \\ \hline
3 & 1 & Average count of Noun phrases per sentence \\ \hline
4 & 1 & Ratio of Noun phrases count to Verb phrases count \\ \hline
4 & 1 & Ratio of Neon phrases count to Subordinate clauses count \\ \hline
6 & 1 & Ratio of New phrases count to Prep phrases count \\ \hline
7 & 1 & Ratio of New phrases count to Adj phrases count to Adj phrases count \\ \hline
8 & 1 & Ratio of Non phrases count to Adv phrases count \\ \hline
9 & 1 & Total count of Verb phrases \\ \hline
10 & 1 & Average count of Verb phrases per sentence \\ \hline
11 & 1 & Average count of Verb phrases per token \\ \hline
12 & 1 & Ratio of Verb phrases count to Num phrases count \\ \hline
13 & 1 & Ratio of Verb phrases count to Subordinate clauses count \\ \hline
14 & 1 & Ratio of Verb phrases count to Prep phrases count to Adv phrases count \\ \hline
15 & 1 & Ratio of Verb phrases count to Adv phrases count \\ \hline
16 & 1 & Ratio of Verb phrases count to Adv phrases count \\ \hline \end{tabular}
\end{table}
Table 10: Discourse features description.

\begin{table}
\begin{tabular}{c|p{28.5pt} p{143.4pt}} \hline \hline
**Idx** & **Dim** & **Feature description** \\ \hline
1 & 1 & Total number of sentences \\ \hline
2 & 1 & Average characters in a sentence \\ \hline
3 & 1 & Average words in a sentence \\ \hline
4 & 1 & Maximum characters in a sentence \\ \hline
5 & 1 & Maximum words in a sentence \\ \hline
6 & 1 & Number of clauses \\ \hline
7 & 1 & Average characters in a clause \\ \hline
8 & 1 & Average words in a clause \\ \hline
9 & 1 & Maximum characters in a clause \\ \hline
10 & 1 & Maximum words in a clause \\ \hline
11 & 30 & Sentence length distribution \\ \hline
12 & 1 & Average syntax tree height \\ \hline
13 & 1 & Maximum syntax tree height \\ \hline
14 & 1 & Syntax tree height \textless= 5 ratio \\ \hline
15 & 1 & Syntax tree height \textless= 10 ratio \\ \hline
16 & 1 & Syntax tree height \textless= 15 ratio \\ \hline
17 & 1 & Syntax tree height \textgreater= 16 ratio \\ \hline
18 & 14 & Dependency distribution \\ \hline \hline  & & \\ \end{tabular}
\end{table}
Table 8: Sentence features description.

\begin{table}
\begin{tabular}{c|p{28.5pt}|p{143.4pt}} \hline \hline
**Idx** & **Dim** & **Feature description** \\ \hline
1 & 1 & Total number of paragraphs \\ \hline
2 & 1 & Average characters in a paragraph \\ \hline
3 & 1 & Average words in a paragraph \\ \hline
4 & 1 & Maximum characters in a paragraph \\ \hline
5 & 1 & Maximum words in a paragraph \\ \hline \hline  & & \\ \end{tabular}
\end{table}
Table 9: Paragraph features description.



* [16] M. C. Bader, R. G. Kudritz, and J. A. Friedman, _A Survey of the Software for Computer Science: The Case of the Study_. PhD thesis, University of California, Berkeley, CA (2009).

[MISSING_PAGE_POST]

 



## Appendix B Templates

Chinese DatasetBased on the _Chinese Curriculum Standards for Compulsory Education_, we devise the following templates:



Figure 5: Similarity difference matrices. We plot the difference matrices of similarity before and after linguistic feature embedding, both with and without SC. The horizontal and vertical coordinates represent the level of linguistic features. By comparing the diagonal of the matrix before and after the similarity calibration (that is, the similarity between linguistic features of the same level), the similarity between analogous categories is drawn closer.