Venue: NAACL 2025

# An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues

Rena Gao\({}^{\heartsuit\diamondsuit}\), Jingxuan Wu\({}^{\clubsuit}\), Xuetong Wu\({}^{\heartsuit}\), Carsten Roever\({}^{\heartsuit}\), Jing Wu\({}^{\spadesuit}\), Long Lv\({}^{\diamondsuit}\), Jey Han Lau\({}^{\heartsuit}\)

###### Abstract

We analyse the cross-lingual transferability of a dialogue evaluation framework that assesses the relationships between micro-level linguistic features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management), originally designed for English-as-a-second-language dialogues. To this end, we develop CNIMA (Chinese Non-Native Interactivity Measurement and **A**utomation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We found the evaluation framework to be robust across distinct languages: English and Chinese, revealing language-specific and language-universal relationships between micro-level and macro-level features. Next, we propose an automated, interpretable approach with low data requirement that scores the overall quality of a second-language dialogue based on the framework. Our approach is interpretable in that it reveals the key linguistic and interactivity features that contributed to the overall quality score. As our approach does not require labelled data, it can also be adapted to other languages for second-language dialogue evaluation.

The University of Melbourne, Parkville, 3052, Australia The Chinese University of Hong Kong, Shatin, NT, Hong Kong, China Central South University, Changsha, 410017, China Tiangong University, Tianjin, 300387, China {rena.gao,carsten}@unimelb.edu.au, {wfyift,jeyhan.lau}@gmail.com wujingxuan@cuhk.edu.cn, wujing@tiangong.edu.cn, lvlongcsu.edu.en

## 1 Introduction

In the context of second language (SL) assessment, speaking has always been considered an essential ability Smith et al. (2022); Allwood (2008); Huang et al. (2020), but prior studies have predominantly focused on written correction Paiva et al. (2022) or pronunciation from ASR McGhee et al. (2024). Given the lack of researches that capture the unique linguistic features of SL speakers in dialogues (especially in open-domain interactive conversations), this leaves conversational interaction assessment under-explored and ultimately contributes to a limited understanding of conversational fluency and interactivity for SL speakers from different languages.

An exception is Gao et al. (2025), who propose a two-level framework for assessing the interactivity ability of English-as-a-second-language (ESL) speakers in open-domain conversations. The framework introduces micro-level word/utterance features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management) and they find that the micro-level features are highly predictive of the macro-level labels. However, they conduct the study only in the context of ESL, raising questions about the transferability of the evaluation framework to other languages besides English. Furthermore, they do not introduce an automated approach for this evaluation, as the analysis relies on human annotations.

Our work aims to address these shortcomings by: (1) testing the cross-lingual transferability of the evaluation framework by validating it on an annotated Chinese-as-a-second-language (CSL) dialogue dataset; and (2) introducing an automated, interpretable approach with low data requirement that scores the overall quality of a second-language dialogue. Our automated approach is interpretable as it highlights the key features that lead to the overall score, and it has a low data requirement because it does not require labelled data. As it can be adapted to other languages easily, our method paves the way for a new tool for assessing SL conversations.1 To summarise our contributions:

Footnote 1: The dataset and related codes can be accessed at [https://github.com/RenGao/CSL2024](https://github.com/RenGao/CSL2024)

* We release CNIMA (**C**hinese **N**on-Native **I**nteractivity **M**easurement and **A**utomation), an annotated CSL dialogue dataset with 10K dialogues, based on the evaluation framework of Gao et al. (2025) originally designed for ESL dialogues.
* We evaluate the cross-lingual transferability of the evaluation framework and find it to be ro

bust across different languages. We further reveal language-specific and language-universal relationships between micro-level and macro-level features, highlighting the subtle differences between CSL and ESL dialogues.
* We introduce an automated, interpretable approach that predicts the micro- and macro-level features and the overall quality scores of second-language dialogues. Our best method demonstrates strong performance, creating a new tool for second language assessment. Importantly, the method can be adapted to other languages as it does not require labelled data.

## 2 Related work

### Assessment of SL Spoken Conversation

Mainstream second language assessments in current industrial practice have mainly focused on grammatical accuracy, pronunciation standardization and vocabulary richness; for example, in TOEFL iBT, PTE Academic and Cambridge IELTS (Xu, 2018; Paiva et al., 2022; Xu et al., 2021). However, few speaking assessments emphasised the importance of interaction in dialogues for second language speakers and learners (Khabazbashi et al., 2021) and aspects such as how speakers manage the topics in communication (Shaxobiddin, 2024), how speakers perform social roles from speaking interactions (Chen et al., 2023), and how speakers start and end a talk in an acceptable manner (Yap and Sahoo, 2024). There are, however, some exceptions. For example, Dai (2022) develop a test rubric for Chinese Second Language speakers. More recently, Gao and Wang (2024) introduce an interactivity scoring framework inspired by the IELTS speaking assessment. However, these studies only provide a theoretical framework for evaluation and lack an automated pipeline for large-scale assessments, human rating is still a major scoring approach in mainstream industrial practice.

### Automated Scoring System in SL

Automated scoring systems can improve the efficiency of processing and scoring dialogues compared to manual assessment methods, allowing for real-time feedback (Evanini et al., 2017) and explainable suggestions. Some major second language assessment organisations have employed automated speaking scoring, including PTE Pearson Jones and Liu (2023), Duolingo English Test (Burstein et al., 2021), and TOEFL (Gong, 2023), but these automated systems failed to capture the **interactive nature** of SL conversations. That is, they struggle to offer detailed analyses and insights on the common errors and language usage patterns and where SL speakers struggle the most. This motivates a more explainable framework for SL assessment that can provide better feedback and suggestions from the automated scoring.

The main challenge is to design an effective automated evaluation tool. Capturing the nuances of spoken language, such as interaction, attitudes, and cooperation in conversation, is challenging to achieve (Gao and Wang, 2024). Automated scoring systems must handle diverse patterns across varied languages, which can vary widely among different non-native SL speakers across distinct languages, like Asian languages and European languages with distinct differences. Assessing interactive features like turn-taking, interruptions, and response appropriateness further complicates the process. Ultimately, the dynamic nature of conversations makes it challenging for automated systems to accurately evaluate interactive speaking, necessitating sophisticated algorithms and continuous refinement (Engwall et al., 2022; Cumbal, 2024).

## 3 Evaluation Framework

We extend the evaluation framework of Gao et al. (2025) for assessing second-language dialogues, which considers micro-level features and macro-level interactivity labels. One addition we have made is to introduce an overall dialogue quality score. Figure 1 shows an example of CSL dialogue annotation with the evaluation framework. The framework has 17 micro-level features, and it can be further broken into token-level (such as 'Reference Word': _she_ and 'Backchannel': _hmm_) and utterance-level features (such as 'Formulaic response': _How's going_). The micro-level features are annotated as spans (i.e. annotators mark text spans corresponding to a feature). For macro-level interactivity labels, there are four: Topic Management, Tone Choice Appropriateness, Conversation Opening and Conversation Closing, and they are annotated as (dialogue) labels. Briefly, Topic Management refers to the speaker's ability to control and navigate the flow of topics; Tone Choice the suitability of the tone; and Conversation Opening/Closing the naturalness in the initial exchange and conclusion of the discussion. A more elaborate definition can be found in Appendix Table 13. For each 

macro-level interactivity label, the score ranges from 1 to 5 (categorical), and higher scores indicate a more natural and active interactivity quality. Note that for tone choice appropriateness, higher scores denote a casual tone, and lower scores indicate a formal tone.2 In addition to micro- and macro-level features, we introduce an _overall score_ for each dialogue to measure second language speakers' interaction abilities in open-domain conversation. Like the macro-level labels, it is scored from 1 to 5 (categorical). The overall score is designed to capture the holistic quality of a conversation, integrating elements like contextualisation, responsiveness, and communicative purposes across the whole conversation, and a high score reflects the speaker's ability to engage in fluid and meaningful interactions. The full description of each score is detailed in Appendix Table 12.

Footnote 2: The rationale of assigning higher scores to casual tone is that in our experiments, the conversations are designed to be informal discussions, and appropriate use of casual tone signifies a more natural communicative interaction.

## 4 CNIMA Development

### CSL Dialogue Collection

We first extend the CSL dialogue dataset developed by Wu and Roever (2021), which has approximately 8,000 dialogues. We followed a similar process, where we recruited 20 learners of Chinese Mandarin at four different proficiency levels.3 Each participant was assigned to a dyadic group with a partner of a similar proficiency level. Each pair of participants did one elicited conversation task, in which they discussed an assigned topic, and two role-play activities (three tasks in total; dialogue collection instructions can be found in Appendix A.4).4 After collecting the conversations, we recruited in-house workers to segment the conversations based on the discussion topics.5 Including the original 8,000 dialogues from Wu and Roever (2021), our extended CSL dataset has 10,908 dialogues in total as shown in Table 1.

Footnote 3: Proficiency level can be categorized into beginner, lower, intermediate, and high (Wu and Roever, 2021).

### Data Annotation

Given the dialogues, we annotate them for micro-level features, macro-level interactivity labels and overall quality scores based on the evaluation framework introduced in Section 3. To this end, we recruited twelve postgraduate students who are native Chinese speakers. We (first author) first trained the annotators and the annotator training manual can be found in Appendix A.3. During the training, annotators will see one dialogue as an example to understand the requirements and learn how to use the annotation platform (Appendix A.1). After training, each annotator was assigned 950 dialogues, and each dialogue was annotated by two annotators.6 The annotation was conducted over two months, where in the first few weeks we (first author) checked for initial agreement, discussed feedback, and fixed any annotation errors (e.g. missing values in the annotation results) before proceeding

\begin{table}
\begin{tabular}{c c} \hline \hline
**Statistic** & **Count** \\ \hline \#dialogues & 10,908 \\ \#turns (average) & 6 \\ \#turns (max) & 13 \\ \#tokens /w token-level features & 170,852 \\ \#tokens /w utterance-level features & 94,516 \\ \hline \hline \end{tabular}
\end{table}
Table 1: CNIMA Statistics.

Figure 1: An example of a CSL dialogue annotated with the micro-level features, macro-level interactivity labels and overall dialogue quality score.

 

with the full annotation. The annotation process was guided by an annotation guide (Appendix A.3), which provided definitions and examples for each micro-level feature, macro-level interactivity label and the overall dialogue quality score. For the overall score, in addition to providing the score, the annotators are also asked to write down their reasons for justifying their label.

As explained in Section 3, the micro-level features are annotated as spans, and the macro-level labels and overall quality scores as document (dialogue) labels. To aggregate the span annotations by the two annotators for the micro-level features, for each dialogue and each micro-level feature, we iterate through each turn and select the shorter (longer) span between the two annotators if it is a token-level (utterance-level) feature.7 For the macro-level interactivity labels, in dialogues where we have disagreement between the two annotators, we use the _majority label from their larger unsegmented conversation_ as the ground truth label.8 For the overall dialogue quality scores, for each dialogue with disagreement, we (two authors of this paper) manually assess the justification provided by the annotators to determine the ground truth.9

Footnote 7: We do this because token-level features (e.g., ‘Reference word’ (She, her, he) or ‘Tense Choice’ (is doing, done, did) tends to be very short and limited to 1 or 2 words and so the shorter spans are likely to be more accurate Greer (2023). The same reasoning applies to utterance-level spans.

Footnote 8: Recall that these dialogues are segmented from a larger, longer conversation. The majority label is the most frequent label when we pool together all the labels from the segmented, smaller dialogues that belong to the same original conversation. Our rationale for deferring to this majority label is that these are cases where it is difficult to determine the correct dialogue level label using the smaller segmented dialogue (this happens more with topic management than other interactivity labels), and so we look at all the labels across all the segmented dialogues collection that belong to the same conversation.

To measure the annotation quality, for the micro-level features, we calculate agreement between the annotators at the token level for each micro-level feature, i.e., we compute agreement statistics based on the presence or absence of the feature as marked by the annotators for each word token.10 We calculate Pearson correlation coefficient \(r\)Cohen et al. (2009) and Krippendorff's \(\alpha\)Krippendorff (2018) and summarise the results in Table 2. The agreement is above 0.6 over all the features/labels, indicating a good consensus between annotators.

Footnote 10: Note that when resolving the disagreement, we only consider the justification _without_ looking at their macro-level interactivity labels. This is so we create an unbiased ground truth for the overall score that is independent of the macro-level labels.

## 5 Automated Pipeline

We now propose automating the prediction of the micro-level features (step 1), macro-level interactivity labels (step 2), and overall score (step 3); see Figure 2 for an illustration. We experiment with classical machine learning models (Logistic Regression (LR), Random Forest (RF), and Naive Bayes (NB)), fine-tuned Chinese BERT Cui et al. (2021), and GPT-4o OpenAI (2024) with prompts to automate these steps. Note that these predictions are done in sequence, where the output from the previous step is used as input for the next step (e.g. when predicting the macro-level labels, we use the predicted micro-level features as input). We partition CNIMA into train, development and test sets (ratio = 7:1:2) for these experiments.

Step 1This step predicts the spans of the 17 micro-level features given an input dialogue, and we experiment with BERT and GPT-4o here. For BERT, we fine-tune 17 span classifiers (one for each feature) to classify the presence or absence of a feature for each word in the dialogue. For GPT-4o, we do one-shot prompting (i.e. 1 dialogue with the expected output as a demonstration) with

Figure 2: Pipeline for automated scoring of the CSL dialogue on three steps



instructions to generate spans for the token-level and utterance-level features separately.11

Footnote 11: In other words, we have 2 prompts for each dialogue.

Step 2This step predicts the macro-level interactivity labels for topic management, tone choice appropriateness, and conversation opening and closing. Here, we experiment with classical models (LR, RF and NB), BERT and GPT-4o. For each of them, we train four (one for each interactivity label) 5-class classifiers (each label has 5 classes). For the classical models, we follow Gao et al. (2025) and convert the micro-level feature spans into normalised counts and use them as input features to train LR/RF/NB to predict the interactivity labels. For BERT and GPT-4o, the input is the dialogue concatenated with the micro-level feature spans. BERT is fine-tuned as a document classifier, and GPT-4o is one-shot prompted with instructions.

Step 3This step predicts the overall dialogue quality score. As this is a document classification task like step 2, we follow the same process for building LR, RF, NB, BERT and GPT-4o. Note that the classical models (LR, RF, NB) use only the normalised macro-level interactivity score as input features (in other words, they have only 4 input features), while BERT and GPT-4o uses a concatenation of the dialogue, micro-level feature spans, and macro-level interactivity scores as input.

For more details on configurations and prompts, see A.5 for BERT and A.6 for GPT-4o.

## 6 Experiments

We first replicate the ESL experiments of Gao et al. (2025) for CSL and compare their results in Section 6.1. We then assess the performance of our proposed automated approach for SL assessment in Section 6.2.

### Transferability of the Evaluation Framework from ESL to CSL

For the _human-annotated_ micro-level features, Gao et al. (2025) convert them into normalised counts and use them as input features to train an LR, NB and RF classifiers to predict macro-level interactivity labels. They found: (1) strong prediction performance; (2) high-impact micro-level features that are _common across all_ interactivity labels (by interpreting feature importance given by the trained classifiers); and (3) high-impact micro-level features that are _specifically_ predictive for an interactivity label. In the following, we replicate the experiments using our annotated CSL data (CNIMA), to see whether the results transfer across languages; details can be found in Appendix A.10.

For (1), we present the F1 performance of predicting macro-level interactivity labels for CNIMA in Table 3. We see a strong performance, where F1 is over 0.8 in most models over the 4 interactivity labels. These results echo the ESL results in Gao et al. (2025),12 providing evidence that the evaluation framework is robust across languages.

Footnote 12: Gao et al. (2025) found marginally higher performance for conversion opening/closing (F1 scores are over 0.9) and lower performance for topic management and tone choice appropriateness (F1 scores are a little below 0.8).

For (2), we present the results in Table 4. We found that micro-level features such as 'Feedback in Next Turn' and 'Reference Word' are high-impact features for both ESL and CSL -- this underlines their fundamental role in impacting dialogue interactive dynamics, and it is language-universal. Interestingly, however, we also found some differences. For example, 'Noun & Verb Collocation' and 'Routinized Resources' are strong features only for CSL, and this might be because, in Chinese, these fixed terms of expressions are often used to show social closeness in open-domain conversations Roever and Dai (2021). 'Code Switching' and 'Tense Choice', on the other hand, are two strong features only for ESL. This is intuitive, as Chinese has no tenses while English lenses correlate with the social expression in communications Lam (2018). For the full comparison results, see Appendix Table 14.

For (3), Table 5 and 6 present the CSL and ESL results, respectively. We found that for topic management, 'Negotiation of Meaning' and 'Question-Based Responses' are high-impact micro-level features for both ESL and CSL, demonstrating that these are language-universal features important to

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Models** & **Topic** & **Tone** & **Opening** & **Closing** \\ \hline LR & 0.829 & 0.859 & 0.821 & 0.810 \\ RF & 0.831 & 0.816 & 0.858 & 0.852 \\ NB & 0.846 & 0.783 & 0.811 & 0.846 \\ \hline \hline \end{tabular}
\end{table}
Table 3: F1 performance of predicting the macro-level features using human-annotated micro-level features.

 

drive the flow of topics in conversation. For tone appropriateness, we generally see less commonality (exceptions: 'Feedback in Next Turn', and 'Routinized Resources' which appear in both languages), suggesting that languages tend to use different features for managing tones [21]. For conversation opening, we see some similarities (e.g. 'Question-Based Responses') and also divergences (e.g. 'Subordinate Clauses' and 'Adj/Adv. Expressing' for ESL and 'Epistemic Copulas' and 'Epistemic Modals' for CSL). One explanation is that English tends to use adjectives and adverbs to extend a topic (e.g. _Totally, I think..._), while Chinese prefers modals and copulas "\(\text{``}\text{``}\text{$\mathbb{N}^{+}$}\text{$\mathbb{N}^{+}$}\text{$\mathbb{ N}^{+}$}\text{$\mathbb{N}^{+}$}\text{$\mathbb{N}^{+}$}\text{$\mathbb{N}^{+}$}\text{$\mathbb{N}^{+}$}\text{$\mathbb{N}^{+}$}\text{$\mathbb{N}^{+ 

are less tolerant of noise.13

Footnote 13: Note that LR does not use the dialogue as input, so any noise in the intermediate features (micro-level or macro-level) will have a bigger impact.

Taking all these results together, we show that when it comes to assessing SL dialogue quality, it is important to take a pipeline approach to predict important intermediate features (i.e. micro- and macro-level) before predicting the overall quality score. This design also has another advantage: it is more interpretable as given an overall score, we can also look at the intermediate results to understand what and where went wrong, providing more meaningful feedback that can benefit both teachers and second language students. Lastly, the strong performance of GPT4 also has another important implication: we can adapt our SL assessment system to another language _without requiring_ large-scale manually-annotated data.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Topic & Tone & Opening & Closing \\ \hline  & \multicolumn{2}{c}{Logistic Regression} & \\ \hline Negotiation of Meaning* & Routinized Resources* & Epistemic Modals & Backchannels* \\ Subordinate Clauses* & Adj/Adv. Expressing & Formulaic Responses & Adj/Adv. Expressing \\ Noun\&Verb Collocation & Feedback in Next Turn* & Question-Based Responses* & Formulac Responses \\ Question-Based Responses & Formalicate Responses* & Subordinate Classes* & Collaborative Finishes* \\ Subordinate clauses* & Reference Word & Adj/Adv. Expressing* & Epistemic Copulas \\ \hline  & \multicolumn{2}{c}{Naive Bayes} & \\ \hline Non-factive Verb Phrase & **Routinized Resources*** & Adj/Adv. Expressing* & Adj/Adv. Expressing \\ Question-Based Responses & Feedback in Next Turn* & Routinized Resources & Epistemic Modals \\ Adj/Adv. Expressing & **Epistemic Copulas** & **Subordinate Clauses* & Backchannels* \\ Negotiation of Meaning* & Question-Based Responses & Epistemic Copulas & Collaborative Finishes* \\ Subordinate clauses* & Subordinate Clauses* & Question-Based Responses* & Question-Based Responses \\ \hline  & \multicolumn{2}{c}{Random Forest} & \\ \hline Negotiation of Meaning* & **Epistemic Copulas** & Feedback in Next Turn & Feedback in Next Turn \\ Formulae Responses & Backchannels & **Subordinate Clauses* & Subordinate clauses \\ Subordinate Clauses* & Feedback in Next Turn* & Adj/Adv. Expressing* & Collaborative Finishes* \\ Epistemic Copulas & Negotiation of Meaning & **Question-Based Responses*** & **Formulac Responses** \\ Question-Based Responses & **Routinized Resources*** & **Formulaic Responses** & **Backchannels* \\ \hline \hline \end{tabular}
\end{table}
Table 6: ESL: High impact interactivity-specific micro-level features (reproduced from Gao et al. (2024)). For each interactivity label, bold/asterisk indicates overlapping features in two/three classifiers.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Topic & Tone & Opening & Closing \\ \hline Negotiation of Meaning & Feedback in Next Turn & Epistemic Copulas & Backchannels \\ Epistemic Copulas & Collaborative finishes & Formulaic Responses & Collaborative Finishes* \\ Collaborative Finishes & Routinized Resources & Question-Based Responses* & Epistemic Copulas \\ Question-Based Responses & Formulationic Responses* & **Subordinate Clauses* \\ Subordinate clauses* & Reference Word & Adj/Adv. Expressing* & Epistemic Copulas \\ \hline  & \multicolumn{2}{c}{Naïve Bayes} & \\ \hline Non-factive Verb Phrase & Epistemic copulas & Code-switching & Code-switching \\ Collaborative Finishes & Collaborative finishes & Feedback in Next Turn & Epistemic Modals \\ Code–switching & Impersonal subject + non-factive verb & Epistemic Models & Epistemic Copulas \\ Backchannels* & Backchannels & Epistemic Copulas & Collaborative Finishes* \\ Epistemic Copulas & Question-Based Responses & Question-Based Responses* & Question-Based Responses \\ \hline  & \multicolumn{2}{c}{Random Forest} & \\ \hline Noun \& verb collocation & Tense choice & Feedback in Next Turn & Backchannels \\ Negotiation of Meaning & Backchannels & Noun \& Verb Collocation & Subordinate Clauses \\ Backgroundness* & Negotiation of Meaning & Collaborative Finishes & Formulaic responses \\ Question-based Responses & Feedback in Next Turn & Question-Based Responses* & Collaborative Finishes* \\ Collaborative Finishes & Routinized Resources & Formulaic Responses & Question-based Responses \\ \hline \hline \end{tabular}
\end{table}
Table 5: CSL: High impact interactivity-specific micro-level features. For each interactivity label, bold/asterisk indicates overlapping features in two/three classifiers.

 



### Additional Analyses

To understand how macro-level interactivity labels contribute to the overall quality score, we present their feature importance as learned by classical machine learning models in the "Human+LR+LR", "Human+RF+RF" and "Human+NB+NB" models in Table 8. Topic management appears to be the most important predictor, followed by tone appropriateness. Conversation opening and closing, on the other hand, has relatively small weights. These observations are supported by Roever and Ikeda (2023), who similarly found that topic flow and tone are dominant in deciding interaction quality.

Next, we assess how well BERT and GPT4 perform for the first (micro-level feature span prediction) and the second-step predictions (macro-level label classification); full results in AppendixA.5 Table14 and 15 for BERT and GPT4 respectively. To summarise, for most of the 17 micro-level features (step 1), both BERT and GPT4 perform very well (average F1 for BERT and GPT4 is 0.89 and 0.81, respectively). That said, BERT and GPT4 struggle on two features related to non-factive verbs, and this can be attributed to the rarity of these features in the Chinese (Roever and Ikeda, 2023). Overall, these results show that the proposed micro-features transfer very well from English to Chinese.

For the second step, we look at the performance of "BERT+BERT" and "GPT4+GPT4" for predicting the macro-level labels (Appendix Table 16). Across all labels, both BERT and GPT4 perform well, averaging about 0.83 F1 for BERT and 0.78 F1 for GPT4, respectively. Together, these encouraging intermediate evaluation results explain why our pipeline approach can consistently score the overall dialogue quality.

## 7 Conclusion

We introduce CNIMA, a large-scale annotated dataset for Chinese-as-a-Second-Language (CSL) dialogues based on the evaluation framework introduced by Gao et al. (2025). This allows us to test the cross-lingual transferability of the framework, as it is originally designed for English. Our experiments demonstrate that the framework works well for CSL dialogues, suggesting that it is robust across languages and that it is a viable framework for evaluating any second-language conversations. Our analyses also reveal language-specific and language-universal relationships between micro-level linguistic features (such as backchannels) and macro-level interactivity features (such as topic management). Next, we propose an automated, interpretable approach that predicts the micro- and macro-level features and the overall quality of second-language dialogues. To this end, we explore a range of models, from classical machine learning models to LLMs. Our approach is interpretable because it does not only predict the overall dialogue quality score, but also reveals the key micro-level and macro-level features that contribute to the score. Our LLM-based approach performs very well, and as it does not require any labelled data for training, it can be adapted to other languages easily. Ultimately, our method paves the way for an interpretable and practical system for automatic second-language dialogue assessment.

## 8 Limitations

The original speech data is transcribed manually into text. Technically we can use speech-to-text software to automate this, but since we are working with second language conversations, this can result in much noise in the translated text. The scope of the current paper is limited to CSL, but it would be equally interesting to see how the evaluation framework would work for other SL dialogues, which calls on a joint contribution of the community effort on more SL language dialogue datasets, like Japanese second language datasets, Spanish second language datasets and etc.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Models** & **F1** \\ \hline BERT (One-step) & 0.379 \\ GPT4 (One-step) & 0.585 \\ \hline Human+LR+LR & 0.772 \\ Human+BERT+BERT & 0.860 \\ Human+GPT4+GPT4 & 0.842 \\ \hline BERT+LR+LR & 0.329 \\ BERT+BERT+BERT & 0.807 \\ \hline GPT4+LR+LR & 0.667 \\ GPT4+GPT4+GPT4 & 0.791 \\ \hline \hline \end{tabular}
\end{table}
Table 7: F1 performance of the dialogue overall score for the CSL dialogue in three steps in different models

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Models** & **Topic** & **Tone** & **Opening** & **Closing** \\ \hline LR & 0.643 & 0.357 & 0.081 & 0.079 \\ RF & 0.463 & 0.235 & 0.066 & 0.090 \\ NB & 0.560 & 0.290 & 0.106 & 0.117 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Feature Importance of the four interactivity labels across three machine learning models



## Ethics Statement

This study is conducted under the guidance of the ACL Code of Ethics. We manually filtered out potentially offensive content and removed all information related to the identification of annotators.

## References

* Alduais et al. (2022) Ahmed Alduais, Issa Al-Qaderi, and Hind Alfadda. 2022. Pragmatic language development: Analysis of mapping knowledge domains on how infants and children become pragmatically competent. _Children_, 9(9):1407.
* Allwood (2008) Jens Allwood. 2008. Dimensions of embodied communication-towards a typology of embodied communication. _Embodied communication in humans and machines_, pages 257-284.
* Burstein et al. (2021) Jill Burstein, Geoffrey T LaFlair, Antony John Kunnan, and Alina A von Davier. 2021. A theoretical assessment ecosystem for a digital-first assessment--the duolingo english test. _DRR-21-04_.
* Chen et al. (2023) Ming-Bin Chen, Jey Han Lau, and Lea Frermann. 2023. The uncivil empathy: Investigating the relation between empathy and toxicity in online mental health support forums. In _Proceedings of the 21st Annual Workshop of the Australasian Language Technology Association_, pages 136-147.
* Cohen et al. (2009) Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. _Noise reduction in speech processing_, pages 1-4.
* Cui et al. (2021) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 29:3504-3514.
* Cumbal (2024) Ronald Cumbal. 2024. _Robots Beyond Borders: The Role of Social Robots in Spoken Second Language Practice_. Ph.D. thesis, KTH Royal Institute of Technology.
* Dai (2022) David Wei Dai. 2022. _Design and validation of an L2-Chinese interactional competence test_. Ph.D. thesis, University of Melbourne (Australia).
* Engwall et al. (2022) Olov Engwall, Ronald Cumbal, Jose Lopes, Mikael Ljung, and Linnea Mansson. 2022. Identification of low-engaged learners in robot-led second language conversations with adults. _ACM Transactions on Human-Robot Interaction (THRI)_, 11(2):1-33.
* Evainti et al. (2017) Keelan Evainti, Maurice Cogan Hauck, and Kenji Hakuta. 2017. Approaches to automated scoring of speaking for k-12 english language proficiency assessments. _ETS Research Report Series_, 2017(1):1-11.
* Finch et al. (2023) Sarah E. Finch, James D. Finch, and Jinho D. Choi. 2023. Don't forget your ABC's: Evaluating the state-of-the-art in chat-oriented dialogue systems. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15044-15071, Toronto, Canada. Association for Computational Linguistics.
* Gao et al. (2024) Rena Gao, Carsten Roever, and Jey Han Lau. 2024. Interaction matters: An evaluation framework for interactive dialogue assessment on english second language conversations.
* Gao et al. (2025) Rena Gao, Carsten Roever, and Jey Han Lau. 2025. Interaction matters: An evaluation framework for interactive dialogue assessment on English second language conversations. In _Proceedings of the 31st International Conference on Computational Linguistics_, pages 10977-11012, Abu Dhabi, UAE. Association for Computational Linguistics.
* Gao and Wang (2024) Wei Gao and Menghan Wang. 2024. Listenership always matters: active listening ability in l2 business english paired speaking tasks. _International Review of Applied Linguistics in Language Teaching_.
* Gong (2023) Kaixuan Gong. 2023. Challenges and opportunities for spoken english learning and instruction brought by automated speech scoring in large-scale speaking tests: a mixed-method investigation into the washback of speechrater in toef ibt. _Asian-Pacific Journal of Second and Foreign Language Education_, 8(1):25.
* Greer (2023) Tim Greer. 2023. Grammar-in-interaction and its place in assessing interactional competence. _Applied Programatics_, 5(2).
* Huang et al. (2020) Lishan Huang, Zheng Ye, Junghui Qin, Liang Lin, and Xiaodan Liang. 2020. GRADE: Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9230-9240, Online. Association for Computational Linguistics.
* Jones and Liu (2023) Jeremy F Jones and Quanling Liu. 2023. Analyzing test-akers' experiences of high-stakes automated language testing. _English as a Foreign Language International Journal_, 3(1):1-41.
* Khabbazbashi et al. (2021) Nahal Khabbazbashi, Jing Xu, and Evelina D Galaczi. 2021. Opening the black box: Exploring automated speaking evaluation. _Challenges in Language Testing Around the World: Insights for language test users_, pages 333-343.
* Koppendorff (2018) Klaus Koppendorff. 2018. _Content analysis: An introduction to its methodology_. Sage publications.
* Lam (2018) Daniel MK Lam. 2018. What counts as "responding"? contingency on previous speaker contribution as a feature of interactional competence. _Language Testing_, 35(3):377-401.
* Lam (2021) Daniel MK Lam. 2021. Don't turn a deaf ear: A case for assessing interactive listening. _Applied Linguistics_, 42(4):740-764

Charles McGhee, Katherine Knill, and Mark Gales. 2024. Highly intelligible speaker-independent articulatory synthesis.
* OpenAI (2024)OpenAI. 2024. Hello gpt-4o.
* Paiva et al. (2022) Jose Carlos Paiva, Jose Paulo Leal, and Alvaro Figueira. 2022. Automated assessment in computer science education: A state-of-the-art review. _ACM Transactions on Computing Education (TOCE)_, 22(3):1-40.
* Roever and Dai (2021) Carsten Roever and David W Dai. 2021. Reconcceptualizing interactional competence for language testing. _Assessing speaking in context: Expanding the construct and its applications_, pages 23-49.
* Roever and Ikeda (2023) Carsten Roever and Naoki Ikeda. 2023. The relationship between l2 interactional competence and proficiency. _Applied Linguistics_, page amad053.
* Shaxobiddin (2024) Abdullayev Shaxobiddin. 2024. A discourse analysis of modal verbs in modern english: Patterns and functions. _Journal of new century innovations_, 50(2):145-147.
* Smith et al. (2022) Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents. In _Proceedings of the 4th Workshop on NLP for Conversational AI_, pages 77-97, Dublin, Ireland. Association for Computational Linguistics.
* Wu and Roever (2021) Jingxuan Wu and Carsten Roever. 2021. Proficiency and preference organization in second language mandarin chinese refusals. _The Modern Language Journal_, 105(4):897-918.
* Xu (2018) Jing Xu. 2018. Measuring "spoken collocational competence" in communicative speaking assessment. _Language Assessment Quarterly_, 15(3):255-272.
* Xu et al. (2021) Jing Xu, Edmund Jones, Victoria Laxton, and Evelina Galaczi. 2021. Assessing l2 english speaking using automated scoring technology: examining automarker reliability. _Assessment in Education: Principles, Policy & Practice_, 28(4):411-436.
* Yang et al. (2024) Bohao Yang, Kun Zhao, Chen Tang, Liang Zhan, and Chenghua Lin. 2024. Structured information matters: Incorporating abstract meaning representation into llms for improved open-domain dialogue evaluation. _arXiv preprint arXiv:2404.01129_.
* Pap and Sahoo (2024) Foong Ha Pap and Anindita Sahoo. 2024. Versatile copulas and their stance-marking uses in conversational odia, an indo-aryan language. _Lingua_, 297:103641.
* Zhao et al. (2022) Jianqiao Zhao, Yanyang Li, Wanyu Du, Yangfeng Ji, Dong Yu, Michael Lyu, and Liwei Wang. 2022. FlowEval: A consensus-based dialogue evaluation framework using segment act flows. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 10469-10483, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zilio et al. (2017) Leonardo Zilio, Rodrigo Wilkens, and Cedrick Fairon. 2017. Using nlp for enhancing second language acquisition. In _RANLP_, pages 839-846.

 

Appendix

### Software Availability

To contribute to the research community and facilitate further development and collaboration, we have made the source codes of our innovative annotation tool publicly available. The tool, designed with a focus on enhancing the efficiency and accuracy of data annotation processes, has been developed through meticulous research and development efforts. It incorporates a range of features tailored to meet the needs of researchers and practitioners working in fields that require precise and reliable annotation of datasets.

#### Accessing the Source Code

The source codes are hosted on GitHub, a platform widely recognized for its robust version control and collaborative features. Interested parties can access the repository at the following link: [https://anonymous.4open.science/r/AnnotationTool2023-CFE1/README.md](https://anonymous.4open.science/r/AnnotationTool2023-CFE1/README.md). This repository is intended for research usage, underlining our commitment to supporting academic and scientific endeavours.

#### Key Features and Capabilities

Our annotation tool stands out for its user-friendly interface, which simplifies the annotation process and allows users to work more efficiently. Among its key features are:

* **Customizable Annotation Labels:** Users can add their own set of labels to cater to the specific requirements of their projects.
* **Collaborative Annotation Support:** Facilitating teamwork, the tool allows multiple annotators to work on the same dataset simultaneously, ensuring consistency and reducing the time required for project completion.
* **Annotation History Tracking:** All the annotation history, such as changes made, can be tracked, and any further modifications can be done at any time for the user's convenience. Export Functionality: Annotated data can be exported in several formats, accommodating further analysis or use in machine learning models.

### Pages View For Span Annotation Website Interface

Figure 3: Annotation tool Demo





* [19]A. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [20]R. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [21]S. K. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [22]D. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [23]P. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [24]T. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [25]K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [26]C. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [27]G. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [28]H. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [29]L. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.
* [30]E. K. S. A. G. R. H. C. W. and D. E. T. L. A. N. V. and P. F. A. Z. (2004) The role of social networks in social networks. In: Proceedings of the 3rd International Conference on Social Information Theory, pp. 1-1. Cited by: SS1.

[MISSING_PAGE_POST]

 



### Manual

#### 1 Introduction to the annotation task

The research aims to investigate the interactive ability of second-language speakers of Chinese through automated dialogue evaluation. This study has been approved by **Human Ethics**.

In your annotation, two types of dialogue tasks would be included in this study conducted by a pair-wise discussion by second-language speakers of Chinese participants. The first task is an elicited conversation task, in this part, two speakers will share some experience or what they want to deliver based on the instructions (e.g., _share some ideas on how you think of education in your life_). In the second task, two speakers need to role-play through a joint discussion.

The dialogue of the two tasks was both transcribed into text, and you are ready to annotate based on the text. Please notify the researcher if you find any misaligned information in the transcriptions compared with the original recordings during your annotation.


* [18] M. C. Bader, R. J. Brouwer, and D. A. Schiller, _Theoretical Computer Science_**206**(4):397-503, 2000.
* [19] S. F. Gudmund, _A Practical Guide to the Theory of Computing_, volume 1. Springer Verlag, New York, 1997.
* [20] T. H. Kienienien, _Computational Methods in Applied Mathematics_**199**(1):1-10, 1997.
* [21] P. L. Lions, _On the theory of computing_, volume 1. Springer Verlag, New York, 1997.
* [22] V. I. Arnold, _An Introduction to the Theory of Computing_, volume 1. Springer Verlag, New York, 1997.

[MISSING_PAGE_POST]

 

* [10] M. C. Dickson, J. R. K. Ellis and A. S. Thorne, Phys. Rev. Lett. **78**, 2396 (1995);
* [11] E. G. Berger, Nucl. Instrum. Meth. **A465**, 1 (1996).
* [12] P. B. Zerwas, W. H. Freeman and T. M. Weber, Phys. Rev. **D48**, 1 (1993).
* [13] F. Abe et al., Phys. Rev. Lett. **78**, 1189 (1993).

[MISSING_PAGE_POST]

 

* [CC1] CC1: detailed summarization and
* [CC3] Cosine of the conversation.
* [CC4] Cosine of the conversation.

 



## 3 Label detailed definitions

\begin{tabular}{|l|p{142.3pt}|p{142.3pt}|} \hline  & CC2: transit to the closing naturally, but without any summarization of the discussion. \\  & CC3: demonstrate a translation to the end of the conversation. \\  & CC4: transit to the end of the discussion. \\  & CC5: no closing, just stop the conversation. \\ \cline{2-3} overall tone choice: very formal & [OT1] & \\ \cline{2-3} overall tone choice: quite formal and some expressions are not that formal & [OT2] & \\ \cline{2-3} overall tone choice: relatively not formal, most expressions are quite informal & [OT3] & \\ \cline{2-3} overall tone choice: quite informal, but some expressions are still formal & [OT4] & \\ \cline{2-3} overall tone choice: very informal & [OT5] & \\ \cline{2-3} \end{tabular}


* [10] A. K. M. C. R. Wilson, and J. E. W. York, "Theoretical status of the theory of quantum gravity", _Phys. Rev. D_**48**:635-729 (1993)
* [11] S. Weinberg, "A new approach to quantum gravity", _Nucl. Phys. B_**108**:1-13 (1974)
* [12] G. F. Giudice, "Quantum Gravity", _Commun. Math. Phys._**10**:1-18 (1975)
* [13] L. Randall and T. Sundrum, "An introduction to string theory", _JHEP_**04**:043 (2004) 007, hep-th/0405166.
* [14] P. Fayet, "On the hierarchy problem in string theory", _Phys. Lett. B_**100**:1-13 (1975)
* [15] H. Georgi and N. Sakai, "Supersymmetry breaking in superstring theories", _Prog. Theor. Phys. Suppl._**10**:1-13 (1976)
* [16] I. Antoniadis, "Supergravity duality", _Phys. Rept._**100**:1-1 (1976)

[MISSING_PAGE_POST]

 

* [108] M. C. B. K., and J. A. R. Laird, "The role of the language in a database for linguistic data analysis," in _Proceedings of the 27th International Conference on Language Science_, pages 3-46, 2005.
* [109] D. P. Kingma and S. Welling, "Learning to align with neural networks," in _Advances in Neural Information Processing Systems_, volume 1, pp. 139-146, 2014.
* [110] G. Hinton, I. Goodfellow, Y. Bengio, and J. Dean, "Deep residual learning: A deep neural network approach," in _NIPS'09 conference on artificial intelligence_, pages 1398-1405, 2009.

[MISSING_PAGE_POST]

 

* [10] M. C. B. K., and J. A. R. Laird, "Theoretical evaluation of the statistical significance of a model for the determination of the number of parameters in the data," _Journal of Statistical Planning Research_**26**(4):357-389, 1996.
* [11] D. P. Kingma and S. Ba, "Adam: A method for stochastic optimization," in _Proceedings of the Thirtieth International Conference on Machine Learning_, pp. 119-120, 2014.
* [12] T. G. LeCun, Y. Bengio, and I. Sutskever, "A new approach to machine learning," _arXiv preprint arXiv:1512.03085_**, 2015.
* [13] H. Li, X. Zhang, Z. Liu, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Advances in Neural Information Processing Systems_, vol. 10, no. 1, p. 103002, 2015.
* [14] W. Li, X. Zhang, and Q. Sun, "Learning from multiple layers by using convolutional neural network," in _Proceedings of the Thirtieth international conference on computer vision and pattern recognition_, pp. 1-10, 2015.
* [15] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [16] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [17] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [18] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [19] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [20] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [21] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [22] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [23] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [24] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [25] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [26] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [27] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [28] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [29] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [30] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [31] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [32] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [33] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [34] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [35] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [36] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [37] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [38] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [39] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [40] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [41] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [42] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [43] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [44] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [45] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [46] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [47] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [48] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [49] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [50] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [51] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [52] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [53] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [54] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [55] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [56] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [57] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [58] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [59] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [60] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.
* [61] W. Li, X. Zhang, and Q. Sun, "Deep residual learning with deep convolutional neural networks," in _Proceedings of the Thirtieth International Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2015.

 

are groups of words that contain a subject and a verb but do not express a complete thought and therefore cannot stand alone as a sentence. They function within a sentence by providing additional information to the main clause, to which they are connected by subordinating conjunctions (such as "because," "although," "when," "if") or relative pronouns (such as "who," "which," "that"). Subordinate clauses serve various roles in sentences, including acting as adjectives, adverbs, or nouns, and are essential for adding complexity, detail, and nuance to communication. Their use enables speakers and writers to articulate relationships of cause and effect, contrast, condition, time, and more, enriching the expressiveness and depth of language.

 



### Speaking Task Collection Instruction

**Task One: Elicited Conversation**

_Elicited Conversation Task (ECVA)_

_Instruction: . In this section, you will be given a topic. Based on the topic, I would like you to talk together with your partner._

Nowadays, some people like traveling independently, while others like travels with a group of persons. Please discuss:

Do you like traveling individually or traveling with a group of persons? Why?

_(Note: Some ideas are provided below. You can use these ideas in your discussions if you want. If you have some new ideas, you can use your own ideas in the discussion. You do not need to cover all the ideas provided below. Please discuss at least 3 minutes. I will stop you if necessary.)_



**Role Play Task One (VARP1)**

_Role A_

_Note: Please read the following situation carefully and imagine yourself in the following scenario. If you have any questions, please feel free to ask._

You are enrolled in the Chinese culture course for this semester. One assessment task of this course is to write an essay in Chinese. However, you are not confident in your Chinese writing. Your friend's Chinese proficiency is very high. You and your friend live in the same apartment.

Now, you decide to go to your friend's room. How would you ask your friend to help you revise your essay?

_Note: Please read the following scenario and imagine yourself in the following situation. It is your responsibility to respond to the task in a natural and flexible way. Before your interlocutor is on the record, please do not accept/ refuse your friends request. If you have any questions, please feel free to ask._

You need to finish writing several papers this week, as the deadline for submitting the papers is next week. Therefore, you are very busy and do not have enough sleep hours. Your classmates know your Chinese proficiency is quite good. Now, your friend, who lives in the same apartment, rings youroorbell.

 



**Role Play Task Two (VARP2)**

_Rule A_

_Note: Please read the following situation carefully and imagine yourself in the following scenario. If you have any questions, please feel free to ask._

You want to register for HSK examination online. You need to make an online payment by using a Chinese credit card. However, you do not have a Chinese credit card. You want to borrow a credit card from your friend who lives in the same apartment.

Now, you decide to go to your friend's room. How would ask your friend in real life to borrow the credit card?

_Note: Please read the following scenario and imagine yourself in the following situation. It is your responsibility to respond to the task in a natural and flexible way. Before your interlocutor is on the record, please do not accept/ refuse your friends request. If you have any questions, please feel free to ask._

You have a credit card, so you can finish the online payment by using this card. You do not want to lend others your credit card, as you think telling the passwords to other persons is unsafe. Now, your friend, who lives in the same apartment, rings your doorbell.

 



### BERT Configuration

We give detailed experimental settings in the following table. All the experiments can run via a single NVIDIA GeForce RTX 4070 GPU within a reasonable time.

### Prompts for GPT-4o

The following Table 10 shows the prompts for annotating micro-level features with GPT-4o.

#### Prompts for GPT-4o Dialogue Overall Evaluation

The following Table 11 shows the prompts for dialogue overall quality score with GPT-4o.

#### Overall Score Description and Definitions

The following Table 12 shows the descriptions and definitions for dialogue's overall quality score.

#### Four Interactivity Aspects Definitions and Descriptions

Table 13 shows the descriptions and definitions for dialogue's overall quality score.

#### Top-k feature computation method

This is reproduced from Gao et al. (2024). Given that a trained LR, NB and RF classifier all provide weights to indicate the importance of each feature, for each classifier, we first compute _common_ micro-level features across the four interactivity labels:

\[f_{\mathrm{c}}= \operatorname*{top5}\bigl{(}\operatorname*{top10}(f_{\mathrm{topic }})\cap\operatorname*{top10}(f_{\mathrm{tone}})\] \[\cap\operatorname*{top10}(f_{\mathrm{opening}})\cap\operatorname* {top10}(f_{\mathrm{closing}})\bigr{)}\]

where \(\operatorname*{topk}\) is a function that returns the best \(k\) items given by their weights, \(f_{\mathrm{topic}}\) denote the set of micro-level features with their weights for predicting the topic management interactivity label.

For micro-level features that are specific to each of the marco-level interactivity aspects. To that end, for each classifier we compute interactivity-specific features, e.g., for topic management, as follows:

\[\operatorname*{top10}(f_{\mathrm{topic}})-f_{\mathrm{c}} \tag{1}\]




\begin{table}
\begin{tabular}{|p{142.3pt}|l|} \hline
**Span** & **Description and Example** \\ \hline Reference Word & A linguistic term used to avoid repetition and link different parts. \\  & **Example:[(")][(Where are [you] going?)** \\ \hline Noun \& Verb Collocation & Words or phrases that habitually occur together, forming a strong and natural linguistic association. \\  & **Example:[(")][(Can you [do me a favor]?)** \\ \hline Code-switching & The alternation between languages within a single conversation. \\  & **Example:[(")][(I think, well, this is right.) \\ \hline Negotiation of Meaning & Interactive process where speakers clarify and confirm understanding. \\  & **Example: SPK1:[(")][(SPK1: So you didn’t do it! SPK2: [Indeed], I haven’t learned at all.)** \\ \hline Tense Choice to Show Interactive Aims & Using verb tenses to fulfil specific communicative goals. \\  & **Example:[(")][(I can’t [right now] because I still have a lot of work.)** \\ \hline Routinized Resources & Prefabricated linguistic elements are used to manage dialogue interactions efficiently. \\  & **Example:[(")][(As you said)]** \\ \hline Subordinate Clauses & Clauses that provide additional information to the main clause. \\  & **Example:[(")][(I think)][(You are good at math, [but] because I need to work hard myself.)** \\ \hline Backchannels & Brief responses from a listener, such as “uh-huh”. \\  & **Example:[(")][(Yep, yet, sep.)]** \\ \hline Question Responses & Replies in a dialogue that directly or answer a preceding question. \\  & **Example:[(")][(SPK2: [I shall teach myself.])** \\ \hline Formulaic Responses & Conventional phrases in dialogue to respond in familiar situations. \\  & **Example:[(")][(Roughly the same])** \\ \hline Collaborative Finishes & Instances in a dialogue where one speaker completes another speaker’s sentence or thought. \\  & **Example: SPK1:[(")][(SPK1: [Alright, See you_] SPK2:[Goodbye.])** \\ \hline Epistemic Copulas & Phrases that express a speaker's degree of certainty about a statement, often using verbs like ”is” or "seems". \\  & **Example:[(")][(II felt wired to go there alone)]** \\ \hline Epistemic Modals & Modal verbs or phrases that express a speaker's judgment about the possibility, such as “might,” ‘must,”. \\  & **Example:[(")][(You should learn these Chinese by yourself.)** \\ \hline Adjectives \& adverbs of possibility & Adjectives or adverbs to show possibility, like ”Possibly”. \\  & **Example:[(")][(II maybe go back to hometown.])** \\ \hline Non-factive Verb Phase & Expressions that use verbs to convey statements without asserting them as true; verbs “think,” *believe,” or “sem.” \\  & **Example:[(")][(I can barely follow the progress.])** \\ \hline Impersonal Subject & An impersonal subject (such as “it” or “there”) is followed by a non-factive verb and a noun phrase, often express opinions. \\  & **Example:[(")][(It’s hard to tell.)]** \\ \hline Feedback in Next Turn & Using next turn to respond other speaker. \\  & **Example:[(")][(You words make sense.])** \\ \hline \end{tabular}
\end{table}
Table 10: LLM Annotation Prompts for CSL Dialogue Span annotation for Mirco-level features



\begin{table}
\begin{tabular}{l l} \hline
**Interactivity** & **Definition** \\
**Macro-level Features** & the strategies and techniques used \\ Topic Management & to control and navigate the flow of topics \\  & the suitability of the tone used in communication, \\ Tone Choice Appropriateness & ensuring it aligns with the context, audience, \\  & and purpose to convey the intended message \\ Conversation Opening & the initial interaction or exchange that begins a dialogue, \\  & often setting the tone and context for the dialogue \\  & the process of ending a dialogue or interaction, \\ Conversation Closing & which involves signaling the conclusion of the discussion, \\  & summarizing key points, and often expressing a farewell \\ \hline \end{tabular}
\end{table}
Table 13: Definitions of macro-level interactivity features, with higher score emphasising on natural, authentic interaction and active engagement in the dialogue

\begin{table}
\begin{tabular}{c p{284.5pt}} \hline
**Scores** & **Descriptions** \\ \hline
5 & Smooth and fluent daily communication, easy and pleasant through the whole chat \\
4 & Somewhat less fluent communication, but the communication purpose is achieved \\
3 & Slightly awkward communication in some places, such as not being able to understand the other person’s question \\
2 & Overall communication is not fluent and mostly awkward, but some parts can be mutually understood \\
1 & Unable to accurately achieve the communication purpose, awkward conversation, failed to talk throughout the conversation.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Response Type** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline Epistemic Copulas & 0.995 & 0.700 & 0.736 & 0.717 \\ Formulaic Responses & 0.951 & 0.559 & 0.579 & 0.669 \\ Question-based Responses & 0.972 & 0.500 & 0.426 & 0.460 \\ Non-factive Verb Phrase Structure & 0.999 & 0.000 & 0.000 & 0.000 \\ Impersonal Subject + Non-factive Verb + NP & 0.998 & 0.000 & 0.000 & 0.000 \\ Reference Word & 0.987 & 0.982 & 0.983 & 0.982 \\ Routinized Resources & 0.971 & 0.564 & 0.448 & 0.500 \\ Noun \& Verb Collocation in Proper Form & 0.972 & 0.953 & 0.900 & 0.826 \\ Collaborative Finishes & 0.988 & 0.565 & 0.406 & 0.472 \\ Tense Choice to Indicate Interactive Aims & 0.990 & 0.915 & 0.826 & 0.768 \\ Negotiation of Meaning & 0.971 & 0.560 & 0.462 & 0.506 \\ Code-switching for Communicative Purposes & 0.999 & 1.000 & 0.941 & 0.869 \\ Feedback in the Next Turn & 0.951 & 0.679 & 0.741 & 0.708 \\ Epistemic Modals & 0.995 & 0.973 & 0.770 & 0.860 \\ Backchannels & 0.975 & 0.697 & 0.588 & 0.638 \\ Subordinate Clauses & 0.964 & 0.863 & 0.831 & 0.847 \\ Adv \& Adj Expressing & 0.901 & 0.731 & 0.706 & 0.847 \\ \hline \hline \end{tabular}
\end{table}
Table 16: F1 performance of Marco-level four interactivity aspects’ score prediction across different model versions

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Response Type** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline Epistemic Copulas & 0.997 & 0.929 & 0.843 & 0.884 \\ Formulaic Responses & 0.976 & 0.875 & 0.781 & 0.825 \\ Question-based Responses & 0.986 & 0.834 & 0.591 & 0.892 \\ Non-factive Verb Phrase Structure & 0.999 & 0.000 & 0.000 & 0.000 \\ Ingersonal Subject + Non-factive Verb + NP & 0.999 & 0.900 & 0.243 & 0.384 \\ Reference Word & 0.990 & 0.985 & 0.989 & 0.987 \\ Routinized Resources & 0.983 & 0.783 & 0.706 & 0.742 \\ Noun \& Verb Collocation in Proper Form & 0.985 & 0.970 & 0.958 & 0.964 \\ Collaborative Finishes & 0.995 & 0.802 & 0.631 & 0.706 \\ Tense Choice to Indicate Interactive Aims & 0.994 & 0.957 & 0.930 & 0.943 \\ Negotiation of Meaning & 0.991 & 0.849 & 0.713 & 0.775 \\ Code-switching for Communicative Purposes & 0.999 & 0.976 & 0.954 & 0.965 \\ Feedback in the Next Turn & 0.972 & 0.842 & 0.833 & 0.838 \\ Epistemic Modals & 0.997 & 0.909 & 0.945 & 0.926 \\ Backchannels & 0.982 & 0.824 & 0.731 & 0.875 \\ Subordinate Clauses & 0.990 & 0.960 & 0.937 & 0.948 \\ Adv \& Adj Expressing & 0.964 & 0.863 & 0.831 & 0.847 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Predicted performance of micro-level features on fine-tune BERT