An Effective Pronunciation Assessment Approach Leveraging Hierarchical Transformers and Pre-training Strategies (2024)

Abstract: Automatic pronunciation assessment (APA) manages to quantify a second language (L2) learner's pronunciation proficiency in a target language by providing fine-grained feedback with multiple pronunciation aspect scores at various linguistic levels. Most existing efforts on APA typically parallelize the modeling process, namely predicting multiple aspect scores across various linguistic levels simultaneously. This inevitably makes both the hierarchy of linguistic units and the relatedness among the pronunciation aspects sidelined. Recognizing such a limitation, we in this paper first introduce HierTFR1, a hierarchal APA method that jointly models the intrinsic structures of an utterance while considering the relatedness among the pronunciation aspects. We also propose a correlation-aware regularizer to strengthen the connection between the estimated scores and the human annotations. Furthermore, novel pre-training strategies tailored for different linguistic levels are put forward so as to facilitate better model initialization. An extensive set of empirical experiments conducted on the speechocean762 benchmark dataset suggest the feasibility and effectiveness of our approach in relation to several competitive baselines.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1737–1747 August 11-16, 2024 ©2024 Association for Computational Linguistics
Automatic pronunciation assessment (APA) manages to quantify a second language (L2) learner's pronunciation proficiency in a target language by providing fine-grained feedback with multiple pronunciation aspect scores at various linguistic levels. Most existing efforts on APA typically parallelize the modeling process, namely predicting multiple aspect scores across various linguistic levels simultaneously. This inevitably makes both the hierarchy of linguistic units and the relatedness among the pronunciation aspects sidelined. Recognizing such a limitation, we in this paper first introduce HierTFR1, a hierarchal APA method that jointly models the intrinsic structures of an utterance while considering the relatedness among the pronunciation aspects. We also propose a correlation-aware regularizer to strengthen the connection between the estimated scores and the human annotations. Furthermore, novel pre-training strategies tailored for different linguistic levels are put forward so as to facilitate better model initialization. An extensive set of empirical experiments conducted on the speechocean762 benchmark dataset suggest the feasibility and effectiveness of our approach in relation to several competitive baselines.

heading: 1 Introduction
text: With the rising trend of globalization, more and more people are willing or being demanded to learn foreign languages. This surging need calls for developing computer-assisted pronunciation training (CAPT) systems, as they can offer tailored and informative feedback for L2 (second-language)
* Corresponding author.
learners to practice pronunciation skills in a stressfree and self-directed learning manner (Eskenazi 2009; Evanini and Wang, 2013; Evanini et al., 2017; Rogerson-Revell, 2021). As a crucial ingredient of CAPT, automatic pronunciation assessment (APA) aims to evaluate the extent of L2 learners’ oral proficiency and then provide fine-grained feedback on specific pronunciation aspects in response to a target language (Bannò et al., 2022; Chen and Li, 2016; Kheir et al., 2023). A de-facto standard for APA systems is typically instantiated with a “reading-aloud” scenario, where an L2 learner is presented with a text prompt and instructed to pronounce it correctly. To offer in-depth feedback on learners’ pronunciation quality, recent efforts have drawn attention to the notion of multi-aspect and multi-granular pronunciation assessments, which normally devises a unified scoring model to
1 https://github.com/bicheng1225/HierTFR
1737
jointly evaluate pronunciation proficiency at various linguistic levels (i.e., phone-, word-, and utterance-levels) with diverse aspects (e.g., accuracy, fluency, and completeness), as the running example depicted in Figure 1. Methods along this line of research usually follow a parallel modeling paradigm, wherein the Transformer network and its variants serve as the backbone architecture to take as input a sequence of phonelevel pronunciation features and in turn predict multiple aspect scores across various linguistic levels simultaneously via a multi-task learning regime (Chao et al., 2022; Do et al., 2023a; Gong et al., 2022).
Albeit models stemming from the parallel modeling paradigm have demonstrated promising results on a few APA tasks, they still suffer from at least two weaknesses. First, the language hierarchy of an utterance is nearly sidelined, which, for example, assumes that all phones within a word are of equal importance and might insufficiently capture the word-level structural traits. Second, most of these methods largely overlook the relatedness among the pronunciation aspects. As an illustration, we visualize the correlation matrix in Figure 2, which shows the Pearson Correlation Coefficients (PCCs) between any pair of expert annotated aspect scores on the training set. We can observe that except for the aspects of utterancecompleteness and word-stress, the rest pronunciation aspects exhibit strong correlations not only within the same linguistic level but also across different linguistic levels2. Building on these observations, we in this paper present a novel language hierarchy-aware APA model, dubbed HierTFR, which leverages a hierarchical Transformer-based architecture to jointly model the intrinsic multi-level linguistic structures of an utterance while considering relatedness among aspects within and across different linguistic levels. To explicitly capture the relatedness within and across different linguistic levels, an aspect attention mechanism and a selective fusion module are introduced. The proposed model is further optimized with an effective correlation-aware regularizer, which encourages the correlations of predicted aspect scores to match those of their counterparts provided by human annotations. Furthermore, distinct pre-training strategies tailored for three linguistic levels are put forward,
2 Both the aspects of utterance completeness and word stress suffer from label imbalance problems, with more than 90%
so as to boost model initialization and hence reduce the reliance on large amounts of supervised training data. A comprehensive set of experimental results reveal that the proposed model achieves significant and consistent improvements over several strong baselines on the speechocean762 benchmark dataset (Zhang et al., 2021).
In summary, the main contributions of our work are at least three-fold: (1) we introduce HierTFR, a hierarchical neural model for APA, which is designed to hierarchically represent an L2 learner’s input utterance and effectively capture relatedness within and across different linguistic levels; (2) we propose a correlation-aware regularizer for model training, which encourages prediction scores to consider the relatedness among disparate aspects; and (3) extensive sets of experiments carried out on a public APA dataset confirm the utility of our proposed pre-training strategies, which considerably boosts the effectiveness of assessments across various linguistic levels.

heading: 2 Methodology

heading: 2.1 Problem Formulation
text: Given an input utterance U, consisting of a time sequence of audio signals X uttered by an L2 learner, and a reference text prompt T with 𝑀 words and 𝑁 phones, an APA model is trained to estimate the proficiency scores pertaining to multiple pronunciation aspects at various linguistic granularities. Let G = {𝑝,𝑤, 𝑢} be a set of linguistic granularities, where 𝑝,𝑤, 𝑢 stands for the phone-, word-, and utterance-level linguistic units,
of the assessments receiving the highest score (Do et al., 2023a).
respectively. For each linguistic unit 𝑔 ∈ 𝐺 , the APA model learns to predict a set of aspect scores A! = {𝑎" !, 𝑎# !, … , 𝑎$!
! }, where 𝑁! is the number of pronunciation aspects of the linguistic unit 𝑔.

heading: 2.2 Hierarchical Interactive Transformer Architecture
text: The overall architecture of our proposed APA model is schematically depicted in Figure 3, which consists of three ingredients: phone-level modeling, word-level modeling, and utterance pooling modules. After obtaining the representations of various pronunciation aspects, fully-connected neural layers is functioned as the regressors to collectively generate the corresponding aspect score sequence for an input utterance. Phone-level Modeling. For an input utterance U, various pronunciation features are extracted to portray the L2 learner’s pronunciation quality, which includes the goodness of pronunciation

heading: 3 Further details on pronunciation feature extractions can be
text: found in Appendix A.
(GOP)-based features E%&' , as well as prosodic features composed of duration E()* and energy E+,! statistics (Witt and Young, 2000; Hu et al., 2015; Zhu et al., 2022; Shen et al., 2021) 3. All these features are then concatenated and subsequently projected to from a sequence of acoustic features X-. In the meantime, the phone-level text prompt is mapped into an embedding sequence E- via a phone and position embedding layer and then point-wisely added to X- for enriching the phonetic information of X- . The resulting representations H-. are prepend with five trainable “[CLS]” embeddings and in turn fed into a phonelevel transformer to obtain the contextualized representations H- (Vaswani et al., 2017):
X! = W ∙ [E"#$; E%&'; E()*] + 𝐛, (1)
H!+ = X! + E!, (2)
H! = Transformer,-.6H!+7, (3)
where W and 𝐛 are learnable parameters. To assess a sequence of phone-level aspect scores, H(excluding the first 5 embeddings) is forward propagated to the corresponding regressors. The excluded embeddings H":0
- are expected to convey the holistic pronunciation information and are further fed into the subsequent selective fusion mechanism for use in utterance-level assessments.
Word-level Modeling. For the word-level assessments, a word-level attention pooling is used to produce a word representation vector from its corresponding phones, which can be implemented as a multi-head attention layer followed by an average operation. The word-level input representations H1. can be obtained by applying the word-level attention to the phone-level representations X- and H- individually, followed by a linear combination with the word-level textual embeddings E1 . Next, H1. is prepend with five trainable “[CLS]” embeddings and fed into a transformer to calculate the contextualized representations H1 at word-level:
X/ = Atten0123(X!), (4)
H=/ = Atten0123(H!), (5)
H/+ = X/ +H=/ + E/ , (6)
H/ = Transformer0123(H/+ ). (7) Note here that H1 (excluding the first 5 embeddings) is utilized in the word-level assessments while the excluded embeddings H":01 are fed into in subsequent selective fusion mechanism for use in the utterance-level assessments.
After that, an aspect attention mechanism is introduced to capture the relatedness among disparate aspects (Do et al., 2023b; Ridley et al., 2021). This mechanism consists of two sub-layers: a self-gating layer and a multi-head cross-attention layer. Specifically, for the 𝑗-th word-level aspect, the relation-aware representations H91*" are first derived from H1 via a self-gating layer which aims to abstract away from redundant information while considering the information gathered from other aspects. In addition, a multi-head cross-attention (MHCA) process alongside a masking strategy is employed to calculate aspect representations H1" from a collection of all relation-aware aspect representations C*2 = ;H91*# , … , H91*$%< . The following equations illustrate the operations of aspect attention:
H=/! = W4 ∙ H/ + 𝐛4 , (8)
H=/'! = σ@W*! ∙ C / + 𝐛*!B⨂ H= /! , (9)
H/! = MHCA6H=/'! , C'57, (10)
where H91" are aspect-specific representations, and C1 = [H91# , … , H91$%] includes all aspect-specific representations. In MHCA, H91*" is linearly projected to act as the query matrix, while C*2 is linearly projected to form the key and value matrixes. Additionally, the masking strategy ensures that the output representation at a specific position is only influenced by the other aspects of the word unit. Lastly, the aspect representations H1" are taken as input to the corresponding regressor to predict a score sequence for the 𝑗-th word-level pronunciation aspect. Utterance Pooling Module. For the utterancelevel assessments, utterance-level attention pooling is introduced to generate an utterance-level holistic representation from the corresponding input representations, which can be effectively implemented by attention pooling (Peng et al., 2022). In more detail, the utterance-level representation 𝐡) can be obtained by feeding the vector sequences X- , H- , and H1 into an utterance-level attention pooling module individually, followed by an aggregation operation:
?̅?&" = Atten677(X!), (11) ?̅?&# = Atten677(H!), (12) ?̅?&$ = Atten677(H/), (13)
𝐡& = W&6?̅?&" + ?̅?&# + ?̅?&$7 + 𝐛&, (14)
where W), 𝐛) are trainable parameters.
Next, a selective fusion mechanism is proposed to integrate contextualized representations across multiple linguistic levels for the utterance-level pronunciation assessments (Xu et al., 2021). Specifically, for the estimation of 𝑗-th utterancelevel aspect score, an aspect attention operation is first performed on 𝐡) to a produce intermediate representation 𝐡@ )". Note also that the gate values for the phone (𝑔- )"), word (𝑔1 )") and utterance (𝑔)
)") granularities are used to control the extent to which these contextualized representations can flow into the fused representation 𝐡)":
𝑔! &! = 𝜎 @𝐰!! ∙ J𝐡4 !; 𝐡4/; 𝐡K &!L + 𝑏!!B, (15) 𝑔/ &! = 𝜎 @𝐰/! ∙ J𝐡4 !; 𝐡4/; 𝐡K &!L + 𝑏/!B, (16) 𝑔& &! = 𝜎 @𝐰&! ∙ J𝐡4 !; 𝐡4/; 𝐡K &!L + 𝑏&!B, (17)
𝐡&! = 𝑔! &! ∙ 𝐡4 ! + 𝑔/ &! ∙ 𝐡4/ + 𝑔& &! ∙ 𝐡K &! , (18)
where 𝐡3 - and 𝐡31 are 𝑗-th representation vectors of H- and H1; and 𝐰-", 𝐰1", 𝐰)", 𝑏-", 𝑏1", and 𝑏)" are trainable parameters. The fused representation 𝐡)" is then passed to the corresponding regressor to assess the proficiency score for a given utterance-level aspect.

heading: 2.3 Optimization
text: Automatic Pronunciation Assessment Loss. The loss for multi-aspect and multi-granular pronunciation assessment, ℒ4'4, is calculated as a weighted sum of the mean square error (MSE) losses corresponding to different linguistic levels.
ℒ!"! = 𝜆# 𝑁# 9ℒ#!" $" + 𝜆% 𝑁% 9ℒ%!# $# + 𝜆& 𝑁& 9ℒ&!$ $$ , (19)
where ℒ-"& , ℒ1"% , and ℒ)"' are phone-level, word-level, and utterance-level losses for disparate aspects, respectively. The parameters 𝜆-, 𝜆1 , and 𝜆) are adjustable parameters which control the influence of different granularities, and 𝑁- , 𝑁1 , and 𝑁) mark the numbers of aspects at the phone-, word-, and utterance-levels, respectively. Correlation-aware Regularization Loss. The correlation-aware regularization loss is defined as the difference between the correlation matrix of the predicted aspect scores ∑9 and the correlation matrix of the corresponding target labels ∑:
ℒ89' = ℓ(∑=,∑), (20)
where ℓ is the regularization loss function, and each element in ∑9<3 (or ∑<3) is defined as a Pearson correlation coefficient between 𝑖 -th aspect score and 𝑗-th aspect score4. We adopt the MSE criterion for computing ℓ ; the overall loss thus can be expressed by:
ℒ = ℒ:$: + 𝜆ℒ89' , (21)
where 𝜆 ∈ [0, 1] is a tunable parameter, which is experimentally set to 0.01 based on the development set.

heading: 2.4 Pre-training Strategies
text: It is without doubt that a proper initialization is vital for the estimation of a neural model, due mainly to the highly nonconvex nature of the training loss function (Tamborrino et al., 2020;
4 To calculate PCC scores between aspects across different granularities, we duplicate the aspect scores of higher
Lakhotia et al., 2021). At lower linguistic levels, we leverage the mask-predict objective (Ghazvininejad et al., 2019) in the pre-training stage. To this end, we first mask a portion of input text prompt at phone- and word-levels. The corresponding Transformers are then tasked on recovering the masked tokens conditioning on the unmasked prompt sequence and the associated pronunciation representations (i.e., H-. , and H1. ). For the utterance level, we base the proposed pretraining strategy on predicting the relatively high or low accuracy scores for a pair of utterances. Namely, given any two utterances, the objective is to predict whether the former has a higher, lower, or the same accuracy score as the latter. Note here that, utterance pairs are randomly selected from a training batch, and this mechanism is employed to pretrain their utterance-level representations, denoted as 𝐡=)># ) , and 𝐡=)>( ) . Next, we feed the concatenation of these vector representations 𝐡=)>) = [𝐡=)># ) ; 𝐡=)>( ) ] into a three-way classifier, using the cross-entropy loss as the training objective.

heading: 3 Experimental Settings

heading: 3.1 Evaluation Dataset and Metrics
text: We conducted APA experiments on the speechocean762 dataset, which is a publicly available open-source dataset specifically designed for research on APA (Zhang et al., 2021). This dataset contains 5,000 English-speaking recordings spoken by 250 Mandarin L2 learners. The training and test sets are of equal size, and each of them has 2,500 utterances, where pronunciation proficiency scores were evaluated at multiple linguistic granularities with various pronunciation aspects. Each score was independently assigned by five experienced experts using the same rubrics, and the final score was determined by selecting the median value from the five scores. The evaluation metrics include Pearson Correlation Coefficient (PCC) and Mean Square Error (MSE). PCC is the primary evaluation metric, quantifying the linear correlation between predicted and ground-truth scores. A higher PCC score reflects a stronger correlation between the predictions and human annotations. In the following experiments, we report the MSE value in order to evaluate the
granularities to match the aspect scores at the lower granularities.
phoneme-level APA accuracy in comparison with prior arts.

heading: 3.2 Implementation Details
text: For the input feature extraction of the phone-level energy and the duration statistics, we follow the processing flow suggested by Zhu et al. (2022) and Shen et al. (2021), where a phone-level feature is constructed from time-aggregated frame-level features according to the forced alignment. Both the phone- and word-level Transformers for contextual representation modeling consist of 3 processing blocks utilizing multi-head attention with 3 heads and 24 hidden units, respectively. In addition, for the word- and utterance-level attention pooling, we use a single-layer multi-head attention with 3 heads and 24 hidden units. The combination weights used in Eq. (19) for the APA loss ( 𝜆- , 𝜆1 , 𝜆)) are assigned as (1, 1, 1) , respectively. To ensure the reliability of our experimental results, we repeated 5 independent trials, each of which consisted of 100 epochs with different random seeds. The test set results are reported by averaging those achieved by the top 100 best-performing models which are determined based on their PCC scores on the development set.

heading: 3.3 Compared Methods
text: We compare our proposed model (viz. HierTFR) with several families of top-of-the-line methods. Lin et al. (2021) and Kim et al. (2022) are singleaspect assessment models. The former develops a bottom-up hierarchical scorer evaluating the accuracy scores at the utterance level. The latter leverages self-supervised features (Baevski et al., 2020) to describe the learner’s pronunciation traits
and then separately models the corresponding utterance-level aspects with recurrent neural models. In addition, LSTM, GOPT (Gong et al., 2022; Ruy et al. 2023), and HiPAMA (Do et al., 2023b) are multi-aspect and multi-granular pronunciation assessments. First, LSTM and GOPT follow a parallel modeling regime, both of which treat the phone-level input features as a flattened sequence and assess higher level pronunciation scores through stacking LSTM layers or Transformer blocks. Second, Ruy et al. (2023) introduces a unified model architecture that jointly optimizes phone recognition and APA tasks. Lastly, HiPAMA is a hierarchical APA model that more resembles our model than the other methods compared in this paper. Different from our method, HiPAMA extracts high-level pronunciation features from low-level features based on a simple average pooling mechanism. Furthermore, the aspect attention mechanism used in HiPAMA performs on the logistics, whereas our model operates on the intermediate representations.

heading: 4 Experimental Results

heading: 4.1 Main Results
text: Table 1 reports the results on the speechocean762 dataset, which is divided into three parts: the first part shows the results of single-aspect assessment models, the second part presents the results of multi-aspect and multi-granular pronunciation methods, and the third part reports the results of our model. We further provide a comparison with another hierarchical APA model (viz. HiPAMA) in the third part.
First, a general observation is that our approach, HierTFR, excels in all assessment tasks, especially at the linguistic levels of utterance and word. This performance gain confirms that our model works comparably better for capturing the relationships between linguistic units than the other competitive methods. In terms of the utterance-level total score, the single-aspect assessment method (viz. Lin2021) largely falls behind the other multi-aspect and multi-granular pronunciation assessment models, which we attribute to the fact that the single-aspect assessment method is unable to harness the dependency relationships between aspects through the multi-task learning paradigm. By leveraging self-supervised learning features, Kim2022 achieves significant improvements over most APA methods in terms of the utterance-level assessments. Next, we scrutinize the performance of multi-aspect and multi-granular pronunciation assessment methods. Ruy2023 demonstrates significant advancements in the utterance-level fluency and prosody assessments due probably to the joint training of the APA model on the phone recognition task simultaneously. In comparison with the parallel modeling approaches (i.e., GOPT and LSTM), we can observe that HierTFR substantially improves the performance across all tasks, where its performance gains reveal the importance of capturing the hierarchical linguistic structures of an input utterance. Notably, compared to the HiPAMA, our model consistently achieves superior performance on a variety of pronunciation assessment tasks. This superiority stems from our tactfully designed selective fusion mechanism and the correlation-aware loss. The former allows our model to assess utterance-level aspect scores by leveraging information from diverse linguistic levels, while the latter explicitly models the relatedness among different aspects during the optimization.

heading: 4.2 Qualitative Analysis
text: Qualitative Visualization of Relatedness Among Aspects. In the second set of experiments, we examine the relatedness among disparate aspects at both word- and utterance-levels, where the
attention weights of the aspect attention mechanisms were determined based on the development set when assessing a specific aspect score. For the word-level assessments, the distributions of attention weights are in close accordance with the manual scoring rubrics of the speechocean762 dataset. In Figure 4(a), the total aspect serves as a comprehensive assessment and the corresponding weights are contributed from various pronunciation aspects. In contrast, the accuracy aspect measures the percentage of mispronounced phones within a word, leading to the attention weights being more concentrated on a word-level unit itself. Furthermore, the stress score also highly attends to the accuracy aspect, reflecting the strong relation between lexical stress and word-level pronunciation accuracy (Korzekwa et al., 2022). In regard to the relatedness within the utterance-level aspects, inspecting Figure 4(b) we find that the attention weights of the prosody and total aspects scatter across various pronunciation aspects, whereas the attention weights of the accuracy and completeness center primarily on the completeness aspect. One possible reason is that the prosody and total scores both measure highlevel oral skills, and when the human annotators judge the proficiency scores, they also take multiple pronunciation aspects into account simultaneously. Next, the completeness aspect measures the percentage of words with good pronunciation quality in an utterance. This implicitly reflects the intelligibility of a learner's pronunciation and is vital to the accuracy assessment. Qualitative Visualization of Interactions Across Linguistic Levels. In Figure 4(c), we report on the average gate values of utterances for three linguistic granularities by estimating the utterancelevel pronunciation aspect scores based on the development set. We can observe that the phonelevel representations bear high impacts on the utterance-level aspect assessments, in comparison to the other linguistic levels. Next, the word-level
and utterance-level representations exhibit minimal impact on the completeness and total aspects, respectively. One possible reason is that the completeness aspect somehow reflects pronunciation intelligibility, and our model learns to distill the information from the phone- and utterance-level representations. On the other hand, the total aspect evaluates an overall speaking skill. Our model thus tends to capture the subtle information by distilling the fine-grained traits inherent in the phone- and word-levels.

heading: 4.3 Ablation Study
text: To gain insight into the effectiveness of each model component of HierTFR, we conduct an ablation study to investigate their impacts. These variations include excluding the correlation-aware regularizer (w/o CorrLoss), removing the proposed pretraining strategies (w/o Pretrain), omitting the selective fusion mechanism (w/o SFusion), and eliminating the aspect attention mechanism at both word and utterance levels (w/o AspAtt). From Table 2, we can observe that the proposed correlation-aware regularization loss is beneficial for most pronunciation assessment tasks. Next, the proposed pre-training strategies are crucial to obtaining better performance as the model trained without them tends to perform relatively worse for all pronunciation assessment tasks. This highlights the efficacy of the pre-training strategies for hierarchical APA models, thereby alleviating the requirement for large amounts of supervised training data. Third, removing the selective fusion mechanism leads to degradations in the utterancelevel aspect assessments, while removing the aspect attention mechanism deteriorates the performance on word-level aspect assessments.

heading: 5 Related Work
text: Early studies on APA focused primarily on singleaspect assessments, typically through individually constructing scoring modules to predict a holistic
pronunciation proficiency score on a targeted linguistic level or some specific aspect with different sets of hand-crafted features, such as the phone-level posterior probability (Witt and Young, 2000), word-level lexical stress (Ferrer et al., 2015), or various utterance-level pronunciation aspects (Coutinho et al., 2016). More recently, with the rapid progress of deep learning (Vaswani et al., 2017; Raffel et al., 2020; Hsu et al., 2021), several neural scoring models have been successfully developed for multi-aspect and multi-granular pronunciation assessment. Gong et al. (2022) proposed a GOP feature-based Transformer (GOPT) architecture to model pronunciation aspects at multiple granularities with a multi-task learning scheme. Do et al. (2023b) employed a neural scorer with a hierarchical structure to mimic the language hierarchy of an utterance to deliver state-of-the-art performance for APA.

heading: 6 Conclusion
text: In this paper, we have put forward a novel hierarchical modeling method (dubbed HierTFR) for multi-aspect and multigranular APA. To explicitly capture the relatedness between pronunciation aspects, a correlation-aware regularizer loss has been devised. We have further developed model pre-training strategies for our HierTFR model. Extensive experimental results confirm the feasibility and effectiveness of the proposed method in relation to several top-of-theline methods. In future work, we plan to examine the proposed HierTFR model on open-response scenarios, where learners speak freely or respond to a given task or question (Wang et. al., 2018; Park and Choi, 2023). In addition, the issues of explainable pronunciation feedback are also left as a future extension.
Limitations Limited Applicability. In this research, the proposed model focus on the “reading-aloud” pronunciation training scenario, where the assumption is that the L2 learner pronounces a predetermined text prompt correctly, which restricts the applicability of our models to other learning scenarios, such as freely speaking or openended conversations. Lack of Accent Diversity. The used dataset merely contains Mandarin L2 learners, hindering the generalizability of the proposed model and could
be untenable when assessing the L2 learners with diverse accents. The lack of Interpretability. The model of the proposed method simply trains to mimic expert’s annotations without resorting to manual assessment rubrics or other external knowledge, making it not straightforward to provide reasonable explanations for the assessment results.
Ethics Statement We hereby acknowledge that all of the co-authors of this work compile with the provided ACL Code of Ethics and honor the code of conduct. Our experimental corpus, speechocean762, is widely used and publicly available. We think there are no potential risks for this work.

heading: Future Work
text: In future work, we plan to examine the proposed HierTFR model on open-response scenarios, where learners speak freely or respond to a given task or question (Wang et. al. , 2018; Park and Choi, 2023). In addition, the issues of explainable pronunciation feedback are also left as a future extension. Limitations Limited Applicability. In this research, the proposed model focus on the “reading-aloud” pronunciation training scenario, where the assumption is that the L2 learner pronounces a predetermined text prompt correctly, which restricts the applicability of our models to other learning scenarios, such as freely speaking or openended conversations. Lack of Accent Diversity. The used dataset merely contains Mandarin L2 learners, hindering the generalizability of the proposed model and could
be untenable when assessing the L2 learners with diverse accents. The lack of Interpretability. The model of the proposed method simply trains to mimic expert’s annotations without resorting to manual assessment rubrics or other external knowledge, making it not straightforward to provide reasonable explanations for the assessment results