SlowGuess/ACL_2020_OCR huggingface dataset:
Investigating the Effect of Auxiliary Objectives for the Automated Grading of Learner English Speech Transcriptions
Hannah Craighead $^{1*}$  Andrew Caines $^{2}$  Paula Buttery $^{2}$  Helen Yannakoudakis $^{3}$
<sup>1</sup> Computer Laboratory, University of Cambridge, U.K. hjc68@cl.cam.ac.uk
2 ALTA Institute & Computer Laboratory, University of Cambridge, U.K.
{andrew.caines|paula.buttery}@cl.cam.ac.uk
$^{3}$  Dept. of Informatics, King's College London, U.K.
helen.yannakoudakis@kcl.ac.uk
We address the task of automatically grading the language proficiency of spontaneous speech based on textual features from automatic speech recognition transcripts. Motivated by recent advances in multi-task learning, we develop neural networks trained in a multi-task fashion that learn to predict the proficiency level of non-native English speakers by taking advantage of inductive transfer between the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language identification (L1). We encode the transcriptions with both bi-directional recurrent neural networks and with bi-directional representations from transformers, compare against a feature-rich baseline, and analyse performance at different proficiency levels and with transcriptions of varying error rates. Our best performance comes from a transformer encoder with L1 prediction as an auxiliary task. We discuss areas for improvement and potential applications for text-only speech scoring.
The growing demand for the ability to communicate in English means that both academic and commercial efforts are increasing to provide automated tutoring and assessment systems. These educational systems address the increasing need for online resources to help students learn and to map users to the validated proficiency scales which play a critical role in securing education and work opportunities (British Council, 2013).
Language learning applications delivered through smart speakers such as Amazon Alexa and Google Home are a novel form of educational technology. These offer obvious benefits to users in terms of immediacy, interaction and
convenience. However, it remains challenging for application providers to assess language content collected through these means. Audio recordings are not returned to the developers for privacy reasons: instead only text responses are returned, the output of automated speech recognition (ASR) systems. This sets a new task in educational applications: the automated proficiency assessment of speech based on transcriptions alone. In this paper we report on our efforts to grade learner English transcriptions obtained from ASR systems, comparing a feature-rich baseline with neural networks trained on multi-task objectives.
To assess spontaneous speech, automated grading systems tend to use a combination of features extracted from the audio recording and the transcription resulting from ASR. For instance, SpeechRaterTM by the Educational Testing Service uses text-based features based on frequency counts and lexical unigrams – among others, the number of word tokens per second, the length of interpaulal units in words, the vocabulary size normalized by recording duration – and score predictions are made using linear regression (Zechner et al., 2007, 2009; Higgins et al., 2011).
However, without the audio recordings, proficiency scoring must be performed based on the text alone. Thus robust methods for text-only speech scoring need to be developed to ensure the reliability and validity of educational applications in scenarios such as smart speakers. Relatively few automated speech graders use neural approaches that incorporate text-based features from transcripts. Chen et al. (2018) used a linear regression model on the concatenated high-level representation outputs of two separate RNNs for sequential audio and text inputs; Qian et al. (2018) use a bi-directional RNN which uses word embeddings concatenated with an encoding of the given prompt and an attention mechanism over all tokens to predict grades.
* Currently at Google U.K.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2258-2269 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In this work, we address the task of automatically grading the language proficiency of spontaneous speech based on ASR transcriptions only, and seek to investigate the extent to which current state-of-the-art neural approaches to language assessment are effective for the task at hand. Specifically, we make the following contributions:
1. We develop a multi-task framework that leverages inductive transfer between our main task (grading spoken language proficiency) and auxiliary objectives – predicting morphosyntactic labels, the learner's first ('native') language (L1) and language modeling (LM).
2. We investigate the performance of two encoder types for the speech scoring task: bidirectional recurrent neural networks, and bidirectional representations from transformers.
3. We analyze model performance under different conditions: namely, with and without filled pauses included in the transcriptions, with varying rates of word error in the ASR transcriptions, and according to the proficiency of the student response.
4. We make our code publicly available for others to use for benchmarking and replication experiments. $^{1}$
In contrast to feature-based scoring, we instead train neural networks on ASR transcriptions which are labeled with proficiency scores assigned by human examiners, and guide the networks with objectives that prioritize language understanding. To the best of our knowledge, there has been no previous work using text-based auxiliary training objectives in automated speech grading systems.
Automated grading of student responses to exam questions until recently tended to adopt feature-based approaches to score prediction, for instance using distinctive word or part-of-speech  $n$ -grams (Page and Paulus, 1968; Attali and Burstein, 2004; Bhat and Yoon, 2015; Sakaguchi et al., 2015), as well as grammatical errors and phrase-structure rules (Yannakoudakis et al., 2011; Andersen et al.,
2013). More recently, word and character embeddings have served as input to deep neural network models, with a final regression layer predicting the score (Alikaniotis et al., 2016; Taghipour and Ng, 2016; Dong et al., 2017; Jin et al., 2018). The advantage of the latter approach is the relative ease of data pre-processing since text representations are learned through distributional methods rather than hand-crafted features.
The field of NLP has seen advances recently thanks to a shift from fixed word embeddings to contextualized representations such as ELMo (Peters et al., 2018) and those which can be obtained from large transformer models such as BERT (Devlin et al., 2019). Similarly in text scoring, some have incorporated contextualized word embeddings to improve performance (Nadeem et al., 2019). We now apply such approaches to the grading of spoken transcriptions in a scenario where the audio, or information derived from it, is not available. In other words the task is analogous to essay scoring except for the presence of characteristic speech features such as false starts, repetitions and filled pauses (Moore et al., 2015; Carter and McCarthy, 2017).
This poses a particular challenge as most models used in data pre-processing and representation learning have been trained on written not spoken texts (Caines et al., 2017). Furthermore, most existing approaches to speech grading do have access to audio features, and indeed extract a large number of prosodic or duration-based features (Zechner et al., 2009; Higgins et al., 2011; Loukina et al., 2017; Wang et al., 2018a). Prosodic and phonological features extracted from the audio and ASR model are undoubtedly useful for human assessment of speech proficiency and for providing feedback.
On the other hand, previous work suggests that models trained solely on ASR text-based features are competitive with those using only acoustic features or a combination of the two (Loukina and Cahill, 2016). Their interpretation of these results was that the transcription offers some proxy information for prosodic and phonological performance – for instance the presence of hesitation and silence markers, the number of word tokens in the transcription, and the transcription errors which might arise from mispronunciations.
We instead allow our models to learn from auxiliary (morpho-syntactic and other) tasks: multi-task learning has been shown to help in automated essay
$^{1}$ https://github.com/hcraighead/automated-english-transcription-grader; the corpus we work with is not publicly available as it is private exams data, but the code repository allows you to work with any set of English texts and proficiency scores.
Train	Valid	Test	Total
Candidates	691	297	225	1213
Transcriptions	4,589	1,982	1488	8,059
Total words	205,311	91,224	67,832	343,367
Mean response length (words)	44.7	46.0	45.6	42.6
Table 1: Training, validation and test split statistics.
scoring (Cummins and Rei, 2018) and grammatical error detection of learner English essays (Rei and Yannakoudakis, 2017), whilst information about a learner's native language has been shown to help in error detection for English and the grading of Norwegian essays (Rozovskaya and Roth, 2011; Johan Berggren et al., 2019). Furthermore, multi-task learning objectives can allow the model to learn more general features of language and composition, and a much richer set of representations (Sanh et al., 2019), without relying on the availability of any external linguistic tools or annotations at inference time.
We train our models using spoken responses collected from candidates taking Cambridge Assessment's BULATS examination. The spoken section of the BULATS exam tests candidates' proficiency in business English through monologue responses to a series of prompts. The candidate may speak for up to one minute in each response and we include only the prompts which invite spontaneous responses (we exclude the prompts which require reading aloud of given sentences, and prompts asking for personal information about the candidates). There are seven such prompts in each exam. Forty-six unique versions of the BULATS exam are represented in the training and test sets, meaning that there are 322 unique prompts  $(7*46)$ .
Each response has been assigned a score between 0 and 6 by expert human examiners, with scoring increments of .5 available and with each whole integer mapping to a proficiency level on the Common European Framework of Reference for Languages (CEFR): a fail (score of 0), beginner (scores of 1, 2: A1 and A2); intermediate (scores 3, 4: B1 and B2); advanced (scores 5, 6: C1 and C2).
Examiners are required to consider five attributes of each candidate's speaking proficiency: pronun-
ciation, hesitation, language resource, coherence and task achievement. In the transcription-only scenario, we cannot assess the first component, have only a proxy for the second in terms of filled pause occurrence ('umm', 'err', etc), but still have access to the other three components through the ASR transcriptions.
Our data comes from 1213 exam candidates with six first languages in approximately uniform distribution: Arabic, Dutch, French, Polish, Thai and Vietnamese. The distribution of candidates over proficiency levels is approximately normal, with a peak over the intermediate scores (Figure 1). The train/Validation/test split across candidates is roughly  $55:25:20$  as detailed by Table 1.
Each candidate's recordings are transcribed by a teacher-student ASR system with a lattice-free maximum-mutual-information acoustic model (Kanda et al., 2017). The teacher-student training procedure uses Kullback-Leibler divergence between the word sequence posteriors from the student model and a teacher ensemble as the loss function (Wong and Gales, 2016). The result is a computationally efficient ASR system, as the student is able to decode in a single run to a similar level of performance as an ensemble decoder requiring multiple runs (Hinton et al., 2014). There is more information about the ASR system in Wang et al. (2018b).
We also evaluate performance on manual transcriptions of the test set, in order to assess the impact of ASR errors on our models. A native speaker of English was asked to transcribe the recordings as faithfully as possible to include hesitations, disfluencies and partial words. A subset of 230 recordings were transcribed by a second native speaker: inter-annotator agreement on this subset is high (Cohen's  $\kappa = .898$ ). Compared against the annotator's manual transcriptions, the word error rate of the ASR is  $19.5\%$  overall, but with variance from  $32\%$  for speakers with a score of 1, to  $15\%$  for speakers with scores 5 and 6.
To be able to predict morpho-syntactic labels,
$^{2}$ https://www.cambridgeenglish.org/exams-and-tests/bulats; now discontinued and replaced by the Linguuaskill Business exam.
Figure 1: Distribution of proficiency scores in the training and test sets.
Figure 2: Transcription length distributions at different proficiency levels.
we parse the data using UDPipe (Wijffels, 2018), trained on the Universal Dependencies (UD) English Web Treebank 2.4 made up of  $255\mathrm{k}$  words and  $16.6\mathrm{k}$  sentences from weblogs, newsgroups, emails, reviews, and Yahoo! answers (Silveira et al., 2014).
We use UDPipe to automatically generate Penn Treebank part of speech (POS) tags (Taylor et al., 2003) and UDs (Nivre et al., 2016) for our training data. Filled pauses were excluded before parsing, so that they would not affect the parse of other words in the transcription, but were then re-inserted with null parse values, in case they serve as a useful signal to the language proficiency models.
Transcriptions were parsed as whole units: we did not attempt to delimit speech-units. For the most part this results in fairly lengthy, but not impractically long, word sequences. The ASR transcriptions are on average 44 word tokens long  $(\sigma = 33.0)$ , with a minimum of 2 tokens, a maximum of 179, and  $50\%$  of the texts being between 23 and 54 tokens long. As seen in Figure 2, the distribution of transcription length differs according to proficiency level: the failing grades tend to be very short responses, the beginner level responses are a little longer, and the bulk of intermediate responses are between 25 and 50 tokens long (recordings are between 20 and 60 seconds duration).
The speech grader<sup>3</sup> takes a sequence of token embeddings  $[x_1, \ldots, x_n]$  as input and predicts a proficiency level score. Tokens are first converted to vector representations  $x_t$ , and then passed through an encoder. We trial two different encoders: a bidirectional LSTM (Hochreiter and Schmidhuber, 1997) and BERT (Devlin et al., 2019). The encoding is passed through the prediction head, a series of linear layers and activation functions, where the final activation function is bound to the scoring scale (0-6). The model uses mean squared error (MSE) as the loss function  $E_{score}$  for the main task.
LSTM encoder The bi-directional LSTM encoder uses the word-level tokenization provided by UD-Pipe. For each token, the hidden states of the two LSTMs are concatenated, creating a context-aware hidden state  $h_t = \overleftarrow{[h_t; h_t]}$ . The hidden layers that are formed at the final timesteps of the bi-directional LSTM  $(h_1, h_n)$  are concatenated for the scoring prediction head.
BERT encoder The BERT encoder uses a pretrained model checkpoint and tokenizer, specifically bert-base-uncased, provided by the HuggingFace Transformer library (Wolf et al., 2019).
3All of our models were built using PyTorch (Paszke et al., 2019).
Figure 3: Encoder architecture of automated speech grader using a bi-directional LSTM for one time step  $t$ : two auxiliary objective architecture (GR and POS) on the left; LM objective architecture on the right.
BERT's tokenizer uses the WordPiece model (Zhang, 2016), resulting in a much larger vocabulary than the LSTM encoder. BERT embeddings are extracted from a transformer trained with a masked LM objective: a percentage of input tokens are masked and then the network learns to predict the masked tokens. BERT is also trained with a second objective: given two input sequences, it predicts whether one sequence directly follows another. A sequence level embedding is produced by pooling the hidden states of the special first token, [CLS], resulting in a 768 dimensional embedding.
Auxiliary objectives We further extend the model to incorporate auxiliary objectives, and experiment with four different tasks: language modelling (LM), native language prediction (L1), POS-tagging, and UD prediction where we predict the UD type of a dependent with its head (see Section 3). These auxiliary objectives are based on previous work indicating that learning to make such predictions aids in tasks such as essay scoring and grammatical error detection (Cheng et al., 2015; Rei and Yannakoudakis, 2017; Cummins and Rei, 2018; Johan Berggren et al., 2019; Bell et al., 2019).
Specifically, for the last three tasks, we predict a label  $y$  per word  $x_{t}$  (Figure 3; left). Each task  $s$  is assigned an individual prediction head, identical to the scoring head described above, followed by a softmax layer that produces a probability distribution over the set of output labels to replace the bounded scoring activation function. When using BERT, our model only predicts labels for auxiliary objectives on the first token of a word, in an identical fashion to Devlin et al. (2019)'s evaluation of BERT on named entity recognition.
The LM objective is implemented differently for each model. The LSTM (Figure 3; right), has two additional hidden layers (Rei, 2017):  $\overrightarrow{m_t} = \tanh \overrightarrow{W_l h_t}$  and  $\overleftarrow{m_t} = \tanh \overleftarrow{W_l h_t}$ , where  $\overrightarrow{W_l}$  and
LM	L1	POS	UD
LSTM	0.1	0.01	0.005	0.001
BERT	0.05	0.5	0.1	0.01
Table 2: Weighting values for auxiliary objectives scores for the LSTM and BERT encoders.
$\overleftarrow{W_l}$  are direction-specific weight matrices. The surrounding tokens  $w_{t - 1}$  and  $w_{t + 1}$  are then predicted based on each hidden state using a softmax output layer. In contrast, the BERT model implements the same masked language modeling objective as utilized during pre-training. We implement this identically to Devlin et al. (2019):  $15\%$  of tokens in the sequence are randomly selected to be masked, and of those,  $80\%$  are masked,  $10\%$  are replaced with another token and  $10\%$  are unchanged. The loss is only computed over the selected tokens. Note that filled pauses are not utilized for auxiliary objectives.
The overall loss function  $E$  is adapted using a similar approach to Cummins and Rei (2018): a weighted sum of the scoring loss (main task)  $E_{score}$  and the auxiliary task losses  $E_{aux}$ , where  $T$  is the total number of auxiliary tasks. All of the auxiliary tasks use cross-entropy loss where  $y_{x,l}$  is the predicted probability of token  $x$  having label  $l$ , and  $\tilde{y}_{x,l}$  has the value 1 when  $l$  is the correct label for token  $x$  and 0 otherwise.
$$
E _ {a u x} = - \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {l = 1} ^ {L} \tilde {y} _ {t, l} l o g (y _ {t, l}) \qquad (1)
$$
$$
E = (1 - \alpha) \times E _ {\text {s c o r e}} + \alpha \times E _ {\text {a u x}} \tag {2}
$$
Model hyper-parameters are tuned based on MSE on the validation set. The model is optimized using Adam (Kingma and Ba, 2014), with a learning rate of 0.001 that linearly decreases during training, for 3-5 epochs (when trained with no,
RMSE	PCC	≤0.5	≤1.0	RMSE	PCC	≤0.5	≤1.0
Baseline	1.086	0.685	50.7	82.1	1.086	0.685	50.7	82.1
LSTM	BERT
Task	RMSE	PCC	≤0.5	≤1.0	RMSE	PCC	≤0.5	≤1.0
Scoring	1.022	0.681	39.496	69.530	0.921	0.762	45.060	75.134
+LM	1.011†	0.689†	40.282†	70.289†	0.910	0.767	45.665	76.169
+L1	1.014	0.687	39.812	69.765	0.908	0.769†	45.659	76.310
+POS	1.006†	0.693†	40.074	70.356†	0.918	0.763	44.892	75.383
+UD	1.010†	0.689†	39.872†	70.309	0.920	0.762	44.940	75.336
Combo	1.005†	0.690†	40.390†	70.114†	-	-	-	-
Table 3: Evaluation of the baselines, LSTM and BERT encoders for speech grading, with a single-task scoring objective and various auxiliary tasks (LM: language modeling, L1: native language identification, POS: part-of-speech tagging, UD: Universal dependency relations, Combo: POS+UD+L1).  $\dagger$  indicates significant difference (paired t-test,  $\alpha = 0.05$ ) compared to the single-task scoring model.
a single, or multiple auxiliary objectives respectively). Responses are processed in batches of 8 and are padded/truncated to a length of 128. LSTM token embeddings of size 300 are randomly initialized and fine-tuned during training. The LSTM has 3 hidden layers with hidden state sizes of 256 for each direction. Weightings for each of the auxiliary objectives were selected by evaluation on the validation set and are outlined in Table 2.
Baseline model Our baseline approach is a feature-based model of the type which has been used in previous research (Vajjala and Rama, 2018; Yannakoudakis et al., 2018). Specifically, we train a linear regression model and use as features tfidf weighted word and POS  $n$ -grams (up to trigrams), grammatical constructions extracted from the phrase-structure trees, the length of the transcript, and the number of errors, estimated by counting the number of trigrams that are absent from a large background corpus of correct English (Ferraresi et al., 2008).
Evaluation Our primary metric is root-mean-square error (RMSE), which results in real valued average distances from the gold standard examiner scores on our 0-6 scale.
For each model we also report Pearson's correlation coefficient with the true scores and the percent of predictions which are within a half or one score from the reference score  $(\leq 0.5$  and  $\leq 1.0)$ . These can be thought of as tolerable error thresholds where being out-by-two can have severe consequences for the student (for example, affecting employment or education prospects). Bear in
mind that human examiners are thought to correlate on proficiency scoring at about 0.8, and that most exams are graded by a single examiner, and the idea of tolerable error becomes relevant to human as well as machine scoring. It would be a useful exercise to collect within 0.5 and within 1.0 scores from human examiners.
We ran a series of experiments to analyze the impact that data pre-processing and encoder design have on the performance of our automated speech grader. All results presented are computed over 10 repetitions, include filled pause information and use an ASR system with a WER of  $19.5\%$  (see Section 3) unless otherwise stated.
5.1 Encoder
Table 3 compares the results for the two different encoders: LSTM and BERT. Using BERT significantly increases the performance of the speech grader, RMSE reduces by approximately 0.1 and the number of responses graded within 0.5 or 1 point of examiner provided score increases by approximately  $5.5\%$ .
5.2 Auxiliary objectives
Our results, in Table 3, indicate that certain auxiliary objectives can improve the performance of our automated speech grader. The LSTM gains significantly when applying multi-task learning from POS, UD or LM prediction tasks. It is also possible that these objectives help to account for errors in ASR by identifying instances where the expected word or morpho-syntactic label differs from the provided input.
4 Initial experiments showed that fixed pre-trained word embeddings such as GloVe (Pennington et al., 2014) do not improve performance further.
WER of ASR systems used for testing
Figure 4: RMSE of LSTM and BERT speech graders trained and tested on ASR systems of decreasing WER.