Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home
Year: 2023

Abstract:
Enriching the quality of early childhood education with interactive math learning at home systems, empowered by recent advances in conversational AI technologies, is slowly becoming a reality. With this motivation, we implement a multimodal dialogue system to support play-based learning experiences at home, guiding kids to master basic math concepts. This work explores Spoken Language Understanding (SLU) pipeline within a task-oriented dialogue system developed for Kid Space, with cascading Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) components evaluated on our home deployment data with kids going through gamified math learning activities. We validate the advantages of a multi-task architecture for NLU and experiment with a diverse set of pretrained language representations for Intent Recognition and Entity Extraction tasks in the math learning domain. To recognize kids’ speech in realistic home environments, we investigate several ASR systems, including the commercial Google Cloud and the latest open-source Whisper solutions with varying model sizes. We evaluate the SLU pipeline by testing our bestperforming NLU models on noisy ASR output to inspect the challenges of understanding children for math learning in authentic homes.

heading: 1 Introduction and Background
text: The ongoing progress in Artificial Intelligence (AI) based advanced technologies can assist humanity in reducing the most critical inequities around the globe. The recent widespread interest in conversational AI applications presents exciting opportunities to showcase the positive societal impact of these technologies. The language-based AI systems have already started to mature to a level where we may soon observe their influences in mitigating the most pressing global challenges. Education is among the top priority improvement areas identified by the United Nations (UN) (i.e., poverty,
hunger, healthcare, and education). In particular, increasing the inclusiveness and quality of education is within the UN development goals1 with utmost urgency. One of the preeminent ways to diminish societal inequity is promoting STEM (i.e., Science, Technology, Engineering, Math) education, specifically ensuring that children succeed in mathematics. It is well-known that acquiring basic math skills at younger ages builds students up for success, regardless of their future career choices (Cesarone, 2008; Torpey, 2012). For math education, interactive learning environments through gamification present substantial leverages over more traditional learning settings for studying elementary math subjects, particularly with younger learners (Skene et al., 2022). With that goal, conversational AI technologies can facilitate this interactive learning environment where students can master fundamental math concepts. Despite these motivations, studying spoken language technologies for younger kids to learn basic math is a vastly uncharted area of AI.
This work discusses a modular goal-oriented Spoken Dialogue System (SDS) specifically targeted for kids to learn and practice basic math concepts at home setup. Initially, a multimodal dialogue system (Sahay et al., 2019) is implemented for Kid Space (Anderson et al., 2018), a gamified math learning application for deployment in authentic classrooms. During this preliminary real-world deployment at an elementary school, the COVID19 pandemic impacted the globe, and school closures forced students to switch to online learning options at home. To support this sudden paradigm shift to at-home learning, previous school use cases are redesigned for new home usages, and our dialogue system is recreated to deal with interactive math games at home. While the play-based learning activities are adjusted for home usages with a much simpler setup, the multimodal aspects of these games are partially preserved along with the
1https://sdgs.un.org/goals
692
fundamental math concepts for early childhood education. These math skills cover using ones and tens to construct numbers and foundational arithmetic concepts and operations such as counting, addition, and subtraction. The multimodal aspects of these learning games include kids’ spoken interactions with the system while answering math questions and carrying out game-related conversations, physical interactions with the objects (i.e., placing cubes and sticks as manipulatives) on a visually observed playmat, performing specific pose and gesture-based actions as part of these interactive games (e.g., jumping, standing still, air high-five).
Our domain-specific SDS pipeline (see Figure 1) consists of multiple cascaded components, namely Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), Multimodal Dialogue Manager (DM), Natural Language Generation (NLG), and Text-to-Speech (TTS) synchronizing the agent utterances with virtual character animations on Student User Interface (UI). Here we concentrate on the Spoken Language Understanding (SLU) task on kids’ speech at home environments while playing basic math games. Such application-dependent SLU approaches commonly involve two main modules applied sequentially: (i) Speech-to-Text (STT) or ASR module that recognizes speech and transcribes the spoken utterances into text, and (ii) NLU module that interprets the semantics of those utterances by processing the transcribed text. NLU is one of the most integral components of these goal-oriented dialogue systems. It empowers user-agent interactions by understanding the meaning of user utterances via performing domain-specific sub-tasks. Intent Recognition (IR) and Named Entity Recognition (NER) are essential sub-tasks within the NLU module to resolve the complexities of human language and extract meaningful information for the application at hand. Given a user utterance as input, the Intent Classification aims to identify the user’s intention (i.e.,
what the user desires to achieve with that interaction) and categorize the user’s objective at that conversational turn. The Entity Extraction targets locating and classifying entities (i.e., specific terms representing existing things such as person names, locations, and organizations) mentioned in user utterances into predefined task-specific categories.
In this study, we present our efforts to convert the task-oriented SDS (Okur et al., 2022b) designed for school use cases (Aslan et al., 2022) to home usages after COVID-19 and inspect the performance of individual SDS modules evaluated on the home deployment data we recently collected from 12 kids individually at their homes. The current work focuses on assessing and improving the SLU task performance on kids’ utterances at home by utilizing this real-world deployment data. We first investigate the ASR and NLU module evaluations independently. Then, we inspect the overall SLU pipeline (ASR+NLU) performance on kids’ speech by evaluating our NLU tasks on ASR output (i.e., recognized text) at home environments. As the erroneous and noisy speech recognition output would lead to incorrect intent and entity predictions, we aim to understand these error propagation consequences with SLU for children in the math learning domain. We experiment with various recent ASR solutions and diverse model sizes to gain more insights into their capabilities to recognize kids’ speech at home. We then analyze the effects of these ASR engines on understanding intents and extracting entities from children’s utterances. We discuss our findings and observations for potential enhancements in future deployments of this multimodal dialogue system for math learning at home.

heading: 2 Related Work

heading: 2.1 Conversational AI for Math Learning
text: With the ultimate goal of improving the quality of education, there has been a growing enthusiasm for exploiting AI-based intelligent systems to boost students’ learning experiences (Chassignol et al., 2018; Aslan et al., 2019; Jia et al., 2020; Zhai et al., 2021; Baker, 2021). Among these, interactive frameworks that support guided playbased learning spaces revealed significant advantages for math learning (Pires et al., 2019; Sun et al., 2021; Richey et al., 2021), especially for building foundational math skills in early childhood education (Nrupatunga et al., 2021; Skene et al., 2022). To attain this level of interactivity within smart
learning spaces, developing innovative educational applications by utilizing language-based AI technologies is in growing demand (Taghipour and Ng, 2016; Lende and Raghuwanshi, 2016; Raamadhurai et al., 2019; Cahill et al., 2020; Chan et al., 2021; Rathod et al., 2022). In particular, designing conversational agents for intelligent tutoring is a compelling yet challenging area of research, with several attempts presented so far (Winkler and Söllner, 2018; Wambsganss et al., 2020; Winkler et al., 2020; Datta et al., 2020; Okonkwo and Ade-Ibijola, 2021; Wollny et al., 2021), most of them focusing on language learning (Bibauw et al., 2022; Tyen et al., 2022; Zhang et al., 2022).
In the math education context, earlier conversational math tutoring applications exist, such as SKOPE-IT (Nye et al., 2018), which is based on AutoTutor (Graesser et al., 2005) and ALEKS (Falmagne et al., 2013), and MathBot (Grossman et al., 2019). These are often text-based online systems following strict rules in conversational graphs. Later, various studies emerged at the intersection of cutting-edge AI techniques and math learning (Mansouri et al., 2019; Huang et al., 2021; Azerbayev et al., 2022; Uesato et al., 2022; Yang et al., 2022). Among those, employing advanced language understanding methods to assist math learning is relatively new (Peng et al., 2021; Shen et al., 2021; Loginova and Benoit, 2022; Reusch et al., 2022). The majority of those recent work leans on exploring language representations for math-related tasks such as mathematical reasoning, formula understanding, math word problemsolving, knowledge tracing, and auto-grading, to name a few. Recently, TalkMoves dataset (Suresh et al., 2022a) was released with K-12 math lesson transcripts annotated for discursive moves and dialogue acts to classify teacher talk moves in math classrooms (Suresh et al., 2022b).
For the conversational AI tasks, the latest large language models (LLMs) based chatbots, such as BlenderBot (Shuster et al., 2022) and ChatGPT (OpenAI, 2022), gained a lot of traction in the education community (Tack and Piech, 2022; Kasneci et al., 2023), along with some concerns about using generative models in tutoring (Macina et al., 2023; Cotton et al., 2023). ChatGPT is a generalpurpose open-ended interaction agent trained on internet-scale data. It is an end-to-end dialogue model without explicit NLU/Intent Recognizer or DM, which currently cannot fully comprehend the
multimodal context and proactively generate responses to nudge children in a guided manner without distractions. Using these recent chatbots for math learning is still in the early stages because they are known to miss basic mathematical abilities and carry reasoning flaws (Frieder et al., 2023), revealing a lack of common sense. Moreover, they are known to be susceptible to triggering inappropriate or harmful responses and potentially perpetuate human biases since they are trained on internetscale data and require carefully-thought guardrails.
On the contrary, our unique application is a taskoriented math learning spoken dialogue system designed to perform learning activities, following structured educational games to assist kids in practicing basic math concepts at home. Our SDS does not require massive amounts of data to understand kids and generate appropriate adaptive responses, and the lightweight models can run locally on client machines. In addition, our solution is multimodal, intermixing the physical and digital hybrid learning experience with audio-visual understanding, object recognition, segmentation, tracking, and pose and gesture recognition.

heading: 2.2 Spoken Language Understanding
text: Conventional pipeline-based dialogue systems with supervised learning are broadly favored when initial domain-specific training data is scarce to bootstrap the task-oriented SDS for future data collection (Serban et al., 2018; Budzianowski et al., 2018; Mehri et al., 2020). Deep learning-based modular dialogue frameworks and practical toolkits are prominent in academic and industrial settings (Bocklisch et al., 2017; Burtsev et al., 2018; Reyes et al., 2019). For task-specific applications with limited in-domain data, current SLU systems often use a cascade of two neural modules: (i) ASR maps the input audio to text (i.e., transcript), and (ii) NLU predicts intent and slots/entities from this transcript. Since our main focus in this work is investigating the SLU pipeline, we briefly summarize the existing NLU and ASR solutions.

heading: 2.2.1 Language Representations for NLU
text: The NLU component processes input text, often detects intents, and extracts referred entities from user utterances. For the mainstream NLU tasks of Intent Classification and Entity Recognition, jointly trained multi-task models are proposed (Liu and Lane, 2016; Zhang and Wang, 2016; Goo et al., 2018) with hierarchical learning approaches (Wen
et al., 2018; Okur et al., 2019; Vanzo et al., 2019). Transformer architecture (Vaswani et al., 2017) is a game-changer for several downstream language tasks. With Transformers, BERT (Devlin et al., 2019) is presented, which became one of the most pivotal breakthroughs in language representations, achieving high performance in various tasks, including NLU. Later, Dual Intent and Entity Transformer (DIET) architecture (Bunk et al., 2020) is invented as a lightweight multi-task NLU model. On multi-domain NLU-Benchmark data (Liu et al., 2021b), the DIET model outperformed fine-tuning BERT for joint Intent and Entity Recognition.
For BERT-based autoencoding approaches, RoBERTa (Liu et al., 2019) is presented as a robustly optimized BERT model for sequence and token classification. The Hugging Face introduced a smaller, lighter general-purpose language representation model called DistilBERT (Sanh et al., 2019) as the knowledge-distilled version of BERT. ConveRT (Henderson et al., 2020) is proposed as an efficiently compact model to obtain pretrained sentence embeddings as conversational representations for dialogue-specific tasks. LaBSE (Feng et al., 2022) is a pretrained multilingual model producing language-agnostic BERT sentence embeddings that achieve promising results in text classification.
The GPT family of autoregressive LLMs, such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), perform well at what they are pretrained for, i.e., text generation. GPT models can also be adopted for NLU, supporting few-shot learning capabilities, and NLG in task-oriented dialogue systems (Madotto et al., 2020; Liu et al., 2021a). XLNet (Yang et al., 2019) applies autoregressive pretraining for representation learning that adopts Transformer-XL (Dai et al., 2019) as a backbone model and works well for language tasks with lengthy contexts. DialoGPT (Zhang et al., 2020) extends GPT-2 as a large-scale neural response generation model for multi-turn conversations trained on Reddit discussions, whose representations can be exploited in dialogue tasks.
For language representations to be utilized in math-related tasks, MathBERT (Shen et al., 2021) is introduced as a math-specific BERT model pretrained on large math corpora. Later, Math-awareBERT and Math-aware-RoBERTa models (Reusch et al., 2022) are proposed based on BERT and RoBERTa, pretrained on Math Stack Exchange2.
2https://math.stackexchange.com/

heading: 2.2.2 Speech Recognition with Kids
text: Speech recognition technology has been around for some time, and numerous ASR solutions are available today, both commercial and open-source. Rockhopper ASR (Stemmer et al., 2017) is an earlier low-power speech recognition engine with LSTM-based language models, where its acoustic models are trained using an open-source Kaldi speech recognition toolkit (Povey et al., 2011). Google Cloud Speech-to-Text3 is a prominent commercial ASR service powered by advanced neural models and designed for speech-dependant applications. Until recently, Google STT API was arguably the leader in ASR services for recognition performance and language coverage. Franck Dernoncourt (2018) reported that Google ASR could reach a word error rate (WER) of 12.1% on LibriSpeech clean dataset (28.8% on LibriSpeech other) (Panayotov et al., 2015) at that time, which is improved drastically over time. Recently, Open AI released Whisper ASR (Radford et al., 2022) as a game-changer speech recognizer. Whisper models are pretrained on a vast amount of labeled audiotranscription data (i.e., 680k hours), unlike its predecessors (e.g., Wav2Vec 2.0 (Baevski et al., 2020) is trained on 60k hours of unlabeled audio). 117k hours of this data are multilingual, which makes Whisper applicable to over 96 languages, including low-resourced ones. Whisper architecture follows a standard Transformer-based encoder-decoder as many speech-related models (Latif et al., 2023). The Whisper-base model is reported to achieve 5.0% & 12.4% WER on LibriSpeech clean & other.
Although speech recognition systems are substantially improving to achieve human recognition levels, problems still occur, especially in noisy environments, with users having accents and dialects or underrepresented groups like kids. Child speech brings distinct challenges to ASR (Stemmer et al., 2003; Gerosa et al., 2007; Yeung and Alwan, 2018), such as data scarcity and highly varied acoustic, linguistic, physiological, developmental, and articulatory characteristics compared to adult speech (Claus et al., 2013; Shivakumar and Georgiou, 2020; Bhardwaj et al., 2022). Thus, WER for children’s voices is reported two-to-five times worse than for adults (Wu et al., 2019), as the younger the child, the poorer ASR performs. There exist efforts to mitigate these difficulties of speech recognition with kids (Shivakumar et al.,
3https://cloud.google.com/speech-to-text/
2014; Duan and Chen, 2020; Booth et al., 2020; Kelly et al., 2020; Rumberg et al., 2021; Yeung et al., 2021). Few studies also focus on speech technologies in educational settings (Reeder et al., 2015; Blanchard et al., 2015; Bai et al., 2021, 2022; Dutta et al., 2022), often for language acquisition, reading comprehension, and story-telling activities.

heading: 3 Methods

heading: 3.1 Home Learning Data and Use Cases
text: We utilize two datasets for gamified basic math learning at home usages. The first set is a proof-ofconcept (POC) data manually constructed based on User Experience (UX) studies (e.g., detailed scripts for new home use cases) and partially adopted from our previous school data (Okur et al., 2022a). This POC data is used to train and cross-validate various NLU models to develop the best practices in later home deployments. The second set is our recent home deployment data collected from 12 kids (ages 7-8) experiencing our multimodal math learning system at authentic homes. The audio-visual data is transcribed manually, and user utterances in these reference transcripts are annotated for intent and entity types we identified for each learning activity at home. Table 1 compares the NLU statistics for Kid Space Home POC and Deployment datasets. Manually transcribed children’s utterances in deployment data are employed to test our best NLU models trained on POC data. We run multiple ASR engines on audio recordings from home deployment data, where automatic transcripts (i.e., ASR output) are utilized to compute WER to assess ASR model performances on kids’ speech. We also evaluate the SLU pipeline (ASR+NLU) by testing NLU models on ASR output from deployment data.
The simplified home deployment setup includes a playmat with physical manipulatives, a laptop with a built-in camera, a wireless lavalier mic, and a depth camera on a tripod. Home use cases follow a particular flow of activities designed for play-based learning in early childhood education. These activities are Introduction (Meet & Greet), Warm-up Game (Red Light Green Light), Training Game, Learning Game, and Closure (Dance Party). After meeting with the virtual character and playing jumping games, the child starts the training game, where the agent asks for help planting flowers. The agent presents tangible manipulatives, cubes representing ones and sticks representing tens, and instructs the kid to answer ba-
sic math questions and construct numbers using these objects, going through multiple rounds of practice questions where flowers in child-selected colors bloom as rewards. In the actual learning game, the agent presents clusters of questions involving ones & tens, and the child provides verbal (e.g., stating the numbers) and visual answers (e.g., placing the cubes and sticks on the playmat, detected by the overhead camera). The agent provides scaffolding utterances and performs animations to show and tell how to solve basic math questions. The interaction ends with a dance party to celebrate achievements and say goodbyes in closure. Some of our intents can be considered generic (e.g., state-name, affirm, deny, repeat, out-of-scope), but some are highly domain-specific (e.g., answerflowers, answer-valid, answer-others, state-color, had-fun-a-lot, end-game) or math-related (e.g., state-number, still-counting). The entities we extract are activity-specific (i.e., name, color) and math-related (i.e., number).

heading: 3.2 NLU and ASR Models
text: Customizing open-source Rasa framework (Bocklisch et al., 2017) as a backbone, we investigate several NLU models for Intent Recognition and Entity Extraction tasks to implement our math learning conversational AI system for home usage. Our baseline approach is inspired by the StarSpace (Wu et al., 2018) method, a supervised embedding-based model maximizing the similarity between utterances and intents in shared vector space. We enrich this simple text classifier by incorporating SpaCy (Honnibal et al., 2020) pre-
trained language models4 for word embeddings as additional features in the NLU pipeline. CRF Entity Extractor (Lafferty et al., 2001) with BILOU tagging is also part of this baseline NLU. For home usages, we explore the advantages of switching to a more recent DIET model5 for joint Intent and Entity Recognition, a multi-task architecture with two-layer Transformers shared for NLU tasks. DIET leverages combining dense features (e.g., any given pretrained embeddings) with sparse features (e.g., token-level encodings of char n-grams). To observe the net benefits of DIET, we first pass the identical SpaCy embeddings used in our baseline (StarSpace) as dense features to DIET. Then, we adopt DIET with pretrained BERT6, RoBERTa7, and DistilBERT8 word embeddings, as well as ConveRT9 and LaBSE10 sentence embeddings to inspect the effects of these autoencoding-based language representations on NLU performance (see 2.2.1 for more details). We also evaluate pretrained embeddings from models using autoregressive training such as XLNet11, GPT-21213, and DialoGPT14 on top of DIET. Next, we explore recently-proposed math-language representations pretrained on math data for our basic math learning dialogue system. MathBERT (Shen et al., 2021) is pretrained on large math corpora (e.g., curriculum, textbooks, MOOCs, arXiv papers) covering pre-k to college-graduate materials. We enhance DIET by incorporating embeddings from MathBERT-base15 and MathBERT-custom16 models, pretrained with BERT-base original and math-customized vocabularies, respectively. Mathaware-BERT17 and Math-aware-RoBERTa18 mod-
4https://github.com/explosion/spacy-models/ releases/tag/en_core_web_md-3.5.0
5Please check Bunk et al. (2020) for hyper-parameter tuning, hardware specs, and computational costs.
6https://huggingface.co/bert-base-uncased 7https://huggingface.co/roberta-base 8https://huggingface.co/
distilbert-base-uncased 9https://github.com/connorbrinton/ polyai-models/releases 10https://huggingface.co/rasa/LaBSE 11https://huggingface.co/xlnet-base-cased 12https://huggingface.co/gpt2 13Excluded GPT-3 and beyond that are not open-source. 14https://huggingface.co/microsoft/ DialoGPT-medium 15https://huggingface.co/tbs17/MathBERT 16https://huggingface.co/tbs17/MathBERT-custom 17https://huggingface.co/AnReu/math_pretrained_ bert 18https://huggingface.co/AnReu/math_pretrained_ roberta
els (Reusch et al., 2022) are initialized from BERTbase and RoBERTa-base, and further pretrained on Math StackExchange19 with extra LaTeX tokens to better tokenize math formulas for ARQMath-3 tasks (Mansouri et al., 2022). We exploit these representations with DIET to investigate their effects on our NLU tasks in the basic math domain.
For the ASR module, we explore three main speech recognizers for our math learning application at home, which are explained further in 2.2.2. Rockhopper ASR20 is the baseline local approach previously inspected, which can be adjusted slightly for kids. Its acoustic models rely on Kaldi21 generated resources and are trained on default adult speech data. In the past explorations, when Rockhopper’s language models fine-tuned with limited in-domain kids’ utterances (Sahay et al., 2021) from previous school usages, WER decreased by 40% for kids but remained 50% higher than adult WER. Although this small-scale baseline solution is unexpected to reach Google Cloud ASR performance, Rockhopper has a few other advantages for our application since it can run offline locally on low-power devices, which could be better for security, privacy, latency, and cost (relative to cloud-based ASR services). Google ASR is a commercial cloud solution providing high-quality speech recognition service but requiring connectivity and payment, which cannot be adapted or finetuned as Rockhopper. The third ASR approach we investigate is Whisper22, which combines the best of both worlds as it is an open-source adjustable solution that can run locally, achieving new stateof-the-art (SOTA) results. We inspect three configurations of varying model sizes (i.e., base, small, and medium) to evaluate the Whisper ASR for our home math learning usage with kids.

heading: 4 Experimental Results
text: To build the NLU module of our SLU pipeline, we train Intent and Entity Classification models and cross-validate them over the Kid Space Home POC dataset to decide upon the best-performing NLU architectures moving forward for home. Table 2 summarizes the results of model selection experiments with various NLU models. We report the average of 5 runs, and each run involves a 10-fold
19https://archive.org/download/stackexchange 20https://docs.openvino.ai/2018_R5/_samples_
speech_sample_README.html 21https://github.com/kaldi-asr/kaldi 22https://github.com/openai/whisper
cross-validation (CV) on POC data. Compared to the baseline StarSpace algorithm, we gain almost 2% F1 score for intents and more than 1% F1 for entities with multi-task DIET architecture. For language representations, we observe that incorporating DIET with the BERT family of embeddings from autoencoders achieves higher F1 scores relative to the GPT family of embeddings from autoregressive models. We cannot reveal any benefits of employing math-specific representations with DIET, as all such models achieve worse than DIET+BERT results. One reason we identify is the mismatch between our early math domain and advanced math corpora, including college-level math symbols and equations, that these models trained on. Another reason could be that such embeddings are pretrained on smaller math corpora (e.g., 100 million tokens) compared to massive-scale generic corpora (e.g., 3.3 billion words) that BERT models use for training. DIET+ConveRT is the clear winner for intents and achieves second-best but very close results for entities compared to DIET+LaBSE. ConveRT and LaBSE are both sentence-level embeddings, but ConveRT performs well on dialogue tasks as it is pretrained on large conversational corpora, including Reddit discussions. Based on these results, we select DIET+ConveRT as the final multitask architecture for our NLU tasks at home.
Next, we evaluate our NLU module on Kid Space Home Deployment data collected at authentic homes over 12 sessions with 12 kids. Each child goes through 5 activities within a session, as described in 3.1. In Table 3, we observe overall F1% drops (∆) of 4.6 for intents and 0.3 for entities when our best-performing DIET+ConveRT models are tested on home deployment data. These findings are expected and relatively lower than
the performance drops we previously observed at school (Okur et al., 2022c). We witness distributional and utterance-length differences between POC/training and deployment/test datasets. Realworld data would always be noisier than anticipated as these utterances come from younger kids playing math games in dynamic conditions.
To further improve the performance of our Kid Space Home NLU models (trained on POC data) by leveraging this recent deployment data, we experiment with merging the two datasets for training and evaluating the performance on individual deployment sessions via leave-one-out (LOO) CV. At each of the 12 runs (for 12 sessions/kids), we merge the POC data with 11 sessions of deployment data for model training and use the remaining session as a test set, then take the average performance of these runs. That would simulate how combining POC with real-world deployment data would help us train more robust NLU models that perform better on unseen data in future deployment sessions. The overall F1-scores reach 96.5% for intents (2.3% gain from 94.2%) and 99.4% for entities (0.1% gain) with LOOCV, which are promising for our future deployments.
To inspect the ASR module of our SLU pipeline, we experiment with Rockhopper, Google, and Whisper-base/small/medium ASR models evaluated on the same audio data collected during home deployments. Using the manual session transcripts as a reference, we compute the average WER for kids with each ASR engine to investigate the most feasible solution. Table 4 summarizes WER results before and after standard pre-processing steps (e.g., lower casing and punctuation removal) as well as application-specific filters (e.g., num2word and cleaning). The numbers are transcribed inconsistently within reference transcripts plus ASR output (e.g., 35 vs. thirty-five), and we need to standardize them all in word forms. The cleaning
step is applied to Whisper ASR output only due to known issues such as getting stuck in repeat loops and hallucinations (Radford et al., 2022). We seldom observe trash output from Whisper (4-to-7%) having very long transcriptions with non-sense repetitions/symbols, which hugely affect WER due to their length, yet these samples can be easily autofiltered. Even after these steps, the relatively high error rates can be attributed to many factors related to the characteristics of these recordings (e.g., incidental voice and phrases), very short utterances to be recognized (e.g., binary yes/no answers or stating numbers with one-or-two words), and recognizing kids’ speech in ordinary home environments. Still, the comparative results indicate that Whisper ASR solutions perform better on kids, and we can benefit from increasing the model size from base to small, while small to medium is close.
For SLU pipeline evaluation, we test our highestperforming NLU models on noisy ASR output. Table 5 presents the Intent and Entity Classification results achieved on home deployment data where the DIET+ConveRT models run on varying ASR models output. Note that Voice Activity Detection (VAD) is an integral part of ASR that decides the presence/absence of human speech. We realize that the VAD stage is filtering out a lot of audio chunks with actual kid speech with Rockhopper and Google. Thus, our VAD-ASR nodes can ignore a lot of audio segments with reference transcripts (57.9% for Rokchopper, 49.1% for Google). That is less of an issue with Whisper-base/small/medium, missing 7.1%/5.7%/4.4% of transcribed utterances (often due to filtering very long and repetitive trash Whisper output). When we treat these entirely missed utterances with no ASR output as classification errors for NLU tasks (i.e., missing to predict intent/entities when no speech is detected), we can adjust the F1-scores accordingly to evaluate the VAD-ASR+NLU pipeline. These VAD-adjusted F1-scores are compared in Table 5, aligned with the WER results, where NLU on Whisper ASR
performs relatively higher than Google and Rockhopper. For enhanced Intent Recognition in realworld deployments with kids, increasing the ASR model size from small to medium could be worth the trouble for Whisper. Yet, the F1 drop is still huge, from 94.2% with NLU to 73.1% with VADASR+NLU, when VAD-ASR errors propagate into the SLU pipeline.

heading: 5 Error Analysis
text: For NLU error analysis, Table 6 reveals utterance samples from our Kid Space Home Deployment data with misclassified intents obtained by the DIET+ConveRT models on manual/human transcripts. These language understanding errors illustrate the potential pain points solely related to the NLU model performances, as we are assuming perfect or human-level ASR here by feeding the manually transcribed utterances into the NLU. Such intent prediction errors occur in real-world deployments for many reasons. For example, authentic user utterances can have multiple intents (e.g., “Yeah. Can we have some carrots?” starts with affirm and continues with out-of-scope). Some utterances can be challenging due to subtle differences between intent classes (e.g., “Ah this is 70, 7.” is submitting a verbal answer with state-number but can easily be mixed with still-counting too). Moreover, we observe utterances having colors and “flowers” within out-of-scope (e.g., “Wow, that’s a lot of red flowers.”), which can be confusing for the NLU models trained on cleaner POC datasets.
For further error analysis on the SLU pipeline (ASR+NLU), Table 7 demonstrates Intent Recognition error samples from Kid Space Home Deployment data obtained on ASR output with several speech recognition models we explored. These samples depict anticipated error propagation from speech recognition to language understanding modules in the cascaded SLU approach. Please check Appendix A for a more detailed ASR error analysis.

heading: 6 Conclusion
text: To increase the quality of math learning experiences at home for early childhood education, we develop a multimodal dialogue system with playbased learning activities, helping the kids gain basic math skills. This study investigates a modular SLU pipeline for kids with cascading ASR and NLU modules, evaluated on our first home deployment data with 12 kids at individual homes. For NLU, we examine the advantages of a multi-task architecture and experiment with numerous pretrained language representations for Intent Recognition and Entity Extraction tasks in our application domain. For ASR, we inspect the WER with several solutions that are either low-power and local (e.g., Rockhopper), commercial (e.g., Google Cloud), or open-source (e.g., Whisper) with varying model sizes and conclude that Whispermedium outperforms the rest on kids’ speech at authentic home environments. Finally, we evaluate the SLU pipeline by running our best-performing NLU models, DIET+ConveRT, on VAD-ASR output to observe the significant effects of cascaded errors due to noisy voice detection and speech recognition performance with kids in realistic home deployment settings. In the future, we aim to finetune the Whisper ASR acoustic models on kids’ speech and language models on domain-specific math content. Moreover, we consider exploring N-Best-ASR-Transformers (Ganesan et al., 2021) to leverage multiple Whisper ASR hypotheses and mitigate errors propagated into cascading SLU.
Limitations
By building this task-specific dialogue system for kids, we aim to increase the overall quality of basic math education and learning at-home experiences for younger children. In our previous school deployments, the overall cost of the whole school/classroom setup, including the wall/ceilingmounted projector, 3D/RGB-D cameras, LiDAR sensor, wireless lavalier microphones, servers, etc., can be considered as a limitation for public schools and disadvantaged populations. When we shifted our focus to home learning usages after the COVID19 pandemic, we simplified the overall setup for 1:1 learning with a PC laptop with a built-in camera, a depth camera on a tripod, a lapel mic, and a playmat with cubes and sticks. However, even this minimal instrumentation suitable for home setup can be a limitation for kids with lower socioeconomic status. Moreover, the dataset size of our initial home deployment data collected from 12 kids in 12 sessions is relatively small, with around 12 hours of audio data manually transcribed and annotated. Collecting multimodal data at authentic homes of individual kids within our target age group (e.g., 5- to-8 years old) and labor-intensive labeling process is challenging and costly. To overcome these data scarcity limitations and develop dialogue systems for kids with such small-data regimes, we had to rely on transfer learning approaches as much as possible. However, the dataset sizes affect the generalizability of our explorations, the reliability of some results, and ultimately the robustness of our multimodal dialogue system for deployments with kids in the real world.
Ethics Statement
Prior to our initial research deployments at home, a meticulous process of Privacy Impact Assessment is pursued. The legal approval processes are completed to operate our research with educators, parents, and the kids. Individual participants and parties involved have signed the relevant consent forms in advance, which inform essential details about our research studies. The intentions and procedures and how the participant data will be collected and utilized to facilitate our research are explained in writing in these required consent forms. Our collaborators comply with stricter data privacy policies as well.

heading: Acknowledgements
text: We aspire to share our gratitude and acknowledge our former and current colleagues in the Kid Space team at Intel Labs. Particularly: (i) Hector Cordourier Maruri, Juan Del Hoyo Ontiveros, and Georg Stemmer for developing the VAD-ASR node to obtain the ASR output that we use in our SLU pipeline; (ii) Benjamin Bair, Lenitra Durham, Sai Prasad, Giuseppe Raffa, Celal Savur, and Sangita Sharma for designing the HW/SW architectural setup, developing the Wizard UI and game logic nodes, and supporting data collection; (iii) Ankur Agrawal, Arturo Bringas Garcia, Vishwajeet Narwal, and Guillermo Rivas Aguilar for developing the Student UI via the Unity game engine; (iv) Glen Anderson, Sinem Aslan, Rebecca Chierichetti, Pete Denman, John Sherry, and Meng Shi for conducting UX studies and performing interaction design to conceptualize school and home usages; (v) David Gonzalez Aguirre, Gesem Gudino Mejia, and Julio Zamora Esquivel for developing the visual understanding nodes to support this research. We would also want to gratefully acknowledge our field team members from Summa Linguae Technologies, especially Rick Lin and Brenda Tumbalobos Cubas, for their exceptional support in executing the data collection and transcription/annotation tasks in collaboration with our Intel Labs Kid Space team. Finally, we should thank the Rasa team and community developers for their open-source framework and contributions that empowered us to conduct our research.

heading: A Appendix: Additional Error Analysis
text: Please refer to Table 8 for additional error analysis on ASR output from our home deployment data. Here, we compare manually transcribed utterances (i.e., human transcripts) with the speech recognition output (i.e., raw ASR transcripts) using five different ASR models that we investigated in this study. These ASR errors demonstrate the challenges faced in the speech recognition model performances on kids’ speech, which potentially would be propagated into the remaining modules in the conventional task-oriented dialogue pipeline.
We may attribute various factors to these speech recognition errors, often related to our deployment data characteristics. Incidental voices and phrases constitute a good chunk of the overall home deployment data, along with very short utterances to be recognized (e.g., stating names, colors, types of flowers, numbers, and binary answers with one-ortwo words), plus the remaining known challenges present with recognizing kids’ speech in noisy realworld environments.

heading: Future Work
text: In the future, we aim to finetune the Whisper ASR acoustic models on kids’ speech and language models on domain-specific math content. Moreover, we consider exploring N-Best-ASR-Transformers (Ganesan et al. , 2021) to leverage multiple Whisper ASR hypotheses and mitigate errors propagated into cascading SLU. Limitations
By building this task-specific dialogue system for kids, we aim to increase the overall quality of basic math education and learning at-home experiences for younger children. In our previous school deployments, the overall cost of the whole school/classroom setup, including the wall/ceilingmounted projector, 3D/RGB-D cameras, LiDAR sensor, wireless lavalier microphones, servers, etc. , can be considered as a limitation for public schools and disadvantaged populations. When we shifted our focus to home learning usages after the COVID19 pandemic, we simplified the overall setup for 1:1 learning with a PC laptop with a built-in camera, a depth camera on a tripod, a lapel mic, and a playmat with cubes and sticks. However, even this minimal instrumentation suitable for home setup can be a limitation for kids with lower socioeconomic status. Moreover, the dataset size of our initial home deployment data collected from 12 kids in 12 sessions is relatively small, with around 12 hours of audio data manually transcribed and annotated. Collecting multimodal data at authentic homes of individual kids within our target age group (e. g. , 5- to-8 years old) and labor-intensive labeling process is challenging and costly. To overcome these data scarcity limitations and develop dialogue systems for kids with such small-data regimes, we had to rely on transfer learning approaches as much as possible. However, the dataset sizes affect the generalizability of our explorations, the reliability of some results, and ultimately the robustness of our multimodal dialogue system for deployments with kids in the real world