Unsupervised Mismatch Localization in Cross-Modal Sequential Data with Application to Mispronunciations Localization
Authors: Wei Wei
Hengguan Huang
Xiangming Gu
Hao Wang
Ye Wang
Venue: Accepted by TMLR ()

Abstract: Content mismatch usually occurs when data from one modality is translated to another, e.g. language learners producing mispronunciations (errors in speech) when reading a sentence (target text) aloud. However, most existing alignment algorithms assume that the content involved in the two modalities is perfectly matched, thus leading to difficulty in locating such mismatch between speech and text. In this work, we develop an unsupervised learning algorithm that can infer the relationship between content-mismatched cross-modal sequential data, especially for speech-text sequences. More specifically, we propose a hierarchical Bayesian deep learning model, dubbed mismatch localization variational autoencoder (ML-VAE), which decomposes the generative process of the speech into hierarchically structured latent variables, indicating the relationship between the two modalities. Training such a model is very challenging due to the discrete latent variables with complex dependencies involved. To address this challenge, we propose a novel and effective training procedure that alternates between estimating the hard assignments of the discrete latent variables over a specifically designed mismatch localization finite-state acceptor (ML-FSA) and updating the parameters of neural networks. In this work, we focus on the mismatch localization problem for speech and text, and our experimental results show that ML-VAE successfully locates the mismatch between text and speech, without the need for human annotations for model training.

Published in Transactions on Machine Learning Research (12/2022)
Unsupervised Mismatch Localization in Cross-Modal Sequential Data with Application to Mispronunciations Localization
Wei Wei
wei.weilr..co mus. edu
National University of Singapore
Hengguan Huang 1
buang.hengguuee nu.nus. nus. edu
National University of Singapore
Xiangming Gu
riangming Sa. mus. edu
National University of Singapore
Hao Wang
hwu.88@es..utgeers..d edu
Rulgers University
Ye Wang National University of Singapore
wangye@@comp.nus. edu.
Reviewed on OpenReview: https: openreview. net/ forum? id- 29V0no7jKp
Abstract
Content mismatch usually occurs when data from one modality translated to another, e-g- language learners producing mispronunciations (errors in speech) when reading sentence (target text) aloud. However, most existing alignment algorithms assume that the content involved in the two modalities is perfectly matched, thus leading to difficulty in locating such mismatch between speech and text. In this work, we develop an unsupervised learning algorithm that can infer the relationship between content-mismmatceed cross-modal sequen- tial data, especially for speech-text sequences. More specifically, we propose a hierarchical Bayesian deep learning model, dubbed mismatch localization variational autoencoder (ML- VAE), which decomposes the generative process of the speech into hierarchically structured latent variables, indicating the relationship between the two modalities. Training such model is very challenging due to the discrete latent variables with complex dependencies involved. To address this challenge, we propose novel and effective training procedure that alternates between estimating the hard assignments of the discrete latent variables over a specifically designed mismatch localization finite-state acceptor (MI-FSA) and updating the parameters of neural networks. In this work, we focus on the mismatch localization problem for speech and text, and our experimental results show that AL-VAE successfully locates the mismatch between text and speech, without the need for human annotations for model training
1 Introduction
Sequential data is prevalent in daily life and usually comes in multiple modalities simultaneously, such as video with audio, speech with text, etc. Research problems about multi-modal sequential data processing have attracted great allention, especially on the alignment problem, such as aligning a video footage with
ihheee authors contributed equally this work.
1 Codes will available htttts://githuubb www/weivei-ieeeeeeeeieeeeeeeeeee wwwp/eeiveiin//M-VVVaa

---

Published in Transactions on Machine Learning Research (12/2022)
actions (Song et al. 2016) or poses (Kundu et al. 2020 aligning speech with its text scripts Chung et al, 2018), aligning audio with tags Favoryy et. al. 2021), elc.
The content mismatch could come from various aspects. For example, mispronounced words in speech or incorrect human annotation will cause mismatch between speech and text; actors not following scripts will cause mismatch between the video and the pre-scripted action list. Locating such content mismatch an important task and has many potential applications. For example, locating the content mismatch between speech and text can help detect misproninciations produced by the speaker, which is crucial for language learning.
In this work, we focus on the problem of mismatch localization between speech and text inputs. In other words, i.e,, to locate the mispronunciations in the speech produced by a speaker. Such mispronunciation localization task plays an important role in real-world applications such as computer-aided language learning system, where the mispronunciation localization results can be used provide multimodal foedback for the users. Most existing cross-modal alignment algorithms assume that data from variely of modalities are matched to each other (Song et al., 2016 Kundu et al.) 2020 Chung et al. 2018 Favory et al., 2021 ; this strong assumption leads to difficulty in locating the mismatch between the two modalities. In particular, recent speech-text aligmment approaches (Kürzinger et al. 2020; McAuliffe et al.
2017 mostly work under the assumption that the speaker has correctly pronounced all the words and are therefore incapable of detecting mispronunciation. Although earlier studies Finke & Waibel (1997); Hazen (2006); Braunschweiler et al. (2100;; Bell & Renals (2015) have considered the mismatch between speech and text, they require human- -nnoa-td speech to train an acoustic model. However, labeling such data is labor intensive and expensive. Similarly, traditional mispronunciation detection methods (Leung et al., 2019) also need to be trained on large number of human-annolated speech samples from second language (L2) speakers, whose annotation process is even more timeccinsuning and requires professional linguists' support. Furthermore, these studies on mispronunciation detection can only detect which phonemes/wer /words in the text input are mispronounced, without locating them in the speech.
To address these issues, we propose a hierarchical Bayesian deep learning model, dubbed mismatch localiza- tion variational autoencoder (ML-VAE), which aims at performing content mismatch localization between cross-modal sequential data without roquiring any human annotation during the training stage. Our model is hierarchically structured variational autoencoder (VAE) containing several hierarchically structured dis- crete latent variables. These latent variables describe the generative process of speech from L2 speakers and indicate the relationship between the two modalities.
One challenge for MI-VAE is that training such an architecture very difficult due to the discrete latent variables with complex dependencies. To address this challenge, we propose novel and effective training procedure that alternates between estimating the hard assignments of the discrete latent variables over a specifically designed finite-state automaton (FSA) and updating the parameters of neural networks.
The main contributions of our work include:
We propose a hierarchical Bayesian deep learning model, ML-VAE, to address the problem of content mismatch localization from cross-moda sequential data.
Our ML-VAE is the first method that bridges finito-state automata and variational autocncoders;
this is achieved via our proposed mismatch localization finite-state acceptor (ML-FSA), which allows the VAE to locate the mismatch by searching for the best path in MT-FSA.
To address the challenge of inferring the latent discrete variables with complex dependencies involved
in MAE, we propose a novel and effective alternating inference and learning procedure.
We apply MI-VAF to the mispronunciation localization task; experiments on a non-native English corpus to demonstrate
effectiveness in terms of unsupervisedly locating the mispronun- ciation segments in the speech.
2 Related Work
Gross-Modal Sequential Data Alignment There have been several studies on aligning sequential data from different modalities. For example, for video-action alignment, most existing work focuses on learning
2

---

Published in Transactions on Machine Learning Research (12/2022)
spatio-temporal relations among the frames. Dwibedi et al. (0199) proposes self-supervised learning ap-
proach named temporal cycle consistency learning to learn useful features for video action aligmment. Liu
lt al. (2021) proposes to learn a normalized human pose feature to help perform the alignment task. [Song
ct al. 2016) adopts an unsupervised way to align the actions in a video with its text description. In terms of
speech- alignment, a recent study by Kürzinger et al. (2000) proposes using the CTC output to perform
speech-text aligument. However, these aforementioned studies assume a perfect match in the content of the cross-modal sequential data, making it impossible locate the content mismatch from the data.
Speech-text alignment is usually referred to as the forced alignment (FA) task (McAuliffe et al. 2017) in the speech processing community. There is fairly long history of rescarch related to FA. Traditional method is to run the Viterbi decoding algorithm (Forney] 1973) on a Hidden-Marrkv--oodde based (HMM-based) acoustic model with a given text sequence. Even though several FA studies have considered the particular problem setting where the text is not perfectly matched to the speech, they require training an acoustic model using a large amount of human-annotated speoch, which is laborious and costly. For instance, Finke
& Waibel (1997); Moreno et al. (1998); Moreno & Alberti (2009); Bell & Renals (2015); Stan et al. require acoustic model for text-to-speech alignment with a moditied lattice, and Bordel et al.
(2012)
(0112)
requires an acoustic model to obtain the recognized phoneme sequence. Therefore, these methods fail to handle our problem setting, where the speech data from L2 speakers is unlabelled.
Variational Autocncoders and Bayesian Deep Learning Variational autocncoder (VAE) (Kingma & Welling 2013) is proposed to learn the latent representations of real-world data by introducing latent vari- ables. It adopts an encoder to approximate the posterior distribution of the latent variable and and decoder to model the data distribution. VAËs have wide range of applications, such as data generation (Mescheder
ct al. 2017), representation learning (Oord et al. 2017), , etc. As an extension to VAE to handle sequential data (e.g., speech), the variational recurrent neural network (VRNN) (Chung et al.] 2015) introduces latent variables into the hidden states of a recurrent neural network (RNN), allowing the complex temporal depen- dency across time steps to be capturod. Along a similar line of research, [Johnson et al. (2016) proposes the structured VAE, which integrates the conditional random field-like structured probability graphical model with VAFs to capture the latent structures of video data. Factorized hierarchical variational autoencoder (FHVAE) (Hsu et al.] 2017) improves upon VRNN and SVAE through introducing two dependent latent variables at different time scales, which enables the learning of disentangled and interpretable representations from sequential data. However, the hierarchical models mentioned above are trained by directly optimizing the evidence lower bound (ELBO), which may fail when discrete latent variables with complex dependencies are involved. To address this issue, we propose a novel learning procedure to optimize our ML-VAE.
To deal with data from different modalities, Jo al. (2019) proposes a cross-modal VAE to capture both intra-modal and cross-modal associations from input data. Theodoridis et al. (2220) proposes VAE-based method to perform rrosl alignment of latent spaces. However, existing work on processing cross-modal data focuses mainly on learning the relationship between modalities while ignoring the potential mismatch between them. Therefore, we propose the ML-VAE to address this issue. In general, our work belongs to
the category of Bayesian deep learning (BDL) (Wang & Yeung. 2016 2020 Wang et al., 2015 Huang et al. 2021 2022), and is the first. BDL method that performs mismatch localization.
2020
3 Problem Formulation
Here, we propose to generalize the traditional speech-text alignment problem by considering the content mismatch between the input sequences (see Fig. 1.. Specifically, given two cross-modal sequences: (1) a speech feature sequence X' = (x1,..,, IT) (x, E RDi where Dy is the feature dimension) as the source sequence, and (2) a phoneme sequence C = (C1, cL) (a E P, where P is the phoneme set) as the target sequence, our goal is to identify the mismatched target elements of C (i.e., mispronounced phonemes) while locating the corresponding clements in the source sequence (i.c. incorrect speech segments). Concretely, let C' = (c::-.,,(() (L) be the mismatch--cdentiiie larget sequence. We use the notation of to present. mismatched content; then of of if of is identified as mismatched content (i.e., mispronounced phoneme) and & e if oi is identified as matched content (i.c., a correctly pronounced phoneme). The final localization result Ć (c,,.., .r) is therefore a repeated version of C' with each element. of repeating for di times, indicating that
3

---

Published in Transactions on Machine Learning Research (12/2022)
Audio wave /
Speech feature sequence X
Input
Phoneme sequence C
aa.
Mismatch localization result Ć I I
aa aa aa
Output
Figure Problem formulation.
of lasts for di frames. The mismatch localization performance is evaluated by averaging the interscction-over- union for each successfully detected mismatched phoneme, with more details to be introduced in Sec. 8.1
4 Model
ML-VAE is a hierarchical Bayesian deep learning model (Wang & Yeung 2020) based on VAE (Kingma & Welling 2013). Our model aims performing content mismatch localization when aligning cross-modal sequential data. In this work, focus the speech-text mismatch localization task.
This is made possible by decomposing the generative process of the speech into hierarchically structured latent variables, indicating the relationship between the two modalities.
ML-VAF is designed to use a hierarchical Gaussian mixture model (GMM) to model the match match/mismatch mismatch between the sequences. Each Gaussian component corresponds to a type mismatch, and the selection of the Gaussian component depends on the discrete latent variables of ML-VAE.
Latent Variables and Generative Process In the context of speoch-text mismatch localization, since both the mismatch-identiffed phoneme sequence C' and the duration of each phoneme are unobservable, we introduce several latent variables to achieve the goal of mismatch localization for speech-text sequences. Instead of using duration variable to describe the duration of each phoneme, variable sequence B = (b)....bby, where b, means the t-th frame marks the start of phoneme segment, and thus Σt= b L. Besides, binary correctness variable sequence II (M ( TT) is introduced to describe the matched/ peetleed/mmssaattttt mismatt content in speech. Each entry n {0, 1}, with = if the I-th speech frame contains mismatched content (i.e. mispronounced phoneme), and ग. otherwise.
use binary boundary
To model the data gencrative process, at each time step t, we introduce three more latent variables: 1) yes which denotes the estimated phoneme, 2) zt, which represents the Gaussian component indicator, and 3) he which is the speech latent variable.
As shown in Fig. 2 draw the estimated phoneme Ji from categorical distri- bution, and draw the cor- rectness variable Te from a Bernoulli distribution. Mean- while, the boundary variable b, assumed to be drawn from Bernoulli distribution Bernoulli(n)) which is fur- ther parameterized by a Beta distribution Bela(a,8) B).
5
yt
T
Zt
ht
Lt
θ
α
O
bt
nt
B
T
K
Figure 2: Graphical model for ML-VAE
In the GMM part of MI-VAE, 24, indicating the index of the selected Gaussian distribution, one-hot variable drawn from categorical distribution, and he, the latent
4

---

Published in Transactions on Machine Learning Research (12/2022)
variable of speech, is drawn from the selected Gaussian distribution. For each phoneme type, component to model the matched content (i.e., correct pronunciation) and No. components to model different mismatch types (i.e., mispromunciation variants). Such a design is motivated by the following observations: all correct promunciations of the same phoneme are usually similar to each other, while its mispronunciations often severely deviate from the correct one and have multiple variants (e.g., the consonant 'DH' may be mispronounced into 'T', 'D', or 'TH). Thercfore, for N phoneme types, there are N components modeling correct pronunciations and N Nm components modeling mispronunciations, making the GMM part of ML-VAE contain total number of K N + N Nm components.
Given the design introduced above, the generative process of ML-VAE is as follows:
For the 1-th frame - 1, ..2....777.
Draw the estimated phoneme ~ Calegorical(()-
Draw the boundary variable b ~ Bernoulli(y).
Draw the correctness variable nΙ Bernoulli(ne): where tp Beta(a, 9)-
With Vt, b,, and Te. draw from ze|Ut, Te, by (f b,, Te)). where fz(+) denotes learnable neural network.
Given that [A[A]] = 1, select the k-th Gaussian distribution and draw he|ze (un, of)
Draw zelhi fo where fx(+) and fo (-) denote learnable neural networks.
The parameters of the prior distributions of the latent variables (e.g., a, 3) in ML-VAE are determined by the training data.
An Example We will take an audio recording of reading the word 'NOT" as an example to explain the generalive process. More specifically, the phoneme sequence contains three phonemes: n, aa, and The estimated phoneme Yt sampled to estimate the phoneme pronounced by the speaker each time step, The boundary variable is sampled determine whether the speaker starts pronounce the next phoneme at time step 1 (e.g., from to aa). The correctness variable Te is then sampled to determine whether this phoneme (e.g., aa) is correctly pronounced (e.g., whether aa is mispronounced as another phoneme). Based on Yt, bt, and Tt, if the phoneme (e.g., aa) is correctly pronounced, the Gaussian component indicator will select the Ganssian distribution for the correct prommciation to generate the speech latent variable hy: otherwise, based on how the phoneme is mispronounced (e.g, aa mispronounced as ae), 2t will select the corresponding Gaussian distribution generate the speech latent variable he-
Model Architecture ML-VAE contains three components: boundary dctector Φb, phoneme estimator Φp. and speech generator Φn, as shown in Fig. The boundary detector outputs the approximated posterior don (belax), and the phoneme estimator outputs the approximated posterior 4e, (ve(2.). The speech generator aims to reconstruct the input speech feature sequence following the generation process discussed in Sec.
More details introduced in Sec. 6.2
yt
Legend
2t
phoneme estimator
neural network
Øt
n
speech generator
+: data flow
ht
Lt
boundary detector
sampling
7t
b
concatenation
Figure 3: Model architecture for ML- VAE.
5

---

Published in Transactions on Machine Learning Research (12/2022)
5 Mismatch Localization Finite-Stafe Acceptor
In this section, we describe our proposed mismatch localization finite-state acceptor (MI-FSS) that bridges finite-state automata and variational autoencoders, allowing our ML-VAE to locate the mispronounced segment in the speech by efficiently searching the best path in ML-FSA. Our ML-FSA special type of finite-state acceptor (FSA) that describes possible hypothesis of mispronunciations and phoneme boundaries, We show ML-FSA for the 1-th phoneme, a, in the given phoneme sequence in Fig. Such phoneme-level MI-FSA contains set of states and set of stale-lo-slate transitions; the initial state 0 and the accepling state are represented by a bold circle and concentric circles respectively, each transition includes source state, a destination state, a label and a corresponding weight. Spocifically, from the initial state (state 0), it can transit to state 1 or 3 based on promunciation correctness (R stands for correct promunciation and W stands for mispronunciation). This further leads to two different paths, one for correct pronunciation (denoted by q) leading to state 2, and the other one for mispronunciation (denoted by of) leading state 4. In each path, at each time step, since each phoneme may last for several frames, it either still holds the current state (denoted by H), or moves forward to the final state 5 (denoted by S)-
With the help of the phoneme sequence C, we can naturally build a sentence-level ML-FSA by combining the corresponding MI-FSA for each phoneme. With the sentence-level ML-FSA, a dynamic programming (DP) algorithm can be applied to search for the optimal path based on the weights on the transitions. More specifically, the weight of cach transition in MI-FSA can be estimated by the modules in ML-VAE, with more details to be introduced in Sec.
In contrast to weighted finite state transducers (WFSTs) Mohri et al. (2002), which only model the segmentation of audio into acoustic units (e.g., phonemes), our ML-FSA models the correctness of promunciation (R and W), the phoneme type (c,, d), and the phoneme boundary (H and S), which can jointly detect and segment the mispronuncia- tion in the speech and thus perform unsupervised content mismalch localization. Furthermore, WF- STs require training supervised acoustic model for Figure 4: Proposed ML-FSA for the I-th phoneme, q weighting transitions between states, while our ML- FSA adopts the unsupervised MI-VAE calculate all the weights of transitions. The details on locating the mispronunciation using ML-FSA are described in the next section. Note that ML-FSA takes all possible mispronunciation variants as single symbol (ci), which trades fidelity for efficiency. This is advantageous for addressing problem settings: (1) our task focuses distinguishing between the correct pronunciation and mispronuncialion, rather than identifying the mispronunciation variants; (2) combining mispronuncia- tion variants for each phoneme can dramatically reduce the computational cost of the dynamic programming (DP) algorithm scarching for the optimal path.
R
1
G
2
S
II
W
H
S
q
3
6 Learning
The learning of ML-VAE consists of two main components: latent variables V = (,, B,113 and neural network parameters + { db, Φh), where y (y1,..., W)), and &p. Φb; and dn denote parameters of the three modules of MI-VAF: phoneme estimator, boundary detector, and speech generator, respectively.
Following the traditional variational inference and the training objective for FHVAE (Hsu et al. 2017), the ELBO for the joint training objective of ML-VAE can be written as:
T
ELBO- = Σ B log p(xelye, br, Tt]] (g(6e by क [e(e))llp(( by, RT))
(1)
where DKL the function to calculate the Bullback-Leible (KL) divergence, qlve b, n x((2;) is the joint approximate posterior distribution of the latent variables, p(ye, be, me) is the joint prior distribution.
6

---

Published in Transactions on Machine Learning Research (12/2022)
However, unlike FHVAE, learning ML.VAE with such an objective function is very difficult due to ML-VAE's discrete latent variables with complex dependencics, as pointed out by Rolfe (2016). Furthermore, empirically tested an intuitive approach using hard expectation- maximization (EM) algorithun (Moon 1996) and found that the system would reliably converge. As pointed out by Locatello et al. (2019),
Step 1.1 max ()));
Step
max JBn [8,11:4)
Φb
Bll
B
D
p
Step 2.3 min (b)i
Step 2.2 min Li (bb;;
B
C
Step 2.1 min Lp (0p;C)
Figure 5: The training framework of ML-VAE.
lacking inductive bias for unsupervised training could render the model unidentifiable. Therefore, improve this approach by providing pscudo-labels to some latent variables, as introduced by Khemakhem jet al. (2020). Note that our learning algorithm remains unsupervised since does not require any external data other than and Our new training procedure (see Fig. can be described as:
Step 1 (E step). Given the model parameters 4 estimate the hard assignments of the latent variables = {}, B, fi} by:
Step 1.1. Given 6p, estimate D.
Step 1.2. Given b, estimate B and îT.
Step 2 (M step). Given the hard assignments of the latent variables V optimize the model parameters by:
Step 2.1.
Given the phoneme sequence C as the training target, optimize op Step 2.2. Given a forced alignment result B, optimize és-
Step 2.3. Given the estimated hard assignments & latent variables, optimiie dn.
Our approach differs from the standard hard EM algorithm in several aspects. First, since the phoneme sequence C is given under our problem setting, the phoneme estimator dpp can be trained by directly optimizing the cross-entropy loss towards C (Step 2.1). Second, can directly adopt the predictions of Φp to obtain y (Step 1.1). Third, we design an FSA to assist approximating a maximum & posteriori (MAP) estimate of B and ii (Step 1.2). Fourth, we find that directly using B from Step 1.2 deteriorates the model performance; therefore in Step 2.2, we adopt a forced aligmment result B (Tebelskis) 1995 McAuliffe et al. 2017) to train the boundary detector Φb. It is worth noting that obtaining such an alignment in Step 2.2 does not require any external data other than X' and C.
6.1 Step 1: Estimation the Hard Assignments v
Next, we discuss the details of estimating the hard assignments.
Estimation of ) We obtain the estimated phoneme sequence ý with:
5 argmaax Jy(D;; Jy op) = argmaax do (2122)
(2)
where 96,(0' (V1x') the variational distribution calculatod by the phoneme estimator Op approximate the intractable true posterior p()1)2).

---

Published in Transactions on Machine Learning Research (12/2022)
Estimation of B and II via ML-FSA The estimation of B and Iî is done by finding the optimal path in the sentence-level ML-FSA. As introduced in Sec. 5 the weights of transitions of MT-FSA are all estimated by the model ML-VAE. More specifically, from state O, the weights of the two transitions, labeled with At and W, are estimated by the approximate posterior Ios (T. = z. yt, by) and go. (Te = 1| ye, by) respectively. Afterwards, the weights of the next Iwo transitions, labeled with oj and ej are imated by the posterior ge, (y/ a)(2)) and qe, (y: o q (2)))) respectively. Finally, the weights of the last two transitions, marked with H and S, are estimated by the posterior qo, (b. - 0|æ²) and Góo (b, 1/2)) respectively. The details of how these posteriors are estimated will be introduced in Sec. 6.2
Given all the weights in the sentence-level ML-FSA available, typical dynamic programming (DP) algorithm can be used to find the optimal path which yields the maximum probability, whereby B and II are backtracked along the optimal DP path. Meanwhile, the optimal DP path also yields the mismatch localization result č. More details are given Appendix A
6.2 Step 2: Learning Model Parameters 4
In this section, we present how ML-VAE's parameters + = (6p. 0s. Φn are learned given the estimated T obtained from the previous stage. Implementation and parameterization details can be found in Appendix Different objectives are proposed for 6p. Φb, and th.
6.2.1 Boundary Detector
The boundary detector takes the speech feature sequence X' as imput and outputs the probability qa, (z). The boundary variable by is drawn from a Bernoulli distribution parameterized by an auxiliary latent variable Th that is, b ~ Bernoull(n:), where we assume T follows a Beta distribution: T Beta(a, 3). Therefore here we models an auxiliary distribution go, (1)l.)) As introduced, we use the forced alignment result sequence B (b,..., br) as the pseudo-label to aid the training process. The training objective minimize the loss Ls:
Co(or: ()
Σ
(ess, (nd (m)) pog = =b,lne)] > Da (ne (n)l)))))l))
(3)
where Å is a hyperparameter controlling the weight of the KL term, 4o (n (e)) is the approximate posterior distribution, and p(m) is the prior distribution of nh-
Note that no human annotation is required when obtaining the forced alignment result sequence B, since B is oblained with X' and as inputs without using any human annotation of mispronounced phonemes
6.2.2 Phoneme Estimator op
Similar to the boundary detector, the phoneme estimator takes the speech feature sequence X' as input and outputs the probability 2,,(y (y x.). We use the pseudo label Č = (?,, .T) as our training target, which can obtained by extending the phoneme sequence C according to the estimated duration of each phoneme in B. Therefore, the model is optimized by minimizing the nogative logllikcioooo::
Cp(0p:
Σ loglus, (yr = 2e (x:))
(4)
6.2.3 Speech Generator dh
The speech generator dh aims to reconstruct the input speech feature sequence. It takes the speech feature sequence X, along with the estimated values the latent variables y and 8 as input and reconstructs the speech feature sequence following the gencration process discussed in Soction 4
We the variational inference learn the speech generator da and thus the ELBO loss is calculated by:
8

---

Published in Transactions on Machine Learning Research (12/2022)
T (bn) = E
[gs, (hellr)
[logpo, pox (ax( A, DKL. (yon (Ax/ (Melax))l
(5)
where ' controls the weight of the KL, term, gou (halar) is the approximate posterior distribution, and p(h.) is the prior distribution of he-
Besides, also augment the ELBO using the estimated correctness variable sequence fi obtained from Step (E. Step) as supervision signal and the new loss is written as:
Colon: 1) = (o(()) + (
(6)
where D is an importance weight; (on: V) is the negative log-likelihood loss, which is computed by:
T
Li(bn; 11)
Σ
log 2s. (TT t #e/ be):
(7)
where 9ds (Telze,Be is the approximate posterior distribution.
6.3 Overall Learning Algorithm
The overall algorithm to learn ML-VAE is shown in Algorithm 1
Algorithm Learning ML-VAE
Input: Speech feature sequence X, phoneme sequence C Mismatch localization result Ć 1: Initialize the model parameters Φp, db, and Φn-
Output:
2: Obtain the forced alignment result
while not converged do Estimate B and 11 with ML-FSA.
Using Eq. 4 optimize op with the phoneme sequence C.
Using Eq. 3 with the help of B, optimize db.
7: Given fi, optimize dn using Eq. 6
8: end while
9. Obtain the mismatch localization result č with ML-FSA.
10: return C
7 ML-VAE with REINFORCE Algorithm
In practice, we found that the training procedure sometimes not stable due to large gradient variance. We suspect this is due to the sampling process the discrete latent variables (such as TT). To overcome such an issue and better reason about the distribution of the correctness variable during training, we further propose variant of ML-VAE that uses the REINFORCE algorithm (Williams) 1992), which we call MI-VAF-RI..
Following the work by Xu et al. (2015), the reward term is defined as R(1I) = (o ④), and to further reduce the gradient variance, a baseline term b((²) is introduced to calculate the calibrated reward term R(II) = RII) - b(x). 6(2') is estimated by fully-connected neural network and trained by minimizing the mcan squared error (MSE) between b(2') and the uncalibrated reward R(II): MSE(b(A)), R(T)) 1 (ba) R(
Then we define the new optimization objective Ln((on ) whose gradient with respect the model param- eters du be calculated with the Monte Carlo method:

---

Published in Transactions on Machine Learning Research (12/2022)
N V Van Cri -==
Σ i=1
Ve Ca (b) R on log 9dn (II 1 B3]
Ven H
(8)
where H[ is the entropy the distribution of the correctness variable whose gradient can be calculated explicitly. Nmc is the number of Monte Carlo samples, and IT(() is the i-th sample drawn from the posterior distribution I
With the help of the REINFORCE algorithm, Step 7 in Algorithm is replaced by optimizing with Eq. so that Φh is reinforced to sample the correctnoss variable sequence II that produces lower La (dn: V).
8 Experiments
We evaluate our proposed MLEVAE and ML-VAE-RL on the mispronunciation localization task to test their mismatch localization ability. We first conduct experiments on a synthetic dataset Mfismnatch-AudioMMNSST) and then further apply our proposed models to a real-world speech-text dataset (L2-ARCTIC).
8.1 Evaluation Metrics
The F1 score is a commonly used metric to evaluate binary classification tasks, e.g., mispronunciation de- tection (Leung et al. 2019). However, we find that Fl score not suitable to evaluate the mispronunciation localization task, as computed without evaluating if the model successfully locates the detected mispro- nunciations in speech.
As such, we improve upon the traditional Fl score by proposing set of new evaluation metries. For each true positive (TP) case, we calculate the intersection over union (IoU) metric of its corresponding phoneme segment, which demonstrates the performance of localization. IoU is computed by IoU = t. where / and U are the interscction length and the union length obtained by comparing the detected phoneme segment with the ground truth phoneme segment. Then such IoU metrics of all TP cases are summnd and denoted as ML- After calculating TP MP, the TP in equations caleulate our proposed metrics. For example, if there are two TP cases detected by the model, then instead of calculating F1 score with TP 2, we calculate the IoU of these two cases' corresponding phoneme segment, e.g. 0.3 and 0.6 respectively. Then we calculate our proposed metrics with TPx1. 0.3 + 0.6. Our proposed metrics are denoted as mismatch localization precision (PRML), recall (REML), and F1 score (FIML).
calculate the Fl score is replaced with TPML
To be more specific, we first calculate the the intermediate metrics: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). Then for all the TP cases detected by the model, we take the sum of their corresponding segments' intersection over union (IoU), which is denoted as TPML. Then the TPM TPM., REMI. TF4FN, FI 2xPRM1 XREMI. PRML -REML
final metrics compuled PRMr. TP4FF:
8.2 Baselines
To knowledge, there is existing work specifically designed for mispronunciation localization. Therefore, adapt some existing methods related tasks (e.g. ASR) to perform mispromunciation localization, and compare our proposed two ML-VAI models with them:
FA (McAuliffe al. 2017), which performs forced alignment using a deep-nneurrl-neewwork-IMMMM based (DNN-HMM-basoe acoustic model.
Two-Pass-FA Tebelskis 1995), which first performs phoneme recognition based on DNN-HMM- based acoustic model and then performs forced alignment on the recognized phoneme sequence and input speech.
w2v-CTC (Baevski al. 2020 which is built with wav2vec 2.0 and trained with CTC loss; CTC-segmentation (Kürzinger al. 2020) is used to align the recognized phonemes with speech.
10

---

Published in Transactions on Machine Learning Research (12/2022)
All models above trained under the same problem setting where no human annotated data is used, i.c. only the speech feature sequence X' and phoneme sequence C which contains mismatch are used during the training process.
8.3 Synthetic Dataset:
We first lest our proposed models on a synthetic dataset, named as Mismatch--AudiMMNIIS,,, which is built based on the AudioMNIST (Becker al.) 2018) dataset. Our synthetic dataset contains 3000 audio samples, each produced by concatenating three AudioMNIST. To simulate the content mismatch between audio and the corresponding text annotations, we randomly select 20.1% of the spoken digits as mismatched content, thereby labeling them as random digits. The total duration of Missmatch-AudioMMNN ANdio is 2.67 hours, The dataset is split into training, validation, and test sets by a 60:20:20 ratio, and the durations of the three sets are 96.3, 33.2, and 30.6 minutes respectively.
spoken digits randomly selected from the original
Mispromunciation Local- ization Results Table [ shows the mispronunciation localization results of ML- VAE and MI-VAE-RL on the synthetic dataset Mismatch- ANISO We can see that all three baselines perform poorly for our mismatch lo- calization task. More specifi- cally, the vanilla forced align- ment model (FA) fails to locate any mispronunciations due its assumption that all the digits are correctly much superior to the baseline pronounced. Both of our proposed models (MI-VAE and MI-VAF-RI.)
Table 1: Mispronunciation localization results on Mismatch-AudioMNNSS
Model
# Params
PRML%
REML %
Fl %
FA
3.3M
0.00
0.00
0.00
Two-Pass-FA
3.3M
6.22
2.43
2.28
w2v-CTC
352.5M
1.72
3.84
2.30
ML-VAE (ours)
24M
28.42
27.60
27.67
ML- VAE-RE (ours)
25M
30.67
30.32
30.28
models, Two-Pass-FAA and w2v-CTC, demonstrating the effectiveness of the proposed method in handling the unsupervised mismatch localization task. Note that in our problem settings, our training data con- tains mislabeled spoken digits and such mislabelling is unknown. This poses great challenges for acoustic model training. By further looking into the recognition results of Two-Pass-FA and w2v-CTC, we find that their recognition results from the acoustic model are very poor terms of the mislabeled spoken digits, consequently limiting the baseline models' ability detect such mislabels in the second stage processing.
8.4 Rerl- Dataset: L2-ARCTIC
We further apply ML-VAIE and ML-VAE-RL to real-world dataset: the L2-ARCTHH dataset (Zhao et al. 2018), which is a non-native English corpus containing 11026 utterances from 24 non-native speakers. Note that to evaluate mismatch localization, each annotated phoneme in the dataset needs to come with an onset and offsct time label; to the best of our knowledge, the L2-ARCTIC dataset is currently the only real-world dalasel suitable for this problem selting.
Mispronunciation Lo- calization Results As shown Table 2 similar to the results Mismatch- AudioMNIST, Our ML-VAE
Table 2: Mispromunciation localization results on L2-ARCTIC.
Model
# Params
PR %
REML
FIME %
models dramatically out- perform Two-Pass-FA and w2v-CTC in all Uhree met- rics. Compared with original ML-VAE, the variant with REINFORCE algorithm
FA
3.3M
0.00
0.00
0.00
Two-Pass- Pao-Pass-FA
3.3M
0.64
0.96
0.77
w2y-CTC
352.5M
1.29
2.26
1.64
ML-VAE (ours)
24M
6.46
11.97
8.39
ML- VAE-RL (ours)
25M
8.20
13.57
10.22
yields better localization performance of mispronunciation.
11

---

Published in Transactions on Machine Learning Research (12/2022)
Interestingly, though w2v-CTC is based on powerful wav2vec 2.0 acoustic model, which shows superior performance for ASR, it does not work well on the mispronunciation localization task; even with the largest number of parameters, it only slightly outperforms Two-Pass-FA, and underperforms our ML-VAE by large margin. Our further analysis on w2v-CTC's output shows that such poor performance is mainly caused by the dataset which contains unknown content mismatches between the speech data and the corresponding text; w2v-CTC is fine-tuned on the speech data with mispronunciations, and therefore its recognition results tend to contain mispronunciations as well. For example, if speaker always mispronounces the phoneme 'b' as 'd', w2v-CTC model trained with such data tends to predict 'd', instead of 'b' during inference. Another reason for w2v-CTC's poor performance is that the CTC-segmentation algorithm assumes correct pronunciations and therefore does not work well on speech-text sequences containing mispronunciations
Some more experiments are performed to demonstrate how traditional alignment methods fail on speech and text inputs with mispronunciations and such experimental results can be found in Appendix F
We present ablation study results to demonstrate the effectiveness of our proposed learn- ing algorithm. Table 3 shows the results of ML-VAE-RL optimized with B instead of B. The first row shows that directly using B leads to very poor mispromuneiation localization performance. The second row shows that our proposed optimization method effectively improves upon the traditional joint optimization method.
Ablation Study
Table 3: Ablation study results.
Model
PRML. %
RFML% %
FI %
Ablation 1: ML- VAE-RL using B instead of B for optimization
2.31
1.53
1.84
Ablation 2: HIL-VAE-RL w/ joint optimization
4.90
2.89
3.64
Ablation 3: ML-VAE-BE w/ B y estimated separately
7.31
10.85
8.73
ML-VAE (ours)
6.46
11.97
8.39
MI-VAE-RL (ours)
8.20
13.57
10.22
The third ablation study compares estimation method with traditional one, which estimates and fl separately based on single-path FSA. Details on this traditional algorithm can be found in Appendix B Compared with this ablation model, our proposed ML-VAF-RI, yields better results. This proves the effectiveness of our proposed estimation algorithm, which specifically designed for the mismatch localization task.
9 Conclusions and Future Work
In this work, we present a hierarchical Bayesian deep learning model address the mismatch localization problem in cross-modal sequential data. More specifically, two variants are proposed, namely MT-VAF and MLL-VAE-RL. We also propose a learning algorithm to optimize our proposed ML-VAE. We focus on applying ML-VAE to the speech-text mismatch localization problem and propose set of new metrics to evaluate the model's mispronunciation localization performance. Our experimental results show that it can achieve superior performance than existing methods, including the powerful wav2vec 2.0 acoustic model. We also perform ablation studies to verify the effectiveness of our training algorithm. One of the future research directions of this project how to extend ML-FSA to address the case where multiple correct pronuncations exist.
10 Acknowledgement
We would like to thank our action editor Brian Kingsbury and the reviewers for their detailed and constructive comments and foedback. HW is partially supported by NSF Grant ITS-2127918 and an Amazon Faculty Research Award. This project is funded in part by a research grant A-0008153-00-00 from the Ministry of Education in Singapore.
12

---

Published in Transactions on Machine Learning Research (12/2022)
References
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.114 2020.
Sören Bocker, Marcel Ackermann, Scbastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. In- terpreting and Explaining Deep Neural Networks for Classification of Audio Signals. 2018. URL
http://arxiv.. wrw. abaaaaaa//////// 03418
Peter Bell and Steve Renals. A system for automatic alignment of broadcast media capt ions using weighted finite-state transducers. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 675 680. IEEE, 2015.
Germán Bordel, Mikel Penagarikano, Luis Javier Rodrriguez-Puentes, and Amparo Varona. A simple and efficient method to align very long speech signals to acoustically imperfect transcriptions. In Thirteenth Annual Conference of the International Speech Communication Association, 2012.
Norbert Braunschweiler, Mark J F Gales, and Sabine Buchholz. Lightly supervised recognition for automatic alignment of large coherent speech recordings. In Eleventh Annual Conference of the International Speech Communicattoo Associalion, 2010.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980 2988, 2015.
Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass, Unsupervised cross-modal alignment of speech and text embodding spaces. arXiv preprint arXiv:1805.0744 2018.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In CVPR, 2019.
Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, and Xavier Serra. Learning contextual tag embed- dings for cross-modal alignment of audio and lags. In ICASSP, 2021.
Michael Finke and Alex Waibel. Flexible transcription alignment. In 1997 IEEE Workshop Automatic Speech. Recognition and Understanding Proceedings, pp. 34-40. IEEE, 1997.
G David Forney. The viterbi algorithm. Proceedings of the TEFE, 61 (3): 278, 1973.
Timothy J Hazen. Automatic aligmment and error correction of human generated transcripts for long speech recordings. In Ninth International Conference on Spoken Language Processing, 2006.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable rep- resentations from sequential data. In Advances in newral information processing systems, pp. 1878 1889, 2017.
Hengguan Huang, Fuzhao Xue, Hao Wang, and Ye Wang. Deep graph random process for relational-thinking based speech recognition. In ICML, 2020.
Hengguan Huang, Hongfu Liu, Hao Wang, Chang Xiao, and Ye Wang. Strode: Stochastic boundary ordinary differential equation. In ICML, 2021.
Hengguan Huang, Xiangming Gu, Hao Wang, Chang Xiao, Hongfu Liu, and Ye Wang. Extrapolative contimuous-time bayesian neural network for fast training-froe froe test-time adaptation. 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.0114. 2016.
Dae Ung Jo, BycongJu Lee, Jongwon Choi, Haanju Yoo, and Jin Young Choi. Cross-modal variational auto-encoder with distributed latent spaces and associators. aixiv preprint Xiv: 1905. 12867, 2019.
13

---

Published in Transactions on Machine Learning Research (12/2022)
Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. Advances in neural information processing systems, 29:2946 2954, 2016.
Nyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In AISTATS, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Jogendra Nath Kundu, Ambarcosh Revanur, Govind Vitthal Waghmare, Rahul Mysore Venkatosh, and R Venkatesh Babu. Unsupervised Cross-Modal Alignment for Multi-Person 3D Pose Estimation. ECCV, 2020.
Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. Cle-segmentation large corpora for german end-to-end speech recognition. In SPECOM, 2020.
Tao Lei, Yu Zhang, and Yoav Artzi. Training rmss as fast as cnns. 2018.
Wai-Kim Leung, Xunying Liu, and Helen Meng. CNN-RNN-CTC based end-to-end mispronunciation detec- tion and diagnosis. In TCASSP, 2019.
Jingyuan Liu, Mingyi Shi, Qifeng Chen, Hongbo Fu, and Chiew-Lan Tai. Normalized Human Pose Features for Human Action Video Alignment. In ICCV, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gumnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. ICML, 2019.
Michael McAuliffe, Michacla Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal
Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In INTERSPEECH, 2017.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In ICML, 2017.
Mchryar Mohri, Fernando Percira, and Michael Riley. Weighted finite-state transducers speech recognition. Computer Speech |& Language, 16(1):69 88, 2002.
Todd K Moon. The expectation-maxiiizatioo algorithm. IEEE Signal processing magazine, 13(6):47-60, 1996.
Pedro J Moreno and Christopher Alberti. A factor automaton approach for the forced aligmment of long speech recordings. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4869-4872. IEEE, 2009.
Pedro J Moreno, Christopher F Joerg, Jean-Manuel Van Thong, and Oren Glickman. A recursive algorithm for the forced alignment of very long audio segments. In ICSLP, volume 98, pp. 2711-2714, 1998.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXio preprint arXie:1711.00933 2017.
Jason Tyler Rolfe. Discrete variational autoencoders. XrXiv preprint arXie:1609.cm 022.00022000..000 2016.
Young Chol Song, Iftekhar Naim, Abdullah Al Mamun, Kaustubh Kulkarni, Parag Singla, Jicbo Luo, Danicl Gildea, and Henry A Kautz. Unsupervised Aligument of Actions in Video with Text Descriptions. In IJCAI, pp. 2025-2000 2235-231,-41 2016.
Adriana Stan, Peter Bell, and Simon King. A grapheme-based method for automatic alignment of speech and text data. In 2012 IEEE Spoken Language Technology Workshop (SLT), pp. 286 290. IEEE, 2012.
Joseph Michael Tebelskis. Speech recognition using neural networks. Carnegie Mellon University, 1995.
14

---

Published in Transactions on Machine Learning Research (12/2022)
Thomas Theodoridis, Theocharis Chatzis, Vassilios Solachidis, Kosmas Dimitropoulos, and Petros Daras. Cross-modal variational alignment of latent spaces. In CVPR, 2020.
Hao Wang and Din Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, 28(12) 2016.
Hao Wang and Dit Yeung. survey on bayesian deep learning. ACM Computing Surveys (CSUR), 53 (5):1 37, 2020.
Hao Wang, Naiyan Wang, and Yan Yeung. Collaborative deep learning for recommender systems. In KDD, 1235 1244, 2015.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255) 1992.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, allend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048 2057. PMLR, 2015.
Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, and others. The HTK book. Cambridge university engineering department, 3(15):112, 2002.
Guanlong Zhao, Sinem Sonsaat, Alif O Silpachai, Ivana Lucic, Evgeny Hunnhharey-Khddlayyeen John Levis, and Ricardo Miteerezz-sunaa L2-ARCTIC: A non-native English speech corpus. Perceplion Sensing Instrumentation Lab, 2018
15

---

Published in Transactions on Machine Learning Research (12/2022)
A MAP estimation of B and πi
Given the sentence-level ML-FSA introduced in Sec. maximizing the following posterior distribution:
the MAP estimate of and can be obtained by
plyla) p(y)p(a))))
p(uclei ... M1-1)p((alle))
plun (x()))
((ve)
(9)
p(yullt..... Mc-1) 20p (yn late (d(t))
p(ye)
t=1
26p (ye (24))
II plyclvi..... M1-1)
p(ye)
where in step a, p()( is approximated by 4o (B)l In step b, the constant term p(r²) is dropped. p(y.) is the prior yr, which is usually estimated with the frequencies of different phonemes in the training dataset. (ye the approximate posterior, which can be computed using the phoneme estimator. the transition probability which is determined by p(b.) and p(T.). Concrctely, when caleulating the transition probability p(yelyi,.... V/-1) (t-11) given the sentence-level FSA, three different cases are considered: (1) the consecutive frames belong to the same segment; (2) the consecutive two frames belong to different segments, and the current segment matches the phoneme sequence; (3) the consecutive two frames belong to different segments, and the current segment contains content mismatch (i.c., mispronomnced phoneme). Corresponding to these three cases, we have:
p(b, 0), p(b, 1)p(TT 0), if yt + .t-1 and yr E p(b. 1)p(TT 1), if y VI-1 and ye C*
if yı Vt-1
p(ycl)1,...,61111
(10)
where C* (ci..... et, denotes the mismatched elements in the phoneme sequence C (i.e. mispronounced phonemes? 2 p(T, = 0) and p(n, = 1) are approximated by Jo, (T, = 0/ Sr, b,) and To, ( = 1/ yt. bt), which can be calculated using the speech generator. Similarly, p(b, = 0) and p(h, = 1) are approximated by don, (be- = O).,)) and go,(4 (b. = 1|xx), which can be obtained from the boundary detector.
Generally, the MAP estimation of B and fi can be written as:
B, fl argmax
B,11
(11)
p(y. M1 .... 9o-1) Go, (ve (12)
argmax
B.IT
p(y:)
In practice, MAP estimation is achieved by following Eq. II and searching for the optimal path in ML-FSA, which can be solved by a DP algorithm.
Note that strictly speaking, here are approximating p(ylV11.... vt-1) using p(v) o(vl11,--. Vy, Bt-1, 1e-1, C1. 62); where e the known canonical phonemes.
16

---

Published in Transactions on Machine Learning Research (12/2022)
B
Estimating B and ñ' separately
In this section, we describe the algorithm which estimates the hard assignments for B and il separately. This algorithm is used to perform the ablation study.
B.1 Estimation of 1
Being different from our joint estimation algorithm discussed in the main paper, the boundary variable B is first estimated without considering the correctness of the promunciation.
When estimating B, we adopt the single-pathi FSA. Fig. 6 demonstrates the partial FSA for the 1-th phoneme, From the initial state O, there is only one path pointing state 1, standing for the I-th phoneme, et- Then at each time step, since each phoneme may last for several frames, it either still holds at state 1 (denoted by H), or moves forward to the final state 2 (denoted by S). The sentence-level FSA can be constructed by connecting all phoneme-level FSA together, according to the input phoneme sequence C.
0
Figure 6: Single-path FSA.
Based on such an FSA, the weights on the transitions can be estimated by ML-VAE in a similar manner as described in Sec. [6.] Then a DP algorithm can be used to find the optimal path in the sentence-level FSA, after which the estimated B can be obtained by backtracking along the optimal path.
B.2 Estimation of Íí
We then estimate given the B obtained in the above procedure. If we are given the 1-th phoneme segment which starts at u-th frame and ends at the e-th frame, we assume that all frames within this segment share the same pronunciation correctness label, that is, Tu Tu+1 promuciation correctness of the I-th phoneme.
We = oo where T/ denotes the
Therefore, similar Eq. 11 we can obtain the MAP estimation of n/ per segment:
Lo (Vel2rs) plyc)
P(T, = 0) IT:
To (mema p =a
ml = 0 if o/ = 1,
#1 argmax plun bo.......... V1-1)
= argmax of
(p(T = 11
Tap Ae P(n
(12)
t=
where 9op (ye [xt); 4o (v) - q (zee), and do (w) + c212) are the approximate posteriors. V: = denotes that the 1-th frame is correctly pronounced ( 0). yt. # a denotes that the (-th frame mispronounced ( = 1), that is, 4o (vr cx(xz)
Jo, (y: cilxt), and p(y * ci) p(v: =t).
C Case Study of ML-VAE
Besides quantitative analysis, we also provide case study to showcase ML-VAE-RU in Fig. Each rectangle in Fig. 7 represents one phoneme segment. The first and second rows show the input phoneme sequence and the actual phonemes pronounced by the speaker, respectively. The last row shows ML-VAE-RLE mispromunciation localization result. In the last row, a phoneme with a star sign (o.g. ***) indicates that ML-VAE-RL detects mispronounced phoneme (e.g. mispronounced 't'). It is shown that ML-VAE successfully detects three out of four mispromunciations with reasonable localization result..
17

---

Published in Transactions on Machine Learning Research (12/2022)
study YKWK a0442
Input phonemes
ae eh err
err
ng
dh
Pronounced phonemes
ae dh err
s aw
ng
dh
t
Model output
g+
eh
t+ dh
err
Case study for the last phonemes of the utterance YKWK a0442. The first and second rows show the imput phonemes and the actual phonemes pronounced by the speaker, respectively. The last shows ML-VAEL VAE's prodicted mispronunciation localization result.
Figure 7:
D Case Study of Ablation Models
Fig. Dhswwws the case study with the outputs of three ablation models presented in Table It is shown that the joint optimization model yields the worst outputs because the poor alignment results. Compared with the other two ablation models, ML- VAE clearly outperforms them in terms of localizing the mispronounced phonemes.
18

---

Published in transactions on Machine Learming Research (12/2022)
Figure 8: Case study for the last ten phonemes of the utterance YKWK a0442. The first and second rows show the imput phonemes and the actual phonemes pronounced by the speaker, respectively. The third row shows ML-VAE's predicted mispronmnciation localization result. The last three rows show the outputs of the three ablation models as presented in Table 3 To save space, the phoneme err is denoted as in this figure.
hg
g
30 eh
dh
01g dh
dh
rgg
ng
Case study for utteronce YKWV 30442
ch
dh
eh eh
Prono Input WL-VAE-rein
ML-VAE
Ablation2
Ablation
19

---

Published in Transactions on Machine Learning Research (12/2022)
E
Implementation Details
In this section, introduce the implementation details of ML-VAE. The input feature sequence X' is obtained by calculating the 40-dimension FBANK feature Young et al. (2002) from the speech signal. The phoneme sequence is obtained from the text which is read by the speaker.
E.1 Boundary Detector
The boundary detector includes two LSTM layers, each with 512 nodes, followed by two fully connected (FC) layers with 128 nodes and ReLU activations. They are followed by two separate output layers, cach of which has only one node and Softplus activation function. The output of these two layers is used to estimate the parameters a and / to sample Vi, which is further used to sample by The weight of the KT. term d is set as 0.01.
E.2 Phoneme Estimator
The phoneme estimator contains two LSTM layers; each LSTM layer has 512 nodes, followed by two FC layers, each with 128 nodes and ReLU. They are followed by a Softmax layer to give the estimation of the phoneme.
E.3 Speech Generator
Wc adopt an encoder-decoder architecture to implement the speech generator. The encoder consists of three FC layers, each with 32 nodes, followed by four LSTM layers, each with 512 nodes. We use FC layer with 512 nodes and ReLU to estimate the mean and variance of the Gaussian components. The decoder contains four bidirectional SRU layers (Lei et al. 2018), each with 512 nodes. They are followed by two FC layers, each with 120 nodes, to estimate the mean and variance of the data distribution. During training, Xp and X are set to 1 and 0.001, respectively.
Gaussian Component Selection As described in the generative process, the Gaussian component indi- cator 2 is sampled from selyt, Tt, b, Categorical(f (Vr, b,, mz)). In this section, describe the implemen- tation of fz(-), which is defined as f=(e,, by, mT) = softmax(pt # 6z), where P: is scalar estimated by a two-layer multilayer perceptron (MLP) taking as inputs Yt, b,, and Te. Each layer of the MLP contains 128 nodes and a Sigmoid activation function; e is a small constant, which is set as x 10-6 in our experiments; & an hot variable, i.e. dr[s] = 1, and s is computed by:
(j 1) (N.. +1)+1, +1,
if n.
(13)
(j - (Nm + +1)++
if f = 1,
where j denotes the phoneme label of yt, i.e. y = In case that y is mispronounced, k [1, Nm. denotes the k-th mispromunciation variant of the phoneme Vt, which is implemented by Gumbel Softmax function (Jang ct al. 2016), i.c. T, Gunbel(NN(2,, 9t)), where T.[k] = 1 and NN is simple neural network with three 128-node FC layers.
F Experimental Results of the Alignment Task
This section provides some experimental results of aligning speech and text inputs that contain mispro- munciation, as shown in Table 4 Experiments are performed on the real-world datasct L2-ARCTIC. The intersection-overs ion-oerrioon (IoU) is caleulated for every phoneme in the input lext and then averaged to evaluate the alignment performance. The experimental results show that the baselines, both the traditional forced alignment method and the CTC alignment algorithm, fail to yield reasonable alignment results, while our proposed ML-VAE and ML-VAL VL--AA-RL can outperform both baselines in this task. Such results demonstrate the traditional alignment results would fail when there is mismatch in the inputs. For example, if there is
20

---

Published in Transactions on Machine Learning Research (12/2022)
Table 4: Experimental results aligning speech and text inputs with misproumunciation on the L2-ARCTCC dataset.
Model
Average IoU (%)
Two-Pass-FA
4.68
w2v-CTC
5.73
ML-VAE (ours)
44.27
MI-VAE-RIA (ours)
53.02
mispromunciation in the middle of sentence, it may cause errors traditional alignment methods (e.g.. forced alignment). Then such errors may accumulate and further degrade the alignment performance of the phonemes after the mispronounced one.
21


Reviews:
[{"review_id": "wWI7dqNQOX", "reviewer": "Reviewer_a3sc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The literature review in Section 1 and Section 2 is currently satisfactory to me.\n\nExperimental results are encouraging. \n\nEdit: The authors carefully integrated feedback into the revised draft.\n\n--\n\nIt's not clear to me whether it's acceptable to only distinguish between correct pronunciation and mispronunciation, as discussed at the end of Section 5. For real-world application, we may need more complex variants, but it's unclear how a more complex version of modeling mispronunciation would work in practice, under the authors' framework. \n\nThe only task experimented in this paper is mispronunciation alignment, but it's not clear if it is clearly motivated. What would be the real use cases? Would there be other more impactful tasks for speech-text mismatch alignment that can be addressed by the authors' approach? \n\nRL details are lacking. It would be more clear if the authors expand on the discussion on modeling baseline (the term that gets subtracted from reward) and how exactly it is learned. Small concern: Is there a reason the authors used naive REINFORCE instead of more recent approaches like PPO? \n- Motivation of using RL is also lacking. \n- The objective (ML-VAE + RL) has clear typos. \n\nSome equations / notations are confusing...\n- On page 4, what are the parameters for f's / how are they parametrized and learned?\n- As discussed above, the ML-VAE + RL objective is confusing / has clear typos.\n- Equation 3 is also confusing to me; it will be great if the authors expand the equation / make sure it is correctly written and clearly explained. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Most of the algorithms that align two modalities (e.g., text and speech) assume that the content of the two modalities are perfectly matched. This work investigates the case where there is a content-mismatch between cross-modal sequential data, by proposing an unsupervised learning algorithm — a hierarchical Bayesian deep learning model called ML-VAE (mismatch localization VAE).\n\nDirectly optimizing the ELBO is infeasible according to the authors, so they propose a new approach to optimize the ML-VAE. The training alternates between (1) estimating the \"hard assignments of the discrete latent variables over a specifically designed finite-state automaton\" (Section 6.1) and (2) updating the neural net parameters (Section 6.2).\n\nThe graphical model and generating process are demonstrated in Figure 2. \n\nIn addition, RL is also used, as a variant of the proposed algorithm.\n\nThe experiments are done on identifying mispronunciations in speech-text mismatch; the experiments are done on non-native English corpus. Qualitative analysis is done.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I think the authors should make it more clear in the title and the abstract that the proposed approach is only experimented on speech-text mismatch localization.\n- Add motivation for only experimenting on mispronunciation alignment / why people would care about this task. \n- A lot more discussion on RL (including motivation for integrating RL, how is baseline is trained, how samples are drawn, as well as potentially other RL algorithms).\n- Carefully proofreading the equations (including page 4, Equation 3) and address my confusion above. ", "broader_impact_concerns": "I think it's worth discussing what specific real-world applications the authors' approach will benefit. Further, discuss the potential disadvantages to non-native speakers and how the authors can prevent misuse of the software. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2iXknqIoPz", "reviewer": "Reviewer_NADY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n\n- This paper is attempting to solve a problem not previously solved in literature without any human-annotated examples in a unsupervised manner. Being able to solve such a hard problem of alignment and mistake localization in a unsupervised manner can be very useful.\n- The proposed scheme of learning a hierarchical VAE with discrete latent variables using a finite-state acceptor can have applications even beyond this specific problem.\n- The empirical performance of the proposed scheme relative to the baselines demonstrates very significant empirical improvements.\n\n\n\n**Weaknesses**\n\n- In my opinion, various technical details need better presentation for the reader to follow the proposed scheme. For example,\n  - The Problem formulation (sec 3) can use (i) a conceptual/mathematical description of the data (and all relevant information like size of phoneme set, etc) available for learning, (ii) evaluation of the quality of a particular alignment to the ground-truth alignment (discussing the novel $\\langle x \\rangle_{ML}$ metrics more elaborately here). Also, given that the same term can be pronounced (correctly) with speech sequences of different length, how is that handled in the problem formulation (or why it is a non-issue)? What are the domains of the $x_i, i = 1, \\ldots, T$ and $c_j, j = 1, \\ldots, L$?\n  - In Model (sec 4), it would be useful to discuss domain of $y_t, z_t, h_t$ and how they are mapped to/from the domains of $x_i, c_j$. How does the ML-VAE generative process to create $x_t$ depend on the input $c_j$? Is $y_t$ sampled from the input $\\mathcal{C}$?\n  - In ML-VAE with Reinforcement Learning (sec 7), it would be better to first highlight the challenges in Algorithm 1 and how a RL based training would allow us to circumvent those challenges.\n  - Further examples discussed below.\n- The discussion of related work states that forced alignment (FA) generally requires an acoustic model trained with human-annotated speech. In that case, it is not clear how the proposed scheme is able to obtain a forced alignment $\\bar{\\mathcal{B}}$ in Algorithm 1? The ablation study highlights the need for this FA so this part needs more explicit discussion for the reader to truly understand how the proposed scheme is able to get around this challenge.\n- It is not clear why mismatch localization is critical in the alignment problem -- if the input speech and phenome sequences are appropriately aligned (even though some phenomes are mispronounced in the speech sequence), it is not clear why this is not sufficient and what might be some applications where we need to identify the mispronunciations in the speech. With mispronunciation, there might still exist a perfect match between speech and phoneme sequences but the speech part is non-canonical; it is not clear why matching non-canonical speech perfectly to phoneme is insufficient. This needs significant motivation in my opinion. Mispronunciation-robust alignment is definitely very important but requiring the localization of the mispronunciation needs motivation. Moreover, focusing on mispronunciations might be tricky since the \"canonical\" pronunciation is often regional and many terms have multiple canonical pronunciations even with native speakers.\n- On a related note, it is not clear if existing alignment schemes are unsuccessful (in alignment only) if the problem contains mispronunciations or if they are able to perform the alignment without localizing mispronunciations. The presented metrics $PR_{ML}$ etc do not show that the baselines struggle with the alignment problem, and it might have been better to also report scores that evaluate the alignment. It is not clear that the mispronunciations are the issue and that the baselines are struggling with the alignment because of the mispronunciations.\n- The Mismatch-AudioMNIST problem appears to be somewhat of a different task than the mispronunciation localization problem since, in this case, the mismatch pronunciation is **very very different** from the true pronunciation -- they are different numbers -- while in general speech-phoneme alignment problems the mispronunciations won't be very different from the true pronunciations. It would be great to get a better motivation as to why this is a useful problem to consider.\n\n\n\n*Various specific clarity comments and questions*:\n\n- Are their citations highlighting the failures of FHVAE, VRNN and SVAE with discrete latent variables? Are their empirical results in the paper that demonstrate this limitation?\n- Can the Jo et al. (2019) and Theodoridis et al. (2020) perform alignment without mismatch localization (if there is a match)? Or are these schemes not meant for alignment at all?\n- Is $N_m$ a hyper-parameter? How should it be set on a per-phoneme basis? How does this choice affect performance?\n- On page 4, \"Draw $x_t | h_t \\sim \\mathcal{N}(f_\\mu (h_k), f_\\sigma(h_k)$\" should it be $h_t$ instead of $h_k$?\n- Given the model description in Sec 4, the ML-FSA requires better introduction and contextualization. Also, what are the (transition) weights that need to be learned (so that we can apply DP to find the optimal path). Is this done on a phoneme level or sentence level (as mentioned that \"With the help of [...] $\\mathcal{C}$, we can [...] build a sentence-level ML-FSA ...\"? Why process at phoneme level vs sentence level?\n- In the learning (sec 6), where does the \"prior distribution\" $p(y_t, b_t, \\pi_t)$ come from and how critical is its selection? How is prior $p(v_t)$ set? How are all the other priors specified?\n- We need more clarity on the learnable parameters $\\phi_p, \\phi_b, \\phi_h$ (in sec 6) and their connection to the model (described in sec 4) with $\\alpha, \\gamma, \\mu, f_z, f_\\mu, f_\\sigma$ etc. There appears to be some mismatches in the notation.\n- There are lots of unclear notation in sec 7 such as $\\log(\\Pi = \\Pi^{(i)} | \\mathcal{X}, \\mathcal{Y}, \\mathcal{B})$ and $MSE(\\mathcal{R}(\\Pi), b(\\mathcal{X}))$.\n- The RL loss seems very confusing where $\\mathcal L_{h} (\\phi_h, \\hat{\\Psi})$ is in the loss but is also in there as part of $R(\\Pi)$, and $R(\\Pi)$ is also in the $\\mathcal{L}_{bl}$ term. Also, it is not clear what is meant by \"baseline term estimated by a fully-connected neural network\" for $b(\\mathcal{X})$ since it is not clear what term is being estimated and how the estimation is being done.\n- The RL variation seems to introduce unnecessary complexity. Also it is not clear if it is replacing the role of th ML-FSA and how learning in Sec 6 (figure 4) is modified when this RL loss is utilized. Is it just a change in Step 2.3 or does the whole process change?\n- What is $\\alpha$ and $\\beta$ in sec 8.1? The boundary detector appears to be modeled with a $\\gamma$ (sec 4) or a $v_t$ (sec 6.2.1) so it is not clear what is being done in 8.1.\n- How is a Softplus (which is an elementwise activation function) converting the vector output of the decoder into a scalar in 8.1?\n- It appears that very specific architectures are used for the different components of the model. It would be useful to motivate why such choices are made? Is it based on some existing architectures or is it a result of a architecture search? Also it would be good to understand how sensitive the performance is to the selected architectures (especially since we are counting number of parameters).\n- The choice of the $f_z$ function and the related definition of $\\delta_t$, $s$, $j, k$ appear to be quite involved and we do not see any intuition or justification behind these choices. Can this be better motivated or justified?\n- Is $TP_{ML}$ same as $TP_{IoU}$? Are the TP computed based on the phoneme sequence or the final localization results?\n- How are forced alignment baselines utilized (\"adapted\") for mismatch localization? How are the necessary acoustic models built without the human-annotated data (mentioned in sec 2)?\n- In addition to ML-metrics, can we also look at the alignment scores? Like how well were the baselines and ML-VAE able to align $\\mathcal{C}$ on top of $\\mathcal{X}$?\n- The example given for w2v-CTC would also be a challenging situation for the proposed ML-VAE, right. Since the learning process has no way to know that the pronounced 'd' should be a 'b'? How is the proposed scheme (or any scheme for that matter) able to connect a phoneme to the correct pronunciation without enough examples of the correct pronunciations.\n- What is meant by \"w2v-CTC is fine-tuned on speech data ...\"? I thought all the learning was done with the training data provided? What is meant by fine-tuning here?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the problem of jointly aligning speech and text while appropriately locating mispronunciations in an unsupervised manner with only access to the speech sequence and the text phoneme sequence. The survey of related work presented in the paper states that there are existing techniques to align speech and text or to identify mispronunciations but most require human-annotated speech. There are no existing schemes for labeling the mispronunciations in the alignment without human-annotated examples.\n\nThis paper briefly describes the problem formulation and then proposes a hierarchical model based on the Variational Auto-Encoders (VAE) which maintains discrete latent variables corresponding to (i) the alignment of the speech and phoneme sequences, (ii) the correctness of the pronunciations, and (iii) the form of the mispronunciations, while continuous latent variables model the speech encoding-decoding. Given the discrete latent variables, the paper proposes the use of a finite-state acceptor and a decomposed learning/optimization scheme where the different components are optimized alternately while keeping the others fixed, and the finite-state acceptor is used to optimize for the alignment and correctness variables. The paper presents the specific decomposed optimization problems that need to be solved iteratively. In addition to the above scheme, a reinforcement learning based loss function is also presented and used.\n\n\nAfter presenting the architecture for the different components in the hierarchical model, the proposed scheme is evaluated against some baselines adapted to this problem of jointly aligning and locating mispronunciations and is shown to have significantly improved performance over the baselines for a problem-specific modified evaluation metric. Beyond quantitative comparisons and additional ablation studies, the paper also presents a qualitative example demonstrating how the proposed scheme is able to align and locate mispronunciations simultaneously.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are three main adjustments to the submission which would secure my recommendation:\n\n- Motivation for the need for joint alignment and  mispronunciation localization beyond just being able to align in a pronunciation-robust manner, and highlighting how existing alignment schemes (without mispronunciation localization) struggle with pronunciations while aligning.\n- Addressing the presentation and technical clarity issues mentioned above (definitely including how the forced alignment in the proposed scheme is obtained without the requirement of human-annotated speech mentioned in Sec 2).\n- Better explaining and motivating the baselines considered (such as how they were adapted) and highlight why they fail (this is partially done but the explanation is not clear; see above clarity comments/questions).\n", "broader_impact_concerns": "There is a potential broader impact for the considered problem where \"mispronunciation identification\" for non-native speakers might be used in some way to disadvantage non-native speakers. Again, this concern stems from my lack of motivation for the problem and can be clarified by the authors. Another related broader impact pertains to how the canonical pronunciation is chosen since they can be different in different regions and still be \"correct\" and sufficient for communication. A model trained with data from one region might incorrectly identify and localize mismatches for speech from a different region.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Esplhs5khA", "reviewer": "Reviewer_mR9q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n\tThis work solves an interesting and important application problem on locating the mismatch between text and speech. The proposed ML-VAE algorithm does not require perfect match assumption and human-annotation, but shows the highest accuracy in the experiments, which are solid advantages. The lit review and experiments look clear and comprehensive to me. \n\t\n**Weaknesses:**\n\tThere are many unclear and possibly incorrect points in the formulation of the proposed methods, as shown in the **requested changes (critical for acceptance)** below, so I answer \"no\" to \"Claims And Evidence\" until the authors address them. \n\t", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a hierarchical Bayesian deep learning model, ML-VAE, to address the problem of content mismatch localization from cross-modal sequential data, which is the first method that bridges finite-state automata and variational autoencoders; this is achieved via our proposed mismatch localization finite-state acceptor (ML-FSA), which allows the ML-VAE to locate the mismatch by searching for the best path in ML-FSA. To address the challenge of inferring the latent discrete variables with complex dependencies involved in ML-VAE, this work proposes a novel and effective alternating inference and learning procedure. Furthermore, experiments on a non-native English corpus to demonstrate ML-VAE’s effectiveness in terms of unsupervisedly locating the mispronunciation segments in the speech. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested changes (critical for acceptance):** You may add clarifications for the following questions in the paper. \n\t\t\n(1) About the list of ML-VAE generative process right before \"**An Example**\" in page 4: \n\n(1.1) Should $h_k$ be $h_t$ in $f_{\\mu}(h_k),f_{\\sigma}(h_k)$? \n\t\n(1.2) If $\\pi_t=0$, it seems that $z_t$ can be uniquely determined by the correct phoneme $y_t$, yes? If yes, then it seems that $f_z(y_t,b_t,0)$ is not needed and we could simply keep $f_z(y_t,b_t)$ only for $\\pi_t=1$. Clarification might be added to the paper. The rules in **Gaussian Component Selection** in Section 8.3 may also be accordingly adjusted. \n\t\n(1.3) Is it more convenient to use $z_t\\in\\{1,2,\\ldots,N\\}$? If not, then you could clarify that $z_t\\in\\{0,1\\}^{K}$ is an one-hot variable. \n\t\n(1.4) What are the parameters of the 3 neural networks $f_z$, $f_{\\mu}$ and $f_{\\sigma}$? They seem to be missing in the learning process? \n\t\n(2) Section 5 about ML-FSA:\n\t\n(2.1) It seems that only one state transition occur from time $t$ to $t+1$, yes?\n\t\n(2.2) What is the meaning of each state? In particular, what's the difference between states 1 VS 2, 3 VS 4? Once state 5 is reached, does it mean the end of $c_{\\ell}$ and beginning (state 0) of $c_{\\ell+1}$? \n\t\n(3) In the training procedure in Section 6, How is the forced alignment result $\\overline{\\mathcal{B}}$ obtained? What's the difference between $\\overline{\\mathcal{B}}$ and $\\hat{\\mathcal{B}}$? \n\t\n(4) Could you add derivation of the posterior of $\\mathcal{B},\\Pi$ in eq. (3)? Why are $b_t$ and $\\pi_t$ missing in this posterior? \n\t\n(5) For eq. (4), why not computing the probabilities of $b_t$ and $\\pi_t$ exactly via $Categorical(\\alpha)$ and $Bernoulli(\\gamma)$? How can we ensure that these probabilities are close to posteriors $q_{\\phi_b}$ and $q_{\\phi_h}$ respectively? I suspect $p(y_t|y_1,\\ldots,y_{t-1})$ may ought to be $p(y_t|y_1,\\ldots,y_{t-1},x_1,\\ldots,x_t)$ as (3) is to maximize posterior. \n\t\n(6) I did not get the meanings of the parameters $\\phi_p$, $\\phi_b$ and $\\phi_h$ until I saw the posterior notations $q_{\\phi_p}(\\mathcal{Y}|\\mathcal{X})$, etc. from Section 6.1. Could you bring these notations to the beginning of Section 6 to more clearly introduce $\\phi_p$, $\\phi_b$ and $\\phi_h$ as parameters of the neural networks describing posteriors $q_{\\phi_p}(\\mathcal{Y}|\\mathcal{X})$, etc? \n\t\n(7) How is the objective fucntion (5) obtained? Is it from ELBO in eq. (1)? If yes, should $b_t=\\overline{b}_t$ be $x_t$ to match ELBO?  \n\t\n(8) In Section 7, in the objective function $\\mathcal{L} _ {rl}$, should $\\log(\\Pi=\\Pi^{(i)}|\\mathcal{X},\\mathcal{Y},\\mathcal{B})$ be $\\log q_{\\phi_h}(\\Pi=\\Pi^{(i)}|\\mathcal{X},\\mathcal{Y},\\mathcal{B})$? How is MSE defined (better to be written in formula if possible)? \n\t\n(9) In Section 8.2, is the phoneme estimator network used to describe $q_{\\phi_p}(y_t|x_t)$? What functions/posteriors are described by the encoder and decoder of the boundary detector and speech detector? You could write down notations like $q_{\\phi_p}(y_t|x_t)$ to be clear. \n\t\n(10) In Section 9.1, do you take each time frame $t$ as a sample? Do you take correct or incorrect pronunciation as positive? Could you define IoU metric, better to use equation if possible or cite a related work that contains the definition? \n\t\n(11) In Section 9.2, I remember the FA method requires human anotation as said in the introduction, right? If yes, how to apply it with only $\\mathcal{X}$ and $\\mathcal{C}$? Also, do Two-Pass-FA and w2v-CTC require human anotation? You may clarify them in Section 9.2.\n\t\t\n**Requested changes (not critical for acceptance):** \n\t\n(1) In Section 3, how is the speech feature $x_t$ obtained, or what's the meaning of $x_t$? (e.g. volume, frequency, etc.) Also, I guess the phoneme sequence $\\mathcal{C}$ is correct pronunication obtained from the text, yes? You might clarify these in the paper. \n\t\n(2) Add \"s\" after \"propose\" and \"adopt\" in the first paragraph of Section 2. Do similar changes in the last two paragraphs of Section 2. \n\t\n(3) In the third paragraph of Section 2, change \"an decoder\" to \"a decoder\". \n\t\n(4) If possible, could you list some popular mismatch types in the second paragraph after Figure 2? \n\t\n(5) In $\\mathbb{E}$ of eq. (1), should $x$ be $x_t$?", "broader_impact_concerns": "I have concerns about the following unlikely but seemingly possible ethical consequences:\n\t\n(1) Could the audio datasets reveal the identities of the speakers? Do the speakers allow their speech to be studied and publicized?  \n\n(2) If employers use your techique to test interviewees' mispronunciation rate and accordingly decide whether to hire them (e.g. as a teacher/compere), could the interviewees whose accent lacks training samples be discriminated? For example, could your algorithm exaggerate their mispronunciation rate? ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors addressed most of the issues raised in the very detailed reviews provided by the three reviewers in their revision submitted on 10 Nov. 2022, and then on 25 Nov. 2022 they provided a second revision (after the reviewers had made their formal recommendations) that addresses remaining questions on derivation of the posterior and the use of the ground-truth phoneme sequences.

One question the action editor is left with is how this framework could be extended to handle the case where multiple pronunciations of a word are considered correct. I believe it would be possible to extend the ML-FSA, but in practice there might be challenges due to the presence of multiple correct paths through it.
Decision: Accept (poster)