SSDM: Scalable Speech Dysfluency Modeling
Authors: Jiachen Lian
Xuanru Zhou
Zoe Ezzes
Jet M.J. Vonk
Brittany T. Morin
David Paul Galang Baquirin
Zachary A. Miller
Maria Luisa Gorno-Tempini
Gopala Anumanchipalli
Venue: NeurIPS 2024 poster (2024)
Keywords: Speech Dysfluency
Disfluency
Stutter
Alignment
Articulatory
Scaling
Primary area: speech_and_audio
TL;DR: A speech processing framework that supports language learning, speech therapy and disorder screening.

Abstract: Speech dysfluency modeling is the core module for spoken language learning, and speech therapy. However, there are three challenges. First, current state-of-the-art solutions~~\cite{lian2023unconstrained-udm, lian-anumanchipalli-2024-towards-hudm} suffer from poor scalability. Second, there is a lack of a large-scale dysfluency corpus. Third, there is not an effective learning framework. In this paper, we propose \textit{SSDM: Scalable Speech Dysfluency Modeling}, which (1) adopts articulatory gestures as scalable forced alignment; (2) introduces connectionist subsequence aligner (CSA) to achieve dysfluency alignment; (3) introduces a large-scale simulated dysfluency corpus called Libri-Dys; and (4) develops an end-to-end system by leveraging the power of large language models (LLMs). We expect SSDM to serve as a standard in the area of dysfluency modeling. Demo is available at \url{https://berkeley-speech-group.github.io/SSDM/}.

SSDM: Scalable Speech Dysfluency Modeling
Jiachen Lian', Xuanru Zhou', Ezzes', Jet Vonk Brittany Morin', David Baquirin', Zachary Miller', Maria Luisa Gomo Tempini', Gopala Anumanchipalli
UC Berkeley, Zhejiang University, UCSF {j iachemlian, gopala)@berkeley. edu edu
Instruction:
What do you think of the pronundielio??
You wish to know all about my grandtather
LTU-AS (13 B)
The pranunciation clear and understand, which makis to along speaker saying.
SALMONN (13 B)
pronunciation speech easy It great effort.
couple of additional lips to enhance clarity. Emphasize the "k" sound in "know" slightly more. Overall, it's a strong effort.
pronunciation
again,
ChatGPT
You are trying to pronounce "you wish to know all about my grandlather" There are a few probleme. For the word 'you," there is stutter of "y" at 0.60 seconds, For word "all." there block 2.92 seconds. For the "grandfather"" phonetic error "d" seconds. Try speak slowly or consider using speech therapy exerciees that target these errors.
Ours
SSDM
https://shorturt.abb
Figure 1: SSDM. Comparison to other methods
Abstract
Speech dysfluency modeling is the core module for spoken language learning, and speech therapy. However, there are three challenges. First, current state-of-the-art solutions M 2
suffer from poor scalability. Second, there is a lack of a large-scale dysfluency corpus. Third, there is an effective learning framework. In this paper, we propose SSDM: Scalable Speech Dysfluency Modeling, which (1) adopts articulatory gestures as scalable forced aligmment; (2) introduces connectionist subsequence aligner (CSA) to achieve dysfluency alignment; (3) introduces large- scale simulated dysfluency corpus called Libri-Dys; and (4) develops an end-to-end system by leveraging the power of large language models (LLMs).. We expect SSDM to serve as a standard in the area of dysfluency modeling. Demo is available at htttps://berkeleee speeeeccleyyyeechhhhheo group. gi github. io/SSDM/
Introduction
Speech dysfluency modeling is key for diagnosing speech disorders, supporting language learning. and enhancing therapy [1. In the U.S., over 2 million individuals live with aphasia (3). while globally, dyslexia affects approximately one ten people
(4). The U.S. speech therapy market projected to reach USD 6.93 billion by 2030 [.. This growth parallels developments in Automatic Speech Recognition (ASR), valued at USD 12.62 billion in 2023 (6), and Text-to-Speet valued at USD 3.45 billion (7]. Morcover, the global language learning market is anticipated to be USD 337.2 2 billion by 2032 [8]. Conversely, substantial investments have been made in training speech-language pathologists (SLPs) 9 10]. and the high cost of treatment often remains out of
(TTS),
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input: Dysfluent or Normal Speech
Reference Text: wish know about grandlather
}
[[lbu [[lbte-[[[:to |I0
Instruction:
What do you think of the pronunciation?
K-Moars
Acoustic Encoder
Forced Alighmant
Text Encodor
Acipplor
14 o
Gestural Encoder
Connectionat Subsequence Alignr
My
Multimodal Takenzze
C8Aa(T.C)
Gestures (1: d
Gestural Ii
Dysfluency Alignment
LLaMA
LORA
Fow f
2v|8|C)
Eagen 1111
Resporse: For the word "you," Where stufter of "y" 0.60 seconds. "all." there block "grandfather" them stutter 5.60 saconds, there phonetic error for "d" 5.90 seconds.
D-Campple
[3-temple
word
Mult-Scale
Gestural Decoder
Figure 2: SSDM architecture
reach for many low-income families [TH.15] Therefore, there is crucial need for an Al solution that makes advanced speech therapy and language learning available and affordable for everyone.
Speech dysfluency modeling detects various dysfluencies (stuttering, replacements, insertions, dele- tions, etc) both word and phoneme levels, with accurate timing and typically using reference text [ (see Figs for examples). Fundamentally, it is spoken language understanding problem. Recent advancements have been driven by large-scale developments [1631]] However, these ef- forts often focus on scaling gaaseegr performance metrics rather than deeply listening and understanding the nuances of human speech.
Traditional approaches to dysfluency modeling have relied on hand-crafted features 32 36 Recent advancements have introduced end-to-end classification tasks at both utterance [37
48 and frame
levels [49 50 However, these methods often overlook internal dysfluency features like alignment [] propose
and struggle to detect and localize multiple dysfluencies within single utterance. 21D-Alignment, non-monotonic approach that effectively encodes dysfluency type and timing. Nonetheless, initial experiments show that this method struggles with scalability, limiting its further development. To address these concerns, we revisit this problem and summarize contributions as follows:
We revisit speech representation learning from a physical perspective and propose neural articula- tory gestural scores, discovered to be scalable representations for dysfluency modeling.
We introduce the Connectionist Subsequence Aligner (CSA), a differentiable and stochastic forced aligner that links acoustic representations and text with dysfluency- alignment.
We enable end-to-end learning by leveraging the power of large language models.
We open-source the large-scale simulated dataset Libri-Dys to facilitate further research.
Articulatory Gesture is Scalable Forced Aligner
2.1 Background
Revisit Speech Representation Learning Self-supervised speech representations [51] large-scale ASR [16 18 20), codec models (52.6] 67]. and speech language models (SLMs) [1 have emerged as universal paradigms across tasks and languages. However, high computing costs of scaling efforts affordable for academia researchers. In this work, we propose learning speech representations grounded in fundamental physical laws [68 69). This approach characterizes speech representations by the kinematic patterns of articulatory movements, method we refer as gestural modeling.

---

Gestural Modeling The concept of gesture, as defined by 770 711. refers to articulatory movements in acoustic space, similar to body gestures in humans. [I introduced gestures as dictionary of basic articulatory movements and gestural scores, representing the duration and intensity of these movements. This principle resembles the gait library and optimization used in roboties [72]. The computational modeling of gestures was first developed by [73], using sparse matrix factorization to decompose EMA data [76] into interpretable components. Further research by [77] and 77 streamlined this into end-to-end neural approach. Gestural scores serve as speech representations. We discovered that they serve as scalable dysfluent phonetic f aligner.
74
75
Scalable Dysfluent Phonetic Foreed Aligner Dysfluency modeling requires detecting both the type and timing of dysftuencies, necessitating the use of forced alignment [. This alignment is often non-monotonic (e.g., stuttering). Thus, previous monotonic alignment methods [79] [80] [8) perform poorly in the dysfluency domain. The primary challenge is the inherent uncertainty in what the speaker actually said, compounded by invariably inaccurate reference texts, explained in [ Effective research in this area focuses on non non-monotonic alignment modeling. [82] introduces the WFST 3 to capture dysfluencies such sound repetition. However, it assumes the actual speech does not deviate significantly from the reference text. 1 proposed 2D- 2D-aligument as final dysfluent representation. Nevertheless, this method, and its extension 22. suffers from scalability issues: increasing training data does not lead to further improvements. In this work, we revisit the monotonic alignment to tackle the scalability problem. To achieve this, we need a scalable representation, and a scalable monotonic aligner (Sec. (3) This section focuses on the first part and proposes Neural Variational Gestural Modeling to deliver gestural scores H scalable dysfluent speech representations. We also provide a visualization of gestures and gestural scores in Appendix. A.I
2.2 Neural Variational Gestural modeling
Despite theoretical support 770 68 69]. gestural scores have not yet become universal speech representation [51] due to several limitations. First, gestural modeling requires extensive, often unavailable, articulatory kinematic data. Second, there is not an effective learning framework. Third, the commonly used EMA data, sampled sparsely from human articulators [8) suffer from information loss. To overcome these challenges, we proposed Neural Variational Gestural Modeling. This model uses an offline inversion module /Sec.2.2.1 to capture articulatory data, and a gestural VAE extract gestural scores (Sec. (.2.2). which then refined through joint self- dilf-disillatiin with acoustic posteriors and textual priors (Sec. 2.2.3). This method ensures that the resulting gestural scores are effective and scalable dysfluent speech representation. (Evidenced in Sec.
2.2.1 Universal Acoustic to Articulatory Inversion (UAAI)
Since the real articultory data are typically unavailable, we employ a state-of-the-an acoustic-to- articulatory inversion (AAI) model (88) pretrained on MNGU0 (84). The model takes 16kHz raw waveform input and predicts 50Hz EMA features. Details are listed in Appendix. A.2.I
2.2.2 Gestural Variational Autoencoders
Any motion data X [X,, X2 .... Xt) can be decomposed into motion kernels G E Rxxxx and an activation function H E kxt using convolutional matrix factorization (CMF) (75), where X ~ -T- C(i)- H Here, represents time, T the kernel window size, d the channel size, and K the number of kernels. When X is articulatory data, G corresponds to K gestures and H to the gestural scores (Visualization in Appendix A. and (..12)). This work focuses on three aspects: (1) joint modeling of articulatory-speeifc duration and intensity, (2) self-distillation from both acoustic and textual data, and (3) multi-scale decoding of gestures and gestural scores.
Variational Inference We employ point-level variational inference for qo (H)X), meaning for each point (R: i) in H E Kxt, we model its posterior (o(H..)))). This approach results in K x f posteriors for each gestural score H, where k = I...., K and i = I,..., t. We use pointwise inference for gestural scores due to its properties, such as overlapping durations across articulators and stochastic variations across accents. We will refer to this patchwise rather than pointwise, as we are modeling a patch embedding for each point (k, i). In practice, we introduce an additional latent vector as variational augmentation 89] where P is patch size. This setup formulates the duration
Z E P posterior qo ( k X). intensity posterior qss (T. Z X and latent posterior 4s (Zk.iXX) Patchwise operation is detailed Appendix A.2.2 Consequently. our gestural encoder encodes the Ai X X)9g (1
joint posterior 90 k. D k [ 9 k
2
k z ** (g.t))ge( (zk. LX).
3

---

VAE Objective After variational inference, our decoder pa P= P. recon- structs X using duration D, intensity I, and gesture G. The evidence lower bound (ELBO) and its derivation are provided in Eq. 1 and Appendix A.4 respectively. The posterior ( X)), modeled via vanilla variational inference [90], assumes standard normal priors for p(Z*.t).
). The mechanisms
of the duration and intensity encoders, (o( xx.,)) and (o. Xk.),, are detailed in
Sec. 2.2.2 and Sec. 2.2.2 Details on the decoder (x I, G,G) are discussed in Sec. 2.2.2
CELBO
E (Z, [log po (X D.)
(ki)) [KL (go(Z DD.., (=(p)))lp(( Dh. [ba))
(I)
Duration Posterior go (e( k x We employ the Gumbel softmax 911 reformulate the
duration posterior go (D 12 .X.. Let Hhi Lº denote the logits across all C discrete duration
classes (values) for patch (kt, i). For each class j. we obtain Gumbel noise 2 log(- - Log(-Log(UJ]) log(U,))
where Uj Uniform(0. 1). We then define * (log(z).)) + "/T where temperature parameter. Finally, we obtain the Gumbel softmax transformation as approximation of the duration posterior 2 We set p(Dk.)) = 1/C, where C is the number of discrete duration classes.
Background and detailed methodology can be viewed in Appendix. A.2.2
exp R
4 ki (
Σ3
exp
Intensity Posterior qo qo(!4 xk.) After sampling / qo ( Zk. xx,,),, the model ap- /20p""//4*"">/p
plies a per-gesture, region-wise impact. This can be formulated in Eq. where represents the local window of impact, _ is the sampled impact value, and D the duration of the gesture. We actually applied Sigmoid function to deliver positive intensity values. The Hann function is used to apply the impact smoothly within the local window. The motivation behind this formulation is that most patches (k. i) are not activated, reflecting the sparse nature of human speech production and co-articulation 77 77.. Visualizations can be checked in Appendix A.2.2
H - &
Hann (Sigmoid(Ik: qo (o
Xk:))), ( (
D ~ go (h. 2 X)) (3)
Online Sparse Sampling Given the limited number of patches contributing to gestural scores 771,
we localize the impact within specific window. We define a Combined Score S = al + 6Dk.1
where T and & represent impact and duration, respectively, and a and b are hyperparameters. This score ranks the importance of each patch, with indices for each gesture computed as Frow i) rank(-Sk:: Sk., within row k). Setting Mrow as the number of patches selected, we apply sparse mask Mrow (Eq. 4 to derive the final sparse gestural scores, detailed in Eq. 4 This entire online sparse sampling process is differentiable. The parameters a, b, and myow elaborated in the Appendix. For simplicity, we denote this process as (i, k) ~ S, with visualizations in Appendix(A.2) A.2.3
Mia = {0
if Trow i) S Myro otherwise
H
where
(4)
row
Multi-scale Gestural Decoder The decoder reconstructs X = 1X,, ..., X.] E Raxt from ges- tures G E RTxdxK and gestural scores H Kxt In this work, retain the CMF operation (77) and extend it to multiple deep layers. We also introduce multi-scale mechanism, which has proven to
be robust tokenizer for various speech tasks 92. 93 62
94
Denote: f faowa,o- f f fig
as downsampleeupsampll modules with scales of 1/2 or 1/4. The convolutive matrix factorization
operator A * B means ET= A(i)- B where A E R Txdx and activation function B E R kxt Then our multi-scale decoder defined in Eq. 5 where fiams represents any neural network, details of which can be found in the Appendix. Up to this point, G) (Eq. I defined. We provide more details in Appendix A.2.4
I means no resolution change, and
8= -=Er(L1224) Súp.0 (fram..
(G+ fddr, ( (H))
(5)
2.2.3 Gestural Scores as Phonetic Representations
After obtaining gestural scores, we predict phoneme alignment for dysfluency modeling. For clean speech, alignment is acquired using the Montreal Forced Aligner (MFA) [80], while for dysfluent

---

speech, it is simulated (see Section 5 The direct prediction of phoneme aligmment from handcrafted features or self-supervised learning (SSL) units [5]] is limited due scalability issues with dysfluent speech, discussed further in Sec. 6 We utilize 4X downsampled gestural scores (from decoding), denoted as H, matching the resolution of acoustic features [95] Let [Ti, ....... Ty/ represent the phoneme alignment, where I' 1/4. Employing the Glow algorithm (96), transform H into
т, expressed as f6 (H), optimized via a softmax crossentropy objective 4phm
Self-Distillation We distill gestural scores from pretrained acoustic features adapted to match gestural scores' dimensions. Instead of directly measuring the distance between acoustic embeddings and gestural scores, we use the alignment-condition
995 which are then
gestural prior as an acoustic-conditioned gestural posterior. The reference text C IC, C,...., CI. ( is processed by a
Cz), ), with the gestural posterior modeled via the change of variable property fq. as described in Eq. Intuition, detailed methodology and visualization can be view in Appendix. A.3]
text encoder to yield the latent Gaussian posterior (ng :0 C+), (1u 0g ())) "....., (ug .00
Po (HIC) (T( det
(a a8
(01
) R. Ii -1 V (Tilll Co ,0,0 (o00)2) det (018 aH
)
(6)
Conversely, given the acoustic embedding A = (2,, Ar). text encoder is employed to output the latent Gaussian posterior (1/) "00", oo""), "00",000 ...., (mD" .o""") ")").. The posterior
qe [ A) can be derived in a similar manner. The overall distillation loss is then presented in Eq.
Cdist = KL ((T()))))l
where go = शे II. II N (Miivo' (o
Both K₁ and K2 are normalization terms. The overall loss objective for neural variational gestural modeling is shown in Eq. 8 where λ2, A are balancing factors.
LVVE = - LELRO + >2hppn + A
(8)
3
Connectionist Subsequence Aligner (CSA) for Dysfluency Modeling
3.1 Monotonic Alignment is effective Dysfluency Aligner
Given the reference text C [C1, C2.--.. Cr.) and dysfluent phonetic alignment T [71, 72, Tel the alignment between C and r is typically non-monotonic. For example, when people say "pl- please," " it is monotonically aligned with "p-l-e-a-s-e." Prior work [ 82) on mon-monotonic dystluent modeling has its limitations, as discussed in Sec. S.1 In this work, we focus on monotonic alignment and argue that it is effective dystluency aligner. The intuition straightforward: we seek an aligner y 21.2..... L} - 2, t')) such that for each i 21. L}. Eq. 9 holds. The aligner y maps elements in C consecutive subsequences in 7 without overlap. This property is beneficial for dysfluency detection, as for each element in C, we can determine the presence of dysfluencies such as insertion, deletion, repetition, block, replacement, etc., based on (Ct).
1 S 8 Seses St
(Ca) = [Te,, Tectl. Te.] where
4 < 8i+1
Vi E (1,2, L 1] Vi E (1,2.....
(9)
8 < 8441.4 < C,41
L. 1-1] 1]
3.2 Local Subsequence Alignment (LSA) Achieves Semantic Dystluency Alignment
All monotonic aligners satisfy Eq 9 which serves as a necessary condition. However, we also desire (CC)) to be semantically aligned with C., Consider the aforementioned example: one preferred alignment 7(p)=[p1l.p). indicating the presence of stutter. contrast, if =fpp)=(p.l......aa.) becomes challenging to identify any reasonable dysfluency, despite still satisfying Eq/9] In this work, we propose that Local Subsequence Alignment (LSA) is an effective approach for achieving semantically aligned 7. Before delving into the main topic. we propose and introduce two terms: (i) Global Sequence Aligner (GSA), where the cost function involves the alignment of all elements the sequence; this includes most sequence aligners such as DTW 197.99, CTC [79], and MFA 80); and (ii) Local Sequence Aligner (ISSA), where the cost function involves only a subset of elements. One representative is longest common subsequence (LCS) alignment [100 [01].
5

---

LSA (LCS)
update cast only If
C₁ -
Reference
AH
EN
LSA
EH
Dysfluent
GSA update
(C,
Figure 3: LSA(LCS) delivers dysfluent alignment that is more semantically aligned.
Intuition Fig. 3 (left) illustrates the effectiveness of LSA as a dysfluency aligner. The reference text C, stress-free phoneme transcription [02] of word "references", contrasts with the dysfluent phonetic alignment т, which includes impairments like insertions of fillers and repetitions. LCS (100) and DTW (GSA, (97]) results are depicted in red and blue, respectively. LSA alignment (C,) shows higher semantic alignment with C, compared to DTW's 7 (C.). which includes misaligned elements like an unwarranted alignment of "F". LSA's superiority stems from cost function, which updates only for matching dysfluency-awan boundaries, while DTW updates for all pairs, often unrelated to dysfluency boundaries. Detailed analysis are available in Appendix A..7
Problem Statement Taking LCS into our framework presents three challenges: First, the high dimensionality of C' and T requires suitable emission and transition probability models. Second, LCS cost function is non-differentiabl. Third, multiple LCS alignments necessitate effective modeling of joint distribution. To address these, introduce Connectionist Subsequence Aligner (CSA).
3.3 Connectionist Subsequence Aligner (CSA) Formulation
Objective From gestural score Á, we obtain phonetic aligmment T f [T, T2. .... Te/ In practice, both t and C' are embeddings instead of explicit labels, where C = =CC
Cz. are
sampled from the text encoder N ( (2) = 1, .... L, proposed in Sec 2.2.3 Let t" denote the sequence length after removing duration from the original length 1'. Duration will be reincorporated post-alignment. The alignment between C and is already defined in Eq We introduce another notation r, where ((T;) the aligned token in C. ((r) [T(T),, (TT)) ] represents the final alignment with respect to C, in comparison to alignment (C), which is with respect to т. There are possibly multiple (N) alignments LSA (C). where j = L, .... N. Our goal is
Single Alighment
Forward
Backwarc
gtrlash
Transition Skip
Anigi
Pwisyo
Emission Copy
Dys
g
71
T2
T3
T4
TB
TC-1 T
T++11
Figure 4: CSA
N to optimize model 0 obtain the largest joint distribution of alignments Σ/ 2 (C) However, unlike CTC [79), we can't 't search alignments explicitly as the monotonic constraints are different. We propose approximating LSA. Let I' (7) be one approximate LSA alignment, and assume there are N possible LSA alignments: (7( where j -,..., N. Our final objective formulated Eq.
Σ po (C)JC,) C.7
E
Σ po ( C,r
E
max E Po (P(((((T)T)) (10)
Σ
Approximate LSA Alignments I' (T) We define ndd the emission probability (()) and transition probability po(T 77). Let Cs denote the embedding sampled from the distribution
N (o ()2)) (Sec 2.2.3.. The emission probability is given in Eq. 11 We approximate the
transition probability using a separate neural network (T(T)(TT))
y = -o(C(TT) ~
(11)

---

It is possible to list all LCS aligmments (r), where i == = I,..., .N, via soft alignments [99], which are also differentiable. However, we propose that by simply introducing the LCS constraint on the vanilla CTC [79 objective, the LCS can be implicitly applied, which we call the Connectionist Subsequence Aligner (CSA). Let us consider Figure (left) for intuition. For a single alignment (T), the emission probability and transition probability will only be applied if C, is already aligned (Ci in the figure). We refer to these Transition Skip and Emission Copy. Now, let us move to the LCS-coontraieed forward- backdadr algorithm [79]. Taking the forward algorithm (Figure 4 (mid)) for illustration, Emission Copy is reflected in Ct via an identity multiplier on a,-1,j- Transition Skip is reflected on both 0;-1,j and -11---- where we apply a transition on a/-1.j-1 This also implicitly leverages language modeling. We also consider all previous tokens Cj-2.... Cj- ..... Co: however, no transition is applied, but discounted factor 8 is utilized instead. This indicates significant jump (deletion), which we denote as a dysfluency module, although all other modules model dysfluencies equally. Forward and backward algorithms are displayed in Eq. [2 and Eq. 13
ag
3J 1-15
00
0
i-1 yhd (po(Cf-119)) 1(k=1) + (8x1))
(12)
B/ = 30
gitli
8 10
gitl b... (po(C (x=1) + (Rx11)
(13)
We initialize 0n,1 = Be,, 1, a(i, 1) 0
3(i, 0 Vi < 1", and ((1,))
0 Vj < L. Our CSA objective is displayed in Eq. 14 reference tokens and time stamps.
where we take the summation over all
N Σ 1=1
a 3
LCSA
E C
po (TTT
(14)
you
==1
Sampling As the alignment [T( is required for the next module, it is necessary to sample it during training. Traditional beam search methods are impeded by reduced inference speeds. To mitigate this, we employ the Longest Common Subsequence (L.C)) algorithm offline C' and 7 derive the alignments. The final alignment is denoted as ((C)) = S --=) as presented in Eq. 9 This methodology yields sequence of inputs in the form of CSA-O (Cf)) .... (C (C))
4 Language Models and Overall Training Objective
Following LTU 23 we utilize speech representations (alignment) [(Cf, (C,)),,..,,,,,,..
(Sec. 3.3 along with word-level timestamps, reference text C, and instruction C¹, as input to LLaMA-7B [03]. During the training process, we incorporate annotations that include word disfluency with timestamps. Our approach strictly adheres to the procedures outlined in 23 and employs Vicuna instruction tuning [104 with LoRA [105]. As this is not our core contribution, we
provide details Appendix A.8 We use the same autoregressive training objective as (3 denoted
as LIAN. The overall loss objective for SSDM is shown in Eq. 15
LSSDM LVAR LOSA + LlAN
(15)
5 Libri-Dys: Open Sourced Dysfluency Corpus
Traditional rule-based simulation methods [ 37 50) operate in acoustic space, and the generated samples are not naturalistie. We developed a new pipeline that simulates in text space. To achieve this, we first convert a sentence into an IPA phoneme sequence. Then, we develop TTS rules for phoneme editing to simulate dysfluency, providing five types of dysfluency: Repetition(phonemee word), Missing(phoneme & word), Block, Replacement and Prolongation. These rules are applied to the entire LibriTTS dataset [106), allowing the voice of generated speech to vary from the 2456 speakers included in the LibriTTS. The TuS-s entire pipeline, dataset statistics, MOS evaluation and phoneme recognition results are available in Appendix A.9 Overall Libri-Dys is 7X larger than LibriTTS, with total size of 3983 hours. Data is opensourced at sattps://bitt 1y/ 4aoLdWU
7

---

Experiments
6.1 Data Setup
For training, we use VCTK++[I and Libri-Dys datasets. For testing, we randomly sample 10% of the training data. Additionally, incorporate nfv7 data from our clinical collaborations, which includes 38 approximately hour of speech. Further details provided in Appendix A.
more than the 3 speakers in prior studies [1 2]. is
6.2 Experiments Setup
The neural gestural VAE (Eq CSA (Eq 14), and language modeling components are trained sequentially, with each stage completed before the next begins. Subsequently, we perform end-to-end learning to implement curriculum learning. Our objective is to evaluate the dysfluent intelligibiliity and scalability of our proposed gestural scores, as well as the dysfluency detection performance of each proposed module. We evaluate phonetic transcription and alignment using the framewise F1 Score and Duration-Awan Phoneme Error Rate (dPER). The Fl Score measures how many phonemes are correctly predicted, while dPER extends the traditional Phoneme Error Rate (PER) by assigning specific weights to different types of errors. For dysfluency evaluation, besides F1 Score, we also report the time-aware Matching Score (MS), which measures both type and temporal accuracy, with temporal matching considering the Intersection over Union (IoU) threshold of 0.5. Detailed training configurations can be found in Appendix A.12
6.3 Scalable Intelligibility Evaluation
Method
Eval
dPER (1.. [L (S. dPER
dPER (S. JI JPER (1S. dPER
SPI
Training Parss
VC7K++
Fibe7775 (790%)
Libri-Pye (309%)
Labr--Pys (607))
Libri-Pyy (1076)
HORERT
Libr-Dys
85.2
40.3
50.3
$8.2
47.4
67.2
80.8
42.3
87.2
43.4
40.2
$7.8
42.9
0.18
0.29
-0.1
VCTK++
36.8
Libi-Dys
88.1
44.5
$8.9
45.6
85.0
433
88.5
43.3
$8.9
43.0
0.32
-0.09
GS-only
VCTK+*
88.1
41.9
88.1
42.2
83.3
41.9
88.9
41.9
$9.4
4017
0.39
-0.36
Dys
44.5
85.5
42.2
GS wio dist
VCTK++
91.4
50.0
91.6
38.5
91.5
38.8
92.0
57.2
97.6
37.1
0.38
-0.67
Libi-Dys
42.4
GS w/
VCTK++
91.5
39.0
38.3
91.7
38.6
92.1
37.0
37.0
-0.64
Libi--ys
88.2
40.9
88.9
40,9
89.0
40.8
89.2
39.0
90.8
39,0
0.56
-0.72
Table Scalable Dysfluent Phonetic Transcription Evaluation
We evaluate phonetic transcription (forced alignment) performance using simulated data from VCTK++[I and our proposed Libri-Dys dataset. The framewise F1 score and dPER [ are used as evaluation metrics. Five types of training data are used: VCTK++, LibriTTS (100%, (106). Libri-Dys (30%), Libri-Dys (60%), and Libri-Dys (100%). HuBERT [08] SSL units and H-UDM alignment (WavLM (95)) fine-tuned with MFA [80] targets are adopted. Additionally,
examine Gestural Scores (GS), GS-only refers to gestural VAE training (Eq [ GS w/o dist excludes Ldist and GS w/ dist includes it, following Eq Results are presented in Table T H-UDM consistently outperforms HuBERT due to the WavLM backbone. Gestural scores from Eq. I show inferior results due to sparse sampling. However, GS demonstrates better scalability compared to SSL units. Using phoneme alignment loss Cphn significantly increases intelligibility, matching SSL unit results. GS outperforms SSL units with more training data. The inclusion of the self-distillation objective yields the best performance and scalability. Scaling factors SF1 for Fl score and SF2 for dPER are computed (c b) x 0.3 + (b- a) 0.4 for results (a, b, from Libri-Dys [30%, 60%, 100%]. In terms of intelligibillit, Gestural Score delivers the best scalability.
Fosl
Tratnin Data
VL-K444
Latr)
(1007%)
Libri-Dy (107%)
Latri-Py (60%)
Lami-Dp (100))
I1 UDM 2
VCTK-+
Libi-Dys
74.8
78.3
63.9
60.
75.0
82.5
62.9
63.9
84.3
77.3
60
6h
$6.3
75.0
62.3
65.3
34.1
75.9
65.2
61.1
-0.61
-0.07
0.64
-0:5
SSDM
Libr Dys
VCTK=*
$4.5
78.9
64.3
68.3
$7.5
79.0
68.2
69.1
89.5
79.3
69.3
69.7
80.5
89.0
69.9
69.5
31.4
89.2
70.2
70.4
0.76
0.26
0.19
0.17
v/o LLMMA
Liki-Dys
VCTK.*
84.5
78.2
64.0
68.1
78.3
$6.9
696
6%.0
78.8
88.4
69.2
69.7
88.7
79.6
69.8
69.3
SU.7
88.9
69.9
70.0
0.18
0,63
0.25
0U07
w/ DTW
Libi-Dys
VCTK+*
75.9
80.3
65.6
609
83.5
76.3
67.4
65.9
84.
76.7
67.5
66.2
77.9
85.0
66.6
68.
85.1
780
684
67.2
0.38
0.34
0.32
G8
Libr Dys
VCTK=*
84.3
76.9
64.1
$6.9
77.0
65.0
65.4
77.5
87.4
67.8
66.2
87.1
76.6
66.3
68.
87.3
68.4
66.5
-0099
0.21
0.1
w/ Cum
Libi
VCTK=+
85.6
65.1
87.1
79.4
68.5
88.8
69.9
89.2
70.2
90.0
71.9
0.39
063
Table 2: Scalable Dysfluent Detection Evaluation (Simulation)

---

6.4 Scalable Dysfluency Evaluation
We follow (2) by using Fl (type match) and MS (matching score). The matching score is defined as follows: if the IoU (Intersection over Union) between the predicted time boundary and the annotations is greater than or equal to 0.5, and the type also matches, is considered detected. We use H-UDM (2), the current ofte-of-theeart time-aware dysfluency detection model, as the baseline. Under our SSDM framework, we include several ablations: (1) We remove LLaMA and use a template matching algorithm [2 on top of CSA alignments: (2) We replace CSA with softDTW 199; (3) We replace gestural scores with WavLM [95] units; (4) We adopt curriculum training, first training the gestural VAE, CSA, and LLaMA separately, then training them end-to-end. For language model outputs, we set the prompt and use [109] to automatically extract both types and time information from the response. The results Table 2 show similar trends in terms of both performance and scalability (SFI and SF2). Notably, we observe that LLaMA modeling does not contribute significantly, while both gestural scores and CSA (especially the latter) contribute the most. is also noted that dysfluent phonetic intelligibility, as shown in Table I is highly correlated with detection performance.
6.5 State-of-the-art Dysfluency Detection
We select the optimal configuration and compare it with state-of of-the-art speech understanding systems. For fair comparison, we fine-tune LTU-AS- 13B (24 and SALMONN-13B [27 using the same instructions but with pure speech input (AST [TO for LTU-AS and Whisper 17 for SALMONN). Additionally, we attach a time embedding to model temporal aspects. Detailed information is available in Appendix A.8 We also test real nfvPPA speech, with results presented in Table Current large- scale models [24 109] show limited performance in dysfluent speech detection, as shown in Fig. [ The detection of nfvPPA speech remains challenging due to the significant gap between simulated and group github......... 1o/ L//SDM//
real disordered speech. See our demo at phttps:///aat htteee//beeeeeyyypeeeeeeeeee
Eval Dati
LTU-AS-138
LTUU-SS-3B-FT
SALMONN-13R
[3]
SALMONN- 13B-FT
CharGPT [[N]]
SSDM
SSDM w/ Carn
FISS, 7) f)
FITS,
MSVS.
P16%, 11
MS(S,
F11S,
MS/9,
f) MS(S, 1>
F1/9. 1)
FI/S.
MS/1.
VCTK++
72
0
12.2
1.7
73
0
14.2
0.5
25.3
0
89.2
70.2
90,0
71.9
Litri Dys
89
0
9.7
1.7
77
0
11.0
25
18.3
0
$1.4
70.4
$1.6
71.0
Table 3: Detection results from state-of-the-ard models.
p
I l((e) z
6.6 Dysfluency Visualization
1
We attempt visualize dystluency in gestural space, as shown in Fig. 5 The correct text is "please" (p i: z), while the real dysfluent speech is (p z). We apply GradCAM [111] to visual- ize the gradient of gestural scores H, shown the right figure. We select the specific gestural scores corresponding to the vowel 'i' (e), and then visualize the corresponding gesture. On the gestural score, the gradient is negative in the center, indicating that the tongue is attempting to move down, which is the incorrect direction for articulation. This observation is meaningful as it provides insight into the dysfluency. Our system also offers explainability and has the potential to serve as a more interactive language learning tool.
Figure 5: Gestural Dysfluency Visualization
Limitations and Conclusions
In this work, we proposed SSDM (Scalable Speech Dysfluency Modeling), which outperforms the current best speech understanding systems by significant margin. However, there are still several limitations. First, we utilize LLMs, whose contribution is mariginal and whose potential has not been fully leveraged. We suspect this is due to the granularity of tokens, and we believe it would be beneficial to develop a phoneme-level language model to address this issue. Second, the current data scale is still inadequate, which is further constrained by computing resources. Third, we believe that learnable WFST 83 82 could provide a more efficient and natural solution to this problem. yet it has not been extensively explored. Fourth, it is worthwhile to explore representations based on real-time Magnetic Resonance Imaging (ttRII [122] or gestural scores [78]. These approaches might enable the avoidance of the distillation process. Recent concurrent works have been focusing on reaedn-based 113 114) and token-based (T5 approaches. It would be useful to explore the combination of these to leverage advantages each side.

---

Acknowledgments and Disclosure of Funding
Thanks for support from UC Noyce Initiative, Society of Hellman Fellows, NIH/NIDCD, and the Schwab Innovation fund.
References
[1] Jiachen Lian, Carly Feng, Naasir Farooqi, Steve Li, Anshul Kashyap. Cheol Jun Cho, Peter Wu, Robbie Netzorg, Tingle Li, and Gopala Krishna Anumanchipalli, Unconstrained dysfluency modeling for dysfluent speech transcription and detection. 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1 1-8. IEEE, 2023.
[2] Jiachen Lian and Gopala Anumanchipalli Towards hierarchical spoken language disfluency modeling. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 539-551, St. Julian's Malta, March 2024. Association for Computational Linguistics.
[3] National Institute on Deafness and Other Communication Disorders. Aphasia. //www wwwwd. mih, htw/hhealth/aphaatie 2017.
https
[4] Cross River Therapy. Dyslexia statistics. sssearch/byssloiiassttttssssssss
https//www.crossrivrrbbaayy com/
2024.
[5] Fortune Business Insights. U.s. speech therapy market size, share & covid-19 impact analysis, by type (speech disorders, language disorders, neurological conditions, swallowing disorders, and others), by age (pediatrics and adults), and country forecast, 2023-2030. https: :/pp:://wwr www. fortunebbsiinesssiiiigg.... cow.:uutt.cccoooooooeeoeeee -
105574 2024.
[6] Fortune Business Insights. Speech and voice recognition market size, share & industry anal- ysis, by technology (voice recognition and speech recognition), by deployment (cloud and on-premise), by end-user (healthcare, it and telecommunications, automotive, bfsi, government & legal, education, retail & ecommerce, media & entertainment, and others), and regional fore- cast, 2024-2032.
com/ wtw.sighttt...ocuuuttt.e........ Industry-reports.
speech- ooperpali.cccooooerereoottttoooe
101382
101382 2024.
[7] Global text-to-speech market size, share, trends, forecast: By offering: Software/solution, service; by mode of deployment: On-premises, cloud; by type: Neural and custom, non-neural; by language type: English, chinese, spanish, hindi, arabic, others; by en-
terprise size; by end use; regional analysis; competitive landscape; 2024-2032. https
7/www. exportmarkkerresearc.. www. to-pert market 2024.
[8] Tech Report. Global language learning market statistics in 2024. https://techreport com/
2024.
[9] Forbes Advisor. How to become a speech pathologsst: p step- guide. https://
www.forbesccm htt.://ww...at.oooo/o.cc/aaaaaaa 2023.
[10] Trustedhealth. htgpsp/ssppppppaaaaaaaooooonnonn http:///nn.aaooo..ooooopopoaeata
waw.boo//eeeaa...ng.....o.ooooor /dyslexichelp.
[11] DyslexicHelp. The unfortunate reality: Medical insurance does not cover dyslexia. https
2024.
[12] Mayo Clinic. Dttp:aaaatt//////.aaaaoaooaoaooa
2022.
[13] Intensive howpaeseeccorreeeeeeeeeeeeeepeee program.
10

---

[14] UCF. Aphasia house. https://hhallcchprrrsssooooo ucf Packet- 20200 2020.ppf.....
upt.at//sttee/////00000////11222 otp.pppaaiiaaiooooopppppaaaaaooa 2020.
[15] Reciprocal scaffolding: A context for communication treatment in aphasia. https:// aphasiology.o http://tt......ooo//////o/oaa100
pdf
[16] Yu Zhang, Daniel S Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, al. Bigssl: Exploring the frontier of large- scale superviie learning for automatic speech recognition. IEEE Journal of Selected Topics in Signal Processing. 16()):111--152, 2022.
[17] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492-285188 PMLR, 2023.
[18] Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01033 2023.
[19] Jiachen Lian, Alexei Baevski, Wei-Ning Hsu, and Michael Auli. Av-data2vee: Self-supervised learning of audio-visual speech representations with contextualized target representations. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023.
[20] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoory Vyas, Maryam Fazel-Zarandi, et al. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97): 2024.
[21] Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. Slam: A unified encoder for speech and language modeling via speech-text joint pre-training. arXiv preprint 10. 2021.
[22] Alexis Conncau, Min Ma. Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798 805. IEEE, 2023.
[23] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint orXiv:2305.107990 2023.
[24] Yuan Gong, Alexander H Liu, Hongyin Luo. Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
[25] Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Za- Ián Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint 2rxiv:2306.12922 2023.
[26] Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul Rubenstein, et al. Slm: Bridge the thin gap be- tween speech and text foundation models. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
[27] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:23100 13289, 2023.
[28] Chien-yu Huang, Ke-Han Lu, Shih- Wang, Chi- Yuan Hsiao, Chun- Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-superb: Towards dynamic, collaborative, and comprehensive instruction ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1216-. IEEE, 2024.
benchmark for speech. In
11

---

[29] Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arxiv:2402.018831 2024.
[30]
OpenAL. Gpt4-o. http:///pppaaat.mommmmmnaaaalaap
2024.
[31] Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, Kyu J Han, and Katrin Kirchhoff. Speechverse: A large-scale generalizable audio language model, 2024.
[32] Ooi Chia Ai, M. Hariharan, Sazali Yaacob, and Lim Sin Chee. Classification of speech dysfluencies with mfce and Ipee features. Expert Systems with Applications, 39(2)2157-22155, 2012.
[33] Lim Sin Chee, Ooi Chia Ai, M. Hariharan, and Sazali Yaacob. Automatic detection of prolongations and repetitions using Ipcc. In 2009 International Conference for Technical Postgraduates (TECHPOS), pages 1-4, 2009.
[34] Iman Esmaili, Nader Jafarnia Dabanloo, and Mansour Vali. Automatic classification of speech dysfluencies in continuous speech based on similarity measures and morphological image processing tools. Biomedical Signal Processing and Control, 23:104-114, 14,, 2016.
[35] Melanie Jouaiti and Kerstin Dautenhahn. Dysfluency classification in speech using a biological sound perception model. In 2022 9th International Conference on Soft Computing & Machine Intelligence (ISCMI). pages 173-177, 2022.
[36] Tedd Kourkounakis, Amirbossein Hajavi, and Ali Etemad. Detecting multiple speech disfluen- cies using a deep residual network with bidirectional long short-term memory. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
2020.
[37] Tedd Kourkounakis, Amirhossein Hajavi, and Ali Etemad, Fluentnet: End-to-ent detection of stuttered speech disfluencies with deep learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22299699999, 2021.
[38] Sadeen Alharbi, Madina Hasan, Anthony JH Simons, Shelagh Brumfitt, and Phil Green. Sequence labeling to detect stuttering events in read speech. Computer Speech & Language. 62:101052, 2020.
[39] Melanie Jouaiti and Kerstin Dautenhahn. Dysfluency classification in stuttered speech using deep learning for real-time time applications. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6482-6486 2022.
[40] Stacey Oue, Ricard Marxer, and Frank Rudzicz. Automatic dysfluency detection in dysarthric speech using deep belief networks. In Jan Alexandersson, Ercan Altinsoy, Heidi Christensen, Peter Ljunglöff, François Portet, and Frank Rudzicz, editors, Proceedings of SLPPAT 2015: 6th Workshop on Speech and Language Processing for Assistive Technologies, pages 60-64, Dresden, Germany, September 2015, Association for Computational Linguistics.
[41] Sebastian Peter Bayerl, Dominik Wagner, Elmar Nöth, and Korbinian Riedhammer. Detecting dysfluencies in stuttering therapy using wav2vec 2.0. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 2868-2872. ISCA, 2022.
[42] Peter Howell and Stevie Sackin. Automatic recognition of repetitions and prolongations in stuttered speech. In Proceedings of the first World Congress on fluency disorders, volume 2, pages 372-374. University Press Nijmegen Nijmegen, The Netherlands, 1995.
[43] Sadeen Alharbi, Anthony Simons, Shelagh Brumfitt, and Phil Green. Automatic recognition of children's 's read speech for stuttering application. In 6th. Workshop Child Computer Interaction (WOCCI 2017), eds. Evanini, M. Najafian, S. Safavi and K. Berkling, pages 1-6. International Speech Communication Association (ISCA), 2017.
12

---

[44] Tian-Swee Tan, Helbin-Libon A. K. Ariff, Chee-Ming Ting, and Sh-Hussaii Salleh. Ap- plication of malay speech technology malay speech therapy assistance tools. In 2007 International Conference on Intelligent and Advanced Systems, pages 330-334, 2007.
[45] Sebastian P. Bayerl, Maurice Gerezuk, Anton Batliner, Christian Bergler, Shahin Amiriparian Björ W. Schuller, Elmar Nöth, and Korbinian Riedhammer. Classification of stuttering the compare challenge and beyond. Comput. Speech Lang., 81:101519, 2023.
[46] Sebastian P. Bayerl, Dominik Wagner, Ilja Baumann, Florian Hönig, Tobias Bocklet, Elmar Nöth, and Korbinian Riedhammer. stutter seldom comes alone cross-corpus stuttering detection as a multi-label problem. CoRR, abs/2305. 19255, 2023.
[47] Ankit Dash, Nikhil Subramani, Tejas Manjunath, Vishruti Yaragarala, and Shikha Tripathi. Speech recognition and correction of stuttered speech. In 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 1757-1760, 2018.
[48] Payal Mohapatra, Bashima Islam, Md Tamzeed Islam, Ruochen Jiao, and Qi Zhu. Efficient stuttering event detection using siamese networks. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[49] Olabanji Shonibare, Xiaosu Tong, and Venkatesh Ravichandran. Enhancing asr for stuttered speech with limited data using detect and pass. arXiv preprint
2022.
[50] John Harvill, Mark Josegowa-Johnson, and Changdong Yoo. Frame-level stutter detection. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, volume 2022, pages 2843 2447, 2022.
[51] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-War Wen Li, Karen Livescu, Lars Maaløe, et al. Self- supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):117--12U1 2022.
[52] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound-stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing. 30:495-507, 2021.
[53] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: language modeling approach to audio generation. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523 22223,253,,222 2023.
[54] Felix Kreuk et al. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209. 253552 2022.
[55] Alexandre Défossez et al. High fidelity neural audio compression. arXiv preprint #xXiv:2210.13438 2022.
[56] Chengyi Wang et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.0211 2023.
[57] Ziqiang Zhang al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.0m 033926, 2023.
[58] Tianuui Wang et al. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305. 16107, 2023.
[59] Zalán Borsos et al. Soundstorm: Efficient parallel audio generation. arXiv preprint rmXiv:2305.09636, 2023.
[60] Yi-Chiao Wu et al. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5, IEEE, 2023.
[61] Dongehao Yang et al. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02755, 2023.
13

---

[62] Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. I6692, 2023.
[63] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvggan. In A. Oh, T. Naumann, Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 27980-27993. Curran Associates, Inc., 2023.
[64] Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. ICASSP 2024 -024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 591- 595, 2024.
[65] Qian Chen et al. Lauragpt: Listen, attend, understand, and regencrate audio with gpt. arXiv preprint arXiv:2310.04675 2023.
[66] Xiaofei Wang et al. Speechx: Neural codec language model as versatile speech transformer. arXiv preprint arXiv:2308. 06873, 2023.
[67] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Defossez. Simple and controllable music generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 47704 47720. Curran Associates, Inc., 2023.
[68] Isaac Newton, Newton's law of motion. https:///ewwwkippeeeee wwwp:aa..gwgwwiii/iiaeeeeeeeeeee Nite///o////
27s laws of motion
[69] Catherine P Browman and Louis Goldstein. Gestural specification using dynamically-deffned articulatory structures. Journal of Phonetics, (8(3):299-320) 1990.
[70] Catherine P Browman and Louis Goldstein. Articulatory gestures as phonological units. Phonology, 6(2):201-211, 1989.
[71] Catherine P Browman and Louis Goldstein. Articulatory phonology: An overview. Phonetica, 49(3-4):155-180, 1992.
[72] Jessy W Grizzle, Christine Chevallereau, Aaron D Ames, and Ryan W Sinnet. 3d bipedal robotic walking: models, feedback control, and open problems. IFAC Proceedings Volumes, 43(14):505-532,222 2010.
[73] Vikram Ramanarayanan, Louis Goldstein, and Shrikanth S Narayanan. Spatio-temporal articulatory movement primitives during speech production: Extraction, interpretation, and validation. The Journal of the Acoustical Society of America, 134(2):1378-13)) 2013.
[74] Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 5(9), 2004.
[75] Paul D O' grady and Barak A Pearlmutter. Convolutive non- non-negative matrix factorisation with a sparseness constraint. In 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, pages 427-432. IEEE, 2006.
[76] Alan A Wrench. A mmulti-channelmuuli--se speaker articulatory database for continuous speech recognition research. Phonus., 2000.
[77] Jiachen Lian, Alan W Black, Louis Goldstein, and Gopala Krishna Anumanchipalli. Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition. In Proc. Interspeech 2022, pages 4686-4690, 2022.
[78] Jiachen Lian, Alan Black, Yijing Lu, Louis Goldstein, Shinji Watanabe, and Gopala Anumanchipalli. Articulatory representation learning via joint factor analysis and neural matrix factorization. In ICASSP 2023-2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
14

---

[79] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[80] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sondereg- ger. Montreal forced aligner: Trainable text-speech alignment using kaldi. Interspeech, volume 2017, pages 498-502 5028502,22 2017.
[81] Loic Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Dup- penthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint urXiv:2312.05187, 2023.
[82] Theodoros Kouzelis, Georgios Paraskevopoulos, Athanasios Katsamanis, and Vassilis Kat- souros. Weakly-superriied forced alignment of disfluent speech using phoneme-level modeling. In Proc. INTERSPEECH 2023, pages 1563-1567, 2023.
[83] Mehryar Mohri, Fernando Pereira, and Michael Riley. Weighted finite-state transducers in speech recognition. Computer Speech & Language, 16(1 (6(1))69-888 2002.
[84] Korin Richmond, Phil Hoole, and Simon King. Announcing the electromagnetic articulog- raphy (day 1) subset of the mngu0 articulatory corpus. In Twelfth Annual Conference of the International Speech Communication Association, 2011.
[85] Alan Wrench. Mocha: multichannel articulatory database. http::/www.ccn .d. ac. uk/
ressarca//ppojjett/rttic/c/oaaa html 1999.
[86] Mark Tiede, Carol Y Espy-Wilson, Dolly Goldenberg, Vikramjit Mitra, Hosung Nam, and Ganesh Sivaraman. Quantifying kinematic aspects of reduction a contrasting rate production task. The Journal of the Acoustical Society of America, 141(5_ 141(511111p0011 Supplement):33580 35800, 2017.
[87] An Ji, Jeffrey Berry, and Michael T Johnson. The electromagnetic articulography mandarin accented english (ema-mae) corpus of acoustic and 3d articulatory kinematic data. 2014 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 7719-7723. IEEE, 2014.
[88] Cheol Jun Cho, AbdeIrahman Mohamed, Alan W Black, and Gopala K. Anumanchipalli, Self- supervised models of speech infer universal articulatory kinematics. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 12061-12065. IEEE, 2024.
[89] Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, pages 1660-1669. PMLR, 2020.
[90] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[91] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax. arXiv preprint 161.v1611.00144. 2016.
[92] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprini arXiv:2210. 13438, 2022.
[93] Jiatong Shi, Hirofumi Inaguma, Xutai Ma, Ilaa Kulikov, and Anna Sun. Multi-resolution hubert: Multi-resolluioon speech self-supervised learning with masked unit prediction. arXiv preprint arXiv:2310.02720, 2023.
[94] Jaehyeon Kim, Keon Lee, Seungjun Chung, and Jaewoong Cho. Clam-tts: Improving neural codec language model for zero-shot text-to-speech. arXiv preprint arXiv:2404.0278. 2024.
15

---

[95] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):105-11118, 2022.
[96] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
[97] Hiroaki Sakoe. Dynamii-programming approach to continuous speech recognition. In 1971 Proc. the International Congress of Acoustics, Budapest, 1971.
[98] Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing. 26(1):43 49, 1978.
[99] Marco Cuturi and Mathieu Blondel. Soft-dtw: differentiable loss function for time-series. In International conference on machine learning, pages 894-903, PMLR, 2017.
[100] Daniel Hirschberg. Algorithms for the longest common subsequence problem. Journal of the ACM (JACM), 24(4):664-615 1977.
[101] Lasse Bergroth, Harri Hakonen, and Timo Raita. A survey of longest common subsequence algorithms. In Proceedings Seventh International Symposium on String Processing and Information Retrieval. SPIRE 2000, pages 39-48. IEEE, 2000.
[102] Cmu phoneme dictionary.
cs cmu wdw/egi-iiie//eeee hattl/ggi-bincmmaetee bin/cmuuicct
[103] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971,
[104] Zhuohan Chiang, Zi Li, Ying Lin, Zhanghao Sheng, Hao Wu, Lianmin Zhang, and Zheng. Vicuna: An open-source chatbot impressing gpt-4 with 902023.
[105] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -Zuu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[106] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. In Gernot Kubin and Zdravko Kacic, editors, INTERSPEECH, pages 1526-1530. ISCA, 2019.
[107] Maria Luisa Gorno-Tempin! Tempini, Argye E Hillis, Sandra Weintraub, Andrew Kertesz, Mario Mendez, Stefano Cappa, Jennifer Ogar, Jonathan D Rohrer, Steven Black, Bradley Boeve, al. Classification of primary progressive aphasia and its variants. Neurology, 1001)):00--111, 2011.
[108] Wei-Ning Hsu, Benjamin Bolte, Yao- Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460 14400, 2021.
[109] ChatGPT. Chatgpt. https: : //ch//cctt openai. com/ 2022.
[110] Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 202/, pages 571-575, 2021.
[111] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision. pages 618-626, 2017.
[112] Peter Wu, Tingle Li, Yijing Lu, Yubin Zhang, Jiachen Lian, Alan W Black, Louis Goldstein, Shinji Watanabe, and Gopala K. Anumanchipalli, Deep Speech Synthesis from MRI-Based Articulatory Representations, In Proc. INTERSPEECH 2023, pages 5132-5136. 2023.
16

---

[113] Xuanru Zhou, Anshul Kashyap, Steve Li, Ayati Sharma, Brittany Morin, David Baquirin, Jet Vonk, Zoe Ezzes, Zachary Miller, Maria Tempini. Jiachen Lian, and Gopala Anumanchipalli Yolo-stutter: End-to-end region-wise speech dysfluency detection. In Interspeech 2024, pages 937-941, 2024.
[114] Xuanru Zhou, Cheol Jun Cho, Ayati Sharma, Brittany Morin, David Baquirin, Jet Vonk, Zoe Ezzes, Zachary Miller, Boon Lead Tee, Maria Luisa Gorno Tempini, et al. Stutter-solver: End-to-end multi-lingual dysfluency detection. arXiv preprint arXiv:2409.080 09621,, 2024.
[115] Xuanru Zhou, Jiachen Lian, Cheol Jun Cho, Jingwen Liu, Zongli Ye, Jinming Zhang, Brittany Morin, David Baquirin, Jet Vonk, Zoe Ezzes, et al. Time and tokens: Benchmarking end-to-end speech dysfluency detection. arXiv preprint arXiv:2409. 13582, 2024.
[116] Cheol Jun Cho, Peter Wu, Tejas S Prabhune. Dhruv Agarwal, and Gopala K Anumanchipalli Articulatory encodec: Vocal tract kinematics codec for speech. arXiv preprint arXiv:2406.12990 2024.
[117] Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, al. Cstr vetk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2019.
[118] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech, Advances in neural information processing systems, 32, 2019.
[119] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.0455 2020.
[120] Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077 2020.
[121] Matthew Lc, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
[122] Jeff Donahue, Sander Dieleman, Mikolaj Binikowki,, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. arXiv preprint 20X/v:2000.03355, 2020.
[123] Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non-attentive tacotron: Robust and controllable neural its synthesis including unsupervised duration modeling. arviv preprint ar arXiv:2010.04300 043001, 2020.
[124] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[125] Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 1962.. Curran Associates, Inc., 2023.
[126] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International conference on machine leurning, pages 2722-2730. PMLR, 2019.
[127] Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker its and zero-shot voice conversion for everyone. International Conference on Machine Learning, pages 2709-2720. PMLR, 2022.
17

---

[128] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with ad- versarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530-5540. PMLR, 2021.
[129] Korin Richmond, Phil Hoole, and Simon King. Announcing the electromagnetic articulog- raphy (day 1) subset of the mnguO articulatory corpus. In Twelfth Anmual Conference of the International Speech Communication Association, 2011.
[130] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[131] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent-a new approach to self supervisd learning. Advances in neural information processing systems, 33:21271 12221271-21284, 2020.
[132] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[133] Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee. Distilhubert: Speech representation learn- ing by layer-wise distillation hidden-unit bert. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7087-7091. IEEE, 2022.
[134) Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN) pages 1-8. IEEE, 2021.
[135] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vee: general framework for self-supervised learning speech, vision and language. In International Conference Machine Learning, pages 1218-1312. PMLR, 2022.
[136] Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli Efficient self-supervised learn- ing with contextualized target representations for vision, speech and language. In International Conference on Machine Learning, pages 1416-1429. PMLR, 2023.
[137] Yifan Peng, Yui Sudo, Shakeel Muhammad, and Shinji Watanabe. Dphubert: Joint distillation and pruning of self-supervised speech models. arXiv preprint 23Xiv:2305.177 2023.
[138] Alexander H Liu, Heng-Jui Chang. Michael Auli, Wei-Ning Hsu, and Jim Glass. Dinosr: Self- distillation and online clustering for self-supervised speech representation learning. Advances in Neural Information Processing Systems, 36, 2024.
[139] Cheol Jun Cho, Abdelrahman Mohamed, Shang- Wen Li, Alan W Black, and Gopala K Anumanchipalli. Sd-hubert: Sentence-level self-distillation induces syllabic organization in hubert. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 12076 12080. IEEE, 2024.
[140] Jacqueline Ann Bauman- Bauman-Waängler. Articulation and phonology in speech sound disorders a clinical focus. Pearson Education, Inc., 2020.
[141] John E. Bernthal, Nicholas W. Bankson, and Peter Flipsen. Articulation and phonological disorders: Speech sound disorders children. Pearson, 2017.
[142] Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiali Yao, Antonios Anastasopoulos, David R Mortensen, Graham Neubig, Alan Black, and Metze Florian. Universal phone recognition with a multilingual allophone system. In ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8249-8253. IEEE, 2020.
18

---

[143] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[144] Jiachen Lian, Chunlei Zhang, and Dong Yu. Robust disentangled variational speech repre- sentation learning for zero-shot voice conversion. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
[145] Jiachen Lian, Chunlei Zhang, Gopala Krishna Anumanchipalli, and Dong Yu. Towards Improved Zero-shot Voice Conversion with Conditional DSVAE. In Proc. Interspeech 2022, 2022.
[146] Jiachen Lian, Chunlei Zhang, Gopala Krishna Anumanchipalli, and Dong Yu. Utts: Unsu- pervised tts with conditional disentangled sequential variational auto-encoder. arXiv preprint arXiv:2206.0253 02512, 2022.
[147] Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa- Johnson, and Shiyu Chang. Contentvec: An improved self-supervised speech representation by disentangling speakers. ICML, 2022.
[148] Hycong-Seok Choi, Jinhyeok Yang. Juheon Lee, and Hyeongju Kim. Nansy++: Unified voice synthesis with neural analysis and synthesis. ICLR, 2022.
[149] Yang Gao, Jiachen Lian, Bhiksha Raj, and Rita Singh. Detection and evaluation of human and machine generated speech in spoofing attacks on automatic speaker verification systems. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 544-551. IEEE, 2021.
19

---

A Appendix supplemental material
A.1 Gestural Modeling Visualization
A.1.1
What are gestures and gestural scores?
The raw articulatory data X E [22x(, where 1 represents time with a sampling rate of 200 Hz, includes x and y coordinates of six articulators: Upper Lip, Lower Lip, Lower Incisor, Tongue Tip, Tongue Blade, and Tongue Dorsum. Here, X is sourced from 2... As motion data, X can be decomposed into gestures G and gestural scores H. T" represents window size that is much smaller than t, set at T' = 200 ms. Fig. 6 provides an example with only three gestures: G1 corresponds to a 200 ms trajectory of upper lip movement, G2 to the lower lip, and G3 to the tongue dorsum. This is illustrative; in our work, we use 40 gestures, approximately the size of the CMU phoneme dictionary [02] excluding stress. The gestural score H 3x3 where corresponds to three gestures with corresponding indices. The first row represents the duration and intensity of gesture 1, the upper lip movement, and similarly for gestures and 3. After decomposition, we obtain gestural scores as representations, which include duration and intensity, supporting co-articulation from different articulators with potential overlap. These scores are typically sparse and under certain conditions, they are also interpretable (77]. The next question is: where do these gestures come from? We performed k-means clustering on the original data X. Specifically, we stacked every 200 ms segment of data from X into supervector. Then, applied k-means clustering to all these supervectors across the entire dataset X. A simple example to understand gestures and gestural scores might be this: if simplified dictionary includes the gestures "lower lower lip" and "raise upper lip," each with a duration second and normalized intensity, these would occur simultaneously for that duration.
Gestures
Gestural Scores
G E IR Tx12x3 = O 11, G| - G. G3.
H E 3xt
Kmeans
G1 E 12xT
12
3
G2 E R 12N7
12
Kmeans
Decompose
G3 E R 12xT
12
Kmeans
12.
Upper Li (UL)
Lower Lio (LL)
Lower (LI)
Tongue To (TT)
Tongue Blade (TB)
Tongue Donsum (1D)
Data
Visualization
Articulatory data
X E 12x
Figure 6: Gestures, Gestural Scores, Raw Data Visualization
A.1.2 How does decomposition work?
As shown in Fig 7 convolutive matrix factorization (C5F) decomposes X E R222xt into gestures
G e R2x12x3 and gestural scores H R3xtt The original CMF algorithm 75 iteratively updates
G' and H. For illustrative purposes, let us consider the reverse process. Given G, we select Gi;, i
20

---

3 for - 1, 12.
Taking G|:, 0, as an example, then [[]][[] 0, H : k +T. where denotes the element-wise product sum.
Gh,0,1" e
Comvolatio
Gla; gaay
X = GBA
A R
Gis 11 Gr e ea tat
12-
R
Carvolition
Gb44 Gg R taxt
Figure 7: Illustration of Convolutive Matrix factorization
A.2 Neural Variationl Gestural Modeling
77 proposed using a neural network to predict H and replacing the traditional CMF process with a one-layer I-D convolutional neural network. However, strictly adhering to CMF results simplistic neural network architecture that limits the expressiveness of the learned gestural scores. We employ advanced probabilistic encoder to predict H. implicitly modeling duration and intensity. Subsequently, we use a multi-scale decoder to simulate the CMF process and recover X. We will discuss the details in the following sections.
A.2.1 Universal Acoustic to Articulatory Inversion (UAAI)
Since the real articultory data are typically unavailable, we employ state-of-the-arr acoustic-to- articulatory inversion (AAI) model (88) pretrained on MNGUO [8]. The model takes 16kHz raw waveform input and predicts 50Hz EMA features. We then upsample them to 200Hz. Although the AAI model was pretrained on single speaker EMA, is a universal template for all speakers (88). A concurrent study [116) further demonstrates that by performing thesis, the single-speaker EMA representations derived from multi-speaker speech corpora, such as LibriTTS [106] and VCTK (117), maintain sufficient level of intelligibility. Note that the entire system should considered a speech-only system, as it does not require the use of any real EMA data during its operation.
Corech-t-EEMA-t--pppe
resyn-
A.2.2 Implicit Duration and Intensity Modeling
However, there three major differences between gestural scores and TTS duration modeling. First, unlike TTS, which enforces a monotonic alignment where each phoneme has single duration, gestural scores permit independent gestures with overlapping durations, capturing co-articulation complexities [71]. Second, while TTS aligns text with speech, gestural scores lack a reference target for each gesture's duration. Third, durations in gestural scores are linked with intensities. Therefore, traditional differential duration modeling methods such as regression-bast based approaches (118)121), Gaussian upsampling [22 and variational dequantization 128 lead to unstable training in our setup.
Here visualize duration and intensity prediction modules in Fig.
Given the input X E R 12xt
latent encoder is utilized to derive latent representations Z E R P)x Subsequently, Z
21

---

is reshaped into three- ditenninmnnio representation Z E R 12 Pxt, where P denotes the patch embedding size for each patch index (k, i), with (in our configuration, K 40), and is the time index. Fach patch embedding is concatenated with [[k: :l. which scalar, to form a P embedding. This composite then processed through the Intensity Encoder and Duration Encoder to predict their respective posteriors. For the intensity posterior, values are treated as floats. As gestural scores must always remain positive for enhanced interpretabilityy. Sigmoid function is applied to the sampled intensity Ihi The duration predictor operates as a classifier, where we establish the class set as [1, 2,3...... ,........000 thus constituting 50-way classification problem. Due to the cinddffeerentiblle nature of the sampling process, we employ Gumbel Softmax [91] to generate the duration posterior. Consequently, for each point (k, t) in the gestural score, obtain continuous positive intensity Sigmoid (I and
I,..., K representing the gesture index
discrete duration D A Hanning window is applied across the entire window: H' - ++ 2 Hann (Sigmoid ~ ( 1 xx )), Dhr
(o(D". I X)). The Hanning window is
defined as w(n)) = Sigmoid (annid((A::)) ( (1 cos f and D
2mm
making it differentiable with respect to both
D
01:000
Hann (Sigmonii (1+)) (*))
1
Sample
X e Rit
(k,
X E tlrt
If & R
Latent Encoder
Intensity Encader
D
J4, a
Simplo
Encoder
Softmax
())
(D 1222 X)
1,2,, 12,5 1,2,...
Figure 8: Duration and Intensity Modeling
A.2.3 Sparse Sampling
The raw gestural scores H, as shown in Fig. A.2.2 represent a dense matrix. For each position (I. i). a Hann window is applied, reflecting the inherent sparsity observed in human speech articulation [71]. To enhance this sparsity, we implement a sparse sampling method. As illustrated in Fig. 9 we define a ranking score 5t. k. + bD* where the coefficients are set to a select the top myow points based on the highest ranking scores. From this, we derive Mask matrix M Mrow, which is applied to the original H to produce sparse gestural score H E Kx
and b 1. We then
A.2.4 Multi-Scale Gestual Decoder
Traditional neural convolutional matrix factorization (CMF) 777 uses a single-layer neural network, which significantly limits the expressiveness of gestural modeling.
In this new decoder, we consider two sampling factors, and r According to Eq. 5 Fig. 10
fully visualizes the multi-scale gestural decoder architecture. Note that the final representation H downsampled by factor of 4 to ensure consistency with the acoustic features from WavLM 995
22

---

From One Patch
Raw Gestural Scores H Rx
Sparse Gestural Scores
g
(K,5)
Rank Score S alA! bDt
Mllee 2
Mask Maa
Figure Sparse Sampling
=0JH,G)=x0D...
X c 10
Up Sample
Σ
Convolution
X = lixt
Gestural Encoding (Latent Encoding)
(Duration) (Intensity)
Sparse
Sooms (ft)
Final Representation:
Figure 10: Multi-Scal. Gestural Decoder
A.3
Self-Distillation
Background and Intuition Electromagnetic articulography (EMA) data, sourced from either real samples [[29] or UAAI [88), are typically sparsely sampled, leading to information loss. While concurrent work [166] achieves satisfactory intelligibility mostly the word level, our objective is to enhance phonetic understanding. Furthermore, gestural scores precisely delineate phoneme boundaries [77. This prompts the exploration of additional constraints to synergize these aspects. Self-distillation has been successful in computer vision 132 and speech [133] revealing emergent properties like unsupervised image segmentation 132 and speech semantic segmenta-
tion 139
Methods We perform self-distillation frame, indicated by the time index i in Fig. From the acoustic adaptor, we introduce the gestural score Å, which is downsampled by factor of four, aligning it with the same resolution as speech features (95). This allows us to derive the posterior gol [i] A(]).. From the text encoder, which processes phonemes, we input the predicted phoneme embedding 7 into the textual distribution parameters b og obtaining pe (TT(CC for each time step i.
23

---

Subsequently, through a flow and change of variable, we derive the prior distribution pe CH((C) KL divergence is then applied between pe (H [C] and go / [i])) to facilitate self-distillation.
Dysfuency Targets: SIL SH (Fram Simulation)
Reference
Normal Targets: Sill W IH SH
Phonemizer
Acoustic
Softmax Loes
Text Encoder
Adaptor
Downsampled Gestural Soure
H
T [ Thy-.
72)
From 12
q6 (E Ale))
A(8))
pe(n)
RL Divergence
(Self Distilation)
Figure 11: Colf-Distillation Paradigm
A.4 ELBO with latent variables
IX) log / pe (X D,),
(16)
log f 40 (0, D, T po (X, dZ dD dT
(17)
2,1X),
loo (2,
(18)
(,(2,
Po(X,
2E,,(2,,111)))
log
(Jensen's 's inequality)
(19)
(o(2,
(Z.D.Ii) [lo Po (XID,I, G) Po E. (Z.D.II)) log Po( D. 1)
(20)
D. (.(Z.DIII))
(2, [lo Po (XID,
(21)
B.HKNNS KI. (6 kh D..., pk: 1 (X)lp DRis, pk.1))
(22)
-Eo(ZZ,D. Po Po (X (X),,I,))
(23)
- E(KKi)~ KL (go ( [X)]I((Zh))
(24)
KL (as (Dhi zk, )l)((())))
(25)
KL (gs (Th.)) X, 2t.
(26)
CELBO
(27)
24

---

A.5
LCS Pseudo Code
Algorithm I Find Longest Common Subsequence (LCS)
Require: Target sequence seq1, Source sequence seq2
Ensure: Longest Common Subsequence (LCS) alignment
1: Initialize a 2D array lengths of size (len(seql) + +1) (len(seq2) 1) with zeros
2: for each element i in seq1 do
3: for each element in seq2 do
4: seq [i] = seq2 [j] then
5: lengths [i+1] []+1] lengths [i] [j] +
6: else
7: lengths [i+1] [j+1] maxngtt [i+1] [j], lengths [i] (j+1])
8: end if
9: end for
10: end for
Initialize an empty list align_cca_result store the LCS alignment len(seq2)
12: Set x = len(seq]),
while x * 0 and & do
14: if lengths [y] = lengths [x-1] [y] then
x:
16: else if lengths [x] [y] = lengths [x] [y-1] then
17:
align_lcs_reste _esuutessstl_.eeeeeeeeeee append ((ennnd(((see22)) (seq2[y-11) seq1 (x-1]))
18:
y/ - 1-
else
20:
align_lcss lesg__cc__resslle www.enn....pppnddleeeeeeea aww.add (y-) seql (x-]))
21: ==--1
22: == y-
23: end if
24: if = and y / 0 then
25:
baign_lcs_resil append (seq2[y-1], seql[0])) break
26:
27:
end if
28: if + a 0 then
29: align_lcs aligultcs_reeult...
result........ append (0) seq1 (x-1]))
break end if end while
31:
33:
align_lca se__lcs_ressll reverse()
return align_ les_lcs__re result
25

---

A.6 DTW Pseudo Code
Algorithm 2 Dynamic Time Warping (DTW)
Require: Sequence seq1, Sequence seq2)
Ensure: Alignment of seq1 and seq2
1: lem(seel), m <- len(seq2)
2: Initialize dtw_matrix of size (n + 1) x (m + 1) with do
3: dtw_matrix([] [o]
4: for i= to n. do
5: for j m do
6: cost <- distance(seql(jii1) [
7:
dtw_matrix [j] + cost +
[j], dtw_matrix [i] [j-1].
[tw_matrix 8: end for
9: end for
Initialize an empty list alignment to store the alignment
11: + n.j + m
while i > and > do
13: if seq1 [i-1] = seq2 [j-1] then
wwtpmennt.ppenne ((seq []- seq1 (i-1]))
15:
i-i-1
j+j-1
17:
else
dtw hew_matrix(ie [i-1] [j] = min(dtw_matrix [j], dtw_matrix [j-1],
dtw dtw_matrix [j-1]] then
19:
iti-1
20:
else if dtw [tw_matrix [j-1] == min(dta_matrix(i-1) [j], [i] [j-1].
dtw [ew_matrix(i-1]]] [1] then
21:
alignment.cm append (seq2[j-1)) ( seq1 (i-1]))
22:
j + -
23:
else
24:
alignment.appen ( (seq2[j-1) seq1 (i-1]))
25:
26:
j
end if
28:
end if
i
0 and j j> 0 then while j > do
30:
alignment. hwtpmment.apendde... j+
(seq2[j-1] (oppend(se seq1 [0]))
32:
end while
34:
break
35:
end if
36: if 0 a j= =
then
37:
alignment. append ((seq2[0) seq1 [0]))
38:
break
39: end if
40:
end while
41: Siggmentt.reerssee
42: return alignment
26

---

A.7
Why LSA is better?
Fig. provides an example and illustrates the intuition behind why LSA is a better candidate for dysfluency modeling. The reference text is the phoneme transcription of the word "references". To better illustrate this, the chosen dysfluent speech is significantly impaired, containing insertion of filler sounds such as "uh", repetition of sounds like "R", "EH", "AH", "IH", and insertion of sounds such as "S" and "HR", along with deletion of sounds like "F".
Let us examine the results from LSA and GSR. We aim to obtain all elements (phonemes) in the dysfluent alignment т for each phoneme in the reference text C. LSA captures most dysfluencies through such per-reference alignment. For instance, the alignment (uh, R) to R indicates insertion. Similarly, (EH, R. EH) aligning EH primarily indicates repetition. Up to this point, GSA exhibits similar performance, aligning (uh, R. EH, S, R) to R. which also indicates repetition. However, significant difference emerges thereafter. For the phoneme F in the reference, no phonemes are aligned in LSA, which is correct as it is missing. Conversely, GSA aligns (ER, AH, AH) to F, which is unreasonable. For the phoneme AH the reference, the LSA alignment (AH, AH, ER, AH) indicates repetition, which GSA fails to capture. Similarly, the repetition of IH is accurately captured by LSA but missing in GSA. Our main point that, although dysfluency alignment with the reference text is non-monotonic, aligning corresponding phonemes with each phoneme in the reference monotically enables fine-grained dysfluency analysis, which is naturally captured by LSA. Note that Fig. 3 we use LCS [100] and DTW (97] for illustration.
Also look at Fig. 3 (right), we select a subsequence C C2, C3, C.l =(ER, AH. SI and 5 [T,, T2, 73. Tal =[AH, AH, ER, AH, NJ from Fig. B (left). and provide an illustrative breakdown Fig. B (right). LCS updates the cost function only when C3 = Ti AH and C4 = = N. excluding the remaining phonemes [72, Ta] = [AH, ER, AH] from the cost function, as they are not essential for determining the alignment boundary. This is particularly relevant for 73- = ER, which is unrelated to the reference phoneme C3 = AH. In contrast, DTW considers all phonemes = [TI T2, 73, 74 =[AH, AH, ER, AH] equally. While 72 AH and T4 AH are not crucial for deciding the boundary. their inclusion leads to lower cost and higher weight in contributing to the final alignment. Therefore, LSA's selective cost function updates prove more effective for dysfluency alignment compared to DTW's equal consideration of all phonemes. Pseudo code provided in Appendix. I and Appendix. respectively.
A.8 Language Models
Reference Text:
Instruction:
-o Now
You wish to know all about my grandfather
What are the problems of the pronunciation?
pgiof
Text Encoder
Text Encoder
Additional Prompt
Connedioniel Subsequance Aligner
Time Eneoelig
Encoder
CSAa(r,
LLaMA
LoRA
Sample
(C)rx(C)) ....................... (CE.)(C())
Annotation 1: Word "please"" stutter of ' 0.32 seconds.
Annotation "cal", prolongation
seconds.
Training
Annotation Word "please", Insertion 'x', 0.19s.
Response: For the word "you." there stutter of "Y at 0.60 seconds. For the word 'al.' there block 2.92 seconds. the word "grandlather" stutter seconds, there phonetic
Inference
5.90 seconds.
Figure 12: Instruction Tuning
27

---

A.9
Dysfluency Simulation
A.9.1 TTS-rules
We inject dysfluency in the text space following these rules:
Repetition (oorneme&word): The first phoneme syllable of a randomly picked word was repeated 2-4 times, with pauselengths varying between 0.5 to 2.0 seconds.
Missing (phoneme&word) We simulated two phonological processes that characterize disordered speech [40] weak syllable delction (deletion of random unstressed syllable based on stress markers and final consonant deletion.
Block: A duration of silence between 0.5-2.0 seconds was inserted after randomly chosen word between the sentence.
We simulated fronting, stopping, gliding, deaffrication processes that characterize disordered speech [41 by replacing random phoneme with one that would mimic the phonological processes mentioned above.
Replacement (phoneme):
Prolongation (phoneme); The duration of a randomly selected phoneme in the utterance was extended by factor randomly chosen between 10 to 15 times its original length, as determined by the duration model.
A.9.2 Simulation pipeline
The simulation pipelines can be divided into following steps: (i) Dysfluency injection: We first convert ground truth reference text of LibriTTS into IPA sequences via the phonemizer then add different types of dysfluencies at the phoneme level according to the TTS rules. (ii) StyleTTS2 inference: We take dysfluency-injected IPA sequences as inputs, conduct the StyleTTS2 [25] inference procedure and obtain the dysfluent speech. (iii) Annotation: We retrieve phoneme alignments from StyleTTS2 duration model, annotate the type of dysfluency on the dysfluent region.
We show two samples (waveform and corresponding annotation) on the right side of the figure above.
2)StyleTTS2 inference
3)Annotation
Refference text: You wish know all about my grandfather.
LIPA Sequence:
ju: w to ou o:l aus maz g ond
1/Dysflunecy Injection
Wrrep:
You (wish wish] know all about my grandfather.
You
about my grandfather.
Blook:
Ju w'f to od eb, aut mar gl annjpause do-
dyverTss
You wish
(Tgae
know al
about my
grandishel
Pirep:
ju of [t....L. aja n'on ou l lb aut max gi zeddfa da.
P-miss:
ju: w'll to n'ou ool eb. ault) maz gi aendfa:de. de.
P-replace:
w/ of ta ail eb, aut max mend[mjarde das
Pprolong:
to You't [[xxtndd]
gi aedio da.
m:00
[sporep]
Figure 13: Simulation Pipeline
A.9.3 Datasets Statisties
The specific statistics of Libri-Dys are listed in Table. compared with VCTK++. Figure. T4 presents comparison between our simulated dataset and two existing simulated dysfluency datasets: VCTK++ [ and LibriStutter [37]. indicates that our dataset surpasses the datasets both hours and the types of simulated dysfluencies. Note that since we build dataset based publicly available corpus LibriTTS [166] and styletts2 [25] it satisfies safeguards criterion.
28

---

Table Types Dysfluency Data in VCTK++ and Libri-Dys
Dysfluency
# Samples
Percentage VCTK++
Samples Libri-Dys
Percentage Libri Dys
VCTK++
Prolongation
43738
33.28
288795
13.24
Block
43959
33.45
345853
15.97
Replacement
0
295082
13.63
Repetition (Phoneme)
43738
33.28
340916
15.75
Repetition (Word)
0
301834
13.94
Missing (Phoneme)
296076
13.68
Missing (Word)
296303
13.69
Total Hours of Audio
130.66
3983.44
Libr-Dya
10"3
hours
VCTK+
10^2
LibriStulter
10"1
# Types
Figure 14: Existing Simulated dysfluency datasets
A.9.4 Evaluation
To evaluate the rationality and naturalness of Libri-Dysefluency and use VCTK++ for comparison, we collected Mean Opinion Score (MOS, 1-5) ratings from 12 people. The final results are as displayed in Table. 5 Libri-Dys was perceived to be far more natural than VCTK+ (MOS of 1. compared 2.14).
Table 5: MOS for VCTK+ & Libri-Dys Samples
Dysfluency Type
VCTK++ MOS
Libri-Dys MOS
Block
2.66 = 0.94
3.20 1.26
Missing (phoneme)
N/A N/A 1.33 = 0.47
4.66 1.06
Missing (word)
4.80 + 0.63
Prolong
3.83 0.89
Repetition (phoneme)
1.33 = 0.43
4.93
Repetition (word)
N/A
3.73
Replacement
N/A
3.90 +.99
Overall
2.14 0.64
4.15 + 0.93
A.9.5 Phoneme Recognition
In order to verify the intelligibility of Libri-Dys, we use phoneme recognition model 142 to evaluate the original LibriTTS test-clean subset and various types of dysfluent speech from Libri-Dys. The Phoneme Error Rate (PER) is calculated and presented in Table. 6
29

---

Table Phoneme Transcription Evaluation on LibriTTSS and Libri-Dys
LibriTTS
Libri Dya
Type
W/P-Repetiiin
W/P-Missing
Block
Prolongation
Replacement
PER (%
6.106
6.365 / 11.374
8.663 / 6.537
12.874
6.226
8.001
A.10
Experiments
A.10.1
nfvPPA
In looking for clinical populations to test pipeline, we decided to focus patients with neurode- generative disease called nonfluent variant primary progressive aphasia (nfvPPA). This phenotype falls under the umbrella of primary progressive aphasia (PPA), which is neurodegenerative disease characterized by initially having most prominent disturbances to speech and language functions. PPA has three distinctive variants that correspond with unique clinical characteristics and differential patterns of brain atrophy: semantic (svPPA), logopenic (IvPPA), and nonfluent (nfvPPA) [07]. Dis- turbances speech fluency occur due multiple underlying causes subsuming different speech and language subsystems in all these variants; however, the variant most commonly associated with dysfluent speech nfvPPA. This phenotype is characterized by primary deficits in syntax, motor speech (i.e., in this case, apraxia of speech), or both, and it is this association with apraxia of speech that makes nfvPPA an apt clinical target for assessing automatic processing of dysfluent speech.
Our collaborators regularly recruit patients with this disease as a part of an observational research study where participants undergo extensive speech and language testing with qualified speech- language pathologist. This testing includes a comprehensive motor speech evaluation, which includes oral mechanism exam, diadochokinetic rates, maximum phonation time, multisyllabie word reading, word of increasing length, passage reading, and connected speech samples. For our present purposes, have decided to analyze the speech of participants reading aloud the Grandfather Passage, a passage often used clinically to assess motor speech due to its inclusion of nearly all phonemes of the English language. We have recordings for 10 participants with nfvPPA under IRB with consents signed for educational use. Passage recordings are conducted using high quality microphones for both in-person and remote visits. We randomly select 10 recordings and calculated the occurrences of various dysfluency types within them. The distribution is shown in Fig. [5] Note that PfvPPA data will not be released.
40
30
20
10
Missing
Block
Repetition
Replacement
Prolongation
Figure Dysfluency distribution in nfvPPA
A.11 Model Configurations
The EMA features are denoted as X = [X1, X2,..,, Xz). where X; E IRd, These features represent the positions of 6 articulators, including the upper lip, lower lip, lower incisor, tongue body, tongue tip, and tongue dorsum, with both and coordinates. Consequently, d is equal to 12.
30

---

A.11.1
Acoustic Adaptor
We use WavLM [26] large pretrained acoustic encoder. The acoustic adaptor is simple linear layer with dimensions (784, 40), where 40 is the number of gestures used in this paper. The output of the acoustic adaptor is H E IR [
A.11.2 Gestural Encoder
The latent encoder 4 4-layer transformer with an input size of 12, hidden size of 96, and output dimension of 144. The latent representation is Z E [222x22t, where the patch size is P. Sinusoidal positional encoding [43] is added the input of each transformer layer to provide position information. The intensity encoder is a three-layer MLP with input dimension 12, hidden dimensions [24, 48], and outputs a scalar. The duration predictor a 3-layer transformer with input dimension 12, hidden size 48, and outputs 50-class distribution (0-49 duration bins). Sinusoidal positional encoding added to the input of each transformer layer.
A.11.3 Gestural Decoder
Downsampling is performed by average pooling (every 2 or 4 frames). Upsampling is performed using a deconvolutional layer (also known as transposed convolution) with scale rate of either 2 or The deconvolutional layer has kernel size of (3, 3), stride of (2, 2) (4, depending on the scale rate, and padding of (1, I) to maintain the spatial dimensions. The convolutional weight has the same shape the gestures G 112x40x40, where the window size is 200ms. ftrans.0 is 4-layer transformer encoder with input dimension 12, hidden size 96, and output dimension 12. Sinusoidal positional encoding added the input of each transformer layer.
A.11.4 CSA
The flow ff is glow (96] that takes input size 40 and output size 64, which is the phoneme embedding size. For Cphn- we predict CMU phoneme targets [022] from MFA or simulation. Note that use an offline IPA2CMU dictionary convert IPA to Paato--IAA-Covveeee
//tttp:://ithub com www.commcccooccc htmm//argoouuu.. We use the same text (phoneme encoder) as [119] but with output embeddings size 64. The transition probability CC)) is simply (64, 64) linear layer with sigmoid activation.
Component
Architecture
Details
GLOW Model (
Invertible Flow
Input size: 40, Output size: 64, Flow steps:
Actnomn Layer
Scule R40 Bias E R40
Invertible 1x1 Convolution
Weight matrix W 4000040
Affine Coupling Layer
Split input into two parts of size 20 Affine transformation network:
2 FC layers, 64 hidden units, ReLUU
Table 7: Detailed GLOW model architecture
A.11.5
Language Modeling
In Fig A.8 we use the same text encoder 24] To compute the time information, we use frame rate of 50Hz and provide at the word level (for each C; in the reference text). This time information then passed to the same text encoder. The embedding sizes are all 4090 [103] 24 We follow 24
by using a rank of 8 and 16 in LoRA [105]. All other settings remain the same.
Note that for CSA alignments, we concatenate each 7; with its corresponding word C2 alignments, resulting in a 128 228diimenioonl vector. Another encoder, one-layer MLP (128-4096), maps CSA embeddings into the textual space.
We also have an additional prompt to summarize the actual pronunciation (word, phoneme) and time. The prompt we are using is:
Given the following text, extract the dysfluent words, the type of dysfluency, and the time of occurrence. Return the result as a list of triples where each triple contains (word, type of dysfluency, time).
31

---

A.12 Training Configurations
In Eq.
2
In Eq
a b 1, Myow = 3.
In Eq. 6 and Eq. 7 we simply set K K2 1.
In Eq.
8
A1
X2
Ag =.
In Eq.
12 and Eq. = 0.9.
We first separately train the gestural VAE (Eq(). Subsequently, we train the CSA (Eq 14 followed by training with LAAN. Finally, we retrain the entire model an end-to-end manner. For each step. we use the Adam optimizer and decay the learning rate from 0.001 at rate of 0.9 every 10 steps until convergence. The training conducted using two A6000 GPUs.
For the VAE and language modeling steps, it takes 40 hours to complete the entire Libri-Dys training. CSA training only takes 5 hours converge, where only the linear layer transition probability Co(C) is trainable. The same training duration applies to SALMONN [27] and LTU-AS 24 with fine-tuning. Note that used pretrained models for SALMONN and LTU-AS.
A.13 Speaker-dependent Behavioral Modeling
Speech dysfluency modeling is fundamentally a clinical problem and, consequently, a speaker- dependent issue. While have not conducted per-speaker analysis at this juncture, future research will explore both speaker-dependent and speaker-independen representations [144|18) for clinical analysis. To address potential ethical concerns, have implemented essential voice anonymization techniques [49 in our processing of disordered speech.
A.14 Discussion about Concurrent works
As concurrent work, YOLO-Stutter [113] approaches dysfluency modeling an object detection problem. The authors utilize a simulated corpus and output dysfluency type and timing informa- tion. Stutter-Solve [144 further extends this approach, employing similar pipeline for cross- lingual (English-Chinese) joint simulation and prediction. Notably, Stutter-Solver outperformed H-UDM (2]. Another recent publication, Time-and-Token
[115), treats the problem automatic speech recognition, mapping each dysfluency to a token and achieving performance comparable YOLO-Stutter [I13]. Our model primarily emphasizes scalability and user-friendly interface design. Additionally, it establishes a foundation for future researchers to explore in-context learning capabilities. We intend to conduct comparisons with these aforementioned works in future research.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have claimed our contribution and scope in both Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of the work in Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
See Sec Sec these information.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have introduced our experiments setup, training and model configurations in See 6 A
and
A.12
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [No]
We have open sourced our data (Sec. 5). For code, we are waiting for the other approval.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. and A.12 for these information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the MOS evaluation of Libri-Dys (Table. 5 we calculated the mean and standard deviation of people's 's ratings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See A.12 for these information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We mentioned them in the beginning of introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We mentioned in Appendix. A.9.3 and Appendix, (aaedii..AII..... A..10.1 about dataset safe- guard statement. Currently we will not open source the model until the approval comes.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We obtained data and code in a legitimate way and cited all the work involved.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Assets (code and model) are well described in Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: For nfvPPA data collection, instructions are provided in Appendix. 1.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our used nfvPPA data is under IRB with consents signed for educational use A.....I.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "d6iWXbET74", "reviewer": "Reviewer_YZds", "summary": "This paper presents SSDM (Scalable Speech Dysfluency Modeling), a novel approach to modeling speech dysfluency, which is essential for applications in spoken language learning and speech therapy. The authors identify three main challenges in the current state-of-the-art: poor scalability of existing solutions, the absence of a large-scale dysfluency corpus, and the lack of an effective learning framework. To address these issues, SSDM incorporates several key innovations:\n\n1. Articulatory Gestures as Scalable Forced Aligner: The paper proposes using neural articulating gestural scores as scalable representations for dysfluency modeling, grounded in physical laws of speech production.\n\n2. Connectionist Subsequence Aligner (CSA): Introduced as a differentiable and stochastic forced aligner to achieve dysfluency alignment by linking acoustic representations and text with dysfluency-aware alignment.\n\n3. Libri-Dys Dataset: The authors have developed a large-scale simulated dysfluency corpus called Libri-Dys to facilitate further research in the field.\n\n4. End-to-End System with Large Language Models (LLMs): The system leverages the power of large language models for end-to-end learning, aiming to enhance the understanding of human speech nuances.", "questions": "1. Is there any ablation study on the number of gesture number?\n2. Is there are any sample or metric showing how well can we recover speech from the articulatory representations.\n3. Is mask in Figure 9 necessary?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. SSDM provides a holistic solution by combining articulatory gestures, a new aligner (CSA), a large-scale dataset (Libri-Dys), and leveraging large language models, which could significantly advance the field.\n2. The work has direct implications for speech therapy and language learning, which are significant areas with real-world impact, especially for individuals with speech disorders.\n3. By open-sourcing the Libri-Dys dataset, the authors contribute to the community by providing a valuable resource for further research and development.", "weaknesses": "1. The paper writing is unclear, which makes it quite hard to understand the whole works. For the Figure 2, the author should make it clear which part is used in training and which part is used in inference. Also there should be more explanations on difference of the line and dashed line. Moreover, a detailed overview of the whole system about the training and inference, respectively, is necessary for the complex system like this.\n\n2. From the results in Table 2, it seems that LLAMA is not a necessary part for the system to be effective. I wonder if there is any traditional method on this task, i.e., the output of traditional system is the category label of error type and its duration. Comparison with such systems is also needed.\n\n3. Since the baselines in Section 6.5 uses only speech input, it seems to be an unfair comparison, which explains the large gap between the proposed method and baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nWe are pleased to address your proposed weaknesses and questions.\n\n**For weaknesses**:\n\n1. We acknowledge that the numerous modules and Figure 2 may be challenging to grasp. The training process involves all modules in Figure 2. For inference, we follow this path: speech -> gestural scores -> CSA alignments + instruction -> LLaMA -> response, removing modules like self-distillation and gestural decoders. Solid lines represent formal I/O processes in neural networks, while dashed lines denote other cases, such as loss functions. In the *general* rebuttal, we also provide additional explanations about our entire pipeline for further review. We will clarify it further in the revised version. \n\n2. Our results show a small improvement with LLaMA, which we are still investigating. One hypothesis is that LLaMA was trained at the token or word level, whereas SSDM operates at the phoneme level. A phoneme-level LLaMA might yield more significant improvements. We do have a traditional method (as reviewer mentioned, the category label output), which is exactly the H-UDM [2]. H-UDM outputs the type and time of dysfluencies without integrating a \"smoothed\" language model. Tables 1 and 2 compare H-UDM in terms of scalability and detection effectiveness.\n\n3. We compared three baselines: LTU-AS, SALMONN, and ChatGPT, which represent state-of-the-art speech understanding models. LTU-AS and SALMONN were trained via instruction tuning with both speech and **text** inputs, like our approach. So we believe it is fair comparison. For ChatGPT, we used the GPT-4o version from the iPhone app with a speech I/O interface, though it's unclear if it still uses Whisper. This comparison is fair given the extensive training data for these models. For a better fair comparison, we also fine-tuned LTU-AS-13B-FT and SALMONN-13B-FT with both speech and text inputs, but the performance boost was limited.\n\n**For questions**:\n\n1. Previous research [1] explored gesture numbers of 20, 40, 60, and 80, finding that 40 gestures yielded the lowest PER on MNGU0 dataset. While this may not apply to our larger Libri-Dys dataset, we adopted 40 gestures as default, which also aligns with the number of CMU phonemes. Using [1]'s code, we tested higher gesture numbers (100, 200, 1000) on MNGU0. Although intelligibility didn't degrade, interpretability decreased, making it difficult to associate gestures with specific articulatory movements. This interpretability is crucial for providing articulatory-based pronunciation feedback (Sec. 6.6). Thus, we used 40 gestures in our formal experiments. We will explore numbers below 100 in a revision.\n\n2. There are two types of articulatory representations: a. Articulatory kinematics data X from UAAI; b.Gestural scores H derived from X. Both serve as \"representations\" of speech and are considered \"articulatory representations.\"\n\n  * Articulatory data X, sparsely sampled from six human articulators, primarily conveys speech content but may miss acoustic details like speaker pitch. Research [4] shows that articulatory kinematics can serve as a universal speech codec. When combined with speaker embedding and signals like pitch, it can fully recover intelligible speech, achieving decent WER, CER, and MOS scores for resynthesized speech (LibriSpeech, VCTK, and AISHELL) as shown in Tables 2 and 3 of [4].  Note that [4] also primarily considers articulatory data as speech content, playing the same role that HuBERT plays in [5] or wav2vec2 plays in [6].\n\n  * Gestural score H from articulatory data X:\nDue to slight information loss in articulatory data, there is also inevitable loss in the gestural score H, as we focus on reconstructing articulatory data X. Our goal is to leverage its scalability and interpretability. To address this, we use self-distillation, allowing H to distill more acoustic information from pretrained WavLM representations. We believe this makes our articulatory representations H nearly fully intelligible. Although we did not directly evaluate H's information retention, we used two implicit metrics:\n\n    * In [3], the original reconstruction loss when recovering X from H is about 20% with 40 gestures, corresponding to a phoneme error rate of 11.02 (7.52 on melspec). This indicates a loss of intelligibility in the gestural score. With our self-distillation method, the reconstruction loss drops to 2.08%, showing nearly full intelligibility. We plan to include more analysis in the updated paper.\n    * Table 1 reports phoneme classification accuracy (F1 score), showing that our gestural score H outperforms HuBERT representations and achieves full intelligibility from speech.\n\n3. Yes. Gestural scores H differ from traditional speech representations due to physical constraints. [1] defined speech as temporal activations of sparse gestures (articulators), making gestural scores naturally sparse. We applied a sparse constraint for physical consistency. Without the sparse constraint (mask), we still achieved good detection results. However, two points are noteworthy:\n\n  * Scalability is affected: Without the mask, we tested that scalability becomes SF1=0.14 and SF2=0.17, matching HuBERT (Table 1).\n\n  * Interpretability is lost: Without the sparse constraint, gestural scores lose interpretability, making articulatory pronunciation feedback (Sec 6.6) impossible.\n\n    In essence, while removing the mask doesn't significantly impact detection, it affects scalability and eliminates interpretability benefits.\t\n\n[1] \"Articulatory phonology: an overview\"\n  \n[2] \"Towards Hierarchical Spoken Language Dysfluency Modeling\"\n\n[3] \"Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition\"\n\n[4] \"Articulatory Encodec: Vocal Tract Kinematics as a Codec for Speech\"\n\n[5] \"Speech Resynthesis from Discrete Disentangled Self-Supervised Representations\"\n\n[6] \"Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9QgNhiPpbK", "reviewer": "Reviewer_nbR5", "summary": "This paper looks at the problem of disfluency events in speech. Their core contributions are:\n* Instead of learning dysfluencies that rely on high-compute SSL models, they characterize speech representations using a type of gestural modeling baed on kinematic/articulatory movements. They rely on a pre-trained acoustic-toarticulatory inversion (AAI) model that regresses from speech to EMA features. They align information from the gestures (using the AAI model) with this and self-distillation. \n* They describe a forced aligner (and connectionist subsequence aligner) to align dysfluencies within their sequences. \n* They introduce a synthetic dataset built on LibriTTS with almost 4k hours worth of data. \n\nThe results across tasks including disfluency detection and (dysfluent) text to speech are compelling.", "questions": "There are results in Table 3 corresponding to \"ChatGPT.\" This isn't explained anywhere in the paper. It's unclear how a text-only model like ChatGPT could be used in this setting.", "limitations": "Yes they do a reasonable job.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this is a really interesting and deep paper on a topic that doesn't get that much attention. \n* The use of acoustic-to-articulatory inversion features is interesting given how they're used in a scalable/universal manner. \n* The various connection to LLMs is relevant and interesting, even if they aren't strictly necessary \n* The dataset looks like it could be really valuable for the community\n* The diagrams do a nice job of outlining what is an otherwise complicated system\n* I thought the additional TTS experiments in the appendix were really compelling. I wonder why they weren't included in the main part of the paper.", "weaknesses": "My biggest gripe is that the overall system is very complicated, which is not a problem in itself, but it's challenging to understand how everything fits together and how why certain design decisions were made. There is SO much work in this paper, but it can be very hard to follow especially if you include the appendices. Many of the technical pieces are described in detail, but don't have proper motivation. For example, why do you need self-distillation? I have a sense of how it works but not why it's necessary. I found that there are many concepts like this.  \n\nSimilarly, some terms are not explained. For example, the metric dPER should have a description -- I see the reference but you need at least a brief explanation in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks reviewer for the thoughtful summarization and comments. \n\n* Regarding the TTS question proposed in Strengths: We summarize that we performed textual dysfluency editing and simply input that text into a pretrained StyleTTS2 model, which is one of the open-sourced state-of-the-art TTS works. Our contribution is primarily in textual engineering, and we did not tune the model itself. Therefore, we believe this part of the contribution, at least technically, might not be novel enough to be considered one of our main contributions. We did observe some unnatural simulations, such as robot-like noise in stutter simulation, which we believe requires a learning method to overcome. For example, we were considering developing a GAN objective and fine-tuning the StyleTTS2 framework to address these issues. However, we have not yet accomplished this. Once we achieve a new technical-level breakthrough in TTS, we would be happy to list that as a core contribution. Due to page limits and the existence of other technical contributions (VAE, CSA, etc.), we do not have much space to elaborate further here. At this moment, we believe the core contribution might still be the open-sourced data. We hope to develop an updated TTS framework in our future work, and any suggestions are welcome!\n\n* We acknowledge that our entire pipeline is complex. We attempted to simplify the system design initially but did not achieve the desired performance boost that we now have. As of now, all modules we proposed or utilized play an essential role and cannot be replaced. However, we are continuing our efforts to make SSDM simpler, more transferable, and more deployable. The reason we include so many details in the appendix is that we feel some essential modules, like gestures, are not well-familiarized in the speech domain. We hope that in the future, as people begin to realize the existence and importance of articulatory speech modeling, this article will be naturally simplified. \n\n* Regarding specific questions, we believe we have introduced both motivation and methods in detail. However, some of this information might be included in the appendix due to page limits. For example, regarding self-distillation, there are two motivations we wish to explore:\n\n  * Articulatory kinematics data are sparsely sampled from human articulators. This definitely results in some information loss, including intelligibility loss, pitch, or speaker identity information loss. The advantage is that it is scalable, as proven in Table 1, and interpretable for pronunciation training, as shown in section 6.6. So, how can we take advantage of these benefits while also guaranteeing intelligibility? We have \"borrowed\" more intelligibility from WavLM via self-distillation. This motivation was briefly discussed in lines 79-84 and detailed in Appendix A.3.\n\n  * It has been evidenced in [1] that gestural score H implicitly captures linguistic boundaries such as phoneme segmentation. We also drew intuition from DINO [2], which showed that self-distillation in visual representations can implicitly enable image object segmentation, a so-called \"Emerging property\". Following work in speech [3] shows that self-distillation can also implicitly enforce word segmentation and better intelligibility [4]. Based on this intuition, we use self-distillation to enforce linguistic boundaries for gestural scores and to improve the upper bound of intelligibility that WavLM already achieves. These points are discussed in detail in Appendix A.3, where we also list other papers in this area. We mentioned in the main paper on line 159 that you can refer to our Appendix. However, we still feel sorry that due to page limits, we can only list this information in the appendix.\n\n* We admit that dPER was indeed not well explained. Although we cited its reference, we should have explained it in our paper. Thank you for the suggestion!\n\n* Regarding the question, ChatGPT was initially proposed as a language model only. However, in 2023, it also included speech I/O and became available (only) in mobile apps. So we now refer to ChatGPT as a general assistant that can process multiple modalities. We used the iPhone ChatGPT app, the 4-o, and conducted the inference experiments in May this year. However, we also note that the 4-o version with speech I/O that is currently available in the app is not the true 4-o speech demo they presented recently. \n\n[1] \"Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition\"\n\n[2] \"Emerging Properties in Self-Supervised Vision Transformers\"\n\n[3] \"SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT\"\n\n[4] \"DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kR0O44VYth", "reviewer": "Reviewer_px45", "summary": "This paper proposes an approach on understanding speech with disfluency.", "questions": "No.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is sound.", "weaknesses": "This work is an application work focused on a very narrow domain in the speech area. It doesn't seem to have sufficient interest for the audience of NeurIPS. It probably fits better for another speech-focused venue such as Interspeech or ICASSP.\n\nUpdated: There is also concerns on the correctness of the evaluation during the author-reviewer discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe understand that confusion or misunderstanding may arise from the definition of this research area. We believe we can agree that in the speech research community, the majority of people are working on TTS, ASR, or spoken language models (SLMs) due to the breakthrough of generative AI and LLM techniques. Only a few people are working on some rare research areas like our domain: dysfluent speech modeling. Because of this, most of the works we have heard about recently are related to TTS, ASR, LLMs, SLMs, etc., especially for those who have just entered the speech area for research and are excited about it. We believe this is the fundamental reason you might think this is a narrow domain rather than a popular or general one. This is totally understandable and makes sense. \n\n\nHowever, we would like to explain our motivation further. The fact that not many people are working on this topic certainly indicates it is a \"narrow domain,\" at least at this moment. However, it does not mean it is not important. We also indicate in the abstract that the research of dysfluency modeling is still at its \"LeNet moment.\" So at the time when LeNet was invented, maybe other people also thought it was a narrow domain and not important. That's why Hinton's neural network was not taken seriously until many years later.\n\nBefore elaborating on our motivation for this work, we might pose a general question: Why are so many people working on popular areas like TTS or SLMs in the speech domain? We think it's fundamentally driven by the needs of the world, or more specifically, the market size. Because there is a huge need for some techniques, money will follow, and many people are needed in this area. Usually, people working together will create much more impactful work, such as those highly influential speech works we cited in the related work section.\n\nIf this is true, we have also done early market investigation showing that the market size for dysfluent speech (speech therapy and spoken language learning) is also substantial. We already reported the numbers in the introduction. The exact figure for the speech therapy market value (6.93B) is even larger than the TTS market size (3.45B) and is comparable to ASR. This means that, in terms of needs and market size, dysfluent speech modeling is as important as other areas such as LLMs, TTS, SLMs, ASR, etc. Also, we reported at the beginning of the introduction that 1 in 10 kids globally suffer from dyslexia. What does this mean? It means if I have a child in the future, there's a 1/10 probability that my child will also suffer from dyslexia. Since there is not an effective AI framework, I would absolutely encourage my child to try this SSDM to get a diagnosis. This 1/10 probability applies to all people globally. In this case, maybe this is even a more important area than popular areas such as TTS or ASR. We're not trying to compare which area is more important, which would be meaningless. We just mention that this is not a narrow domain and instead will become an increasingly important one.\n\nBut you might further ask, since it's such an important area, why aren't more people working on it? We mentioned this in the introduction and related work sections, as well as in the demo on the first page. The current large-scale SLMs such as LTU-AS, SALMONN, ChatGPT have limited ability to tackle this problem. Although we mentioned that, technically, dysfluent modeling is a spoken language understanding problem that could be classified under either ASR or SLMs, the work including [1,2] and our method indicate that the efforts needed to bridge the gap between ASR/SLMs and dysfluent modeling are much greater. It is a super difficult problem. Because of this difficulty, we personally think dysfluent speech modeling should be separated as an independent research area. Even in some speech conferences such as Interspeech/ICASSP/SLT/ASRU, this area has not been defined. So when we were suggested to submit this work to Interspeech/ICASSP, there might still not be a suitable venue for us.\nRegarding the specific point that the reviewer has raised, we checked the official NeurIPS requirements again and found:\n\n\"The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024) is an interdisciplinary conference that brings together researchers in machine learning, neuroscience, statistics, optimization, computer vision, natural language processing, life sciences, natural sciences, social sciences, and other adjacent fields. We invite submissions presenting new and original research on topics including but not limited to the following: Applications (e.g., vision, language, speech and audio, Creative AI).\"\n\nSo we would think the NeurIPs still encourages the speech application work submission. But if this is the core concern and still not clear, we would also love to invite the program chairs or area chairs to look through our work again and to analyze our qualifications. \n\nWe have clarified our points regarding the reviewer's primary concerns. We understand that the confusion happens because there are indeed not too many people working in this area. However, we are making efforts to make this area more well-known in the future and to attract a wider audience. So feel free to discuss this further. If possible, we would also love to invite you to help announce this paper to a broader audience in the future to benefit more patients with speech disorders.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nmb7qErdSL", "reviewer": "Reviewer_oU79", "summary": "This is an extraordinarily well-written paper that addresses speech disfluency modeling. Given a recording of a person’s speech and the reference text transcription, the proposed model named SSDM outputs a natural text explanation of the pronunciation problems in specific words and sounds in the recording. This is an extremely challenging task in the speech domain given the sparsity of annotated data and the high cost of producing accurate annotations. The paper as a whole is very novel. I will highlight the main novelties below. \n\nThe authors propose a cascade of modeling approaches, drawing upon the learnings from representation learning (VAE, self-distillation), TTS (duration prediction, glow), ASR (CTC, alignment), and self-supervised modeling (acoustic encoders, pre-trained LLMs). It is clear that the authors have put significant effort into developing an interpretable method, which is commendable. They represent the speech signal as a sequence of “gestures” that correspond to articulatory movements, for which pre-trained encoders, VAE, and self-distillation are critical to learn. In order to align the acoustic and text representations well, they find it is necessary to leverage local sequence alignment approaches and they extend the popular CTC objective to perform approximate longest common subsequence alignment in the embedding space. Then, the aligned acoustic-text sequences are provided as the input to LLaMa that is fine-tuned with LoRA for the task of disfluency description. The LLM part of the paper actually acts as the icing on the cake; the main contributions are found in the representation learning across audio, text, and gesture domains, and the alignment method. \n\nIn order to train the model, the authors propose a synthetic data creation approach to generate accurately-labeled disfluent speech with TTS. The resulting SSDM model displays state-of-the-art performance in VCTK++ (accented English), Libri-Dys (the synthetic dataset), and nfvPPA dataset which contains real disordered speech, outperforming multiple baselines (LTU-AS, SALMONN, ChatGPT).", "questions": "* Following up on the issue of using 100% synthetic data, would it be possible to leverage SSDM for weak labeling of data in the wild to develop a non-synthetic dataset? How reliable would SSDM be for this task in author's opinion? I'm asking only for an assessment and not for an experiment, as I realize that's easily a separate follow-up research material.\n* A loose thought, but perhaps the system developed by the authors may be used to evaluate TTS outputs, given that some popular classes of TTS models (e.g. decoder-only approaches) suffer from non-monotonic alignment issues. Do the authors anticipate any risk related to overfitting to TTS used for generating Libri-Dys?", "limitations": "The authors present a fair assessment of the model's limitations.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Important problem with a big potential for impact that is notoriously difficult to tackle with existing approaches. The authors achieved strong results on the presented benchmarks, which are suitable for evaluating their model.\n* Clever application of synthetic data to a task which is known to suffer from a lack of data availability. This choice is very well motivated by the authors who point out that largely dominant alignment-based method suffer from the assumptions about monotonicity and reference text correctness. It is evident the synthetic data creation benefitted heavily from expert knowledge of phonetics and speech production. Bonus points for releasing the dataset as open-source.\n* Very well motivated application of monotonic aligners used to identify subsequences instead of individual symbols, and the choice of LCS, further developed by the authors into CCS.\n* The appendices contain plentiful information about the details of their approach and experimental setup, but the main body of the paper is written such that it is not necessary to jump into any of the appendices to understand it.", "weaknesses": "I couldn’t find any major weakness in the paper, but it could benefit from some revision (see detailed comments).\n\nAs for minor weaknesses: \n* The model seems to be trained on 100% of synthetic data. It is unclear to what extend it overfits to the specific TTS model instance used to curate Libri-Dys. This question is partially answered by the evaluation on nfvPPA dataset where some degradation of performance is observed compared to Libri-Dys test data.\n* As the authors note, the LLM component seems under-utilized. It is unclear whether the LLM overfits to the answer patterns in Libri-Dys dataset. It would be interesting to check to what extent the LLM was able to preserve its original in-context learning and reasoning capabilities to enable the user to have an in-depth discussion about their disfluency patterns.\n\n* p.5, l.163, typo in “variationa”\n* p.5, l. 172, was “i” supposed be present in the formula after “we seek an aligner: …”?\n* p.5, incosistent naming L_dist vs L_dis\n* eq. 10 the expectation over C and tau is missing a comma after the approximation is introduced\n* Figure 4 should have at least a rudimentary description of each of the three presented blocks.\n* p.7 l.228, should be Eq. 13 not Eq. 2.1\n* sec 3.3, I got a bit lost in the explanation of how LCS can be implicitly added to CTC. I think it would be easier to grasp for the reader if the authors explicitly marked which terms in the equations are added/modified vs vanilla CTC objective. I also suggest to explicitly discuss what is the LCS constraint (p.7, l. 217).\n* p.9, l.322-323, elaborate what do you mean by “We suspect this is due to the granularity of tokens“\n* The references [4-16] have placeholders instead of proper bibliographic information\n* The last appendix: “SALMOON” should be “SALMONN”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the time and effort put into giving such in-depth comments.\n\n* Yes. Our model was trained entirely on synthetic data. We chose high-quality open-source multi-speaker TTS models like VITS and StyleTTS2 to mitigate potential overfitting, though we lack concrete evidence of overfitting to specific TTS models. We did tests using VITS on VCTK data and they showed similar results, with slightly lower detection accuracy but still a significant gap between simulated and nfvPPA data. We may have overfitted to fluent TTS models, as no large-scale dysfluent TTS model currently exists.\n\n  * Regarding the questions, we think those are great approaches to try. We all know that current multi-speaker TTS models are trained within a limited set of speakers. We do see efforts in scaling TTS, such as Voicebox, SeedTTS, etc., that were trained on a wider range of speakers. However, the scaling effort is still limited, and those pretrained models are not planned to be open-sourced at this moment. So for a stronger and broader scaling plan, using SSDM on speech in the wild to generate pseudo labels should be a powerful and effective way for further scaling planning. Thank you so much for this insightful suggestion.\n\n  * We strongly agree that SSDM can be used for TTS output evaluation. We can think of two ways to evaluate it: First, for the normal fluent TTS task, we would expect fewer dysfluencies to be detected given ground truth text and speech as input, with our \"pronunciation prompt\" as additional input. Second, we can also check the CSA alignments output, and we would expect the CSA alignments to be as monotonic as possible.\n\n  * We believe that the evaluation results also depend on the TTS method used for simulation. Popular open-source TTS methods such as StyleTTS2 or VITS are non-AR based methods, which might implicitly inject more monotonicity for speech-text alignments. This might have double-sided effects: (1) The simulated Libri-Dys might not be that natural because of the forced monotonic effect that was naturally encoded. (2) There might also be bias in that our SSDM will be naturally monotonically affected, so some non-monotonic outputs from AR methods might not be affected. These are our assumptions. We should also try AR-based methods to generate simulation data, which should train our SSDM to then evaluate TTS outputs from both AR and non-AR based methods. We think this is a pretty interesting research direction.\n\n* Regarding our CSA alignments: The intuition is that vanilla CTC treats each speech frame - text pair $y^{i,j}$ equally $\\forall i,j$, or CTC treats each alignment equally. However, LCS only cares about the \"first element\" for each sub-sequence and will not take the remaining elements into the loss objective, as shown in Fig. 3. So our purpose is to create some inductive bias to make $y^{i,j}$ not equally considered for different $i,j$.\n\n  * The leftmost sub-figure in Figure 4 gives an algorithm example. Assume we have a true alignment where text $C_2$ is aligned to $y^{2,2}, y^{3,2}, y^{4,2}$. In this example, the emission probability for $y^{3,2}$ and $y^{4,2}$ is actually not important since they are not considered in the loss objective in the off-line LCS algorithm. So there will be two changes in comparison to vanilla CTC: The transition probability should ideally happen between $y^{5,3}$ and $y^{2,2}$ instead of $y^{4,2}$, so $y^{3,2}, y^{4,2}$ are skipped and are called transition skip. However, it is hard for us to really do such a skip operation in the coding. We performed another slack, where we can just make $y^{4,2} = y^{3,2} = y^{2,2}$ so that we still model transition probability between $y^{5,3}$ and $y^{4,2}$. This is what emission copy means. \n\n  * Then let's look at the forward algorithm in the middle part of Figure 4. We have $\\alpha^{i,j}$ primarily determined by $\\alpha^{i-1,j}$ and $\\alpha^{i-1,j-1}$. $\\alpha^{i-1,j} -> \\alpha^{i,j}$ denotes emission copy. Here we would assume $\\alpha^{i-1,j-1}$ has already copied $\\alpha^{i-2,j-1}, \\alpha^{i-3,j-1},...$, and the transition between $\\alpha^{i-1,j-1}$ and $\\alpha^{i,j}$ will be able to represent any transition between $\\alpha^{i-k,j-1}$ and $\\alpha^{i,j}$. So in this method, LCS is recursively encoded. Then we still take into consideration the other text $\\alpha^{i-1,j-k}$ with a shrink factor, as this implicitly captures some dysfluencies such as missing. The approximation algorithm is developed primarily based on heuristic analysis, and there is no strict proof of why this approximation is truly LCS-aware. However, our experiments may have already proven this assumption. We are still working on the theoretical aspects to make it more solid. We will make this clearer and add more captions for Figure 4 in the updated work.\n\n* We are still investigating the LLM part. We suspect that LLaMA might be treating this task as a \"phoneme transcription\" task instead of understanding, since the alignments from CSA already capture highly dysfluency-aware information. LLaMA might just be performing transcription on that. However, we are conducting experiments to do a more in-depth check as you suggested, focusing on in-context learning and reasoning abilities. We also suspect that this may be \"due to the granularity of tokens\" since LLMs were trained at word or token level (usually corresponding to many frames for speech), while phonemes are at a finer-grained level. So perhaps SSDM would benefit more from a phoneme-level or other hierarchical LLaMA.\n\n* In p.5, l.172, we double-checked and believe there are some symbol issues. We believe there is no \"i\" since \"i\" is defined in Eq. 9. The alignment function should be $\\gamma: {C_1,C_2,...C_L} \\rightarrow \\mathcal{P}(\\tau_1,...\\tau_t')$, which means we map $C_i$ to a subset of the sequence $\\tau_1,...\\tau_t'$, where $i=1,2..,L$.\n\n* All other typos you proposed are valid, and we will correct them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers:

Given the complexity of our system design, we will elaborate more on our pipeline in this general rebuttal. 

While we believe the paper's writing is generally clear, we recognize there is room for improvement. Our core method involves generating gestural scores from speech, aligning these scores with text supervision, and then feeding this alignment into LLaMA for training and inference. 

We use gestural scores rather than traditional speech features like SSL because they are more scalable for this task. Alignment is necessary because dysfluencies are naturally encoded in speech-text alignments. We incorporate LLaMA to make the entire system end-to-end and to explore whether external LLMs can enhance dysfluency detection. To develop gestural scores, we employed a VAE for expressive posterior modeling and a duration model. We use sparse sampling (mask strategy) to make gestural scores consistent with theoretical support [1]. The reasons for using self-distillation are twofold: (1) articulatory information from AAI suffers from information loss; and (2) self-distillation implicitly derives emergent abilities such as image segmentation and word segmentation, which can improve the semantics of the gestural scores. We believe the overall logic is well-explained in the paper. 

However, we admit there are aspects we must improve to enhance clarity, as you suggested. Firstly, Figure 2 contains many modules, and we should clearly differentiate between training and inference steps. Due to page limitations, we did not initially separate these, which unfortunately caused confusion. We would like to clarify further: The training process involves all modules in Figure 2. We have two training paradigms, as mentioned in Sec. 6.2. The first method trains VAE, CSA, and LLaMA separately. VAE training includes gestural posterior modeling, self-distillation, and dysfluency alignment classification, up to dysfluency alignment. The dysfluency alignment uses pre-simulated forced alignment as a target (dashed line). The loss objective is defined in equation 8. We then fix the VAE and use dysfluency alignment (green blocks) and reference text embedding as input to train CSA with the objective defined in equation 14. Next, we fix VAE + CSA and use the output alignment embedding from CSA, along with language instructions and targets (shown in response), to fine-tune LLaMA with LoRA. The second training method involves training the entire system (VAE+CSA+LLaMA) end-to-end, using the separately trained models as initialization. For inference, we follow the path: speech -> gestural scores -> CSA alignments + instruction -> LLaMA -> response. Modules like self-distillation (acoustic adaptor parts), and gestural decoders are no longer needed during inference. 

We apologize for any confusion this may have caused. Although these steps are detailed in the paper, we should provide better visualizations for training and inference separately. In our updated paper, we will either simplify Figure 2 and clearly label training and inference steps, or provide more detailed instructions in the appendix. We welcome any advice on this matter. We will provide a detailed caption to explain this or simplify Figure 2 in the updated work.

Meta-review: 1 strong accept, 2 acccepts and 1 reject
the reject reviewer wrote very little validation about why rejecting, other than it is better suited for ICASSP or Interspeech. This is not a reason to reject a paper.
The accepts all point to interesting idea and results, as well as a paper being well written.
Decision: Accept (poster)