SSDM: Scalable Speech Dysfluency Modeling

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:31:53

Abstract: Speech dysfluency modeling is the core module for spoken language learning, and speech therapy. However, there are three challenges. First, current state-of-the-art solutions~~\cite{lian2023unconstrained-udm, lian-anumanchipalli-2024-towards-hudm} suffer from poor scalability. Second, there is a lack of a large-scale dysfluency corpus. Third, there is not an effective learning framework. In this paper, we propose \textit{SSDM: Scalable Speech Dysfluency Modeling}, which (1) adopts articulatory gestures as scalable forced alignment; (2) introduces connectionist subsequence aligner (CSA) to achieve dysfluency alignment; (3) introduces a large-scale simulated dysfluency corpus called Libri-Dys; and (4) develops an end-to-end system by leveraging the power of large language models (LLMs). We expect SSDM to serve as a standard in the area of dysfluency modeling. Demo is available at \url{https://berkeley-speech-group.github.io/SSDM/}.

Main paper: 
 introduction: 
 Speech dysfluency modeling is key for diagnosing speech disorders, supporting language learning, and enhancing therapy [1] . In the U.S., over 2 million individuals live with aphasia [3], while globally, dyslexia affects approximately one in ten people [4] . The U.S. speech therapy market is projected to reach USD 6.93 billion by 2030 [5] . This growth parallels developments in Automatic Speech Recognition (ASR), valued at USD 12.62 billion in 2023 [6] , and Text-to-Speech (TTS), valued at USD 3.45 billion [7] . Moreover, the global language learning market is anticipated to be USD 337.2 billion by 2032 [8] . Conversely, substantial investments have been made in training speech-language pathologists (SLPs) [9, 10] , and the high cost of treatment often remains out of 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Response: For the word "you," there is a stutter of "y" at 0.60 seconds.For the word "all," there is a block at 2.92 seconds. For the word "grandfather," there is a stutter at 5.60 seconds, and there is a phonetic error for "d" at 5.90 seconds. [11] [12] [13] [14] [15] . Therefore, there is a crucial need for an AI solution that makes advanced speech therapy and language learning available and affordable for everyone.Speech dysfluency modeling detects various dysfluencies (stuttering, replacements, insertions, deletions, etc) at both word and phoneme levels, with accurate timing and typically using a reference text [1] . (see Figs. 1 for examples). Fundamentally, it is a spoken language understanding problem. Recent advancements have been driven by large-scale developments [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] . However, these efforts often focus on scaling coarse-grained performance metrics rather than deeply listening to and understanding the nuances of human speech.Traditional approaches to dysfluency modeling have relied on hand-crafted features [32] [33] [34] [35] [36] . Recent advancements have introduced end-to-end classification tasks at both utterance [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] and frame levels [49, 50] . However, these methods often overlook internal dysfluency features like alignment [1] and struggle to detect and localize multiple dysfluencies within a single utterance. [1, 2] propose 2D-Alignment, a non-monotonic approach that effectively encodes dysfluency type and timing. Nonetheless, initial experiments show that this method struggles with scalability, limiting its further development. To address these concerns, we revisit this problem and summarize our contributions as follows:• We revisit speech representation learning from a physical perspective and propose neural articulatory gestural scores, discovered to be scalable representations for dysfluency modeling. • We introduce the Connectionist Subsequence Aligner (CSA), a differentiable and stochastic forced aligner that links acoustic representations and text with dysfluency-aware alignment. • We enable end-to-end learning by leveraging the power of large language models. • We open-source the large-scale simulated dataset Libri-Dys to facilitate further research.2 Articulatory Gesture is Scalable Forced Aligner 
background: 
 Revisit Speech Representation Learning Self-supervised speech representations [51] , large-scale ASR [16] [17] [18] 20] , codec models [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] , and speech language models (SLMs) [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] have emerged as universal paradigms across tasks and languages. However, high computing costs of scaling efforts is not affordable for academia researchers. In this work, we propose learning speech representations grounded in fundamental physical laws [68, 69] . This approach characterizes speech representations by the kinematic patterns of articulatory movements, a method we refer to as gestural modeling.VAE Objective After variational inference, our decoder p θ (X|H, G) = P θ (X|D, I, G) reconstructs X using duration D, intensity I, and gesture G. The evidence lower bound (ELBO) and its derivation are provided in Eq. 1 and Appendix A.4, respectively. The posterior q ϕ (Z k,i |X), modeled via vanilla variational inference [90] , assumes standard normal priors for p(Z k,i ). The mechanisms of the duration and intensity encoders, q ϕ (D k,i |Z k,i , X k,i ) and q ϕ (I k,i |Z k,i , X k,i ), are detailed in Sec. 2.2.2 and Sec. 2.2.2. Details on the decoder P θ (X|D, I, G) are discussed in Sec. 2.2.2.L ELBO = E q ϕ (Z,D,I|X) [log p θ (X|D, I, G)]EQUATIONDuration Posterior q ϕ (D k,i |Z k,i , X k,i ) We employ the Gumbel softmax [91] to reformulate the duration posterior q ϕ (D k,i |Z k,i , X). Let π k,i ∈ R C denote the logits across all C discrete duration classes (values) for patch (k, i). For each class j, we obtain Gumbel noise ϵ k,i j = -log(-log(U j )), where U j ∼ Uniform(0, 1). We then define πk,i j = (log(π k,i j ) + ϵ k,i j )/τ , where τ is temperature parameter. Finally, we obtain the Gumbel softmax transformation as an approximation of the duration posterior in Eq.2. We set p(D k,i ) = 1/C, where C is the number of discrete duration classes. Background and detailed methodology can be viewed in Appendix. A.2.2.q ϕ (D k,i =j|Z k,i ,X)≈ exp ( πk,i j ) C l=1exp ( πk,i l )(2)Intensity Posterior q ϕ (I k,i |Z k,i , X k,i ) After sampling I k,i ∼ q ϕ (I k,i |Z k,i , X k,i), the model applies a per-gesture, region-wise impact. This can be formulated in Eq. 3. where H i-D k,i /2:i+D k,i /2,k represents the local window of impact, I k,i is the sampled impact value, and D k,i is the duration of the gesture. We actually applied Sigmoid function to deliver positive intensity values. The Hann function is used to apply the impact smoothly within the local window. The motivation behind this formulation is that most patches (k, i) are not activated, reflecting the sparse nature of human speech production and co-articulation [70, 77] . Visualizations can be checked in Appendix.A.2.2.H i-D k,i 2 :i+ D k,i 2 ,k = Hann Sigmoid(I k,i ∼ q ϕ (I k,i |Z k,i , X k,i )), D k,i ∼ q ϕ (D k,i |Z k,i , X) (3)Online Sparse Sampling Given the limited number of patches contributing to gestural scores [71] , we localize the impact within a specific window. We define a Combined Score S k,i = aI k,i + bD k,i , where I k,i and D k,i represent impact and duration, respectively, and a and b are hyperparameters. This score ranks the importance of each patch, with indices for each gesture computed as r row (k, i) = rank(-S k,i within row k). Setting m row as the number of patches selected, we apply a sparse mask M row (Eq. 4) to derive the final sparse gestural scores, detailed in Eq. 4. This entire online sparse sampling process is differentiable. The parameters a, b, and m row are elaborated in the Appendix.For simplicity, we denote this process as (i, k) ∼ S, with visualizations in Appendix A.2.3.EQUATIONMulti-scale Gestural Decoder The decoder reconstructs X = [ X1 , X2 , ..., Xt ] ∈ R d×t from gestures G ∈ R T ×d×K and gestural scores H ∈ R K×t . In this work, we retain the CMF operation [77] and extend it to multiple deep layers. We also introduce multi-scale mechanism, which has proven to be a robust tokenizer for various speech tasks [92, 93, 62, 94] . Denote:f 1/2 down,θ , f 1/4 down,θ , f 2 up,θ , f 4 up,θas downsample/upsample modules with scales of 1/2 or 1/4. The convolutive matrix factorization operator A * B meansT -1 i=0 A(i) • - → B iwhere A ∈ R T ×d×K and activation function B ∈ R K×t . Then our multi-scale decoder is defined in Eq. 5, where r = 1 means no resolution change, and f trans represents any neural network, details of which can be found in the Appendix. Up to this point, p θ (X|D, I, G) (Eq. 1) is defined. We provide more details in Appendix A.2.4.EQUATIONAfter obtaining gestural scores, we predict phoneme alignment for dysfluency modeling. For clean speech, alignment is acquired using the Montreal Forced Aligner (MFA) [80] , while for dysfluent speech, it is simulated (see Section 5) . The direct prediction of phoneme alignment from handcrafted features or self-supervised learning (SSL) units [51] is limited due to scalability issues with dysfluent speech, discussed further in Sec. 6. We utilize 4X downsampled gestural scores (from decoding), denoted as Ĥ, matching the resolution of acoustic features [95] . Let τ = [τ 1 , τ 2 , . . . , τ t ′ ] represent the phoneme alignment, where t ′ = t/4. Employing the Glow algorithm [96] , we transform Ĥ into τ , expressed as τ = f G θ ( Ĥ), optimized via a softmax crossentropy objective L phn .Self-Distillation We distill gestural scores from pretrained acoustic features [95] , which are then adapted to match gestural scores' dimensions. Instead of directly measuring the distance between acoustic embeddings and gestural scores, we use the alignment-conditioned gestural prior as an acoustic-conditioned gestural posterior. The reference text C = [C 1 , C 2 , . . . , C L ] is processed by a text encoder to yield the latent Gaussian posterior(µ C1 θ , σ C1 θ ), (µ C2 θ , σ C2 θ ), . . . , (µ C L θ , σ C L θ ), with the gestural posterior modeled via the change of variable property f G θ as described in Eq. 6. Intuition, detailed methodology and visualization can be view in Appendix. A.3.EQUATIONConversely, given the acoustic embedding A = [A 1 , A 2 , . . . , A L ], a text encoder is employed to output the latent Gaussian posterior (µ A1 θ , σ A1 θ ), (µ A2 θ , σ A2 θ ), . . . , (µA t ′ θ , σ A t ′ θ ).The posterior q θ ( Ĥ|A) can be derived in a similar manner. The overall distillation loss is then presented in Eq. 7.L dist = KL q θ ( Ĥ|A)∥p θ ( Ĥ|C) , where q θ ( Ĥ|A) = 1 K2 t ′ i=1 t ′ j=1 N Ĥi ; µ Aj θ , (σ Aj θ ) 2(7) Both K 1 and K 2 are normalization terms. The overall loss objective for neural variational gestural modeling is shown in Eq. 8, where λ 1 , λ 2 , λ 3 are balancing factors.L VAE = -λ 1 L ELBO + λ 2 L phn + λ 3 L dist (8)3 Connectionist Subsequence Aligner (CSA) for Dysfluency Modeling the alignment between C and τ is typically non-monotonic. For example, when people say "plplease," it is non-monotonically aligned with "p-l-e-a-s-e." Prior work [1, 82] on non-monotonic dysfluent modeling has its limitations, as discussed in Sec. 2.1. In this work, we focus on monotonic alignment and argue that it is effective dysfluency aligner. The intuition is straightforward: we seek an aligner γ : {1, 2, . . . , L} → P({1, 2, . . . , t ′ }) such that for each i ∈ {1, 2, . . . , L}, Eq. 9 holds. The aligner γ maps elements in C to consecutive subsequences in τ without overlap. This property is beneficial for dysfluency detection, as for each element in C, we can determine the presence of dysfluencies such as insertion, deletion, repetition, block, replacement, etc., based on γ(C i ).γ(C i ) = [τ si , τ si+1 , . . . , τ ei ] where    1 ≤ s i ≤ e i ≤ t ′ e i < s i+1 ∀i ∈ {1, 2, . . . , L -1} s i < s i+1 , e i < e i+1 ∀i ∈ {1, 2, . . . , L -1} (9) 
local subsequence alignment (lsa) achieves semantic dysfluency alignment: 
 All monotonic aligners satisfy Eq.9, which serves as a necessary condition. However, we also desire γ(C i ) to be semantically aligned with C i . Consider the aforementioned example: one preferred alignment is γ(p)=[p,l,p], indicating the presence of a stutter. In contrast, if γ(p)=[p,l,p,l,e,a,s], it becomes challenging to identify any reasonable dysfluency, despite still satisfying Eq.9. In this work, we propose that Local Subsequence Alignment (LSA) is an effective approach for achieving semantically aligned γ. Before delving into the main topic, we propose and introduce two terms: (i) Global Sequence Aligner (GSA), where the cost function involves the alignment of all elements in the sequence; this includes most sequence aligners such as DTW [97] [98] [99] , CTC [79] , and MFA [80] ; and (ii) Local Sequence Aligner (LSA), where the cost function involves only a subset of elements. One representative is longest common subsequence (LCS) alignment [100, 101] . Intuition Fig. 3 (left) illustrates the effectiveness of LSA as a dysfluency aligner. The reference text C, a stress-free phoneme transcription [102] of word "references", contrasts with the dysfluent phonetic alignment τ , which includes impairments like insertions of fillers and repetitions. LCS (LSA, [100] ) and DTW (GSA, [97] ) results are depicted in red and blue, respectively. LSA alignment γ LSA (C i ) shows higher semantic alignment with C i compared to DTW's γ GSA (C i ), which includes misaligned elements like an unwarranted alignment of "F". LSA's superiority stems from its cost function, which updates only for matching dysfluency-aware boundaries, while DTW updates for all pairs, often unrelated to dysfluency boundaries. Detailed analysis are available in Appendix A.7.Problem Statement Taking LCS into our framework presents three challenges: First, the high dimensionality of C and τ requires suitable emission and transition probability models. Second, LCS cost function is non-differentiable. Third, multiple LCS alignments necessitate effective modeling of joint distribution. To address these, we introduce Connectionist Subsequence Aligner (CSA).Objective From gestural score Ĥ, we obtain phonetic alignment τ = f G θ ( Ĥ) = [τ 1 , τ 2 , . . . , τ t ′′ ]. In practice, both τ and C are embeddings instead of explicit labels, where C = [C 1 , ..., C L ] are sampled from the text encoder N (µ Ci θ , (σ Ci θ ) 2 ), i = 1, ..., L, as proposed in Sec.2.2.3. Let t ′′ denote the sequence length after removing duration from the original length t ′ . Duration will be reincorporated post-alignment. The alignment between C and τ is already defined in Eq.9. We introduce another notation Γ, where Γ(τ i ) is the aligned token in C. Γ(τ ) = [Γ(τ 1 ), ..., Γ(τ t ′′ )] represents the final alignment with respect to C, in comparison to alignment γ(C), which is with respect to τ . There are possibly multiple (N ) alignments γ LSA j (C), where j = 1, ..., N . Our goal is Approximate LSA Alignments Γ ′ (τ ) We define y i,j as the emission probability p(C j |τ i ), and transition probability p θ (τ j |τ i ). Let C S j denote the embedding sampled from the distribution N (µ Cj θ , (σ Cj θ ) 2 ) (Sec.2.2.3). The emission probability is given in Eq. 11. We approximate the transition probability using a separate neural network p θ (τ j |τ i ).EQUATIONIt is possible to list all LCS alignments Γ ′ j (τ ), where i = 1, ..., N , via soft alignments [99] , which are also differentiable. However, we propose that by simply introducing the LCS constraint on the vanilla CTC [79] objective, the LCS can be implicitly applied, which we call the Connectionist Subsequence Aligner (CSA). Let us consider Figure 4 (left) for intuition. For a single alignment Γ ′ j (τ ), the emission probability and transition probability will only be applied if C i is already aligned (C 1 in the figure). We refer to these as Transition Skip and Emission Copy. Now, let us move to the LCS-constrained forward-backward algorithm [79] . Taking the forward algorithm (Figure 4 (mid)) for illustration, Emission Copy is reflected in α i,j via an identity multiplier on α i-1,j . Transition Skip is reflected on both α i-1,j and α i-1,j-1 , where we apply a transition on α i-1,j-1 . This also implicitly leverages language modeling. We also consider all previous tokens C j-2 , ..., C j-k , ..., C 0 ; however, no transition is applied, but a discounted factor δ k is utilized instead. This indicates a significant jump (deletion), which we denote as a dysfluency module, although all other modules model dysfluencies equally. Forward and backward algorithms are displayed in Eq. 12 and Eq. 13.EQUATIONEQUATIONWe initialize α 1,1 = β t ′′ ,L = 1, α(i, 1) = 0 ∀i > 0, β(i, 1) = 0 ∀i < t ′′ , and β(1, j) = 0 ∀j < L. Our CSA objective is displayed in Eq. 14, where we take the summation over all reference tokens and time stamps.EQUATIONSampling As the alignment Γ ′ (τ ) is required for the next module, it is necessary to sample it during training. Traditional beam search methods are impeded by reduced inference speeds. To mitigate this, we employ the Longest Common Subsequence (LCS) algorithm offline on C e and τ e to derive the alignments. The final alignment is denoted as γ(C S i ) = [τ S si , . . . , τ S ei ], as presented in Eq. 9. This methodology yields a sequence of inputs in the form of CSA-O = [(C S 1 , γ(C S 1 )), . . . , (C S L , γ(C S L ))]. 
language models and overall training objective: 
 Following LTU [23] , we utilize speech representations (alignment)[(C S 1 , γ(C S 1 )), . . . , (C S L , γ(C S L))] (Sec. 3.3), along with word-level timestamps, reference text C, and instruction C I , as input to LLaMA-7B [103] . During the training process, we incorporate annotations that include per-word disfluency with timestamps. Our approach strictly adheres to the procedures outlined in [23] and employs Vicuna instruction tuning [104] with LoRA [105] . As this is not our core contribution, we provide details in Appendix A.8. We use the same autoregressive training objective as [23] , denoted as L LAN . The overall loss objective for SSDM is shown in Eq. 15.EQUATION)5 Libri-Dys: Open Sourced Dysfluency CorpusTraditional rule-based simulation methods [1, 37, 50] operate in acoustic space, and the generated samples are not naturalistic. We developed a new pipeline that simulates in text space. To achieve this, we first convert a sentence into an IPA phoneme sequence. Then, we develop TTS rules for phoneme editing to simulate dysfluency, providing five types of dysfluency: Repetition(phoneme & word), Missing(phoneme & word), Block, Replacement and Prolongation. These rules are applied to the entire LibriTTS dataset [106] , allowing the voice of generated speech to vary from the 2456 speakers included in the LibriTTS. The TTS-rules, entire pipeline, dataset statistics, MOS evaluation and phoneme recognition results are available in Appendix A.9. Overall Libri-Dys is 7X larger than LibriTTS, with a total size of 3983 hours. Data is opensourced at https://bit.ly/4aoLdWU.6 Experiments 
data setup: 
 For training, we use VCTK++ [1] and Libri-Dys datasets. For testing, we randomly sample 10% of the training data. Additionally, we incorporate nfvPPA [107] data from our clinical collaborations, which includes 38 participants-significantly more than the 3 speakers in prior studies [1, 2] . It is approximately 1 hour of speech. Further details are provided in Appendix A.10.1.The neural gestural VAE (Eq.8), CSA (Eq.14), and language modeling components are trained sequentially, with each stage completed before the next begins. Subsequently, we perform end-to-end learning to implement curriculum learning. Our objective is to evaluate the dysfluent intelligibility and scalability of our proposed gestural scores, as well as the dysfluency detection performance of each proposed module. We evaluate phonetic transcription and alignment using the framewise F1 Score and Duration-Aware Phoneme Error Rate (dPER). The F1 Score measures how many phonemes are correctly predicted, while dPER extends the traditional Phoneme Error Rate (PER) by assigning specific weights to different types of errors. For dysfluency evaluation, besides F1 Score, we also report the time-aware Matching Score (MS), which measures both type and temporal accuracy, with temporal matching considering the Intersection over Union (IoU) threshold of 0.5.Detailed training configurations can be found in Appendix A.12.Method Eval Data F1 (%, ↑) dPER (%, ↓) F1 (%, ↑) dPER (%, ↓) F1 (%, ↑) dPER (%, ↓) F1 (%, ↑) dPER (%, ↓) F1 (%, ↑) dPER (%, ↓) SF1 (%, ↑) SF2 (%, ↓) Training Data VCTK++ LibriTTS (100%) Libri-Dys (30%) Libri-Dys (60%) Libri-Dys (100%)HuBERT We evaluate phonetic transcription (forced alignment) performance using simulated data from VCTK++ [1] and our proposed Libri-Dys dataset. The framewise F1 score and dPER [1] are used as evaluation metrics. Five types of training data are used: VCTK++, LibriTTS (100%, [106] ), Libri-Dys (30%), Libri-Dys (60%), and Libri-Dys (100%). HuBERT [108] SSL units and H-UDM alignment (WavLM [95] ) fine-tuned with MFA [80] targets are adopted. Additionally, we examine Gestural Scores (GS We follow [2] by using F1 (type match) and MS (matching score). The matching score is defined as follows: if the IoU (Intersection over Union) between the predicted time boundary and the annotations is greater than or equal to 0.5, and the type also matches, it is considered detected. We use H-UDM [2] , the current state-of-the-art time-aware dysfluency detection model, as the baseline. Under our SSDM framework, we include several ablations: (1) We remove LLaMA and use a template matching algorithm [2] on top of CSA alignments; (2) We replace CSA with softDTW [99] ;(3) We replace gestural scores with WavLM [95] units; (4) We adopt curriculum training, first training the gestural VAE, CSA, and LLaMA separately, then training them end-to-end. For language model outputs, we set the prompt and use [109] to automatically extract both types and time information from the response.The results in Table 2 show similar trends in terms of both performance and scalability (SF1 and SF2). Notably, we observe that LLaMA modeling does not contribute significantly, while both gestural scores and CSA (especially the latter) contribute the most. t is also noted that dysfluent phonetic intelligibility, as shown in Table 1 , is highly correlated with detection performance.We select the optimal configuration and compare it with state-of-the-art speech understanding systems.For fair comparison, we fine-tune LTU-AS-13B [24] and SALMONN-13B [27] using the same instructions but with pure speech input (AST [110] for LTU-AS and Whisper [17] for SALMONN). Additionally, we attach a time embedding to model temporal aspects. Detailed information is available in Appendix A.8. We also test on real nfvPPA speech, with results presented in Table 3 . Current largescale models [24, 27, 109] show limited performance in dysfluent speech detection, as shown in Fig. 1 .The detection of nfvPPA speech remains challenging due to the significant gap between simulated and real disordered speech. See our demo at https://berkeley-speech-group.github.io/SSDM/. We attempt to visualize dysfluency in gestural space, as shown in Fig. 5 . The correct text is "please" (p l i: z), while the real dysfluent speech is (p l e z). We apply GradCAM [111] to visualize the gradient of gestural scores H, shown in the right figure. We select the specific gestural scores corresponding to the vowel 'i' (e), and then visualize the corresponding gesture. On the gestural score, the gradient is negative in the center, indicating that the tongue is attempting to move down, which is the incorrect direction for articulation. This observation is meaningful as it provides insight into the dysfluency. Our system also offers explainability and has the potential to serve as a more interactive language learning tool.Eval Data LTU-AS-13B [24] LTU-AS-13B-FT SALMONN-13B [27] SALMONN-13B-FT ChatGPT [109] SSDM SSDM w/ Curri F1(%, ↑) MS(%, ↑) F1(%, ↑) MS(%, ↑) F1(%, ↑) MS(%, ↑) F1(%, ↑) MS(%, ↑) F1(%, ↑) MS(%, ↑) F1(%, ↑) MS(%, ↑) F1(%, ↑) MS(%, ↑) VCTK++ 7. 
limitations and conclusions: 
 In this work, we proposed SSDM (Scalable Speech Dysfluency Modeling), which outperforms the current best speech understanding systems by a significant margin. However, there are still several limitations. First, we utilize LLMs, whose contribution is mariginal and whose potential has not been fully leveraged. We suspect this is due to the granularity of tokens, and we believe it would be beneficial to develop a phoneme-level language model to address this issue. Second, the current data scale is still inadequate, which is further constrained by computing resources. Third, we believe that learnable WFST [83, 82] could provide a more efficient and natural solution to this problem, yet it has not been extensively explored. Fourth, it is worthwhile to explore representations based on real-time Magnetic Resonance Imaging (rtMRI) [112] or gestural scores [78] . These approaches might enable the avoidance of the distillation process. Recent concurrent works have been focusing on region-based [113, 114] and token-based [115] approaches. It would be useful to explore the combination of these to leverage advantages on each side.A Appendix / supplemental material A.1 Gestural Modeling Visualization A.1.1 What are gestures and gestural scores?The raw articulatory data X ∈ R 12×t , where t represents time with a sampling rate of 200 Hz, includes x and y coordinates of six articulators: Upper Lip, Lower Lip, Lower Incisor, Tongue Tip, Tongue Blade, and Tongue Dorsum. Here, X is sourced from UAAI 2.2.1. As motion data, X can be decomposed into gestures G and gestural scores H. T represents a window size that is much smaller than t, set at T = 200 ms. Fig. 6 provides an example with only three gestures: G 1 corresponds to a 200 ms trajectory of upper lip movement, G 2 to the lower lip, and G 3 to the tongue dorsum. This is illustrative; in our work, we use 40 gestures, approximately the size of the CMU phoneme dictionary [102] excluding stress. The gestural score H ∈ R 3×t , where 3 corresponds to three gestures with corresponding indices. The first row represents the duration and intensity of gesture 1, the upper lip movement, and similarly for gestures 2 and 3. After decomposition, we obtain gestural scores as representations, which include duration and intensity, supporting co-articulation from different articulators with potential overlap. These scores are typically sparse and under certain conditions, they are also interpretable [77] . The next question is: where do these gestures come from? We performed k-means clustering on the original data X. Specifically, we stacked every 200 ms segment of data from X into a supervector. Then, we applied k-means clustering to all these supervectors across the entire dataset X. A simple example to understand gestures and gestural scores might be this: if a simplified dictionary includes the gestures "lower lower lip" and "raise upper lip," each with a duration of 1 second and normalized intensity, these would occur simultaneously for that duration. where * denotes the element-wise product sum. [77] proposed using a neural network to predict H and replacing the traditional CMF process with a one-layer 1-D convolutional neural network. However, strictly adhering to CMF results in a simplistic neural network architecture that limits the expressiveness of the learned gestural scores. We employ an advanced probabilistic encoder to predict H, implicitly modeling duration and intensity. Subsequently, we use a multi-scale decoder to simulate the CMF process and recover X. We will discuss the details in the following sections. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We mentioned them in the beginning of introduction.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We mentioned in Appendix. A.9.3 and Appendix. A.10.1 about dataset safeguard statement. Currently we will not open source the model until the approval comes.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We obtained data and code in a legitimate way and cited all the work involved.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Assets (code and model) are well described in our Appendix. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Since the real articultory data are typically unavailable, we employ a state-of-the-art acoustic-toarticulatory inversion (AAI) model [88] pretrained on MNGU0 [84] . The model takes 16kHz raw waveform input and predicts 50Hz EMA features. We then upsample them to 200Hz. Although the AAI model was pretrained on single speaker EMA, it is a universal template for all speakers [88] . A concurrent study [116] further demonstrates that by performing speech-to-EMA-to-speech resynthesis, the single-speaker EMA representations derived from multi-speaker speech corpora, such as LibriTTS [106] and VCTK [117] , maintain a sufficient level of intelligibility. Note that the entire system should be considered a speech-only system, as it does not require the use of any real EMA data during its operation.However, there are three major differences between gestural scores and TTS duration modeling. First, unlike TTS, which enforces a monotonic alignment where each phoneme has a single duration, gestural scores permit independent gestures with overlapping durations, capturing co-articulation complexities [71] . Second, while TTS aligns text with speech, gestural scores lack a reference target for each gesture's duration. Third, durations in gestural scores are linked with intensities. Therefore, traditional differential duration modeling methods such as regression-based approaches [118] [119] [120] [121] ,Gaussian upsampling [122] [123] [124] [125] , and variational dequantization [126] [127] [128] lead to unstable training in our setup.Here we visualize our duration and intensity prediction modules in Fig. 8 . Given the input X ∈ R 12×t , a latent encoder is utilized to derive latent representations Z ∈ R (12×P )×t . Subsequently, Z is reshaped into a three-dimensional representation Z ∈ R 12×P ×t , where P denotes the patch embedding size for each patch index (k, i), with k = 1, . . . , K representing the gesture index (in our configuration, K = 40), and i is the time index. Each patch embedding is concatenated with X[k, i], which is a scalar, to form a P + 1 embedding. This composite is then processed through the Intensity Encoder and Duration Encoder to predict their respective posteriors. For the intensity posterior, values are treated as floats. As gestural scores must always remain positive for enhanced interpretability, a Sigmoid function is applied to the sampled intensity I k,i . The duration predictor operates as a classifier, where we establish the class set as [1, 2, 3, . . . , 50], thus constituting a 50-way classification problem. Due to the non-differentiable nature of the sampling process, we employ Gumbel Softmax [91] to generate the duration posterior. Consequently, for each point (k, i) in the gestural score, we obtain a continuous positive intensity Sigmoid(I k,i ) and a discrete duration D k,i . A Hanning window is applied across the entire window:H i-D k,i 2 :i+ D k,i 2 ,k = Hann Sigmoid(I k,i ∼ q ϕ (I k,i |Z k,i , X k,i )), D k,i ∼ q ϕ (D k,i |Z k,i , X) . The Hanning window is defined as w(n) = Sigmoid(I k,i ) 1 -cos 2πn D k,i -1, making it differentiable with respect to both I k,i and D k,i . The raw gestural scores H, as shown in Fig. A.2.2, represent a dense matrix. For each position (k, i), a Hann window is applied, reflecting the inherent sparsity observed in human speech articulation [71] . To enhance this sparsity, we implement a sparse sampling method. As illustrated in Fig. 9 , we define a ranking score S k,i = aI k,i + bD k,i where the coefficients are set to a = 10 and b = 1. We then select the top m row points based on the highest ranking scores. From this, we derive a Mask matrix M row , which is applied to the original H to produce a sparse gestural score H ∈ R K×t .Traditional neural convolutional matrix factorization (CMF) [77] uses a single-layer neural network, which significantly limits the expressiveness of gestural modeling.In this new decoder, we consider two sampling factors, r = 2 and r = 4. According to Eq. 5, Fig. 10 fully visualizes the multi-scale gestural decoder architecture. Note that the final representation Ĥ is downsampled by a factor of 4 to ensure consistency with the acoustic features from WavLM [95] . Background and Intuition Electromagnetic articulography (EMA) data, sourced from either real samples [129] or UAAI [88] , are typically sparsely sampled, leading to information loss. While concurrent work [116] achieves satisfactory intelligibility mostly at the word level, our objective is to phonetic understanding. Furthermore, gestural scores precisely delineate phoneme boundaries [77] . This prompts the exploration of additional constraints to synergize these aspects. Self-distillation has been successful in computer vision [130] [131] [132] and speech [133] [134] [135] [136] [137] [138] [139] , revealing emergent properties like unsupervised image segmentation [132] and speech semantic segmentation [139] .Methods We perform self-distillation per frame, indicated by the time index i in Fig. 11 . From the acoustic adaptor, we introduce the gestural score Ĥ, which is downsampled by a factor of four, aligning it with the same resolution as speech features [95] . This allows us to derive the posterior q θ ( Ĥ[i]|A[i]). From the text encoder, which processes phonemes, we input the predicted phoneme embedding τ into the textual distribution parameters µ C θ , σ C θ , obtaining p θ (τ i |C) for each time step i.Subsequently, through a flow and change of variable, we derive the prior distribution p θ ( Ĥ|C). KL divergence is then applied between p θ ( Ĥ[i]|C) and q θ ( Ĥ[i]|A[i]) to facilitate self-distillation. = log q ϕ (Z, D, I|X) p θ (X, Z, D, I) q ϕ (Z, D, I|X) dZ dD dI= log E q ϕ (Z,D,I|X) p θ (X, Z, D, I) q ϕ D, I|X)≥ q ϕ (Z,D,I|X) log p θ (X, Z, D, I) q ϕ (Z, D, I|X) (Jensen's inequality)EQUATIONEQUATIONEQUATIONEQUATION)EQUATIONA. for each element j in seq2 do4: if seq1[i] == seq2[j] then 5: lengths[i+1][j+1] = lengths[i][j] + 1 6: else 7: lengths[i+1][j+1] = max(lengths[i+1][j], lengths[i][j+1]) 8:end if9:end for 10: end for 11: Initialize an empty list align_lcs_result to store the LCS alignment 12: Set x = len(seq1), y = len(seq2) 13: while x ̸ = 0 and y ̸ = 0 do 14: for j = 1 to m doif lengths[x][y] == lengths[x-1][y] then 15: x = x -1 16: else if lengths[x][y] == lengths[x][y-1]6: cost ← distance(seq1[i-1], seq2[j-1]) 7: dtw_matrix[i][j] ← cost + min(dtw_matrix[i-1][j], dtw_matrix[i][j-1], dtw_matrix[i-1][j-1]) 8:end for 9: end for 10: Initialize an empty list alignment to store the alignment 11: i ← n, j ← m 12: while i > 0 and j > 0 do if i == 0 and j > 0 then if i > 0 and j == 0 then Fig. 3 provides an example and illustrates the intuition behind why LSA is a better candidate for dysfluency modeling. The reference text is the phoneme transcription of the word "references". To better illustrate this, the chosen dysfluent speech is significantly impaired, containing insertion of filler sounds such as "uh", repetition of sounds like "R", "EH", "AH", "IH", and insertion of sounds such as "S" and "ER", along with deletion of sounds like "F".13: if seq1[i-1] == seq2[j-1] then 14: alignment.append((seq2[j-1], seq1[i-1])) 15: i ← i -1 16: j ← j -1 17: else 18: if dtw_matrix[i-1][j] == min(dtw_matrix[i-1][j], dtw_matrix[i][j-1], dtw_matrix[i-1][j-1]) then 19: i ← i -1 20: else if dtw_matrix[i][j-1] == min(dtw_matrix[i-1][j], dtw_matrix[i][j-1], dtw_matrix[i-1][j-1]) then 21: alignment.append((seq2[j-1], seq1[i-1]))Let us examine the results from LSA and GSR. We aim to obtain all elements (phonemes) in the dysfluent alignment τ for each phoneme in the reference text C. LSA captures most dysfluencies through such per-reference alignment. For instance, the alignment (uh, R) to R indicates insertion. Similarly, (EH, S, R, EH) aligning to EH primarily indicates repetition. Up to this point, GSA exhibits similar performance, aligning (uh, R, EH, S, R) to R, which also indicates repetition. However, a significant difference emerges thereafter. For the phoneme F in the reference, no phonemes are aligned in LSA, which is correct as it is missing. Conversely, GSA aligns (ER, AH, AH) to F, which is unreasonable. For the phoneme AH in the reference, the LSA alignment (AH, AH, ER, AH) indicates repetition, which GSA fails to capture. Similarly, the repetition of IH is accurately captured by LSA but is missing in GSA. Our main point is that, although dysfluency alignment with the reference text is non-monotonic, aligning corresponding phonemes with each phoneme in the reference monotically enables fine-grained dysfluency analysis, which is naturally captured by LSA. Note that in Fig. 3 , we use LCS [100] and DTW [97] for illustration. Response: For the word "you," there is a stutter of "y" at 0.60 seconds. For the word "all," there is a block at 2.92 seconds. For the word "grandfather," there is a stutter at 5.60 seconds, and there is a phonetic error for "d" at 5.90 seconds.Annotation 1: Word "please," stutter of "p", 0.32 seconds.Annotation 2: Word "call", prolongation "kɔ" at 0.48 seconds.Annotation N: Word "please", insertion of "ʌ", 0.19s....... We inject dysfluency in the text space following these rules:• Repetition (phoneme&word): The first phoneme or syllable of a randomly picked word was repeated 2-4 times, with pauselengths varying between 0.5 to 2.0 seconds. • Missing (phoneme&word): We simulated two phonological processes that characterize disordered speech [140] -weak syllable deletion (deletion of a random unstressed syllable based on stress markers1 ) and final consonant deletion. • Block: A duration of silence between 0.5-2.0 seconds was inserted after a randomly chosen word in between the sentence. • Replacement (phoneme): We simulated fronting, stopping, gliding, deaffrication -processes that characterize disordered speech [141] -by replacing a random phoneme with one that would mimic the phonological processes mentioned above. • Prolongation (phoneme): The duration of a randomly selected phoneme in the utterance was extended by a factor randomly chosen between 10 to 15 times its original length, as determined by the duration model.The simulation pipelines can be divided into following steps: Reference text: You wish to know all about my grandfather.IPA Sequence: juː wˈɪʃ tə nˈoʊ ˈɔːl ɐbˌaʊt maɪ ɡɹˈaendfɑːðɚ. W-rep: You [wish wish] to know all about my grandfather. W-miss: You wish [to] know about my grandfather.Figure 13 : Simulation PipelineThe specific statistics of Libri-Dys are listed in Table . 4, compared with VCTK++. Figure . 14 presents a comparison between our simulated dataset and two existing simulated dysfluency datasets: VCTK++ [1] and LibriStutter [37] . It indicates that our dataset surpasses the datasets in both hours and the types of simulated dysfluencies. Note that since we build dataset based on publicly available corpus LibriTTS [106] and styletts2 [125] , it satisfies safeguards criterion. In looking for clinical populations to test our pipeline, we decided to focus on patients with a neurodegenerative disease called nonfluent variant primary progressive aphasia (nfvPPA). This phenotype falls under the umbrella of primary progressive aphasia (PPA), which is a neurodegenerative disease characterized by initially having most prominent disturbances to speech and language functions. PPA has three distinctive variants that correspond with unique clinical characteristics and differential patterns of brain atrophy: semantic (svPPA), logopenic (lvPPA), and nonfluent (nfvPPA) [107] . Disturbances to speech fluency can occur due to multiple underlying causes subsuming different speech and language subsystems in all of these variants; however, the variant most commonly associated with dysfluent speech is nfvPPA. This phenotype is characterized by primary deficits in syntax, motor speech (i.e., in this case, apraxia of speech), or both, and it is this association with apraxia of speech that makes nfvPPA an apt clinical target for assessing automatic processing of dysfluent speech.Our collaborators regularly recruit patients with this disease as a part of an observational research study where participants undergo extensive speech and language testing with a qualified speechlanguage pathologist. This testing includes a comprehensive motor speech evaluation, which includes an oral mechanism exam, diadochokinetic rates, maximum phonation time, multisyllabic word reading, word of increasing length, passage reading, and connected speech samples. For our present purposes, we have decided to analyze the speech of participants reading aloud the Grandfather Passage, a passage often used clinically to assess motor speech due to its inclusion of nearly all phonemes of the English language. We have recordings for 10 participants with nfvPPA under IRB with consents signed for educational use. Passage recordings are conducted using high quality microphones for both in-person and remote visits. We randomly select 10 recordings and calculated the occurrences of various dysfluency types within them. The distribution is shown in Fig. 15 . Note that nfvPPA data will not be released. The EMA features are denoted as X = [X 1 , X 2 , ..., X t ], where X i ∈ R d . These features represent the positions of 6 articulators, including the upper lip, lower lip, lower incisor, tongue body, tongue tip, and tongue dorsum, with both x and y coordinates. Consequently, d is equal to 12.In Eq. 2, τ = 2.In Eq. 4, a = b = 1, m row = 3.In Eq. 6 and Eq. 7, we simply setK 1 = K 2 = 1.In Eq. 8, λ 1 = λ 2 = λ 3 = 1.In Eq. 12 and Eq. 13, δ = 0.9.We first separately train the gestural VAE (Eq.8). Subsequently, we train the CSA (Eq.14), followed by training with L LAN . Finally, we retrain the entire model in an end-to-end manner. For each step, we use the Adam optimizer and decay the learning rate from 0.001 at a rate of 0.9 every 10 steps until convergence. The training is conducted using two A6000 GPUs.For the VAE and language modeling steps, it takes 40 hours to complete the entire Libri-Dys training.CSA training only takes 5 hours to converge, where only the linear layer transition probability p θ (C i |C j ) is trainable. The same training duration applies to SALMONN [27] and LTU-AS [24] with fine-tuning. Note that we used pretrained models for SALMONN and LTU-AS.Speech dysfluency modeling is fundamentally a clinical problem and, consequently, a speakerdependent issue. While we have not conducted per-speaker analysis at this juncture, our future research will explore both speaker-dependent and speaker-independent representations [144] [145] [146] [147] [148] for clinical analysis. To address potential ethical concerns, we have implemented essential voice anonymization techniques [149] in our processing of disordered speech.As concurrent work, YOLO-Stutter [113] approaches dysfluency modeling as an object detection problem. The authors utilize a simulated corpus and output dysfluency type and timing information. Stutter-Solver [114] further extends this approach, employing a similar pipeline for crosslingual (English-Chinese) joint simulation and prediction. Notably, Stutter-Solver outperformed H-UDM [2] . Another recent publication, Time-and-Tokens [115] , treats the problem as automatic speech recognition, mapping each dysfluency to a token and achieving performance comparable to YOLO-Stutter [113] . Our model primarily emphasizes scalability and user-friendly interface design. Additionally, it establishes a foundation for future researchers to explore in-context learning capabilities. We intend to conduct comparisons with these aforementioned works in future research.Answer Justification: See A.12 for these information.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/timmahrt/pyslehttps://pypi.org/project/phonemizer/ 


Decision: 1 strong accept, 2 acccepts and 1 reject
the reject reviewer wrote very little validation about why rejecting, other than it is better suited for ICASSP or Interspeech. This is not a reason to reject a paper.
The accepts all point to interesting idea and results, as well as a paper being well written.

Reviews: review: paper_summary: paper_summary: This paper presents SSDM (Scalable Speech Dysfluency Modeling), a novel approach to modeling speech dysfluency, which is essential for applications in spoken language learning and speech therapy. The authors identify three main challenges in the current state-of-the-art: poor scalability of existing solutions, the absence of a large-scale dysfluency corpus, and the lack of an effective learning framework. To address these issues, SSDM incorporates several key innovations:

1. Articulatory Gestures as Scalable Forced Aligner: The paper proposes using neural articulating gestural scores as scalable representations for dysfluency modeling, grounded in physical laws of speech production.

2. Connectionist Subsequence Aligner (CSA): Introduced as a differentiable and stochastic forced aligner to achieve dysfluency alignment by linking acoustic representations and text with dysfluency-aware alignment.

3. Libri-Dys Dataset: The authors have developed a large-scale simulated dysfluency corpus called Libri-Dys to facilitate further research in the field.

4. End-to-End System with Large Language Models (LLMs): The system leverages the power of large language models for end-to-end learning, aiming to enhance the understanding of human speech nuances.
strength_weakness: strength_weakness: 1. The paper writing is unclear, which makes it quite hard to understand the whole works. For the Figure 2, the author should make it clear which part is used in training and which part is used in inference. Also there should be more explanations on difference of the line and dashed line. Moreover, a detailed overview of the whole system about the training and inference, respectively, is necessary for the complex system like this.

2. From the results in Table 2, it seems that LLAMA is not a necessary part for the system to be effective. I wonder if there is any traditional method on this task, i.e., the output of traditional system is the category label of error type and its duration. Comparison with such systems is also needed.

3. Since the baselines in Section 6.5 uses only speech input, it seems to be an unfair comparison, which explains the large gap between the proposed method and baselines.
questions: questions: 1. Is there any ablation study on the number of gesture number?
2. Is there are any sample or metric showing how well can we recover speech from the articulatory representations.
3. Is mask in Figure 9 necessary?
limitations: limitations: N/A
paper_summary: paper_summary: This paper looks at the problem of disfluency events in speech. Their core contributions are:
* Instead of learning dysfluencies that rely on high-compute SSL models, they characterize speech representations using a type of gestural modeling baed on kinematic/articulatory movements. They rely on a pre-trained acoustic-toarticulatory inversion (AAI) model that regresses from speech to EMA features. They align information from the gestures (using the AAI model) with this and self-distillation. 
* They describe a forced aligner (and connectionist subsequence aligner) to align dysfluencies within their sequences. 
* They introduce a synthetic dataset built on LibriTTS with almost 4k hours worth of data. 

The results across tasks including disfluency detection and (dysfluent) text to speech are compelling.
strength_weakness: strength_weakness: My biggest gripe is that the overall system is very complicated, which is not a problem in itself, but it's challenging to understand how everything fits together and how why certain design decisions were made. There is SO much work in this paper, but it can be very hard to follow especially if you include the appendices. Many of the technical pieces are described in detail, but don't have proper motivation. For example, why do you need self-distillation? I have a sense of how it works but not why it's necessary. I found that there are many concepts like this.  

Similarly, some terms are not explained. For example, the metric dPER should have a description -- I see the reference but you need at least a brief explanation in this paper.
questions: questions: There are results in Table 3 corresponding to "ChatGPT." This isn't explained anywhere in the paper. It's unclear how a text-only model like ChatGPT could be used in this setting.
limitations: limitations: Yes they do a reasonable job.
paper_summary: paper_summary: This paper proposes an approach on understanding speech with disfluency.
strength_weakness: strength_weakness: This work is an application work focused on a very narrow domain in the speech area. It doesn't seem to have sufficient interest for the audience of NeurIPS. It probably fits better for another speech-focused venue such as Interspeech or ICASSP.

Updated: There is also concerns on the correctness of the evaluation during the author-reviewer discussion.
questions: questions: No.
limitations: limitations: Yes.
paper_summary: paper_summary: This is an extraordinarily well-written paper that addresses speech disfluency modeling. Given a recording of a person’s speech and the reference text transcription, the proposed model named SSDM outputs a natural text explanation of the pronunciation problems in specific words and sounds in the recording. This is an extremely challenging task in the speech domain given the sparsity of annotated data and the high cost of producing accurate annotations. The paper as a whole is very novel. I will highlight the main novelties below. 

The authors propose a cascade of modeling approaches, drawing upon the learnings from representation learning (VAE, self-distillation), TTS (duration prediction, glow), ASR (CTC, alignment), and self-supervised modeling (acoustic encoders, pre-trained LLMs). It is clear that the authors have put significant effort into developing an interpretable method, which is commendable. They represent the speech signal as a sequence of “gestures” that correspond to articulatory movements, for which pre-trained encoders, VAE, and self-distillation are critical to learn. In order to align the acoustic and text representations well, they find it is necessary to leverage local sequence alignment approaches and they extend the popular CTC objective to perform approximate longest common subsequence alignment in the embedding space. Then, the aligned acoustic-text sequences are provided as the input to LLaMa that is fine-tuned with LoRA for the task of disfluency description. The LLM part of the paper actually acts as the icing on the cake; the main contributions are found in the representation learning across audio, text, and gesture domains, and the alignment method. 

In order to train the model, the authors propose a synthetic data creation approach to generate accurately-labeled disfluent speech with TTS. The resulting SSDM model displays state-of-the-art performance in VCTK++ (accented English), Libri-Dys (the synthetic dataset), and nfvPPA dataset which contains real disordered speech, outperforming multiple baselines (LTU-AS, SALMONN, ChatGPT).
strength_weakness: strength_weakness: I couldn’t find any major weakness in the paper, but it could benefit from some revision (see detailed comments).

As for minor weaknesses: 
* The model seems to be trained on 100% of synthetic data. It is unclear to what extend it overfits to the specific TTS model instance used to curate Libri-Dys. This question is partially answered by the evaluation on nfvPPA dataset where some degradation of performance is observed compared to Libri-Dys test data.
* As the authors note, the LLM component seems under-utilized. It is unclear whether the LLM overfits to the answer patterns in Libri-Dys dataset. It would be interesting to check to what extent the LLM was able to preserve its original in-context learning and reasoning capabilities to enable the user to have an in-depth discussion about their disfluency patterns.

* p.5, l.163, typo in “variationa”
* p.5, l. 172, was “i” supposed be present in the formula after “we seek an aligner: …”?
* p.5, incosistent naming L_dist vs L_dis
* eq. 10 the expectation over C and tau is missing a comma after the approximation is introduced
* Figure 4 should have at least a rudimentary description of each of the three presented blocks.
* p.7 l.228, should be Eq. 13 not Eq. 2.1
* sec 3.3, I got a bit lost in the explanation of how LCS can be implicitly added to CTC. I think it would be easier to grasp for the reader if the authors explicitly marked which terms in the equations are added/modified vs vanilla CTC objective. I also suggest to explicitly discuss what is the LCS constraint (p.7, l. 217).
* p.9, l.322-323, elaborate what do you mean by “We suspect this is due to the granularity of tokens“
* The references [4-16] have placeholders instead of proper bibliographic information
* The last appendix: “SALMOON” should be “SALMONN”
questions: questions: * Following up on the issue of using 100% synthetic data, would it be possible to leverage SSDM for weak labeling of data in the wild to develop a non-synthetic dataset? How reliable would SSDM be for this task in author's opinion? I'm asking only for an assessment and not for an experiment, as I realize that's easily a separate follow-up research material.
* A loose thought, but perhaps the system developed by the authors may be used to evaluate TTS outputs, given that some popular classes of TTS models (e.g. decoder-only approaches) suffer from non-monotonic alignment issues. Do the authors anticipate any risk related to overfitting to TTS used for generating Libri-Dys?
limitations: limitations: The authors present a fair assessment of the model's limitations.

Comments: title: Response to follow-up- questions
comment: Dear Reviewer,

Thank you for the questions you proposed. We acknowledge that there are some aspects we did not clearly explain in the paper. However, there are no technical errors, and we will do our best to clarify these points for you.

* In our original annotations, the time annotation for a specific type of dysfluency is a region, not a timestamp. We have open-sourced our data in our manuscript, and you can verify this manually. Note that for some dysfluencies, such as missing words, the start and end times might be the same or very close. For example, it might be 2:00s-2:00s or 2:00s-2:06s. To reduce confusion, here is an exact sample of our JSON format labeling: {"word": "stella", "start": 0.005, "end": 2.005, "phoneme": "t", "start": 1.125, "end": 1.175, "type": "replace", ...}

* Figure 12's "annotation part" might be confusing as we only intended to show readers where the dysfluency starts. Introducing additional information like duration (e.g., "2:00s-2:06s") might make it more confusing since the current system is already quite complex. You may have noticed that we have time-embedding per frame, so it is definitely region-wise modeling. The figure is an illustration of annotation, not the actual annotation. For real annotations, please refer to our open-sourced data. However, if this causes further confusion, we will update Figure 12 in our future work. We welcome any advice on improving it.

* The actual output from the model is a region, not a timestamp. We use this region to compute the MS score, which is consistent with [1,2]. Note that for some specific types of dysfluencies, such as missing words, the output is exactly a timestamp or pretty short region. In these cases, we still report exact matching during MS computation, replacing IoU. However, it is essentially still IoU. (duration-free F1)

* We have received suggestions from doctors that the starting time or end time is usually more important. Therefore, we filter out those regions and only keep the starting timestamp as the final visualization for friendly user interface. However, region-wise prediction and evaluation are essential. And if you think keeping region information during visualization stage is critical, we will definitely do that. There is no conflict between these approaches.

In conclusion, we do not have the technical errors you mentioned. The misunderstanding might stem from our lack of detail or clarity in the paper. We will definitely add these experimental details in the updated work.

We sincerely hope this clarifies your technical concerns. Feel free to discuss!
Thank authors for the additional responses. Based on these responses, I have a concern on the technical correctness of the evaluation and the clarity of the writing on them. Both Fig 1 and Sec A.11.5 show that the proposed method predicts a single timestamp for each disfluency instance, which would be insufficient for calculating an intersection-over-union ratio. The authors' first response didn't address the question; the second response stated that they outputted a time region instead of a single timestamp, which is different from the paper.

I updated my ratings based on the technical concerns.
Dear reviewer:

Thank you for your attention to our research. We greatly respect your perspective on our research domain, which we acknowledge is somewhat narrow. However, we cannot construct a meaningful rebuttal without understanding your specific concerns. We would appreciate clarification on any technical flaws you've identified, such as issues with our experimental design or methodology. We are eager to address these concerns or any confusion you might have. We also welcome any suggestions you might offer.
We are happy to provide a more detailed explanation of our methods and to re-explain our work and its importance using clear, accessible language. We will explain our research step-by-step, from background to technical details, to enhance your understanding.

* Approximately 1 in 10 people suffer from speech disorders such as aphasia or dyslexia.

* When people suffer from speech disorders, they typically seek treatment at hospitals. However, the cost is often prohibitive, usually exceeding $1,000 for a single diagnosis, which can take over 5 hours. Additionally, there is a shortage of doctors and speech-language pathologists (SLPs). Consequently, such costs are often unaffordable for low-income families. Even in countries where the government covers these fees, treatment is time-consuming, and many people lack access due to the scarcity of specialists.

* This raises the question: Can AI automate this process? AI has the potential to provide an efficient and more affordable solution.
For AI to automate speech disorder diagnosis, it needs to perform deep transcription of speech, including dysfluencies such as insertions, omissions, replacements, stutters, prolongations, and phonetic errors. Accurate timestamping is also crucial. SLPs have informed us that these are the official diagnostic procedures.

* Here's an example of what an AI model should output: Given an impaired speech sample, we expect our system to produce: "Y-You [block] wi-[stutter]-sh [missing] know all [prolongation] about my grandfather". In contrast, state-of-the-art models like Whisper, ChatGPT, or Salmonn typically produce a perfect transcription: "You wish to know all about my grandfather", which is not useful for our purposes. This demonstrates that current large-scale Automatic Speech Recognition (ASR) models show limited performance on these tasks.

* To underscore the importance of this research: Recall that a single treatment session costs about \\$1,000 and takes approximately 5 hours. Our AI solution could potentially save both patients and healthcare providers (or governments) \\$1,000 and 5 hours per session. Given that 1 in 10 people have disordered speech, the market size is substantial. To our knowledge, AI exploration in this area is limited. Therefore, our work has broad impact, including both clinical and business implications.

Now, let's discuss the technical aspects of our work. We will introduce these step-by-step, and please don't hesitate to ask for clarification.

Our pipeline has two inputs: dysfluent speech and either golden truth text or a prompt.
* We obtain speech representations from dysfluent speech.
    * 2.1. While we could use traditional speech representations such as HuBERT, we found that our gestural scores are more scalable for this task, as demonstrated in Table 1. Therefore, we employ gestural scores instead of traditional SSL units.
    * 2.2. To obtain gestural scores, we first need articulatory kinematics data. We use Acoustic to Articulatory Inversion (AAI) to derive articulatory data X from speech. We then developed a VAE framework to generate gestural scores H from X. Gestural scores are sparse and duration-aware, so we developed a duration predictor, intensity predictor, and sparse sampler. We're happy to discuss any of these steps in more detail if you have questions.

* Connectionist Subsequence Alignment (CSA): We discovered that the Longest Common Subsequence (LCS) algorithm naturally captures speech dysfluencies, as shown in Figure 3. For the ground truth phoneme "AH", it aligns to the speech "AH AH ER AH", indicating a repetition of "AH" and an insertion of "ER". Based on this discovery, we developed a stochastic and differentiable LCS, named CSA, based on the vanilla CTC algorithm to provide dysfluency-aware alignments. 

* CSA takes gestural scores H and text as input and outputs an alignment. This alignment is then input into LLaMA with pre-designed instructions to achieve end-to-end learning.

 * Due to the lack of large-scale data, we developed a large-scale, TTS-based simulation dataset called Libridys, comprising about 4,000 hours of speech. We trained our entire model on Libridys. We then performed evaluations on Libridys, VCTK++, and real disordered speech data (nfvPPA).

We look forward to your feedback and are happy to provide further explanations or clarifications as needed. Please let us what step might confuse you.
Dear Reviewer:

We again acknowledge that the writing is not as straightforward as it could be due to the complexity of the subject matter, and we are making efforts to improve its visualization (global rebuttal and future work). Apart from that, please let us know if your primary concerns/questions have been addressed. Feel free to discuss or ask any other questions you may have so that we can provide prompt feedback!
Thank you for your comments. We feel there are still some misunderstandings that we'd like to clarify and hopefully we can work together to make it clear. 

* Regarding your primary concern

While we indeed mentioned that not many people are working in this direction, it doesn't mean it's a narrow research domain. We used "narrow" in quotes. Since you also admitted "it did not mean that this domain was not important," an important domain should expect to receive enough attention at NeurIPS. We think it's normal that for some domains in NeurIPS papers, many people are working on them, while for others, fewer people are involved. So we might not evaluate it based on the number of people working in that area, but rather on how important it is, which you somewhat admitted. Our point is that as long as it's an important area, it should be eligible for NeurIPS submission. We would greatly appreciate feedback on our actual technical contributions (gesture, VAE, CSA, etc). Additionally, whether a paper fits the NeurIPS venue or not should have been reflected to area chairs before the review starts, not at this moment. So we believe you've already realized the importance of this work to this extent and you are just concerning the number of people working on this area. If you agree with this, we believe your primary concern might be addressed. Feel free to discuss this further. We suggest looking at the potential impact of this work rather than the number of people working on it.

* Regarding your secondary concerns:

We call it a "LeNet" moment because this is an important area where there's no large-scale, high-quality data and no effective learning framework. We're happy to revise this. Also, in the first paragraph of the introduction, we indeed mentioned the importance of this paper beyond the domain of speech (l.15 - l.23 for its huge clinical and business impact). Let us know your suggestions, and we're happy to revise, remove, or add more.

* Regarding minor questions:

Q1. (1) We admit that the evaluation metric isn't mentioned much in the main context, primarily due to space constraints. This is a weakness we acknowledge. We'll add it to the main context in the updated work.

Q1. (2) By human correction, we only fixed some grammatical presentation issues. For example, if the output says "3.40s," we would only keep its numerical part "3.40". So the "results" are not corrected by humans.

Q2. This is a good question. We definitely predict a boundary. However, when start and end times almost overlap, we combine them for visualization. This works especially for missing, deletion, or insertion cases. We output a region like "2.20s-2.21s". We also have region-wise targets. So we still use the same IoU metric. Sorry for the confusion; these are details that weren't included. We'll include them in the main context and make it clearer.

Please let us know whether your primary, secondary, or all concerns have been addressed. We're happy to discuss further if you have more questions.
Thank authors for the response.

Re: Relevance

The authors agreed that this paper was on a narrow research domain, while it did not mean that this domain was not important. My concern here is on the relevance of this paper to the general audience of NeurIPS. The authors' response did not address this concern.

The author claimed this paper to be a "LeNet moment". However, I honestly do not see a connection. If the authors do believe so, please add a section in the paper to explain the potential impact of this work beyond the domain of dysfluent speech modeling and beyond the domain of speech.

Re: Techincal questions

Q1: 1) This is an important part of the evaluation protocol so that it should be explained in the main context, instead of in appendix. 2) How much is the impact of the human correction? Is it even a fair evaluation, while the reported results include human correction?

Q2: This question is not address. [1] predicts time boundaries for each disfluency instance, so that they could calculate an intersection-over-union ratio against the reference time ranges. This paper predicts a single timestamp rather than a time range for each instance, then how was intersection-over-union calculated?
Yeah, it makes perfect sense to have some practical experimental attempts, which we are trying as ongoing efforts. Really appreciate the valuable suggestions!
Yes sure. Thanks for all the suggestions!
Thank you for your rebuttal. I am satisfied with most of the answers. Regarding the question about the ability to leverage SSDM for weak labeling, I was more asking whether you believe, based on your existing experiments and experience with it, whether SSDM is robust enough to actually succeed in creating a high quality weakly labeled dataset. I realize without actually trying (and possibly iterating many times) it is difficult to make a definitive statement, but any intuitions, expected outcomes, known points of failure, etc. would be interesting to learn. I believe such discussion would make for a good "future work" addition in the paper.
Thanks for the detailed response. For the final version, please include text for the last two bullets describing dPER and the ChatGPT approach as these are both missing from the text.
We have attempted to address the main concerns in rebuttal part. Thanks so much for your follow-up questions. Here are our explanations for these questions. (We attempted to submit it during the rebuttal period, but due to technical issues, we were unable to do so until now. )

1. LLMs truly give a general diagnosis or feedback. We applied pre-designed prompt engineering to obtain time and type information. This was mentioned in Appendix 11.5. However, during inference, we also perform human intervention to check the actual outputs after prompt engineering for 100% controllability.

2. This is a good question. I think the previous paper [1] provides detailed explanations for those metrics. We did not detail them again here. However, we do have a visualization at the end of our demo page (link is in the abstract). We will make this more clearly visible in either the main paper or appendix in the future. Thank you!

3. We would like to stress this question more. It is true that we have been working on inference for disordered speech with baseline work [2], which is presumably better than other SLMs such as SALMONN, etc. However, the data is from the clinical domain and is extremely sensitive. We temporarily lost access to some of that data for some reasons. Thus, we only evaluated about 2 speakers, not the full set by the time of submission. The accuracy (F1, MS) is much lower than ours. Unfortunately, we did not report the full results from [2]. After paper submission, we regained access and now have the exact numbers. For H-UDM in [2], it is 55.6% F1 score and 46.3 for MS. We will update these numbers in the revised work. Thank you so much for pointing out this question!

4. This is also a question from another reviewer, so we can use the same clarification here: ChatGPT was initially proposed as a language model only. However, in 2023, it also included speech I/O and became available (only) in mobile apps. So we now refer to ChatGPT as a general assistant that can process multiple modalities. We used the iPhone ChatGPT app, the 4-o version, and conducted the inference experiments in May this year. However, we also note that the 4-o version with speech I/O currently available in the app is not the true 4-o speech demo they presented recently.
Sorry for the brevity in the previous review.

My main concern on this submission is: it is an application work focused on a very specific task on a narrow domain, which doesn't seem to have sufficient interest for the general audience of NeurIPS.

According to this paper (Sec. 1 and Sec. 2.1), speech dysfluency modeling is a task that detects both the types and the timing of disfluency in speech recordings given the reference text. By this definition, it is a very specific and narrow domain task, with forced alignment as the core technique.

All the key related works on this task as referenced by this paper in Sec. 1 [1, 2, 32, 33, 34, 35, 36, 27, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50] and [82] in Sec 2.1, including the "current state-of-the-art solutions [1, 2]", were published in various domain specific venues, notably in ICASSP and Interspeech.

Because this work is quite task-specific, I personally don't have an expertise for a thorough assessment for its technical contribution on this task.

That said, in addition to the main concern, I do have a few technical questions:

1) The proposed method uses an LLM (Llama) as the last components, which outputs in free form text. How were the numeric-based evaluation (F1, MS, etc.) done with the free-form text response?

2) According to Figure 2, the proposed method predicts only a single timestamp as the timing for each disfluency instance. What does this timestamp represent, starting, ending, or something else? Either way, without being a time range, how does IoU (Intersection over Union)-based Matching Score calculated in the evaluations?

3) In Sec 6.5 (and Table 3), the paper claims the proposed method as "State-of-the-art Dysfluency Detection". However, it's not compared to either of the "current state-of-the-art solutions [1, 2]". Although there is a comparison to [2] in Table 2, but Table 2 only include results on simulated datasets, without using the real disfluency dataset (nfvPPA). Does such evaluation support the claim?

4) In Table 3, ChatGPT is used as a baseline. Isn't ChatGPT a text-based model? How does it work for detecting disfluency in speech?