Time-Accurate Speech Rich Transcription with Non-Fluencies

Authors: Jiachen Lian, Xuanru Zhou, Zoe Ezzes, Jet M.J. Vonk, Brittany T. Morin, David Paul Galang Baquirin, Zachary A. Miller, Maria Luisa Gorno-Tempini, Gopala Anumanchipalli
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: non-fluency, dysfluency, alignment, speech transcription, clinical

Abstract:
Speech is a hierarchical collection of text, prosody, emotions, dysfluencies, etc. Automatic transcription of speech that goes beyond text (words) is an underexplored problem.
We focus on transcribing speech along with non-fluencies (dysfluencies). The current state-of-the-art pipeline \citep{lian2024ssdmscalablespeechdysfluency} suffers from complex architecture design, training complexity, and significant shortcomings in the local sequence aligner, and it does not explore in-context learning capacity. In this work, we propose SSDM 2.0, which tackles those shortcomings via four main contributions:
(1) We propose a novel \textit{neural articulatory flow} to derive highly scalable speech representations.
(2) We developed a \textit{full-stack connectionist subsequence aligner} that captures all types of dysfluencies.
(3) We introduced a mispronunciation prompt pipeline and consistency learning module into LLM to leverage dysfluency \textit{in-context pronunciation learning} abilities.
(4) We curated Libri-Dys \citep{lian2024ssdmscalablespeechdysfluency} and open-sourced the current largest-scale co-dysfluency corpus, \textit{Libri-Co-Dys}, for future research endeavors.
\hl{In clinical experiments on pathological speech transcription, we tested SSDM 2.0 using nfvPPA corpus primarily characterized by \textit{articulatory dysfluencies}.}
Overall, SSDM 2.0 outperforms SSDM and all other dysfluency transcription models by a large margin. See our project demo page at \url{https://srnf2.github.io/}.

Decision: withdrawn
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730067845735

thread_type:
official_review

<thread_object>
creator: reviewer_ch6n

<content>
summary:
The paper presents a novel pipeline to model and identify speech disfluencies. In tandem it presents a sizable dataset containing disfluent utterances for further work in the field.

strengths:
The paper offers several novel techniques to assist in ongoing work for modeling speech disfluencies. I particularly find their development of the neural articulatory flow model fruitful for in-depth modeling of patient vocalizations during therapy. Each method utilized in their pipeline is of significant interest for a standalone paper on the topic, and has sizable justification for consideration in work in this field. In addition, their development of a dysfluency corpus is incredibly important for the field of dysfluency modeling (most datasets are rather sparse and limited in coverage, there is a critical need in the research community for more datasets for study).

weaknesses:
This paper has a significant organizational issue that makes it difficult for an outside reader to follow. The vast majority of necessary information for a researcher outside disfluent speech research to understand the paper has been relocated to the appendix. This makes reading and evaluation of the paper rather difficult. For example, the researchers have chosen nfvPPA as a primary case study for their experiments in a field that could just as well consider Parkinsons and other variants of Aphasia as a case study. This information is not presented to the reader in the paper proper; it is instead moved to Appendix A.2. This is information that should be more clearly presented. 

While this would not be an issue for a specialized conference, ICLR is general enough that consideration should be made for outside researchers to follow. Care should be made for establishing clear cases of dysfluency and conditions that cause it. As well, given that articulation modeling is a key subject of the paper, it should be presented clearly in the background section.

questions:
Suggestion: reconsider what elements in the main paper and which elements in the appendix should be moved. Please take consideration that your colleagues may not have knowledge of dysfluency research or even models of language production.

Could you clarify why your models consider codecs? If was difficult to follow why these should serve as the audio input format.

Have you considered the potential bias involved in evaluating speech production and how it would dovetail with use of Speech LMs? While in aphasic cases this is more clear cut, I'm suspicious about how the work may mislabel dialect speakers as dysfluent speakers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662435415

thread_type:
official_review

<thread_object>
creator: reviewer_f1se

<content>
summary:
This paper proposes a framework named SSDM 2.0 to better transcribe the speech with dysfluencies. This framework includes Neural Articulatory Flow (NAF) to extract scalable and efficient dysfluency-aware speech representation, Full-Stack Connectionist Subsequence Aligner(FCSA) for dysfluency alignments, and Mispronunciation In-Context Learning and Consistency Learning in langauge
model for dysfluency transfer. On top of that, the author open-sources a large simulated speech co-dysfluency corpus called Libri-Dys-Co.

Experiments are designed to evaluate the scalability and efficacy of the proposed NAF representation and the performance of SSDM 2.0 in detecting dysfluencies in speech.

strengths:
* Novel methodology. The paper proposes multiple new methodologies such as Neural Articulatory Flow which encodes semi-implicit speech representations that align closely with human speech production and Fullstack Connectionist Subsequence Aligner(FCSA) which develops a comprehensive alignment mechanism capable of handling diverse dysfluencies, improving the precision of non-fluency detection. The proposed methods are shown to significantly reduce computational complexity, allowing the model to scale effectively with larger datasets. 
* Comprehensive evaluation. The author designs evaluation and benchmarks to cover proposed methods.
* Contribution of new resources. This paper open-sources a large-scale co-dysfluency corpus (Libri-Co-Dys) with over 6,000 hours of annotated speech, providing a valuable resource for future research in dysfluency transcription and detection.
* Potential real world application: The proposed framework could accurately transcribe and detect dysfluencies and has significant implications for speech therapy, language screening, and spoken language learning.

weaknesses:
* The writing style of this paper comes with overly formal language and verbose phrasing. One instance is the beginning of section 3: "Notwithstanding, the neural gestural system proposed by Lian et al. (2024) is encumbered by complex design and challenges in equilibrating multiple training objectives such as self-distillation." The usage of "notwithstanding", "encumbered" and "equilibrate" is not necessary and they all have synonyms that are commonly used. The phrasing of the sentence is not concise. Making the paper overall less comprehensible to the readers.
*  Framework complexity. Even though the paper claims that SSDM 2.0 has 10x less training complexity of its precedent work SSDM, the integration of NAF, FCSA, and NICL adds more layers of complexity to the model. There is lack of complexity comparison of other types of baseline systems to provide a reference of where it stands in all viable options.
* The presentation of the paper needs to improve:
  + Many concepts mentioned in this paper are not self-contained and the reader has to refer to other papers. For instance, AAI.
  + Given the layers of complexity brought by multiple different modules/methods in the framework, an overview of the it should be first given based on Figure 2 before breaking it down to individual components.
* Lack of ablation studies. Table 1 only shows difference with or without neural articulatory flow loss and post-interpretable training, there is no assessment of the significance of other proposed components.
* Lack of real-world data. There is a over-reliance on synthetic data which arises questions about how the system performs in real world scenarios.

questions:
+ I suggest removing the claim of complexity improvement from the paper if there is no enough space to explore in details.
+ In section 6.7, I think a more fair baseline would be SSDM 2.0 removing the NICL tuning steps.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689160135

thread_type:
official_review

<thread_object>
creator: reviewer_wcgm

<content>
summary:
The paper presents a novel approach to transcribing speech that includes non-fluencies (such as repetitions, blocks, and sound replacements) beyond traditional text transcription. The authors introduce SSDM 2.0, which improves upon previous models by offering a highly scalable speech representation system called Neural Articulatory Flow, a comprehensive Full-Stack Connectionist Subsequence Aligner (FCSA) for aligning dysfluent speech with text, and a new method for in-context pronunciation learning that enhances dysfluency detection. Additionally, they open-source the largest co-dysfluency corpus, Libri-Co-Dys, for future research. SSDM 2.0 outperforms existing dysfluency transcription models while reducing training complexity significantly. The paper demonstrates the scalability and efficiency of the proposed system in both single and co-dysfluency transcription tasks.

strengths:
1. The paper introduces SSDM 2.0, a novel model that goes beyond traditional transcription to accurately capture non-fluencies, filling a gap in existing speech transcription research. It introduces creative components like Neural Articulatory Flow for scalable speech representation and the Full-Stack Connectionist Subsequence Aligner for precise dysfluency alignment.
2. SSDM 2.0 is thoroughly evaluated with rigorous experimentation across multiple datasets.
3. The paper clearly defines its focus on time-accurate dysfluency transcription and outlines the shortcomings of previous models.
4. The model’s ability to transcribe co-dysfluencies has broad implications for fields such as speech therapy, language learning, and automatic speech recognition.

weaknesses:
The paper includes a great deal of detail. However, there are several aspects that could be further clarified:
1. The evaluation details of the HuBERT and Soundstorm models in Table 1 are not explained in the paper. For instance, the original HuBERT paper only tested on the LibriSpeech and LibriLight datasets. How was the HuBERT model in this paper trained, and what training data was used?
2. The Acoustic Encoder of the model is WavLM, but Table 1 only shows results for HuBERT. Since WavLM generally outperforms HuBERT, the authors should consider adding more experiments comparing WavLM with NAF.
3. While the paper introduces many new modules, the comparison of baseline models could be more diverse. For example, the authors could include results from directly using a speech encoder (e.g., HuBERT, Whisper encoder) to extract speech representations as input for the LLM, followed by fine-tuning with LoRA. This would provide a clearer understanding of whether the added modules effectively improve the model's performance.
4. Although many new modules are proposed, there is a lack of comprehensive ablation studies to verify whether each module is necessary. For example, Equation 11 includes six different loss terms, but there are no corresponding experiments demonstrating whether each loss term contributes to performance improvement. While some related experiments are included later in the paper, they remain incomplete.

questions:
1. How was the HuBERT model trained in this paper, and what specific training data was used, given that the original HuBERT paper only evaluated on LibriSpeech and LibriLight datasets?
2. Why does Table 1 only show results for HuBERT when the Acoustic Encoder is WavLM, and can the authors provide additional comparisons between WavLM and NAF, considering WavLM generally outperforms HuBERT?
3. Could the authors include more diverse baseline model comparisons, such as using a speech encoder like HuBERT or Whisper encoder with fine-tuning via LoRA, to better assess the impact of the new modules on performance improvement?
4. Can the authors provide more comprehensive ablation studies to verify the necessity of each proposed module, including experiments showing the impact of each loss term in Equation 11 on model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798707481

thread_type:
official_review

<thread_object>
creator: reviewer_i1ks

<content>
summary:
In this work, the authors tackle the problem of fine-grained speech transcription along with non-fluencies (dysfluencies). Towards this, they present a new system SSDM 2.0 (that builds on top of SSDM, Lian et al. 2024) with some key modifications including a new articulatory feature-driven flow-based model, a full-stack connectionist subsequence aligner for improved coverage of dysfluencies and mispronunciation in-context learning using LLMs. This system consistently outperforms SSDM on multiple evaluation benchmarks.

strengths:
* The problem is well-motivated and the authors propose multiple new and innovative techniques (SSDM 2.0) to tackle the problem of time-accurate transcription of speech with dysfluencies.
* The authors open-source a large dysfluency corpus (Libri-Co-Dys) to encourage further work in this domain.
* SSDM 2.0 is significantly more performant (both in terms of accuracy and efficiency) compared to prior state-of-the-art (SSDM).

weaknesses:
* The writing could be improved overall, especially the main technical parts in Sections 3, 4.
  * Rather than diving right into the details of each module in Section 3, it would have been useful for the reader to first get a complete description of the full SSDM 2.0 framework and the role of each module (neural articulatory flow, full-stack CSA, etc.) in the overall pipeline before learning about individual details.
  * Notation can be made cleaner in many places (e.g., inconsistent definition of H with H \in R^{K \times T} in Section 3.1 and H \in R^{T \times K} in Section 3.2), and illustrative examples would be useful especially for full-stack CSA.
  * There are also many references to Lian et al. 2024 which disrupts the overall flow and makes it less self-contained.

questions:
* The authors should motivate the gestural scores in H, right in Section 3.1 -- How is it rule-generated, what is K, why is it sparse and how does H relate to articulatory gestures? Since X is already articulatory data, how does the information in H differ from that in X?  More about H only appears scattered across Sections 3.2 and 3.3.

* Can the authors further motivate their choice of using a count encoder + index generator to identify active regions? One could devise alternate schemes like using discretized binary gates (using Gumbel softmax) for every element in X_i to denote whether or not it is part of an active region. It would be useful to know if the proposed design was motivated by some articulatory or speech production priors.

* From the proposed SSDM 2.0 architecture and the demo page, the LLM responses are descriptive and in natural language. Apart from objective measures of F1/WPER, did the authors consider evaluating these free-form responses containing pronunciation feedback via a human evaluation?

* Libri-Co-Dys (with multiple dysfluencies) is presumably harder than Libri-Dys (with a single dysfluency). However, the F1 scores using SSDM 2.0 on Libri-Dys appear to be no worse than that on Libri-Co-Dys (across 30, 60, 100 training data settings). Could the authors comment on this?

* Is there any fraction of Libri-Co-Dys that does not contain dysfluencies? If not, it is useful to check how SSDM 2.0 performs on fluent instances. Please comment.

* Minor comment: There are no arrows coming out of 3 and 4 (inside yellow circles) in Figure 4 (under Pre-Alignment Training).

* Section 6.8 could be moved entirely to Appendix A.5.

* The authors should do a careful editorial pass of the draft. Listing some examples of typos below:
  * ARTCULATORY --> ARTICULATORY (Title of Section 3.2)
  * Traing Data --> Training data (Section 6.2)
  * Title of Section 6.4 does not parse
</content>
</thread_object>

</thread>
<thread>
creation date:
1731999861267

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response Regarding Paper Update

comment:
Dear Reviewers:

> Thank you for your thorough and in-depth reviews of our paper. We have updated the manuscript by reorganizing sections and adding additional experiments. Below is a global response summarizing the changes made to address your reviews. Note that we only discuss reviews that led to updates in the paper; questions directly addressed in responses are not listed here.
> 
> For clarity, we use different colors to highlight updates made according to each reviewer's feedback:
>
>> Reviewer cH6N: yellow
>
>> Reviewer f1SE: green
>
>> Reviewer wcgM: blue
>
>> Reviewer i1kS: red
>
> When multiple reviewers raised the same concern, we randomly selected one color to represent the change.
> Summary of changes in the updated manuscript:
>
>> (1) Enhanced clarity regarding dysfluency and clinical background in the introduction (highlighted in yellow, addressing Reviewer cH6N's concerns)
>
>> (2) Improved and simplified introduction to articulatory modeling in Sections 3 and 3.1 (highlighted in green and red, addressing concerns from Reviewers cH6N, f1SE, and i1kS)
>
>> (3) Added new Section 2 summarizing the SSDM 2.0 pipeline (highlighted in red, addressing concerns from Reviewers f1SE and i1kS)
>
>> (4) Added experiments on accented speech (VCTK, Common Voice, GLOBE), addressing Reviewer cH6N's major question and Reviewer f1SE's concern about real-world data experiments (highlighted in yellow, results in Appendix A.16 and Table 10)
>
>> (5) Tested SSDM 2.0 on real-world stuttered speech datasets UCLASS and SEP28K (highlighted in green, addressing Reviewer f1SE's concerns, results in Appendix A.17 and Table 11)
>
>> (6) Added extensive ablation studies across modules (NAF, FCSA, NICL) and loss objectives (addressing questions from Reviewers wcgM and f1SE, results in Appendix A.18 and Table 12)
>
>> (7) Added WavLM ablation in Table 1 (highlighted in blue, as suggested by Reviewer wcgM)
>
>> (8) Added w/o NICL ablation in Section 6.7 (highlighted in green, as suggested by Reviewer f1SE)
>
>> (9) Added fluent speech (LibriTTS) dysfluency ablation to explore false positives (highlighted in red, as suggested by Reviewer i1kS, results in Appendix A.15 and Table 9)
>
>> (10) Relocated Related Work section to Appendix B and adjusted image and table sizes to optimize space
>
>> (11) Corrected typos as indicated by specific reviewers (marked in corresponding colors)
>
>  We are happy to address any additional concerns or remaining questions that reviewers may have. Please feel free to reach out for further clarification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732169912080

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response Regarding the Second Update of the Paper

comment:
Dear Reviewers,

We have made a second update to our paper as suggested by reviewer cH6N, with the changes highlighted in deeper yellow. Here are the updates:
> 
> * In Introduction, we updated the first two paragraphs again by constraining our focus to pathological speech disorders only, specifically articulatory-based dysfluencies. The abstract and conclusion have also been slightly edited.
> 
> * In Section 3, we stated that articulatory representation learning is the natural way for articulatory dysfluency modeling.
> 
> * We removed codec experiments from Table 1 to reduce confusion.
> 
There are no updates regarding other reviewers' reviews since the last update. We really appreciate the  in-depth suggestions that we have received to make the manuscript more solid. Please feel free to propose any questions or suggestions!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733126482863

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Reviews from Authors

comment:
Dear reviewers,

Thank you for your valuable reviews. As we have not received further discussion regarding our rebuttal, **we believe this indicates that all concerns and weaknesses raised by reviewers have been adequately addressed**. Given that we have addressed all concerns, we would greatly appreciate your consideration in raising the scores accordingly. We remain available to provide any additional clarification should you have new suggestions. 

Sincerely
</content>
</thread_object>

</thread>


Metareview: