Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder

Authors: Muhammad Shakeel, Yui Sudo, Yifan Peng, Chyi-Jiunn Lin, Shinji Watanabe
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Speaker diarization, speech separation, multi-speaker speech recognition, overlapped speech recognition, end-to-end, multitask learning

Abstract:
The rapid progress of single-task architectures has dominated recent developments in multi-talker speech processing, prompting the need for unified approaches. This paper introduces a unified multi-speaker encoder (UME), a novel model architecture that jointly learns representations for diarization, separation, and multi-speaker automatic speech recognition (ASR) tasks using a shared pre-trained foundational speech encoder. We leverage the hidden representations from multiple layers of UME to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME achieves substantial improvements over the single-task state-of-the-art (SOTA) baselines dedicated to speaker diarization, speech separation, and multi-speaker ASR. Notably, for speaker diarization, UME achieved SOTA performance by lowering the diarization error rate (DER) from 3.24 to 2.19 on the Libri2Mix dataset. Furthermore, our results in multi-speaker ASR outperform the previous results, reducing the concatenated minimum-permutation word error rate (cpWER) from 11.9 to 9.2 on the LibriSpeech2Mix evaluation set.

Decision: reject
Reviewer scores: 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730286051167

thread_type:
official_review

<thread_object>
creator: reviewer_wjfq

<content>
summary:
This paper presents a method of combining diarization, separation and ASR tasks with multi-layer feature learning in a unifed model. Experiments have been conducted to evaluate the effectiveness of the proposed method.

strengths:
The paper is written clearly and easy to follow.

weaknesses:
1) The novelty is limited. As introduced in Section 2, multi-layer feature learning and joint training of speech tasks have been investigated in plenty of previous studies. The main contribution of this paper is to include more speech tasks into joint training and to combine it with multi-layer feature learning, which is limited in my opinion.
2) As showin in experimental results, the task weights for joint training in Eq.(16) play an important role in model performance. Thus, how to optimize these weights should be carefully explained.

questions:
According to the results shown in Table 3 and 4, the layer weights for the three tasks were quite similar in both joint training configurations. Therefore, it may not be necessary to use task-dependent layer weights. How about the performance of using unifed layer weights across different tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730312196701

thread_type:
official_review

<thread_object>
creator: reviewer_7dga

<content>
summary:
The authors proposed a "Unified Multi-Speaker Encoder (UME)" for tasks such as speech separation, speech diarization, and multi-speaker ASR. Specifically, the UME employs a pre-trained OWSM encoder and is jointly fine-tuned across all three tasks. The latent embeddings from multiple layers of the UME are combined using a weighted sum with learnable coefficients. However, the novelty of this approach is limited, and its performance is not as good as that of state-of-the-art methods

strengths:
This manuscript attempts to construct a unified encoder for speech-related tasks in multi-talker scenarios. This research question is necessary and important.

weaknesses:
- The manuscript lacks novelty. Numerous works have already discussed the use of multi-layer feature fusion/weighted sum, such as "Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification," "Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers," and "Resource-Efficient Transfer Learning from Speech Foundation Model Using Hierarchical Feature Fusion." Compared to the aforementioned studies, this manuscript makes no novel contributions to methodology. Instead, it leverages existing techniques to address three speech-related tasks in multi-speaker scenarios.

- The authors overclaimed achieving state-of-the-art performance in speaker diarization, speech separation, and multi-speaker ASR. 
  - However, for speech diarization, they only compared their method with ConvTasNet (2019) and did not benchmark against more recent methods such as Mossformer or Mossformer2. 
  - For multi-speaker ASR, the proposed method achieved a WER of 9.2% on LibriSpeechMix. In contrast, the state-of-the-art performance is 3.43%, as reported in "Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System."  
  - For speaker diarization, the authors only compared their method with EEND-based methods. The performance of their UME when combined with non-E2E method is not explored. 

- The author reported the model's performance solely on two-speaker simulated overlapped speech for all three tasks. The performance in real-world scenarios and with more speakers remains under investigation.

questions:
1. Can you explain the novelty of your approach compared to existing multi-layer weighted sum methods, aside from fine-tuning the model on the three multi-speaker-related tasks?
2. Why did you not compare your method with the latest research?
3. Why did you not report the results from other papers that demonstrate better performance?
4. It would be beneficial to discuss the performance on a broader range of datasets, rather than limiting the evaluation to only two-speaker LibriMix and LibriSpeechMix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387712334

thread_type:
official_review

<thread_object>
creator: reviewer_xggr

<content>
summary:
This work proposes a method where one speech foundation model is used as backbone to three task-specific heads. Unlike the standard SUPERB protocol, these heads are trained in a multi-task setup to perform speaker diarization, speech separation, and multi-speaker speech recognition.

strengths:
This work tackles the inherently difficult problem of solving different, orthogonal speech technology tasks. A real-world ASR system needs to solve these, and currently this is done with a pipeline approach of multiple models. Unifying this pipeline of speech separation, speaker diarization, speech recognition, etc into a single model is beneficial to the community.

weaknesses:
I do not believe this work uses a fair evaluation protocol, as it is stated in Section 4.1 that the the speaker diarization test set uses 100% overlapped speech. From my understanding of DER, this means that the model needs to simply predict that each of the 2 speakers are talking all the time. The low but not perfect scores in Table 1 are then simply due to noise in the ground-truth labels? 

The paper is well-structured, but there are some sentences which are not clear or contain typos. See e.g., 
* line 37-38 "A key limitation of training tasks independently is that inter-dependencies cannot be leveraged"
* line 40 "Most existing speech-processing frameworks address this limitation with unified, joint training architectures" (also, 'unified, joint training architecture' is vague)
* line 48: delete " so much", add, e.g., "do not work well on"
* line 154 "gorund truth" 
* line 156 "where a T-length..."
* line 298 "metrics" -> "metric"
* line 367 "Unlike WavLM ... outperforms WavLM" This sentence is hard to parse, split it up, e.g.: "WavLM uses overlapped speech mixtures, while OWSM is just trained on clean speech. We observe that... OWSM > WAVLM"
* line 369 "We explain" -> "We speculate" or "We hypothesize" 

I believe reproduction of this work is difficult due to missing details on dataset generation and model training, see my questions below.

questions:
* Can the authors comment on the use of 100% overlapped speech for evaluation of SD? Does the SUPERB benchmark for SD also use 100% overlapped speech?
* What model type was used for OWSMv3.1(base, small, medium , LR)? From Figure 3 I assume medium, but I do not think the text states this anywhere.
* What does a "flat start" mean in line 342?
* For how many epochs/steps did you train in total?
* How much VRAM is needed to train your model, which GPU(s) did you use, and how long did it take to train?
* What batch size did you use, and how were these sampled? 
* What is the minimum, maximum, and average duration of utterances in your train/eval dataset(s)?
* I think it is unexpected (From Figure 3 and Figure 4) that the weights are nearly identical between task heads. Do the authors know what weights are found when training in a single-task setup, and whether they are similar or different to the values displayed in these figures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708849807

thread_type:
official_review

<thread_object>
creator: reviewer_rojk

<content>
summary:
The authors introduce a method to concurrently train models for speech recognition, speech separation, and speaker diarization, utilizing embeddings from a pre-trained speech foundation model (SFM). They leverage a weighted sum of the model’s layer embeddings as input for each task. Experimental results indicate that this joint training approach enhances the performance of all three tasks.

strengths:
The paper introduces a unified architecture that tackles Speech Separation, Speaker Diarization, and ASR tasks simultaneously. Experimental results confirm that this multitask framework enhances the performance of each task, demonstrating their mutual benefit.

weaknesses:
As noted by the authors, previous works have utilized SFMs like WavLM for SS and ASR tasks. Although the proposed extension to cover all three tasks shows metric improvements compared to other SFMs, the innovation is minimal.

Additionally, the authors have only tested their approach on a simulated dataset. A more comprehensive evaluation using datasets like AMI, ICSI, or LibriCSS would better validate their method.

questions:
1. Clarification on Figure 1:
Figure 1 appears misleading. Equation (3) shows different weights for different tasks, yet Figure 1 depicts only a single weight. Can you clarify this discrepancy?


2. Training with Real Data:
How do you propose to use real data for training the model, given that datasets for these tasks (e.g., ASR and speech separation) are typically disjoint? For example datasets for ASR do not generally contain ground truth for speech separation. How will this disjoint nature impact the joint training of the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732343910827

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of changes in the revised manuscript

comment:
**Summary of changes in the revised manuscript**

We have revised the manuscript and uploaded an updated version incorporating the following changes:

* Corrected the variable label in figure 1.
* Added the minimum, maximum, and average durations of utterances in Section 4.1.
* Clarified details about the implementation, experimental setup, and the choice of weights for task-specific losses.
* Included Libri3Mix results for completeness in Tables 2, 3, and 4.
* Added a discussion of the Libri3Mix experiments in Section 5.2.
* Added recovered speech examples for the three-speaker case in Appendix A.3.
* Provided layer weights for single-task models in the Appendix A.1 and moved the previous layer weights figure there due to space constraints.
* Added a discussion on the limitations of using simulated datasets in the Limitations section.

**Note:** All updates in the revised paper are highlighted in red text for easy reference.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper Introduces a unified architecture that simultaneously tackles Speech Separation, Speaker Diarization, and ASR tasks, enhancing the performance of each task through a multitask framework. It addresses the practical need for a unified model in real-world ASR systems, which currently relies on a pipeline approach with multiple models. It demonstrates that solving different, orthogonal speech technology tasks together can be mutually beneficial, improving overall system performance. The paper is clearly written and easy to follow, making the complex concepts accessible to readers.

However,  the manuscript leverages existing techniques without making novel contributions to methodology, similar to previous works on multi-layer feature fusion and joint training. The proposed extension to cover all three tasks (Speech Separation, Speaker Diarization, and ASR) shows metric improvements but lacks significant innovation compared to previous works. The approach is only tested on a simulated dataset, lacking comprehensive evaluation on real-world datasets like AMI, ICSI, or LibriCSS. The authors overclaim SOTA performance without adequate benchmarking against more recent methods for speaker diarization, speech separation, and multi-speaker ASR. The model's performance is only reported for two-speaker simulated overlapped speech, with real-world scenarios and more speakers remaining untested.