Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
Authors: Taejin Park
Ivan Medennikov
Kunal Dhawan
Weiqing Wang
He Huang
Nithin Rao Koluguri
Krishna C Puvvada
Jagadeesh Balam
Boris Ginsburg
Venue: ICML 2025 poster (2025)
Keywords: Automatic Speech Recognition
Speech to Text
Speaker Diarization
Multi-speaker ASR
Multi-talker ASR
Primary area: applications->language_speech_and_dialog
TL;DR: Introducing Sortformer, a novel model that resolves speaker permutations in multi-speaker ASR using Sort Loss and arrival-time sorting, enabling seamless integration of speaker recognition into speech-to-text systems.

Abstract: Sortformer is an encoder-based speaker diarization model designed for supervising speaker tagging in speech-to-text models. Instead of relying solely on permutation invariant loss (PIL), Sortformer introduces Sort Loss to resolve the permutation problem, either independently or in tandem with PIL. In addition, we propose a streamlined multi-speaker speech-to-text architecture that leverages Sortformer for speaker supervision, embedding speaker labels into the encoder using sinusoidal kernel functions. This design addresses the speaker permutation problem through sorted objectives, effectively bridging timestamps and tokens to supervise speaker labels in the output transcriptions. Experiments demonstrate that Sort Loss can boost speaker diarization performance, and incorporating the speaker supervision from Sortformer improves multi-speaker transcription accuracy. We anticipate that the proposed Sortformer and multi-speaker architecture will enable the seamless integration of speaker tagging capabilities into foundational speech-to-text systems and multimodal large language models (LLMs), offering an easily adoptable and user-friendly mechanism to enhance their versatility and performance in speaker-aware tasks. The code and trained models are made publicly available through the NVIDIA NeMo Framework.

Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
Taejin Park 1
Ivan Medennikov Kunal Dhawan 1
Weiqing Wang"
He Huang
Nithin Rao Koluguri
Krishna C. Puvvada 1
Jagadeesh Balam
Boris Ginsburg'
Abstract
Random Permutation
Arrival Time Order
Speakers
Speakers
Sortformer is an encoder-based speaker diariza- tion model designed for supervising speaker tag- ging in speech-to-text models. Instead of rely- ing solely on permutation invariant loss (PIL), Sortformer introduces Sort Loss to resolve the permutation problem, either independently or in tandem with PIL. addition, we propose a streamlined multi-speaker speech-to-text architec- ture that leverages Sortformer for speaker super- vision, embedding speaker labels into the encoder using sinusoidal kernel functions. This design ad- dresses the speaker permutation problem through sorted objectives, effectively bridging timestamps and tokens to supervise speaker labels in the output transcriptions. Experiments demonstrate that Sort Loss can boost speaker diarization per- formance, and incorporating the speaker super- vision from Sortformer improves multi-speaker transcription accuracy. We anticipate that the pro- posed Sortformer and multi-speaker architecture will enable the seamless integration of speaker tag- ging capabilities into foundational speech-to-Bt -totetttttttttttt systems and multimodal large language models (LLLMs), offering easily adoptable and user- friendly mechanism enhance their versatility and performance in speaker-aware tasks. The code and trained models are made publicly avail- able through the NVIDIA NeMo Framework.
Time
Loss Function
CPn
Function
L
End and-to-End Dier taaion
M
Multi-speaker Audio Recording
Figure 1. Sortformer resolves permutation problem by following the arrival-time order of the speech segments.
scription services, a growing number of applications require speaker annotations because natural language understanding (NLU) modules need to recognize speakers to gain deeper understanding of conversations and interactions. Moreover, as modern machine learning models demand large amounts of training data, the need for automatic annotation systems has grown significantly.
The rising demand for speaker annotations underscores the need for robust speaker tagging- also known as speaker di- arization, which is the process of estimating generic speaker labels by assigning audio segments to individual speakers. In the context of automatic speech recognition (ASR), multi- speaker ASR (also referred to as speaker-atrributed attributed ASR or multi-talker ASR in the literature) requires the speaker diarization process, either directly or indirectly, to transcribe spoken words with speaker annotations alongside the gener- ated text. As ASR models continue be streamlined and become more accurate, speaker diarization is progressively integrated into the ASR framework performed simulta- neously during the ASR decoding process, enabling rich transcription with conversational context.
1. Introduction
With recent advances in deep neural networks and large language models (LLMss), automatic speech recognition (ASR) being deployed across broader range of industrial applications, enabling numerous new cases. In tran-
Despite recent advances in speaker diarization and multi- speaker ASR, these systems have been typically trained, deployed, and evaluated separately from ASR models due to challenges such as data scarcity and application diversity. Collecting annotated multi-talks conversational speech is significantly more difficult than acquiring images or single- speaker speech data, particularly for low-resource languages or privacy-sensitee domains such as medical applications. Additionally, multi-speaker ASR use cases often require models to perform inference on multi-how audio samples,
Eqqual contribution 'NVIDIA, Santa Clara, USA. Correspon- dence to: Taejin Park <tacjinp@ nvidia.com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sortformer: A Novel Approach for Permutation- Resolvve Speaker Supervision in Speech-to-Text Systems
while acquiring such long-form training data more challenging.
Sorted Speaker Tokens
<splO> oh <spki> great <spk0> did <spid> you <spk0> grow <spk0> <spk0> there <spk2> yeah <apk3s <spk3> <spk2> <epk3>> *spk2> <spkš= <spk2>
Although these cascaded multi-speaker ASR systems achieve competitive performance, optimizing or fine-tuning high-performance multi-speaker ASR systems for specific domains remains considerable challenge, as demonstrated by evaluations such as CHiME challenges (Barker al., 2017; 2018; Watanabe al., 2020; Cornell et al., 2023). On the other hand, end-to-end multi-speaker ASR models without explicit speaker diarization modules have been proposed (Kanda et al., 2020b; Shi et al., 2024) which are based on the serialized output training (SOT) tech- nique. However, training such end-to-end multi-speaker ASR systems requires speaker-annotatee multi-speaker data, which relatively scarce and challenging to collect and annotate. As result, the performance of end-to-end multi- speaker ASR systems tends to lag behind that of cascaded systems (Kanda et al., 2022b).
LCE
Cross-Entropy Loss
Transformer Decoder
Adapter
Sinusoidal Speaker Kernels
Speaker Kernel Encoded ASR Encoder State
Fast Conformer ASR Encoder
Sortformer
Enc. Adapter
To address these challenges, we propose Sortformer', in- troducing Sort Loss and techniques for bridging times- tamps with text tokens. Despite the popularity of end-to- end speaker diarization systems, such speaker diarization models have not been able to be seamlessly integrated into ASR models or multimodal LLMs. To overcome this, we introduce arrival time sorting (ATS) approach, where speaker tokens from ASR outputs and speaker timestamps from diarization outputs are sorted by arrival times to resolve permutations (see Figure! 1).
Multispeaker Audio Recording
Figure 2.
The overall dataflow of speaker supervision from Sort- former model integrated into the proposed MS-ASR system.
RT03 evaluation (Tranter al., 2003), combined ASR word timestamps with speaker segmentations or attempted use phrase dictionaries CCanseco-Rodrigee which showed limited success. Recent advances, like TS- VAD (Medennikov et al., 2020a), revolutionized cascaded multi-speaker ASR by employing target-speaker voice ac- tivity detection into the multi-speaker ASR pipeline, in- fluencing subsequent systems (Wang Li, 2022: Yang et al., 2024). These approaches achieved strong results in real-world challenges (Wang et al., 2021; Niu et al., 2024), cementing their relevance in modular systems.
al., 2004),
Our proposed method enables multi-speaker ASR systems multimodal LLMs to be trained fine-tuned while sig- nificantly improving in speaker tagging accuracy with rel- atively small amount of fine-tuning. A key ad advantage is that multi-speaker ASR training can leverage a standard token- level cross-entropy loss, facilitated by the permutation- resolved speaker supervision of the Sortformer model. This approach makes multi-speaker ASR training functionally equivalent standard mono-speaker ASR training and fine- tuning, requiring only minimal architectural adjustments. Additionally, our method climinates the need for word-leve or segment-leve timestamps, significantly reducing anno- tation requirements. Furthermore, Sortformer can function independently as an end-to-end speaker diarization model.
To streamline the speaker diarization system, slew of end- to-end neural diarization (EEND) models were proposed, framing speaker labeling as frame-wise classification task with permutation invariant training (PIT) loss (Yu al., 20178 7b; Fujita et al., 2019). Subsequent works (Fujita et al., 2020; Takashima et al., 2021) introduced flexible output dimensions to accommodate varying numbers of speak- ers, while hybrid approaches like EEND-EDA (Horiguchi et al., 2020) and attention-based models (Chen et al., 2024) achieved greater accuracy.
2. Related Works
2.1. Speaker Diarization
Before multi-speaker ASR gained prominence, speaker diarization handled the task of identifying "who spoke when" without transcription. Early systems, such as the
2.2. Multi-speaker ASR
Early studies on multi-speaker ASR -SS-eexepplified by (Yu et al., 2017a; Qian et al., 2018) used separate components for source separation and transcription. Jointly trainable systems (Shafey al., 2019) later introduced speaker at-
htttp:: http://uugiigga.....ooo...dadddd cortformese 4opk
2

---

Sortformer: A Novel Approach for CommutatiinnResolveee Speaker Supervision in Speech-to-Text Systems
tribution alongside text generation. Key advancements like SOT (Kanda al., 2020b) enabled simpler models by leveraging attention mechanisms, extending to token-level SOT (t-SOT) for streaming (Kanda et al., 2022a). Recent systems, such as (Shi et al., 2024), adopt non-PIT loss schemes like dominance ranking. Strictly end-to-end sys- tems often face limitations in handling speaker counting and domain-speciff datasets (Shafey et al., 2019; Wang et al., 2024). Cascaded systems like Transcribe-to- to-Diarize (Kanda et al., 2022b) combine diarization and ASR with SOT, while modular systems incorporating clustering steps (Cornell et al., 2023) still show strong performance. However, such systems require extensive tuning, highlighting the need for more adaptable architectures.
While PIL shows promising results in the aforementioned tasks, PlL-bbased end-to-end speaker diarization model is more challenging to integrate into ASR systems. Since PIL requires a specialized loss function at the model's output layer, it limits its applicability when training multi-speaker ASR models for multiple tasks simultaneously using the same ground truth. For example, if a model is trained for tasks like speech summarization, speech translation, and multi-speaker ASR concurrently, this constraint mandates specialized loss calculation mechanism specifically de- signed for the multi-speaker ASR task. In contrast, the sorting-basel bbang approach does not impose special require- ments on the loss function. Once the speaker tokens in the ground truth labels are sorted, the model can be trained using the standard cross-entropy function on text tokens (see Fig. 2). This approach improves ease of use and adapt- ability, especially for those unfamiliar with complex model architectures.
2.3. Limitations of Previous Approaches
Despite the abundance of high-performing end-to-end di- arization and ASR models (Fujita et al., 2019; Horiguchi et al., 2022a: Chen et al., 2024). there have been limited efforts create a synergistic effect by integrating both models within differentiable computational graph. To the best of our knowledge, our proposed system is the first to integrate an end-to-end diarization system with an end-to-end multi-speakkr ASR model at the computational graph level. Challenge-winning cascaded or modular sys- tems (Cornell et al., 2023; Medennikov et al., 2020b; Niu et al., 2024) demonstrate remarkable performance, where speaker diarization and ASR are processed sequentially with additional source separation modules (Boeddecker et al., 2018: žmolíková et al., 2019). However, these systems are difficult to optimize because each component often needs to be tailored for specifcc datasets. Our approach focuses on ease of deployment and adaptability, where a multi-speaker ASR model is trained in the same way as mono-speaker ASR models, based on token objectives and cross-entropy loss.
3.2. Diarization Model Multi-label Binary Classifier
We propose model designed for the simultaneous estima- tion of class presences from a sequence of input tokens while the class labels follow the arrival time of cach speaker's 's first segment. Consider a set of frame-wise D-dimensional em- bedding vectors, {x where X E IRP = 1, T represents the frame index. Given the input sequence, the model is expected to generate the class presence vector sequence (6,TT= where & E IR
1....... T. In this context, & = [y1,e. ..., yK,e] denotes the class presences of K classes (K potential speakers) time t, where ykst E (0, 1} indicates the speech activity of the k-thh speaker at the t-th frame.
K T
P ((E Erl XT)
IIIIII (y xT) (1)
k=1
Sortformer assumes the conditional independence of ykue given the embedding vectors (features). Therefore, Sort- former employs Sigmoid instead of Softmax unlike the activation function for the output layer in the Transformer encoder (Vaswani et al., 2017). This assumption is formal- ized as in Eq. (1).
3. Proposed Approach: Sortformer
3.1. Permutation Problem in Diarization
Speaker diarization or speaker- ASR always ac- companies issues of permutation matching between inferred speaker and the ground-truth speaker during training for calculating losses or evaluation processes to find the right speaker mapping. To tackle this issue, the concept of PIL or PIT was first popularized by the two studies (Kolbæk et al., 2017; Yu et al., 2017b) for the task of speech source separation, which inevitably requires the model to handle PIL calculation. Following this, (Fujita al., 2019) adopted the concept of PIL for the task of speaker diarization, and later improved it in (Horiguchi et al., 2022a) by employing sqquence-to-sequemee model to generate the attractors by training the model with PIL.
Under this framework, the task is construed as a multi-label classification problem, which is amenable modeling via neural network, denoted by fe. The model is defined as follows:
P P ..... Pr] = fe (x,,...,TT)
(2)
where P. = [pr..,.... Pk,]] E [0. 1]Kr represents the pos- terior probabilities of the presence of K classes at frame index t, P is a K by T matrix that contains the columns of p. vectors and fe represents the Sortformer model with a set of parameters O. Each ÿk.t is defined as a binarized
3

---

Sortformer: A Novel Approach for Crrmuttaton-Resolvee Speaker Supervision in Speech-to-Text Systems
single example is defined as:
Arrival Sort
0 500 090 1110
Permutation
LBCE ().P) -- (ylo)(p) + log(1 p)) (3)
120cos
Ground Truth Labels
1co0e 20D
2000000
where E {0. 1 is the true speaker label for the example,
000 10:01111111
02080 100
and E [o, 1] is the predicted speaker probability for the positive class.
11 300
0 00000000
Hybrid Loss
Permutation Invariant Loss
Sort-Loss
Permutation Invariant Loss Hereafter, we refer to the function that computes PIL as Lrn. The definition of PIL can be described follows: Let Y (y yx] R be the ground truth speaker presence matrix, and P = qq, . qK] R be the predicted speaker presence matrix where K denotes the number of speakers and T denotes the number of frames. LPn. aims to find the permutation и that minimizes the error between the predicted matrix and the ground truth. Mathematically, defined as:
Prediction
Transformer Encoder Blocks
Sort
NEST Encoder
Cest-Conformer Encoder Blocks
Positional Encoding
Len. (Y P) min {LBCE (Y P)
(4)
xell
Pre-encade Layer
where IT is the set of all possible permutations of the indices {I,...,K] K]. and Y is the matrix Y permuted according to the permutation function म, i.e., Y, = .y=())....,, If we express the Eq. (4) using the speaker-wise class presence vector yı and speaker-wise posterior speaker prob- ability qu. the equation becomes:
Multi-speaker Audio Recording
Figure 3. Sortformer architecture with hybrid loss.
Leu. (Y, P) min well
LBCE (V qu)
(5)
value from Pk,t- Eq. (2) can be rewritten in matrix form by concatenating the vectors as X = [x1, X XT], where
K
K
each column of Y represents the class presence time
min
LECE (UrCk)..
(6)
i.e. Y [6 .......,. where {5+ ( is the estimation of class presences.
TK
k=1.m1
Sort Loss Sort Loss is designed to compare predicted outputs with true labels, typically sorted by arrival time order or another relevant metric. The key distinction Sort- former introduces compared to the previous end-to-end diarization systems such EEND-SA (Fujita et al., 2019), EEND-EDO (Horiguchi al., 2022a) lies in the organiza- tion of class presence matrix Y. Let V be a function that measures the arrival time of the first speaker segment for the corresponding speaker bin,
we decompose Ý in terms of speaker identity, the class presence matrix becomes Y = 82,. 9x] [T where ý& row-wise speaker presence vector, 8: [ūv.1 $k.2.........11 for the k-th speaker. Similarly, P can also be decomposed as [qi, 92: 92. qx] where de is row-wise speaker presence posterior probabilities, q: |P.1,PP,2: PK.T] for the k-th speaker. Thus, in its simplest form, the model can be represented P (X), where X E DDX the input sequence and P E KXT is matrix containing the speaker presence posterior probabilities.
(yy)) = min(t' ya.e + 00, [1, 1
(7)
where #f is the frame index of the first speaker segment for the k-th speaker. Sortformer is expected to generate values Y for each speaker index k where the following condition holds:
3.3. Loss Calculation
Binary Cross-Entropy The loss values for the individual sigmoid output Ps.t in the aforementioned model, repre- sented by fe (X). are calculated using the binary cross- entropy (BCE) function. Let PR.: represent the class pres- ence posterior probability in Eq. (2), where Pk.c E [0, 1]. simplify the notation, we drop and The BCE loss for
v(S)) < v < ... <V()))
(8)
which indicates that the model function fe learns to gen- erate the class presence output Y with row indices sorted in arrival time order. Let n the sorting function applied

---

Sortformer: A Novel Approach for SrimuutationResollver Speaker Supervision in Speech-to-Tex Text Systems
to the indices (1...., Kj and Y, be the vector y sorted according to the arrival time order sorting function 17, i.e.,
Speaker1
Speaker-2
Speaken-3
Speaker-4
ASR Embedding
n YY) Y, (yn(1)) -.. Yn(K))
(9)
Speaker-encoded Embedding
Using the arrival time function defined in Eq. (7), accord- ingly, the following conditions hold in the ground truth vectors Yn(k) for all K speakers:
Speaker-3
( (()) S (V)(2)) s... (V)(K)
(10)
Thus, sort-loss with the sorting function 1 is defined mathe- matically as:
Figure 4. Sinusoidal kernels are applied to represent speaker super- visions on top of the ASR embeddinggs.
K
Cson (Y,P) Loce (Y.: = K Σ
Coce(yn(s)..a)) qa), (11)
2017) inherently exhibits permutation equivariance when positional embeddings are omitted (see Appendix Sections E and F). Therefore, Sortformer employs positional em- beddings to provide the model with a sense of sequence ordering.
where Yn(k) is the vector of true labels that are sorted in arrival time order resulting in the sorted index 1y(k), q is the vector of predicted outputs, LECE (y M()) qs) represents the loss for the k-th speaker, and K is the total number of speakers.
4. Bridging Timestamps and Tokens
4.1. Efficice- Training with Adapters
Hybrid Loss While Sortformer can be trained solely with
Sort Loss, there is a limitation that arrival time estimation is not always correct. This issue becomes more pronounced as the number of speakers increases during the training session. Note that Sortformer models can be trained using Sort Loss only, PII. only, or a hybrid loss by setting the weight between these two losses. The hybrid loss Laybrid can be described as follows:
To effectively leverage the knowledge from pretrained ASR model, we incorporate adapters for multi-speaker ASR tasks, as outlined in (Wang et al., 2024). A common challenge with fully fine-tuning pretrained ASR model on new tasks is that it tends to forget previous tasks. In our case, the primary distinction between single-speaker and multi- speaker ASR lies in the insertion of speaker tokens into the single-speaker transcripts. Consequently, preserving the previously acquired knowledge becomes crucial for multi- speaker ASR. This makes the use of adapters, as described in (Houlsby et al., 2019), a more effective approach.
Luybrid Lson + (1 a)) CeIL:
(12)
where a is empirically determined weighting factor.
3.4. Transformer Encoder Learns to Sort
4.2. Speaker Supervision with Speaker Kernel
Sort Loss, along with sorted target objectives, enables the model to learn the sorting of arrival times it generates frame-level speaker labels. Therefore, model trained with Sort Loss can be viewed as performing neural sorting, as the sorting operation is integrated into the Transformer's matrix multiplication process. Sortformer models are trained by minimizing the Sort Loss, which used to train fe:
The most crucial part of integrating the speaker diarization model and the ASR model is how the words or tokens are assigned to speaker labels. In our framework, the diarization result is treated as speaker encoding by injecting informa- tion through differentiable kernels. Fig. 4 shows how sinu- soidal kernels are added to the original ASR encoder states. Let Th be the speaker kernel for the k-th speaker. We define Kk, (in((2nkz)/)) M/M), 7 = [KA.1 K KKMMM ]. and I = [n. 72........ 7k]T, where M is the dimension of the ASR encoder state, the embedding vector bin index, and r E KKM contains the kernels for K speakers. We employ additive kernels based on the aforementioned sinusoidal functions. The following equation represents the kernel-base speaker encoding:
Lson fe()) LBCE (Y,, fe (x)).
(13)
It is worth noting that our model differs from conventional Transformer-basee end-to-end diarization systems, such as EEND-SA (Fujita et al., 2019) and EEND-ED (Horiguchi et al., 2022a) through its use of positional embeddings. These baseline systems do not require positional embed- dings, as the ordering of speaker labels is not relevant to these end-to-end diarization models. However, the multi- head self-attention (MHA) in Transformers (Vaswani et al.,
A
Ä
T.P. .P.
(14)
A
2
5

---

Sortformer: A Novel Approach for Permutation-Resolyye Speaker Supervision in Speech-to-Text Systems
Multi-speaker Transcription
SOT focuses on speaker change point token <cs> with serialized outputs, while SST does not employ speaker change points but employs sorted speaker tokens to assign generic speaker index (e.g., <spk3>) for each word. On the other hand, the word-level objective simply places speaker tokens before each and every word.
great oh you grow up there
yeah
born and raised
have to
Segment Level Objective (SST)
<spk0> <spki>> did you grow there <spk2>> yeah <spk3> you have <spk2> born and raised
Word Level Objective (SST)
Permutation Resolution via Sorting In this work, all multi-speaker ASR training sessions use the same cross- entropy loss as conventional single-speaker ASR mod- cls. without relying on permutation tive permutation-handling losses. Instead, the permutation problem resolved by directly matching the model's logit outputs with speaker and word tokens that are sorted by speaker arrival time, as illustrated in Figure 2 and Figure 5. Speaker tokens in the ground truth transcriptions are sorted by arrival time during data preparation, and the Sortformer module is designed to generate speaker predictions in the same arrival-time order, ensuring correct alignment with these ground truth labels. During the multi-speaker ASR training, Sortformer can either be fine-tuned, or its weights can be kept frozen. Regardless of whether Sortformer's weights are fine-tuned or frozen, the training of the multi- speaker ASR system is governed solely by the cross-entropy loss applied to the output tokens.
<<pk0>> great <spki> oh <spki> did <spki> you <spki>> grow <spki> up <<apk1> there <apk2> yeah <apk3> no <spk3> you <apl3>> have capk2> born
<spk3>> to <apl2>> and <apk2>> raised
Serialized Output Training (SOT) style
or alterna-
great <CS> oh did you up there <CS> yoah bomn and raised <CS> 50 you have
Figure 5. Three types of transcriptions for multi-speaker ASR model training.
where A is the encoder state (also referred to as the ASR embedding) from the encoder part of the ASR model, and ||A||2 2 denotes the L.2 norm of each column (feature vector) in A. A is the speaker-encoded encoder state matrix with dimensions M by T. P is the output from Eq. (2), which we refer to as speaker supervision.
4.3. Sorted Speaker Tojen-Objectives in Transcript
Sorted Serialized Transcript We use speaker tokens that represent the generic speaker labels such <spkO>, <spk1>, <spkK>. These speaker tokens appear single tokens in both predicted text and ground truth text. In the ground truth text, these tokens are also sorted in arrival time order, mean- ing the first appearing speaker is assigned <spkO> where the second appearing speaker is assigned <spk1> and so on. Therefore, if both a word and the corresponding speaker's 's speech segment are recognized correctly, these speaker tokens and speaker kernels are aligned by the decoder. As result, the ASR model and Sortformer diarization model can be trained or fine-tuned using a standard cross-entropy loss on these sorted tokens (including speaker tokens), without requiring a separate PIT or PIL mechanism for the ASR decoding stage. We refer to the transcriptions that include sorted word-level objectives as Sorted Serialized Transcript (SST). Our approach does not require word-level timestamps for training, thanks to the word-timestamp approximation scheme (see Appendix C for more details).
5. Experimental Results
5.1. Diarization Model Training
1. DATASETS
For training data of the Sortformer end-to-end diarizer, we use combination 2,030 hours of real data (Fisher English Training Speech Part1 and 2 (Cieri et al., 2004), AMI Corpus Individual Headset Mix (IHM) (Kraaij et al., 2005) using the train and dev split from (Landini et al., 2022). DIHARD3- dev (Ryant et al., 2020), VoxConverse-v0.3 (Chung et al., 2020), ICSI (Janin et al., 2003), AISHELL-4 (Fu et al., 2021), NIST SRE 2000 CALLHOME Part1 (Przybocki & Martin, 2001) which we refer as CALLHOME) and 5150 hours of audio mixture data (created using using LibriSpeech (Panayotov et al., 2015) and NIST SRE04- -oorilez- 2014) as source datasets) generated by an open-source speech data simulator (Park et al., 2023).
10 (Doddington et al., 2000;
Word level vs. Segment level In our proposed framework for training multi-speaker ASR models, speaker tokens can be applied at two different levels as described in Figure 5. A speaker token is placed in front of every word. The order of words is determined by comparing the onset (start time) of each word. The segment-level objective is conceptually similar to the SOT (Kanda et al., 2020b; 2021) style, while the speaker tokens used in our work are sorted speaker indices and not the change of speaker token, In comparison,
All parameters for audio mixture generated using an open- source speech data simulator (Park et al., 2023) are default settings except that the overlap ratio is set to 0.12 and the average silence ratio is set to 0.1. We evaluate the model per- formance on DIHARD3 eval (Ryant et al., 2020) (referred to
We use the two-fold splits from the Kaldi x-vector recipe (Przy- bocki & Martin, 2001) where Part1 is used for training and fine-tuning and Pa 2art2 for evaluation
6

---

Sortformer: A Novel Approach for Permutation Resollved Speaker Supervision in Speech-to-Text Systems
Table 1. DER results on speaker diarization. All evaluations include overlapping speech. Dataset name, number of speakers, and collar length are shown. Underlined values are the best-performing Sortformer evaluations. single Sortformer model trained for each loss type and evaluated on thrce datasets. Systems marked with a cross (f) involve a clustering phase and are not strictly end-to-end.
DH3
CALLHOME-
part2
CH109
Diarization Systems
Model Size
Post
7ispk S4,
ni 0.25 s
nsp==
nipp =4, 0.25
Proc.
0.0 s
0.25
0.25
(Park et al., 2022) tMSDD
31,1M
29.40
11.41
16.45
19.49
8.24
(Horiguchi al., 2022a;b) EEND-EDA
6.4M
15.55
7.83
12.29
17.59
(Chen et al., 2022) WWaYLM-L+EEND- (Horiguchi al., 2022b) (Chen et al., 2024) AED-EEND
317M
6.46
10.69
11.84
10.7M
13.64
7.11
11.88
14.37
11.6M
6.18
11.51
18.44
(Chen al., 2024) AED-EEND-EE
11.6M
6.93
11.92
17.12
X
18.33
7.28
11.57
18.80
5.66
Sortformer-
123M
V
17.04
6.94
10.30
17.52
6,89
X
17.88
7.42
12.68
19.42
9.08
Sortiormer-Sort-Loss 123M
V
17.10
6.52
10.36
17.40
10.85
X 16.28
6.49
10.01
14.14
6.27
Sorfformer-yybrid-Loss 123M
V
14.76
5.87
8.46
12.59
6.86
as DH3 in Table 1), CALLHOME- part2 (Przybocki & Mar- tin, 2001), and a two-speaker subset of 109 sessions from Callhome American English Speech (CHAES) (Canavan al., 1997), which we refer to CH109. In DIHARD3- eval, we include only sessions with four or fewer speakers.
only. We use 90-second long training samples and a batch size of 4. We use adam W (Loshchilov, 2017) optimizer with a learning rate of 10-4 and weight decay of 10-3 3 The minimum learning rate is 10-6 We use 2,500 steps of warmup where inverse square-root annealing is employed for learning-rate scheduling. A dropout rate of 0.5 is used for Transformer encoder layers and feedforward layers, and 0.1 is used for NEST encoders. We do not employ any special augmentation schemes such as SpecAugment (Park et al., 2019). All Sortformer models are trained on nodes of 8 NVIDIA Tesla V100 GPUs. Relative to pure Permu- tation Invariant Loss (PIL) training, which has an average epoch time of 1,020 seconds, our sorting-basee approach introduces minimal training time overhead. The average epoch duration increases by 0.22% to 1,022.28 seconds for a pure sort loss configuration, and increases by 2.26% to 1,043.1 seconds for hybrid loss setup.
5.1.2. DATA CLEANING
For the Fisher English Training Speech (Cieri et al., 2004), AMI (Kraaij et al., 2005), and NIST SRE 04-10 datasets (Doddington et al., 2000; Gondiiez-Rddriguez, 2014), we refined the speaker annotations by applying multilingual speech activity detection (SAD) model from an open-source toolkit and a pretrained Sortformer diarizer model to gain more accurate and tight boundaries where the minimal amount of silence exists between the onset and offset of speech and the segment start and end. For datasets such as AMI (Kraaij et al., 2005), ICSI (Janin et al., 2003), AISHELL-4 (Fu et al., 2021) where more than four speakers exist and/or session lengths are far longer than the 90-second limit, we truncated the dataset into 90-second short segments and retained only those segments containing less than or equal to four speakers.
5.2. Results on Speaker Diarization Task
Table shows the experimental results of diarization eval- uation on Sortformer diarizer. We evaluate three models trained with three different loss types: PIL only, Sort Loss only, and hybrid loss with = 0.5 in Eq. (12). We train the Sortformer model to handle up to 4 speakers, so we compare the popular neural diarizers that are reporting speaker-wise diarization error rate (DER) on each dataset. In addition, is crucial to remind that Sortformer is not individually fine-tuned on three evaluation datasets, un- like the systems in EEND-EDA (Horiguchi et al., 2022a), EEND-GLA (Horiguchi al., 2022b) and WayLM+EEND- VC (Chen al., 2022). We apply timestamp postprocessing that mitigates the errors generated from collar length and annotation style of the datasets. See Appendix B for details of the postprocessing. A noteworthy result from Table is
5.1.3. TRAINING SETUP
Our model based on the L-size NEST (Huang et al., 2025) encoder (115M parameters). We use 18 layers of Transformer (Vaswani al., 2017) encoder blocks with hidden size of 192, and two feed-forward layers with four sigmoid outputs on top of it (See Fig. 3). In total, including the NEST encoder, we evaluate a 123M parameter Sortformer model. We employ a two-stage training strategy on the Sortformer model: pretraining stage with both real and simulated data, and fine-tuning stage with real data
7

---

Sortformer: A Novel Approach for Cermuutaion-Resolveed Speaker Supervision in Speech-to-Tex Systems
Table 2. Evaluation results of Sortformer-MS-Camee
short segments from AMI test and CH109. Underlined numbers are the best performing setups without adapter. Except the baseline, the ASR encoder and decoder are fine-tuned in all systems.
Model
Train
Infer Speaker
Diar.
System Index
Obj. Level
Param.
Speaker
Model
Adapter Dim.
AMI-test ( 4-spks) WER
CH109 (2-pkss)
Size
Supervision
Supervision
Fine-tune
cpWER
WER
cpWER
baseline
170M
26.93%
21.81%
1
word
170M
19.67%
32.94%
18.57%
24.80%
2
word
293M
Sortformer
Sortformer Sortformer
x
20.08%
28.17%
18.65%
22.22%
3
word
293M
Sortformer Ground Truth
19.47%
32.74%
19.53%
26.97%
4
word
293M
Sortformer
19.48%
26.83%
18.74%
24.39%
5
segment 1.12B
Sortformer Sortformer
Sortformer
X
2556
18.58%
28.59%
17.74%
22.19%
6
word 1.12B
Sortformer
X
256
18.04%
26.71%
16.46%
21.45%
Table 3. Comparison of WER on the LibriSpeechMix dataset. Evaluations marked with a cross (f) are tested on audio mixtures with fixed delay for each speaker in the dataset.
WER (cpWER) (Watanabe et al., 2020) for real-life multi- speaker recording experiments. See Appendix for detailed descriptions of the evaluation metries.
WER
Param. Spk. Size
For comparative studies, we evaluate our model Lib- riSpeechMix (Kanda et al., 2020b), which is the most popular artificial audio mixture dataset for testing harsh overlap speech for multi-speaker ASR systems. We fol- low the train, validation, and test set split as described in (Kanda et al., 2020b) and created 2M audio mixtures from the LibriSpeech (Panayotov et al.. 2015) corpus. For the LibriSpeechMiix dataset, WER is reported following the methodology described in (Kanda al., 2020b;a).
ASR Systems
Spv.
1mix
2mix
3mix
Canary ASR
170M
x
2.19
21.37
48.71
(Puvvada et al., 2024)
IB
X
1.65
20.49
47.32
SOT-ASR
135.6M
X
4.6
11.2
24.0
(Kanda et al., 2020b)
SOT-ASR-SQR
135.6M
X
4.2
8.7
20.2
(Kanda et al., 2020a)
DOM-SOT
33M
x
5.17
5,56t
9.96t
(Shi al., 2024)
MT-LLM
(Meng et al., 2025)
8.4B
/
2.3
5.2
10.2
5.3.2.
TRAINING SETUP
MMS-Canayy
170M
x
2.74
6.55
12.14
For the real-life multi-speaker recording experiments, we build upon the Canary architecture (Puvvada et al., 2024), extending its ca capabilities to process multi-speaker input. We use the 170M variant for fine-tuning experiments and the 1B model for our adapter experiments following the adapter approach outlined in (Wang et al., 2024). For all these experiments, a single NVIDIA RTX 6000 Ada GPU is used.
Sortformer-MS-Canal
293M
2.26
4.61
9.05
that Sort Loss alone achieves performance comparable to that of the traditional PIL-trained model. Because Sort Loss offers competitive training signal, combining it with PIL in a hybrid loss allows the model to leverage strengths from both, leading to performance that surpasses models trained with either loss alone.
We train the Canary- models for 50K steps on the multi-speaker ASR training data blend, using a batch size of 64. Both the Fast-Conformer encoder and Transformer decoder parameters are fully fine-tuned. Speaker informa- tion is integrated through the Sortformer model, whose output is combined with the ASR encoder embedding via a sinusoidal kernel. For the Canary- model, all other model parameters are frozen, and only the adapter parameters in the encoder and decoder are learned over 75K updates on the multi-speaker ASR training data blend, starting from random initialization. All models are trained using the AdamW (Loshchilov, 2017) optimizer, with weight decay of 10-3 inverse square root annealing, a warm-up up of 2,500 steps, a peak learning rate of 3.10-4, and a minimum learning rate of 10-6.
5.3. Multi-speaker ASR Training Data
1. DATASETS
The training dataset used for real-life multi-speaker record- ing experiments includes the AMI (Kraaij et al., 2005) Individual Headset Mix (IHM) train split, which has been used in previous research. It also includes the ICSI (Janin et al., 2003) dataset, the DipCo dataset (Segbroeck et al., 2020), and a 30K segment subset of the Fisher English Training Speech Part 1 and 2 dataset. The first three sets collectively contain 138 hours of multi-speaker speech, with up to four speakers per sample. The Fisher dataset comprises 2,000 hours of two-speaker data. To address the speaker data imbalance, 30K samples are randomly selected from the Fisher dataset and incorporated into our four-speaker data blend. The resulting combined training corpus consists of 230 hours of multi-speaker audio, with a maximum of four speakers per sample. We report word error rate (WER) and concatenated caiimum-permutttio
For the LibriSpeechMix experiments, we use System 2 from Table with the 170M ASR model, the model without an adapter while using the SOT-style speaker token objective to test the effectiveness of the SOT approach. First, we fine-
8

---

Sortformer: A Novel Approach for Commutaatin-Resollvee Speaker Supervision in Speech-to-Text Systems
tune the Sortformer model from the evaluation in Table 2 the 960-hour LibriSpeechMix training dataset. Then we run I 180K steps of fine-tuning of the ASR model while keeping the Sortformer model frozen, obtaining the Sortformer-Me Canary model. As baseline, we also fine-tune the Canary- 170M model on LibriSpeechMix data for the same number of updates without any speaker supervision from Sortformer. This model is referred to as MS-Canary Table 3.
30% for 2-mix and 25% for 3-mix, while showing minor degradation on single-speaker 1-mix audio when compared to the baseline Canary-170M model.
5.4.3. RUNTIME PERFORMANCE
Runtime evaluations were performed on a single NVIDIA RTX A6000 Ada GPU, The stand-aloue Sortformer diariza- tion model was benchmarked on the LibriSpeechMix test- 3mix dataset (42,514.9 14.94s 4.9s total audio, using 10-run averages). In multi-speakeer ASR tasks (batch size: 100), integrating Sortformer supervision with the MS-Canary system (170M parameters) increased processing time by a mere 0.78%, increasing from 297.891s to 300.2135 for the Sortformer- MS- Als-Caary system (293M parameters).
5.4. Results on Multi-speaker ASR
5.4.1. ABLATION STUDY DESIGN
We perform an ablation study on real-life multi-speaker recordings to gauge the contribution of each component. As baseline system, we use the CaMary-170M (Puvvada et al., 2024) ASR model in its original form without any fine-tuning. Table 2 shows the various setups we evaluate to show the contributions of each component. The baseline system is a single-speaker Canary- 17MM (Puvvada et al., 2024) model that is not trained on the multi-speaker ASR dataset. The original Canary- 170M model does not have speaker tokens; therefore, epWER is not calculated. See Appendix D for detailed description about WER calculation. The baseline system shows how challenging the evaluation set is for the vanilla ASR model. System 1 is the most primitive model where neither speaker supervision nor adapters are used. System and System 3 are the models where Sortformer diarization module is plugged in while Sortformer model weights are frozen in System and fine- tuned in System 3. Finally, System 4 is system trained with ground-truth speaker labels fed through a speaker kernel but Sortformer is used as speaker supervision during inference. System 5 and System 6 are the multi-speaker ASR models trained with the adapter technique in (Wang et al., 2024), with the Canary- IB model. Systems 5 and 6 show the best-performing setup in both segment- level and word-level objectives. However, System 5 not only shows degradation in segment- objectives, but we also observe this decline across all types of settings and datasets.
6. Conclusion
In this paper, we propose Sortformer, encoder-type diarization model designed for integration with speech-to- text systems. By learning an arrival-time sorting mech- anism, Sortformer enables permutation-resolved speaker supervision, thereby supporting cross-entropy loss-based training and unifying multi-speaker ASR frameworks with the principles of monaural ASR frameworks. In addition, we show that combining Sort Loss with PIL as a hybrid loss improves stand-alone diarization performance when trained solely on PIL. Finally, we demonstrate that the proposed framework can improve multi-talker ASR benchmarks to a system without speaker supervision. Future work will explore streaming systems, target-speaker ASR features, and multi-task capabilities such as translation and summa- rization. We hope our proposed work serves as accessible baseline to inspire further research in multi-speaker ASR.
Impact Statement
This research introduces Sortformer, a novel encoder-based diarization model that addresses the challenging speaker permutation problem in multi-speaker STT systems. By employing Sort Loss function based on speaker arrival times, Sortformer enables permutation-rreoovee speaker supervision. This innovation streamlines the integration of speaker tagging, allowing STT models to be trained with standard cross-entropy loss, similar to simpler mono- speaker systems, and reduces complex annotation needs. Experiments confirm improved diarization performance and transcription accuracy. The broader impact lies in making advanced multi-speaker ASR more accessible and easier to deploy. We foresee Sortformer accelerating the development of more versatile and accurate speaker-aware applications, paving the way for seamless integration into foundational STT models and L.Mss, thus benefiting a wide range of interactive technologies.
5.4.2. COMPARATIVE EVALUATION
For evaluation on the LibriSpeechMix benchmark, shown in Table 3, we compare our proposed model with other top- performing systems in the literature that report WERs across all three mixture sets with the same model. See Appendix D for the WER calculation in Table 3. Senfformer--S-CCanaae achieves the best performance on multi-talker datasets (2- mix and 3-mix), while the baseline single-speaker Canary ASR models (170M and 1B) show slightly better perfor- mance on 1-mix (single speaker). Furthermore, the improve- indicates that we can successfully integrate 1 123M- -aaammmree Sort- former model, yielding relative error rate reduction of
ment from MS-Canary to Sortformer-MS-Caree
9

---

Sortformer:
A Novel Approach for Pormutation- Speaker Supervision in Speech-to-Text Systems
References
Doddington, G. R., Przybocki, M. A., Martin, A. F., and Reynolds, D. A. The nist speaker recognition evaluation- overview, methodology, systems, results, perspective. Speech communication, 31(233)) 31(2-3):225-2555 200..
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next eext-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2623-2631, 2019.
Fu, Y., Cheng, L., Lv, S., Jv, Y., Kong, Y., Chen, Z., Hu, Y., Xie, L.., Wu, J., Bu, H., Xu, X., Du, J., and Chen, J. Aishell-4: An open source dataset for speech enhance- ment, separation, recognition and speaker diarization in conference scenario. In Proc. INTERSPEECH, pp. 3665- 202.heeccch221119999
Barker, J., Watanabe, S., Vincent, E.,, and Trmal, The Fifth CHiME Speech Separation and Recognition Challenge: Dataset, Task and Baselines. In Proc. INTERSPEECH, pp. 1561-1565, 2018.
3669, 2021. doi:
Fujita, Y., Kanda, N., Horiguchi, S., Xue, Y., Nagamatsu, K., and Watanabe, End-to-End Neural Speaker Diarization with
Barker, J. P., Marxer, R., Vincent, E., and Watanabe, S. The chime challenges: Robust speech recognition in everyday environments. New era for robust speech recognition: Exploiting deep learning, pp. 327-344, 2017.
In Proc. ASRU, pp. 296-303, 2019.
Fujita, Y., Watanabe, S.,, Horiguchi, S., Xue, Y., Shi, J., and Nagamatsu, K. Neural speaker diarization with speaker- wise chain rule. arXiv preprint arXiv:2006.01 7766, 2020.
Boeddecker, C., Heitkaemper, J., Schmalenstroeer, J., Drude, L., Heymann, J., and Kaeb-Umbacke R. Front-end processing for the chime-5 dinner party scenario. In Proc. CHiME 2018, pp. 35-40, 2018 8. doi: 10.21437/CHiME. 2018-8.
Ronzalez-Rodriguez, Evaluating automatic speaker recog- nition systems: An overview of the nist speaker recogni- tion evaluations (1996-2014). Loquens, 1(1)=0007-e007, 2014.
Canavan, A., Graff, D., and Zipperlen, G. CALLHOME American English speech, 1997. URL https: catalog. ldc. www.n.edu/I..e....... 2025-05-23.
Horiguchi, S., Fujita, Y., Watanabe, S., Xue, Y., and Naga- matsu, K. End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attrac- tors. In Proc. INTERSPEECH, pp- 269 273, 2020. doi: 10.2 00224437/ntesspecch220000
Accessed:
Canseco-Rodriguez L., Lamel, L., and Gauvain, J.-L. Speaker Diarization from Speech Transcripts. In Proc. ICSLP, pp. 3-7, 2004.
Horiguchi, S., Fujita, Y., Watanabe, S., Xue, Y., and Garcia, P. Encoder-decoder based attractors for end-to-end neural diarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 10:1499-1507, 2022a.
Chen, S., Wang, C.. and et al., Z. C. WavLM: Large- Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 16(6), 2022.
Horiguchi, S., Watanabe, S., García, P., Takashima, Y., and Kawaguchi, Y. Online neural diarization of unlimited numbers of speakers using global and local attractors. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 31:706-720, 2022h.
Chen, Z., Han, B., Wang. S., and Qian, Y. Attention- based encoder-decoder end-to-end neural diarization with embedding enhancer. IEEE/ACM Transactions Audio, Speech, and Language Processing, 32: 16:1666-1649, 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Ceaumeter-Efficieee Transfer Learning for NLP. In Proc. ICML,, pp. 2790-2799, 2019.
Chung. J. S., Huh, J.. Nagrani, A., Afouras, T., and Zisser- man, A. Spot the conversation: Speaker diarisation in the wild. In Proc. INTERSPEECH, pp. 299-303, 2020. doi: 10.21 0,21144///eerpeecch220.. 2020-
Huang, H., Park, T., Dhawan, K., Medennikov, I,, Puvvada, K. C., Koluguri, N. R., Wang, W., Balam, J., and Gins- burg, Nest: Self-supervised fast conformer as all- purpose seasoning speech processing tasks. In Proc. ICASSP, pp. 1-5. IEEE, 2025.
Cieri, C., Miller, D., and Walker, K. The Fisher Corpus: A Resource for the Next Generations of Speech-to-text. In Proc. LREC. pp. 69-71, 2004.
Cornlll, S., Wiesner, M. S., Watanabe, S., Raj, D., Chang, X., Garcia, P.. Masuyam, Y., Wang, Z.- Squartini, S., and Khudanpur, S. The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios. In Proc. CHiME 2023, pp. 1-6, 2023.
Janin, A., Baron, D.,. Edwards, J. Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E.,, Stolcke, A., et al. The icsi meeting corpus. In Proc. ICASSP, volume 1, pp. I-I. JEEE, 2003.
10

---

Sortformer: A Novel Approach for Permutation-Resolvee Speaker Supervision in Speech-to-Text Systems
Kanda, N., Gaur, Y., Wang, X., et al. Joint Speaker Count- ing, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers. In Proc. INTERSPEECH, 2020a.
transcribe speech in multi-talker scenarios with versatile instructions. In Proc. ICASSP, pp. 1-5. IEEE, 2025.
Niu, S., Wang, R., Du, J., Yang, G., Tu, Y., Wu, S., Qian, S., Wu, H., Xu, H., Zhang, X., Zhong, G., Yu, X., Chen, J., Wang, M., Cai, D., Gao, T., Wan, G., Ma, F,, Pan, J., and Gao, J. The uste-nereslip systems for the chime-8 notsofar- challenge. In CHiME 2024, pp. 31-36, 2024. doi: 10.24477/CHMME2022477
Kanda, N.. Gaur, Y., al. Serialized Output Training for End-to-End Overlapped Speech Recognition. In Proc. INTERSPEECH, pp. 2797-2801, 2020b.
Kanda, N., Chang, X., and et al., Y. G. Investigation End- to-End Speaker-Attributed ASR for Continuous Multi- Talker Recordings. In Proc. SLT, 2021.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain audio books. In Proc. ICASSP, pp. $206-5210 IEEE, 2015.
Kanda, N., Wu, J., et al. Streaming Multi-Talker ASR with Token-Level Serialized Output Training. In Proc. INTERSPEECH, pp. 3774 3778, 2022a.
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. Specaugment A simple data augmentation method for automatic speech recognition. In Proc. INTERSPEECH, pp. 2613-2617, 2019. doi: 10.21 10.21447//meerppechh001999
Kanda, N., Xiao, X., and et al.. Y. G. Transcribe-to-Diie -iiariie:: Neural Speaker Diarization for Unlimited Number of Speakers Using End-to- -nd Speaker-Attribute ASR. In Proc. ICASSP, pp. 8082-8086, 2022b.
Park, T. J., Koluguri, N. R., Balam, J., and Ginsburg, B. Multi-scale speaker diarization with dynamic scale weighting. In Proc. INTERSPEECH, pp. 5080-5084, 2022. doi: (0.21457/hnerspeeehh2222
Kolbæk, M. Yu, D., Tan, Z.-H., and Jensen, J. Mul- titalker speech separation with utterance-level permuta- tion invariant training of deep recurrent neural networks. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing. 25(10):101111913 2017.
Park, T. J.. Huang, H., Hooper, C., Koluguri, N. R., Dhawan, K., Jukié, A., Balam, J., and Ginsburg. B. Property- multi-speaker data simulation: A probabilistic modelling technique for synthetic data generation. In CHIME 2023, pp. 82-86, 2023. doi: 10.21437/CHHIME22233-U
Kraaij, W., Hain, T., Lincoln, M., and Post, W. The ami meeting corpus. In Proc. International Conference on Methods and Techniques in Behavioral Research, pp. 1-4, 2005.
Przybocki, M. and Martin, A. 2000 NIST Speaker Recog- nition Evaluation, 2001. URL https://catalog: ldc 05-23.
Landini, F., Profant, J., Diez, M., and Burget, L. Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: theory, implementation and analysis on stan- dard tasks. Computer Speech & Language, 71:101254, 2022.
Accessed: 2025-
Puvvada, K. C., Zelasko, P., Huang, H., Hrinchuk, O.,
Koluguri, N. R., Dhawan, K., Majumdar, S., Rastorgueva, E., Chen, Z., Lavrukhin, V., Balam, J., and Ginsburg, B. Less is more: Accurate speech recognition & translation without web-scale data. In Proc. INTERSPEECH, pp. 3964-3968, 2024. doi: 00214457/nersyeccc.222--222
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1111.0510/ 2017.
Medennikov, I., Korenevsky, M., Prisyach, T., Khokhlov, Y., Korenevskaya, M., Sorokin, I., Timofeeva, T., Mitro- fanov, A., Andrusenko, A., Podluzhny, I., Laptey, A., and Romanenko, A. Target-speaker voice activity detection: novel approach for multi-speaker diarization in dinner party scenario. In Proc. INTERSPEECH, pp. 274 278, 2020a. doi: 10.21457/nterspeechh2222 1602.
Qian, Y., Chang, X., and Yu, D. Single-channel Multi-talker Speech Recognition with Permutation Invariant Training. Speech Communication, 104:1-11, 2018.
Ryant, N., Church, K., Cieri, C., Du, J., Ganapathy, S., and Liberman, M. Third DIHARD challenge evaluation plan. arXiv preprint arXiv:2006.0584
Medennikov, I., Korenevsky, M., Prisyach, T., Khokhlov, Y., Korenevskaya, M., Sorokin, I., Timofeeva, T., Mitrofanov, A., Andrusenko, A., Podluzhny, I., et al. The STC System for the CHiME-6 Challenge. In Proc. CHiME 2020, 2020b.
2020.
Segbroeck, M. V., Zaid, A., Kutsenko, K., Huerta, C., Nguyen, T., Luo, X., Hoffmeister, B., Trmal, J., Omol- ogo, M., and Maas, R. DiPCo - dinner party cor- pus. In Proc. INTERSPEECH, pp. 434-436, 2020. doi: 10224477/merspeech.220-022 202002220....028000
Meng, L., Hu, S., Kang, J., Li, Z., Wang, Y., Wu, W., Wu, X., Liu, X., and Meng, H. Large language model can

---

Sortformer: A Novel Approach for Permutaaion-Resolyee Speaker Supervision in Speech-to-Text Systems
Shafey, L. E., Soltau, H., and Shafran, Joint Speech Recognition and Speaker Diarization via Sequence Trans- duction. In Proc. INTERSPEECH, pp. 396-400, 2019.
Yu, D., Chang, X., and Qian, Y. Recognizing Multi-Talker Speech with Permutation Invariant Training. In Proc. INTERSPEECH, pp. 2456-2460, 2017a.
Shi, Y., Li, L., Yin, S., Wang, D., and Han, J. Serialized output training by learned dominance. In Proc. INTER- SPEECH, pp. 712-716, 2024. doi: 10.21457/mneesspechh. 2024-1710.
Yu, D., Kolbæk, M., Tan, Z.-H., and Jensen, Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In Proc. ICASSP, pp. 241- 245. IEEE, 2017b.
Takashima, Y., Fujita, Y., Watanabe, S., Horiguchi, S., Gar- cía, P., and Nagamatsu, K. End-to-end speaker diarization conditioned on speech activity and overlap detection. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 849-856. IEEE, 2021.
mollfovvá K., Delcroix, M., Kinoshita, K., Ochiai, T, Nakatani, T., Burget, L.,, and Černocký, Speakerbeam: Speaker aware neural network for target speaker extrac- tion in speech mixtures. IEEE Journal of Selected Topics in Signal Processing, 13(4):800-8) -814, 2019.
Tranter, S. E., Yu, K., Reynolds, D. A., Evermann, G., Kim, D. Y., and Woodland, P. C. An Investigation into the Interactions between Speaker Diarisation Systems and Automatic Speech Transcription. CUED/F-INEENG/TR 464, 2003.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention all you need. Advances in neural information processing systems, 30, 2017.
Von Neumann, T., Boeddeker, C. B., Delcroix, M., and Haeb- Umbach, R. McetEval: A toolkit for computation of word error rates for meeting transcription systems. In CHiME 2023, pp. 27-32, 2023. doi: 10.21437CCHMME.2223-4
Wang, W. and Li, M. Incorporating end-to-end framework into target-speaker voice activity detection. In Proc. ICASSP. pp. 8362-8366. IEEE, 2022.
Wang, W., Cai, D., Lin, Q., Yang, L., Wang, J., Wang, J., and Li, M. The DKU-DukcECE-Lenovo system for the diarization task of the 2021 VoxCeleb Speaker Recognition Challenge. arXiv e-prints, pp. arXiv-2109, 2021.
Wang, W., Dhawan, K., Park, T., Puvvada, K. C., Meden- nikov, Majumdar, S., Huang, H., Balam, J., and Ginsburg, B. Resource-efficient adaptation of speech foundation models for multi-speaker asr. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 1224- 1231. IEEE, 2024.
Watanabe, S., Mandel, M., Barker, J., et al. CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings. In Proc. CHiME 2020, 2020.
Yang, G., He, M., Niu, S., Wang, R., Yue, Y., Qian, S., Wu, Du, J., and Lee, C.-H. Neural speaker diarization using memory-aware multi multi-speaker embedding with sequence- to-sequence architecture. In Proc. ICASSP, pp. 11626- 11630. IEFE, 2024.
12

---

Sortformer: A Novel Approach for SramuutationResollve Speaker Supervision in Speech-to-Text Systems
A. Data Cleaning
In our proposed multi-speaker ASR model, speaker tokens are generated by the model to predict the corresponding speaker label for each word. During training, we clean the data according the following rules:
We segment the long-form audio into shorter segments, each ranging between 10 and 20 seconds.
Words the transcripts are sorted based on their arrival time, even when they overlap, as illustrated in Figure 5. Overlapping speech results in more frequent speaker changes.
If word-level timestamps are missing, we simulate them from segment-level timestamps. Specifically, we split the words into syllables and assume that each syllable has the same duration. The timestamp of each word then estimated based on the number of syllables of each word and the average duration of each syllable.
Speaker tokens are assigned based on the arrival time of each speaker, starting from <spkO>, followed by <spk1>, <spk2>, and so on.
Samples with more than 1-second overlap at the beginning or end are excluded.
Samples where the first speaker only has or two filler words at the beginning excluded.
B. Postprocessing of Speaker Diarization Segments (Timestamps)
We upply timestamp postprocessing that mitigates the errors generated from collar length and annotation style differences from multiple datasets. Our postprocessing step consists of:
1. Onset threshold: The threshold for detecting the beginning of speech.
2. Offset threshold: The threshold for detecting the end of speech.
3. Onset padding: The duration added at the beginning of each speech segment.
4. Offset padding: The duration added the end of each speech segment.
5. Minimum duration (on): The minimum duration required to retain a speech segment, used to remove short non-speech segments.
6. Minimum duration (off): The minimum duration required to retain a non-speech segment, used to remove very short speech segments.
The parameters are tuned on two different splits of datasets: Set-A on DIHARD3 (Ryant et al., 2020) Dev split and Set-B on CALLHOME Partl (Przybocki & Martin, 2001). Then Set-A parameters are applied to DIHARD3-eval, and Set-B is applied to CALLHOME Part2 and CH109. This speaker diarization postprocessing scheme is inspired by the postprocessing procedure in (Medennikov et al., 2020a). We optimize these floating-point postprocessing parameters with Optuna( (Aiiba et al., 2019) software.
C. Word Timestamp Approximation
We employ a syllable-based word timestamp approximation technique for word-level objectives. After end-to-end ASR models gained popularity, such RNNT-based models and attention encoder-decoder (AED) models, the ASR training process no longer requires word-by by-rrd timestamp (alignment of words). Thus, securing speech datasets with word timestamps is difficult, and it becomes even more challenging when it comes to multi-speaker conversations because overlaps make it hard to be aligned with the forced aligners. Therefore, we propose a method to train model without providing the model with timestamps by approximating the timestamps:
L lend Lata
(15)
E.
7
word
d; 0, + N
=8,, 8, + An.]
(16)
13

---

Sortformer: A Novel Approach for Permutation- Resollved Speaker Supervision in Meeech-to-Text Systems
Original Segment-level Transcript
California does experience rain, but it's not always frequent.
yeah
that's right
Syllable-level Break Down
cal
for
nia
does
ex
pe
ence
rain
but
It's
not
al
ways
fre
quent
yeah
that's
right
Sorted Serialized Transcript
<spk0> california <spk0> does <spkO> experience <spk1> yeah <spk0> rain <spkO> but <spk0> not <spk1> that's <spk0> always <spk1> right <spk0> frequent
Figure Process of generating pseudo word timestamp and sorted serialized transcript.
where N is the total number of syllables in segment, 6 is the start time of the i-th word, bast and fend are start and end time of the segment. represents the segment length, À N denotes the average syllable duration (speaking rate), defined the segment length divided by the total number of syllables within that segment. This rate, denoted by X, provides average measure of how quickly syllables are spoken during the segment. Hence, À value is used to normalize the segment length and derive the start time and end time of each word. T, represents the timestamp of the i-th word, and denotes the number of syllables in the i-th word.
Figure shows how ford-timmstsmps are calculated in the absence word-by-word timestamps. We split the words into syllable levels and assume that each syllable has the same duration. Thus, the timestamp of each word then estimated based the number of syllables of each word and the average duration of each syllable. The proposed wird-timestamp approximation ensures the approximated word timestamps are comparable to the original word timestamps.
D. Word Error Rate (WER) Calculation
When measuring the accuracy of multi-speaker ASR models, we need to assess both speaker tagging accuracy and word error rate (WER) itself, which comprises insertion, deletion, and substitution errors. However, in some of our experiments, we evaluate monaural ASR on multi-speaker recordings or artificial audio mixtures. Additionally, in Table 3, we report multi-speaket ASR system results using cpWER (Watanabe et al., 2020) for real-life multi-speaker recordings, well as method referred to as WER in previous studies(Kan) Kanda et al., 2020b:a) that introduced the LibriSpeechMix dataset. Furthermore, we report WERs on the monaural ASR systems tested on multi-speaker datasets. The WER values reported in the literature (Shi et al., 2024; Meng et al., 2025) could lead to misconceptions, as different groups of authors exhibit discrepancies in their descriptions. Here, we clarify the schemes we use to calculate WER values.
1. cpRER (Concatenated Pernuuttttnntttioonnnnn WER): cpR multi-speaker ASR multi-speaker recordings: The WER metric evaluates speech recognition and diarization jointly through the following three steps:
Concatenation:
Merging all utterances per speaker in both the reference and hypothesis.
Permutation Scoring: Computing WER across all possible speaker permutations (e.g.. 24 permutations for 4 speakers).
Optimal Selection: Selecting the permutation with the lowest WER as the final score.
As proposed in (Watanabe et al., 2020), epWER inherently captures diarization errors and is used as the primary evaluation metrie. Additionally, utterance-level emror breakdowns reported for detailed analysis.
2. WER for monaural ASR on multi-speaker recordings (in Table 2): For the baseline model, Canary 170M, we use word timestamps from annotations or forced-alignment results to sort the words based on their start times. We then evaluate WER the same manner as standard monaural ASR evaluation.
14

---

Sortformer: A Novel Approach for Cermutation-Resolvee Speaker Supervision in Speech-to-Text Systems
3. WER for multi-speaker ASR multi-speaker recordings (in Table 2): For all systems in Table 2, we remove the speaker tokens from both hypothesis and reference and measure WER values from the speaker-token removed hypothesis and reference pairs.
4. WER of monaural ASR for the LibriSpeechMix dataset (in Table 3): Baseline Canary ASR models (170M and IB) fall into this category. use the optimal reference combination (ORC) WER from the open-source toolkit (von Neumann al., 2023). In ORC WER, speaker labels are ignored, and WER measured based solely on hypothesis and reference transcript pairs. Although speaker labels are ignored, WER in SOT approaches still reflects speaker tagging accuracy, since SOT concatenates all word outputs for each speaker.
5. WER of Multi-Speaker ASR for the LibriSpeechMix Dataset (in Table 3): The multi-speaker (MS) version of Canary falls into this category. use cpWER from the open-source toolkit (von Neumann et al., 2023). This aligns with the WER definitions (Kanda et al., 2020a; Shi al., 2024), where a mapping that delivers the lowest WER between predicted speakers and the ground-truth speakers is found, and then the WERs for all speakers summed.
E. Permutation Properties
E.1. Definitions
Here are definitions and properties needed for clearly describing the permutation equivariance in the multi-head self-attention (MHA) mechanism in Transformer architectures. Note that we assume no positional embeddings are used on any input matrix or embedding.
Definition E.1 (Permutation Function T). Let E Z+ be a positive integer. A permutation is defined as any bijective transformation of the finite set {l,..., n into itself. Thus, permutation is a function æ {1,2, .1,,,...,1. - { n}
such that, for every integer E {l....,,7 n. there exists exactly one integer j E {1...., [2} for which
x 7() =
(17)
... n be a finite set of n integers. A permutation is a bijective function from S to itself, т S + S. The permutation matrix Ps E IRxx" corresponding to the permutation æ is defined such that its entry in the i-th row and j-th column, denoted as (P=)e,. is given by:
Definition E.2 (Permutation Matrix P+). Let 5 f1,2,...
(1 if xj)
(Pa)
(18)
otherwise
This can also be written using the Kronecker delta symbol as follows:
(Pa) = 61,=()
(19)
The permutation matrix PT can be expressed by employing standard basis vectors. Let be the k-th standard basis vector in IR"" (a column vector with a 1 in the k-th position and Os elsewhere). The permutation matrix P, can be constructed by arranging the standard basis vectors en(j) as its columns:
P. = (1)
CT(2)
(T(n)
(20)
This means the j-th column of Pz is the vector Cx(s)-
Definition E.3 (Spatial Permutation). Given spatial permutation æ, the transformation Tμ of feature map X E Boxd is given by:
Tx(X) P.X
(21)
where P. E RRax is the permutation matrix.
15

---

Sortformer: A Novel Approach for Soimuutatio-Resollvve Speaker Supervision in Speech-to-Text Systems
E.2. Properties
Property 1 (Permutation Invariance). Let X' be and be a codomain. function f 2 + y said be permutation invariant if, for any subset {Ii,,... k S X' and any permutation of the indices {1, (.., M, the following holds:
f (mi, ( FM) f (o (T(M))
(22)
This property means that the output of is independent of the order of the elements in its input set. In matrix form, operator A [gdxn - [R is spatially permutation invariant if:
A(T,(X)) - A(X),
(23)
for any input X and any spatial permutation
Property 2 (Permutation Equivariance). Let ı be a permutation of {1, [1,2,..., n.. and let f IR" - R function. Then, is said to be permutation equivariant for every input x = (21, 22, ..., (n) IR", it holds that
[(x(x)) (x(()),
(24)
where xxx represents the permutation of the components of according to æ, and x (f((x)) represents the permutation of the components of (x) in the same manner. We can describe this property in matrix form as follows: A function F Raxd + Raxd is permutation equivariant if for any input matrix X Raad and any permutation matrix Pa E Rx (representing permutation of the items/rows), the following holds:
F(P,X) PxF(X)
(25)
F. Permutation in Multi-head Self Attention
F1. Multi-head Self Attention Structure
The MHA architecture proposed in (Vaswani et al., 2017) is a key component of the Transformer model. Let be the input (sequence) length, d the model (embedding) dimension per token, and h the number of parallel attention heads. We first project the inputs into query, key, and value matrices Q, V e gaxd, where Q = xw K = XW and V = XWV Subsequently, we apply scaled dot-product attention independently in each the heads (of width da = d/h), yielding head outputs O....., O₂ Raxdx, Finally, we concatenate and linearly re-pre re-project:
MHA K, V) Concat .... On,) w
(26)
where D E dx) is trainable matrix. Each i-th head, also referred to as self-attention is then defined as:
O Attention Q, K. V)
(27)
Attention XW XW. XW!)
(28)
XW (XW))
softmax
XW.
(29)
Vdle
where Wº w WV gaxds are trainable parameter matrices.
F.2. Proof of Permutation Equivariance
Proof. Permutation equivariance means that you permute the input, the output should be permuted in the same way. Mathematically, for function and permutation matrix P, this is expressed as:
f(Px) PP
(30)
Let PQ, PK, PV be the permuted version of the query, key, and value matrices Q K, V E Raxd,
16

---

Sortformer: A Novel Approach for Ieciunttttio-Resollvee Speaker Supervision in Opeech-to-Text Systems
PQ, PKK PV
(31)
where a permutation matrix. The attention mechanism with the permuted inputs can be described as:
PXW (PXW:)) dd PXW X PT dd
O = softmax
PXWV
(32)
softmax
PXW.
(33)
Using the property of the softmax function:
XW (WK)
P. softmax
pd P"
(34)
dd
Since PT p= I (the identity matrix, because permutation matrix):
XW (W((WK) X
P. softmax
XW
(35)
dp
XW XW
P. softmax
XWV
(36)
Vdk
Hence:
O PO;
(37)
Concatenating across all heads:
Concat Concat(O/,...... Of...... 0²,) Concat( PO,)
(38)
p. Concat .I.....00 ..)
(39)
Finally, the equation can be arranged as:
MHA(PQ, PK, PV) = P. MHA(Q, K, V)
(40)
This holds the definition of (30) and shows that the MHA mechanism is permutation equivariant, as the output under any permutation of the inputs simply the same permutation applied the original output.
17


Reviews:
[{"review_id": "no2JbGlnTq", "reviewer": "Reviewer_cJRr", "summary": "The paper proposes Sortformer, an encoder-based neural model designed for permutation-resolved speaker diarization integrated into speech-to-text (STT) systems. Its core innovation is the introduction of a Sort Loss that addresses the traditional permutation invariance problem in speaker diarization by sorting speakers based on their arrival time, rather than solely relying on permutation invariant loss (PIL). Additionally, the paper introduces a novel multi-speaker speech-to-text architecture which embeds sorted speaker labels using sinusoidal kernel functions directly into the encoder's hidden states. Experimental results demonstrate that using Sort Loss, especially in combination with traditional permutation invariant loss (PIL), boosts diarization and multi-speaker transcription accuracy. The authors claim that this framework simplifies the training of multi-speaker ASR to be as straightforward as mono-speaker ASR, facilitating integration into multimodal large language models (LLMs).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Response to Claims and Evidence\n\n### 1-1. Robustness\n\nWhile the reviewer may find our evaluation insufficient, we believe the robustness of Sort Loss is demonstrated through the datasets we used up to a certain degree:\n\n- **DIHARD3** (\"Diarization is Hard\") covers 11 challenging domains, including noisy and reverberant scenarios (e.g., restaurants, street interviews, web videos). To our knowledge, no other benchmark is more comprehensive for noisy and reverberant speaker diarization.\n\n- **LibriSpeechMix** is specifically designed for overlapping speech evaluation, with overlap rates ranging up to 90%, including three-speaker overlaps. It is widely used in multi-talker ASR research.\n\n- We selected widely accepted benchmark datasets to ensure reliable comparisons. Evaluating on untested datasets would compromise result validity. We encourage the reviewer to reconsider after reviewing the datasets' features.\n\n### 1-2. Comment on NLU Task Evaluation\n\nWe do not think that evaluating multi-speaker ASR on NLU tasks is a critical omission:\n\n- In our target applications (e.g., meeting transcriptions, patient-doctor dialogues, and real-time speaker-tagged transcriptions), the primary output (text with speaker labels) is often consumed directly. Thus, metrics like DER and cpWER are more relevant.\n\n- Designing such an evaluation would require extensive methodology descriptions, which is beyond the scope of this 8-page paper. While NLU-based assessment is an interesting direction, we believe it warrants a separate study.\n\n\n\n\n## 2. Response to Experimental Designs Or Analyses\n\n> \"..LibriSpeechMix .. with fixed delays\":\n\nThe reviewer’s concern about “fixed delays” is incorrect. LibriSpeechMix is artificially mixed, but each session uses **randomized delays**. No fixed delays are applied.\n\n\n\n\n\n## 3. Response to Questions For Authors\n\n### 3-1. Runtime Evaluations\n\nInference time on stand-alone diarization model:\n\nDataset: LibriSpeechMix test-3mix (2,620 files, total duration: 42,514.9s) on an NVIDIA RTX A6000, with 10-run averages.\n\nStand-alone diarization (batch size=1, collar=0.0, overlap included eval):\n\n- Pyannote-Diarization-3.1 (most popular open-source diarization model):  \n  4m39.6s (RTFx=152.06), DER=0.2144, Speaker Counting Accuracy=0.4985\n\n- Proposed Sortformer (123M params):  \n  2m14.6s (RTFx=316.01), DER=0.1346, Speaker Counting Accuracy=0.9763\n\nMulti-speaker ASR (batch size=100):\n\n- MS-Canary (170M params): 297.891s  \n- Sortformer-MS-Canary (293M params): 300.213s  \n  → Only 0.78% runtime increase (x1.0078) from adding Sortformer supervision.\n\n### 3-2. Sorting Inaccuracies and Its Effect on Performance\n\nThe model does not make sorting errors; the model only makes errors in diarization:\n\n- The Sortformer model does not make \"sorting errors\". We have never seen an example where Sortformer generates speaker segments’ arrival time in the wrong order. If trained with enough data and time, assigning speaker index in arrival time order is an easy task for the model.\n\n- The error comes from “missed” or “false alarm” predictions—i.e., short segments missed or non-existing segments falsely detected. This type of error occurs in all diarization systems.\n\n- Therefore, the starting time of the first speech segments of each speaker (t1, t2, t3, t4) are always in order: t1 < t2 < t3 < t4, even if the diarization is wrong.\n\n- For this reason, we are not able to perform “sorting error-based analysis.” We can only evaluate diarization errors and cpWER.\n\n### 3-3. Scenarios Where Sorting-Based Mapping Could Be Inaccurate\n\n#### a. Training Phase\n\n- Table 1 shows performance gaps between Sort-Loss-only and Hybrid models, revealing sorting-related disadvantages. Both PIL and Sort Loss have imperfections, but they complement each other (acting as mutual regularizers).\n\n#### b. MS-ASR Training/Inference\n\n- Errors typically occur when speakers begin with very short utterances (1–2 words).\n- This increases cpWER through incorrect speaker assignments.\n- Note: Such errors affect al diarization and MS-ASR systems, not just ours.\n\n### 4. Relation to DOM-SOT and Comparison\n\n- \"Arrival time\" is not the only quantity that can be used to resolve permutations. Other options include: Total speaking time (DOM-SOT), End time, Speaking rate can be used.\n\n- However, only arrival time determines the permutation at the start and does not require future input. For example, using \"total speaking time\" requires input from start to end, unsuitable for streaming or divide-and-conquer scenarios.\n\n- DOM-SOT work is not tackling the same problem. DOM-SOT proposes a different loss criterion for multi-talker ASR but does not investigate supervision of a speaker diarization model.", "claims_and_evidence": "- The main claim—that Sort Loss resolves the speaker permutation ambiguity effectively—is well-supported through multiple experiments across standard diarization benchmarks (DIHARD3, CALLHOME, CH109, and LibriSpeechMix). The performance improvement when using hybrid loss (Sort Loss + PIL) is clearly demonstrated.\n- The paper clearly states and experimentally validates that the proposed system simplifies the integration with existing speech-to-text architectures, as indicated by improved results when combined with models like Canary-170M and Canary-1B.\n- Despite thorough experimental evidence, the paper does not clearly address the robustness of Sort Loss under highly complex, realistic scenarios involving noisy, reverberant, or strongly overlapping speech. Thus, claims about robustness and generalizability may be overstated or insufficiently explored.\n- It is not fully demonstrated whether improvements in diarization accuracy directly translate to significantly better downstream NLU tasks, such as conversation summarization or information extraction, which are key use cases motivating the research.", "methods_and_evaluation_criteria": "- The methods (Sort Loss, hybrid loss, and sinusoidal kernels) are well justified and designed specifically to address the permutation problem in speaker diarization. The paper thoroughly uses established datasets (CALLHOME, DIHARD3, LibriSpeechMix, AMI, ICSI) that reflect realistic and diverse scenarios.\n- However, the proposed evaluation criteria, while sensible, could be enhanced by additional metrics assessing latency, resource efficiency, and robustness to domain shifts, which are critical in real-world scenarios but currently missing from evaluation criteria.", "theoretical_claims": "The paper provides detailed theoretical justifications related to permutation invariance and equivariance properties of multi-head attention mechanisms in Transformers (Appendices E and F). The proofs for these properties are checked thoroughly and appear mathematically sound and correct.", "experimental_designs_or_analyses": "- The experimental design is largely sound, particularly the choice of datasets. Nevertheless, there are weaknesses: the authors use an artificially created LibriSpeechMix dataset with fixed delays, which is less representative of real-world spontaneous conversational scenarios. Evaluating the model on more realistic, noisy, or spontaneous speech scenarios (such as more diverse conversational datasets) would provide stronger evidence of real-world applicability.\n- The training setup lacks extensive details about hyperparameter selection (particularly the choice of α in hybrid loss, the impact of dropout rates, and the reasons for not employing data augmentation techniques like SpecAugment), leaving unanswered questions regarding the robustness of the presented results.", "relation_to_broader_scientific_literature": "Sortformer effectively addresses existing gaps in multi-speaker ASR and speaker diarization literature:\n- It explicitly discusses the limitations of PIL-based diarization methods (EEND series) regarding integration complexity into ASR systems.\n- It effectively positions itself within recent literature, notably SOT (Serialized Output Training) methods, multi-task learning in speech models, and transformer-based diarization architectures.\n- Clear reference to recent competitive models (EEND variants, WavLM, DOM-SOT) is well contextualized.\n\nThe paper is notably strong in placing its novel Sort Loss clearly relative to prior work.", "essential_references_not_discussed": "The authors have generally cited essential works thoroughly.", "other_strengths_and_weaknesses": "**Strengths**\n- The introduction of Sort Loss and integration into ASR frameworks is a significant conceptual innovation.\n- Rigorous empirical results thoroughly demonstrate effectiveness and robustness.\nPractical Significance: Easily integrable and adaptable to various downstream speech applications and multimodal LLM systems, increasing practical relevance.\n\n**Weaknesses**\n- Computational overhead or runtime comparisons were not thoroughly addressed, leaving open questions regarding practical efficiency.\n- Sort Loss inherently assumes accurate estimation of arrival times. Error analysis of mis-sorting scenarios is somewhat limited, raising potential robustness concerns under challenging real-world conditions.", "other_comments_or_suggestions": "Please refer to the previous sections.", "questions_for_authors": "- Can you provide detailed computational runtime benchmarks for Sortformer integration into ASR, especially compared to standard diarization modules? Understanding practical deployment feasibility is critical.\n- How robust is Sortformer to initial sorting inaccuracies? Have you conducted experiments analyzing the system's degradation under varying levels of sorting errors, and if so, what were your findings?\n- Could you clarify explicitly how Sort Loss compares conceptually and empirically to recent dominance-based methods like DOM-SOT [1]? Would such methods complement or potentially outperform Sort Loss under certain conditions?\n\n[1] Shi, Ying, et al. \"Serialized Output Training by Learned Dominance.\" Proc. Interspeech 2024. 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDl9TFuYsI", "reviewer": "Reviewer_rtb3", "summary": "This paper proposes a model called sortformer and a sorting loss to achieve joint speaker diarization and ASR without the need for permutation invariance loss. The proposed model can still be trained with PIL and it can also be combined with the sorting loss. In terms of modeling, the speaker label probabilities are obtained by setting up the output layer as a multilabel output layer with sigmoids. To get the sorting loss, speaker are labeled based on their appearance order in time. Hence the first speaker is labeled as class 0, the next one as class 1, etc. Once the ground truth speakers are also sorted in a similar way, the sorting loss uses binary CE. Speaker supervision to the model is provided by the kernel based speaker encodings. Training of the joint ASR + diarization model can be done either at word or segment level. \n\nExperiments use real and simulated speech mixtures for training and tested on various test sets in terms of DER for speaker diarization and WER for ASR. The results presented suggest competitive performance as compared to the existing approaches with a simpler training framework.\n\n## update after rebuttal\nI would like to keep my score after the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Experimental Designs Or Analyses:\n\nHere is model size information for the models we listed in Table 1. We will add the model size to Table 1.\n\n- **MSDD**: 31.1 M  \n- **EEND-EDA**: 6.4M  \n- **WavLM-L + EEND-VC**: 317M+  \n- **EEND-GLA-Large**: 10.7M  \n- **AED-EEND**: 11.6M  \n- **AED-EEND-EE**: 11.6M  \n\n## Response to Other Strengths And Weaknesses:\n\n2-1. Comments on speaker embedding experiments: We have done a few experiments by concatenating speaker embedding (TitaNet) for the same multispeaker ASR task. However, we were not able to obtain any improvements with both frozen and fine-tunable speaker embedding extractors. The reason we are making a guess is as follows:\n\na. The speaker embedding model is trained to distinguish 10K to 100K different speakers and the speaker representations are too complicated to make synergistic effect on 2~4 speakers in a session.  \n\nb. Not only speaker tagging accuracies, speaker embedding interferes with the ASR output states (ASR embedding from Fast Conformer Encoder) and damages the WER itself.  \n\nc. Since there is already plenty of content included in the paper, we did not include the speaker embedding experiments which have no improvements. In addition, concatenating speaker embedding for speaker adaptation or multi-talker ASR is not a novel idea, which has been already tried in many of previous studies. We did not think that showing speaker-embedding concatenation experiments bolster the main idea of this paper and the other contents should be prioritized to explain and demonstrate the proposed method.  \n\n2-2. Speaking of the speaker embedding model, Sortformer is initialized with the NEST (Nemo Encoder for Speaker Tasks) model that is self-supervised learned on large size unlabeled dataset and NEST can be trained to perform speaker verification/identification. In a way, we are already taking advantage of speech representation pretrained on large amounts of datasets.\n\n3. The second section of the green speaker should still be spk3. Not only our system but this applies to all the speaker diarization systems or multi-speaker ASR. If you want to have a better understanding on multi-speaker ASR tasks, please watch the supplementary demo video that contains multi-speaker ASR in action.\n\n## Response to Other Comments Or Suggestions:\n\n- We will update Figure 5 to intuitively explain the issues you highlighted in the final version.\n\n## Response to Questions For Authors:\n\n1. Please see rebuttal for “Other Strengths And Weaknesses”.  \n2. Please see rebuttal for “Other Strengths And Weaknesses”.  \n3. We speculate that this happens because word-level objectives produce more gradient for speaker tagging than segment level objectives due to its token count.  \n\na. The segment-level objective includes a speaker token whenever there is a speaker switch (i.e., speaker change), whereas in the word-level objective, each word is accompanied by a speaker token.  \n\nb. In the proposed MS-Canary system, we compute the cross-entropy (CE) loss on the multispeaker text output (consisting of both speaker and text tokens) for model training.  \n\nc. Consequently, the word-level objective places greater importance on correct speaker assignment in addition to accurate word recognition compared to the segment-level objective. This leads to an improved cpWER.\n\n## Side Note\n\nPlease also take a look at the other rebuttals for other reviewers regarding runtime measurements.", "claims_and_evidence": "Seems correct.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Read the equations quickly. They seem to be correct.", "experimental_designs_or_analyses": "In table 1, different loss functions and the use of post-processing are compared, resulting in hybrid loss with post-processing leading to the best performance (expected). In 2- and 3-speaker conditions, Sortformer outperforms existing approaches. For n=4 speakers, the performance is slightly behind than a previous study but still the numbers are comparable.  \n\nFrom Table 2, it seems that the bigger model (model 6 vs. 2) performs better. Hence, it would be good to know the sizes of the models in Table which outperformed the Sortformer model on some test sets. Could you please add those details to Table 1? \n\nASR experiments on LibrispeechMix also show promising WERs.", "relation_to_broader_scientific_literature": "Sortformer, and especially the sorting loss can be used as a complementary tool for applications that require permutation invariant training. The most common use case is the speaker diarization domain but the idea can be utilized in other type of applications, too. \nUse of sinusoidal embeddings for speaker guidance is also an interesting idea.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "+ Strengths: The paper proposes a simple but working solution to permutation invariance issues in speaker diarization. The proposed loss could be used by itself or in combination with PIL which can be useful for other model training applications.\n\n- Weakness: A few additional experimental experiments could have made the paper more stronger. For example, what if we replaced the sinusoidal guidance with some sort of real speaker embeddings from a speaker recognition system? \n\n- In terms of clarity, the paper is clear overall. However, a few details might have been made clearer. For example, Fig. 5, shows a scenario where the green speaker speaks twice with a different speaker (spk4) in between those two segments. In the proposed system, how do we treat the second section of the green speaker? Do we still say spk3 or do we increment the count and say spk5?", "other_comments_or_suggestions": "- Fig. 5. might be updated in light of the question I raised above.", "questions_for_authors": "1) What if we replaced the sinusoidal guidance with some sort of real speaker embeddings from a speaker recognition system, how would the diarization and ASR results look like? \n\n2) Fig. 5, shows a scenario where the green speaker speaks twice with a different speaker (spk4) in between those two segments. In the proposed system, how do we treat the second section of the green speaker? Do we still say spk3 or do we increment the count and say spk5?     \n\n3) In Section 5.4, it is mentioned that \"System 5 not only shows degradation in segment-level objectives,\" Do you have an explanation for this result which you can add to the text?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ndLn0CwgTc", "reviewer": "Reviewer_98zA", "summary": "The authors introduce Sortformer, a model built on a transformer-based encoder and trained using a hybrid loss that combines permutation invariant loss (PIL) and the newly proposed Sort Loss. Sort Loss is formulated as a binary cross-entropy loss, calculated between the sorted speaker presence labels in a sequence and the encoder’s output sequence. Once trained, the model can be seamlessly integrated with a multi-speaker ASR system, which is trained using standard cross-entropy loss. The method was evaluation on diarization and multi-speaker ASR tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Claims And Evidence\n\n### 1. Response to the reviewer's claim that the statement \"Sort Loss solves the permutation problem is not fully supported\":\n\na. What we cannot achieve without PIL+ alpha=0.5 is the \"maximized diarization performance\", not \"resolved permutation\".  \nb. Therefore, we believe we can say that \"Sort Loss based training\" resolves the permutation problem in speaker diarization training and multispeaker ASR training.  \nc. Even if we use only Sort Loss without PIL, the Sortformer still performs with the sorting capability and PIL is a helping hand.\n\n### 2. WER/DER with Sort Loss\nShowing the SOTA diarization with Sort Loss is not the contribution we claim.\n\n### 3. Response to missing deeper analysis:\n\na. Given page constraints, we focused on demonstrating how Sort Loss enables diarization supervision using standard ASR token-level cross-entropy loss.  \n\nb. Table 1 shows all comparable systems exhibit similar DER degradation with more speakers, as each additional speaker introduces compounding diarization errors with the similar proportion as our proposed system. The sole exception (WavLM-EEND-VC) is not an end-to-end speaker diarization model, weakening the validity of direct comparisons.\n\n\n\n\n## Response to Experimental Designs Or Analyses\n\n### 1. Alpha parameter tuning: \nWe did grid search on this alpha parameter then found that 0.5 is the best performing parameter. We will add the grid-search results in the appendix in the final version. \n\n### 2. Runtime measurements:\n\n#### a. Training Time Impact vs pure PIL training\n\n- Pure Sort Loss: +0.23% (17.038 vs 17.0 min/epoch)  \n- Hybrid Loss: +2.26% (17.385 vs 17.0 min/epoch)\n\n#### b. Inference Time of Multispeaker ASR:\nLibriSpeechMix test-3mix (2,620 files, total duration: 42,514.9s) on an NVIDIA RTX A6000, batch size 100, with 10-run averages.\n\n- MS-ASR with Sortformer: 300.213s  \n- MS-ASR without Sortformer 297.891s  \n- Adding Sortformer causes 0.78% overhead in runtime (1.0078x)\n\n#### c. Inference on standalone diarization:\nNo added inference time.\n\n\n## Response to Other Strengths And Weaknesses\n\n### 1. In the Table 1, we wanted to show:\n\na. Showing Sort Loss achieves SOTA is not the key contribution. \nb. Sort-Loss can perform at a comparable level with PIL-trained models.  \nc. Hybrid Loss outperforms PIL based method, without losing \"learned arrival-time sorting capability\".\n\n### 2. Sort Loss shows no improvement over PIL\nWe believe this is a minor weakness because we clearly state the goal of Sort Loss in the paper. Resolve permutation during multispeaker ASR training bridging diarization timestamps and tokens. Improving the PIL method with Sort Loss is not the main contribution we claim.\n\n### 3-1. Robustness:\nOne of our evaluation dataset, DIHARD3, as the name suggests \"Diarization is Hard\", includes 11 domains that are very challenging to diarize such as noisy restaurant conversations, web-video and street interviews. DIHARD3 is the most noisy and reverberant diarization evaluation dataset.\n\n### 3-2. Low-resource language:\nNIST SRE 2000 dataset includes multi-lingual speech (Mandarin Chinese, Vietnamese, Spanish, Tagalog, etc) and DIHARD3 also includes Arabic, Mandarin, Min Nan, Portuguese, Russian, and Polish language. Therefore, the evaluation on these datasets involve diarization performance on low-resource languages.\n\n### 3-3. Missing evaluations  \nHence, we do not think that the Sortformer diarization model is completely missing evaluations on noise robustness and low-resource languages.\n\n## Response to Other Comments Or Suggestions\n\n1. We realized this margin crash only after the submission. We will fix this in the final version of the paper.  \n2. \"A\" in Equation (14) is the output state (also referred to as ASR embedding) from Fast Conformer encoder in Figure 2. We will specify this in the final version.\n\n## Response to Questions For Authors\n\n### 1. “What is the loss used for Multi-speaker ASR Training Data?” \nThe answer to this question is “Cross Entropy Loss” and it is already clearly mentioned multiple times in the manuscript:\n\na. The last paragraph in Section 1 (Introduction) “multi-speaker ASR training ... tokenlevel cross-entropy loss”  \nb. Figure 2. See “Cross-Entropy Loss” at the top with the loss function symbol “L_{CE}”.  \nc. End of Section 2.3 “Our approach focuses ... based on token objectives and cross-entropy loss”  \nd. End of Section 3.3 “the model can be trained ... cross-entropy function..”  \ne. In Section 6 (Conclusion), “... thereby supporting cross-entropy loss-based training and unifying the multi-speaker ASR framework...”\n\n### 2. Limitations of our proposed system:\n\na. Currently, our proposed system has a limited inference length of 45 seconds and limited maximum of 4 speakers in a session.  \nb. The implementation we use for the experimental results runs Sortformer and FastConformer encoder in serial manner. This adds up inference time to the total inference time although it is less than 1%.", "claims_and_evidence": "* The claim that Sort Loss alone solves the permutation problem independently of PIL is not fully supported by the results. The experiments indicate that Sort Loss can complement PIL but cannot replace it. This is evident from the use of the weight parameter $\\alpha$ = 0.5, which gives equal importance to the PIL term.\n\n* The proposed model is designed for both speaker diarization and multi-speaker ASR. It has been evaluated on relevant benchmarks, and the results show that while it helps reduce word error rate (WER) in multi-speaker ASR, its impact on improving diarization error rate (DER) is less significant.\n\n* A deeper analysis of the proposed loss function is needed. Specifically, it would be helpful to examine how it affects training time and why its performance degrades as the number of speakers increases.", "methods_and_evaluation_criteria": "The method was evaluation on diarization and multispeaker asr tasks.\nOn diarization taks - the authors used 3 test sets and compared to six recent baselines.\nOn multispeaker ASR - the model was tested on AMI test and CH109 test set and compared to baseline on LibriSpeechMix.\nEvaluation looks reasonable to me.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The design and evaluation looks reasonable to me.\nThere are missing analysis sections: $\\alpha$ parameter tuning, time measurements, robustness to noise, error analysis.", "relation_to_broader_scientific_literature": "* Sortformer is one of the first models to integrate speaker diarization directly into ASR models using a differentiable sorting-based loss.\n\n*  The Sorted Serialized Transcript (SST) approach simplifies multi-speaker ASR training to be similar to mono-speaker ASR training.", "essential_references_not_discussed": "The essential references are discussed.", "other_strengths_and_weaknesses": "**Strengths**:\n* The paper is well-written with clear explanations and visualizations.\n* The proposed method shows strong performance on multi speaker ASR task (Table 3)\n\n\n**Weaknesses:**\n* According to Table 1 the proposed model works well for 2-3 speakers and the performance degrades as for 4 speakers.\n* Compared to PIL - the proposed loss doesn’t propose any improvements for diarization task (Table 1 - *Sortformer-PIL* and *Sortformer-Sort-Loss* rows)\n* Limited Robustness Analysis: Needs more tests on low-resource languages and extreme noise conditions.", "other_comments_or_suggestions": "* Figure 4 and Table 1 - need a space margin at the bottom\n* Equation 14 - not clear what is $A$", "questions_for_authors": "* What is the loss used for Multi-speaker ASR Training Data? Please describe it in the paper.\n\n* Are there any limitations of the proposed model? Please add a section in the text or in the appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFwA0GMGWv", "reviewer": "Reviewer_US9G", "summary": "This paper proposes a new multi-speaker speech diarization and recognition model, with a loss function bridging timestamps with the tokenzied texts, built on classification loss, permutation invariant loss, and the newly-proposed sort loss. The sort loss looks like an variant of PIL, where the label is placed in certain order, referred by either time or any relevant metric.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Claims and Evidence\n\n1. Sort loss is not a complementary or supportive loss to the system. Only by including Sort Loss, the model learns to arrange speaker predictions in arrival-time order. \n\n   a. If we only use PIL, the model does not have arrival-time sorting capability.  \n   b. We clearly state the goal of Sort Loss in the abstract: It is designed to bridge the gap between ASR tokens and diarization timestamps during “multispeaker ASR training”. And this is achieved by training the diarization model using Sort Loss or Sort+PIL hybrid loss.  \n   c. Therefore, it is fair to say rather PIL (in the hybrid loss) is playing a role as a supplementary loss since it additionally corrects the mapping between predictions and labels since sort-prediction could be erroneous in some cases.\n\n2. The “minimal architectural adjustments” are clearly explained throughout the paper.\n\n   a. Section 2.3, Section 3.1 and Figure. 2 Explain the downside of modular or pipelined multi-speaker ASR systems and we explain how our proposed method can train a multi-speaker ASR model without applying any specialized loss but using the standard token-level cross-entropy loss.  \n   b. If you want to visually check the “minimal architectural adjustments”, see “Sortformer” module and sinusoidal kernels in Figure 2. The original modules in the vanilla Carany model (FastConformer and Transformer Decoder) are not altered in terms of architecture. We optionally use adapters to boost the performance with less amount of trainable weights.  \n   c. In summary, our proposed approach maintains the original ASR model’s architecture, except the part where we inject encoder output with speaker-kernel injection.\n\n## Response to Theoretical Claim\n\n1. We found couple of errors in the appendix F:\n\n   a. In F2 and F3, all QW^{Q}, KW^{K}, VW^{W} should be changed to XW^{Q}, XW^{K}, XW^{W}.  \n   b. Regardless of the mistake a, we did not put a necessary condition “Q is learned parameters” to permutation invariance (See-[https://people.tamu.edu/~sji/classes/attn-slides.pdf]). Thus, the proof F.2 is incorrect and we will remove the proof of permutation invariance (F.2) in the final version. We do not use learnable query Q in our proposed system either.  \n   c. The Permutation Equivariance property (when there is no positional embedding) seems enough for supporting the need for positional embedding.\n\n## Response to Relation To Broader Scientific Literature\n\nWe believe that the reviewer’s claim “this may limit the contribution of the paper to multi-speaker ASR where there is no overlap between the speakers (and preferably, long pauses between them)” can hardly be considered a valid criticism, because: \n\n1. The supplementary demo video clip is designed for reviewers to intuitively understand multi-speaker ASR tasks in action, not to assess the functionality and performance of the proposed system.\n\n2. There is plenty of evidence that our proposed system works without such issue:\n\n   a. More than half of the samples in the diarization datasets DIHARD3, NIST-SRE-2000, CH109, are longer than 5 minutes, and these datasets have a very frequent long gap between each speaker’s segment. In the multi-speaker ASR test set AMI Corpus, the overlap ratio is 14% and the rest of 86% speech is non-overlapping speech.  \n   b. Based on these reasons, the point raised by the reviewer’s claim “this may limit the contribution” doesn't seem like a reasonable concern.\n\n## Response to Other Strengths And Weaknesses\n\n2. The additional computational cost could be discussed as follows:\n\n   a. Training Time Impact vs pure PIL training  \n   ∙ Pure Sort Loss: +0.23% (17.038 vs 17.0 min/epoch)  \n   ∙ Hybrid Loss: +2.26% (17.385 vs 17.0 min/epoch)  \n\n   b. Inference Time of Multispeaker ASR: LibriSpeechMix test-3mix (2,620 files, total duration: 42,514.9s) on an NVIDIA RTX A6000, batch size 100, with 10-run averages.  \n   - MS-ASR with Sortformer: 300.213s  \n   - MS-ASR without Sortformer 297.891s  \n   - Adding Sortformer causes 0.78% overhead in runtime (1.0078x)\n\n   c. Inference on standalone diarization: No added inference time compared to PIL trained models.\n\n   d. Our view on additional computational cost:  \n   (1) Training Sortformer with Hybrid Loss (Sort Loss + PIL) increases runtime by only 2.26% (x1.0226) compared to PIL-only training, while Sort Loss alone adds just 0.23% (x1.0023).  \n   (2) For inference, standalone diarization (Sortformer) adds no extra time versus the PIL-trained model.  \n   (3) Multispeaker ASR inference time increases marginally by 0.78% (x1.0078) when using Sortformer.  \n   (4) On the LibriSpeechMix dataset, our system achieves a 25.6% relative error rate reduction (7.14% → 5.31%).  \n   (5) Given the significant performance gains, the added computational cost is quite negligible, making the claim of \"significant computational cost\" not very convincing.\n\n3. The reviewer’s comment seems unfinished: “For example, the”.", "claims_and_evidence": "One important claim in the paper, or one of the main selling point of the paper, is that the SortFormer model cna reduce the workload of architectural adjustments of the original EEND model, and make the multi-speaker ASR training equivalent to mono-speaker one. However, this brings two problems:\n1. The speaker supervision of the multi-speaker is still involved in the training, the sort loss brings it cleverly to the label side and use the loss function as a complmentary one of the permutation loss. So, either the claim itselfm or the novelty of the claim seems limited.\n2. The paper does not show explicitly about how the model has been \"minimally\" adjusted.", "methods_and_evaluation_criteria": "About the proposed method. The reviewer thinks think the novelty of the method is limited, especially when the sort loss\n1. Still take advantage of the speaker supervision, and increases the computational load\n2. Does not show significant advantage by itself.\n\nThe evaluation criteria in terms of datasets and metrics are good, following state-of-the-art approaches.", "theoretical_claims": "There are several theoretical claims in the appendix F.2. about permutation of multi-head self attention (MHA):\n1. MHA is permutation invariant\n2. MHA is permutation equivariant\n\nHowever, the 1st claim's proof seems not correct, especially 29) to 30) needs more detailed explanation.", "experimental_designs_or_analyses": "The reviewer checks all the experimental results and setups. \n\n1. The main problem is that under diarization results, sort loss works best when being complementary with the original PIT loss, and does not perform that well when using it alone, whose reason is not discussed in detail. \n\n2. Besides, the training data for different models seems not well-unified and properly arranged. Since this is architectural and training strategy improvement, the authors should emphasize and be more regor on this issue.", "relation_to_broader_scientific_literature": "The key contribution of the paper is about the proposed multi-class loss function, which takes good advantage of the speaker labels and timestamps. However, this may limit the contribution of thenpaper to multi-speaker ASR where there is no overlap between the speakers (and preferrably, long pauses between them). This is not shown in the demo (supplementary material) needs to be validated.", "essential_references_not_discussed": "The reviewer thinks there is no essential reference missed.", "other_strengths_and_weaknesses": "There are several other minor weaknesses of the paper.\n1. The significance of the method.\n2. The efficiency of the method. The method itself involves token sorting accoridng to certain metric multiple times. The reviewer wonders if this will significantly increase the computational cost\n3. Lacking clarity. There are multiple places in the paper lacking clarity or causes confusion. For example, the", "other_comments_or_suggestions": "The reviewer does not have further comment or question about this paper, as such have been enlisted in detail in the earlier questions.", "questions_for_authors": "The questions for the authors of this paper has been enlisted in the above questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Sortformer which is an encoder-based model for speaker diarization designed for integration with multi-speaker ASR. The authors propose Sort Loss to sort speakers based on their arrival time to resolve the speaker permutation problem. Sort Loss can work either by itself or jointly with the conventional permutation invariance loss (PIL). On top of that, the authors also introduces sinusoidal kernels to encode speaker identities to provide speaker supervision in the training. As a result, multi-speaker ASR can be trained analogously to the mono-speaker ASR case. Experimental results show the effectiveness of Sortformer in improved performance on speaker diarization and multi-speaker ASR.  Most of the concerns raised by the reviewers have been cleared in the rebuttal by the authors (including an error in proving the PI property of MHA under the setting used in the work).   Overall, the work is interesting and the performance is good, especially when using the hybrid loss consisting of both sort loss and PIL.
Decision: Accept (poster)