## 2 Related Work
### Conventional Multi-Speaker Speech Recognition Methods  
Several methods have been proposed to generate multiple, speaker-specific texts from mixed recordings of multi-talkers with overlapped speeches. The conventional methods can be classified into the following four types. 1. A combination of single-speaker ASR models and existing acoustic source separation methods, such as deep clustering (DPCL) [2], permutation invariant training (PIT) [3; 23], using \(x\)-vectors [24], self-supervised learning [25], and the joint learning of separation masks and speech recognition models [26]. 2. Directly building a multi-speaker speech recognition system using mixed speech as the input and multiple text streams as learning targets [4; 27; 28; 29]. 3. The addition of speaker embedding vectors as indicators for the speakers in the input mixture [7; 8; 9; 30] combined with the multi-speaker ASR system in (b) This is called speaker-attributed training for automatic speech recognition (SAT-ASR) [8]. 4. Addition of a splitting encoder network that generates speaker-specific speech embedding vectors, using them as inputs to the single-speaker ASR decoder [5; 6; 31]. No retraining of a single-speaker ASR decoder is necessary. As illustrated in Figure 2, Type (a) is implemented by combining existing source separation and ASR models, and no retraining or fine tuning is required. Its limitation is that the ASR performance degrades according to the distortion in the separated speech signals caused by the adopted monaural source separation method. The ASR performance is no better than the other types with fine tuning [4; 5]. The second type constructs a single-input and multiple-output (SIMO) black box, with its inputs and outputs being mixed speech signals and multiple word sequences spoken by the individual speakers, respectively. This SIMO black box is usually implemented by either hybrid deep neural networks [4] or end-to-end ASR [5; 6]. This method ensures decent multi-speaker ASR performance if a high enough amount of training data is given. However, each of the training samples should be mixed with all the other speaker samples to simulate all the possible combinations of talkers; thus, the training time may become \(O(MN)\), where \(M\) is the number of speakers and \(N\) is the number of samples, while it is \(O(N)\) for single-speaker ASR models. The third type, as shown in Figure 2-(c), is called speaker-attributed ASR [8] and appends speaker embedding vectors to the input speech features. The additional embedding vectors let the ASR be informed on the characteristics of overlapped speakers. It is expected to more separately train the multi-speaker ASR models, resulting in faster convergence, as well as improved performance [8; 9]. It requires additional training of the speaker encoder with mixed-input recordings so the computational overhead of the whole training is the same order as Type (b)--\(O(MN)\). The last method, Type (d), was proposed to overcome the heavy-computation problem. The single-speaker ASR is constructed by a Transformer network [6], and its decoder is used without retraining. The \"Split Encoder\" shown in Figure 2-(d) was trained by a set of mixed-speech signals in a way that it can encode the mixed input into multiple representations, \(\mathbf{H}^{(1)}\) and \(\mathbf{H}^{(2)}\), to be compatible with the ASR decoder. The multiple text outputs, denoted by \(\mathbf{y}^{(1)}\) and \(\mathbf{y}^{(2)}\), where the parenthesized superscripts are speaker indices, were used as target labels, and only the split encoder was fully trained while the SISO ASR decoder was fixed. Training the ASR decoder takes \(O(N)\) time, as does \(O(MN)\) for the split encoder, but it takes much less time than the ASR decoder. Therefore, the computational overhead was greatly reduced when compared to the SIMO ASR approach. This paper proposes a novel multi-speaker ASR method that takes advantages of Types (c) and (d): a combination of speaker-attributed training and encoder-decoder-based multi-speaker ASR.