## 3 Elaborated Model of the Proposed Methods
### Proposed Speaker-Attributed Training
\(\text{Enc}_{\text{diff}}^{(s)}\) splits the mixed features of \(S\) speakers so that single-speaker information remains in each \(\mathbf{H}^{(s)}\). The last \(\text{Enc}_{\text{rec}}\) transforms each \(\mathbf{H}^{(s)}\) into the embeddings \(\mathbf{G}^{(s)}\) to be compatible with the pretrained decoder \(\text{Dec}_{\text{rec}}\). The encoder weights were shared over all the input \(\mathbf{H}^{(s)}\). Finally, \(\text{Dec}_{\text{rec}}\) maps each \(\mathbf{G}^{(s)}\) to the output \(\mathbf{y}^{(s)}\), which is typically a word sequence for ASR. The pretrained decoder \(\text{Dec}_{\text{rec}}\) was fixed and shared over all of the input. \[\mathbf{G}^{(s)} = \text{Enc}_{\text{rec}}\Big{(}\mathbf{H}^{(s)}\Big{)}\,, \tag{7}\] \[\mathbf{y}^{(s)} = \text{Dec}_{\text{rec}}\Big{(}\mathbf{G}^{(s)}\Big{)}\,. \tag{8}\]  
The three-stage encoder design has several advantages. The recognition encoder, \(\text{Enc}_{\text{rec}}\), is focused on matching the previous encoder output to the recognition decoder \(\text{Dec}_{\text{rec}}\). Their weights are shared so that the number of trainable parameters can be reduced. The speaker splitting is performed by \(\text{Enc}_{\text{diff}}\), and merging the speaker embedding vectors and speech features is performed by \(\text{Enc}_{\text{mix}}\) only. By assigning different roles, it can provide faster training and improved performance. Another advantage is that the pretrained ASR decoder and speaker encoder-decoder is used without retraining. Moreover, those modules are trained by single-talker recordings, and only encoders require multi-talker recordings for fine tuning, as shown by the dotted box in Figure 3.  
The whole procedure of the proposed speaker-attributed training is shown in Algorithms 1-3. The Procedure Forward in Algorithm 1 is a step-by-step illustration of the forward inference in \[\mathbf{E}=\left\{\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e} _{T}\right\} = \text{Enc}_{\text{speaker}}(\mathbf{X})\,,\]0 The inputs are a set of mixed feature vectors \(\mathbf{X}\) in Equation (1), where the parameter sets are \(\Theta_{0}\) and \(\Theta_{1}\) and the outputs are the predicted word sequences \(\hat{\mathbf{Y}}\) for the given input \(\mathbf{X}\). \(\Theta_{0}\) consists of a speaker encoder and decoder to extract speaker embedding vectors, i.e., \(\text{Enc}_{\text{speaker}}\) and \(\text{Dec}_{\text{speaker}}\), and the pretrained speech recognition decoder is \(\text{Dec}_{\text{rec}}\). \(\Theta_{1}\) is a set of \(\text{Enc}_{\text{mix}}\), \(\text{Enc}_{\text{diff}}^{(1\dots S)}\), and \(\text{Enc}_{\text{rec}}\), which are required by Equations (5)-(7). The Procedure Backward in Algorithm 2 requires the predicted and target word sequences \(\hat{\mathbf{Y}}\) and \(\mathbf{Y}\). It first computes the loss between \(\hat{\mathbf{Y}}\) and \(\mathbf{Y}\), and then it performs a single backpropagation learning step to minimize the loss (denoted by \(L\)). The Procedure SAT in Algorithm 3 defines the whole training process. It requires an input of mixed features \(\mathbf{X}\), target word sequences \(\mathbf{Y}\), and parameter set \(\Theta_{0}\). These inputs are fixed throughout the entire training process. The output of Procedure SAT, \(\Theta_{1}\), is a set of a mixed encoder, speaker-differentiation encoder, and recognition encoder. In Line 2 of Algorithm 3, they are created by an appropriate initialization method. In Lines 4-5, the input features \(\mathbf{X}\) is passed through the network constructed by the parameters \(\Theta_{0}\) and \(\Theta_{1}\) by calling a sub-procedure Forward, and set \(\Theta_{1}\) is updated by calling Backward. These are repeated until the convergence of the training loss \(L\). The detailed implementation is publicly available at [[https://github.com/craft8244/MSSR](https://github.com/craft8244/MSSR)]([https://github.com/craft8244/MSSR](https://github.com/craft8244/MSSR)), accessed on 17 August 2024.