## 3 Elaborated Model of the Proposed Methods  
### Proposed Speaker-Attributed Training  
The proposed model combines automatic speech recognition (ASR) using the encoder-decoder architecture of the Transformer network [6] and speaker-attributed training with speaker embedding vectors [7,8]. As shown in Figure 3, speaker encoder and decoder blocks of a Transformer network, \(\text{Enc}_{\text{speaker}}\) and \(\text{Dec}_{\text{speaker}}\), were trained using the ASR training dataset. The subscript'speaker' indicates that the encoder-decoder extracts speaker embedding vectors from the input recording. The training samples were single-talker utterances, and their targets were speaker identity labels that were used to obtain representations for the speaker differences in order to be maximally emphasized. The speaker embedding block consists of the trained \(\text{Enc}_{\text{speaker}}\) and \(\text{Dec}_{\text{speaker}}\), which is represented as  
\[\mathbf{X} = \left\{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{T}\right\}, \tag{1}\] \[\mathbf{E}=\left\{\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e} _{T}\right\} = \text{Enc}_{\text{speaker}}(\mathbf{X})\,,\] (2) \[\mathbf{Q}_{\text{mix}}=\left\{\mathbf{q}_{1},\mathbf{q}_{2}, \dots,\mathbf{q}_{T}\right\} = \text{Dec}_{\text{speaker}}(\mathbf{E})\,, \tag{3}\]  
where \(T\) is the number of the feature vectors from the time-varying input recording, \(\mathbf{x}_{t}\) is the extracted feature at the analysis frame index \(t\), \(\mathbf{e}_{t}\) is the encoder output, and \(\mathbf{q}_{t}\) is the extracted speaker embedding vector at the index \(t\). The embedding vector \(\mathbf{q}_{t}\) is also referred to as speaker query vector [8,32] because it identifies the speaker in the input speech. The capital notations, such as \(\mathbf{X}\), \(\mathbf{E}\), and \(\mathbf{Q}_{\text{mix}}\), represent the set of all \(T\) vectors, and they are often implemented by matrices. We added'mix' to \(\mathbf{E}\) as a subscript to imply that this set of speaker embedding vectors is used by the ASR encoder \(\text{Enc}_{\text{mix}}\). In the case of the \(d\)-vector method [33,34], the extraction was autoregressive, so the embedding vector at a time was dependent on the previous vectors only, and this can be expressed by  
\[\mathbf{q}_{t}=\text{Dec}_{\text{speaker}}^{(d-\text{vec})}(\left\{\mathbf{q} _{1},\dots,\mathbf{q}_{t-1}\right\},\ \mathbf{e}_{t})\,,\quad t=1,\dots,T\,. \tag{4}\]  
As show in Figure 3, the ASR encoder was divided into three encoders with different roles. The acoustic feature \(\mathbf{X}\) and the speaker embedding \(\mathbf{Q}_{\text{mix}}\) were concatenated and were called vertical stacking [7]. The first encoder \(\text{Enc}_{\text{mix}}\) maps the concatenated input to a higher-dimensional representation \(\mathbf{H}_{\text{mix}}\),  
\[\mathbf{H}_{\text{mix}}=\text{Enc}_{\text{mix}}(\mathbf{X},\,\mathbf{Q}_{ \text{mix}})\,. \tag{5}\]  
Figure 3: Overview of the proposed speaker-attributed training ASR system. The gray blocks were trained by single-speaker recordings, and these were fixed when the white blocks were trained with multi-speaker recordings. The same \(\text{Enc}_{\text{rec}}\) and \(\text{Dec}_{\text{rec}}\) were used with different inputs, and they are represented by the dotted link denoted by *_shared_. The boxed parts (**) require fine-tuning with multi-speaker utterances. The role of \(\text{Enc}_{\text{mix}}\) is in emphasizing the information of the speakers by merging the speaker embedding \(\mathbf{Q}_{\text{mix}}\) and \(\mathbf{X}\). From the higher-dimensional representation \(\mathbf{H}_{\text{mix}}\), the speaker-differentiating encoder \(\text{Enc}_{\text{diff}}^{(s)}\) extracted the speech of each speaker,  
\[\mathbf{H}^{(s)}=\text{Enc}_{\text{diff}}^{(s)}(\mathbf{H}_{\text{mix}})\,,\quad s =1,\dots,S, \tag{6}\]  
where \(s\) and \(S\) represent the speaker index and the total number of speakers, respectively. As shown in Figure 3, \(S=2\).