## 3 Elaborated Model of the Proposed Methods
### Speaker-Attributed Training Using Speaker Profiles
Procedure Beta in Algorithm 4 is a step-by-step description of the attention-weighted speaker embedding vector derivation in Equations (9)-(11). The computed similarities represent the attention of the \(\mathbf{q}\) to the enrolled profile vectors. It computes cosine similarities for all embedding vectors in the profile matrix \(\mathbf{P}\), and it replaces the speaker embedding vector \(\mathbf{q}\) with the linear combination of the profile embedding vectors. Procedure Forward in Algorithm 5 is a forward inference with a profile matrix. Most of the steps are identical to Algorithm 1 except that the speaker embedding \(\mathbf{Q}_{\text{mix}}\) of the mixed-speech features was replaced with the attention-weighted embedding vectors, as shown in Lines 6-9. There was no change in the backpropagation steps and training iterations, so Algorithms 2 and 3 can be used without any modification. The detailed implementation is also publicly available at [[https://github.com/craft8244/MSSR](https://github.com/craft8244/MSSR)]([https://github.com/craft8244/MSSR](https://github.com/craft8244/MSSR)), accessed on 17 August 2024. ```
0:\(\mathbf{q},\Theta_{1}\), \(\mathbf{P}\)\(\triangleright\) speaker embedding vector, speaker profile matrix
1:procedureBeta(\(\mathbf{q}\), \(\mathbf{P}\)):
2:\(K\leftarrow\left\lvert\mathbf{P}\right\rvert\)\(\triangleright\) number of profile vectors
3:for\(k=1,\dots,K\)do:
4:\(b_{k}\leftarrow\mathbf{q}\cdot\mathbf{p}_{k}/(\left\lVert\mathbf{q}\right\rVert \left\lVert\mathbf{p}_{k}\right\rVert)\)\(\triangleright\) compute cosine similarity
5:endfor
6:for\(k=1,\dots,K\)do:
7:\(\beta_{k}\leftarrow\exp(b_{t,k})/\sum_{j=1}^{K}\exp\bigl{(}b_{t,j}\bigr{)}\)\(\triangleright\) softmax normalization
8:endfor
9:return\(\tilde{\mathbf{q}}=\left(\sum_{k=1}^{K}\beta_{k}\mathbf{p}_{k}\right)\)\(\triangleright\) return attention-weighted speaker embedding
10:endprocedure
```  
**Algorithm 4** Attention-weighted speaker embedding vector extraction  
```
0:\(\mathbf{X}=[\mathbf{x}_{1}\dots\mathbf{x}_{T}]\), \(\Theta_{0}\), \(\Theta_{1}\), \(\mathbf{P}\)\(\triangleright\) overlapped speech and transformer encoder/decoders, speaker profile matrix
1:procedureForward(\(\mathbf{X}\), \(\Theta_{0}\), \(\Theta_{1}\), \(\mathbf{P}\)):
2:\(\left(\text{Enc}_{\text{speaker}},\text{Dec}_{\text{speaker}},\text{Dec}_{ \text{rec}}\right)\leftarrow\Theta_{0}\)
3:\(\left(\text{Enc}_{\text{mix}},\text{Enc}_{\text{diff}}^{(1\dots\mathbf{x})}, \text{Enc}_{\text{rec}}\right)\leftarrow\Theta_{1}\)\(\triangleright\) fetch transformer parameters
4:\(\mathbf{E}\leftarrow\text{Enc}_{\text{speaker}}(\mathbf{X})\)\(\triangleright\) generate a matrix of multi-speaker encoder outputs
5:\([\mathbf{q}_{1}\dots\mathbf{q}_{T}]\leftarrow\text{Dec}_{\text{speaker}}( \mathbf{E})\)\(\triangleright\) convert to a matrix of embedding vectors
6:for\(t=1,\dots,T\)do:
7:\(\tilde{\mathbf{q}}_{t}\leftarrow\text{Beta}(\mathbf{q}_{t},\mathbf{P})\)\(\triangleright\) attention-weighting
8:endfor
9:\(\tilde{\mathbf{Q}}_{mix}=[\tilde{\mathbf{q}}_{1}\dots\tilde{\mathbf{q}}_{T}]\)\(\triangleright\) construct attention-weighted embedding matrix
10:\(\mathbf{H}_{\text{mix}}\leftarrow\text{Enc}_{\text{mix}}(\mathbf{X},\tilde{\mathbf{Q}}_{ \text{mix}})\)\(\triangleright\) map to high-dimensional features
11:for\(s=1,\dots,S\)do:
12:\(\mathbf{H}^{(s)}\leftarrow\text{Enc}_{\text{diff}}^{(s)}(\mathbf{H}_{\text{ mix}})\)\(\triangleright\) extract single-speaker features
13:\(\mathbf{G}^{(s)}\leftarrow\text{Enc}_{\text{rec}}\Bigl{(}\mathbf{H}^{(s)}\Bigr{)}\)\(\triangleright\) convert to decoder-compatible features
14:\(\mathbf{\hat{y}}^{(s)}\leftarrow\text{Dec}_{\text{rec}}\Bigl{(}\mathbf{G}^{(s)}\Bigr{)}\)\(\triangleright\) generate speech recognition outputs
15:endfor
16:return\(\mathbf{\hat{Y}}=\left\{\mathbf{\hat{y}}^{(1)},\dots\mathbf{\hat{y}}^{(S)}\right\}\)
17:endprocedure
```  
**Algorithm 5** Forward inference using speaker profile