## 4 Experimental Performance Evaluation
### ASR Model Training  
All proposed multi-speaker speech recognition models were implemented by the _ESPnet_ framework [42] with the _PyTorch_[41] backend. The acoustic features consisted of 80-dimensional mel-scale log filterbank energies, including pitch features, deltas, and deltas. The input features were created by concatenating the speaker embedding vector with the acoustic features. Consequently, the dimension of the input feature depended on the type of speaker embedding vectors adopted. For instance, in the case of \(i\)-vectors, the input dimension became 480 (400 for the speaker embedding and 80 for the acoustic feature). In the case of \(x\)-vectors, the dimension was 592 (512 + 80), and it was 336 (256 + 80) for the \(d\)-vectors. In the Transformer-based multi-speaker decoder ASR model, there are 18 layers in the conventional method [43; 44].  
For the proposed methods, \(\text{Enc}_{\text{mix}}\) acts as the CNN (convolutional neural network) embedding layer. \(\text{Enc}_{\text{diff}}\) and \(\text{Enc}_{\text{rec}}\) consist of four and eight Transformer layers, respectively. For all Transformers, the attention size was set to 256, the number of attention heads to 4, and the fully-connected feed-forward network size to 2048. The training of the Transformer model was conducted with the Adam optimizer and Noam learning rate decay, as outlined in [3]. The backend ASR, \(\text{Dec}_{\text{rec}}\), was initialized with a pretrained single-speaker speech recognition model from the _ESPnet_ recipe of the _LibriSpeech_ dataset. During training, it was kept frozen for the first 15 epochs to ensure stability. At the end of all ASR model training, a Transformer language model from the _ESPnet_ recipe of the _LibriSpeech_ dataset was employed on the pretrained single-speaker _LibriSpeech_ dataset model without fine tuning. Similarly, as introduced in Section 4.1, all speaker embedding extractors were also used without additional training on the pretrained models from the VoxCeleb 1 and 2 speaker recognition datasets [40].  
To train the baseline ASR model and the proposed models, we used four units of NVIDIA RTX TITAN with a 24 GB GPU memory. The single-speaker ASR model based on Transformer architecture has 113 million parameters, and its training data consist of 460 h of _train-clean-100_ and _train-clean-360_ subsets of _LibriSpeech_[36]. In terms of training time, it required 223 h, which is equivalent to 892 GPU hours because four GPUs were used. The baseline multi-speaker ASR [43; 44] has 90.6 million parameters, and 30.6 million of them were taken from a single-speaker ASR Transformer model. The remaining 60 million parameters were trained by multi-speaker utterances. The multi-speaker training set consisted of 270 h, as shown in Table 1, and it took 320 GPU hours to train the multi-speaker modules. For the proposed SAT methods, 0, 11.2, and 4.2 million additional parameters were required for the \(i\)-, \(d\)-, and \(x\)-vectors, respectively. They took 320, 336, and 328 additional GPU hours to train. The number of trainable parameters and the total training hours are compared in detail in Table 2.