## 4 Experimental Setup
### Model Configuration  
The encoder-decoder module of the ESPNET toolkit [33] is used to develop the proposed JSUM that blends supervised and unsupervised adversarial learning, as shown in Figure 2. The shared encoder uses the unsupervised, pre-trained wav2 vec 2.0 model. This pre-training process is implemented in fairseq [34]. In the wav2 vec 2.0 model [35], there are two transformer architectures: (1) Base model, with 12 layers of transformer, model dimension is 768, internal dimension is 3072, and with 8 attention heads; (2) Large model,  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Lang** & **kk** & **tr** & **ky** & **tt** & **du** & **zh** & **ru** & **it** & **es** & **fr** \\ \hline Unsup.pretrain (h) & 2 & 11 & 17 & 17 & 29 & 50 & 55 & 90 & 168 & 353 \\ Sup.train (h) & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 \\ Evaluation (h) & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Data duration of 10 languages in CommonVoice.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{4}{c}{**Duration**} \\ \cline{2-5}  & **Train** & **Dev** & **Test** & **Total** \\ \hline Librispeech & 960.9 & 10.7 & 10.5 & 982.1 \\ KSC & 318.4 & 7.1 & 7.1 & 332.6 \\ TIMIT & 2.9 & 0.3 & 0.2 & 3.4 \\ THUYG-20 & 20.2 & 1.1 & 2.4 & 23.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Specifications of LibriSpeech, KSC, TIMIT, and THUYG-20 data sets.  
with 24 layers of transformer, model dimension is 1024, internal dimension is 4096, and with 16 attention heads. This paper uses the Base model to pre-train the encoder. Decoder-1 consists of a single layer GRU containing 300 cells, as shown in Figure 3.  
Decoder-2 consists of a generator and a discriminator, as shown in Figure 4. The generator is a single non-causal convolution with a kernel size of 4. The input is a segment of dimension 512 representing \(S\), and the output is a \(\mid\!O\!\mid\) dimensional vector, which represents the probability distribution on the phoneme vocabulary. The subsequent generator predictions are combined and then fed into the discriminator, and the softmax normalization is applied. The discriminator consists of three causal convolutional blocks with a kernel size of 6 and a hidden size of 384. The input to the discriminator is the output of the generator. The output is a single logit for each time step, indicating the likelihood that the sample came from the distribution of the data. The generator and discriminator were optimized by [PERSON] using \(\beta_{1}\) = 0.5 and \(\beta_{2}\) = 0.94, respectively. The discriminator and generator were optimized alternately during this training. The discriminator's weight attenuation is 1 \(\times\) 10\({}^{-4}\), while the generator does not use heavy attenuation. The discriminator's weight decay is set to 1 \(\times\) 10\({}^{-4}\), while the generator does not use weight attenuation. The learning rates of the generator and discriminator are set to 1 \(\times\) 10\({}^{-5}\) and 1 \(\times\) 10\({}^{-4}\), respectively, and remain constant throughout the training process.  
In each training batch, a joint CTC-Attention loss for supervised data and a generative adversarial loss for unsupervised data were calculated using mixed samples from both supervised and unsupervised data sets. A total of 150 k steps were trained. In each training step, Decoder-1 used 80 pairs of paired samples randomly selected from the supervised  
Figure 4: Schematic diagram of Decoder-2.  
Figure 3: Schematic diagram of Decoder-1.  
data, and Decoder-2 used 80 samples randomly selected from the unlabeled audio data and 80 text samples randomly selected from the unlabeled text data. The entire training process was performed on a single V100 and took about 18 h.