## 3 Speech Recognition Results
### Experimental Setups  
All speech recognition experiments were performed using the ESPnet toolkit [23]. Most of the hyper-parameters followed the default settings provided by the toolkit, especially the recipe of LibriSpeech [33,34]. Each model was jointly trained with the CTC objective function as an auxiliary task [24,26]. We used four 1080 Ti GPUs for the training stage and did not use external language models for the decoding stage. All experiments employed 83-dimensional features, including 80-dimensional log-Mel filterbank coefficients with 3-dimensional pitch features per frame. The features were normalized by the mean and variance for the training set. We removed utterances having more than 3000 frames or more than 400 syllables due to GPU memory efficiency.  
The RNN and Transformer were mostly configured by following the settings previously provided [27,34]. For RNN, the encoder was composed of 2 VGG blocks [27] followed by 5 layers of bidirectional long short-term memory (BLSTM) with 1024 cells in each layer and direction. The attention layer used a location-aware attention mechanism [35] with 1024 units. The decoder was 2 layers of unidirectional long short-term memory (LSTM) with 1024 cells. The training was performed using AdaDelta [36] with early stopping and dropout regularization [27]. For the decoding, we used a beam search algorithm with a beam size of 20 and CTC weight of 0.5.  
For Transformer, the encoder used 12 self-attention blocks stacked on the 2 VGG blocks, and the decoder used 6 self-attention blocks. For every Transformer layer, we used 2048-dimensional feedforward networks. For multi-head attention, we employed two configurations: 4 attention heads with 256 dimensions (\"small Transformer\") and 8 attention heads with 512 dimensions (\"large Transformer\"). The training was performed using Noam [6] without early stopping. For regularization, we also adopted warmup-steps, label smoothing, gradient clipping, and accumulating gradients described previously [27]. For the decoding, we used a beam search algorithm with beam sizes of 10 and 60 and CTC weights of 0.5 and 0.4 for small and large Transformer, respectively. More detailed configurations are described in Appendix A.  
Transcriptions for training and evaluation were generated by the following text processing: first, we used the orthographic transcription in the original transcript consisting of the dual transcription. After that, all punctuation marks such as period and question marks and non-speech symbols such as overlap, breath, and laughter, except for unknown words, are removed from the transcription. Finally, we generate a transcription by removing the \"/\" symbol for the filler-words and the \"+\" symbol for repeated-words and self-correction.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Subset** & **Hours** & **No. Utterances** &
\begin{tabular}{c} **No. Speakers** \\ **(Male/Female)** \\ \end{tabular} & **Filenames** \\ \hline Train & 965.2 & 620,000 & 2000 (923/1077) & KsponSpeech\_00001-620000 \\ Dev & 3.9 & 2545 & 1348 (619/729) & KsponSpeech\_620001-622545 \\ Eval-clean & 2.6 & 3000 & 60 (30/30) & KsponSpeech\_E00001-E03000 \\ Eval-other & 3.8 & 3000 & 60 (30/30) & KsponSpeech\_E03001-E06000 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Data subsets in KsponSpeech.