## 1 Introduction  
Recent advances in deep learning architectures with a high degree of parameterization have led to the development of end-to-end approaches, which have demonstrated their effectiveness in achieving state-of-the-art performance in automatic speech recognition (ASR) tasks, as shown in several studies [1; 2; 3]. However, end-to-end speech recognition technology is fundamentally data-driven and necessitates a significant amount of supervised data to achieve satisfactory performance. Unfortunately, collecting such data is a time-consuming and expensive process, making it difficult to leverage this architecture for training speech recognition systems in low-resource languages.  
Current work on end-to-end speech recognition focuses on two approaches: CTC [4; 5; 6] and attention-mechanism-based encoder-decoder [7; 8; 9]. CTC is advantageous as it permits duplicate and blank labels, does not necessitate exact alignment between input and output, and utilizes both forward and backward algorithms. However, CTC assumes independence between the outputs of different time steps, which may not be true for some sequential problems. On the other hand, attention-based encoder-decoders directly learn the mapping from acoustic frames to character sequences by utilizing encoders to convert input acoustic features into sequences of high-level hidden features. The attention module calculates attention weights between the previous decoder output and each frame of the encoder output, allowing the decoder to generate its output from previous output labels along with context vectors. At each output time step, the model emits a character based on the input and target characters' historical context. Attention-based codecs do not rely on assumptions of conditional independence and generally outperform CTC without the need for language models. Nonetheless, the model's training is challenging due to misalignment issues in long input sequences, and the estimated alignment in the attention mechanism is susceptible to noise.  
In the realm of computer vision and natural language processing, researchers have been investigating various hybrid architectures [10; 11; 12] that join CTC and attention mechanisms. These hybrid architectures have been shown to be effective in improving alignment errors and accelerating convergence in speech recognition tasks [13; 14]. Several studies have successfully applied these hybrid models to such tasks. Compared to single-task models, multi-task learning methods can train multiple subtasks of a high-level task by sharing parameters between tasks to some degree, which enhances the generalization ability of the original task.  
This paper proposes an end-to-end system (JSUM) based on multi-task learning using a joint loss approach inspired by previous research. In this approach, supervised learning and unsupervised adversarial learning are considered separate tasks, and their joint loss is used for multi-task learning. The supervised learning task uses a hybrid architecture of CTC and attention mechanisms, while the unsupervised learning task segments unlabeled audio using self-supervised phonetic representations and learns to map these representations to phonemes via adversarial training. The experimental results demonstrate that the proposed multi-task learning model can effectively utilize both supervised and unsupervised datasets within a unified end-to-end framework, which is beneficial for improving the use of limited data. Additionally, the results indicate that unsupervised cross-language representation learning in shared encoders can influence the multi-task learning model's performance and demonstrate its effectiveness in both single-language and cross-language scenarios.