# Real-Time Communication Aid System for Korean Dysarthrie Speech
## 3 Proposed Real-Time Communication Aid System for Korean Dysarthric Speech
### Korean TTS Model  
Based on the recognition results from the ASR model, we construct a TTS model to generate Korean non-dysarthric speech. For the TTS model architecture, we use Jointly Training FastSpeech2 and HiFi-GAN for end-to-end Text-to-Speech (JETS) [37]. JETS combines the popular TTS architecture FastSpeech2 [38] and the GAN-based speech synthesis model HiFi-GAN [39] for joint training. This approach simplifies the traditional two-stage TTS training process into a single unified training process, allowing for high-quality speech synthesis results. The joint training structure is advantageous for better connecting text and speech with complex phonological rules in Korean. Unlike other TTS models, it additionally utilizes a GAN-based generation structure to regenerate speech, which reduces the noise in the speech output and produces a more natural and high-quality speech. Furthermore, it  
Figure 5: Flowchart of the learning process in GTN-CTC.  
utilizes additional prediction mechanisms such as a pitch predictor to further fine-tune the intonation and more naturally represent the various contextual variations and intonation patterns of Korean endings (question, plain, etc.). This provides an advantage for Korean speech synthesis where intonation is important. The overall learning process of JETS is shown in Figure 6.  
The components of JETS can be broadly divided into three parts: FastSpeech2, HiFiGAN, and the alignment module. In the first stage, FastSpeech2 takes text input and learns the duration \(d\), pitch p, and energy e for each text token through a variance adaptor. The corresponding loss function, \(L_{var}\), is defined in Equation (9) and is based on the predicted values of the feature sequences \(\hat{d}\), \(\hat{p}\), and \(\hat{e}\):  
\[L_{var}=\parallel d-\hat{d}\parallel_{2}+\parallel p-\hat{p}\parallel_{2}+ \parallel e-\hat{e}\parallel_{2} \tag{9}\]  
FastSpeech2 combines the text embeddings and the variance features predicted by the variance adaptor to generate the Mel spectrogram M.  
Then, in the alignment module, the alignment between the Mel spectrogram M and the text embedding E is learned. The objective function for alignment learning can be simplified as shown in Equation (10), and it is computed as shown in Equation (11):  
\[L_{align}=-\log P(M\Big{|}E) \tag{10}\]  
\[P(M\mid E)=\prod\
olimits_{i=1}^{N}P(M_{i}\mid E) \tag{11}\]  
Here, \(N\) represents the number of frames in the Mel spectrogram. This alignment process ensures that the temporal structure of the Mel spectrogram matches the sequence of text embeddings.  
Finally, HiFi-GAN takes the output of the decoder as input and synthesizes the raw waveform. During this process, the adversarial learning between the generator and the discriminator results in the GAN loss function, denoted as \(L_{GAN}\). The overall loss function L for JETS integrates the contributions from the variance adaptor, alignment module, and HiFi-GAN, as defined in Equation (12):  
\[L=\lambda_{GAN}L_{GAN}+\lambda_{var}L_{var}+\lambda_{align}L_{align} \tag{12}\]  
Here, \(\lambda_{GAN}\), \(\lambda_{var}\), and \(\lambda_{align}\) represents the weight of each loss function. Therefore, based on the overall loss function L defined as in Equation (12), JETS converts text to Mel spectrograms through FastSpeech2, learns the alignment between text and Mel spectrograms through the alignment module, and finally generates high-quality raw audio through HiFi-GAN, implementing an efficient and integrated TTS model.  
Figure 6: Flowchart of the learning process in JETS.