# Real-Time Communication Aid System for Korean Dysarthrie Speech
## 3 Proposed Real-Time Communication Aid System for Korean Dysarthric Speech
### Conformer-Based Korean Dysarthric ASR  
This section discusses the Korean dysarthic ASR model for a real-time communication aid system. The goal is to develop a highly accurate model to ensure the system can be effectively applied in real-world social communication scenarios involving dysarthria. To achieve this, we aim to enhance and develop the existing state-of-the-art Korean dysarthic speech recognition model and demonstrate its superiority through performance comparisons with the improved model. The ASR model training is based on the hybrid CTC/attention model [28]. It has a similar structure to the model built in [27], but uses a conformer structure [32], which is a combination of a convolution and transformer structure, rather than only a transformer structure, as the hierarchy for modelling the attention mechanism. The conventional transformer structure, which uses VGG to extract feature vectors for the input sequence, suffers from low computational efficiency and difficulty in processing long sequences. In addition, since VGG and the transformer operate independently, there is a lack of integration between local and global information at the encoder level. The conformer encoder, on the other hand, combines self-attention and convolution in parallel, which solves computational efficiency and memory issues and captures both global and local patterns within the input sequence, enabling it to effectively learn the temporal dynamics of dysarthic speech and visual features of the speech structure. In particular, Korean is a language where investigation is highly important and word order is relatively free, and it is important to learn a long range of dependencies to understand the context. Therefore, through the dual features of self-attention and convolution, the conformer can effectively model long contextual dependencies while learning phoneme and syllable-level features, which is advantageous for distinguishing subtle differences in Korean pronunciation. Dysarthric speech has highly irregular temporal patterns between pronunciations, with large variability in stress, rhythm, and pitch. The conformer effectively captures these temporal variations, and its local-global learning structure provides an environment in which the complexity of dysarthic speech can be better understood and modeled. Figure 3 illustrates the entire training process of the ASR model. The process begins with inputting speech data \(x(t)\) in wav format, which is then transformed into a log-Mel  
Figure 2: Structural diagram of our proposed real-time dysarthria communication aid system. spectrogram, as described in Equations (1)-(3). In Equation (1), \(\omega(n)\) represents the window function, \(\tau\) denotes the time index, and \(\omega\) corresponds to the frequency index:  
\[\mathrm{X}(\tau,\omega)=\sum_{n=-\infty}^{\infty}x(n)\cdot\omega(n-\tau)\cdot e ^{-j\omega n} \tag{1}\]  
The resulting spectrogram is passed through a Mel filterbank H to generate the Mel spectrogram \(S_{Mel}\):  
\[S_{Mel}=H\cdot|X(\tau,\omega)|^{2} \tag{2}\]  
Subsequently, the log transformation is applied to compress the dynamic range, yielding the log-Mel spectrogram \(S_{LogMel}\):  
\[S_{LogMel}=\log(S_{Mel}+\epsilon), \tag{3}\]  
Here, \(\epsilon\) is a small positive constant added to prevent numerical instability and ensure that the logarithm operation does not encounter undefined values when \(S_{Mel}\) approaches zero. This log-Mel spectrogram serves as the input to the conformer encoder. Both the encoder and decoder perform positional embedding at the early stages of the model, which vectorizes the word order and the order of acoustic features to use as positional information. In particular, during the positional embedding process in the conformer encoder, a CNN is incorporated to more finely learn the local patterns present in the positional information of the dysarthric speech sequence. The CNN generates feature maps reflecting local patterns, which better capture local information and structural relationships at each position.