# Real-Time Communication Aid System for Korean Dysarthrie Speech
## 3 Proposed Real-Time Communication Aid System for Korean Dysarthric Speech
### Conformer-Based Korean Dysarthric ASR
The vectors converted through embedding are then passed through the encoder layer, which consists of 12 layers in total. The learning process for each layer block is shown in Figure 4.  
Figure 4: Flowchart of the learning process for conformer encoder layer block. Figure 3: Flowchart of the entire learning process for the Korean dysarthric ASR model. In the first step of the layer block, multi-head self-attention (MHSA) [33] is applied. For the given input \(S_{LogMel}\), query (Q), key (K), and value (V) vectors are generated for each word in the input vector sequence. These vectors are then processed using the attention mechanism defined in Equation (4):  
\[\text{Attention}(\text{Q},\text{K},\text{V})=softmax\bigg{(}\frac{Q\text{K}^{T }}{\sqrt{d_{k}}}\bigg{)}V \tag{4}\]  
Here, \(d_{k}\) represents the dimension of the key vector. The attention mechanism computes a weighted sum of the value vectors, where the weights are determined by the similarity between the query and key vectors. This enables the model to selectively emphasize relevant features in the input sequence. Following MHSA, the output is passed through two Feed-Forward Network (FFN) layers [34], which apply nonlinear transformations to further refine the extracted features. The computation in each FFN layer, \(FFN_{1}\) and \(FFN_{2}\), is described by Equation (5):  
\[FFN_{1,2}=Swish(XW_{1}+b_{1})W_{2}+b_{2} \tag{5}\]  
Here, \(X\) represents the input, \(W_{1}\) and \(W_{2}\) are the weight matrices of the linear layers, and \(b_{1}\) and \(b_{2}\) are the bias vectors. The activation function used in this model is Swish [35], which has been shown to improve model performance in various deep learning tasks. The next component, the convolution module, is designed to capture local patterns within the input sequence. This module uses a pointwise convolution and a depthwise convolution for each kernel. These operations are followed by batch normalization and activation functions to stabilize and enhance learning. The resulting features are passed through a normalization layer. The normalization layer integrates multiple operations, including layer normalization, residual connections, and dropout, to produce the final output of the block. The overall process is summarized in Equation (6):  
\[\text{Y}=Dropout(LayerNorm(X+Sublayer(X))) \tag{6}\]  
Here, \(Sublayer(X)\) encompasses all preceding operations, such as MHSA, FFN, and the convolution module. The residual connection ensures stable gradient flow, while layer normalization mitigates internal covariate shifts, and dropout reduces overfitting. Based on the output obtained through the conformer encoder, the decoder generates the output text sequence. Using the MHSA mechanism, the decoder selectively emphasizes important information from the speech sequence during text generation. In this process, CTC learns the alignment between the output of the encoder and the text sequence, ultimately generating the final output text. As a CTC algorithm, we utilize the graph transformer network-based CTC (GTN-CTC) [36] algorithm. Compared to traditional CTC approaches, GTN-CTC is effective in handling long input sequences and large datasets, and its graph structure enables it to manage complex dependencies, resulting in faster computation and better memory efficiency. In addition, Korean has a very diverse morphology with a combination of endings and articles, which are often semantically analyzed on a syllable-by-syllable basis. While the simple linear sequence matching approach of the traditional CTC can handle speech data at the phoneme or syllable level, it is less efficient in a language like Korean, where the number of possible combinations due to phonological variations (alliteration, allophones, contractions, elision) is high. GTN-CTC uses a graph structure to more flexibly and efficiently represent the relationship between pronunciation models and word sequences under these variations and can cope well with the complex syllable structures and word order variations in Korean.