## 4 LM Domain Adaptation Using Semantically Similar Sentences  
We tried to cover the pronunciation variations and to balance the corpus in terms of AM. Still, due to the lack of CS data occurring in real life, we should take the infrequent occurrence of English words into account in terms of linguistic modeling. For that reason, we considered LM domain adaption using semantically similar sentences as the best way to approximate a target domain in real life. When similar sentences from a large text corpus were searched for, they had to include English words in which we were interested. Figure 4 shows the overall structure of the LM domain adaptation. As shown on the right side of the dotted line, the shallow fusion method incorporating a domain LM was used. The remaining problem was how to extract CS sentences that were semantically similar to the target domain. For this, it is reasonable to utilize a development set (dev. set) as a clue for the target domain. In this experiment, AI and economics lecture domains were chosen as the targets. The semantically similar sentences were extracted from the following three steps:  
* Step 1: Sentences containing very rare English words (domain adaptation 1)  
The domain adaptation DB consists of CS sentences containing very rare English words. In general, the lower the frequency, the less the ambiguity. For example, deep learning is almost definitely AI-related. Accordingly, sentences containing low-frequency English words should be included in a domain DB. Very rare English words can be found by counting occurrences from the general domain. The English words in the dev. set should then be compared to the very rare English words. If there is a match, the sentence is included in the domain DB. Figure 5 shows the steps in detail. * Step 2: Sentences containing more than two English words (domain adaptation 2)  
In general, if there are English words in a sentence, the topic of the sentence is likely close to the target domain. Hence, these sentences should be given preference for inclusion in a domain DB. For this reason, we extracted CS sentences that had many English words from the general domain text corpus. Duplicates of words in a sentence were not allowed. For example, one of the general domain sentences, \"  
* Step 3: Sentences semantically similar to the target domain (domain adaptation 3)  
Recently, bidirectional encoder representations from transformers (BERT) [26] represented the semantic relationships of words in an embedding space well. In this study, we utilized KorBERT [27], which is specialized in Korean, and extracted CS sentences that were semantically similar to the dev. set from the general domain text corpus. Euclidian  
Figure 5: Block diagram of the sentences containing very rare English words. Dev. set, development set; DB, database. Figure 6: Block diagram of the sentences containing more than two English words. Dev. set, development set; DB, database. cosine similarity was adopted to measure the degree of similarity. The cosine similarity for arbitrary sentence vectors \(\mathbf{a}\) and \(\mathbf{b}\) is defined as follows:  
\[\cos\ (\mathbf{a},\mathbf{b}) =\frac{\mathbf{a}\cdot\mathbf{b}}{||\mathbf{a}||\ ||\mathbf{b}||} \tag{1}\]  
Figure 7 describes the steps with the cosine similarity. The extracted sentences for the domain DB were converted into Konglish using the rules. Finally, the domain DB was combined with the base LM using shallow fusion [28], as shown in Equation (2):  
\[\mathbf{y}^{*} =\ \operatorname*{argmax}_{\mathbf{y}}\log p(\mathbf{y}|\mathbf{x})+\lambda_{ \text{Base}}\log p_{\text{LM}_{\text{Base}}}(\mathbf{y})+\lambda_{\text{Domain}}\log p_{ \text{LM}_{\text{Domain}}}(\mathbf{y}) \tag{2}\]  
where \(\mathbf{x}=\ (x_{1},x_{2},\cdots,\ x_{n})\) is a sequence vector consisting of \(n\) elements, \(\mathbf{y}\) is an existing output sequence vector, \(\mathbf{y}^{*}\) is an expected output sequence vector, and \(\lambda_{\text{Base}}\) and \(\lambda_{\text{Domain}}\) are the weights of the base LM and the domain LM, respectively.