## 3 FAQ Classification Using an Encoder-Decoder Neural Network  
Figure 2 shows the overall architecture of the proposed FAQ classification model. As shown in Figure 2, the proposed model consists of an embedding layer, a transformer encoder with attentions, and an RNN decoder. To make the proposed model robust to lexical disagreements, the embedding layer consists of three types of embedding vectors: Fixed word embedding vectors, fine-tuned word embedding vectors, and character-level word embedding vectors using a CNN. We expect that the fixed word embedding vectors represent domain-independent meanings of each word, and the fine-tuned word embedding vectors represent domain-specific meanings of each word. For example, we hope that \"transfer\" has the domain-independent meaning \"move something\" and the domain-specific meaning \"send money\" in a banking domain. We also expect that the character-based word embedding vectors alleviate lexical disagreement problems that are raised by spelling errors. For example, we hope that the misspelled word \"remitence\" has a similar vector representation with \"remittance.\" In Figure 2, \(W_{0}\) and \(E_{0}\) are [CLS] (a special symbol added in front of every input example) and an embedding of [CLS], respectively. \(W_{i}\) except \(W_{0}\) and \(E_{i}\) except \(E_{0}\) are the \(i\)-th word in a sentence, and its embedding vector concatenated with three types of word embedding vectors, respectively. Figure 3 exemplifies three types of word embedding vectors. In Figure 3, \(e_{i}\), \(\hat{e}_{i}\), and \(e_{i}^{char}\) are a fixed word embedding vector, a fine-tuned word embedding vector, and a character-level word embedding vector of the \(i\)-th one among \(n\) w  
Figure 3: Embedding layer of the proposed model. The Korean sentence “VR \(\circ\)\(\circ\)\(\circ\)?” means “What is VR?” in English. Figure 2: Overall architecture of the proposed model. \(S\) (i.e., an input query or a predefined question in a FAQ set), respectively. \(e_{i}^{char}\) is generated by a CNN, as shown in the following equation. \[e_{i}^{char}\ =CNN(c_{1},c_{2},\dots,c_{j},\dots,c_{l}), \tag{1}\]  
where \(c_{j}\) is the \(j\)-th one among \(l\) characters in a word \(w_{i}\). In this paper, a character refers to the Korean characters called _jamo_. A final word embedding vector \(E_{i}\) is represented by the concatenation of \(e_{i}\), \(\hat{e}_{i}\), and \(e_{i}^{char}\), as shown in the following equation:  
\[E_{i}=\Big{[}e_{i};\hat{e}_{i};e_{i}^{char}\Big{]}. \tag{2}\]  
To supplement word embedding vectors with contextual information, we adopt an encoder-decoder neural network in which word embedding vectors are encoded by a transformer's encoder [16]. The output \(T_{i}\) of the transformer's encoder is represented by a multi-head scaled dot-product self-attention mechanism, as shown in the following equations. \[Q=K=V=E \tag{3}\]  
\[Attention(Q,K,V)=softmax\Bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Bigg{)}V, \tag{4}\]  
where \(E\) is one among \(n\) word embedding vectors, and \(Q\), \(K\), and \(V\) are a query, a key, and a value for calculating attentions, respectively. Then, \(d_{k}\) is the size of \(E\) for scaling dot-products. In Equation (4), the query, key, and value are the same vectors according to Equation (3). This case is called self-attention, relating different positions of a single sequence \(E_{1}\), \(E_{2},\dots,E_{n}\), to compute a representation of the sequence. Self-attention has been successfully used in various NLP tasks, such as machine translation, machine reading comprehension, abstractive document summarization, etc. The query, key, and value are first linearly transformed into \(N\) heads. Then, each head is entered into Equation (4). Therefore, the self-attention is calculated \(N\) times, making it so-called multi-headed.