S <sup>2</sup> ‐Net: Machine reading comprehension with SRU‐based self‐matching networks

ETRI JournalVolume 41, Issue 3 p. 371-382 ORIGINAL ARTICLEFree Access S2-Net: Machine reading comprehension with SRU-based self-matching networks Cheoneum Park, Cheoneum Park orcid.org/0000-0001-5386-0483 Department of Computer Science, Kangwon National University, Chuncheon, Rep. of KoreaSearch for more papers by this authorChangki Lee, Corresponding Author Changki Lee leeck@kangwon.ac.kr Department of Computer Science, Kangwon National University, Chuncheon, Rep. of Korea Correspondence Changki Lee, Kangwon National University, Chuncheon, Rep. of Korea. Email: leeck@kangwon.ac.krSearch for more papers by this authorLynn Hong, Lynn Hong SKtelecom, Seoul, Rep. of KoreaSearch for more papers by this authorYigyu Hwang, Yigyu Hwang MindsLab, Seoul, Rep. of KoreaSearch for more papers by this authorTaejoon Yoo, Taejoon Yoo MindsLab, Seoul, Rep. of KoreaSearch for more papers by this authorJaeyong Jang, Jaeyong Jang LG Uplus, Seoul, Rep. of KoreaSearch for more papers by this authorYunki Hong, Yunki Hong Naver, Seongnam, Rep. of KoreaSearch for more papers by this authorKyung-Hoon Bae, Kyung-Hoon Bae LG Uplus, Seoul, Rep. of KoreaSearch for more papers by this authorHyun-Ki Kim, Hyun-Ki Kim SW and Contents Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this author Cheoneum Park, Cheoneum Park orcid.org/0000-0001-5386-0483 Department of Computer Science, Kangwon National University, Chuncheon, Rep. of KoreaSearch for more papers by this authorChangki Lee, Corresponding Author Changki Lee leeck@kangwon.ac.kr Department of Computer Science, Kangwon National University, Chuncheon, Rep. of Korea Correspondence Changki Lee, Kangwon National University, Chuncheon, Rep. of Korea. Email: leeck@kangwon.ac.krSearch for more papers by this authorLynn Hong, Lynn Hong SKtelecom, Seoul, Rep. of KoreaSearch for more papers by this authorYigyu Hwang, Yigyu Hwang MindsLab, Seoul, Rep. of KoreaSearch for more papers by this authorTaejoon Yoo, Taejoon Yoo MindsLab, Seoul, Rep. of KoreaSearch for more papers by this authorJaeyong Jang, Jaeyong Jang LG Uplus, Seoul, Rep. of KoreaSearch for more papers by this authorYunki Hong, Yunki Hong Naver, Seongnam, Rep. of KoreaSearch for more papers by this authorKyung-Hoon Bae, Kyung-Hoon Bae LG Uplus, Seoul, Rep. of KoreaSearch for more papers by this authorHyun-Ki Kim, Hyun-Ki Kim SW and Contents Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this author First published: 07 April 2019 https://doi.org/10.4218/etrij.2017-0279Citations: 12 Funding information This work was supported by Institute for Information & Communications Technology Promotion (IITP) grants funded by the Korean government (MSIT) (2018-0-00605, Artificial Intelligence Contact Center Solution; 2013-0-00131, Development of Knowledge Evolutionary WiseQA Platform Technology for Human Knowledge Augmented Services). AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Machine reading comprehension is the task of understanding a given context and finding the correct response in that context. A simple recurrent unit (SRU) is a model that solves the vanishing gradient problem in a recurrent neural network (RNN) using a neural gate, such as a gated recurrent unit (GRU) and long short-term memory (LSTM); moreover, it removes the previous hidden state from the input gate to improve the speed compared to GRU and LSTM. A self-matching network, used in R-Net, can have a similar effect to coreference resolution because the self-matching network can obtain context information of a similar meaning by calculating the attention weight for its own RNN sequence. In this paper, we construct a dataset for Korean machine reading comprehension and propose an S2-Net model that adds a self-matching layer to an encoder RNN using multilayer SRU. The experimental results show that the proposed S2-Net model has performance of single 68.82% EM and 81.25% F1, and ensemble 70.81% EM, 82.48% F1 in the Korean machine reading comprehension test dataset, and has single 71.30% EM and 80.37% F1 and ensemble 73.29% EM and 81.54% F1 performance in the SQuAD dev dataset. 1 INTRODUCTION Machine reading comprehension refers to a machine's ability to understand a given context and determine the correct answer to a question in the context of question-answering applications. For example, machine reading comprehension should be able to understand the context “2004년 건조기 시장에… 의류 건조기 중 LG전자는 점유율 77.4%로 1위를 차지했다. (Machine dryer market in 2004 … LG Electronics ranked first with 77.4% of clothes dryers.)” and determine the correct answer output “LG 전자 (LG Electronics)” in the context of a question like “국내 건조기 시장 점유율 1위 누구야? (Who is number 1 in the Korean dryer market share?)”. Machine reading comprehension has been performed on datasets, such as SQuAD of Stanford, bAbi of Facebook, and MS-MARCO of Microsoft 1-3. Deep-learning models, such as DrQA, fastQA, R-Net, AoA reader, bidirectional flow (BiDAF), and Match-LSTM 4-9, are currently the focus of research. These deep-learning models perform encoding and matching of a given context and question, and they use a pointer network model 10 based on an attention mechanism 11 to generate boundary indices (start and end positions) of the correct answer. The self-matching network used in R-Net has a similar effect to coreference resolution because the self-matching network can obtain contextual information of a similar meaning by calculating the attention weight for its own recurrent neural network (RNN) sequence. A simple recurrent unit (SRU) is a model that solves the vanishing gradient problem in RNNs by using a neural gate, such as a gated recurrent unit (GRU) 12 or long short-term memory (LSTM) 13. SRU simplifies the calculation process of memory cells compared to GRU and LSTM by eliminating previous hidden states from the gate input; moreover, it performs parallelization and CUDA-level optimization to show a similar speed to that of a convolutional neural network (CNN), and is 5−10 times faster than LSTM optimized for the CUDA deep neural network library. Moreover, SRU includes a highway network to improve the performance when stacking multilayers 14. In this paper, we construct a dataset for Korean machine reading comprehension and propose an SRU-based self-matching network (S2-Net) to the encoder RNN using multilayer SRU. To summarize, the main contributions of our model are Model combination and performance enhancement: Our model is based on DrQA and fastQA, and it performs additional feature extraction on the question input sequence. We model hidden states by adding alignment weights between words with similar meanings by applying a self-matching network. Our model shows better performance than DrQA for the Korean dataset and SQuAD dataset in Chapter 6. SRU application for speed improvement: The deep-learning model requires more computation as the depth of the network increases. We apply SRU to improve the training and testing operation speed of S2-Net, a more complex model than DrQA or fastQA. Korean dataset construction and model application: In this paper, we show Korean in the SQuAD style, such as the MindsMRC dataset. We also show the performance for the Korean and SQuAD datasets using S2-Net. 2 SIMPLE RECURRENT UNIT SRU is a new recurrent unit model that solves the vanishing (or exploding) gradient problem caused by backpropagation of RNNs with a neural gate, such as GRU and LSTM, and improves the speed by removing the previous hidden state from the gate input 14. SRU uses an input gate it, forget gate ft, reset gate rt, and highway network. The equation is as follows. (1) (2) (3) (4) (5) (6) The input gate it determines whether to reflect the input information by performing element-wise multiplication with , and the forget gate ft determines whether to reflect the previous internal state information by performing element-wise multiplication with ct−1. is the result of linearly transforming input xt by weight Wf, where it is it = 1 − ft given by (2) and ft is the result of performing a feed-forward neural network (FFNN) on input xt and applying the sigmoid function. The forget gate of the existing RNN models (GRU, LSTM) includes the previous hidden state Rht−1 as ft = σ(Wrxt + Rht−1 + bt). In contrast, SRU applies FFNN to the gate calculation to reduce the amount of computation and enable parallel computation. ct is an internal state that controls the information transfer from the input xt and the previous internal state ct−1, and applies the activation function g(·) to produce the output of the internal state. The hidden state ht is the result of executing the highway network for the internal state output and the input xt. In this case, the internal state output g(ct) determines the extent to which the internal state output is reflected in the hidden state by performing element-wise multiplication with reset gate rt. The input xt is calculated using () and element-wise multiplication to determine whether to reflect the input xt. Figure 1 shows the SRU. Figure 1Open in figure viewerPowerPoint Simple recurrent unit 3 KOREAN READING COMPREHENSION WITH SRU-BASED SELF-MATCHING NETWORKS (S2-NET) To perform machine reading comprehension, each model is provided a dataset composed of questions (Q), passages (P), and answers (Y). Each question has m words as Q = {q1, q2, …, qm}, and each passage consists of n words as P = {p1, p2, …, pn}. S2-Net encodes the question and passage to vector representation, and the pointer network outputs a start boundary y1(Pstart) and end boundary y2(Pend). In this paper, we use S2-Net to perform Korean machine reading comprehension, and the S2-Net model is shown in Figure 2. S2-Net performs feature embedding for paragraphs and questions in the feature layer, and performs paragraph encoding and question encoding in the hidden layer. Self-attention is applied to a paragraph encoder vector in the self-matching layer, and the modeling layer models the paragraph encoder vector. Then, output layer points to the answer. The formula for the feature layer is as follows. (7) (8) and are the input feature vectors with word embedding, and the extracted features are as follows ( is given as the input instead of in the case of a question). Figure 2Open in figure viewerPowerPoint SRU-based self-matching network (S2-Net) architecture Word embedding: femb(pt) = E(pt) Character embedding: fc_emb(pt) = CNN(pt) Exact match: Token feature: ftf(pt) = TF(pt)/T Aligned sentence embedding: The embedding layer E(pt) is responsible for mapping each word to its corresponding n-dimensional representation. We used a word-embedding method that uses 2-year news articles of approximately 100,000 words learned from the neural network language model (NNLM) 15. In this paper, we use a CNN 16 CNN(pt) for character embedding CNN(pt), which sets an arbitrary initial value. The character embeddinglayer maps each word to a high-dimensional vector space. Each character in a word is embedded into vectors using n-size filters, and each output is max-pooled to obtain a fixed-size vector for each word. The exact match feature 4, 5 is a binary feature (1 or 0), which verifies that word pt in the paragraph is included in the question, and each word of the paragraph and the question consists of “morpheme/POS tag”. The token feature normalizes the frequency of each word (TF(pt)/T, where T is the length of each sequence for the question or passage). The aligned sentence embedding calculates a matching context vector that is multiplied by both the context encoder vector and an alignment score for the paragraph and question representation. We use a single dense layer α(.) with ReLU nonlinearity. The layer is applied to the output of the word-embedding layer. Our model makes by encoding the question feature vector . To use in the output layer, we apply attention mechanism on the encoding hidden state . The hidden states of the question sentence to create an alignment vector bj, and we compute the question vector q used for both the hidden state of the question sentence and the alignment vector bj as in the following formula. w is a weight vector to learn. (9) (10) The hidden layer that performs paragraph encoding consists of bidirectional SRU (BiSRU), and the formula is as follows, where is the hidden state encoded for the paragraph input hidden state . (11) is used as an input to the modeling layer , and is concatenated with the context vector Ct of the self-matching layer, such as , to encode. is the gated attention-based recurrent network (R-Net), which performs the element-wise product for and gt, which is the nonlinear gate layer with the sigmoid function applied. The formula for the modeling layer is as follows. (12) (13) (14) The self-matching layer based on self-matching attention makes context vector ct from the whole passage for words in passage. The alignment score is calculated by assigning a given sequence as the input (that is, itself) and adjusts the attention weight by calculating a high alignment score between similar hidden states in the input sequence. (15) (16) (17) S2-Net computes the query vector q and , which is created on the modeling layer, as a bilinear sequence attention, and determines the correct answer location for the question in the paragraph. At this time, the output result is the position of the start (Pstart) and end (Pend) of the answer span, is a weight matrix for the start position, and is a weight matrix for the end position, as follows. (18) 4 COMPARISON WITH RELATED NETWORKS Existing studies to solve machine reading comprehension include the pointer networks based on the attention mechanism, and there are models such as DrQA, fastQA, R-Net, and BiDAF. The proposed S2-Net improves the learning speed and performance for machine reading comprehension by using multiple layers of SRU in the encoder RNN, and we add character (or syllable) representation, some features for the question sentence, and the self-matching layer of R-Net to our model. 4.1 FastQA FastQA is simply composed of an embedding layer, encoder RNN, and answer layer. In the embedding layer, word representations and a highway network are applied to the input sequence, and an exact match for each word and aligned sentence representation for question features are extracted and concatenated. In this study, we perform encoding based on SRU in hidden layers. The SRU includes the highway network; thus, there is no need for an additional highway layer, and more layers can be stacked because of the small amount of computation. The encoder RNN of FastQA uses bidirectional LSTM and shares the weight matrix of the question and paragraph. Thereafter, FFNN is performed, where the weight matrix is applied independently to the question and paragraph vectors. In this study, question and paragraph matching are performed by extracting both the question and paragraph for the aligned sentence representation feature rather than sharing them when encoding the vector of the question and the paragraph. Finally, in the answer layer, FastQA constructs a question vector q, as in this paper, and it applies the ReLU-based two-layer FFNN with the paragraph-encoding vector to output the beginning and end of the answer. In this study, we calculate the question vector q and the modeling vector as bilinear sequence attention without the ReLU-based two-layer FFNN. 4.2 DrQA DrQA consists of a document-retriever module for finding documents related to a question on the web and a document-reader module that performs machine reading comprehension to determine the correct answer to the question from the documents found. Encoder RNN of DrQA is composed of bidirectional GRU. In this study, we apply the SRU, which has a greater learning speed and higher performance as the layers, including the highway network, are stacked. DrQA does not include the self-matching layer, does not use character representation, and uses the feature vector composed of word representation, exact match, token features, and aligned question representation. Here, the additional token features used are not only TF(pt) but also the part-of-speech (POS) information POS(pt) and the named-entity information NER(pt). In this paper, we did not use any additional features for NER and POS tag information because the input word is a morpheme/POS tag. The token feature is defined below. (19) The layer after the paragraph encoding uses the same method in this study (DrQA), with the exception of the self-matching layer. The input of the function to calculate the output result is given by the paragraphs feature embedding pt instead of the encoding of the modeling layer of this study. (20) (21) 4.3 R-Net R-Net matches the question and paragraph in a gated attention-based matching layer to create a paragraph representation containing the meaning of the question and calculates an alignment score for the matched paragraph representation based on a self-matching attention mechanism. Then, it outputs a result by multiplying the alignment score with an encoded hidden state. In the case of R-Net, word and character representation is used as described in this paper, but instead of the aligned sentence representation, the attention weight is calculated in the question-paragraph matching layer and applied to a hidden state to perform the modeling for the hidden state. R-Net models the question encoding as self-attention and obtains the probability distribution of the attention scores of the output results along with the paragraph encoding when calculating the answer attention weights by pointer networks in the output layer. R-Net applies the concatenation score to all attention mechanisms, but unlike the attention score method of R-Net, this study uses pointer networks based on bilinear sequence attention to output the position of the answer boundary. 4.4 BiDAF BiDAF is a six-layer hierarchical process model based on a bidirectional attention-flow mechanism. The bidirectional attention flow refers to Query2Context and Context2Query 8. Here, H is the hidden state from the context word vectors and U is hidden state from query word vectors. When calculating the alignment score, Query2Context and Context2Query are calculated together with the paragraph encoding H using β(·) to create the attention weight G. Then, bidirectional RNN is performed by inputting G at the modeling layer to produce a new encoding M. (22) (23)where β(·) is a trainable vector function, such as FFNN after concatenation of inputs 8. The calculated G and encoded M are concatenated to each other at the output layer to output the answer to the question, and the concatenated hidden state is calculated with the linear attention weight to obtain the start (Pstart) and end (Pend) of the answer. (24) (25) In this study, unlike the bidirectional attention-flow mechanism of BiDAF, we create a paragraph-encoding vector using the aligned sentence representation feature, which is modeled in the self-matching layer to make the input , and bilinear sequence attention is performed in the output layer. 5 KOREAN MACHINE READING COMPREHENSION DATASET The dataset of Korean machine reading comprehension consists of paragraphs and question-answer pairs collected from the news and Wikipedia for entertainment and the general domain, and follows a format similar to the SQuAD dataset, as shown in Figure 3. Figure 3Open in figure viewerPowerPoint Korean machine reading comprehension dataset structure The dataset (MindsMRC dataset) structure of the Korean machine reading comprehension is based on JSON. In Figure 3, “set” is the key that contains the entire dataset; “data” is the list containing all the data in the current dataset; and “paragraphs” is the list of paragraphs, questions, and answers corresponding to the title of the document. “Context_original” is the raw text collected from the news or Wikipedia, “dp” is a list containing the dependency-parsing results of the text corresponding to context_original, “qas” is a list containing the question and answer information of the document, and “context” is a list containing the morpheme information corresponding to the raw text. The qas is composed of the “question” and “answer” information of the document. It includes the questions, including the morpheme result of the raw question text, and the answer key, including information corresponding to the question. The answer to the question is an answer that consists of answer_start and answer_end, which are the beginning and end index information, respectively, of the answer in the document. An example of the Korean machine reading comprehension dataset is presented in Table 1. Table 1. Example of Korean machine reading comprehension dataset Title 장마철에도 빳빳하게… 물 만난 의류건조기 Stiff in the rainy season … well-sold clothes dryers Paragraphs Context_original 2004년 건조기 시장에 가장 먼저 뛰어든 LG전자를 비롯해 올해 초 삼성전자와 중견 기업까지 건조기 판매에 나서면서 국내 건조기 생산량은 급격히 늘고 있다. 건조기의 대당 판매가격을 고려했을 때 1~2년 내에 연간 시장 규모는 1조 원을 넘을 것으로 예상된다. 국내 건조기 시장은 LG전자가 주도하고 있다. 가격비교사이트 다나와리서치에 따르면 올 1월부터 6월까지 판매된 의류 건조기 중 LG전자는 점유율 77.4%로 1위를 차지했다. 가스식-전기식을 모두 판매하는 LG전자는 올해 초부터 전기식 건조기 사업에 주력하고 있다. 회사는 올해 용량과 사용 편의성을 업그레이드한 트롬 전기식 건조기 신제품 2종을 출시했다. 올해 제품에는 냉매를 순환시켜 발생한 열을 활용하는 ‘인버터 히트펌프’ 기술을 적용했다. LG Electronics, which was the first to enter the dryer market in 2004, started to sell dryers to Samsung Electronics and mid-sized companies earlier this year. Considering the selling price of dryers, the annual market size is expected to exceed KRW 1 trillion within 1-2 years. The domestic dryer market is dominated by LG Electronics. According to Danawa Research, LG Electronics ranked first, with a market share of 77.4%, among clothes dryers sold from January to June this year. LG Electronics, which sells both gas and electric products, has focused on the electric dryer business from the beginning of this year. The company has released two new TROMM electric dryers that have upgraded capacity and usability this year. This year's products use inverter heat-pump technology that utilizes the heat generated by circulating the refrigerant. Context [[[‘2004/sn’, ‘년/nnb’], [‘건조기/nng’], [‘시장/nng’, ‘에/jkb’], [‘가장/mag’], [‘먼저/mag’], [‘뛰어들/vv’, ‘ㄴ/etm’], [‘LG/sl’, ‘전자/nng’, ‘를/jko’], [‘비롯하/vv’, ‘어/ec’], [‘올해/nng’], [‘초/nnb’], [‘삼성전자/nng’, ‘와/jc’], [‘중견/nng’], [‘기업/nng’, ‘까지/jx’], [‘건조기/nng’], [‘판매/nng’, ‘에/jkb’], [‘나서/vv’, ‘면서/ec’], [‘국내/nng’], [‘건조기/nng’], [‘생산량/nng’, ‘은/jx’], [‘급격히/mag’], [‘늘/vv’, ‘고/ec’], [‘있/vx’, ‘다/ef’, ‘./sf’]],…]] Question_original 한국 건조기 시장 점유율 1위 어딘지 알려줘 Who is in the first place in the Korean dryer market? Question [[[‘한국/nng’], [‘건조기/nng’], [‘시장/nng’], [‘점유율/nng’], [‘1/sn’, ‘위/nnb’], [‘어디/np’, ‘이/vcp’, ‘ㄴ지/ec’], [‘알려주/vv’, ‘어/ec’]]] Answers text_original LG전자 LG Electronics text [[[‘LG/sl’, ‘전자/nng’]]][[[‘LG/sl’, ‘Jeonja/nng’]]] answer_start 95 answer_end 97 Table 1 lists examples of title, paragraphs, context_original, context, question_original, question, and answer information. The title represents the title of the document (news or Wikipedia), and the paragraphs represent the passage of the document and the question-answer pair. The context_original and question_original are raw texts of the paragraphs and questions, and we produce the results of morphological analysis, such as the context and question, to learn and predict with S2-Net. The answers contain the raw text (text_original) of the answer and the result (text) of the morphological analysis and consist of the start position (answer_start) and end position (answer_end) of the answer text in the paragraph. 6 EXPERIMENTAL RESULT The proposed S2-Net was implemented in PyTorch, and the experiments was performed on a computer with Intel i7-4790 CPU (3.60 GHz), 32-GB RAM, TITAN X (Pascal), and Ubuntu 16.04 OS. The dataset used in the experiment consisted of 55,088 documents and 139,583 questions for the news domain, and 7,119 documents and 17,948 questions for the Wikipedia domain. The training, development (dev), and test sets were divided as listed in Table 2. Table 2. Count of dataset Training set Dev set Test set Number of paragraphs 55,986 6,221 6,220 Number of questions 141,424 8,054 8,054 In this study, we conducted the following experiments on Korean machine reading comprehension using S2-Net. Training was conducted using Adam 17, and the learning rate was set at 0.1. The activation functions for the hidden layers and the attention layer are all tanh, and all the RNN layers used the SRU (CUDA-level optimization). Dropout was fixed at 0.2, the number of dimensions of character embedding was 50, the number of dimensions of word embedding was 100, and the number of dimensions in the hidden layer was 128. The CNN for character embedding used window-size (2, 3, 4, 5, 6) filters, and the size of the filter was set to 30. The size of the minibatch was set to 32, and performance evaluation was performed with the dev set for each epoch. We used the performance measurements of exact match (EM) and F1 1. In this study, we implemented BiDAF and BiDAF+self-matching (BiDAF+SM) with PyTorch and applied DrQA to the Korean dataset. The performance of each RNN type for BiDAF and BiDAF+SM is listed in Table 3, and the number of layer stacks is (3, 1) (hidden layer, modeling layer). At this time, BiDAF and BiDAF+SM do not use features. Table 3. Performance for each RNN type of BiDAF and BiDAF+SM (dev, %) Model Layer stack Modeling layer stack RNN type EM F1 BiDAF 3 1 GRU 55.60 68.82 LSTM 56.61 69.15 SRU 56.46 69.52 BiDAF +SM 3 1 GRU 53.33 66.43 LSTM 54.73 68.63 SRU 54.47 68.74 The experimental results showed that the performance of BiDAF is the lowest, at 68.82% F1, and the performance is improved in the order of LSTM and SRU. The SRU was 69.52% F1 (56.46% EM) and exhibited the best performance among the RNN types. In the case of BiDAF+SM, the performance was improved in the order of GRU, SRU, and LSTM. When we used LSTM, the BiDAF+SM showed 68.74% F1, which shows good performance among the RNN types. The BiDAF+SM model exhibited a performance of 54.73% EM using LSTM (RNN type). Table 4 lists the performance of RNN types for DrQA 4, 14, BiDAF, and BiDAF+SM. In all three models, the features (fexact_match(p), ftf(p), faligned(p)) used in DrQA, were added. DrQA did not use character embedding on CNN, and the hidden layer and modeling layer stacks are presented in Table 4. Table 4. Performance for each RNN type of DrQA, BiDAF, and BiDAF+SM (using features of DrQA, dev, %) Model Layer stack Modeling layer stack RNN type EM F1 DrQA 3 1 GRU 62.58 75.64 LSTM 63.58 76.00 SRU 64.22 77.04 BiDAF 3 1 GRU 64.87 77.45 LSTM 65.69 77.64 SRU 66.00 78.89 BiDAF +SM 3 1 GRU 65.42 77.72 LSTM 66.31 78.38 SRU 64.53 77.81 The experimental results showed that the performance of DrQA improved in the order of GRU, LSTM, and SRU, and when we used the SRU, the DrQA model showed 77.04% F1, which is the best performance among the RNN types. BiDAF and BiDAF+SM showed the same pattern as the experimental results in Table 3. The BiDAF model, which was added to features of DrQA, had 78.89% F1, which is an improvement of 9.37% over the result of BiDAF in Table 3. Second, the performance of BiDAF+SM was 78.38% F1. In the case of EM performance, the BiDAF+SM model was 66.31% better than the others. Table 5 lists the optimized results of the stacking layer for the DrQA, BiDAF, and BiDAF+SM presented in Table 4. The RNN type used was SRU. We performed optimization using the features of DrQA (DrQA: fexact_match(p), ftf(p), faligned(p)) and the features of our model (Ours: fexact_match(p), ftf(p), faligned(p), fexact_match(q), ftf(q), faligned(p), character CNN). Table 5. Optimization of layer stack and features for DrQA, BiDAF, and BiDAF+SM (dev, %) Model Layer stack Modeling layer stack Features EM F1 DrQA 3 1 DrQA 64.22 77.04 5 2 DrQA 64.99 77.98 Ours 67.38 79.90 BiDAF 3 1 DrQA 66.00 78.89 5 2 DrQA 65.94 78.78 Ours 65.94 78.54 BiDAF +SM 3 1 DrQA 64.53 77.81 5 2 DrQA 57.76 71.72 Ours 63.71 76.68 The experimental results showed that the optimal performance of the DrQA model, which was better than other models, was 67.38% EM and 89.90% F1 when we used the features proposed in this paper, and the stack layer was [5, 1]. BiDAF exhibited the best performance of 66.00% EM and 78.89% F1 when using the stack layer [3, 1] and the features of DrQA, and BiDAF+SM also showed 64.53% EM and 77.81% F1 when using the method of BiDAF. Table 6 presents the performance of the proposed S2-Net by RNN type. The stack layer was [3, 1] and we used all the features proposed in this paper. Table 6. Performance for each RNN type of S2-Net (dev, %) Model Layer stack [hidden, modeling] RNN type EM F1 document/s S2-Net [3, 1] GRU 65.74 78.01 70 LSTM 63.29 75.73 51 SRU 68.33 80.87 177 S2-Net exhibited better pe