prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the model performance on target language reading comprehension?
Context 1: Introduction
Reading Comprehension (RC) has become a central task in natural language processing, with great practical value in various industries  In recent years, many large-scale RC datasets in English BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 have nourished the development of numerous powerful and diverse RC models BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11  The state-of-the-art model BIBREF12 on SQuAD, one of the most widely used RC benchmarks, even surpasses human-level performance  Nonetheless, RC on languages other than English has been limited due to the absence of sufficient training data  Although some efforts have been made to create RC datasets for Chinese BIBREF13, BIBREF14 and Korean BIBREF15, it is not feasible to collect RC datasets for every language since annotation efforts to collect a new RC dataset are often far from trivial  Therefore, the setup of transfer learning, especially zero-shot learning, is of extraordinary importance 
Existing methods BIBREF16 of cross-lingual transfer learning on RC datasets often count on machine translation (MT) to translate data from source language into target language, or vice versa  These methods may not require a well-annotated RC dataset for the target language, whereas a high-quality MT model is needed as a trade-off, which might not be available when it comes to low-resource languages 
In this paper, we leverage pre-trained multilingual language representation, for example, BERT learned from multilingual un-annotated sentences (multi-BERT), in cross-lingual zero-shot RC  We fine-tune multi-BERT on the training set in source language, then test the model in target language, with a number of combinations of source-target language pair to explore the cross-lingual ability of multi-BERT  Surprisingly, we find that the models have the ability to transfer between low lexical similarity language pair, such as English and Chinese  Recent studies BIBREF17, BIBREF12, BIBREF18 show that cross-lingual language models have the ability to enable preliminary zero-shot transfer on simple natural language understanding tasks, but zero-shot transfer of RC has not been studied  To our knowledge, this is the first work systematically exploring the cross-lingual transferring ability of multi-BERT on RC tasks 
Zero-shot Transfer with Multi-BERT
Multi-BERT has showcased its ability to enable cross-lingual zero-shot learning on the natural language understanding tasks including XNLI BIBREF19, NER, POS, Dependency Parsing, and so on  We now seek to know if a pre-trained multi-BERT has ability to solve RC tasks in the zero-shot setting 
Zero-shot Transfer with Multi-BERT ::: Experimental Setup and Data
We have training and testing sets in three different languages: English, Chinese and Korean  The English dataset is SQuAD BIBREF2  The Chinese dataset is DRCD BIBREF14, a Chinese RC dataset with 30,000+ examples in the training set and 10,000+ examples in the development set  The Korean dataset is KorQuAD BIBREF15, a Korean RC dataset with 60,000+ examples in the training set and 10,000+ examples in the development set, created in exactly the same procedure as SQuAD  We always use the development sets of SQuAD, DRCD and KorQuAD for testing since the testing sets of the corpora have not been released yet 
Next, to construct a diverse cross-lingual RC dataset with compromised quality, we translated the English and Chinese datasets into more languages, with Google Translate  An obvious issue with this method is that some examples might no longer have a recoverable span  To solve the problem, we use fuzzy matching to find the most possible answer, which calculates minimal edit distance between translated answer and all possible spans  If the minimal edit distance is larger than min(10, lengths of translated answer - 1), we drop the examples during training, and treat them as noise when testing  In this way, we can recover more than 95% of examples  The following generated datasets are recovered with same setting 
The pre-trained multi-BERT is the official released one  This multi-lingual version of BERT were pre-trained on corpus in 104 languages  Data in different languages were simply mixed in batches while pre-training, without additional effort to align between languages  When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged 
Zero-shot Transfer with Multi-BERT ::: Experimental Results
Table TABREF6 shows the result of different models trained on either Chinese or English and tested on Chinese  In row (f), multi-BERT is fine-tuned on English but tested on Chinese, which achieves competitive performance compared with QANet trained on Chinese  We also find that multi-BERT trained on English has relatively lower EM compared with the model with comparable F1 scores  This shows that the model learned with zero-shot can roughly identify the answer spans in context but less accurate  In row (c), we fine-tuned a BERT model pre-trained on English monolingual corpus (English BERT) on Chinese RC training data directly by appending fastText-initialized Chinese word embeddings to the original word embeddings of English-BERT  Its F1 score is even lower than that of zero-shot transferring multi-BERT (rows (c) v s  (e))  The result implies multi-BERT does acquire better cross-lingual capability through pre-training on multilingual corpus  Table TABREF8 shows the results of multi-BERT fine-tuned on different languages and then tested on English , Chinese and Korean  The top half of the table shows the results of training data without translation  It is not surprising that when the training and testing sets are in the same language, the best results are achieved, and multi-BERT shows transfer capability when training and testing sets are in different languages, especially between Chinese and Korean 
In the lower half of Table TABREF8, the results are obtained by the translated training data  First, we found that when testing on English and Chinese, translation always degrades the performance (En v s  En-XX, Zh v s  Zh-XX)  Even though we translate the training data into the same language as testing data, using the untranslated data still yield better results  For example, when testing on English, the F1 score of the model training on Chinese (Zh) is 53 8, while the F1 score is only 44 1 for the model training on Zh-En  This shows that translation degrades the quality of data  There are some exceptions when testing on Korean  Translating the English training data into Chinese, Japanese and Korean still improve the performance on Korean  We also found that when translated into the same language, the English training data is always better than the Chinese data (En-XX v s  Zh-XX), with only one exception (En-Fr v s  Zh-Fr when testing on KorQuAD)  This may be because we have less Chinese training data than English  These results show that the quality and the size of dataset are much more important than whether the training and testing are in the same language or not 
Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Machine Translation
Table TABREF8 shows that fine-tuning on un-translated target language data achieves much better performance than data translated into the target language  Because the above statement is true across all the languages, it is a strong evidence that translation degrades the performance We notice that the translated corpus and untranslated corpus are not the same  This may be another factor that influences the results  Conducting an experiment between un-translated and back-translated data may deal with this problem 
Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Other Factors
Here we discuss the case that the training data are translated  We consider each result is affected by at least three factors: (1) training corpus, (2) data size, (3) whether the source corpus is translated into the target language  To study the effect of data-size, we conducted an extra experiment where we down-sampled the size of English data to be the same as Chinese corpus, and used the down-sampled corpus to train  Then We carried out one-way ANOVA test and found out the significance of the three factors are ranked as below: (1) > (2) >> (3)  The analysis supports that the characteristics of training data is more important than translated into target language or not  Therefore, although translation degrades the performance, whether translating the corpus into the target language is not critical 
What Does Zero-shot Transfer Model Learn? ::: Unseen Language Dataset
It has been shown that extractive QA tasks like SQuAD may be tackled by some language independent strategies, for example, matching words in questions and context BIBREF20  Is zero-shot learning feasible because the model simply learns this kind of language independent strategies on one language and apply to the other?
To verify whether multi-BERT largely counts on a language independent strategy, we test the model on the languages unseen during pre-training  To make sure the languages have never been seen before, we artificially make unseen languages by permuting the whole vocabulary of existing languages  That is, all the words in the sentences of a specific language are replaced by other words in the same language to form the sentences in the created unseen language  It is assumed that if multi-BERT used to find answers by language independent strategy, then multi-BERT should also do well on unseen languages  Table TABREF14 shows that the performance of multi-BERT drops drastically on the dataset  It implies that multi-BERT might not totally rely on pattern matching when finding answers 
What Does Zero-shot Transfer Model Learn? ::: Embedding in Multi-BERT
PCA projection of hidden representations of the last layer of multi-BERT before and after fine-tuning are shown in Fig  FIGREF15  The red points represent Chinese tokens, and the blue points are for English  The results show that tokens from different languages might be embedded into the same space with close spatial distribution  Even though during the fine-tuning only the English data is used, the embedding of the Chinese token changed accordingly  We also quantitatively evaluate the similarities between the embedding of the languages  The results can be found in the Appendix 
What Does Zero-shot Transfer Model Learn? ::: Code-switching Dataset
We observe linguistic-agnostic representations in the last subsection  If tokens are represented in a language-agnostic way, the model may be able to handle code-switching data  Because there is no code-switching data for RC, we create artificial code-switching datasets by replacing some of the words in contexts or questions with their synonyms in another language  The synonyms are found by word-by-word translation with given dictionaries  We use the bilingual dictionaries collected and released in facebookresearch/MUSE GitHub repository  We substitute the words if and only if the words are in the bilingual dictionaries 
Table TABREF14 shows that on all the code-switching datasets, the EM/F1 score drops, indicating that the semantics of representations are not totally disentangled from language  However, the examples of the answers of the model (Table TABREF21) show that multi-BERT could find the correct answer spans although some keywords in the spans have been translated into another language 
What Does Zero-shot Transfer Model Learn? ::: Typology-manipulated Dataset
There are various types of typology in languages  For example, in English the typology order is subject-verb-object (SVO) order, but in Japanese and Korean the order is subject-object-verb (SOV)  We construct a typology-manipulated dataset to examine if the typology order of the training data influences the transfer learning results  If the model only learns the semantic mapping between different languages, changing English typology order from SVO to SOV should improve the transfer ability from English to Japanese  The method used to generate datasets is the same as BIBREF21 
The source code is from a GitHub repository named Shaul1321/rnn_typology, which labels given sentences to CoNLL format with StanfordCoreNLP and then re-arranges them greedily 
Table TABREF23 shows that when we change the English typology order to SOV or OSV order, the performance on Korean is improved and worsen on English and Chinese, but very slightly  The results show that the typology manipulation on the training set has little influence  It is possible that multi-BERT normalizes the typology order of different languages to some extent 
Conclusion
In this paper, we systematically explore zero-shot cross-lingual transfer learning on RC with multi-BERT  The experimental results on English, Chinese and Korean corpora show that even when the languages for training and testing are different, reasonable performance can be obtained  Furthermore, we created several artificial data to study the cross-lingual ability of multi-BERT in the presence of typology variation and code-switching  We showed that only token-level pattern matching is not sufficient for multi-BERT to answer questions and typology variation and code-switching only caused minor effects on testing performance 
Supplemental Material ::: Internal Representation of multi-BERT
The architecture of multi-BERT is a Transformer encoder BIBREF25  While fine-tuning on SQuAD-like dataset, the bottom layers of multi-BERT are initialized from Google-pretrained parameters, with an added output layer initialized from random parameters  Tokens representations from the last layer of bottom-part of multi-BERT are inputs to the output layer and then the output layer outputs a distribution over all tokens that indicates the probability of a token being the START/END of an answer span 
Supplemental Material ::: Internal Representation of multi-BERT ::: Cosine Similarity
As all translated versions of SQuAD/DRCD are parallel to each other  Given a source-target language pair, we calculate cosine similarity of the mean pooling of tokens representation within corresponding answer-span as a measure of how much they look like in terms of the internal representation of multi-BERT  The results are shown in Fig  FIGREF26 
Supplemental Material ::: Internal Representation of multi-BERT ::: SVCCA
Singular Vector Canonical Correlation Analysis (SVCCA) is a general method to compare the correlation of two sets of vector representations  SVCCA has been proposed to compare learned representations across language models BIBREF24  Here we adopt SVCCA to measure the linear similarity of two sets of representations in the same multi-BERT from different translated datasets, which are parallel to each other  The results are shown in Fig FIGREF28 
Supplemental Material ::: Improve Transfering
In the paper, we show that internal representations of multi-BERT are linear-mappable to some extent between different languages  This implies that multi-BERT model might encode semantic and syntactic information in language-agnostic ways and explains how zero-shot transfer learning could be done 
To take a step further, while transfering model from source dataset to target dataset, we align representations in two proposed way, to improve performance on target dataset 
Supplemental Material ::: Improve Transfering ::: Linear Mapping Method
Algorithms proposed in BIBREF23, BIBREF22, BIBREF26 to unsupervisedly learn linear mapping between two sets of embeddings are used here to align representations of source (training data) to those of target  We obtain the mapping generated by embeddings from one specific layer of pre-trained multi-BERT then we apply this mapping to transform the internal representations of multi-BERT while fine-tuning on training data 
Supplemental Material ::: Improve Transfering ::: Adversarial Method
In Adversarial Method, we add an additional transform layer to transform representations and a discrimination layer to discriminate between transformed representations from source language (training set) and target language (development set)  And the GAN loss is applied in the total loss of fine-tuning 
Supplemental Material ::: Improve Transfering ::: Discussion
As table TABREF33 shows, there are no improvements among above methods  Some linear mapping methods even causes devastating effect on EM/F1 scores 

ground_truth: Table TABREF6, Table TABREF8

when testing on English, the F1 score of the model training on Chinese (Zh) is 53.8,  F1 score is only 44.1 for the model training on Zh-En

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they obtain the new context represetation?
Context 2: Introduction
Relation classification is the task of assigning sentences with two marked entities to a predefined set of relations  The sentence “We poured the <e1>milk</e1> into the <e2>pumpkin mixture</e2> ”, for example, expresses the relation Entity-Destination(e1,e2)  While early research mostly focused on support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 , recent research showed performance improvements by applying neural networks (NNs) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 on the benchmark data from SemEval 2010 shared task 8 BIBREF8  
This study investigates two different types of NNs: recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as well as their combination  We make the following contributions:
(1) We propose extended middle context, a new context representation for CNNs for relation classification  The extended middle context uses all parts of the sentence (the relation arguments, left of the relation arguments, between the arguments, right of the arguments) and pays special attention to the middle part 
(2) We present connectionist bi-directional RNN models which are especially suited for sentence classification tasks since they combine all intermediate hidden layers for their final decision  Furthermore, the ranking loss function is introduced for the RNN model optimization which has not been investigated in the literature for relation classification before 
(3) Finally, we combine CNNs and RNNs using a simple voting scheme and achieve new state-of-the-art results on the SemEval 2010 benchmark dataset 
Related Work
In 2010, manually annotated data for relation classification was released in the context of a SemEval shared task BIBREF8   Shared task participants used, i a , support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1   Recently, their results on this data set were outperformed by applying NNs BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6  
zeng2014 built a CNN based only on the context between the relation arguments and extended it with several lexical features  kim2014 and others used convolutional filters of different sizes for CNNs  nguyen applied this to relation classification and obtained improvements over single filter sizes  deSantos2015 replaced the softmax layer of the CNN with a ranking layer  They showed improvements and published the best result so far on the SemEval dataset, to our knowledge 
socher used another NN architecture for relation classification: recursive neural networks that built recursive sentence representations based on syntactic parsing  In contrast, zhang investigated a temporal structured RNN with only words as input  They used a bi-directional model with a pooling layer on top 
Convolutional Neural Networks (CNN)
CNNs perform a discrete convolution on an input matrix with a set of different filters  For NLP tasks, the input matrix represents a sentence: Each column of the matrix stores the word embedding of the corresponding word  By applying a filter with a width of, e g , three columns, three neighboring words (trigram) are convolved  Afterwards, the results of the convolution are pooled  Following collobertWeston, we perform max-pooling which extracts the maximum value for each filter and, thus, the most informative n-gram for the following steps  Finally, the resulting values are concatenated and used for classifying the relation expressed in the sentence 
Input: Extended Middle Context
One of our contributions is a new input representation especially designed for relation classification  The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context  Since in most cases the middle context contains the most relevant information for the relation, we want to focus on it but not ignore the other regions completely  Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context  Due to the repetition of the middle context, we force the network to pay special attention to it  The two contexts are processed by two independent convolutional and max-pooling layers  After pooling, the results are concatenated to form the sentence representation  Figure FIGREF3 depicts this procedure  It shows an examplary sentence: “He had chest pain and <e1>headaches</e1> from <e2>mold</e2> in the bedroom ” If we only considered the middle context “from”, the network might be tempted to predict a relation like Entity-Origin(e1,e2)  However, by also taking the left and right context into account, the model can detect the relation Cause-Effect(e2,e1)  While this could also be achieved by integrating the whole context into the model, using the whole context can have disadvantages for longer sentences: The max pooling step can easily choose a value from a part of the sentence which is far away from the mention of the relation  With splitting the context into two parts, we reduce this danger  Repeating the middle context increases the chance for the max pooling step to pick a value from the middle context 
Convolutional Layer
Following previous work (e g , BIBREF5 , BIBREF6 ), we use 2D filters spanning all embedding dimensions  After convolution, a max pooling operation is applied that stores only the highest activation of each filter  We apply filters with different window sizes 2-5 (multi-windows) as in BIBREF5 , i e  spanning a different number of input words 
Recurrent Neural Networks (RNN)
Traditional RNNs consist of an input vector, a history vector and an output vector  Based on the representation of the current input word and the previous history vector, a new history is computed  Then, an output is predicted (e g , using a softmax layer)  In contrast to most traditional RNN architectures, we use the RNN for sentence modeling, i e , we predict an output vector only after processing the whole sentence and not after each word  Training is performed using backpropagation through time BIBREF9 which unfolds the recurrent computations of the history vector for a certain number of time steps  To avoid exploding gradients, we use gradient clipping with a threshold of 10 BIBREF10  
Input of the RNNs
Initial experiments showed that using trigrams as input instead of single words led to superior results  Hence, at timestep INLINEFORM0 we do not only give word INLINEFORM1 to the model but the trigram INLINEFORM2 by concatenating the corresponding word embeddings 
Connectionist Bi-directional RNNs
Especially for relation classification, the processing of the relation arguments might be easier with knowledge of the succeeding words  Therefore in bi-directional RNNs, not only a history vector of word INLINEFORM0 is regarded but also a future vector  This leads to the following conditioned probability for the history INLINEFORM1 at time step INLINEFORM2 : DISPLAYFORM0
Thus, the network can be split into three parts: a forward pass which processes the original sentence word by word (Equation EQREF6 ); a backward pass which processes the reversed sentence word by word (Equation ); and a combination of both (Equation )  All three parts are trained jointly  This is also depicted in Figure FIGREF7  
Combining forward and backward pass by adding their hidden layer is similar to BIBREF7   We, however, also add a connection to the previous combined hidden layer with weight INLINEFORM0 to be able to include all intermediate hidden layers into the final decision of the network (see Equation )  We call this “connectionist bi-directional RNN” 
In our experiments, we compare this RNN with uni-directional RNNs and bi-directional RNNs without additional hidden layer connections 
Word Representations
Words are represented by concatenated vectors: a word embedding and a position feature vector 
Pretrained word embeddings  In this study, we used the word2vec toolkit BIBREF11 to train embeddings on an English Wikipedia from May 2014  We only considered words appearing more than 100 times and added a special PADDING token for convolution  This results in an embedding training text of about 485,000 terms and INLINEFORM0 tokens  During model training, the embeddings are updated 
Position features  We incorporate randomly initialized position embeddings similar to zeng2014, nguyen and deSantos2015  In our RNN experiments, we investigate different possibilities of integrating position information: position embeddings, position embeddings with entity presence flags (flags indicating whether the current word is one of the relation arguments), and position indicators BIBREF7  
Objective Function: Ranking Loss
Ranking  We applied the ranking loss function proposed in deSantos2015 to train our models  It maximizes the distance between the true label INLINEFORM0 and the best competitive label INLINEFORM1 given a data point INLINEFORM2   The objective function is DISPLAYFORM0
with INLINEFORM0 and INLINEFORM1 being the scores for the classes INLINEFORM2 and INLINEFORM3 respectively  The parameter INLINEFORM4 controls the penalization of the prediction errors and INLINEFORM5 and INLINEFORM6 are margins for the correct and incorrect classes  Following deSantos2015, we set INLINEFORM7   We do not learn a pattern for the class Other but increase its difference to the best competitive label by using only the second summand in Equation EQREF10 during training 
Experiments and Results
We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8   It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other)  8,000 sentences have been distributed as training set and 2,717 sentences served as test set  For evaluation, we applied the official scoring script and report the macro F1 score which also served as the official result of the shared task 
RNN and CNN models were implemented with theano BIBREF12 , BIBREF13   For all our models, we use L2 regularization with a weight of 0 0001  For CNN training, we use mini batches of 25 training examples while we perform stochastic gradient descent for the RNN  The initial learning rates are 0 2 for the CNN and 0 01 for the RNN  We train the models for 10 (CNN) and 50 (RNN) epochs without early stopping  As activation function, we apply tanh for the CNN and capped ReLU for the RNN  For tuning the hyperparameters, we split the training data into two parts: 6 5k (training) and 1 5k (development) sentences  We also tuned the learning rate schedule on dev 
Beside of training single models, we also report ensemble results for which we combined the presented single models with a voting process 
Performance of CNNs
As a baseline system, we implemented a CNN similar to the one described by zeng2014  It consists of a standard convolutional layer with filters with only one window size, followed by a softmax layer  As input it uses the middle context  In contrast to zeng2014, our CNN does not have an additional fully connected hidden layer  Therefore, we increased the number of convolutional filters to 1200 to keep the number of parameters comparable  With this, we obtain a baseline result of 73 0  After including 5 dimensional position features, the performance was improved to 78 6 (comparable to 78 9 as reported by zeng2014 without linguistic features) 
In the next step, we investigate how this result changes if we successively add further features to our CNN: multi-windows for convolution (window sizes: 2,3,4,5 and 300 feature maps each), ranking layer instead of softmax and our proposed extended middle context  Table TABREF12 shows the results  Note that all numbers are produced by CNNs with a comparable number of parameters  We also report F1 for increasing the word embedding dimensionality from 50 to 400  The position embedding dimensionality is 5 in combination with 50 dimensional word embeddings and 35 with 400 dimensional word embeddings  Our results show that especially the ranking layer and the embedding size have an important impact on the performance 
Performance of RNNs
As a baseline for the RNN models, we apply a uni-directional RNN which predicts the relation after processing the whole sentence  With this model, we achieve an F1 score of 61 2 on the SemEval test set 
Afterwards, we investigate the impact of different position features on the performance of uni-directional RNNs (position embeddings, position embeddings concatenated with a flag indicating whether the current word is an entity or not, and position indicators BIBREF7 )  The results indicate that position indicators (i e  artificial words that indicate the entity presence) perform the best on the SemEval data  We achieve an F1 score of 73 4 with them  However, the difference to using position embeddings with entity flags is not statistically significant 
Similar to our CNN experiments, we successively vary the RNN models by using bi-directionality, by adding connections between the hidden layers (“connectionist”), by applying ranking instead of softmax to predict the relation and by increasing the word embedding dimension to 400 
The results in Table TABREF14 show that all of these variations lead to statistically significant improvements  Especially the additional hidden layer connections and the integration of the ranking layer have a large impact on the performance 
Combination of CNNs and RNNs
Finally, we combine our CNN and RNN models using a voting process  For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes  In case of a tie, we pick one of the most frequent classes randomly  The combination achieves an F1 score of 84 9 which is better than the performance of the two NN types alone  It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations 
Comparison with State of the Art
Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models  Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features 
Conclusion
In this paper, we investigated different features and architectural choices for convolutional and recurrent neural networks for relation classification without using any linguistic features  For convolutional neural networks, we presented a new context representation for relation classification  Furthermore, we introduced connectionist recurrent neural networks for sentence classification tasks and performed the first experiments with ranking recurrent neural networks  Finally, we showed that even a simple combination of convolutional and recurrent neural networks improved results  With our neural models, we achieved new state-of-the-art results on the SemEval 2010 task 8 benchmark data 
Acknowledgments
Heike Adel is a recipient of the Google European Doctoral Fellowship in Natural Language Processing and this research is supported by this fellowship 
This research was also supported by Deutsche Forschungsgemeinschaft: grant SCHU 2246/4-2 
Context 3: Introduction
Machine Reading Comprehension (MRC), as the name suggests, requires a machine to read a passage and answer its relevant questions  Since the answer to each question is supposed to stem from the corresponding passage, a common MRC solution is to develop a neural-network-based MRC model that predicts an answer span (i e  the answer start position and the answer end position) from the passage of each given passage-question pair  To facilitate the explorations and innovations in this area, many MRC datasets have been established, such as SQuAD BIBREF0 , MS MARCO BIBREF1 , and TriviaQA BIBREF2   Consequently, many pioneering MRC models have been proposed, such as BiDAF BIBREF3 , R-NET BIBREF4 , and QANet BIBREF5   According to the leader board of SQuAD, the state-of-the-art MRC models have achieved the same performance as human beings  However, does this imply that they have possessed the same reading comprehension ability as human beings?
OF COURSE NOT  There is a huge gap between MRC models and human beings, which is mainly reflected in the hunger for data and the robustness to noise  On the one hand, developing MRC models requires a large amount of training examples (i e  the passage-question pairs labeled with answer spans), while human beings can achieve good performance on evaluation examples (i e  the passage-question pairs to address) without training examples  On the other hand, BIBREF6 revealed that intentionally injected noise (e g  misleading sentences) in evaluation examples causes the performance of MRC models to drop significantly, while human beings are far less likely to suffer from this  The reason for these phenomena, we believe, is that MRC models can only utilize the knowledge contained in each given passage-question pair, but in addition to this, human beings can also utilize general knowledge  A typical category of general knowledge is inter-word semantic connections  As shown in Table TABREF1 , such general knowledge is essential to the reading comprehension ability of human beings 
A promising strategy to bridge the gap mentioned above is to integrate the neural networks of MRC models with the general knowledge of human beings  To this end, it is necessary to solve two problems: extracting general knowledge from passage-question pairs and utilizing the extracted general knowledge in the prediction of answer spans  The first problem can be solved with knowledge bases, which store general knowledge in structured forms  A broad variety of knowledge bases are available, such as WordNet BIBREF7 storing semantic knowledge, ConceptNet BIBREF8 storing commonsense knowledge, and Freebase BIBREF9 storing factoid knowledge  In this paper, we limit the scope of general knowledge to inter-word semantic connections, and thus use WordNet as our knowledge base  The existing way to solve the second problem is to encode general knowledge in vector space so that the encoding results can be used to enhance the lexical or contextual representations of words BIBREF10 , BIBREF11   However, this is an implicit way to utilize general knowledge, since in this way we can neither understand nor control the functioning of general knowledge  In this paper, we discard the existing implicit way and instead explore an explicit (i e  understandable and controllable) way to utilize general knowledge 
The contribution of this paper is two-fold  On the one hand, we propose a data enrichment method, which uses WordNet to extract inter-word semantic connections as general knowledge from each given passage-question pair  On the other hand, we propose an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the above extracted general knowledge to assist its attention mechanisms  Based on the data enrichment method, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them  When only a subset ( INLINEFORM0 – INLINEFORM1 ) of the training examples are available, KAR outperforms the state-of-the-art MRC models by a large margin, and is still reasonably robust to noise 
Data Enrichment Method
In this section, we elaborate a WordNet-based data enrichment method, which is aimed at extracting inter-word semantic connections from each passage-question pair in our MRC dataset  The extraction is performed in a controllable manner, and the extracted results are provided as general knowledge to our MRC model 
Semantic Relation Chain
WordNet is a lexical database of English, where words are organized into synsets according to their senses  A synset is a set of words expressing the same sense so that a word having multiple senses belongs to multiple synsets, with each synset corresponding to a sense  Synsets are further related to each other through semantic relations  According to the WordNet interface provided by NLTK BIBREF12 , there are totally sixteen types of semantic relations (e g  hypernyms, hyponyms, holonyms, meronyms, attributes, etc )  Based on synset and semantic relation, we define a new concept: semantic relation chain  A semantic relation chain is a concatenated sequence of semantic relations, which links a synset to another synset  For example, the synset “keratin n 01” is related to the synset “feather n 01” through the semantic relation “substance holonym”, the synset “feather n 01” is related to the synset “bird n 01” through the semantic relation “part holonym”, and the synset “bird n 01” is related to the synset “parrot n 01” through the semantic relation “hyponym”, thus “substance holonym INLINEFORM0 part holonym INLINEFORM1 hyponym” is a semantic relation chain, which links the synset “keratin n 01” to the synset “parrot n 01”  We name each semantic relation in a semantic relation chain as a hop, therefore the above semantic relation chain is a 3-hop chain  By the way, each single semantic relation is equivalent to a 1-hop chain 
Inter-word Semantic Connection
The key problem in the data enrichment method is determining whether a word is semantically connected to another word  If so, we say that there exists an inter-word semantic connection between them  To solve this problem, we define another new concept: the extended synsets of a word  Given a word INLINEFORM0 , whose synsets are represented as a set INLINEFORM1 , we use another set INLINEFORM2 to represent its extended synsets, which includes all the synsets that are in INLINEFORM3 or that can be linked to from INLINEFORM4 through semantic relation chains  Theoretically, if there is no limitation on semantic relation chains, INLINEFORM5 will include all the synsets in WordNet, which is meaningless in most situations  Therefore, we use a hyper-parameter INLINEFORM6 to represent the permitted maximum hop count of semantic relation chains  That is to say, only the chains having no more than INLINEFORM7 hops can be used to construct INLINEFORM8 so that INLINEFORM9 becomes a function of INLINEFORM10 : INLINEFORM11 (if INLINEFORM12 , we will have INLINEFORM13 )  Based on the above statements, we formulate a heuristic rule for determining inter-word semantic connections: a word INLINEFORM14 is semantically connected to another word INLINEFORM15 if and only if INLINEFORM16  
General Knowledge Extraction
Given a passage-question pair, the inter-word semantic connections that connect any word to any passage word are regarded as the general knowledge we need to extract  Considering the requirements of our MRC model, we only extract the positional information of such inter-word semantic connections  Specifically, for each word INLINEFORM0 , we extract a set INLINEFORM1 , which includes the positions of the passage words that INLINEFORM2 is semantically connected to (if INLINEFORM3 itself is a passage word, we will exclude its own position from INLINEFORM4 )  We can control the amount of the extracted results by setting the hyper-parameter INLINEFORM5 : if we set INLINEFORM6 to 0, inter-word semantic connections will only exist between synonyms; if we increase INLINEFORM7 , inter-word semantic connections will exist between more words  That is to say, by increasing INLINEFORM8 within a certain range, we can usually extract more inter-word semantic connections from a passage-question pair, and thus can provide the MRC model with more general knowledge  However, due to the complexity and diversity of natural languages, only a part of the extracted results can serve as useful general knowledge, while the rest of them are useless for the prediction of answer spans, and the proportion of the useless part always rises when INLINEFORM9 is set larger  Therefore we set INLINEFORM10 through cross validation (i e  according to the performance of the MRC model on the development examples) 
Knowledge Aided Reader
In this section, we elaborate our MRC model: Knowledge Aided Reader (KAR)  The key components of most existing MRC models are their attention mechanisms BIBREF13 , which are aimed at fusing the associated representations of each given passage-question pair  These attention mechanisms generally fall into two categories: the first one, which we name as mutual attention, is aimed at fusing the question representations into the passage representations so as to obtain the question-aware passage representations; the second one, which we name as self attention, is aimed at fusing the question-aware passage representations into themselves so as to obtain the final passage representations  Although KAR is equipped with both categories, its most remarkable feature is that it explicitly uses the general knowledge extracted by the data enrichment method to assist its attention mechanisms  Therefore we separately name the attention mechanisms of KAR as knowledge aided mutual attention and knowledge aided self attention 
Task Definition
Given a passage INLINEFORM0 and a relevant question INLINEFORM1 , the task is to predict an answer span INLINEFORM2 , where INLINEFORM3 , so that the resulting subsequence INLINEFORM4 from INLINEFORM5 is an answer to INLINEFORM6  
Overall Architecture
As shown in Figure FIGREF7 , KAR is an end-to-end MRC model consisting of five layers:
Lexicon Embedding Layer  This layer maps the words to the lexicon embeddings  The lexicon embedding of each word is composed of its word embedding and character embedding  For each word, we use the pre-trained GloVe BIBREF14 word vector as its word embedding, and obtain its character embedding with a Convolutional Neural Network (CNN) BIBREF15   For both the passage and the question, we pass the concatenation of the word embeddings and the character embeddings through a shared dense layer with ReLU activation, whose output dimensionality is INLINEFORM0   Therefore we obtain the passage lexicon embeddings INLINEFORM1 and the question lexicon embeddings INLINEFORM2  
Context Embedding Layer  This layer maps the lexicon embeddings to the context embeddings  For both the passage and the question, we process the lexicon embeddings (i e  INLINEFORM0 for the passage and INLINEFORM1 for the question) with a shared bidirectional LSTM (BiLSTM) BIBREF16 , whose hidden state dimensionality is INLINEFORM2   By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the passage context embeddings INLINEFORM3 and the question context embeddings INLINEFORM4  
Coarse Memory Layer  This layer maps the context embeddings to the coarse memories  First we use knowledge aided mutual attention (introduced later) to fuse INLINEFORM0 into INLINEFORM1 , the outputs of which are represented as INLINEFORM2   Then we process INLINEFORM3 with a BiLSTM, whose hidden state dimensionality is INLINEFORM4   By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the coarse memories INLINEFORM5 , which are the question-aware passage representations

ground_truth: They use two independent convolutional and max-pooling layers on (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. They concatenated the two results after pooling to get the new context representation.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do the authors hypothesize that humans' robustness to noise is due to their general knowledge?
Context 3: 
Refined Memory Layer  This layer maps the coarse memories to the refined memories  First we use knowledge aided self attention (introduced later) to fuse INLINEFORM0 into themselves, the outputs of which are represented as INLINEFORM1   Then we process INLINEFORM2 with a BiLSTM, whose hidden state dimensionality is INLINEFORM3   By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the refined memories INLINEFORM4 , which are the final passage representations 
Answer Span Prediction Layer  This layer predicts the answer start position and the answer end position based on the above layers  First we obtain the answer start position distribution INLINEFORM0 : INLINEFORM1 INLINEFORM2
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents the refined memory of each passage word INLINEFORM4 (i e  the INLINEFORM5 -th column in INLINEFORM6 ); INLINEFORM7 represents the question summary obtained by performing an attention pooling over INLINEFORM8   Then we obtain the answer end position distribution INLINEFORM9 : INLINEFORM10 INLINEFORM11
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents vector concatenation  Finally we construct an answer span prediction matrix INLINEFORM4 , where INLINEFORM5 represents the upper triangular matrix of a matrix INLINEFORM6   Therefore, for the training, we minimize INLINEFORM7 on each training example whose labeled answer span is INLINEFORM8 ; for the inference, we separately take the row index and column index of the maximum element in INLINEFORM9 as INLINEFORM10 and INLINEFORM11  
Knowledge Aided Mutual Attention
As a part of the coarse memory layer, knowledge aided mutual attention is aimed at fusing the question context embeddings INLINEFORM0 into the passage context embeddings INLINEFORM1 , where the key problem is to calculate the similarity between each passage context embedding INLINEFORM2 (i e  the INLINEFORM3 -th column in INLINEFORM4 ) and each question context embedding INLINEFORM5 (i e  the INLINEFORM6 -th column in INLINEFORM7 )  To solve this problem, BIBREF3 proposed a similarity function: INLINEFORM8
where INLINEFORM0 is a trainable parameter; INLINEFORM1 represents element-wise multiplication  This similarity function has also been adopted by several other works BIBREF17 , BIBREF5   However, since context embeddings contain high-level information, we believe that introducing the pre-extracted general knowledge into the calculation of such similarities will make the results more reasonable  Therefore we modify the above similarity function to the following form: INLINEFORM2
where INLINEFORM0 represents the enhanced context embedding of a word INLINEFORM1   We use the pre-extracted general knowledge to construct the enhanced context embeddings  Specifically, for each word INLINEFORM2 , whose context embedding is INLINEFORM3 , to construct its enhanced context embedding INLINEFORM4 , first recall that we have extracted a set INLINEFORM5 , which includes the positions of the passage words that INLINEFORM6 is semantically connected to, thus by gathering the columns in INLINEFORM7 whose indexes are given by INLINEFORM8 , we obtain the matching context embeddings INLINEFORM9   Then by constructing a INLINEFORM10 -attended summary of INLINEFORM11 , we obtain the matching vector INLINEFORM12 (if INLINEFORM13 , which makes INLINEFORM14 , we will set INLINEFORM15 ): INLINEFORM16 INLINEFORM17
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents the INLINEFORM4 -th column in INLINEFORM5   Finally we pass the concatenation of INLINEFORM6 and INLINEFORM7 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM8   Therefore we obtain the enhanced context embedding INLINEFORM9  
Based on the modified similarity function and the enhanced context embeddings, to perform knowledge aided mutual attention, first we construct a knowledge aided similarity matrix INLINEFORM0 , where each element INLINEFORM1   Then following BIBREF5 , we construct the passage-attended question summaries INLINEFORM2 and the question-attended passage summaries INLINEFORM3 : INLINEFORM4 INLINEFORM5
where INLINEFORM0 represents softmax along the row dimension and INLINEFORM1 along the column dimension  Finally following BIBREF17 , we pass the concatenation of INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM6   Therefore we obtain the outputs INLINEFORM7  
Knowledge Aided Self Attention
As a part of the refined memory layer, knowledge aided self attention is aimed at fusing the coarse memories INLINEFORM0 into themselves  If we simply follow the self attentions of other works BIBREF4 , BIBREF18 , BIBREF19 , BIBREF17 , then for each passage word INLINEFORM1 , we should fuse its coarse memory INLINEFORM2 (i e  the INLINEFORM3 -th column in INLINEFORM4 ) with the coarse memories of all the other passage words  However, we believe that this is both unnecessary and distracting, since each passage word has nothing to do with many of the other passage words  Thus we use the pre-extracted general knowledge to guarantee that the fusion of coarse memories for each passage word will only involve a precise subset of the other passage words  Specifically, for each passage word INLINEFORM5 , whose coarse memory is INLINEFORM6 , to perform the fusion of coarse memories, first recall that we have extracted a set INLINEFORM7 , which includes the positions of the other passage words that INLINEFORM8 is semantically connected to, thus by gathering the columns in INLINEFORM9 whose indexes are given by INLINEFORM10 , we obtain the matching coarse memories INLINEFORM11   Then by constructing a INLINEFORM12 -attended summary of INLINEFORM13 , we obtain the matching vector INLINEFORM14 (if INLINEFORM15 , which makes INLINEFORM16 , we will set INLINEFORM17 ): INLINEFORM18 INLINEFORM19
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters  Finally we pass the concatenation of INLINEFORM3 and INLINEFORM4 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM5   Therefore we obtain the fusion result INLINEFORM6 , and further the outputs INLINEFORM7  
Related Works
Attention Mechanisms  Besides those mentioned above, other interesting attention mechanisms include performing multi-round alignment to avoid the problems of attention redundancy and attention deficiency BIBREF20 , and using mutual attention as a skip-connector to densely connect pairwise layers BIBREF21  
Data Augmentation  It is proved that properly augmenting training examples can improve the performance of MRC models  For example, BIBREF22 trained a generative model to generate questions based on unlabeled text, which substantially boosted their performance; BIBREF5 trained a back-and-forth translation model to paraphrase training examples, which brought them a significant performance gain 
Multi-step Reasoning  Inspired by the fact that human beings are capable of understanding complex documents by reading them over and over again, multi-step reasoning was proposed to better deal with difficult MRC tasks  For example, BIBREF23 used reinforcement learning to dynamically determine the number of reasoning steps; BIBREF19 fixed the number of reasoning steps, but used stochastic dropout in the output layer to avoid step bias 
Linguistic Embeddings  It is both easy and effective to incorporate linguistic embeddings into the input layer of MRC models  For example, BIBREF24 and BIBREF19 used POS embeddings and NER embeddings to construct their input embeddings; BIBREF25 used structural embeddings based on parsing trees to constructed their input embeddings 
Transfer Learning  Several recent breakthroughs in MRC benefit from feature-based transfer learning BIBREF26 , BIBREF27 and fine-tuning-based transfer learning BIBREF28 , BIBREF29 , which are based on certain word-level or sentence-level models pre-trained on large external corpora in certain supervised or unsupervised manners 
Experimental Settings
MRC Dataset  The MRC dataset used in this paper is SQuAD 1 1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 )  Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models  The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models  Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage 
Implementation Details  We tokenize the MRC dataset with spaCy 2 0 13 BIBREF30 , manipulate WordNet 3 0 with NLTK 3 3, and implement KAR with TensorFlow 1 11 0 BIBREF31   For the data enrichment method, we set the hyper-parameter INLINEFORM0 to 3  For the dense layers and the BiLSTMs, we set the dimensionality unit INLINEFORM1 to 600  For model optimization, we apply the Adam BIBREF32 optimizer with a learning rate of INLINEFORM2 and a mini-batch size of 32  For model evaluation, we use Exact Match (EM) and F1 score as evaluation metrics  To avoid overfitting, we apply dropout BIBREF33 to the dense layers and the BiLSTMs with a dropout rate of INLINEFORM3   To boost the performance, we apply exponential moving average with a decay rate of INLINEFORM4  
Model Comparison in both Performance and the Robustness to Noise
We compare KAR with other MRC models in both performance and the robustness to noise  Specifically, we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets  As for the comparative objects, we only consider the single MRC models that rank in the top 20 on the SQuAD 1 1 leader board and have reported their performance on the adversarial sets  There are totally five such comparative objects, which can be considered as representatives of the state-of-the-art MRC models  As shown in Table TABREF12 , on the development set and the test set, the performance of KAR is on par with that of the state-of-the-art MRC models; on the adversarial sets, KAR outperforms the state-of-the-art MRC models by a large margin  That is to say, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them 
To verify the effectiveness of general knowledge, we first study the relationship between the amount of general knowledge and the performance of KAR  As shown in Table TABREF13 , by increasing INLINEFORM0 from 0 to 5 in the data enrichment method, the amount of general knowledge rises monotonically, but the performance of KAR first rises until INLINEFORM1 reaches 3 and then drops down  Then we conduct an ablation study by replacing the knowledge aided attention mechanisms with the mutual attention proposed by BIBREF3 and the self attention proposed by BIBREF4 separately, and find that the F1 score of KAR drops by INLINEFORM2 on the development set, INLINEFORM3 on AddSent, and INLINEFORM4 on AddOneSent  Finally we find that after only one epoch of training, KAR already achieves an EM of INLINEFORM5 and an F1 score of INLINEFORM6 on the development set, which is even better than the final performance of several strong baselines, such as DCN (EM / F1: INLINEFORM7 / INLINEFORM8 ) BIBREF36 and BiDAF (EM / F1: INLINEFORM9 / INLINEFORM10 ) BIBREF3   The above empirical findings imply that general knowledge indeed plays an effective role in KAR 
To demonstrate the advantage of our explicit way to utilize general knowledge over the existing implicit way, we compare the performance of KAR with that reported by BIBREF10 , which used an encoding-based method to utilize the general knowledge dynamically retrieved from Wikipedia and ConceptNet  Since their best model only achieved an EM of INLINEFORM0 and an F1 score of INLINEFORM1 on the development set, which is much lower than the performance of KAR, we have good reason to believe that our explicit way works better than the existing implicit way 
Model Comparison in the Hunger for Data
We compare KAR with other MRC models in the hunger for data  Specifically, instead of using all the training examples, we produce several training subsets (i e  subsets of the training examples) so as to study the relationship between the proportion of the available training examples and the performance  We produce each training subset by sampling a specific number of questions from all the questions relevant to each passage  By separately sampling 1, 2, 3, and 4 questions on each passage, we obtain four training subsets, which separately contain INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 of the training examples  As shown in Figure FIGREF15 , with KAR, SAN (re-implemented), and QANet (re-implemented without data augmentation) trained on these training subsets, we evaluate their performance on the development set, and find that KAR performs much better than SAN and QANet  As shown in Figure FIGREF16 and Figure FIGREF17 , with the above KAR, SAN, and QANet trained on the same training subsets, we also evaluate their performance on the adversarial sets, and still find that KAR performs much better than SAN and QANet  That is to say, when only a subset of the training examples are available, KAR outperforms the state-of-the-art MRC models by a large margin, and is still reasonably robust to noise 
Analysis
According to the experimental results, KAR is not only comparable in performance with the state-of-the-art MRC models, but also superior to them in terms of both the hunger for data and the robustness to noise  The reasons for these achievements, we believe, are as follows:
Conclusion
In this paper, we innovatively integrate the neural networks of MRC models with the general knowledge of human beings  Specifically, inter-word semantic connections are first extracted from each given passage-question pair by a WordNet-based data enrichment method, and then provided as general knowledge to an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the general knowledge to assist its attention mechanisms  Experimental results show that KAR is not only comparable in performance with the state-of-the-art MRC models, but also superior to them in terms of both the hunger for data and the robustness to noise  In the future, we plan to use some larger knowledge bases, such as ConceptNet and Freebase, to improve the quality and scope of the general knowledge 
Acknowledgments
This work is partially supported by a research donation from iFLYTEK Co , Ltd , Hefei, China, and a discovery grant from Natural Sciences and Engineering Research Council (NSERC) of Canada 

ground_truth: Yes

Yes