prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the clinical datasets used in the paper?
Context 1: Introduction
During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive  This expansion of data sharing and processing capabilities is both a blessing and a curse  Data helps build better information systems for the digital era and enables further research for advanced data management that benefits the society in general  But the use of this very data containing sensitive information conflicts with private data protection, both from an ethical and a legal perspective 
There are several application domains on which this situation is particularly acute  This is the case of the medical domain BIBREF0  There are plenty of potential applications for advanced medical data management that can only be researched and developed using real data; yet, the use of medical data is severely limited –when not entirely prohibited– due to data privacy protection policies 
One way of circumventing this problem is to anonymise the data by removing, replacing or obfuscating the personal information mentioned, as exemplified in Table TABREF1  This task can be done by hand, having people read and anonymise the documents one by one  Despite being a reliable and simple solution, this approach is tedious, expensive, time consuming and difficult to scale to the potentially thousands or millions of documents that need to be anonymised 
For this reason, numerous of systems and approaches have been developed during the last decades to attempt to automate the anonymisation of sensitive content, starting with the automatic detection and classification of sensitive information  Some of these systems rely on rules, patterns and dictionaries, while others use more advanced techniques related to machine learning and, more recently, deep learning 
Given that this paper is concerned with text documents (e g  medical records), the involved techniques are related to Natural Language Processing (NLP)  When using NLP approaches, it is common to pose the problem of document anonymisation as a sequence labelling problem, i e  classifying each token within a sequence as being sensitive information or not  Further, depending on the objective of the anonymisation task, it is also important to determine the type of sensitive information (names of individuals, addresses, age, sex, etc ) 
The anonymisation systems based on NLP techniques perform reasonably well, but are far from perfect  Depending on the difficulty posed by each dataset or the amount of available data for training machine learning models, the performance achieved by these methods is not enough to fully rely on them in certain situations BIBREF0  However, in the last two years, the NLP community has reached an important milestone thanks to the appearance of the so-called Transformers neural network architectures BIBREF1  In this paper, we conduct several experiments in sensitive information detection and classification on Spanish clinical text using BERT (from `Bidirectional Encoder Representations from Transformers') BIBREF2 as the base for a sequence labelling approach  The experiments are carried out on two datasets: the MEDDOCAN: Medical Document Anonymization shared task dataset BIBREF3, and NUBes BIBREF4, a corpus of real medical reports in Spanish  In these experiments, we compare the performance of BERT with other machine-learning-based systems, some of which use language-specific features  Our aim is to evaluate how good a BERT-based model performs without language nor domain specialisation apart from the training data labelled for the task at hand 
The rest of the paper is structured as follows: the next section describes related work about data anonymisation in general and clinical data anonymisation in particular; it also provides a more detailed explanation and background about the Transformers architecture and BERT  Section SECREF3 describes the data involved in the experiments and the systems evaluated in this paper, including the BERT-based system; finally, it details the experimental design  Section SECREF4 introduces the results for each set of experiments  Finally, Section SECREF5 contains the conclusions and future lines of work 
Related Work
The state of the art in the field of Natural Language Processing (NLP) has reached an important milestone in the last couple of years thanks to deep-learning architectures, increasing in several points the performance of new models for almost any text processing task 
The major change started with the Transformers model proposed by vaswani2017attention  It substituted the widely used recurrent and convolutional neural network architectures by another approach based solely on self-attention, obtaining an impressive performance gain  The original proposal was focused on an encoder-decoder architecture for machine translation, but soon the use of Transformers was made more general BIBREF1  There are several other popular models that use Transformers, such as Open AI's GPT and GPT2 BIBREF5, RoBERTa BIBREF6 and the most recent XLNet BIBREF7; still, BERT BIBREF2 is one of the most widespread Transformer-based models 
BERT trains its unsupervised language model using a Masked Language Model and Next Sentence Prediction  A common problem in NLP is the lack of enough training data  BERT can be pre-trained to learn general or specific language models using very large amounts of unlabelled text (e g  web content, Wikipedia, etc ), and this knowledge can be transferred to a different downstream task in a process that receives the name fine-tuning 
devlin2018bert have used fine-tuning to achieve state-of-the-art results on a wide variety of challenging natural language tasks, such as text classification, Question Answering (QA) and Named Entity Recognition and Classification (NERC)  BERT has also been used successfully by other community practitioners for a wide range of NLP-related tasks BIBREF8, BIBREF9 
Regarding the task of data anonymisation in particular, anonymisation systems may follow different approaches and pursue different objectives (Cormode and Srivastava, 2009)  The first objective of these systems is to detect and classify the sensitive information contained in the documents to be anonymised  In order to achieve that, they use rule-based approaches, Machine Learning (ML) approaches, or a combination of both 
Although most of these efforts are for English texts –see, among others, the i2b2 de-identification challenges BIBREF10, BIBREF11, dernon2016deep, or khin2018deep–, other languages are also attracting growing interest  Some examples are mamede2016automated for Portuguese and tveit2004anonymization for Norwegian  With respect to the anonymisation of text written in Spanish, recent studies include medina2018building, hassan2018anonimizacion and garcia2018automating  Most notably, in 2019 the first community challenge about anonymisation of medical documents in Spanish, MEDDOCAN BIBREF3, was held as part of the IberLEF initiative  The winners of the challenge –the Neither-Language-nor-Domain-Experts (NLNDE) BIBREF12– achieved F1-scores as high as 0 975 in the task of sensitive information detection and categorisation by using recurrent neural networks with Conditional Random Field (CRF) output layers 
At the same challenge, mao2019hadoken occupied the 8th position among 18 participants using BERT  According to the description of the system, the authors used BERT-Base Multilingual Cased and an output CRF layer  However, their system is $\sim $3 F1-score points below our implementation without the CRF layer 
Materials and Methods
The aim of this paper is to evaluate BERT's multilingual model and compare it to other established machine-learning algorithms in a specific task: sensitive data detection and classification in Spanish clinical free text  This section describes the data involved in the experiments and the systems evaluated  Finally, we introduce the experimental setup 
Materials and Methods ::: Data
Two datasets are exploited in this article  Both datasets consist of plain text containing clinical narrative written in Spanish, and their respective manual annotations of sensitive information in BRAT BIBREF13 standoff format  In order to feed the data to the different algorithms presented in Section SECREF7, these datasets were transformed to comply with the commonly used BIO sequence representation scheme BIBREF14 
Materials and Methods ::: Data ::: NUBes-PHI
NUBes BIBREF4 is a corpus of around 7,000 real medical reports written in Spanish and annotated with negation and uncertainty information  Before being published, sensitive information had to be manually annotated and replaced for the corpus to be safely shared  In this article, we work with the NUBes version prior to its anonymisation, that is, with the manual annotations of sensitive information  It follows that the version we work with is not publicly available and, due to contractual restrictions, we cannot reveal the provenance of the data  In order to avoid confusion between the two corpus versions, we henceforth refer to the version relevant in this paper as NUBes-PHI (from `NUBes with Personal Health Information') 
NUBes-PHI consists of 32,055 sentences annotated for 11 different sensitive information categories  Overall, it contains 7,818 annotations  The corpus has been randomly split into train (72%), development (8%) and test (20%) sets to conduct the experiments described in this paper  The size of each split and the distribution of the annotations can be consulted in Tables and , respectively 
The majority of sensitive information in NUBes-PHI are temporal expressions (`Date' and `Time'), followed by healthcare facility mentions (`Hospital'), and the age of the patient  Mentions of people are not that frequent, with physician names (`Doctor') occurring much more often than patient names (`Patient')  The least frequent sensitive information types, which account for $\sim $10% of the remaining annotations, consist of the patient's sex, job, and kinship, and locations other than healthcare facilities (`Location')  Finally, the tag `Other' includes, for instance, mentions to institutions unrelated to healthcare and whether the patient is right- or left-handed  It occurs just 36 times 
Materials and Methods ::: Data ::: The MEDDOCAN corpus
The organisers of the MEDDOCAN shared task BIBREF3 curated a synthetic corpus of clinical cases enriched with sensitive information by health documentalists  In this regard, the MEDDOCAN evaluation scenario could be said to be somewhat far from the real use case the technology developed for the shared task is supposed to be applied in  However, at the moment it also provides the only public means for a rigorous comparison between systems for sensitive health information detection in Spanish texts 
The size of the MEDDOCAN corpus is shown in Table   Compared to NUBes-PHI (Table ), this corpus contains more sensitive information annotations, both in absolute and relative terms 
The sensitive annotation categories considered in MEDDOCAN differ in part from those in NUBes-PHI  Most notably, it contains finer-grained labels for location-related mentions –namely, `Address', `Territory', and `Country'–, and other sensitive information categories that we did not encounter in NUBes-PHI (e g , identifiers, phone numbers, e-mail addresses, etc )  In total, the MEDDOCAN corpus has 21 sensitive information categories  We refer the reader to the organisers' article BIBREF3 for more detailed information about this corpus 
Materials and Methods ::: Systems
Apart from experimenting with a pre-trained BERT model, we have run experiments with other systems and baselines, to compare them and obtain a better perspective about BERT's performance in these datasets 
Materials and Methods ::: Systems ::: Baseline
As the simplest baseline, a sensitive data recogniser and classifier has been developed that consists of regular-expressions and dictionary look-ups  For each category to detect a specific method has been implemented  For instance, the Date, Age, Time and Doctor detectors are based on regular-expressions; Hospital, Sex, Kinship, Location, Patient and Job are looked up in dictionaries  The dictionaries are hand-crafted from the training data available, except for the Patient's case, for which the possible candidates considered are the 100 most common female and male names in Spain according to the Instituto Nacional de Estadística (INE; Spanish Statistical Office) 
Materials and Methods ::: Systems ::: CRF
Conditional Random Fields (CRF) BIBREF15 have been extensively used for tasks of sequential nature  In this paper, we propose as one of the competitive baselines a CRF classifier trained with sklearn-crfsuite for Python 3 5 and the following configuration: algorithm = lbfgs; maximum iterations = 100; c1 = c2 = 0 1; all transitions = true; optimise = false  The features extracted from each token are as follows:
[noitemsep]
prefixes and suffixes of 2 and 3 characters;
the length of the token in characters and the length of the sentence in tokens;
whether the token is all-letters, a number, or a sequence of punctuation marks;
whether the token contains the character `@';
whether the token is the start or end of the sentence;
the token's casing and the ratio of uppercase characters, digits, and punctuation marks to its length;
and, the lemma, part-of-speech tag, and named-entity tag given by ixa-pipes BIBREF16 upon analysing the sentence the token belongs to 
Noticeably, none of the features used to train the CRF classifier is domain-dependent  However, the latter group of features is language dependent 
Materials and Methods ::: Systems ::: spaCy
spaCy is a widely used NLP library that implements state-of-the-art text processing pipelines, including a sequence-labelling pipeline similar to the one described by strubell2017fast  spaCy offers several pre-trained models in Spanish, which perform basic NLP tasks such as Named Entity Recognition (NER)  In this paper, we have trained a new NER model to detect NUBes-PHI labels  For this purpose, the new model uses all the labels of the training corpus coded with its context at sentence level  The network optimisation parameters and dropout values are the ones recommended in the documentation for small datasets  Finally, the model is trained using batches of size 64  No more features are included, so the classifier is language-dependent but not domain-dependent 
Materials and Methods ::: Systems ::: BERT
As introduced earlier, BERT has shown an outstanding performance in NERC-like tasks, improving the start-of-the-art results for almost every dataset and language  We take the same approach here, by using the model BERT-Base Multilingual Cased with a Fully Connected (FC) layer on top to perform a fine-tuning of the whole model for an anonymisation task in Spanish clinical data  Our implementation is built on PyTorch and the PyTorch-Transformers library BIBREF1  The training phase consists in the following steps (roughly depicted in Figure ):
Pre-processing: since we are relying on a pre-trained BERT model, we must match the same configuration by using a specific tokenisation and vocabulary  BERT also needs that the inputs contains special tokens to signal the beginning and the end of each sequence 
Fine-tuning: the pre-processed sequence is fed into the model  BERT outputs the contextual embeddings that encode each of the inputted tokens  This embedding representation for each token is fed into the FC linear layer after a dropout layer (with a 0 1 dropout probability), which in turn outputs the logits for each possible class  The cross-entropy loss function is calculated comparing the logits and the gold labels, and the error is back-propagated to adjust the model parameters 
We have trained the model using an AdamW optimiser BIBREF17 with the learning rate set to 3e-5, as recommended by devlin2018bert, and with a gradient clipping of 1 0  We also applied a learning-rate scheduler that warms up the learning rate from zero to its maximum value as the training progresses, which is also a common practice  For each experiment set proposed below, the training was run with an early-stopping patience of 15 epochs  Then, the model that performed best against the development set was used to produce the reported results 
The experiments were run on a 64-core server with operating system Ubuntu 16 04, 250GB of RAM memory, and 4 GeForce RTX 2080 GPUs with 11GB of memory  The maximum sequence length was set at 500 and the batch size at 12  In this setting, each epoch –a full pass through all the training data– required about 10 minutes to complete 
Materials and Methods ::: Experimental design
We have conducted experiments with BERT in the two datasets of Spanish clinical narrative presented in Section SECREF3 The first experiment set uses NUBes-PHI, a corpus of real medical reports manually annotated with sensitive information  Because this corpus is not publicly available, and in order to compare the BERT-based model to other related published systems, the second set of experiments uses the MEDDOCAN 2019 shared task competition dataset  The following sections provide greater detail about the two experimental setups 
Materials and Methods ::: Experimental design ::: Experiment A: NUBes-PHI
In this experiment set, we evaluate all the systems presented in Section SECREF7, namely, the rule-based baseline, the CRF classifier, the spaCy entity tagger, and BERT  The evaluation comprises three scenarios of increasing difficulty:
[noitemsep]
- Evaluates the performance of the systems at predicting whether each token is sensitive or non-sensitive; that is, the measurements only take into account whether a sensitive token has been recognised or not, regardless of the BIO label and the category assigned  This scenario shows how good a system would be at obfuscating sensitive data (e g , by replacing sensitive tokens with asterisks) 
- We measure the performance of the systems at predicting the sensitive information type of each token –i e , the 11 categories presented in Section SECREF5 or `out'  Detecting entity types correctly is important if a system is going to be used to replace sensitive data by fake data of the same type (e g , random people names) 
- This is the strictest evaluation, as it takes into account both the BIO label and the category assigned to each individual token  Being able to discern between two contiguous sensitive entities of the same type is relevant not only because it is helpful when producing fake replacements, but because it also yields more accurate statistics of the sensitive information present in a given document collection 
The systems are evaluated in terms of micro-average precision, recall and F1-score in all the scenarios 
In addition to the scenarios proposed, a subject worth being studied is the need of labelled data  Manually labelled data is an scarce and expensive resource, which for some application domains or languages is difficult to come by  In order to obtain an estimation of the dependency of each system on the available amount of training data, we have retrained all the compared models using decreasing amounts of data –from 100% of the available training instances to just 1%  The same data subsets have been used to train all the systems  Due to the knowledge transferred from the pre-trained BERT model, the BERT-based model is expected to be more robust to data scarcity than those that start their training from scratch 
Materials and Methods ::: Experimental design ::: Experiment B: MEDDOCAN
In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text  Specifically, we include the results of a domain-independent NLNDE model (S2), and the results of a model enriched with domain-specific embeddings (S3)  Finally, we include the results obtained by mao2019hadoken with a CRF output layer on top of BERT embeddings  MEDDOCAN consists of two scenarios:
[noitemsep]
- This evaluation measures how good a system is at detecting sensitive text spans, regardless of the category assigned to them 
- In this scenario, systems are required to match exactly not only the boundaries of each sensitive span, but also the category assigned 
The systems are evaluated in terms of micro-averaged precision, recall and F-1 score  Note that, in contrast to the evaluation in Experiment A, MEDDOCAN measurements are entity-based instead of tokenwise  An exhaustive explanation of the MEDDOCAN evaluation procedure is available online, as well as the official evaluation script, which we used to obtain the reported results 
Results
This section describes the results obtained in the two sets of experiments: NUBes-PHI and MEDDOCAN 
Results ::: Experiment A: NUBes-PHI
Table shows the results of the conducted experiments in NUBes-PHI for all the compared systems  The included baseline serves to give a quick insight about how challenging the data is  With simple regular expressions and gazetteers a precision of 0 853 is obtained  On the other hand, the recall, which directly depends on the coverage provided by the rules and resources, drops to 0 469  Hence, this task is unlikely to be solved without the generalisation capabilities provided by machine-learning and deep-learning models 
Regarding the detection scenario –that is, the scenario concerned with a binary classification to determine whether each individual token conveys sensitive information or not–, it can be observed that BERT outperforms its competitors  A fact worth highlighting is that, according to these results, BERT achieves a precision lower than the rest of the systems (i e , it makes more false positive predictions); in exchange, it obtains a remarkably higher recall  Noticeably, it reaches a recall of 0 979, improving by more than 4 points the second-best system, spaCy 
The table also shows the results for the relaxed metric that only takes into account the entity type detected, regardless of the BIO label (i e , ignoring whether the token is at the beginning or in the middle of a sensitive sequence of tokens)  The conclusions are very similar to those extracted previously, with BERT gaining 2 1 points of F1-score over the CRF based approach  The confusion matrices of the predictions made by CRF, spaCy, and BERT in this scenario are shown in Table   As can bee seen, BERT has less difficulty in predicting correctly less frequent categories, such as `Location', `Job', and `Patient'  One of the most common mistakes according to the confusion matrices is classifying hospital names as `Location' instead of the more accurate `Hospital'; this is hardly a harmful error, given that a hospital is actually a location  Last, the category `Other' is completely leaked by all the compared systems, most likely due to its almost total lack of support in both training and evaluation datasets 
To finish with this experiment set, Table also shows the strict classification precision, recall and F1-score for the compared systems  Despite the fact that, in general, the systems obtain high values, BERT outperforms them again  BERT's F1-score is 1 9 points higher than the next most competitive result in the comparison  More remarkably, the recall obtained by BERT is about 5 points above 
Upon manual inspection of the errors committed by the BERT-based model, we discovered that it has a slight tendency towards producing ill-formed BIO sequences (e g, starting a sensitive span with `Inside' instead of `Begin'; see Table )  We could expect that complementing the BERT-based model with a CRF layer on top would help enforce the emission of valid sequences, alleviating this kind of errors and further improving its results 
Finally, Figure shows the impact of decreasing the amount of training data in the detection scenario  It shows the difference in precision, recall, and F1-score with respect to that obtained using 100% of the training data  A general downward trend can be observed, as one would expect: less training data leads to less accurate predictions  However, the BERT-based model is the most robust to training-data reduction, showing an steadily low performance loss  With 1% of the dataset (230 training instances), the BERT-based model only suffers a striking 7-point F1-score loss, in contrast to the 32 and 39 points lost by the CRF and spaCy models, respectively  This steep performance drop stems to a larger extent from recall decline, which is not that marked in the case of BERT  Overall, these results indicate that the transfer-learning achieved through the BERT multilingual pre-trained model not only helps obtain better results, but also lowers the need of manually labelled data for this application domain 
Results ::: Experiment B: MEDDOCAN
The results of the two MEDDOCAN scenarios –detection and classification– are shown in Table   These results follow the same pattern as in the previous experiments, with the CRF classifier being the most precise of all, and BERT outperforming both the CRF and spaCy classifiers thanks to its greater recall  We also show the results of mao2019hadoken who, despite of having used a BERT-based system, achieve lower scores than our models  The reason why it should be so remain unclear 
With regard to the winner of the MEDDOCAN shared task, the BERT-based model has not improved the scores obtained by neither the domain-dependent (S3) nor the domain-independent (S2) NLNDE model  However, attending to the obtained results, BERT remains only 0 3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors  Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies 
Conclusions and Future Work
In this work we have briefly introduced the problems related to data privacy protection in clinical domain  We have also described some of the groundbreaking advances on the Natural Language Processing field due to the appearance of Transformers-based deep-learning architectures and transfer learning from very large general-domain multilingual corpora, focusing our attention in one of its most representative examples, Google's BERT model 
In order to assess the performance of BERT for Spanish clinical data anonymisation, we have conducted several experiments with a BERT-based sequence labelling approach using the pre-trained multilingual BERT model shared by Google as the starting point for the model training  We have compared this BERT-based sequence labelling against other methods and systems  One of the experiments uses the MEDDOCAN 2019 shared task dataset, while the other uses a novel Spanish clinical reports dataset called NUBes-PHI 
The results of the experiments show that, in NUBes-PHI, the BERT-based model outperforms the other systems without requiring any adaptation or domain-specific feature engineering, just by being trained on the provided labelled data  Interestingly, the BERT-based model obtains a remarkably higher recall than the other systems  High recall is a desirable outcome because, when anonymising sensible documents, the accidental leak of sensible data is likely to be more dangerous than the unintended over-obfuscation of non-sensitive text 
Further, we have conducted an additional experiment on this dataset by progressively reducing the training data for all the compared systems  The BERT-based model shows the highest robustness to training-data scarcity, loosing only 7 points of F1-score when trained on 230 instances instead of 21,371  These observation are in line with the results obtained by the NLP community using BERT for other tasks 
The experiments with the MEDDOCAN 2019 shared task dataset follow the same pattern  In this case, the BERT-based model falls 0 3 F1-score points behind the shared task winning system, but it would have achieved the second position in the competition with no further refinement 
Since we have used a pre-trained multilingual BERT model, the same approach is likely to work for other languages just by providing some labelled training data  Further, this is the simplest fine-tuning that can be performed based on BERT  More sophisticated fine-tuning layers could help improve the results  For example, it could be expected that a CRF layer helped enforce better BIO tagging sequence predictions  Precisely, mao2019hadoken participated in the MEDDOCAN competition using a BERT+CRF architecture, but their reported scores are about 3 points lower than our implementation  From the description of their work, it is unclear what the source of this score difference could be 
Further, at the time of writing this paper, new multilingual pre-trained models and Transformer architectures have become available  It would not come as a surprise that these new resources and systems –e g , XLM-RoBERTa BIBREF19 or BETO BIBREF20, a BERT model fully pre-trained on Spanish texts– further advanced the state of the art in this task 
Acknowledgements
This work has been supported by Vicomtech and partially funded by the project DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER,UE) 
Context 2: Introduction
Deep Neural Networks (DNN) have been widely employed in industry for solving various Natural Language Processing (NLP) tasks, such as text classification, sequence labeling, question answering, etc  However, when engineers apply DNN models to address specific NLP tasks, they often face the following challenges 
The above challenges often hinder the productivity of engineers, and result in less optimal solutions to their given tasks  This motivates us to develop an NLP toolkit for DNN models, which facilitates engineers to develop DNN approaches  Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas 
To satisfy the requirements of all the above three personas, the NLP toolkit has to be generic enough to cover as many tasks as possible  At the same time, it also needs to be flexible enough to allow alternative network architectures as well as customized modules  Therefore, we analyzed the NLP jobs submitted to a commercial centralized GPU cluster  Table TABREF11 showed that about 87 5% NLP related jobs belong to a few common tasks, including sentence classification, text matching, sequence labeling, MRC, etc  It further suggested that more than 90% of the networks were composed of several common components, such as embedding, CNN/RNN, Transformer and so on 
Based on the above observations, we developed NeuronBlocks, a DNN toolkit for NLP tasks  The basic idea is to provide two layers of support to the engineers  The upper layer targets common NLP tasks  For each task, the toolkit contains several end-to-end network templates, which can be immediately instantiated with simple configuration  The bottom layer consists of a suite of reusable and standard components, which can be adopted as building blocks to construct networks with complex architecture  By following the interface guidelines, users can also contribute to this gallery of components with their own modules 
The technical contributions of NeuronBlocks are summarized into the following three aspects 
Related Work
There are several general-purpose deep learning frameworks, such as TensorFlow, PyTorch and Keras, which have gained popularity in NLP community  These frameworks offer huge flexibility in DNN model design and support various NLP tasks  However, building models under these frameworks requires a large overhead of mastering these framework details  Therefore, higher level abstraction to hide the framework details is favored by many engineers 
There are also several popular deep learning toolkits in NLP, including OpenNMT BIBREF0 , AllenNLP BIBREF1 etc  OpenNMT is an open-source toolkit mainly targeting neural machine translation or other natural language generation tasks  AllenNLP provides several pre-built models for NLP tasks, such as semantic role labeling, machine comprehension, textual entailment, etc  Although these toolkits reduce the development cost, they are limited to certain tasks, and thus not flexible enough to support new network architectures or new components 
Design
The Neuronblocks is built on PyTorch  The overall framework is illustrated in Figure FIGREF16   It consists of two layers: the Block Zoo and the Model Zoo  In Block Zoo, the most commonly used components of deep neural networks are categorized into several groups according to their functions  Within each category, several alternative components are encapsulated into standard and reusable blocks with a consistent interface  These blocks serve as basic and exchangeable units to construct complex network architectures for different NLP tasks  In Model Zoo, the most popular NLP tasks are identified  For each task, several end-to-end network templates are provided in the form of JSON configuration files  Users can simply browse these configurations and choose one or more to instantiate  The whole task can be completed without any coding efforts 
Block Zoo
We recognize the following major functional categories of neural network components  Each category covers as many commonly used modules as possible  The Block Zoo is an open framework, and more modules can be added in the future 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported 
Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc  Furthermore, attention mechanisms are widely used in neural networks  Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc  Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability 
Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6  
Metrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported  For sequence labeling task, F1/Accuracy are supported  For knowledge distillation task, MSE/RMSE are supported  For MRC task, ExactMatch/F1 are supported 
Model Zoo
In NeuronBlocks, we identify four types of most popular NLP tasks  For each task, we provide various end-to-end network templates 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Text Classification and Matching  Tasks such as domain/intent classification, question answer matching are supported
Context 3: Introduction
Machine Reading Comprehension (MRC), as the name suggests, requires a machine to read a passage and answer its relevant questions  Since the answer to each question is supposed to stem from the corresponding passage, a common MRC solution is to develop a neural-network-based MRC model that predicts an answer span (i e  the answer start position and the answer end position) from the passage of each given passage-question pair  To facilitate the explorations and innovations in this area, many MRC datasets have been established, such as SQuAD BIBREF0 , MS MARCO BIBREF1 , and TriviaQA BIBREF2   Consequently, many pioneering MRC models have been proposed, such as BiDAF BIBREF3 , R-NET BIBREF4 , and QANet BIBREF5   According to the leader board of SQuAD, the state-of-the-art MRC models have achieved the same performance as human beings  However, does this imply that they have possessed the same reading comprehension ability as human beings?
OF COURSE NOT  There is a huge gap between MRC models and human beings, which is mainly reflected in the hunger for data and the robustness to noise  On the one hand, developing MRC models requires a large amount of training examples (i e  the passage-question pairs labeled with answer spans), while human beings can achieve good performance on evaluation examples (i e  the passage-question pairs to address) without training examples  On the other hand, BIBREF6 revealed that intentionally injected noise (e g  misleading sentences) in evaluation examples causes the performance of MRC models to drop significantly, while human beings are far less likely to suffer from this  The reason for these phenomena, we believe, is that MRC models can only utilize the knowledge contained in each given passage-question pair, but in addition to this, human beings can also utilize general knowledge  A typical category of general knowledge is inter-word semantic connections  As shown in Table TABREF1 , such general knowledge is essential to the reading comprehension ability of human beings 
A promising strategy to bridge the gap mentioned above is to integrate the neural networks of MRC models with the general knowledge of human beings  To this end, it is necessary to solve two problems: extracting general knowledge from passage-question pairs and utilizing the extracted general knowledge in the prediction of answer spans  The first problem can be solved with knowledge bases, which store general knowledge in structured forms  A broad variety of knowledge bases are available, such as WordNet BIBREF7 storing semantic knowledge, ConceptNet BIBREF8 storing commonsense knowledge, and Freebase BIBREF9 storing factoid knowledge  In this paper, we limit the scope of general knowledge to inter-word semantic connections, and thus use WordNet as our knowledge base  The existing way to solve the second problem is to encode general knowledge in vector space so that the encoding results can be used to enhance the lexical or contextual representations of words BIBREF10 , BIBREF11   However, this is an implicit way to utilize general knowledge, since in this way we can neither understand nor control the functioning of general knowledge  In this paper, we discard the existing implicit way and instead explore an explicit (i e  understandable and controllable) way to utilize general knowledge 
The contribution of this paper is two-fold  On the one hand, we propose a data enrichment method, which uses WordNet to extract inter-word semantic connections as general knowledge from each given passage-question pair  On the other hand, we propose an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the above extracted general knowledge to assist its attention mechanisms  Based on the data enrichment method, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them  When only a subset ( INLINEFORM0 – INLINEFORM1 ) of the training examples are available, KAR outperforms the state-of-the-art MRC models by a large margin, and is still reasonably robust to noise 
Data Enrichment Method
In this section, we elaborate a WordNet-based data enrichment method, which is aimed at extracting inter-word semantic connections from each passage-question pair in our MRC dataset  The extraction is performed in a controllable manner, and the extracted results are provided as general knowledge to our MRC model 
Semantic Relation Chain
WordNet is a lexical database of English, where words are organized into synsets according to their senses  A synset is a set of words expressing the same sense so that a word having multiple senses belongs to multiple synsets, with each synset corresponding to a sense  Synsets are further related to each other through semantic relations  According to the WordNet interface provided by NLTK BIBREF12 , there are totally sixteen types of semantic relations (e g  hypernyms, hyponyms, holonyms, meronyms, attributes, etc )  Based on synset and semantic relation, we define a new concept: semantic relation chain  A semantic relation chain is a concatenated sequence of semantic relations, which links a synset to another synset  For example, the synset “keratin n 01” is related to the synset “feather n 01” through the semantic relation “substance holonym”, the synset “feather n 01” is related to the synset “bird n 01” through the semantic relation “part holonym”, and the synset “bird n 01” is related to the synset “parrot n 01” through the semantic relation “hyponym”, thus “substance holonym INLINEFORM0 part holonym INLINEFORM1 hyponym” is a semantic relation chain, which links the synset “keratin n 01” to the synset “parrot n 01”  We name each semantic relation in a semantic relation chain as a hop, therefore the above semantic relation chain is a 3-hop chain  By the way, each single semantic relation is equivalent to a 1-hop chain 
Inter-word Semantic Connection
The key problem in the data enrichment method is determining whether a word is semantically connected to another word  If so, we say that there exists an inter-word semantic connection between them  To solve this problem, we define another new concept: the extended synsets of a word  Given a word INLINEFORM0 , whose synsets are represented as a set INLINEFORM1 , we use another set INLINEFORM2 to represent its extended synsets, which includes all the synsets that are in INLINEFORM3 or that can be linked to from INLINEFORM4 through semantic relation chains  Theoretically, if there is no limitation on semantic relation chains, INLINEFORM5 will include all the synsets in WordNet, which is meaningless in most situations

ground_truth: MEDDOCAN, NUBes-PHI

MEDDOCAN, NUBes 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What neural network modules are included in NeuronBlocks?
Context 2: 
Sequence Labeling  Predict each token in a sequence into predefined types  Common tasks include NER, POS tagging, Slot tagging, etc 
Knowledge Distillation BIBREF7   Teacher-Student based knowledge distillation is one common approach for model compression  NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT 
Extractive Machine Reading Comprehension  Given a pair of question and passage, predict the start and end positions of the answer spans in the passage 
User Interface
NeuronBlocks provides convenient user interface for users to build, train, and test DNN models  The details are described in the following 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
I/O interface  This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc 
Model Architecture interface  This is the key part of the configuration file, which defines the whole model architecture  Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks  To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo 
Training Parameters interface  In this part, the model optimizer as well as all other training hyper parameters are indicated 
Workflow
Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks  Users only need to write a JSON configuration file  They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo  This configuration file is shared across training, test, and prediction  For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file  Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks  These new blocks can be further shared across all users for model architecture design  Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI 
Experiments
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15   The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved 
Sequence Labeling
For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task  This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC  We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17   Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF  The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance 
GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks  We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models  As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts 
Knowledge Distillation
We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine  We refer to this dataset as Domain Classification Dataset  Each sample in this dataset consists of two parts, i e , a question and a binary label indicating whether the question belongs to a specific domain  Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed  By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier 
WikiQA
The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering  This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia  We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models  The results are shown in Table TABREF41   The models built in NeuronBlocks achieved competitive or even better results with simple model configurations 
Conclusion and Future Work
In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch  NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users  To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks  Such design achieves a balance between generality and flexibility  Extensive experiments have verified the effectiveness of this approach  NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches 
As an open-source toolkit, we will further extend it in various directions  The following names a few examples 
Context 3:  Therefore, we use a hyper-parameter INLINEFORM6 to represent the permitted maximum hop count of semantic relation chains  That is to say, only the chains having no more than INLINEFORM7 hops can be used to construct INLINEFORM8 so that INLINEFORM9 becomes a function of INLINEFORM10 : INLINEFORM11 (if INLINEFORM12 , we will have INLINEFORM13 )  Based on the above statements, we formulate a heuristic rule for determining inter-word semantic connections: a word INLINEFORM14 is semantically connected to another word INLINEFORM15 if and only if INLINEFORM16  
General Knowledge Extraction
Given a passage-question pair, the inter-word semantic connections that connect any word to any passage word are regarded as the general knowledge we need to extract  Considering the requirements of our MRC model, we only extract the positional information of such inter-word semantic connections  Specifically, for each word INLINEFORM0 , we extract a set INLINEFORM1 , which includes the positions of the passage words that INLINEFORM2 is semantically connected to (if INLINEFORM3 itself is a passage word, we will exclude its own position from INLINEFORM4 )  We can control the amount of the extracted results by setting the hyper-parameter INLINEFORM5 : if we set INLINEFORM6 to 0, inter-word semantic connections will only exist between synonyms; if we increase INLINEFORM7 , inter-word semantic connections will exist between more words  That is to say, by increasing INLINEFORM8 within a certain range, we can usually extract more inter-word semantic connections from a passage-question pair, and thus can provide the MRC model with more general knowledge  However, due to the complexity and diversity of natural languages, only a part of the extracted results can serve as useful general knowledge, while the rest of them are useless for the prediction of answer spans, and the proportion of the useless part always rises when INLINEFORM9 is set larger  Therefore we set INLINEFORM10 through cross validation (i e  according to the performance of the MRC model on the development examples) 
Knowledge Aided Reader
In this section, we elaborate our MRC model: Knowledge Aided Reader (KAR)  The key components of most existing MRC models are their attention mechanisms BIBREF13 , which are aimed at fusing the associated representations of each given passage-question pair  These attention mechanisms generally fall into two categories: the first one, which we name as mutual attention, is aimed at fusing the question representations into the passage representations so as to obtain the question-aware passage representations; the second one, which we name as self attention, is aimed at fusing the question-aware passage representations into themselves so as to obtain the final passage representations  Although KAR is equipped with both categories, its most remarkable feature is that it explicitly uses the general knowledge extracted by the data enrichment method to assist its attention mechanisms  Therefore we separately name the attention mechanisms of KAR as knowledge aided mutual attention and knowledge aided self attention 
Task Definition
Given a passage INLINEFORM0 and a relevant question INLINEFORM1 , the task is to predict an answer span INLINEFORM2 , where INLINEFORM3 , so that the resulting subsequence INLINEFORM4 from INLINEFORM5 is an answer to INLINEFORM6  
Overall Architecture
As shown in Figure FIGREF7 , KAR is an end-to-end MRC model consisting of five layers:
Lexicon Embedding Layer  This layer maps the words to the lexicon embeddings  The lexicon embedding of each word is composed of its word embedding and character embedding  For each word, we use the pre-trained GloVe BIBREF14 word vector as its word embedding, and obtain its character embedding with a Convolutional Neural Network (CNN) BIBREF15   For both the passage and the question, we pass the concatenation of the word embeddings and the character embeddings through a shared dense layer with ReLU activation, whose output dimensionality is INLINEFORM0   Therefore we obtain the passage lexicon embeddings INLINEFORM1 and the question lexicon embeddings INLINEFORM2  
Context Embedding Layer  This layer maps the lexicon embeddings to the context embeddings  For both the passage and the question, we process the lexicon embeddings (i e  INLINEFORM0 for the passage and INLINEFORM1 for the question) with a shared bidirectional LSTM (BiLSTM) BIBREF16 , whose hidden state dimensionality is INLINEFORM2   By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the passage context embeddings INLINEFORM3 and the question context embeddings INLINEFORM4  
Coarse Memory Layer  This layer maps the context embeddings to the coarse memories  First we use knowledge aided mutual attention (introduced later) to fuse INLINEFORM0 into INLINEFORM1 , the outputs of which are represented as INLINEFORM2   Then we process INLINEFORM3 with a BiLSTM, whose hidden state dimensionality is INLINEFORM4   By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the coarse memories INLINEFORM5 , which are the question-aware passage representations 
Refined Memory Layer  This layer maps the coarse memories to the refined memories  First we use knowledge aided self attention (introduced later) to fuse INLINEFORM0 into themselves, the outputs of which are represented as INLINEFORM1   Then we process INLINEFORM2 with a BiLSTM, whose hidden state dimensionality is INLINEFORM3   By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the refined memories INLINEFORM4 , which are the final passage representations 
Answer Span Prediction Layer  This layer predicts the answer start position and the answer end position based on the above layers  First we obtain the answer start position distribution INLINEFORM0 : INLINEFORM1 INLINEFORM2
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents the refined memory of each passage word INLINEFORM4 (i e  the INLINEFORM5 -th column in INLINEFORM6 ); INLINEFORM7 represents the question summary obtained by performing an attention pooling over INLINEFORM8   Then we obtain the answer end position distribution INLINEFORM9 : INLINEFORM10 INLINEFORM11
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents vector concatenation  Finally we construct an answer span prediction matrix INLINEFORM4 , where INLINEFORM5 represents the upper triangular matrix of a matrix INLINEFORM6   Therefore, for the training, we minimize INLINEFORM7 on each training example whose labeled answer span is INLINEFORM8 ; for the inference, we separately take the row index and column index of the maximum element in INLINEFORM9 as INLINEFORM10 and INLINEFORM11  
Knowledge Aided Mutual Attention
As a part of the coarse memory layer, knowledge aided mutual attention is aimed at fusing the question context embeddings INLINEFORM0 into the passage context embeddings INLINEFORM1 , where the key problem is to calculate the similarity between each passage context embedding INLINEFORM2 (i e  the INLINEFORM3 -th column in INLINEFORM4 ) and each question context embedding INLINEFORM5 (i e  the INLINEFORM6 -th column in INLINEFORM7 )  To solve this problem, BIBREF3 proposed a similarity function: INLINEFORM8
where INLINEFORM0 is a trainable parameter; INLINEFORM1 represents element-wise multiplication  This similarity function has also been adopted by several other works BIBREF17 , BIBREF5   However, since context embeddings contain high-level information, we believe that introducing the pre-extracted general knowledge into the calculation of such similarities will make the results more reasonable  Therefore we modify the above similarity function to the following form: INLINEFORM2
where INLINEFORM0 represents the enhanced context embedding of a word INLINEFORM1   We use the pre-extracted general knowledge to construct the enhanced context embeddings  Specifically, for each word INLINEFORM2 , whose context embedding is INLINEFORM3 , to construct its enhanced context embedding INLINEFORM4 , first recall that we have extracted a set INLINEFORM5 , which includes the positions of the passage words that INLINEFORM6 is semantically connected to, thus by gathering the columns in INLINEFORM7 whose indexes are given by INLINEFORM8 , we obtain the matching context embeddings INLINEFORM9   Then by constructing a INLINEFORM10 -attended summary of INLINEFORM11 , we obtain the matching vector INLINEFORM12 (if INLINEFORM13 , which makes INLINEFORM14 , we will set INLINEFORM15 ): INLINEFORM16 INLINEFORM17
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents the INLINEFORM4 -th column in INLINEFORM5   Finally we pass the concatenation of INLINEFORM6 and INLINEFORM7 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM8   Therefore we obtain the enhanced context embedding INLINEFORM9  
Based on the modified similarity function and the enhanced context embeddings, to perform knowledge aided mutual attention, first we construct a knowledge aided similarity matrix INLINEFORM0 , where each element INLINEFORM1   Then following BIBREF5 , we construct the passage-attended question summaries INLINEFORM2 and the question-attended passage summaries INLINEFORM3 : INLINEFORM4 INLINEFORM5
where INLINEFORM0 represents softmax along the row dimension and INLINEFORM1 along the column dimension  Finally following BIBREF17 , we pass the concatenation of INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM6   Therefore we obtain the outputs INLINEFORM7  
Knowledge Aided Self Attention
As a part of the refined memory layer, knowledge aided self attention is aimed at fusing the coarse memories INLINEFORM0 into themselves  If we simply follow the self attentions of other works BIBREF4 , BIBREF18 , BIBREF19 , BIBREF17 , then for each passage word INLINEFORM1 , we should fuse its coarse memory INLINEFORM2 (i e  the INLINEFORM3 -th column in INLINEFORM4 ) with the coarse memories of all the other passage words

ground_truth: Embedding Layer, Neural Network Layers, Loss Function, Metrics

Embedding Layer, Neural Network Layers, Loss Function, Metrics

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do the authors hypothesize that humans' robustness to noise is due to their general knowledge?
Context 3:  However, we believe that this is both unnecessary and distracting, since each passage word has nothing to do with many of the other passage words  Thus we use the pre-extracted general knowledge to guarantee that the fusion of coarse memories for each passage word will only involve a precise subset of the other passage words  Specifically, for each passage word INLINEFORM5 , whose coarse memory is INLINEFORM6 , to perform the fusion of coarse memories, first recall that we have extracted a set INLINEFORM7 , which includes the positions of the other passage words that INLINEFORM8 is semantically connected to, thus by gathering the columns in INLINEFORM9 whose indexes are given by INLINEFORM10 , we obtain the matching coarse memories INLINEFORM11   Then by constructing a INLINEFORM12 -attended summary of INLINEFORM13 , we obtain the matching vector INLINEFORM14 (if INLINEFORM15 , which makes INLINEFORM16 , we will set INLINEFORM17 ): INLINEFORM18 INLINEFORM19
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters  Finally we pass the concatenation of INLINEFORM3 and INLINEFORM4 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM5   Therefore we obtain the fusion result INLINEFORM6 , and further the outputs INLINEFORM7  
Related Works
Attention Mechanisms  Besides those mentioned above, other interesting attention mechanisms include performing multi-round alignment to avoid the problems of attention redundancy and attention deficiency BIBREF20 , and using mutual attention as a skip-connector to densely connect pairwise layers BIBREF21  
Data Augmentation  It is proved that properly augmenting training examples can improve the performance of MRC models  For example, BIBREF22 trained a generative model to generate questions based on unlabeled text, which substantially boosted their performance; BIBREF5 trained a back-and-forth translation model to paraphrase training examples, which brought them a significant performance gain 
Multi-step Reasoning  Inspired by the fact that human beings are capable of understanding complex documents by reading them over and over again, multi-step reasoning was proposed to better deal with difficult MRC tasks  For example, BIBREF23 used reinforcement learning to dynamically determine the number of reasoning steps; BIBREF19 fixed the number of reasoning steps, but used stochastic dropout in the output layer to avoid step bias 
Linguistic Embeddings  It is both easy and effective to incorporate linguistic embeddings into the input layer of MRC models  For example, BIBREF24 and BIBREF19 used POS embeddings and NER embeddings to construct their input embeddings; BIBREF25 used structural embeddings based on parsing trees to constructed their input embeddings 
Transfer Learning  Several recent breakthroughs in MRC benefit from feature-based transfer learning BIBREF26 , BIBREF27 and fine-tuning-based transfer learning BIBREF28 , BIBREF29 , which are based on certain word-level or sentence-level models pre-trained on large external corpora in certain supervised or unsupervised manners 
Experimental Settings
MRC Dataset  The MRC dataset used in this paper is SQuAD 1 1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 )  Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models  The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models  Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage 
Implementation Details  We tokenize the MRC dataset with spaCy 2 0 13 BIBREF30 , manipulate WordNet 3 0 with NLTK 3 3, and implement KAR with TensorFlow 1 11 0 BIBREF31   For the data enrichment method, we set the hyper-parameter INLINEFORM0 to 3  For the dense layers and the BiLSTMs, we set the dimensionality unit INLINEFORM1 to 600  For model optimization, we apply the Adam BIBREF32 optimizer with a learning rate of INLINEFORM2 and a mini-batch size of 32  For model evaluation, we use Exact Match (EM) and F1 score as evaluation metrics  To avoid overfitting, we apply dropout BIBREF33 to the dense layers and the BiLSTMs with a dropout rate of INLINEFORM3   To boost the performance, we apply exponential moving average with a decay rate of INLINEFORM4  
Model Comparison in both Performance and the Robustness to Noise
We compare KAR with other MRC models in both performance and the robustness to noise  Specifically, we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets  As for the comparative objects, we only consider the single MRC models that rank in the top 20 on the SQuAD 1 1 leader board and have reported their performance on the adversarial sets  There are totally five such comparative objects, which can be considered as representatives of the state-of-the-art MRC models  As shown in Table TABREF12 , on the development set and the test set, the performance of KAR is on par with that of the state-of-the-art MRC models; on the adversarial sets, KAR outperforms the state-of-the-art MRC models by a large margin  That is to say, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them 
To verify the effectiveness of general knowledge, we first study the relationship between the amount of general knowledge and the performance of KAR  As shown in Table TABREF13 , by increasing INLINEFORM0 from 0 to 5 in the data enrichment method, the amount of general knowledge rises monotonically, but the performance of KAR first rises until INLINEFORM1 reaches 3 and then drops down  Then we conduct an ablation study by replacing the knowledge aided attention mechanisms with the mutual attention proposed by BIBREF3 and the self attention proposed by BIBREF4 separately, and find that the F1 score of KAR drops by INLINEFORM2 on the development set, INLINEFORM3 on AddSent, and INLINEFORM4 on AddOneSent  Finally we find that after only one epoch of training, KAR already achieves an EM of INLINEFORM5 and an F1 score of INLINEFORM6 on the development set, which is even better than the final performance of several strong baselines, such as DCN (EM / F1: INLINEFORM7 / INLINEFORM8 ) BIBREF36 and BiDAF (EM / F1: INLINEFORM9 / INLINEFORM10 ) BIBREF3   The above empirical findings imply that general knowledge indeed plays an effective role in KAR 
To demonstrate the advantage of our explicit way to utilize general knowledge over the existing implicit way, we compare the performance of KAR with that reported by BIBREF10 , which used an encoding-based method to utilize the general knowledge dynamically retrieved from Wikipedia and ConceptNet  Since their best model only achieved an EM of INLINEFORM0 and an F1 score of INLINEFORM1 on the development set, which is much lower than the performance of KAR, we have good reason to believe that our explicit way works better than the existing implicit way 
Model Comparison in the Hunger for Data
We compare KAR with other MRC models in the hunger for data  Specifically, instead of using all the training examples, we produce several training subsets (i e  subsets of the training examples) so as to study the relationship between the proportion of the available training examples and the performance  We produce each training subset by sampling a specific number of questions from all the questions relevant to each passage  By separately sampling 1, 2, 3, and 4 questions on each passage, we obtain four training subsets, which separately contain INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 of the training examples  As shown in Figure FIGREF15 , with KAR, SAN (re-implemented), and QANet (re-implemented without data augmentation) trained on these training subsets, we evaluate their performance on the development set, and find that KAR performs much better than SAN and QANet  As shown in Figure FIGREF16 and Figure FIGREF17 , with the above KAR, SAN, and QANet trained on the same training subsets, we also evaluate their performance on the adversarial sets, and still find that KAR performs much better than SAN and QANet  That is to say, when only a subset of the training examples are available, KAR outperforms the state-of-the-art MRC models by a large margin, and is still reasonably robust to noise 
Analysis
According to the experimental results, KAR is not only comparable in performance with the state-of-the-art MRC models, but also superior to them in terms of both the hunger for data and the robustness to noise  The reasons for these achievements, we believe, are as follows:
Conclusion
In this paper, we innovatively integrate the neural networks of MRC models with the general knowledge of human beings  Specifically, inter-word semantic connections are first extracted from each given passage-question pair by a WordNet-based data enrichment method, and then provided as general knowledge to an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the general knowledge to assist its attention mechanisms  Experimental results show that KAR is not only comparable in performance with the state-of-the-art MRC models, but also superior to them in terms of both the hunger for data and the robustness to noise  In the future, we plan to use some larger knowledge bases, such as ConceptNet and Freebase, to improve the quality and scope of the general knowledge 
Acknowledgments
This work is partially supported by a research donation from iFLYTEK Co , Ltd , Hefei, China, and a discovery grant from Natural Sciences and Engineering Research Council (NSERC) of Canada 

ground_truth: Yes

Yes