prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their highest recall score?
Context 1: Introduction
BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year  We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year)  In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions  We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task  Our system scored near the top for factoid questions for all the batches of the challenge  More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task  Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set  Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments  In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions )
The QA task is organized in two phases  Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets)  Exact answer generation is required for factoid, list, and yes/no type question 
BioASQ organizers provide the training and testing data  The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2)  The test data is split between phases A and B  The Phase A dataset consists of the questions, unique ids, question types  The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types  Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers  Answers for the list type question are evaluated using precision, recall, and F-measure 
Related Work ::: BioAsq
Sharma et al  BIBREF3 describe a system with two stage process for factoid and list type question answering  Their system extracts relevant entities and then runs supervised classifier to rank the entities  Wiese et al  BIBREF4 propose neural network based model for Factoid and List-type question answering task  The model is based on Fast QA and predicts the answer span in the passage for a given question  The model is trained on SQuAD data set and fine tuned on the BioASQ data  Dimitriadis et al  BIBREF5 proposed two stage process for Factoid question answering task  Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences  These candidate sentences are represented in the form of features, and are then ranked by the binary classifier  Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge  For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0 4325’  Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0 6103’ in one of the test batches for Factoid Question Answering task 
Related Work ::: A minimum background on BERT
BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model  Given a sentence as an input, contextual embedding for the words are returned  The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks  For a question answering task, question and paragraph (context) are given as an input  A BERT standard is that question text and paragraph text are separated by a separator [Sep]  BERT question-answering fine tuning involves adding softmax layer  Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context)  This process is represented in Figure FIGREF4 
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction  In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation  For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1 
Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings
A ‘word embedding’ is a learned representation  It is represented in the form of vector where words that have the same meaning have a similar vector representation  Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus  Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e g  the sentiment of the sentence  However, contextual word embedding models like BERT also takes context of the word into consideration 
Related Work ::: Comparison of BERT and Bio-BERT
‘BERT’ and BioBERT are very similar in terms of architecture  Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles  Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings 
BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers  Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large  According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1 1 question answering data BIBREF7 can achieve F1 Score of 90 9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88 5  One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon  Lee et al  BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab txt), as a result biomedical jargon is not included in word-piece vocabulary  Modifying word-piece vocabulary (vocab txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified 
In our future work we would like to build pre-trained ‘BERT’ model from scratch  We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia  Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary  We will consider this scenario in the future, or wait for the next version of BioBERT 
Experiments: Factoid Question Answering Task
For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features  Fig  FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context)  As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure  The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context)  On test data, the fine tuned model generates $n$-best predictions for each question  For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence  Variable $n$ is configurable  In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model 
Experiments: Factoid Question Answering Task ::: Setup
BioASQ provides the training data  This data is based on previous BioASQ competitions  Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition  We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers  The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing 
The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer  The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context  For finding ‘start_index’ we used built-in python function find()  The function returns the lowest index of the actual answer present in the context(paragraph)  If the answer is not found ‘-1’ is returned as the index  The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question 
Example (Question, Answer and Paragraph from BIBREF8):
Question: Which drug should be used as an antidote in benzodiazepine overdose?
Answer: 'Flumazenil'
Paragraph(context):
"Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data  OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK  Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants  A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK  METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD  RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil  Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure  Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration  Seizure frequency in patients not treated with flumazenil was 0 3%" 
Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'  Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question  In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context  Unfortunately, we could not identify readily available tools that can achieve this goal  In our future work, we look forward to handling these scenarios effectively 
Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively  But 'SQuAD' data set is much more general and does not include biomedical question answering data 
Experiments: Factoid Question Answering Task ::: Training and error analysis
During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0  We started training the model with 495 available train data and 35 test data by setting number of epochs to 50  After training with these hyper-parameters training accuracy(exact match) was 99 3%(overfitting) and testing accuracy is only 4%  In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98 5% and test accuracy moved to 5%  We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%  In the next iteration set number of epochs to 12 and achieved train accuracy of 57 7% and test accuracy 23 3%  Repeated the experiment with 11 epochs and found training accuracy to be 57 7% and test accuracy to be same 22%  In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%  Hence optimum number of epochs is taken as 12 epochs 
During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer  On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer  We removed 70% of those instances in order to make train data more balanced  In the new train data set we are left with '411' question answering data instances  This time we got the highest test accuracy of 26% at 11 epochs  We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place  Initially, hyper-parameter- 'batch size' is set to ‘400’  Later it is tuned to '32'  Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases 
Example (from BIBREF8)
Question: Which mutated gene causes Chediak Higashi Syndrome?
Exact Answer: ‘lysosomal trafficking regulator gene’ 
The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’ 
In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2 0) and BioAsq train data  For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1  Test accuracy of the model boosted to 44%  In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%  The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy 
Our Systems and Their Performance on Factoid Questions
We have experimented with several systems and their variations, e g  created by training with specific additional features (see next subsection)  Here is their list and short descriptions  Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details 
When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering  The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods  Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available 
Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present:
training on BioAsq data vs  training on SQuAD
using the BioAsq snippets for context vs  using the documents from the provided URLs for context
adding or not the LAT, i e  lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below) 
For Yes/No questions (only) we experimented with the entailment methods 
We will discuss the performance of these models below and in Section 6  But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative)
During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question 
Example: (from BIBREF8)
Question: Hy's law measures failure of which organ?
Actual Answer: ‘Liver’ 
The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme  The model returns an enzyme, when the question asked for the organ name  To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis  In an example given in the last source we read:
POETS & POETRY: He was a bank clerk in the Yukon before he published "Songs of a Sourdough" in 1907 
The focus is the part of the question that is a reference to the answer  In the example above, the focus is "he" 
LATs are terms in the question that indicate what type of entity is being asked for 
The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs 
(   ) In the example, LATs are "he", "clerk", and "poet" 
For example in the question "Which plant does oleuropein originate from?" (BIBREF8)  The LAT here is ‘plant’  For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts  In this example, the expectation is that answer returned by the model is a plant  Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low  As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT 
We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used  We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4  This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type 
There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'  We assumed that LAT is a ‘Noun’ and follows the question word  Often it was also a subject (nsubj)  This process is illustrated in Fig FIGREF15 
LAT computation was governed by a few simple rules, e g  when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’  These rules are different for each "Wh" word 
Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’  The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’ 
For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e g  'enzyme' in Which enzyme is targeted by Evolocumab?  When the word immediately following the question word is not a Noun, e g  in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’ 
For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself  For the word ‘How', e g  in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’ 
Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly  Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed  The training used BioBERT with the LAT feature as part of the input string  The errors it introduces usually involve finding the wrong element of the correct type e g  wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text  We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above  Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results 
Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative)
Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting  This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score) 
In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2 0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data  This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch  The latter was also trained on SQuAD, but not on BioAsq  We suspect that the reason could be the simplistic nature of the find() function described in Section 3 1  So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance 
Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative)
In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq  However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets 
In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer 
Performance on Yes/No and List questions
Our work focused on Factoid questions  But we also have done experiments on List-type and Yes/No questions 
Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy
We started by answering always YES (in batch 2 and 3) to get the baseline performance  For batch 4 we used entailment  Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned  In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score)  We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question 
Performance on Yes/No and List questions ::: For List-type the URLs have negative impact
Overall, we followed the similar strategy that's been followed for Factoid Question Answering task  We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets)  Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers  In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0 7033’ but low precision of 0 1119, leaving open the question of how could we have better balanced the two measures 
In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'  Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers  BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer  Considering lengthy snippets in to the list of answers would reduce the mean precision score  As a final step, duplicate snippets in the answer pool are removed  For example, consider these top 3 answers predicted by the system (before post-processing):
{
"text": "dendritic cells",
"probability": 0 7554540733426441,
"start_logit": 8 466046333312988,
"end_logit": 9 536355018615723
},
{
"text": "neutrophils, macrophages and
distinct subtypes of dendritic cells",
"probability": 0 13806867348304214,
"start_logit": 6 766478538513184,
"end_logit": 9 536355018615723
},
{
"text": "macrophages and distinct subtypes of dendritic",
"probability": 0 013973475271178242,
"start_logit": 6 766478538513184,
"end_logit": 7 24576473236084
},
After execution of post-processing heuristics, the list of answers returned is as follows:
["dendritic cells"],
["neutrophils"],
["macrophages"],
["distinct subtypes of dendritic cells"]
Summary of our results
The tables below summarize all our results  They show that the performance of our systems was mixed  The simple architectures and algorithm we used worked very well only in Batch 3  However, we feel we can built a better system based on this experience  In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly 
Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments
System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data 
Summary of our results ::: List Questions
For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good 
Summary of our results ::: Yes/No questions
The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline)  The results (weak) are in Table 3 
Discussion, Future Experiments, and Conclusions ::: Summary:
In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT  After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations  The latter we tried to address with mixed results using feature engineering  Overall these experiments allowed us to secure a best and a second best score in different test batches  Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach 
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance 
All our data and software is available at Github, in the previously referenced URL (end of Section 2) 
Discussion, Future Experiments, and Conclusions ::: Future experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
We also see potential for incorporating domain specific inference into the task e g  using the MedNLI dataset BIBREF14  For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e g  BIBREF16) and possibly deeper discourse representations (e g  BIBREF17) 
APPENDIX
In this appendix we provide additional details about the implementations 
APPENDIX ::: Systems and their descriptions:
We used several variants of our systems when experimenting with the BioASQ problems  In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems  So, we apologize for the names that do not reflect the modifications, and necessitate this list 
APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering:
We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: System description for QA1:
We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id  We ignored the content present in the documents (document URLS were provided in the original test data)  The model is finetuned with BioASQ data 
data preprocessing is done in the same way as it is done for test batch-1  Model fine tuned on BioASQ data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1:
System is finetuned on the SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3:
System is finetuned on the SQuAD 2 0 [reference] and BioASQ dataset[] For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2:
Fine tuning process is same as for ‘UNCC_QA_1 ’  Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data  System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set 
APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS:
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: List Type Questions:
We attempted List type questions starting from test batch ‘2’  Used similar approach that's been followed for Factoid Question answering task  For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data 
For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers  system ‘QA1’ achieved low F-Measure score:‘0 0786’ in the second test batch  In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5 2) and achieved a better F-measure score: ‘0 2862’ in the final test batch set 
In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing  System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data  for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context 
In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing  In case of system ‘FACTOIDS’ snippets in the test data were used to generate context  for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data 
In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing  for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions:
For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions  Later, we employed ‘Sentence Entailment’ techniques(refer section 6 0) for the fourth and fifth test batch sets  Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5) 
APPENDIX ::: Additional details for Yes/No Type Questions
We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type  The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question  If we find one 'No' is returned as answer, else 'Yes' is returned  (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question 
Flow Chart for Yes/No Question answer processing is shown in Fig FIGREF51
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions
There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP' 
Assumptions:
1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word 
2) The LAT word is a Subject  (This clearly not always true, but we used a very simple method)  Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass' 
3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’ 
4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively 
Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs  The figures show the grammatical structures used for this purpose 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1:
Question with question word ‘How’ 
For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately)  If an adjective is absent, word 'How' is considered as ‘LAT’  When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’  Note: The part of speech tag to identify adjectives is 'JJ'  For Other possible question words like ‘whose’  ‘LAT’/Focus word is question words itself 
Example Question: How many selenoproteins are encoded in the human genome?
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word 
Example Question: Which enzyme is targeted by Evolocumab?
Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word 
When the word immediately following the question word is a noun, the window size is set to ‘3’  This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word  Else the word that is present very next to the question word is considered as ‘LAT’ 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun' 
Example Question: What is the function of the protein Magt1?
Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word 
When the very next word following the question word is not a Noun, window size is set to ‘5’  Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as ‘LAT’  Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’ 
Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent  But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10  Whether this in turn produces improvements in this particular task is an open question 
APPENDIX ::: Proposing Future Experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network  In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’  Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus  To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
Context 2: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/ Deep neural networks have been widely used in text classification and have achieved promising results BIBREF0 , BIBREF1 , BIBREF2   Most focus on content information and use models such as convolutional neural networks (CNN) BIBREF3 or recursive neural networks BIBREF4   However, for user-generated posts on social media like Facebook or Twitter, there is more information that should not be ignored  On social media platforms, a user can act either as the author of a post or as a reader who expresses his or her comments about the post 
In this paper, we classify posts taking into account post authorship, likes, topics, and comments  In particular, users and their “likes” hold strong potential for text mining  For example, given a set of posts that are related to a specific topic, a user's likes and dislikes provide clues for stance labeling  From a user point of view, users with positive attitudes toward the issue leave positive comments on the posts with praise or even just the post's content; from a post point of view, positive posts attract users who hold positive stances  We also investigate the influence of topics: different topics are associated with different stance labeling tendencies and word usage  For example we discuss women's rights and unwanted babies on the topic of abortion, but we criticize medicine usage or crime when on the topic of marijuana BIBREF5   Even for posts on a specific topic like nuclear power, a variety of arguments are raised: green energy, radiation, air pollution, and so on  As for comments, we treat them as additional text information  The arguments in the comments and the commenters (the users who leave the comments) provide hints on the post's content and further facilitate stance classification 
In this paper, we propose the user-topic-comment neural network (UTCNN), a deep learning model that utilizes user, topic, and comment information  We attempt to learn user and topic representations which encode user interactions and topic influences to further enhance text classification, and we also incorporate comment information  We evaluate this model on a post stance classification task on forum-style social media platforms  The contributions of this paper are as follows: 1  We propose UTCNN, a neural network for text in modern social media channels as well as legacy social media, forums, and message boards — anywhere that reveals users, their tastes, as well as their replies to posts  2  When classifying social media post stances, we leverage users, including authors and likers  User embeddings can be generated even for users who have never posted anything  3  We incorporate a topic model to automatically assign topics to each post in a single topic dataset  4  We show that overall, the proposed method achieves the highest performance in all instances, and that all of the information extracted, whether users, topics, or comments, still has its contributions 
Extra-Linguistic Features for Stance Classification
In this paper we aim to use text as well as other features to see how they complement each other in a deep learning model  In the stance classification domain, previous work has showed that text features are limited, suggesting that adding extra-linguistic constraints could improve performance BIBREF6 , BIBREF7 , BIBREF8   For example, Hasan and Ng as well as Thomas et al  require that posts written by the same author have the same stance BIBREF9 , BIBREF10   The addition of this constraint yields accuracy improvements of 1–7% for some models and datasets  Hasan and Ng later added user-interaction constraints and ideology constraints BIBREF7 : the former models the relationship among posts in a sequence of replies and the latter models inter-topic relationships, e g , users who oppose abortion could be conservative and thus are likely to oppose gay rights 
For work focusing on online forum text, since posts are linked through user replies, sequential labeling methods have been used to model relationships between posts  For example, Hasan and Ng use hidden Markov models (HMMs) to model dependent relationships to the preceding post BIBREF9 ; Burfoot et al  use iterative classification to repeatedly generate new estimates based on the current state of knowledge BIBREF11 ; Sridhar et al  use probabilistic soft logic (PSL) to model reply links via collaborative filtering BIBREF12   In the Facebook dataset we study, we use comments instead of reply links  However, as the ultimate goal in this paper is predicting not comment stance but post stance, we treat comments as extra information for use in predicting post stance 
Deep Learning on Extra-Linguistic Features
In recent years neural network models have been applied to document sentiment classification BIBREF13 , BIBREF4 , BIBREF14 , BIBREF15 , BIBREF2   Text features can be used in deep networks to capture text semantics or sentiment  For example, Dong et al  use an adaptive layer in a recursive neural network for target-dependent Twitter sentiment analysis, where targets are topics such as windows 7 or taylor swift BIBREF16 , BIBREF17 ; recursive neural tensor networks (RNTNs) utilize sentence parse trees to capture sentence-level sentiment for movie reviews BIBREF4 ; Le and Mikolov predict sentiment by using paragraph vectors to model each paragraph as a continuous representation BIBREF18   They show that performance can thus be improved by more delicate text models 
Others have suggested using extra-linguistic features to improve the deep learning model  The user-word composition vector model (UWCVM) BIBREF19 is inspired by the possibility that the strength of sentiment words is user-specific; to capture this they add user embeddings in their model  In UPNN, a later extension, they further add a product-word composition as product embeddings, arguing that products can also show different tendencies of being rated or reviewed BIBREF20   Their addition of user information yielded 2–10% improvements in accuracy as compared to the above-mentioned RNTN and paragraph vector methods  We also seek to inject user information into the neural network model  In comparison to the research of Tang et al  on sentiment classification for product reviews, the difference is two-fold  First, we take into account multiple users (one author and potentially many likers) for one post, whereas only one user (the reviewer) is involved in a review  Second, we add comment information to provide more features for post stance classification  None of these two factors have been considered previously in a deep learning model for text stance classification  Therefore, we propose UTCNN, which generates and utilizes user embeddings for all users — even for those who have not authored any posts — and incorporates comments to further improve performance 
Method
In this section, we first describe CNN-based document composition, which captures user- and topic-dependent document-level semantic representation from word representations  Then we show how to add comment information to construct the user-topic-comment neural network (UTCNN) 
User- and Topic-dependent Document Composition
As shown in Figure FIGREF4 , we use a general CNN BIBREF3 and two semantic transformations for document composition   We are given a document with an engaged user INLINEFORM0 , a topic INLINEFORM1 , and its composite INLINEFORM2 words, each word INLINEFORM3 of which is associated with a word embedding INLINEFORM4 where INLINEFORM5 is the vector dimension  For each word embedding INLINEFORM6 , we apply two dot operations as shown in Equation EQREF6 : DISPLAYFORM0
where INLINEFORM0 models the user reading preference for certain semantics, and INLINEFORM1 models the topic semantics; INLINEFORM2 and INLINEFORM3 are the dimensions of transformed user and topic embeddings respectively  We use INLINEFORM4 to model semantically what each user prefers to read and/or write, and use INLINEFORM5 to model the semantics of each topic  The dot operation of INLINEFORM6 and INLINEFORM7 transforms the global representation INLINEFORM8 to a user-dependent representation  Likewise, the dot operation of INLINEFORM9 and INLINEFORM10 transforms INLINEFORM11 to a topic-dependent representation 
After the two dot operations on INLINEFORM0 , we have user-dependent and topic-dependent word vectors INLINEFORM1 and INLINEFORM2 , which are concatenated to form a user- and topic-dependent word vector INLINEFORM3   Then the transformed word embeddings INLINEFORM4 are used as the CNN input  Here we apply three convolutional layers on the concatenated transformed word embeddings INLINEFORM5 : DISPLAYFORM0
where INLINEFORM0 is the index of words; INLINEFORM1 is a non-linear activation function (we use INLINEFORM2 ); INLINEFORM5 is the convolutional filter with input length INLINEFORM6 and output length INLINEFORM7 , where INLINEFORM8 is the window size of the convolutional operation; and INLINEFORM9 and INLINEFORM10 are the output and bias of the convolution layer INLINEFORM11 , respectively  In our experiments, the three window sizes INLINEFORM12 in the three convolution layers are one, two, and three, encoding unigram, bigram, and trigram semantics accordingly 
After the convolutional layer, we add a maximum pooling layer among convolutional outputs to obtain the unigram, bigram, and trigram n-gram representations  This is succeeded by an average pooling layer for an element-wise average of the three maximized convolution outputs 
UTCNN Model Description
Figure FIGREF10 illustrates the UTCNN model  As more than one user may interact with a given post, we first add a maximum pooling layer after the user matrix embedding layer and user vector embedding layer to form a moderator matrix embedding INLINEFORM0 and a moderator vector embedding INLINEFORM1 for moderator INLINEFORM2 respectively, where INLINEFORM3 is used for the semantic transformation in the document composition process, as mentioned in the previous section  The term moderator here is to denote the pseudo user who provides the overall semantic/sentiment of all the engaged users for one document  The embedding INLINEFORM4 models the moderator stance preference, that is, the pattern of the revealed user stance: whether a user is willing to show his preference, whether a user likes to show impartiality with neutral statements and reasonable arguments, or just wants to show strong support for one stance  Ideally, the latent user stance is modeled by INLINEFORM5 for each user  Likewise, for topic information, a maximum pooling layer is added after the topic matrix embedding layer and topic vector embedding layer to form a joint topic matrix embedding INLINEFORM6 and a joint topic vector embedding INLINEFORM7 for topic INLINEFORM8 respectively, where INLINEFORM9 models the semantic transformation of topic INLINEFORM10 as in users and INLINEFORM11 models the topic stance tendency  The latent topic stance is also modeled by INLINEFORM12 for each topic 
As for comments, we view them as short documents with authors only but without likers nor their own comments  Therefore we apply document composition on comments although here users are commenters (users who comment)  It is noticed that the word embeddings INLINEFORM0 for the same word in the posts and comments are the same, but after being transformed to INLINEFORM1 in the document composition process shown in Figure FIGREF4 , they might become different because of their different engaged users  The output comment representation together with the commenter vector embedding INLINEFORM2 and topic vector embedding INLINEFORM3 are concatenated and a maximum pooling layer is added to select the most important feature for comments  Instead of requiring that the comment stance agree with the post, UTCNN simply extracts the most important features of the comment contents; they could be helpful, whether they show obvious agreement or disagreement  Therefore when combining comment information here, the maximum pooling layer is more appropriate than other pooling or merging layers  Indeed, we believe this is one reason for UTCNN's performance gains 
Finally, the pooled comment representation, together with user vector embedding INLINEFORM0 , topic vector embedding INLINEFORM1 , and document representation are fed to a fully connected network, and softmax is applied to yield the final stance label prediction for the post 
Experiment
We start with the experimental dataset and then describe the training process as well as the implementation of the baselines  We also implement several variations to reveal the effects of features: authors, likers, comment, and commenters  In the results section we compare our model with related work 
Dataset
We tested the proposed UTCNN on two different datasets: FBFans and CreateDebate  FBFans is a privately-owned, single-topic, Chinese, unbalanced, social media dataset, and CreateDebate is a public, multiple-topic, English, balanced, forum dataset  Results using these two datasets show the applicability and superiority for different topics, languages, data distributions, and platforms 
The FBFans dataset contains data from anti-nuclear-power Chinese Facebook fan groups from September 2013 to August 2014, including posts and their author and liker IDs  There are a total of 2,496 authors, 505,137 likers, 33,686 commenters, and 505,412 unique users  Two annotators were asked to take into account only the post content to label the stance of the posts in the whole dataset as supportive, neutral, or unsupportive (hereafter denoted as Sup, Neu, and Uns)  Sup/Uns posts were those in support of or against anti-reconstruction; Neu posts were those evincing a neutral standpoint on the topic, or were irrelevant  Raw agreement between annotators is 0 91, indicating high agreement  Specifically, Cohen’s Kappa for Neu and not Neu labeling is 0 58 (moderate), and for Sup or Uns labeling is 0 84 (almost perfect)  Posts with inconsistent labels were filtered out, and the development and testing sets were randomly selected from what was left  Posts in the development and testing sets involved at least one user who appeared in the training set  The number of posts for each stance is shown on the left-hand side of Table TABREF12   About twenty percent of the posts were labeled with a stance, and the number of supportive (Sup) posts was much larger than that of the unsupportive (Uns) ones: this is thus highly skewed data, which complicates stance classification  On average, 161
Context 3: Introduction
Automatically answering questions, especially in the open-domain setting (i e , where minimal or no contextual knowledge is explicitly provided), requires bringing to bear considerable amount of background knowledge and reasoning abilities  For example, knowing the answers to the two questions in Figure FIGREF1 requires identifying a specific ISA relation (i e , that cooking is a type of learned behavior) as well as recalling the definition of a concept (i e , that global warming is defined as a worldwide increase in temperature)  In the multiple-choice setting, which is the variety of question-answering (QA) that we focus on in this paper, there is also pragmatic reasoning involved in selecting optimal answer choices (e g , while greenhouse effect might in some other context be a reasonable answer to the second question in Figure FIGREF1, global warming is a preferable candidate) 
Recent successes in QA, driven largely by the creation of new resources BIBREF2, BIBREF3, BIBREF4, BIBREF5 and advances in model pre-training BIBREF6, BIBREF7, raise a natural question: do state-of-the-art multiple-choice QA (MCQA) models that excel at standard tasks really have basic knowledge and reasoning skills?
Most existing MCQA datasets are constructed through either expensive crowd-sourcing BIBREF8 or hand engineering effort, in the former case making it possible to collect large amounts of data at the cost of losing systematic control over the semantics of the target questions  Hence, doing a controlled experiment to answer such a question for QA is difficult given a lack of targeted challenge datasets 
Having definitive empirical evidence of model competence on any given phenomenon requires constructing a wide range of systematic tests  For example, in measuring competence of definitions, not only do we want to see that the model can handle individual questions such as Figure FIGREF1 1 inside of benchmark tasks, but that it can answer a wider range of questions that exhaustively cover a broad set of concepts and question perturbations (i e , systematic adjustments to how the questions are constructed)  The same applies to ISA reasoning; not only is it important to recognize in the question in Figure FIGREF1 1 that cooking is a learned behavior, but also that cooking is a general type of behavior or, through a few more inferential steps, a type of human activity 
In this paper, we look at systematically constructing such tests by exploiting the vast amounts of structured information contained in various types of expert knowledge such as knowledge graphs and lexical taxonomies  Our general methodology works as illustrated in Figure FIGREF1: given any MCQA model trained on a set of benchmark tasks, we systematically generate a set of synthetic dataset probes (i e , MCQA renderings of the target information) from information in expert knowledge sources  We then use these probes to ask two empirical questions: 1) how well do models trained on benchmark tasks perform on these probing tasks and; 2) can such models be re-trained to master new challenges with minimal performance loss on their original tasks?
While our methodology is amenable to any knowledge source and set of models/benchmark tasks, we focus on probing state-of-the-art transformer models BIBREF7, BIBREF9 in the domain of science MCQA  For sources of expert knowledge, we use WordNet, a comprehensive lexical ontology, and other publicly available dictionary resources  We devise probes that measure model competence in definition and taxonomic knowledge in different settings (including hypernymy, hyponymy, and synonymy detection, and word sense disambiguation)  This choice is motivated by fact that the science domain is considered particularly challenging for QA BIBREF10, BIBREF11, BIBREF12, and existing science benchmarks are known to involve widespread use of such knowledge (see BIBREF1, BIBREF13 for analysis), which is also arguably fundamental to more complex forms of reasoning 
We show that accurately probing QA models via synthetic datasets is not straightforward, as unexpected artifacts can easily arise in such data  This motivates our carefully constructed baselines and close data inspection to ensure probe quality 
Our results confirm that transformer-based QA models have a remarkable ability to recognize certain types of knowledge captured in our probes—even without additional fine-tuning  Such models can even outperform strong task-specific models trained directly on our probing tasks (e g , on definitions, our best model achieves 77% test accuracy without specialized training, as opposed to 51% for a task-specific LSTM-based model)  We also show that the same models can be effectively re-fine-tuned on small samples (even 100 examples) of probe data, and that high performance on the probes tends to correlate with a smaller drop in the model's performance on the original QA task 
Our comprehensive assessment reveals several interesting nuances to the overall positive trend  For example, the performance of even the best QA models degrades substantially on our hyponym probes (by 8-15%) when going from 1-hop links to 2-hops  Further, the accuracy of even our best models on the WordNetQA probe drops by 14-44% under our cluster-based analysis, which assesses whether a model knows several facts about each individual concept, rather than just being good at answering isolated questions  State-of-the-art QA models thus have much room to improve even in some fundamental building blocks, namely definitions and taxonomic hierarchies, of more complex forms of reasoning 
Related Work
We follow recent work on constructing challenge datasets for probing neural models, which has primarily focused on the task of natural language inference (NLI) BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18  Most of this work looks at constructing data through adversarial generation methods, which have also been found useful for creating stronger models BIBREF19  There has also been work on using synthetic data of the type we consider in this paper BIBREF20, BIBREF21, BIBREF22  We closely follow the methodology of BIBREF22, who use hand-constructed linguistic fragments to probe NLI models and study model re-training using a variant of the inoculation by fine-tuning strategy of BIBREF23  In contrast, we focus on probing open-domain MCQA models (see BIBREF24 for a related study in the reading comprehension setting) as well as constructing data from much larger sources of structured knowledge 
Our main study focuses on probing the BERT model and fine-tuning approach of BIBREF7, and other variants thereof, which are all based on the transformer architecture of BIBREF25  Related to our efforts, there have been recent studies into the types of relational knowledge contained in large-scale knowledge models BIBREF26, BIBREF27, BIBREF28, which, similar to our work, probe models using structured knowledge sources  This prior work, however, primarily focuses on unearthing the knowledge contained in the underlying language models as is without further training, using simple (single token) cloze-style probing tasks and templates (similar to what we propose in Section SECREF3)  In contrast, we focus on understanding the knowledge contained in language models after they have been trained for a QA end-task using benchmark datasets in which such knowledge is expected to be widespread  Further, our evaluation is done before and after these models are fine-tuned on our probe QA tasks, using a more complex set of QA templates and target inferences 
The use of lexical resources and knowledge graphs such as WordNet to construct datasets has a long history, and has recently appeared in work on adversarial attacks BIBREF14, BIBREF29 and general task construction BIBREF30, BIBREF31  In the area of MCQA, there is related work on constructing questions from tuples BIBREF32, BIBREF3, both of which involve standard crowd annotation to elicit question-answer pairs (see also BIBREF33, BIBREF34)  In contrast to this work, we focus on generating data in an entirely automatic fashion, which obviates the need for expensive annotation and gives us the flexibility to construct much larger datasets that control a rich set of semantic aspects of the target questions 
Dataset Probes and Construction
Our probing methodology starts by constructing challenge datasets (Figure FIGREF1, yellow box) from a target set of knowledge resources  Each of our probing datasets consists of multiple-choice questions that include a question $\textbf {q}$ and a set of answer choices or candidates $\lbrace a_{1},   a_{N}\rbrace $  This section describes in detail the 5 different datasets we build, which are drawn from two sources of expert knowledge, namely WordNet BIBREF35 and the GNU Collaborative International Dictionary of English (GCIDE)  We describe each resource in turn, and explain how the resulting dataset probes, which we call WordNetQA and DictionaryQA, are constructed 
For convenience, we will describe each source of expert knowledge as a directed, edge-labeled graph $G$  The nodes of this graph are $\mathcal {V} = \mathcal {C} \cup \mathcal {W} \cup \mathcal {S} \cup \mathcal {D}$, where $\mathcal {C}$ is a set of atomic concepts, $\mathcal {W}$ a set of words, $\mathcal {S}$ a set of sentences, and $\mathcal {D}$ a set of definitions (see Table TABREF4 for details for WordNet and GCIDE)  Each edge of $G$ is directed from an atomic concept in $\mathcal {C}$ to another node in $V$, and is labeled with a relation, such as hypernym or isa$^\uparrow $, from a set of relations $\mathcal {R}$ (see Table TABREF4) 
When defining our probe question templates, it will be useful to view $G$ as a set of (relation, source, target) triples $\mathcal {T} \subseteq \mathcal {R} \times \mathcal {C} \times \mathcal {V}$  Due to their origin in an expert knowledge source, such triples preserve semantic consistency  For instance, when the relation in a triple is def, the corresponding edge maps a concept in $\mathcal {C}$ to a definition in $\mathcal {D}$ 
To construct probe datasets, we rely on two heuristic functions, defined below for each individual probe: $\textsc {gen}_{\mathcal {Q}}(\tau )$, which generates gold question-answer pairs $(\textbf {q},\textbf {a})$ from a set of triples $\tau \subseteq \mathcal {T}$ and question templates $\mathcal {Q}$, and $\textsc {distr}(\tau ^{\prime })$, which generates distractor answers choices $\lbrace a^{\prime }_{1},   a^{\prime }_{N-1} \rbrace $ based on another set of triples $\tau ^{\prime }$ (where usually $\tau \subset \tau ^{\prime }$)  For brevity, we will use $\textsc {gen}(\tau )$ to denote $\textsc {gen}_{\mathcal {Q}}(\tau )$, leaving question templates $\mathcal {Q}$ implicit 
Dataset Probes and Construction ::: WordNetQA
WordNet is an English lexical database consisting of around 117k concepts, which are organized into groups of synsets that each contain a gloss (i e , a definition of the target concept), a set of representative English words (called lemmas), and, in around 33k synsets, example sentences  In addition, many synsets have ISA links to other synsets that express complex taxonomic relations  Figure FIGREF6 shows an example and Table TABREF4 summarizes how we formulate WordNet as a set of triples $\mathcal {T}$ of various types  These triples together represent a directed, edge-labeled graph $G$  Our main motivation for using WordNet, as opposed to a resource such as ConceptNet BIBREF36, is the availability of glosses ($\mathcal {D}$) and example sentences ($\mathcal {S}$), which allows us to construct natural language questions that contextualize the types of concepts we want to probe 
Dataset Probes and Construction ::: WordNetQA ::: Example Generation @!START@$\textsc {gen}(\tau )$@!END@ 
We build 4 individual datasets based on semantic relations native to WordNet (see BIBREF37): hypernymy (i e , generalization or ISA reasoning up a taxonomy, ISA$^\uparrow $), hyponymy (ISA$^{\downarrow }$), synonymy, and definitions  To generate a set of questions in each case, we employ a number of rule templates $\mathcal {Q}$ that operate over tuples  A subset of such templates is shown in Table TABREF8  The templates were designed to mimic naturalistic questions we observed in our science benchmarks 
For example, suppose we wish to create a question $\textbf {q}$ about the definition of a target concept $c \in \mathcal {C}$  We first select a question template from $\mathcal {Q}$ that first introduces the concept $c$ and its lemma $l \in \mathcal {W}$ in context using the example sentence $s \in \mathcal {S}$, and then asks to identify the corresponding WordNet gloss $d \in \mathcal {D}$, which serves as the gold answer $\textbf {a}$  The same is done for ISA reasoning; each question about a hypernym/hyponym relation between two concepts $c \rightarrow ^{\uparrow /\downarrow } c^{\prime } \in \mathcal {T}_{i}$ (e g , $\texttt {dog} \rightarrow ^{\uparrow /\downarrow } \texttt {animal/terrier}$) first introduces a context for $c$ and then asks for an answer that identifies $c^{\prime }$ (which is also provided with a gloss so as to contain all available context) 
In the latter case, the rules $(\texttt {isa}^{r},c,c^{\prime }) \in \mathcal {T}_i$ in Table TABREF8 cover only direct ISA links from $c$ in direction $r \in \lbrace \uparrow ,\downarrow \rbrace $  In practice, for each $c$ and direction $r$, we construct tests that cover the set HOPS$(c,r)$ of all direct as well as derived ISA relations of $c$:
This allows us to evaluate the extent to which models are able to handle complex forms of reasoning that require several inferential steps or hops 
Dataset Probes and Construction ::: WordNetQA ::: Distractor Generation: @!START@$\textsc {distr}(\tau ^{\prime })$@!END@ 
An example of how distractors are generated is shown in Figure FIGREF6, which relies on similar principles as above  For each concept $c$, we choose 4 distractor answers that are close in the WordNet semantic space  For example, when constructing hypernymy tests for $c$ from the set hops$(c,\uparrow )$, we build distractors by drawing from $\textsc {hops}(c,\downarrow )$ (and vice versa), as well as from the $\ell $-deep sister family of $c$, defined as follows  The 1-deep sister family is simply $c$'s siblings or sisters, i

ground_truth: 0.7033

0.7033

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 2: 1 users were involved in one post  The maximum was 23,297 and the minimum was one (the author)  For comments, on average there were 3 comments per post  The maximum was 1,092 and the minimum was zero 
To test whether the assumption of this paper – posts attract users who hold the same stance to like them – is reliable, we examine the likes from authors of different stances  Posts in FBFans dataset are used for this analysis  We calculate the like statistics of each distinct author from these 32,595 posts  As the numbers of authors in the Sup, Neu and Uns stances are largely imbalanced, these numbers are normalized by the number of users of each stance  Table TABREF13 shows the results  Posts with stances (i e , not neutral) attract users of the same stance  Neutral posts also attract both supportive and neutral users, like what we observe in supportive posts, but just the neutral posts can attract even more neutral likers  These results do suggest that users prefer posts of the same stance, or at least posts of no obvious stance which might cause annoyance when reading, and hence support the user modeling in our approach 
The CreateDebate dataset was collected from an English online debate forum discussing four topics: abortion (ABO), gay rights (GAY), Obama (OBA), and marijuana (MAR)  The posts are annotated as for (F) and against (A)  Replies to posts in this dataset are also labeled with stance and hence use the same data format as posts  The labeling results are shown in the right-hand side of Table TABREF12   We observe that the dataset is more balanced than the FBFans dataset  In addition, there are 977 unique users in the dataset  To compare with Hasan and Ng's work, we conducted five-fold cross-validation and present the annotation results as the average number of all folds BIBREF9 , BIBREF5  
The FBFans dataset has more integrated functions than the CreateDebate dataset; thus our model can utilize all linguistic and extra-linguistic features  For the CreateDebate dataset, on the other hand, the like and comment features are not available (as there is a stance label for each reply, replies are evaluated as posts as other previous work) but we still implemented our model using the content, author, and topic information 
Settings
In the UTCNN training process, cross-entropy was used as the loss function and AdaGrad as the optimizer  For FBFans dataset, we learned the 50-dimensional word embeddings on the whole dataset using GloVe BIBREF21 to capture the word semantics; for CreateDebate dataset we used the publicly available English 50-dimensional word embeddings, pre-trained also using GloVe  These word embeddings were fixed in the training process  The learning rate was set to 0 03  All user and topic embeddings were randomly initialized in the range of [-0 1 0 1]  Matrix embeddings for users and topics were sized at 250 ( INLINEFORM0 ); vector embeddings for users and topics were set to length 10 
We applied the LDA topic model BIBREF22 on the FBFans dataset to determine the latent topics with which to build topic embeddings, as there is only one general known topic: nuclear power plants  We learned 100 latent topics and assigned the top three topics for each post  For the CreateDebate dataset, which itself constitutes four topics, the topic labels for posts were used directly without additionally applying LDA 
For the FBFans data we report class-based f-scores as well as the macro-average f-score ( INLINEFORM0 ) shown in equation EQREF19   DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the average precision and recall of the three class  We adopted the macro-average f-score as the evaluation metric for the overall performance because (1) the experimental dataset is severely imbalanced, which is common for contentious issues; and (2) for stance classification, content in minor-class posts is usually more important for further applications  For the CreateDebate dataset, accuracy was adopted as the evaluation metric to compare the results with related work BIBREF7 , BIBREF9 , BIBREF12  
Baselines
We pit our model against the following baselines: 1) SVM with unigram, bigram, and trigram features, which is a standard yet rather strong classifier for text features; 2) SVM with average word embedding, where a document is represented as a continuous representation by averaging the embeddings of the composite words; 3) SVM with average transformed word embeddings (the INLINEFORM0 in equation EQREF6 ), where a document is represented as a continuous representation by averaging the transformed embeddings of the composite words; 4) two mature deep learning models on text classification, CNN BIBREF3 and Recurrent Convolutional Neural Networks (RCNN) BIBREF0 , where the hyperparameters are based on their work; 5) the above SVM and deep learning models with comment information; 6) UTCNN without user information, representing a pure-text CNN model where we use the same user matrix and user embeddings INLINEFORM1 and INLINEFORM2 for each user; 7) UTCNN without the LDA model, representing how UTCNN works with a single-topic dataset; 8) UTCNN without comments, in which the model predicts the stance label given only user and topic information  All these models were trained on the training set, and parameters as well as the SVM kernel selections (linear or RBF) were fine-tuned on the development set  Also, we adopt oversampling on SVMs, CNN and RCNN because the FBFans dataset is highly imbalanced 
Results on FBFans Dataset
In Table TABREF22 we show the results of UTCNN and the baselines on the FBFans dataset  Here Majority yields good performance on Neu since FBFans is highly biased to the neutral class  The SVM models perform well on Sup and Neu but perform poorly for Uns, showing that content information in itself is insufficient to predict stance labels, especially for the minor class  With the transformed word embedding feature, SVM can achieve comparable performance as SVM with n-gram feature  However, the much fewer feature dimension of the transformed word embedding makes SVM with word embeddings a more efficient choice for modeling the large scale social media dataset  For the CNN and RCNN models, they perform slightly better than most of the SVM models but still, the content information is insufficient to achieve a good performance on the Uns posts  As to adding comment information to these models, since the commenters do not always hold the same stance as the author, simply adding comments and post contents together merely adds noise to the model 
Among all UTCNN variations, we find that user information is most important, followed by topic and comment information  UTCNN without user information shows results similar to SVMs — it does well for Sup and Neu but detects no Uns  Its best f-scores on both Sup and Neu among all methods show that with enough training data, content-based models can perform well; at the same time, the lack of user information results in too few clues for minor-class posts to either predict their stance directly or link them to other users and posts for improved performance  The 17 5% improvement when adding user information suggests that user information is especially useful when the dataset is highly imbalanced  All models that consider user information predict the minority class successfully  UCTNN without topic information works well but achieves lower performance than the full UTCNN model  The 4 9% performance gain brought by LDA shows that although it is satisfactory for single topic datasets, adding that latent topics still benefits performance: even when we are discussing the same topic, we use different arguments and supporting evidence  Lastly, we get 4 8% improvement when adding comment information and it achieves comparable performance to UTCNN without topic information, which shows that comments also benefit performance  For platforms where user IDs are pixelated or otherwise hidden, adding comments to a text model still improves performance  In its integration of user, content, and comment information, the full UTCNN produces the highest f-scores on all Sup, Neu, and Uns stances among models that predict the Uns class, and the highest macro-average f-score overall  This shows its ability to balance a biased dataset and supports our claim that UTCNN successfully bridges content and user, topic, and comment information for stance classification on social media text  Another merit of UTCNN is that it does not require a balanced training data  This is supported by its outperforming other models though no oversampling technique is applied to the UTCNN related experiments as shown in this paper  Thus we can conclude that the user information provides strong clues and it is still rich even in the minority class 
We also investigate the semantic difference when a user acts as an author/liker or a commenter  We evaluated a variation in which all embeddings from the same user were forced to be identical (this is the UTCNN shared user embedding setting in Table TABREF22 )  This setting yielded only a 2 5% improvement over the model without comments, which is not statistically significant  However, when separating authors/likers and commenters embeddings (i e , the UTCNN full model), we achieved much greater improvements (4 8%)  We attribute this result to the tendency of users to use different wording for different roles (for instance author vs commenter)  This is observed when the user, acting as an author, attempts to support her argument against nuclear power by using improvements in solar power; when acting as a commenter, though, she interacts with post contents by criticizing past politicians who supported nuclear power or by arguing that the proposed evacuation plan in case of a nuclear accident is ridiculous  Based on this finding, in the final UTCNN setting we train two user matrix embeddings for one user: one for the author/liker role and the other for the commenter role 
Results on CreateDebate Dataset
Table TABREF24 shows the results of UTCNN, baselines as we implemented on the FBFans datset and related work on the CreateDebate dataset  We do not adopt oversampling on these models because the CreateDebate dataset is almost balanced  In previous work, integer linear programming (ILP) or linear-chain conditional random fields (CRFs) were proposed to integrate text features, author, ideology, and user-interaction constraints, where text features are unigram, bigram, and POS-dependencies; the author constraint tends to require that posts from the same author for the same topic hold the same stance; the ideology constraint aims to capture inferences between topics for the same author; the user-interaction constraint models relationships among posts via user interactions such as replies BIBREF7 , BIBREF9  
The SVM with n-gram or average word embedding feature performs just similar to the majority  However, with the transformed word embedding, it achieves superior results  It shows that the learned user and topic embeddings really capture the user and topic semantics  This finding is not so obvious in the FBFans dataset and it might be due to the unfavorable data skewness for SVM  As for CNN and RCNN, they perform slightly better than most SVMs as we found in Table TABREF22 for FBFans 
Compared to the ILP BIBREF7 and CRF BIBREF9 methods, the UTCNN user embeddings encode author and user-interaction constraints, where the ideology constraint is modeled by the topic embeddings and text features are modeled by the CNN  The significant improvement achieved by UTCNN suggests the latent representations are more effective than overt model constraints 
The PSL model BIBREF12 jointly labels both author and post stance using probabilistic soft logic (PSL) BIBREF23 by considering text features and reply links between authors and posts as in Hasan and Ng's work  Table TABREF24 reports the result of their best AD setting, which represents the full joint stance/disagreement collective model on posts and is hence more relevant to UTCNN  In contrast to their model, the UTCNN user embeddings represent relationships between authors, but UTCNN models do not utilize link information between posts  Though the PSL model has the advantage of being able to jointly label the stances of authors and posts, its performance on posts is lower than the that for the ILP or CRF models  UTCNN significantly outperforms these models on posts and has the potential to predict user stances through the generated user embeddings 
For the CreateDebate dataset, we also evaluated performance when not using topic embeddings or user embeddings; as replies in this dataset are viewed as posts, the setting without comment embeddings is not available  Table TABREF24 shows the same findings as Table TABREF22 : the 21% improvement in accuracy demonstrates that user information is the most vital  This finding also supports the results in the related work: user constraints are useful and can yield 11 2% improvement in accuracy BIBREF7   Further considering topic information yields 3 4% improvement, suggesting that knowing the subject of debates provides useful information  In sum, Table TABREF22 together with Table TABREF24 show that UTCNN achieves promising performance regardless of topic, language, data distribution, and platform 
Conclusion
We have proposed UTCNN, a neural network model that incorporates user, topic, content and comment information for stance classification on social media texts  UTCNN learns user embeddings for all users with minimum active degree, i e , one post or one like  Topic information obtained from the topic model or the pre-defined labels further improves the UTCNN model  In addition, comment information provides additional clues for stance classification  We have shown that UTCNN achieves promising and balanced results  In the future we plan to explore the effectiveness of the UTCNN user embeddings for author stance classification 
Acknowledgements
Research of this paper was partially supported by Ministry of Science and Technology, Taiwan, under the contract MOST 104-2221-E-001-024-MY2 
Context 3: e , the other children $\tilde{c} \ne c$ of the parent node $c^{\prime }$ of $c$  For $\ell > 1$, the $\ell $-deep sister family also includes all descendants of each $\tilde{c}$ up to $\ell -1$ levels deep, denoted $\textsc {hops}_{\ell -1}(\tilde{c},\downarrow )$  Formally:
For definitions and synonyms we build distractors from all of these sets (with a similar restriction on the depth of sister distractors as noted above)  In doing this, we can systematically investigate model performance on a wide range of distractor sets 
Dataset Probes and Construction ::: WordNetQA ::: Perturbations and Semantic Clusters
Based on how we generate data, for each concept $c$ (i e , atomic WordNet synset) and probe type (i e , definitions, hypernymy, etc ), we have a wide variety of questions related to $c$ that manipulate 1) the complexity of reasoning that is involved (e g , the number of inferential hops) and; 2) the types of distractors (or distractor perturbations) that are employed  We call such sets semantic clusters  As we describe in the next section, semantic clusters allow us to devise new types of evaluation that reveal whether models have comprehensive and consistent knowledge of target concepts (e g , evaluating whether a model can correctly answer several questions associated with a concept, as opposed to a few disjoint instances) 
Details of the individual datasets are shown in Table TABREF12  From these sets, we follow BIBREF22 in allocating a maximum of 3k examples for training and reserve the rest for development and testing  Since we are interested in probing, having large held-out sets allows us to do detailed analysis and cluster-based evaluation 
Dataset Probes and Construction ::: DictionaryQA
The DictionaryQA dataset is created from the GCIDE dictionary, which is a comprehensive open-source English dictionary built largely from the Webster's Revised Unabridged Dictionary BIBREF38  Each entry consists of a word, its part-of-speech, its definition, and an optional example sentence (see Table TABREF14)  Overall, 33k entries (out of a total of 155k) contain example sentences/usages  As with the WordNet probes, we focus on this subset so as to contextualize each word being probed  In contrast to WordNet, GCIDE does not have ISA relations or explicit synsets, so we take each unique entry to be a distinct sense  We then use the dictionary entries to create a probe that centers around word-sense disambiguation, as described below 
Dataset Probes and Construction ::: DictionaryQA ::: Example and Distractor Generation 
To generate gold questions and answers, we use the same generation templates for definitions exemplified in Figure TABREF8 for WordNetQA  To generate distractors, we simply take alternative definitions for the target words that represent a different word sense (e g , the alternative definitions of gift shown in Table TABREF14), as well as randomly chosen definitions if needed to create a 5-way multiple choice question  As above, we reserve a maximum of 3k examples for training  Since we have only 9k examples in total in this dataset (see WordSense in Table TABREF12), we also reserve 3k each for development and testing 
We note that initial attempts to build this dataset through standard random splitting gave rise to certain systematic biases that were exploited by the choice-only baseline models described in the next section, and hence inflated overall model scores  After several efforts at filtering we found that, among other factors, using definitions from entries without example sentences as distractors (e g , the first two entries in Table TABREF14) had a surprising correlation with such biases  This suggests that possible biases involving differences between dictionary entries with and without examples can taint the resulting automatically generated MCQA dataset (for more discussion on the pitfalls involved with automatic dataset construction, see Section SECREF5) 
Probing Methodology and Modeling
Given the probes above, we now can start to answer the empirical questions posed at the beginning  Our main focus is on looking at transformer-based MCQA models trained in the science domain (using the benchmarks shown in Table TABREF21)  In this section, we provide details of MCQA and the target models, as well as several baselines that we use to sanity check our new datasets  To evaluate model competence, we look at a combination of model performance after science pre-training and after additional model fine-tuning using the lossless inoculation strategy of BIBREF22 (Section SECREF22)  In Section SECREF24, we also discuss a cluster-level accuracy metric for measuring performance over semantic clusters 
Probing Methodology and Modeling ::: Task Definition and Modeling
Given a dataset $D =\lbrace (\textbf {q}^{(d)}, \lbrace a_{1}^{(d)},   , a_{N}^{(d)}\rbrace ) \rbrace _{d}^{\mid D \mid }$ consisting of pairs of questions stems $\textbf {q}$ and answer choices $a_{i}$, the goal is to find the correct answer $a_{i^{*}}$ that correctly answers each $\textbf {q}$  Throughout this paper, we look at 5-way multiple-choice problems (i e , where each $N=5$) 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Question+Answer Encoder 
To model this, our investigation centers around the use of the transformer-based BIBREF25 BERT encoder and fine-tuning approach of BIBREF7 (see also BIBREF6)  For each question and individual answer pair $q^{(j)}_{a_{i}}$, we assume the following rendering of this input:
which is run through the pre-trained BERT encoder to generate a representation for $ q^{(j)}_{a_{i}}$ using the hidden state representation for CLS (i e , the classifier token) $\textbf {c}_{i}$:
The probability of a given answer $p^{(j)}_{i}$ is then computed as $p^{(j)}_{i} \propto e^{\textbf {v}\cdot \textbf {c}^{(j)}_{i}}$, which uses an additional set of classification parameters $\textbf {v} \in \mathbb {R}^{H}$ that are optimized (along with the full transformer network) by taking the final loss of the probability of each correct answer $p_{i^{*}}$ over all answer choices:
We specifically use BERT-large uncased with whole-word masking, as well as the RoBERTa-large model from BIBREF9, which is a more robustly trained version of the original BERT model  Our system uses the implementations provided in AllenNLP BIBREF39 and Huggingface BIBREF40 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks 
When creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42)  To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models 
Following the notation from BIBREF0, for any given sequence $s$ of tokens in $\lbrace q^{(j)}, a_{1}^{(j)},   ,a_{N}^{(j)}\rbrace $ in $D$, an encoding of $s$ is given as $h_{s}^{(j)} = \textbf {BiLSTM}(\textsc {embed}(s)) \in \mathbb {R}^{|s| \times 2h}$ (where $h$ is the dimension of the hidden state in each directional network, and embed$(\cdot )$ is an embedding function that assigns token-level embeddings to each token in $s$)  A contextual representation for each $s$ is then built by applying an element-wise max operation over $h_{s}$ as follows:
With these contextual representations, different baseline models can be constructed  For example, a Choice-Only model, which is a variant of the well-known hypothesis-only baseline used in NLI BIBREF46, scores each choice $c_{i}$ in the following way:
for $\textbf {W}^{T} \in \mathbb {R}^{2h}$ independently of the question and assigns a probability to each answer $p_{i}^{(j)} \propto e^{\alpha _{i}^{(j)}}$ 
A slight variant of this model, the Choice-to-choice model, tries to single out a given answer choice relative to other choices by scoring all choice pairs $\alpha _{i,i^{\prime }}^{(j)} = \textsc {Att}(r^{(j)}_{c_{i}},r^{(j)}_{c_{i^{\prime }}}) \in \mathbb {R}$ using a learned attention mechanism Att and finding the choice with the minimal similarity to other options (for full details, see their original paper)  In using these partial-input baselines, which we train directly on each target probe, we can check whether systematic biases related to answer choices were introduced into the data creation process 
A Question-to-choice model, in contrast, uses the contextual representations for each question and individual choice and an attention model Att model to get a score $\alpha ^{(j)}_{q,i} = \textsc {Att}(r^{(j)}_{q},r^{(j)}_{c_{i}}) \in \mathbb {R}$ as above  Here we also experiment with using ESIM BIBREF47 to generate the contextual representations $r$, as well as a simpler VecSimilarity model that measures the average vector similarity between question and answer tokens: $\alpha ^{(j)}_{q,i} = \textsc {Sim}(\textsc {embed}(q^{(j)}),\textsc {embed}(c^{(j)}_{i}))$  In contrast to the models above, these sets of baselines are used to check for artifacts between questions and answers that are not captured in the partial-input baselines (see discussion in BIBREF49) and ensure that the overall MCQA tasks are sufficiently difficult for our transformer models 
Probing Methodology and Modeling ::: Inoculation and Pre-training
Using the various models introduced above, we train these models on benchmark tasks in the science domain and look at model performance on our probes with and without additional training on samples of probe data, building on the idea of inoculation from BIBREF23  Model inoculation is the idea of continuing to train models on new challenge tasks (in our cases, separately for each probe) using only a small amount of examples  Unlike in ordinary fine-tuning, the goal is not to learn an entirely re-purposed model, but to improve on (or vaccinate against) particular phenomena (e g , our synthetic probes) that potentially deviate from a model's original training distribution (but that nonetheless might involve knowledge already contained in the model) 
In the variant proposed in BIBREF22, for each pre-trained (science) model and architecture $M_{a}$ we continue training the model on $k$ new probe examples (with a maximum of $k=$ 3k) under a set of different hyper-parameter configurations $j \in \lbrace 1,    , J\rbrace $ and identify, for each $k$, the model $M_{*}^{a,k}$ with the best aggregate performance $S$ on the original (orig) and new task:
As in BIBREF22, we found all models to be especially sensitive to different learning rates, and performed comprehensive hyper-parameters searches that also manipulate the number of iterations and random seeds used 
Using this methodology, we can see how much exposure to new data it takes for a given model to master a new task, and whether there are phenomena that stress particular models (e g , lead to catastrophic forgetting of the original task)  Given the restrictions on the number of fine-tuning examples, our assumption is that when models are able to maintain good performance on their original task during inoculation, the quickness with which they are able to learn the inoculated task provides evidence of prior competence, which is precisely what we aim to probe  To measure past performance, we define a model's inoculation cost as the difference in the performance of this model on its original task before and after inoculation 
We pre-train on an aggregated training set of the benchmark science exams detailed in Table TABREF21, and created an aggregate development set of around 4k science questions for evaluating overall science performance and inoculation costs  To handle the mismatch between number of answer choices in these sets, we made all sets 5-way by adding empty answers as needed  We also experimented with a slight variant of inoculation, called add-some inoculation, which involves balancing the inoculation training sets with naturalistic science questions  We reserve the MCQL dataset in Table TABREF21 for this purpose, and experiment with balancing each probe example with a science example (x1 matching) and adding twice as many science questions (x2 matching, up to 3k) for each new example 
Probing Methodology and Modeling ::: Evaluating Model Competence
The standard way to evaluate our MCQA models is by looking at the overall accuracy of the correct answer prediction, or what we call instance-level accuracy (as in Table TABREF25)  Given the nature of our data and the existence of semantic clusters as detailed in Section SECREF11 (i e , sets of questions and answers under different distractor choices and inference complexity), we also measure a model's cluster-level (or strict cluster) accuracy, which requires correctly answering all questions in a cluster  Example semantic clusters are shown in Table TABREF30; in the first case, there are 6 ISA$^\uparrow $ questions (including perturbations) about the concept trouser n 01 (e g

ground_truth: SVM with unigram, bigram, and trigram features, SVM with average word embedding, SVM with average transformed word embeddings, CNN, ecurrent Convolutional Neural Networks, SVM and deep learning models with comment information

SVM with unigram, bigram, trigram features, with average word embedding, with average transformed word embeddings, CNN and RCNN, SVM, CNN, RCNN with comment information

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is WordNet useful for taxonomic reasoning for this task?
Context 3: , involving knowing that trousers are a type of consumer good and garment/clothing), which a model must answer in order to receive full credit 
Our cluster-based analysis is motivated by the idea that if a model truly knows the meaning of a given concept, such as the concept of trousers, then it should be able to answer arbitrary questions about this concept without sensitivity to varied distractors  While our strict cluster metric is simplistic, it takes inspiration from work on visual QA BIBREF53, and allows us to evaluate how consistent and robust models are across our different probes, and to get insight into whether errors are concentrated on a small set of concepts or widespread across clusters 
Results and Findings
In this section, we provide the results of the empirical questions first introduced in Figure FIGREF1, starting with the results of our baseline models 
Results and Findings ::: Are our Probes Sufficiently Challenging?
As shown in Table TABREF25, most of our partial-input baselines (i e , Choice-Only and Choice-to-Choice models) failed to perform well on our dataset probes across a wide range of models, showing that such probes are generally immune from biases relating to how distractors were generated  As already discussed in Section SECREF13, however, initial versions of the DictionaryQA dataset had unforeseen biases partly related to whether distractors were sampled from entries without example sentences, which resulted in high Choice-Only-GloVe scores ranging around 56% accuracy before a filtering step was applied to remove these distractors 
We had similar issues with the hypernymy probe which, even after a filtering step that used our Choice-to-Choice-GloVe model, still leads to high results on the BERT and RoBERTa choice-only models  Given that several attempts were made to entirely de-duplicate the different splits (both in terms of gold answers and distractor types), the source of these biases is not at all obvious, which shows how easy it is for unintended biases in expert knowledge to appear in the resulting datasets and the importance of having rigorous baselines  We also note the large gap in some cases between the BERT and RoBERTa versus GloVe choice-only models, which highlights the need for having partial-input baselines that use the best available models 
Using a more conventional set of Task-Specific QA models (i e , the LSTM-based Question-to-Choice models trained directly on the probes), we can see that results are not particularly strong on any of the datasets, suggesting that our probes are indeed sufficiently challenging and largely immune from overt artifacts  The poor performance of the VecSimilarity (which uses pre-trained Word2Vec embeddings without additional training) provides additional evidence that elementary lexical matching strategies are insufficient for solving any of the probing tasks 
Results and Findings ::: How well do pre-trained MCQA models do?
Science models that use non-transformer based encoders, such as the ESIM model with GloVe and ELMO, perform poorly across all probes, in many cases scoring near random chance, showing limits to how well they generalize from science to other tasks even with pre-trained GloVe and ELMO embeddings  In sharp contrast, the transformer models have mixed results, the most striking result being the RoBERTa models on the definitions and synonymy probes (achieving a test accuracy of 77% and 61%, respectively), which outperform several of the task-specific LSTM models trained directly on the probes  At first glance, this suggests that RoBERTa, which generally far outpaces even BERT across most probes, has high competence of definitions and synonyms even without explicit training on our new tasks 
Given the controlled nature of our probes, we can get a more detailed view of how well the science models are performing across different reasoning and distractor types, as shown in the first column of Figure FIGREF28 for ESIM and RoBERTa  The ESIM science model without training has uniformly poor performance across all categories, whereas the performance of RoBERTa is more varied  Across all datasets and number of hops (i e , the rows in the heat maps), model performance for RoBERTa is consistently highest among examples with random distractors (i e , the first column), and lowest in cases involving distractors that are closest in WordNet space (e g , sister and ISA, or up/down, distractors of distance $k^{\prime }=1$)  This is not surprising, given that, in the first case, random distractors are likely to be the easiest category (and the opposite for distractors close in space), but suggests that RoBERTa might only be getting the easiest cases correct 
Model performance also clearly degrades for hypernymy and hyponymy across all models as the number of hops $k$ increases (see red dashed boxes)  For example, problems that involve hyponym reasoning with sister distractors of distance $k^{\prime }=1$ (i e , the second column) degrades from 47% to 15% when the number of hops $k$ increases from 1 to 4  This general tendency persists even after additional fine-tuning, as we discuss next, and gives evidence that models are limited in their capacity for certain types of multi-hop inferences 
As discussed by BIBREF26, the choice of generation templates can have a significant effect on model performance  The results so far should therefore be regarded as a lower bound on model competence  It is possible that model performance is high for definitions, for example, because the associated templates best align with the science training distribution (which we know little about)  For this reason, the subsequent inoculation step is important—it gives the model an opportunity to learn about our target templates and couple this learned knowledge with its general knowledge acquired during pre-training and science training (which is, again, what we aim to probe) 
Results and Findings ::: Can Models Be Effectively Inoculated?
Model performance after additional fine-tuning, or inoculation, is shown in the last 3 rows of Table TABREF25, along with learning curves shown in Figure FIGREF29 for a selection of probes and models  In the former case, the performance represents the model (and inoculation amount) with the highest aggregate performance over the old task and new probe  Here we again see the transformer-based models outperform non-transformer models, and that better models correlate with lower inoculation costs  For example, when inoculating on synonymy, the cost for ESIM is around 7% reduced accuracy on its original task, as opposed to $< 1$% and around 1% for BERT and RoBERTa, respectively  This shows the high capacity for transformer models to absorb new tasks with minimal costs, as also observed in BIBREF22 for NLI 
As shown in Figure FIGREF29, transformer models tend to learn most tasks fairly quickly while keeping constant scores on their original tasks (i e , the flat dashed lines observed in plots 1-4), which gives evidence of high competence  In both cases, add-some inoculation proves to be a cheap and easy way to 1) improve scores on the probing tasks (i e , the solid black and blue lines in plot 1) and; 2) minimize loss on science (e g , the blue and black dashed lines in plots 2-4)  The opposite is the case for ESIM (plots 5-6); models are generally unable to simultaneously learn individual probes without degrading on their original task, and adding more science data during inoculation confuses models on both tasks 
As shown in Figure FIGREF28, RoBERTa is able to significantly improve performance across most categories even after inoculation with a mere 100 examples (the middle plot), which again provides strong evidence of prior competence  As an example, RoBERTa improves on 2-hop hyponymy inference with random distractors by 18% (from 59% to 77%)  After 3k examples, the model has high performance on virtually all categories (the same score increases from 59% to 87%), however results still tends to degrade as a function of hop and distractor complexity, as discussed above 
Despite the high performance of our transformer models after inoculation, model performance on most probes (with the exception of Definitions) averages around 80% for our best models  This suggests that there is still considerable room for improvement, especially for synonymy and word sense, which is a topic that we discuss more in Section SECREF6 
Results and Findings ::: Are Models Consistent across Clusters?
Table TABREF32 shows cluster-level accuracies for the different WordNetQA probes  As with performance across the different inference/distractor categories, these results are mixed  For some probes, such as definitions, our best models appear to be rather robust; e g , our RoBERTa model has a cluster accuracy of $75\%$, meaning that it can answer all questions perfectly for 75% of the target concepts and that errors are concentrated on a small minority (25%) of concepts  On synonymy and hypernymy, both BERT and RoBERTa appear robust on the majority of concepts, showing that errors are similarly concentrated  In contrast, our best model on hyponymy has an accuracy of 36%, meaning that its errors are spread across many concepts, thus suggesting less robustness 
Table TABREF30 shows a selection of semantic clusters involving ISA reasoning, as well as the model performance over different answers (shown symbolically) and perturbations  For example, in the the second case, the cluster is based around the concept/synset oppose v 06 and involves 4 inferences and a total 24 questions (i e , inferences with perturbations)  Our weakest model, ESIM, answers only 5 out of 24 questions correctly, whereas RoBERTa gets 21/24  In the other cases, RoBERTa gets all clusters correct, whereas BERT and ESIM get none of them correct 
We emphasize that these results only provide a crude look into model consistency and robustness  Recalling again the details in Table TABREF12, probes differ in terms of average size of clusters  Hyponymy, in virtue of having many more questions per cluster, might simply be a much more difficult dataset  In addition, such a strict evaluation does not take into account potential errors inside of clusters, which is an important issue that we discuss in the next section  We leave addressing such issues and coming up with more insightful cluster-based metrics for future work 
Discussion and Conclusion
We presented several new challenge datasets and a novel methodology for automatically building such datasets from knowledge graphs and taxonomies  We used these to probe state-of-the-art open-domain QA models (centering around models based on variants of BERT)  While our general methodology is amendable to any target knowledge resource or QA model/domain, we focus on probing definitions and ISA knowledge using open-source dictionaries and MCQA models trained in the science domain 
We find, consistent with recent probing studies BIBREF26, that transformer-based models have a remarkable ability to answer questions that involve complex forms of relational knowledge, both with and without explicit exposure to our new target tasks  In the latter case, a newer RoBERTa model trained only on benchmark science tasks is able to outperform several task-specific LSTM-based models trained directly on our probing data  When re-trained on small samples (e g , 100 examples) of probing data using variations of the lossless inoculation strategy from BIBREF22, RoBERTa is able to master many aspects of our probes with virtually no performance loss on its original QA task 
These positive results suggest that transformer-based models, especially models additionally fine-tuned on small samples of synthetic data, can be used in place of task-specific models used for querying relational knowledge, as has already been done for targeted tasks such as word sense disambiguation BIBREF54  Since models seem to already contain considerable amounts of relational knowledge, our simple inoculation strategy, which tries to nudge models to bring out this knowledge explicitly, could serve as a cheaper alternative to recent attempts to build architectures that explicitly incorporate structured knowledge BIBREF55; we see many areas where our inoculation strategy could be improved for such purposes, including having more complex loss functions that manage old and new information, as well as using techniques that take into account network plasticity BIBREF56 
The main appeal of using automatically generate datasets is the ability to systematically manipulate and control the complexity of target questions, which allows for more controlled experimentation and new forms of evaluation  Despite the positive results described above, results that look directly at the effect of different types of distractors and the complexity of reasoning show that our best models, even after additional fine-tuning, struggle with certain categories of hard distractors and multi-hop inferences  For some probes, our cluster-based analysis also reveals that errors are widespread across concept clusters, suggesting that models are not always consistent and robust  These results, taken together with our findings about the vulnerability of synthetic datasets to systematic biases, suggest that there is much room for improvement and that the positive results should be taken with a grain of salt  Developing better ways to evaluate semantic clusters and model robustness would be a step in this direction 
We emphasize that using synthetic versus naturalistic QA data comes with important trade-offs  While we are able to generate large amounts of systematically controlled data at virtually no cost or need for manual annotation, it is much harder to validate the quality of such data at such a scale and such varying levels of complexity  Conversely, with benchmark QA datasets, it is much harder to perform the type of careful manipulations and cluster-based analyses we report here  While we assume that the expert knowledge we employ, in virtue of being hand-curated by human experts, is generally correct, we know that such resources are fallible and error-prone  Initial crowd-sourcing experiments that look at validating samples of our data show high agreement across probes and that human scores correlate with the model trends across the probe categories  More details of these studies are left for future work 

ground_truth: Unanswerable

Yes