ChemistryQA: A Complex Question Answering Dataset from Chemistry

1 INTRODUCTION . Recent years have witnessed huge advances for the question answering ( QA ) task , and some AI agents even beat human beings . For example , IBM Watson won Jeopardy for answering questions which requires a broad range of knowledge ( Ferrucci , 2012 ) . Transformer-based neural models , e.g . XLNet ( Yang et al. , 2019 ) and RoBERTa ( Liu et al. , 2019 ) , beat human beings on both machine reading comprehension and conversational QA task . Ariso System ( Clark et al. , 2019 ) gets an ’ Ace ’ for an eighth-grade science examination and is able to give 80 percent correct answers for 12th-grade science test . Most solutions of the QA task fall into two categories , end-to-end solution and parsing plus execution . The former predicts answers with an end-to-end neural network , e.g. , Reading comprehension QA ( Rajpurkar et al. , 2016 ; 2018 ; Lai et al. , 2017 ) and Science Exam QA ( Clark et al. , 2019 ; 2018 ) . The latter translates a question into a specific structural form which is executed to get the answer . For example , in knowledge-based question answering ( KBQA ) ( Berant et al. , 2013 ; Yih et al. , 2016 ; Saha et al. , 2018 ) questions are parsed into SPARQL-like queries consisting of predicates , entities and operators . In Math Word Problem ( Huang et al. , 2016 ; Amini et al. , 2019 ) questions are translated to stacks of math operators and quantities . However , in the real world , many QA tasks can not be solved by end-to-end neural networks and it is also very difficult to translate questions into any kind of formal representation . Solving Chemical Calculation Problems is such an example . Chemical Calculation Problems can not be solved by end-to-end neural networks since complex symbolic calculations are required . It is also difficult to translate such problems into formal representations , since not all operators in solving processes occur in question stems , which makes it difficult to annotate data and train models . Table 1 shows a question in ChemistryQA . To answer the question in Table 1 , machines need to : 1 ) understand the question and extract variable to be solved and conditions in the question ; 2 ) retrieve and apply related chemistry knowledge , including calculating molarity by mole and volume , balancing a chemical equation and calculating the equilibrium constant K , although there is no explicit statement for ” calculating molarity ” and ” balancing equations ” in the question . The combination of these capabilities is scarcely evaluated well by existing QA datasets . In order to foster the research on this area , we create a dataset of chemical calculation problems , namely ChemstriyQA . We collect about 4,500 chemical calculation problems from https : //socratic.org/ chemistry , covering more than 200 topics in chemistry . Besides the correct answer , we also label the target variable and conditions provided in a question . Such additional labels facilitate potential data augmentation and inferring golden solving process for training . To verify the dataset is consistent with the purpose of evaluating AI ’ comprehensive capability and help other researchers ramp up , we build two baselines as follows . a ) We build a BERT based sequence to sequence model , which take the raw question as input and the answer as output . The first baseline achieves 0.164 precision on ChemistryQA . b ) We create an extraction system which extracts the target variable and conditions from raw questions . The extracted structure information is fed into a graph searching based solver , which performs a sequence of calculating and reasoning to get the final answer . The second baseline achieves 0.169 precision on ChemistryQA . In summary , our contribution of this paper is shown as follows . • We propose a new QA task , ChemistryQA , which requires open knowledge and complex solving processes . ChemistryQA is different with other existing QA tasks , and can not be solved by existing QA methods very well . • We create a ChemistryQA dataset , which contains about 4,500 chemical calculation problems and covers more than 200 topics in chemistry . In this dataset , we provide a novel annotation for questions , which only labels the variable asked and conditions from question stem but not solving processes . This annotation is much easier and cost less effort , and it is flexible for researchers to explore various of solutions as a weakly supervised dataset . • We build two baselines to prove : a ) end-to-end neural networks can not solve this task very well ; b ) the annotation we provided can be used to improve a simple graph search based solver . 2 CHEMISTRYQA DATASET . 2.1 DATA COLLECTION . We collect chemical calculation problems from https : //socratic.org/chemistry . It this website , there are more than 30,000 questions which cover about 225 chemistry related topics , e.g. , Decomposition Reactions , Ideal Gas Law and The Periodic Table . There is an example of annotation page in Appendix A . Figure 2.A shows the original page in Socratic , which contains a raw question , an answer and probably a description of solving process . We filter raw questions by a simple rule , and only keep questions with a numerical value , a chemical formula or a chemical equation as answers . 2.2 DATA ANNOTATION . Unlike similar tasks ’ annotation , we can not collect all the atomic operations needed before starting annotation , since the set of chemical operators is not closed . Therefore , we propose a novel annotation method that only the target variable and all conditions will be labeled in a triple-like format . For instance in Figure 2 , the target variable is labeled as ( subject = reaction , predicate = Equilibrium constant K , object = ? ) , and one of conditions is labeled as ( subject = N2 , predicate = Mole , object = 2.80× 10−4 mol ) . Therefore , for a question in a link , parts to be annotated are question stems , correct answers , the target variable and all conditions . Figure 2.B shows our annotation page for a question link . For questions and answers , we ask annotators to copy them into corresponding forms . If there are typos or obvious mistakes , we also ask annotators to correct them . For the target variable and conditions , we break them down into several types : physical unit , chemical formula , chemical equation , substance name and other . We also design easy-to-understand annotation interfaces , e.g. , ( [ BLANK ( predicate ) ] OF [ BLANK ( subject ) ] IN [ BLANK ( unit or None ) ] ) and ( [ BLANK ( predicate ) ] OF [ BLANK ( subject ) ] = [ BLANK ( object or value ) ] ) for tagging the physical unit from the raw question as variables and conditions , respectively . More detail about other types ’ definitions and annotation interfaces are shown in Appendix A . We employed crowdsourcing for this annotation work . The task was split into 6 batches and assigned to annotators sequentially . We applied some check-and-verify mechanism in first batch to ensure the annotation quality , also help annotators be more familiar with this task . Finally , we have collected 4418 annotated questions within around 336 hours . During the annotating phase , we encourage annotators to use text phrase in original questions whenever possible for chemical formula , chemical equation , substance name , subject and value in physical unit , while for predicates and units , we do not make any restrictions . We maintain two dynamic mappings to convert mentions labeled to identified predicates or unites , which greatly reduces the difficulty of labeling and the total annotation time . For other , there is not any restrictions either , and we only consume identified ones , e.g. , STP . 2.3 DATA ANALYSIS . We divide annotated questions into train , valid and test subsets , and their sizes are 3433 , 485 and 500 , respectively . We make some statistics on the annotated questions in different perspectives as follows . 1 ) According to the types of target variables , we divide questions into 3 classes , physical unit , chemical formula , chemical equation . Table 2 shows examples belonging to different question types , and Table 3 shows the distribution of question types . 2 ) There are 172 unique predicates , 90 unique units and 25 identified other conditions . We conducted detailed statistics on them in Appendix B , 2.4 COMPARING WITH OTHER QA DATASETS . We pick a representative dataset from each type of task to compare with ChemistryQA , including WEBQUESTIONS ( Berant et al. , 2013 ) , RACE ( Lai et al. , 2017 ) , ARC ( Clark et al. , 2018 ) and MathQA ( Amini et al. , 2019 ) . We compare these QA datasets in Answer Type , External Knowledge , Knowledge usage , Calculation and Annotation perspectives , and Table 4 shows the detail . Comparing ChemistryQA with existing QA datasets , ChemistryQA has the following advantages : 1 ) ChemistryQA contains more diverse answer types and excludes the influence of randomness by not providing options . 2 ) There are various knowledge required by ChemistryQA including a ) triplet-like fact , e.g. , substances ’ molar mass , colour and other physical properties , b ) calculation methods between various physical quantities and c ) domain specific skills , e.g. , balancing chemical equations . The knowledge in ChemsitryQA is open and used in various ways , while other datasets use knowledge in single way . 3 ) ChemistryQA only provides triplet like extraction annotation which isolates language understanding and domain knowledge as much as possible . This setting makes annotation and model training easier . 3 METHODS . We provide two completely different baselines : 1 ) an end-to-end neural based solver and 2 ) a solving pipeline composed of an extractor and a graph search based solver . 3.1 END TO END SOLVER . We build a sequence to sequence model , and both of its encoder and decoder are based on BERT model . Both encoder and decoder load from pretrained BERT and share the same vocabulary , more than 30,000 sub-tokens from BERT . To build the decoder , we change the encoder ’ s structure as Vaswani et al . ( 2017 ) did : 1 ) the self-attention of decoder has a triangular mask matrix and 2 ) there is an extra layer of attention performing over outputs of the encoder and hidden states of the decoder . We also append a head of predicting next token to the decoder , which maps hidden states into the vocabulary size space Rv and follows a softmax layer behind it . The end-to-end solver takes the question as the input of encoder and takes the answer as the target of decoder . Questions are split into sub-tokens , and even real numbers also break into sub-tokens . We greedily choose the next token with maximum score after softmax . We append a special token , ’ [ SEP ] ’ , as the end of the target sequence . During inference , the decoding process ends when the decoder outputs ’ [ SEP ] ’ token.This method represents a class of powerful neural networks , which achieved state-of-the-art performance on many QA tasks . 3.2 EXTRACTOR PLUS GRAPH SEARCH BASED SOLVER . As the second baseline , we build an extractor plus solver pipeline . First , we employ the extractor to extract the target variable and conditions from the question text . The target variable and conditions are represented as triplets as described in the above Data Annotation Section . Second , we employ a graph search based solver to take triplets as input and execute pre-defined functions in chemistry domain to get the final answer . Figure 1 shows the structure of the extractor plus solver pipeline .

Summary: This paper proposes a new dataset based on textbook / classroom chemistry questions for complex knowledge retrieval and aggregation. The authors scrape several thousands questions from online repositories and add additional natural language annotations signifying the quantities to be solved for in each question, as well as the declarative knowledge. Two baselines, one end-to-end neural and another symbolic, both fail at this dataset.