ChemistryQA: A Complex Question Answering Dataset from Chemistry
Authors: Zhuoyu Wei
Wei Ji
Xiubo Geng
Yining Chen
Baihua Chen
Tao Qin
Daxin Jiang
Venue:  (2021)

Abstract: Many Question Answering (QA) tasks have been studied in NLP and employed to evaluate the progress of machine intelligence. One kind of QA tasks, such as Machine Reading Comprehension QA,  is well solved by end-to-end neural networks; another kind of QA tasks, such as Knowledge Base QA,  needs to be translated to a formatted representations and then solved by a well-designed solver. We notice that some real-world QA tasks are more complex, which cannot be solved by end-to-end neural networks or translated to any kind of formal representations. To further stimulate the research of QA and development of QA techniques, in this work, we create a new and complex QA dataset, ChemistryQA,  based on real-world chemical calculation questions. To answer chemical questions, machines need to understand questions, apply chemistry and Math knowledge, and do calculation and reasoning. To help researchers ramp up, we build two baselines: the first one is BERT-based sequence to sequence model, and the second one is an extraction system plus a graph search based solver. These two methods achieved 0.164 and 0.169 accuracy on the development set, respectively, which clearly demonstrate that new techniques are needed for complex QA tasks. ChemistryQA dataset will be available for public download once the paper is published.

Under review as a conference paper at ICLR 2021
CHEMISTRYQA: A COMPLEX QUESTION ANSWERING DATASET FROM CHEMISTRY
Anonymous authors Paper under double-blind review
ABSTRACT
Many Question Answering (QA) tasks have been studied in NLP and employed to evaluate the progress of machine intelligence. One kind of QA tasks, such Machine Reading Comprehension QA, is well solved by end-to-end neural networks; another kind of QA tasks, such as Knowledge Base QA, needs to be translated a formatted representations and then solved by a well-designed solver. We notice that some real-world QA tasks are more complex, which cannot be solved by end-to-end neural networks or translated to any kind of formal representations. To further stimulate the research of QA and development of QA techniques, in this work, we create a new and complex QA dataset, ChemissryyAA. based on real-world chemical calculation questions. To answer chemical questions, machines need to understand questions, apply chemistry and math knowledge, and do calculation and reasoning. To help researchers ramp up, we build two baselines: the first one is BERT-based sequence to sequence model, and the second one is an extraction system plus a graph search based solver. These methods achieved 0.164 and 0.169 accuracy on the development set, respectively, which clearly demonstrates that new techniques are needed for complex QA tasks. Chemistry QA dataset will be available for public download once the paper is published.
INTRODUCTION
Recent years have witnessed huge advances for the question answering (QA) task, and some AI agents even beat human beings. For example, IBM Watson won Jeopardy for answering questions which requires a broad range of knowledge (Ferrucei) 2012). Transformer-baat
neural models, e.g.
XLNet (Yang al., (019)) and RoBERTa (Liu et al. 2019), beat human beings on both machine reading comprehension and conversational QA task. Ariso System (Clark et al. 2019) gets an 'Ac'' for eighth-grade science examination and is able give 80 percent correct answers for 12th-grade science test.
Most solutions of the QA task fall into two categories, end-to-end solution and parsing plus execution. The former predicts answers with an end-to-end neural network, e.g., Reading comprehension QA (Rajpurkar et al. 2016: 2018: Lai et al. 2017) and Science Exam QA (Clark al. 2019: 2018). The latter translates a question into a specific structural form which is executed to get the answer. For example, in knowledge-basse
question answering (KBQA) (Berant et al.) 2013 Yih et al. 2016 [Sha ct al. 2018) 1) questions are parsed into SPARQL- queries consisting of predicates, entities and operators. In Math Word Problem (Huang et al. 2016 Amini al. 2019 questions are translated to stacks of math operators and quantities.
However, the real world, many QA tasks cannot be solved by end-to-end neural networks and is also very difficult to translate questions into any kind of formal representation. Solving Chemical Calculation Problems is such an example. Chemical Calculation Problems cannot be solved by end-to-end neural networks since complex symbolic calculations translate such problems into formal representations, since not all operators in solving processes occur in question stems, which makes it difficult to annotate data and train models.
required. It also difficult to
Table D shows a question in ChemistryQA. To answer the question in Table I machines need to: 1) understand the question and extract variable to be solved and conditions in the question; 2) retrieve and apply related chemistry knowledge, including calculating molarity by mole and volume, balancing a chemical equation and calculating the equilibrium constant K, although there is explicit statement

---

Under review as a conference paper at ICLR 2021
Table 1: An Example in ChemistryQA.
Question
At particular temperature 2.00 flask at equilibrium contains 2.80 10-4 mol N2. 2.50 x 10-5 mol O2, and 2.00 x 10-a mol N2O. How would you calculate K at this temperature for the following reaction: Nz(g) + O2(g) 7 (g)
Variable to be solved
Equilibrium constant, K of this reaction
Conditions provided
Volume of the flask is 2.00L..
Mole of N2 is 2.80 x 10 mol.
Mole of O2 is 2.50 10 mol.
Mole of N2O is 2.00 x 10-2 mol.
Reaction equation is N2(g) + O2(g) N2)(g).
Knowledge required
K =
1. N
is the molarity of #, and a,b,c are coefficients of matters.
Molarity Mole Volume.
Balance the reaction equation following atomic conservation theory.
Solving
1. Balance the reaction equation get a.b and
steps
2. Calculate molarities for N2, O2 and N₂O
Calculate following K's formula
Answer
4.08 10
for "calculating molarity" and "balancing equations" in the question. The combination of these capabilities is scarcely evaluated well by existing QA datasets. In order to foster the research on this area, create dataset of chemical calculation problems, namely ChemstriyQA.
We collect about 4,500 chemical calculation problems from chemistryl covering more than 200 topies in chemistry. Besides the correct answer, also label the target variable and conditions provided a question. Such additional labels facilitate potential data augmentation and inferring golden solving process for training.
To verify the dataset is consistent with the purpose of evaluating Al' comprehensive capability and help other researchers ramp up, we build two baselines follows. a) We build BERT based sequence to sequence model, which take the raw question as input and the answer as output. The first baseline achieves 0.164 precision on ChemistryQA. b) We create an extraction system which extracts the target variable and conditions from raw questions. The extracted structure information is fed into a graph searching based solver, which performs a sequence of calculating and reasoning get the final answer. The second baseline achieves O. 166 precision on ChemistryQA.
In summary, our contribution of this paper is shown as follows.
We propose new QA task, ChemistryQA, which requires open knowledge and complex solving processes. ChemistryQA is different with other existing QA tasks, and cannot be solved by existing QA methods very well.
We create ChemistryQA dataset, which contains about 4,500 chemical calculation prob- lems and covers more than 200 topics in chemistry. In this dataset, we provide novel annotation for questions, which only labels the variable asked and conditions from question stem but not solving processes. This annotation much easier and cost less effort, and is flexible for researchers explore various of solutions
weakly supervised dataset.
We build two baselines to prove: a) end-to-end neural networks cannot solve this task very well; b) the annotation we provided can be used to improve a simple graph search based solver.
2 CHEMISTRYQA DATASET
2.1 DATA COLLECTION
We collect chemical calculation problems from website, there are more than 30,000 questions which cover about 225 chemistry related topics, e.g., Decomposition Reactions, Ideal Gas Law and The Periodic Table. There an example of annotation page in Appendix A Figure 2 A shows the original page in Socratic, which contains raw question,
It this
2

---

Under review as a conference paper at ICLR 2021
answer and probably description of solving process. We filter questions by simple rule, and only keep questions with a numerical value, chemical formula or a chemical equation as answers.
2.2 DATA ANNOTATION
Unlike similar tasks' annotation, we cannot collect all the atomic operations needed before starting annotation, since the set of chemical operators is not closed. Therefore, we propose a novel annotation method that only the target variable and all conditions will be labeled in triple-like format. For instance in Figure 2 the target variable is labeled as (subject = reaction, predicate Equilibrium constant K. object ?), and one of conditions is labeled as (subject = N2, predicate Mole, object 2.80 x 10-4 mol).
Therefore, for question in a link, parts to be annotated are question stems, correct answers, the target variable and all conditions. Figure B shows our annotation page for a question link. For questions and answers, ask annotators copy them into corresponding forms. If there typos obvious mistakes, we also ask annotators to correct them. For the target variable and conditions, we break them down into several types: physical unit, chemical formula, chemical equation, substance name and other. We also design easy-to-understand annotation interfaces, e.g., ([BLANK (predicate) OF [BLANK (subject)] IN (BLANK (unit None)[) and ([BLANK (predicate)] OF [BLANK (subject)) [BLANK (object or value)]) for tagging the physical unit from the raw question as variables and conditions, respectively. More detail about other types' definitions and annotation interfaces are shown in Appendix A
We employed crowdsoureing for this annotation work. The task was split into batches and assigned to annotators sequentially. We applied some check-and-vente mechanism in first batch to ensure the annotation quality, also help annotators be more familiar with this task. Finally, we have collected 4418 annotated questions within around 336 hours.
During the annotating phase, we encourage annotators to use text phrase in original questions whenever possible for chemical formula, chemical equation, substance name, subject and value in physical unit, while for predicates and units, we do not make any restrictions. We maintain two dynamic mappings to convert mentions labeled to identified predicates or unites, which greatly reduces the difficulty of labeling and the total annotation time. For other, there is not any restrictions either, and only consume identified ones, e.g., STP. STP.
2.3 DATA ANALYSIS
We divide annotated questions into train, valid and test subsets, and their sizes are 3433, 485 and 500, respectively. We make some statistics on the annotated questions in different perspectives as follows.
1) According the types of target variables. we divide questions into classes, physical unit. chemical formula, chemical equation. Table 2 shows examples belonging to different question types, and Table shows the distribution of question types.
Table 2: Examples under various question types
Question
Answer
Question Type
How many moles of ammonium nitrate
0.14 moles
Physical Unit
in 335 mL of 0.425 M. NH4NO3?
What the empirical formula mag- MgC12
Chemical Foruula
nesium chloride if 0.96 of magnesium
combines with 2.84 of chlorine?
How would you write balanced equa- tion for the combustion of octane, C8H18 with oxygen to obtain carbon dioxide and water?
20SHI8 2502 16002+ 18H20
Chemical Equation
Table 3: Distribution of different question types
Dataset
Physical Unit
Chemical For- Chemical
Total
mula
Equation
Train
2,721
314
398
3.433
Valid
381
55
49
485
Test
392
55
53
500
Total
3,494
424
500
4,418
3

---

Under review as a conference paper at ICLR 2021
2) There are 172 unique predicates, 90 unique units and 25 identified other conditions. We conducted detailed statistics on them in Appendix B
2.4 COMPARING WITH OTHER QA DATASETS
We pick a representative dataset from each type of task to compare with Chemistry QA, including WEBBQUESTIONS(Beeaat et al. 2013), RACE et al. 2017), ARC(Clark et al.) 2018) and MathQA(Amini et 2019). We compare these QA datasets in Answer Type, External Knowledge, Knowledge usage, Calculation and Annotation perspectives, and Table 4 shows the detail.
Table 4: ChemistryQA Compares with existing related QA tasks.
Dataset
Answer Type
Extenal
Knowledge us-
Calculation
Annotation
Knowledge
age
WEBQUESTIONS Entiy, Entities
Open
Graph Search
SPARQL Oper-
SPARQL, An-
ators
swer
RACE
Option
None
None
Language
Un-
Only Answer
derstanding
ARC
Option
Open
Implicit Infer-
None
Only Answer
ence
MathQA
Option
Closed
Language derstanding
Un-
Math Operators
Operator Stack, Answer
ChemistryQA
Value, Formula,
Open
Graph Search, Language Un- derstanding and
Chemical Oper- ators
Variable, Condi- tions, Answer
Equation
Calculating
Comparing ChemistryQA with existing QA datasets, ChemistryQA has the following advantages:
1) ChemistryQA contains more diverse answer types and excludes the influence of randomness by not providing options.
2) There are various knowledge required by ChemistryQA including a) triplet-like fact, e.g., sub- stances' molar mass, colour and other physical properties, b) calculation methods between various physical quantities and c) domain specific skills, e.g., balancing chemical equations. The knowledge in ChemsitryQA is open and used in various ways, while other datasets use knowledge in single way.
3) Chemistry only provides triplet like extraction annotation which isolates language understand- ing and domain knowledge as much as possible. This setting makes annotation and model training easier.
3 METHODS
We provide two completely different baselines: 1) an end-to-end neural based solver and 2) solving pipeline composed of an extractor and a graph search based solver.
3.1 END TO END SOLVER
We build a sequence to sequence model, and both of its encoder and decoder are based on BERT model. Both encoder and decoder load from pretrained BERT and share the same vocabulary, more than 30,000 sub-tokens from BERT. To build the decoder, we change the encoder's structure as Vaswani et al. (201)) did: 1) the self-attention of decoder has triangular mask matrix and 2) there is an extra layer of attention performing over outputs of the encoder and hidden states of the decoder. We also append head of predicting next token to the decoder, which maps hidden states into the vocabulary size space R" and follows a softmax layer behind it. The end-to-end solver takes the question as the input of encoder and takes the answer as the target decoder. Questions are split into sub-tokens, and even real numbers also break into sub-tokens. We greedily choose the next token with maximum score after softmax. We append a special token, '[SEP]', as the end of the target sequence. During inference, the decoding process ends when the decoder outputs '[SEP]' token. This method represents a class of powerful neural networks, which achieved state-of thetthrrrtt performance on many QA tasks.
4

---

Under review as a conference paper at ICLR 2021
particular temperature 2.00 L flask at equilibrium contains 2.80 10 mol Ny... temperature for following reaction: Nz(g) 0g(g) N2((g)
Graph Search
Based Solver
FSA
based
12 Transformers
Question
<reaction, equilibrium constant k,
Functions Domain
Extractor
Encoder
<tlask, volume, 2.00
(Chemistry)
<N2, male, 2.80 10 mal>
Conditions
Transformers Decoder
<reaction, chemical equation, N2(9) + (2(g) Ng)(g)
DFS Solving Engine
Restrain
Finite State Automata
Train
Infer
<reaction, equilibrium constant 4.08 x 10 >
start physical unit 28 equilibrium constant None qe_end ... physical_ _nit 13 13 11 12 mole qc_end ... cneemicaleequatine 30 34 qc_end end
Figure 1: The structure of Extractor plus Graph Search Based Solver pipeline
3.2 EXTRACTOR PLUS GRAPH SEARCH BASED SOLVER
As the second baseline, we build an extractor plus solver pipeline. First, we employ the extractor to extract the target variable and conditions from the question text. The target variable and conditions are represented as triplets as described in the above Data Annotation Section. Second, we employ graph search based solver take triplets as input and execute pre-defined functions chemistry domain to get the final answer. Figure shows the structure of the extractor plus solver pipeline.
3.2.1 FSA BASED EXTRACTOR
take the question input and output triplet like variable and conditions, so a sequence to sequence model is a good candidate. However, such straight forward method is hard to work, because triplets structural and also want extract simple semantic information, i.e., whether a triplet is target variable or condition and types of triplets. Therefore, design Finite State Automata (FSA) restrict the type of each output token. We define 12 separate vocabulary sets, and the Table in Appendix C shows the these vocabulary names, sizes and tokens belonging to them. distinguish between vocabulary and token, we use upper cases to represent vocabulary names and lower cases to represent tokens. START, END, QC END are control symbols. START and END represent the beginning and end of the decoding sequence, while QC_END represents the end of the target variable or a condition block. PHYSICAL_UNIT, CHEMICAL AEQUAAAIO,,, CHEMICAL.FORMULA SUBSTANCE, C.OTHER are types of target variables or conditions. POINTER contains all possible positions of token in question, which can be used to represent the start or end a text span, e.g., subjects, chemical equations and values.
We expect the extractor
For the model, employ standard pretrained BERT based model the encoder. We still use 6 layers transformers as the decoder model, and use a triangular mask matrix as the attention mask. We state transitions represent the relations between output tokens and build FSA model the state transition process. The decoder always outputs "start" as the first token, determines the current state and chooses the next vocabulary based the current output and state. Figure 1 also shows example of the output of FSA based Extractor.
In the training stage, convert triplet-like annotations to the FSA based token sequence, and treat them as the target of the extractor. For example in Figure 1, the target variable, <reaction, equilibrium constant k, ?>. is translated to physical 28 28 cinlibbriumcconsttne and one of conditions, rrectionn, chemical equation, N2(g) + O2(g) + N₂O(g) > is translated to chemical equation 30 qc end. In the inference stage, we obtain the FSA based token sequence from the decoder, and then we convert it to triplets as the input of subsequent solver. Since there is no conflicts FSA, both of directed conversions are determinate and lossless.
None qe_end,
5

---

Under review as a conference paper at ICLR 2021
3.2.2 GRAPH SEARCH BASED SOLVER
We get the target variable and all kinds of conditions from the extractor, and then we want to call sequence of functions to calculate the final answer step by step. However, do not have such information, because the annotators only label target variables and conditions but not the solving processes. According to our observation, specific physical quantities only can be calculated by functions with specific chemistry knowledge. For example in Figure 1, the equilibrium constant k of the reaction can be calculated by k [Na] where [] is the molarity of a substance. Therefore, we can implement a function, noted as CalculateK, which takes the molarity of the three substances and the balanced chemical equation the input and calculates k. However, there is molarity but only mole of substances extracted, and the chemical equation is not balanced. Therefore, We need to implement function, CalculateMolarity, obtain molarity of these substances and function, Balance AllanneeCemiccaEEquatoo to balance the chemical equation before calling CalculateK.
[N
We model this process as a graph search in hyper graph follows: We view triplets nodes in this graph, e.g., <reaction, chemical equation, N2(g) + O2(g) + Ng)(g) >. 2) We view pre-build functions as directed hyper edges, e.g., CalculateK, CalaulateMolarity and BalanceChemicalEqqee tion. A hyper edge directs from the input triplets to the output triplets, e.g., the hyper edge, Calcu- lateK, starts from < N2..molartty, ? < Oz.moolrity.?>. < N20..ollaityy?>>> www.acttinnceeeneee equation, (g) + O2(g) + N2)(g) and directs to <reaction, k, ?>> 3) The solver maintains set of triplets without unknown quantities, and uses conditions obtained from the extractor to initialize the set. 4) The solver starts searching from the target variable. If the inputs can be satisfied by the current triplets, then the solver executes the function and adds the result to the set. Otherwise, the solver performs deep searching to check whether the unsatisfied input can be calculated by any function. Table shows the algorithm, where S is the set of triplets without unknown quantities, I and Of are inputs and outputs of function f, Iu. is subset of Is and Vi E I & S, and F, is the set of functions with as their output triplets' predicate.
Table 5: Graph based search solving algorithm
Input: The target variable q. Condition set C. Function set Output: Answer A
1:
FUNC GraphSearch( TTagget q= 8, p, =? >,>, Triplet set S);
2: Fe = [íis F and the predicate of Oy p)
3: f Fy:
4:
Iu = {ili Iy and i& 5}
5:
for i in Iu:
6:
(=GraphSearchú.S)
7:
if == St, Pr, t, satisfied:
8:
S= SU 8s, P.
9:
irvi E I is satisfied:
10: execute t=())
11: return < <,p,t>
return None
13:Call A=GraphSearch(q.C)
In Appendix D show several functions with inputs and outputs examples. A triplet with specific predicate can be calculated by different more than one functions with different inputs, e.g., RalculateMoleFromMas and CalculateMoleFromlle FromNumber take mass and the number of atoms(or molecules) as input, respectively. We do not need to implement all functions for all predicate combinations, since there may a combination of functions can represent the relationship among predicates. For example, we can call GalculateMooeFrom..l to calculate substance's molarity given its mass. We implemented 78 functions which covers 61 predicates in training set, and the coverage is 35.5%. We list all functions we implemented in Appendix ald will publish them once the paper published.
and CalculateMolarity sequentially
EXPERIMENTS
4.1 SETTING
For the end-to-onn solver, the pretrained uncased BERT base model to initialize its encoder, and use the first 6 layers of BERT to initialize its decoder. We also reuse the tokenizer from BERT for this model, and both encoder and decoder share the sub-token embedding matrix from BERT base.

---

Under review as a conference paper at ICLR 2021
We tuned the hyper parameters follows: learning rate E (1x10=1,5x10 x 1xx10 10 1,000,,,. 2 10 10-5; epoch E {100, 200, 300, 500) and fix batch size as
For the FSA based extractor, replace BERT based decoder's ' vocabulary to 12 customized FSA based vocabularies. We initialize parameters of the extractor by pretrained uncased BERT base model, except the embeddings of tokens from decoder's vocabularies. We randomly initialize the embeddings of tokens from decoder's vocabularies. We also tuned hyper parameters in extractor exactly same we did for the end-to- end solver. For the graph search based solver, we restrain the maximum depth as 5 when perform searching. We train both the end-to-end solver and the extractor on a single P40 GPU.
the same model structure as the end-to-end solver. However,
4.2 EVALUATION AND RESULT
To evaluate the methods, we design different criterion for different types of questions, i.e., physical unit, chemical formula, chemical equation questions. For physical unit questions, if |A-Á] < 0.05, we treat the answer is correct. For both chemical formula and chemical equation, we remove spaces in both A and Á and perform string matching between them. is the ground true answer and Å is the predicted answer.
A
For graph search based solver, we also evaluate the accuracy of the extractor beside the final answer. We evaluate token-level and question-level accuracy for the output sequence from the decoder, respectively calculated by A, N, D. Σ, NN (f, and Ag = > (+.==)))11 where and Nt.
N₂
i are respectively ground true and predicted tokens, & and are ground true and predicted output sequences, N₂ is the number of questions, and Nz is the number of tokens in (s === 8) is true if and only if == i) is true all position in
Table 6: Performances of methods on ChemsitryQA Development Set
Method
Token-level Accuracy Seq-level Accuracy
Answer Accuracy
And to End Solver Extractor GraphSolver
0.164
0.713
0.303
0.169
Table 6 shows the performances of these two methods. We can obtain the following observations:
End-to-end solver achieves 0.164 answer accuracy, which surprises but also implies that the powerful neural network can learn some pattern of questions, including calculating physical quantities, inferring chemical formulas and equations.
b) The FSA based extractor plus graph search based solver achieves 166 answer accuracy even only with 35.5% functions implemented, which implies this framework is effective, and the larger coverage of functions implemented will likely increase the accuracy.
For the extractor, the token-level accuracy is 0.713, but the sequence level accuracy drops to only 0.303. This implies the issue of cascading error is serious, but to improve sequence-level accuracy is very difficult. Thus, the more robust subsequent solver probably needed.
4.3 ANALYSIS
We want analyze reasons from wrong cases and get ratios of them. First, we know there about 69.7% wrong cases come from the wrong extraction results, and then we sample 100 wrong cases which have the correct extraction results from the development set for analyzing reasons. Table shows the ratios of reasons.
From the analysis, we can observe that most of wrong cases are caused by lacking some chemical knowledge in other forms which cannot be handled by the current solver.
5 RELATED WORK
We introduce related work of this paper from two perspectives: 1) There are kinds of examinations employed as benchmark to evaluate machine intelligence. 2) There are some NLP/CV tasks or datasets solved by both end-to-end neural networks and parsing plus execution method.
7

---

Under review as a conference paper at ICLR 2021
Table 7: Reasons of wrong cases and some examples
Reason
Ratio 46%
Examples and Explanation
Infer Chemical Equation or Formula
For question "What mass of oxygen needed for the complete combustion of 8.90 x 10 of methane?", the current solver cannot write the correct chemical equation for the combustion reaction. For questions "How much energy is required to raise the temperature of 3 g of silver from 15 "C to 20 CC?", the current solver cannot tell and maintain more than one different states in the question.
Multiple States
23%
Lack of Knowledge
6%
For question "How many O2 molecules will be con- sumed when one gram of propane combust completely in the air?", it lacks knowledge about what propane's s chemical formula is.
Wrong Label Other Reason
7%
18%
From the perspective of examinations, so far, there have been several tasks proposed based on some question types in specific subjects: 1) Math word problem (Huang et al. [016; Wang et al.] [017]] Amiii et al. 2019) can viewed as semantic parsing task. Most them require models translating question texts to equations or sequences of mathematical operations and then performing simple calculations to get the final answer. However, these datasets do not involve domain knowledge and usually provide strong supervision for the parser, i.e., equations or sequences of operations. 2)
Newtonian physics datasets (Sachan & Xing 2018 Sachan et al.) 2018) involves physical knowledge, but its knowledge is narrow and limited to Newtonian physics. This dataset is not public. 3) Elementary and middle school science examination (Clark) 2015 Clark et al. 2018) contains multiple choice questions involving open knowledge crossing various subjects. However, many questions in the dataset can be solved by retrieval and text matching. Although ARC(Clark et al. (2018) separates a subset of hard questions, the state-of-the-art on ARC is retrieval plus a powerful transformer based model. 3) Biological processes problem dataset (Berant et al.) (2014) provides passages to describe biological processes and asks questions about the passages. Different from ours, this dataset focuses more on evaluating machine reading comprehension, as English reading comprehension data LLai
ct al. 2017) does.
From the perspective of solving methods, besides examinations, there are several datasets can be solved by both end-to-end models and parsing plus execution method. For example, WEBQUES- TIONS et al. (203)) is famous KBQA dataset, and there are both end-to-end models (Bordes)
ct al. 2014 Hao et al. 2017) and semantic parser (Yih et al. 2014 Berant & Liang 2014) working on it. For WEBQUESTIONS, the solving process (i.c., executing SPARQL on Freebase) is fix after obtaining the parsing result, and the correct parsing result must lead to the correct answer. However, for ChemistryQA, there is more than one paths from extraction result the correct answer, and it requires searching on the graph. Another example is Knowledge Base Completion (KBC) task. KBC task can be solved by both end-to-end knowledge graph embedding models (Bordes et al. 2013) and logical inference based method, e.g., Markov Logic Network (Wei et al. 2015). However, the input of KBC not natural language.
6 CONCLUSION
Real world question answering is more complex than existing QA tasks, since requires not only understanding questions well but also interlcaving complex reasoning with knowledge retrieval, which scarcely represented by existing QA datasets. To foster the research on this area, we create ChemstriyQA dataset which contains chemical calculation problems. We implemented two baselines, a sequence to sequence model and a FSA based extractor plus a graph search based solver, which stands for two types of methods, the end to end neural network and the extractor plus solver, respectively. The experiment result shows the extractor- extractor-plus--eee plus-solver baseline can achieve a better performance with only 35.5% functions in domain implemented. Therefore, there is still room for improving in the extractor wwweettor-puus-see wwt---plusssseeeeeee solver method, but it is hard to improve performance for the end to end models.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzil-Ke Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 2357-22272 2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/N11-1245
(ttpstts::
www. ac wwbb. www.oooggggooooooooooooooooooooo anthologyy
N1 1999-11 1245
Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp-
1425, 2014.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, 2013.
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding. Brad Huang, Peter Clark, and Christopher D Manning. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1499-15100 2014.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pp. 2787-2795, 2013.
Antoine Bordes, Sumit Chopra, and Jason Weston. Question answering with subgraph embeddings. arXiv preprint 4rXiv:1406.3676, 2014.
Peter Clark. Elementary school science and math tests as a driver for ai: take the aristo challenge! In AAAL pp. 4019-4021. Citeseer, 2015.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Peter Clark, Oren Etzioni, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, al. From't t''o a on the ny regents science exams: An overview of the aristo project. arXiv preprint arXiv:1909.01958, 2019.
David A Ferrucci. Introduction to "this is watson". IBM Journal of Research and Development, 56 (3.4):1-1, 2012.
Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 221-231 2017.
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei- Ying Ma. How well do computers solve math word problems? large- dataset construction and evaluation. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 887-896, 2016.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv: 046804 2017.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.

---

Under review as a conference paper at ICLR 2021
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 23922, Austin, Texas, November 2016. Association for Computational I. Linguistics. doi: 10.18633/11/D1-- 1264, URL
www ac web. org/
motholooyy/Dl6-225.5
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melhourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. N0.1866/11//1822122 URL https: / /www. ac lweb
oxg/aanthology/P1e 2124
Mrinmaya Sachan and Eric P Xing. Parsing to programs: A framework for situated qa. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 2140-2149, 2018.
Mrinmaya Sachan, Kumar Avinava Dubey, Tom M Mitchell, Dan Roth, and Eric P Xing. Learning pipelines with limited data and domain knowledge: A study in parsing physics problems. In Advances in Neural Information Processing Systems, pp. 140-151, 2018.
Amrita Saha, Vardaan Pahuja, Mitesh M Khapra, Karthik Sankaranarayanan, and Sarath Chandar. Complex sequential question answering: Towards learning to converse over linked question answer pairs with knowledge graph. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Autention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845-854, 2017.
Zhuoyu Wei, Jun Zhao, Kang Liu, Zhenyu Qi, Zhengya Sun, and Guanhua Tian. Large-scale knowledge base completion: Inferring via grounding network sampling over selected instances. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 1331-1340, 2015.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint vrXiv:1906.082.77 2019.
Wen-tau Yih, Xiaodong He, and Christopher Meek. Semantic parsing for single-relation question answering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 643-648, 2014.
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming- Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 201-206, 2016
10

---

Under review as a conference paper at ICLR 2021
A ANNOTATION EXAMPLES, INTERFACES AND PROCESSES
A.1 AN EXAMPLE OF ANNOTATION PAGE
Figure 2 shows page of annotation, the left part is the original web page in socratic. and the right part is the annotation area.
Chanistry
Top
Soard
parcua
lemperelure 10 2,00 moi NZC, How Caliclltto at this temperature for the following reaction. N2(g) -2/g) 2 N2O0ig?
C.reditam Comtee
renaeeee & shea clough Sraarii
dodda 136g
2.20%
Miraac Ochar casqunion chence remas accann was Chacratoo vam Dulvirsors
1810 30
(A)
(B)
Figure 2: An example of the annotation page
A.2 ANNOTATION TYPES
The annotation area contains two parts: one for labeling the target variable and the other one for labeling conditions. For a chemical calculation problem, there is only one target variable but probably more than one conditions. Thus, annotators are free to add condition block. Usually, the interface for question variables and conditions are different even for the same type of annotations. For variable or condition block will be first ask to choose a type of annotation, and then the page will show the interface of it. Table 8 shows interfaces for all types of annotations.
Table 8: Annotation Interfaces As a Question Variable
Annotation
As a Condition
Type
physical unit
[BLANK (predicate)) OF [BLANK (sub-
[BLANK (predicate)) OF (BLANK (sub-
ject)] IN (BLANK (unit or None)]
ject)] (BLANK (object or value)]
chemical
Chemical Formula OF (BLANK (sub-
Chemical Formula OF (BLANK (sub-
formula
ject)] IN NONE
ject)] IS [BLANK (value))
chemical equa-
Chemical Equation OF reaction
Chemical Equation OF reaction IS
tion
[BLANK (value))
substance
Substance IS [BLANK (value))
other
Other Condition IS (BLANK (value))
11

---

Under review as a conference paper at ICLR 2021
A.3 CROWD SOURCING ANNOTATION DETAILS
First, we estimated the annotation time by recording time spent on annotators label a small scale experimental set, and got the average time spent labeling and verifying 4.65 minutes and 3.25 minutes, respectively. In earlier stage, we performed both labeling and verifying on each question. After we got the verified rate is high enough, we only keep the labeling process, which leads to the annotation time of a question is 4.4 minutes. Finally, we obtain 4418 annotated questions with spending about 336 hours.
B PREDICATES, UNITS AND OTHERS IN CHEMISTRYQA
Table 9 shows top predicates and units.
Table Statistics on top predicates and units
Predicate
Count
Unit
Count
mass
2,013
gram
1,911
volume
1,851
mоl/
1,098
temperature
1,099
mol
1,033
mole
1,056
liter
899
molarity
731
OC
872
pressure
709
mliter
832
concentration
323
atm
398
ph
226
kelvin
222
number
226
mmng
168
heat energy
146
kpa
164
We list all predicates in Table 10, list all units in Table and list all other conditions in Table 12.
12

---

Under review as a conference paper at ICLR 2021
Table 10: All predicates in ChemsitryQA
[ch3cooh]
[h+]
[h2so4]
[h3o+]
[nh3]
[nh4+]
[oh-]
absorbance actual yield
absorbance reading
absorptivity
acid concentration
activation barrier
altitude
angle of the
flected beam
area
atomic mass
average oxidation
avogadro constant
barometric pressure
number
boiling point tem-
cell potential
changed factor
charge
coefficient
perature
completion percent
composition
by
concentration
constant
coordination num- ber
cp
degree dissocia- delta vaph tion
deltag
deltag"
deltagf
deltah
deltah(rxn)
deltah"
deltah" f diameter electron configura- tion
"eltahl"
deltas
densily
depth
dilution factor
distance
ecell
electric current
energy
energy transfer rate
enthalpy
enthalpy of combus- tion
enthalpy of forma- tion
enthalpy fusion
enthalpy sublima- tion
enthalpy of vapor- ization
entropy
equilibrium concen- tration formula weight
cquilibrium con-
equilibrium
equivalent weight
first ionisation en- engy
stant
stant ktant
freezing point tem- perature
gauge pressure
half-life
hardness
heat capacity
heat capacity ratio
heat energy
height
initial temperature
ionic product
ionization energy
kbp
kh
kinetic energy
kp
kw
longest wavelength
mass/volume percent
concentration
percent
maximum amount
maximum possible yield
melted time
melting point tem-
molal solubility
molality
perature
molar enthalpy of dissolution
molar enthalpy of vaporization
molar heat
molar mass
molar ratio
molar solubility
molar volume
molarity
molarity percent normality
mole
mole fraction
molecular weight
multiplier
number
o.n
osmolarity
osmotic pressure
oxidation number percent tion
oxidation state
partial pressure
path length
percent
composi-
percent ionization
percent yield
ph
pka
pkb
poh
power
pre-exponential fac-
pressure
quantity
radius
rate
ratio
reaction enthalpy
reaction quotient q
reaction rate con- stant
relative molecular mms speed
size
solubility
solubility product
weight
specific heat
specific heat capac- ity
standard atmo- spheric pressure
standard cmf
standard enthalpy
standard enthalpy
standard enthalpy formation
standard free energy
state
steric number
change
strength
surface tension
temperature
theoretical yield times
thermal energy total pressure
thermochemical
threshold odor num- ber
time
equation
trasmittance
value
van t hoff factor volume
vapor pressure volume percent
vapour density w/w
velocity
voltage
wavelength
13

---

Under review as a conference paper at ICLR 2021
Table 11: All units in ChemsitryQA
(M min) 1)
amperes
amu
atm
bar
cal
call(g
cal/mol
cm/s
cm
dm 3)
dm
cq/l
fluid ounces
g
g/100 ml
g/em"
g/dm"
g/eg
g/1
g/ml
g/mol
gallon
hour
in
in
il[g k)
i/(g Cc)
i/lg "
j/Okg k)
j/lk
j/moll k)
j/mmol
j(c g)
j/g
j/mol
kcal
kcal/g
kcal/mol
kg
kj
kj/lkg Pc)
kj/min
kj/mol
kpa
l/mmll S)
m/s
m3
meg/1
mg
mg/1
mg/ml
milliequivaennts
min
mj
ml
mm
mmbg mmol mol
mol/(1
mol/dm mol/g nm none
mol/kg mol/1
mpa
n
ng
osmol
ppm psi pa
osmol/1 OZ
particles/s
pounds
1)
years' 1)
"c/(kg mol)
Table 12: All other conditions in ChemsitryQA
Avogadro constant Equivalence
Combusted Freezing
Conjugate base Henderson- Hasselbalch
vaporize
Equilibrium
Incomplete combus-
NTP
tion
Neutralization
OTHER burn
Photosynthesis
SATP
STP
Stoichiometrically
combustion
complete combus- tion
constant
lempera-
ture
decomposes ionizing
melt
neutralize
reaction
14

---

Under review as a conference paper at ICLR 2021
C FSA VOCABULARIES AND STATE TRANSITIONS
Table 13 shows the vocabularies used in Finite State Automata of Extractor and some tokens in them.
Table 13: The vocabularies' names and tokens belonging to them.
Vocabulary
Size
Example Tokens
START
["start"]
END
["end"]
QCLEND
["qc_end"]
POINTER
512
"""""""""""""""""""""""""""""""" ",,"""""""""""""""""""""""""""""
PHYSICAL UNIT
("physical unit"]
CHEMICAL EQUATION
('chemical equation"]
CHEMICAL
I"chemical.formlaa
SUBSTANCE
["substance"]
C.OTHER
["c_other"]
PREDICATE
172
["atomic_mas'' "booiii
"energy"" "ooncentration"""heuu "appaccttppattt"""" "molarity", ...]
UNIT OTHER
91
["atm" ""all"""""a""""""" =_))(" "j/(g...)"" "mo""".""""""""""""""""""" "psi" ....]
"cal/(g.-
26
[" ""Avogadro_coms consttntt" "onstan",""oomuusted"" "oaaal"""""Cuuaa"a Coojuuate_hasse"
We define 26 states for FSA. Table 14 shows all states and Figure 3 shows the state transitions.
Table 14: States of FSA
s_start
S.q-pu
suq.pu.subject.star
B.q.pu_subject.end s.c. pu subject end s.q.ef
s.q.pu.predicate
S.q pu unit
S.C.pu pu
S. c.c.p subject start
s pu value start
s.c.pu.value_end
s.c.pu.property
s.q.ce
s.q_end
s.c.ce
S.c.cf
S.c.sub
s.c_other
s.c.ce_start
s ce end
S.C. cf start
s.c. cf end
sub start
S.C. sub end
s_c_other_type
CpoLsuNectt
C.0.supett end
0. prediaa
C. viae start
uni
C valu and
a.d
C00,0d
dd
6.0d
and
Cubillance
mnd
Figure 3: The state transition graph for the FSA
D FUNCTIONS
Table 15 shows several functions' inputs and outputs for example.
15

---

Under review as a conference paper at ICLR 2021
Table 15: Function examples implemented for ChemistryQA
Function Name
Input Predicates
Output Predicates
Calculatek
balanced chemical equation, mo- larity (for all substance in equa- tion)
equilibrium constant k
CalculateMolarity CaleulateMole romMass
mole, volume
molarity
mass, molar
mole
CalulcateMole FromNumber
number of atoms or molecules mole
Balance Caaenneehhemicee Chemical Equatiion
chemical equation
chemical equation
Parse Parse Chemicallquaation
span
chemical equation
We also list all functions we implemented as follows:
FFunc_Name2CE
FormulaCE
Func Equation2CE
Func BalanceChemicallqquttoon
Monee.MoleAtom
Muna.cEE2MolrMass
Moss
Func_Ph2Kw
Mene.e
Nomeeeeoee
Hine_MaasWolar.mas22Mb
Tunn.Molleoolarmmas2Maass
Func_ VolumeeolaarttyMMoe
Func Yollme/NoeeTemmprtuuuuePPsssrr
Func. YollmmeTTomprruuuuPPesseueee
Func. Prr.peessuueeoooiomsmss//eeeeeen
Mime.NModelass22Mlar.nassss
Func Molelatume2Molarity
Func ModéVVllume2Conceettaaioo
Func woroMMoorrtyoolleeeMMoooo
Lime.MooleMolariyyyYolume
FuncePPhAAcid..onceentttiio
Tuue--Acid.ccmmeettrratoo122
PhncoP
Func. Func_Poh222 Poh2Ph
Func Ka2Pka
16

---

Under review as a conference paper at ICLR 2021
Func.Maas VolumeeDttt
Sunn..MMsDDenssty2Volume
Func. Molar
Func MolalityMolar mass2Molarity
Func. Mollaattyylemmeeeuuuee/00ooorrry
Func.KaaDDegrre.oo..ddiiccciiion
Func:2Frrezznngppiinneeeeemarrr
Tiuee-MessVelooctykiieeiiceeee
Func_ KaMoolarty22erccanninnzzzzoooo
Func 2Standard atmospheric pressure
Func MolalityMollar mass2Ww
WwMolar.
Tune.MassMaas2MMsssprrnnnt
Fine:MMoeevdeeeMooopprreent
Tlue.PPl22Alaalii.ccccnnnaaatoo
Func Atom.oliculleMode2Mllle
Func MassSpecific heat capacityTlemperatueeHeett energy
Func Heat energySpecific heat htt.taaaaaaaaaeeee SapacityTemperaauure2..
Func weweeHeeeeeeegyemmeeeeee hwt.:eowwwg..nnneeuuueeeeeeseeep
www..MaassvuuumeeeMaaasccccccccn
VhmnoMMassoolumeeDeeniin
Func. Bptph///pppeessureeeesseeeeppppp
Func Molarity3 2Ka
Func Molarity3 2Kb
Fune ModartiyppuubbooopiiuyyAhhhbbbe
Func Absoorbance2Trassmittaeccc
Func. fomulloolm
17

---

Under review as a conference paper at ICLR 2021
Tunc.MMss22.imme2HHllliie
Func Heat of fusionMMoiePooee2eeeeeedd time
Func ShcMolar mass2Molar heat
Func MolecuuullllMMl
18


Reviews:
[{"review_id": "lIkX2AkkEay", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:  A new QA dataset for chemistry QA consisting of 4500 questions and covering 200 topics. Crowdsourced annotation of variable and conditions. \n\nWeaknesses: -  Strong baseline results are missing. Intermediate steps is missing in the annotations which are really helpful in training an end-to-end model.  The topic distribution is missing from the paper. Experimental results and analysis is not enough. \n\nOverall: The idea of curating and annotating a new dataset for Chemistry QA dataset is good. I feel a stronger baseline would have helped much in understanding and analysing the quality dataset and annotations. Also the question complexity analysis/topic distribution is missing. Overall the paper writing could be improved a lot, in the current version it is difficult to follow.\n\nQuestion: If the whole problem can be converted into a set of triples and conditions they why not use graph based QA techniques?  It will be interesting to see how neural symbolic machines/Neural module network perform on this dataset ? Topic distribution, question type distribution etc are missing.  Any specific reasons for using 12 layers of transformer encoders and 6 layers of transformer decoders in Extractor plus Graph Search Based Solver?  Which graph search algorithm is used in section 3.2.2 and Table 5?\n\n\nTypos:\n It this website,  -> On this website\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iBkRwn-zaS8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*: This paper proposes a new dataset based on textbook / classroom chemistry questions for complex knowledge retrieval and aggregation. The authors scrape several thousands questions from online repositories and add additional natural language annotations signifying the quantities to be solved for in each question, as well as the declarative knowledge. Two baselines, one end-to-end neural and another symbolic, both fail at this dataset.\n\n*Strengths*: The dataset targets the important question of how to build models that can retrieve knowledge while performing complex reasoning.\n\n*Weaknesses*: As-is, the dataset fails to target the knowledge retrieval component---models are either expected to magically know how to calculate the answer, or use hard-coded functions that complete a graph of values. The neural baseline also seems a bit non-standard, raising questions of how well modern systems can actually do on the task; furthermore, the end-to-end neural system is disadvantaged in that it likely has not seen much chemistry-related content during fine-tuning, whereas the symbolic baseline has access to a host of human-defined functions. Furthermore, dataset quality is a bit difficult to assess without more samples.\n\n*Recommendation*: 3 . This benchmark is motivated by the lofty goals of encouraging the development of models that can combine knowledge retrieval, complex reasoning, and language understanding. However, it’s unclear to this reviewer whether it will prove useful in making progress towards such goals---they’re too conflated to be meaningfully evaluated within this context. To improve the benchmark and make it more amenable toward advancing those research goals (versus just being a difficult datasets that current models cannot handle), I’d recommend explicitly targeting and evaluating this knowledge retrieval component as well. For instance, given a specific knowledge-base that’s guaranteed to span the facts necessary to answer the questions, how well can a model (1) retrieve relevant information and (2) use such relevant information to answer questions?\n\nQuestions:\n\n“Chemical Calculation Problems cannot be solved by end-to-end neural networks since complex symbolic calculations are required”: this is a hypothesis---there are many tasks where “complex symbolic calculations are required”, but end-to-end networks excel.\n\nWhat extent of knowledge is required to solve this task? For instance, many old semantic parsing datasets came with databases, and it was guaranteed that within the database, an answer would occur. What would a corresponding knowledge graph for this case look like, and how complex would it be?\n\n“Unlike similar tasks’ annotation, we cannot collect all the atomic operations needed before starting annotation, since the set of chemical operators is not closed.” The set of mathematical operators is also not closed (e.g., in math word problems). Why is this approach better than collecting all the operations represented in the dataset (even if it doesn’t cover all of the operations that one could conceivably see)?\n\nThe annotation interface / process looks quite regular---you aren’t expecting too much variation from the templates given. Given that you can help crowdworkers with these templates, why not just use these templates as the baseline for a formal meaning representation that would encompass the knowledge needed for the task?\n\nCan you give more details about the annotation process, beyond the short paragraph near the end of section 2.2? (“We employed crowdsourcing for this annotation work...around 336 hours)”. I’d be surprised if any crowdworker could label this sort of data well. What quality control filters did you put in place? Can we see more (random) samples of the dataset, so we can better assess its quality?\n\nEnd-To-End Solver: where did you get this model architecture from, such that “This method represents a class of powerful neural networks, which achieved state-of-the-art performance on many QA tasks.”? I’ve never seen BERT used in a seq2seq setting like this (instead, people tend to use models trained on input/output pairs, like BART or T5). I’d like to see how this compares to using BART or T5, since it’s not clear that the BERT initialization would be good for generation.\n\nGraph-Search based Solver: the need to implement specific functions (78, in this case) is significant, and undermines the point of this dataset, in my opinion. There’s no inherent value in learning to solve chemical equations well---the hope is that, in the process of doing so, we’ll get modeling insights into what methods work well and can be generally applied to other knowledge-intensive tasks. This graph-search based solver seems narrowly scoped to ChemistryQA and difficult to adapt to other tasks, and it’s not entirely clear why we should value its results.\n\nToken-level accuracy: Is it guaranteed that the output of the graph-search based solver will be the same length as the gold output? How? Else, how is token-level accuracy computed?", "extra_scores": {}, "extra_text": {}}, {"review_id": "GEWPfnRR8Zs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new dataset for the chemistry domain as a real-world QA task.\nThe authors collected chemistry questions from a web page and used crowdsourcing to annotate the questions with labels and conditions required for solving them.\nAs baseline models, the authors propose an end-to-end neural network model and a symbolic solver that uses pre-defined rules. They demonstrate that the neural model struggles to solve the task but the symbolic solver outperforms it with the pre-defined rules that cover only a part of predicates in the dataset, arguing that their dataset is challenging and can foster the research of real-world problems in the chemical domain.\n\nPros:\n\n- The paper proposes a new dataset, ChemistryQA, that consists of 4,500 questions with labels of target variables and conditions provided in the questions.\n- The paper proposes a graph-search based solver with an extraction mechanism which is carefully implemented for this task.\n- Appendix has thorough lists of the predicates and units in the dataset and functions used in the baseline solver.\n\nCons:\n\n- Overall, I think that the writing of the paper can be improved more. There are some typos and formatting issues, which reduce the paper strength. Besides in Sections 2.1, 2.2, and 3.2.1, the paragraphs refer to figures and tables in Appendix. This seems to violate the submission instructions.\n- My primary concern is on the quality of collected questions. In Section 2.2, the authors say that they performed some check-and-verify mechanism in the first batch, which should be described in detail. Some related questions:\n  + Did the author compute the inter-annotator agreement on sampled questions?\n  + What kind of rules did you define for the verification?\n  + How many workers did work on the annotation in total?\n  + Did the same pool of workers work on the annotation for the first batch and the subsequent batches?\n  + Is there actual human performance on the collected questions? Seemingly, it is not guaranteed that posted questions on the web page are reasonably answerable.\n- The purpose of employing two baseline models is not explained well. In the introduction, the authors say that \"to verify the dataset is consistent with the purpose of evaluating AI' comprehensive capability\". However, their hypotheses for the experiments are not clearly stated.\n- It seems that the authors stopped implementing the predefined functions for the symbolic solver at the point where the solver outperforms the neural network model. The authors could have implemented more, but it is not clearly explained why they only implemented the predefined functions for the 35.5% predicate coverage. What would happen if there are a larger number of functions implemented?\n- Comparison with existing datasets can be elaborated more. For example, because the Option answer type should include different types of entities and the option is just a form, directly comparing it with value, formula, and equation does not make sense to me.\n- In the error analysis, 18% of cases are classified into Other Reason, which does not look like a small number to me. Can the authors break this down into more detailed categories?\n\nTypos:\n\n- Section 2.1, It this website -> In this website\n- Cite Devlin et al. (2019) for using BERT.\n- There is an ill-formatted cell in Table 4.\n- There is inconsistent use of decimal commas (4,500 vs 4418)\n- Add whitespace before citations (e.g., in Section 2.4).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DgHMh7jcBdw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n* This paper presents a question answering dataset called ChemistryQA.  It is different from existing datasets in that ChemistryQA requires open knowledge and complex solving processes.  It provides triplet like extraction annotation which isolates language understanding and domain knowledge. Experimental results show that a neural encoder-decoder model and an extractor-plus-solver do not work well.\n\nStrengths:\n* The dataset contains real-world QA that requires the ability to perform complex chemistry calculation and reasoning. It is difficult for crowdsourcing workers to generate such complex questions.\n* The authors proposed a novel annotation method that target variables and conditions are labeled in a triple-like format.\n\nWeaknesses:\n* The dataset seems small to acquire the ability to perform complex calculation and reasoning. The training, validation, and testing datasets consist of 3,433, 485, and 500 questions, respectively.\n* The paper does not show statistics of the dataset such as the average length of questions and answers and the unique number of answers.\n* The paper does not show the performances broken down by question types. Although the end-to-end solver achieves an answer accuracy of 0.164, I think it is important to show more detail on what it can and cannot do.\n* The authors uses a pre-trained BERT as the encoder of the end-to-end solver and trained the decoder from scratch.  I think pre-trained encoder-decoder models such as T5 and BART are better as the baselines of the end-to-end solver than the model used in this paper.  \n\nReview Summary:\n* The paper is well motivated. ChemistryQA can be a useful dataset to evaluate the ability of chemistry calculation and reasoning, while the dataset seems small to acquire the ability.   I think it can benefit a lot with a more comprehensive analysis of evaluation results of baselines.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new dataset, ChemistryQA which has complex questions requiring scientific and mathematical reasoning. They show that existing SOTA models do not perform well on this dataset thereby establishing the complexity of the dataset. 

The reviewers raised several concerns as summarised below:

1) Writing is not very clear
2) The quality of the dataset is hard to judge as some crucial information about the dataset creation process is missing 
3) The size of the dataset is small
4) some stronger QA baselines need to be included

Unfortunately the authors did not provide a rebuttal. Hence, its current form this paper cannot be accepted. 
Decision: Reject