We test several state-of-the-art (BERT-based) models and find that they are still substantially (20%) below human performance. Our contributions are thus (1) the dataset, containing 3864 richly annotated questions plus a background corpus of 400 qualitative knowledge sentences; and (2) an analysis of the dataset, performance of BERT-based models, and a catalog of the challenges it poses, pointing the way towards solutions.

QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions	We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., "A sunscreen with a higher SPF protects the skin longer.", twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?", plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system's ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20%) below human performance, presenting an open challenge to the NLP community.

Related Work	Despite rapid progress in general question-answering (QA), e.g., , and formal models for qualitative reasoning (QR), e.g., , , there has been little work on reasoning with textual qualitative knowledge, and no dataset available in this area. Although many datasets include a few qualitative questions, e.g., , , the only one directly probing QR is QuaRel . However, although QuaRel contains 2700 qualitative questions, its underlying qualitative knowledge was specified formally, using a small, fixed ontology of 19 properties. As a result, systems trained on QuaRel are limited to only questions about those properties. Likewise, although the QR community has performed some work on extracting qualitative models from text, e.g., , , and interpreting questions about identifying qualitative processes, e.g., , there is no dataset available for the NLP community to study textual qualitative reasoning. QuaRTz addresses this need.

BERT (IR upper bound): Same, but using the ideal (annotated) $K_i$ rather than retrieved $K_i$. 5. BERT-PFT (no knowledge): BERT first fine-tuned (“pre-fine-tuned”) on the RACE dataset , , and then fine-tuned on QuaRTz (questions only, no $K$, both train and test). Questions are supplied as [CLS] question-stem [SEP] answer-option [SEP]. 6. BERT-PFT (IR): Same as BERT (IR), except starting with the pre-fine-tuned BERT. All models were implemented using AllenNLP .

Results	 The results are shown in Table TABREF10, and provide insights into both the data and the models: 1. The dataset is hard. Our best model, BERT-PFT (IR), scores only 73.7, over 20 points behind human performance (95.0), suggesting there are significant linguistic and semantic challenges to overcome (Section SECREF7). 2. A general science-trained QA system has not learned this style of reasoning. BERT-Sci only scores 54.6, just a little above random (50.0). 3. Pre-Fine-Tuning is important. Fine-tuning only on QuaRTz does signficantly worse (64.4) than pre-fine-tuning on RACE before fine-tuning on QuaRTz (73.7). Pre-fine-tuning appears to teach BERT something about multiple choice questions in general, helping it more effectively fine-tune on QuaRTz. 4. BERT already “knows” some qualitative knowledge. Interestingly, BERT-PFT (no knowledge) scores 68.8, significantly above random, suggesting that BERT already “knows” some kind of qualitative knowledge.

Models	The QuaRTz task is to answer the questions given the corpus $K$ of qualitative background knowledge. We also examine a “no knowledge” (questions only) task and a “perfect knowledge” task (each question paired with the qualitative sentence $K_i$ it was based on). We report results using two baselines and several strong models built with BERT-large  as follows: 1. Random: always 50% (2-way MC). 2. BERT-Sci: BERT fine-tuned on a large, general set of science questions . 3. BERT (IR): This model performs the full task. First, a sentence $K_i$ is retrieved from $K$ using $Q_i$ as a search query. This is then supplied to BERT as [CLS] $K_i$ [SEP] question-stem [SEP] answer-option [SEP] for each option. The [CLS] output token is projected to a single logit and fed through a softmax layer across answer options, using cross entropy loss, the highest being selected. This model is fine-tuned using QuaRTz (only). 4.

QuaRTz thus provides a dataset towards this end, allowing controlled experiments while still covering a substantial number of textual relations in an open setting. QuaRTz is available at http://data.allenai.org/quartz/.

Discussion and Analysis ::: Qualitative Reasoning	Can models learn qualitative reasoning from QuaRTz? While QuaRTz questions do not require chaining, 50% involve “flipping” a qualitative relationship (e.g., K: “more X $\rightarrow $ more Y”, Q: “Does less X $\rightarrow $ less Y?”). Training on just the original crowdworkers' questions, where they chose to flip the knowledge only 10% of the time, resulted in poor (less than random) performance on all the flipped questions. However, training on full QuaRTz, where no-flip and flip were balanced, resulted in similar score for both types of question, suggesting that such a reasoning capability can indeed be learned.

To rule out annotation artifacts, we we experimented with balancing the distributions of positive and negative influences, and different train/test splits to ensure no topical overlap between train and test, but the scores remained consistent. 5. BERT can apply general qualitative knowledge to QA, but only partially. The model for the full task, BERT-PFT (IR) outperforms the no knowledge version (73.7, vs. 68.8), but still over 20 points below human performance. Even given the ideal knowledge (IR upper bound), it is still substantially behind (at 79.8) human performance. This suggests more sophisticated ways of training and/or reasoning with the knowledge are needed.

Introduction	Understanding and applying qualitative knowledge is a fundamental facet of intelligence. For example, we may read that exercise improves health, and thus decide to spend more time at the gym; or that larger cars cause more pollution, and thus decide to buy a smaller car to be environmentally sensitive. These skills require understanding the underlying qualitative relationships, and being able to apply them in specific contexts. To promote research in this direction, we present the first open-domain dataset of qualitative relationship questions, called QuaRTz (“Qualitative Relationship Test set”). Unlike earlier work in qualitative reasoning, e.g., , the dataset is not restricted to a small, fixed set of relationships. Each question $Q_i$ (2-way multiple choice) is grounded in a particular situation, and is paired with a sentence $K_i$ expressing the general qualitative knowledge needed to answer it. $Q_i$ and $K_i$ are also annotated with the properties being compared (Figure FIGREF1). The property annotations serve as supervision for a potential semantic parsing based approach. The overall task is to answer the $Q_i$ given the corpus $K = \lbrace K_i\rbrace $.

Discussion and Analysis ::: Linguistic Phenomena	From a detailed analysis of 100 randomly sampled questions, the large majority (86%) involved the (overlapping) linguistic and semantic phenomena below, and illustrated in Tables TABREF3 and TABREF12: Differing comparative expressions ($\approx $68%) between $K_i$ and $Q_i$ occur in the majority of questions, e.g., “increased altitude” $\leftrightarrow $ “higher”  Indirection and Commonsense knowledge ($\approx $35%) is needed for about 1/3 of the questions to relate $K$ and $Q$, e.g., “higher temperatures” $\leftrightarrow $ “A/C unit broken” Multiple Worlds ($\approx $26%): 1/4 of the questions explicitly mention both situations being compared, e.g., $Q_1$ in Table TABREF3. Such questions are known to be difficult because models can easily confuse the two situations . Numerical property values ($\approx $11%) require numeric comparison to identify the qualitative relationship, e.g., that “60 years” is older than “30 years”.

Conclusion	Understanding and applying textual qualitative knowledge is an important skill for question-answering, but has received limited attention, in part due the lack of a broad-coverage dataset to study the task. QuaRTz aims to fill this gap by providing the first open-domain dataset of qualitative relationship questions, along with the requisite qualitative knowledge and a rich set of annotations. Specifically, QuaRTz removes the requirement, present in all previous qualitative reasoning work, that a fixed set of qualitative relationships be formally pre-specified. Instead, QuaRTz tests the ability of a system to find and apply an arbitrary relationship on the fly to answer a question, including when simple reasoning (arguments, polarities) is required. As the QuaRTz task involves using a general corpus $K$ of textual qualitative knowledge, a high-performing system would be close to a fully general system where $K$ was much larger (e.g., the Web or a filtered subset), encompassing many more qualitative relationships, and able to answer arbitrary questions of this kind. Scaling further would thus require more sophisticated retrieval over a larger corpus, and (sometimes) chaining across influences, when a direct connection was not described in the corpus.

Flipping means inverting comparatives (e.g., “more” $\rightarrow $ “less”), values, and other edits as needed to change the answer, e.g., K: More rain causes damper surfaces. Q:More rain causes (A) wetter land (B) drier land Q-flipped: Less rain causes (A) wetter land (B) drier land Flipped questions are created to counteract the tendency of workers to use the same comparison direction (e.g., “more”) in their question as in the seed sentence $K_i$, potentially answerable by simply matching $Q_i$ and $K_i$. Flipped questions are more challenging as they demand more qualitative reasoning (Section SECREF11). Questions marked by workers as poor quality were reviewed by the authors and rejected/modified as appropriate. The dataset was then split into train/dev/test partitions such that questions from the same seed $K_i$ were all in the same partition. Statistics are in Table TABREF9. To determine if the questions are correct and answerable given the general knowledge, a human baseline was computed.

Dataset Collection	QuaRTz was constructed as follows. First, 400 sentences expressing general qualitative relations were manually extracted by the authors from a large corpus using keyword search (“increase”, “faster”, etc.). Examples ($K_i$) are in Table TABREF3. Second, crowdworkers were shown a seed sentence $K_i$, and asked to annotate the two properties being compared using the template below, illustrated using $K_2$ from Table TABREF3: [vskip=0mm,leftmargin=3mm] "The smaller its mass is, the greater its acceleration for a given amount of force."   They were then asked to author a situated, 2-way multiple-choice (MC) question that tested this relationship, guided by multiple illustrations. Examples of their questions ($Q_i$) are in Table TABREF3. Third, a second set of workers was shown an authored question, asked to validate its answer and quality, and asked to annotate how the properties of $K_i$ identified earlier were expressed. To do this, they filled a second template, illustrated for $Q_2$:   Finally these workers were asked to generate a new question by “flipping” the original so the answer switched.

Conclusion ::: Acknowledgements	We are grateful to the AllenNLP and Beaker teams at AI2, and for the insightful discussions with other Aristo team members. Computations on beaker.org were supported in part by credits from Google Cloud.

Sue is taller than Joe so Sue is (A) stronger (B) weaker Sue is shorter than Joe so Sue is (A) stronger (B) weaker Sue is shorter than Joe so Joe is (A) stronger (B) weaker $Q_{n}^{\prime }$ requires reasoning about intensifers that are flipped with respect to $K$ (shorter $\rightarrow $ weaker), and $Q_{n}^{\prime \prime }$ requires entities be tracked correctly (asking about Sue or Joe changes the answer).

Three annotators independently answered a random sample of 100 questions given the supporting sentence $K_i$ for each. The mean score was 95.0%.

Discrete property values ($\approx $7%), often require commonsense to compare, e.g., that a “melon” is larger than an “orange”. Stories ($\approx $15%): 15% of the questions are 3 or more sentences long, making comprehension more challenging. This analysis illustrates the richness of linguistic and semantic phenomena in QuaRTz.

Discussion and Analysis ::: Use of the Annotations	QuaRTz includes a rich set of annotations on all the knowledge sentences and questions, marking the properties being compared, and the linguistic and semantic comparatives employed (Figure FIGREF1). This provides a laboratory for exploring semantic parsing approaches, e.g., , , where the underlying qualitative comparisons are extracted and can be reasoned about.

The Task	Examples of QuaRTz questions $Q_{i}$ are shown in Table TABREF3, along with a sentence $K_{i}$ expressing the relevant qualitative relationship. The QuaRTz task is to answer the questions given a small (400 sentence) corpus $K$ of general qualitative relationship sentences. Questions are crowdsourced, and the sentences $K_{i}$ were collected from a larger corpus, described shortly. Note that the task involves substantially more than matching intensifiers (more/greater/...) between $Q_{i}$ and $K_{i}$. Answers also require some qualitative reasoning, e.g., if the intensifiers are inverted in the question, and entity tracking, to keep track of which entity an intensifier applies to. For example, consider the qualitative sentence and three questions (correct answers bolded): People with greater height are stronger.

Question: What are state-of-the-art models on this dataset?
Answer: BERT (IR), BERT (IR upper bound), BERT-PFT (no knowledge), BERT-PFT (IR)