Scenario-based Question Answering with Interacting Contextual Properties
Authors: Haitian Sun
William W. Cohen
Ruslan Salakhutdinov
Venue: ICLR 2023 poster (2023)
Keywords: Question Answering

Abstract: In the scenario-based Question Answering (QA) task, models are asked to find answers that are appropriate to the user scenarios associated with the question and identify information that is missing from the scenarios but is necessary for the answers to hold. Scenarios commonly include multiple properties of users, such as age, employment status, and income level for the question “How much can I claim from this benefit”. The properties relevant to a potential answer are given in a document, which will state conditions necessary for the answer to hold. Documents also may specify how conditions interact with each other, e.g. with text like “one of the conditions below must apply”. Although understanding the relationship between conditions is crucial for solving this challenging QA task, limited work has been done so far in modeling this. In this paper, we propose the T-Reasoner model, which solves this problem with three jointly learned modules: an entailment module which checks whether a condition has been satisfied by the scenario, a decoding module which locates eligible answers from documents, and a reasoning module which infers the relationship between conditions and performs a reasoning step to determine the logically consistent answers and identify missing conditions. T-Reasoner outperforms strong baselines on a synthetic scenario-based QA dataset and achieves a new state-of-the-art on two scenario-based QA benchmarks, outperforming the prior best models by 3-10 points.

Published as a conference paper at ICLR 2023
SCENARIO-BASED QUESTION ANSWERING WITH INTERACTING CONTEXTUAL PROPERTIES
Haitian Sun
William W. Cohen
Ruslan Salakhutdinov
School of Computer Science Cameeiie Mellon University hait Auttian@cco cmu. edu
Google Brain
School of Computer Science
wcohen@google..om com
Carnegie Mellon University
rsalakhu@ce cmu, edu
ABSTRACT
In the scenario-based Question Answering (QA) task, models are asked find answers that are appropriate to the user scenarios associated with the question and identify information that is missing from the scenarios but is necessary for the answers to hold. Scenarios commonly include multiple properties of users, such age, employment status, and income level for the question "How much can claim from this benefit". The properties relevant to potential answer are given in a document, which will state conditions necessary for the answer to hold. Documents also may specify how conditions interact with each other, e.g. with text like "one of the conditions below must apply". Although understanding the relationship between conditions is crucial for solving this challenging QA task, limited work has been done so far in modeling this. In this paper, we propose the T-Reasoner model, which solves this problem with three jointly learned modules: an entailment module which checks whether a condition has been satisfied by the scenario, a decoding module which locates eligible answers from documents, and a reasoning module which infers the relationship between conditions and performs reasoning step determine the logically consistent answers and identify missing conditions. T-Reasoner outperforms strong baselines on a synthetic scenario- based QA dataset and achieves new state-of-the of------thttttte art two deemario-based QA benchmarks, outperforming the prior best models by 3-10 points.
INTRODUCTION
Many questions can only be answered correctly after some context for the question is supplied or inferred: e.g., "When the next LA Lakers home game" needs temporal context, and "Where the
closest pizza place" needs geographical context. Prior work on contextual QA (Zhang & Choil 202T
Dhingra et al. 2021 Kasai al., 2022 Chen et al. 202T) has focused on tasks in which context is important, but limited: generally a small number of properties of the user that posed the question need be considered (e.g., location and time). However, many important questions depend many more properties of the user. In this paper we consider acenario-based QA, in which questions are augmented with textual "scenario" that describes some properties of the user. For example, in Figure user has posed a question "how much support am I cligible for?", and the answer depends on multiple user properties (namely, their relationship with deceased, and whether they or other relatives have claimed other benefits.) Having multiple contextual properties means these properties can interact. For example, in T the answer depends on conjunction of conditions (e.g. "if both" in Scenario 1) and also disjunction of conditions (e.g. either being a "relative" or a "close friend" in Scenario 2).
In our benchmarks, scenarios are informative but not complete, so the goal of the system is to identify possible answers-i.ee answers that are logically consistent with the scenario as well as any conditions that necessary for the answer to hold which are not entailed by the scenario. For example, in Figure Scennari 1, the system should provide the answer "up to $1200" but must also note that the condition "you didn't claim other benefits" is required by the answer, and not entailed by the scenario. We refer to such conditions unsatisfied conditions. This task is challenging because in addition to finding eligible answers from documents, it also requires models to perform two non-trive non-trivial reasoning tasks. First, it must understand the document well enough to understand conditions given as
Codes and data are available http::://////.... httppp::///// http///gttbbb.tt// thub. www/haitttttoooooooooeeeoeac..at sun/1 Keasoner

---

Published as a conference paper at ICLR 2023
context for the answer (each property that may affect the answer is considered condition), and the logical relationship between these conditions. For example, "the partner of the deceased..." " and "you didn't claim other benefits" to be satisfied (i.e. conjunction), while it requires either a "relative" or "close friend" (i.e. disjunction) in Scenario 2. Second, a model must identify which conditions are entailed by information provided in user scenarios, which are contradicted, and which are not mentioned but are required to support an eligible answer.
1, it requires both
Previous work by Clark et al. (2020b) has shown that pretrained Language Models (LMs), c.g. RoBERTa (Liu et al. 2019), can be finetuned to perform a similar reasoning task over hypothetical statements, I.e. "if' A and B then C". However, conditions used in their experiments are over simplified and sometimes semantically incorrect, e-g. A "Mike is strong" and "Cindy is green". Furthermore, languages used to described the relationship between conditions are easy, and the number of conditions involved in the reasoning process is small. All factors above make the proposed task easy for existing models (Liu et al. 2019: Raffel et al. 2019), but undre-tepsesents the challenges exists in real problems that require reasoning with logically interacting conditions.
Furthermore, previous work (Clark et al., 2020b) makes an assumption that every conditions must be either satisfied or contradicted by the evidence provided in questions. As a result, no "unsatisfied condition" is required in predictions. We do not make such assumption, but instead only provide evi- dences for a subset of conditions, and ask models to predict a logically consistent answer and identify conditions that are required but not yet satisfied, i.e. unsatisfied conditions. Indeed, experiments (Sun) et al. 2021a) show that pretrained language models (LMs), e.g. T5 (Raffel et al. 2019), struggle to predict unsatisfied conditions. Even though an additional module is specifically trained to predict unsatisfied conditions (Gao et al. 2020b Ouyang el al. 2020), their performance is still limited.
We propose simple yet effective model, T-Reasoner, which models the relationship between conditions and performs the reasoning task to verify answers that are consistent with user scenarios and identify conditions that unsatisfied. T-Reasoner contains three main modules, entailment module, a reasoning module, and a decoding module, which are jointly trained. The entailment module predicts whether conditions have been entailed or contradicted by users' scenarios. The reasoning module infers the relationship between conditions then performs reasoning step to decide whether the provided information in user scenarios is sufficient and to identify unsatisfied conditions otherwise. If the answer a free-form text span, T-Reasoner additionally uses a generation module to predict the answer span. T-Reasoner shows excellent reasoning ability on synthetic dataset and outperforms the previous state-of -f-the- models on two Question Answering (QA) datasets, ConditionalQA and ShARC (Sun et al., [021a] [Saeidi et al.] (018),, improving the state-of- state-of-the-art by 3-10 points on answer and unsatisfied condition prediction tasks.
2 RELATED WORK
The task proposed by Clark et al.
(020b) is commonly referred to as deductive reasoning where all information required to find a definite answer provided. Other models have been developed for deductive reasoning with symbolic rules (Cohen 2016: Cohen et al., 2020 Sun et al., 2020 Ren et al. 2020 Ren & Leskovee 2020). Embedding- based methods (Sun al. 2020 Ren et al. 2020 Ren & Leskovee (020) first convert symbolic facts and rules to embeddings and then apply neural network layers on top to softly predict answers. These models differ from our work in that the symbolic structure of the rules is typically known, whereas in our model it is implicit in a document.
Other recent work in deductive reasoning focused tasks where rules and facts expressed in natural language (Talmor et al.) 2020, (aaeed et al. 2021: Clark et al.) 2020b]] Kassner et al., 20200). Such tasks are more challenging because the model has to first understand the logic described in the natural language sentences before performing logical reasoning. Many of these models rely on rules that are produced by templates, or templated rules that have been paraphrased by crowd workers. In our work, the logical interactions analogous to these rules are implicit in real-world documents.
Different from most reasoning tasks, the task considered in this paper provides a list of conditions that, if true, would support an answer. Identifying such conditions is usually called abductive reasoning, as opposed to deductive reasoning. Very limited work has explored abductive reasoning for QA. Previous work (Gao et al. 2020a by Ouyang al. 2020) on the ShARC (Sacidi et al., 2018) dataset propose to solve this problem by predicting a special label "inquire" if there was not enough information to make definite prediction. Specifically, EMT and DISCERN (Gao et al. 2020a b) computed
2

---

Published as a conference paper at ICLR 2023
Document (excerpted):
Funeral Expense Payment up to $1200 may be awarded if both: the partner of the deceased when they died
didn't claim other benefits
you relative close friend the deceased, you may claim to $800 thoir partner did not claim.
Scenarlo 1: My husband has succumbed to cancer.
Scenarlo 2: My mom passed last month and I am her only relative.
Question: How much support am eligible for?
Question:
How much support am cligible for?
Conditions
Relationship
Answers
Conditions
Relationship
Answers
"partner"
"partner"
"didn't claim other benelits"
AND
up $1200"
"didn't claim other benefits"
AND
uPp $1200*
"relative"
"relative"
"close friend'
OR
*close friend"
OR
"partner did not claim'
AND
"up to $800
"partner did not claim"
AND
uPp to $800"
Answer: $1200
Answer: to $800 Unsatisfied Conditions: 0
Unsatisfied Conditions: you didn't other benefits"]
Examples for the scenario-based QA task. The examples have the question but different scenarios which lead to different answers. Diagrams in the blue blocks show desired reasoning process. The conditions green, red, and yellow are entailed, contradicted, and not mentioned the scenario. In Scenario 1, the answer "up to $1200" logically consistent with the scenario. The condition "you didn't 't claim other henefits" is an unsatisfied condition because it not mentioned hut required to he satisfied for the answer. The other condition "partner did not claim" is also not mentioned hut is not an unsatisfied condition hecause whether it is satisfied will not affect the answer. In Scenario 2, there's not unsatisfied condition because existing information provided in the scenario sufficient predict the answer "up $800".
Figure 1:
an entailment vector for each condition and performed weighted sum of those vectors to predict the final answer. DGM (Ouyang al. 2020) additionally introduced GCN-based model to better represent the entailment vectors. Even though these models were able to predict the answer labels as "inquire" when there unsatisfied conditions, further satisfied, unlike our model. Our model is also more scalable than these, as it does not require concatenating full context and question.
of them predict which conditions needed be
MODEL
3.1
TASK: QA WITH CONDITIONS
The scenario-based QA task requires models to find answers that are logically consistent with the provided user scenarios which are potentially incomplete. In this paper, we consider this task in the reading comprehension (RC) setting in which passage that contains relevant information about the question is provided. We leave the open-domain setting of this problem for future work. Specifically, a model takes question, a scenario, and a passage that contains answers and conditions as input and predicts logically consistent answers and their unsatisfied conditions. Let's consider a passage that contains a set conditions all possible combinations of conditions A = {a....., am }. Each answer a; A is restricted by a subset of conditions C. C C. Conditions in C₁ interact with each other under relationship R, (R, is an abstract set which will not be explicitly expressed). A condition group, G; (C2, R&) is pair of C, and R,, which describes answers A, sets of conditions C,'s ss and their relationship R, 's are not explicitly provided in training and testing examples models have to generate them from the passage.
(c.,...,e , c, and the set of eligible answers for a question under
what scenario the answer a; is correct. Note that the list of
We say that a condition group G; is satisfied if its underlying logical statement that consists of C; and R. has been satisfied by the scenario, for example, in Scenario 2 in Figure where the condition group for "up to $800" has been satisfied. Besides being satisfied, a condition group G, has two more possible outcomes: (1) is partially satisfied if some of the conditions have been satisfied but there is still some information missing so the answer is not fully supported, e.g. the condition group of the answer "up to $1200" in Scenario 1 (Figure and (2) Ge is contradicted if one or more
3

---

Published as a conference paper at ICLR 2023
conditions in the group contradicted which causes the answer ineligible, e.g. the condition group of the answer "up to $1200" in Scenario 2 (Figure
An answer o; is logically consistent with the scenario if the underlying condition group G; is satisfied or partially satisfied. We denote the set of logically consistent answers A S A. The set contains zero or more answers - the set A is empty if none of the answers in A is logically consistent with the user scenario. A model should predict an answer from À if Á is not empty, and mark the question as not answerable, otherwise In addition predicting logically consistent answers, we also perform the task of finding unsatisfied conditions C. The set C, should be concise, i.e. it should only include the conditions that are necessary. For example, the condition "have worked for more than 4 years" is an unsatisfied condition because whether it has been satisfied not won't affect the output of the condition group.
In summary, we evaluate a model's prediction of a logically consistent answer a; E Á and the set of unsatisfied conditions C₂ for answer ag i.e. (a,, Cj). Answers and unsatisfied conditions in the output are jointly evaluated This task specifically challenges models' ability in understanding the relationship between conditions and performing logical reasoning process accordingly. We will introduce& a simple and effective model, T-Reasoner, to tackle this challenging reasoning task.
3.2 MODEL
In this section, we will discuss T-Reasoner which consists of an entailment module, reasoning module, and optionally a decod- ing module, perform this challenging QA task in embedding space.
partially satisfied
condition
$1200"
Reasoning
Decoding
Input The model, T-Reasoner, takes ques- tion q with scenario e and passage p as inputs and predicts an answer a; that is log- ically consistent with the user scenario and a list of unsatisfied conditions O.. Since the list of all conditions C' for the question are not provided in the example, we chunk the passage p into pieces and consider each piece of text as a condition c.. Conditions obtained this way may be irrelevant to ques- tions. We rely on the entailment module (see next) to decide whether a condition & is rel- evant and what is its relationship with others. The chunking strategy may be different for different datasets. Please see $4.2 and $4.33 for more information. Briefly, passages are usually chunked into sentences, short pas- sages with 2-3 sentences, or sub-sentences (text phrases).
oe h
Entailment
Entailment
Entailment
$1200 betts <COT> the partner of the de decessed <ICDT>..."
5,
o: "My huaband has sucoumbed to q: "How mush support am digiille for?"
Figure 2: T-Reasoner Overview. The entailment module independently encodes each condition e; (along with its con- textual information s.). 'The entailment module outputs condition embedding that will be input into the reason- ing module decide if conditions have been satisfied and identify unsatisfied conditions. Other computed embeddings h., will be used by the decoding module to generate answer spans (if the question has a free-form answer).
Entailment Module We apply an entailment module check whether each condition c; E C have been entailed by the user scenario. Each condition is checked independently, as opposed to concatenating all conditions into long input and checking them all once. This strategy significantly reduce the computation cost compared to checking all conditions at once, especially if context is long, e.g. legal documents which are tens or hundreds of pages long (see examples 4.3) Specifically.
"An oracle model should be able to predict all answers from Á. We consider a slightly simplified setting in this paper in which model is only required to predict one of the answers. In experiments, the ShARC Sacidi et al. 2018) dataset only contains questions that have a single answer, i.c. |Å] 1. The ConditionalQA 2021a) dataset contains questions that have multiple answers, |Á] >>> 1, so the performance will be sacrificed. We leave the task of predicting all logically consistent answers as future work.
Sun et al.
"Evalaation metrics are different in different datasets Sun al. 202 Sacidi et al. 2018 Please refer
and 4.2 for more details.
4

---

Published as a conference paper at ICLR 2023
the computation complexity of our approach is O(ICI) where [C is the number of total conditions, compared to a complexity of O((C)²) otherwise.
This independent checking strategy, however, separates each condition from its context and thus causes a lost of contextual information for conditions e; and eventually negatively impacts the model's performance. Thus, we extend a condition G; by adding tokens from its surroundings. For example, the condition "the partner of the deceased when they died is expanded "... up to $1200 may be awarded if both: <CDT> the partner of the deceased when they died < <\CDT> you didn't 't claim ...". where <CDT> and < <DT> are two special tokens that mark the beginning and end of the condition G. Apart from making a condition c; more fluent and coherent, the added contextual tokens also make it easier understand the relationship between the current condition G; and other conditions in its neighbours. We may additionally add page titles, section titles. prompts of list items, or table headings, etc., if applicable to the expanded conditions. Please see $4.3 and 4.2 for more details. We denote conditions with extended contextual information as 8; for condition C;
We learn a Transformer model for the entailment module which takes expanded condition s;
and the question y and scenario e as input, and returns a list of vectors s, h,,,,.... ........,,,m.. The first vector s; is a summarization vector which includes several aspects of information: (1) whether the underlying condition e has been satisfied, contradicted, or not mentioned by the user scenario, (2) whether the condition is relevant to the question, and (3) if relevant, what is its relationship with other conditions in its neighbours. These information will be used for reasoning in the future layers. are token embeddings that will used for decoding if needed. Please see
Embeddings ha.1.-. .... hi. the description of the reasoning module for more information.
S; h; hi
Entail (5;, e, e,q)
(1)
One may consider supervising this entailment module by adding classification layers on to explicitly predict the entailment status of condition e and its relationship with other conditions. However, obtaining supervision labels for these auxiliary tasks can be challenging as they are often not provided in the example. Fortunately, show that our proposed model, T-Reasoner, can be trained end-to-end, without such intermediate supervision.
Decoding Module The decoding module generates an eligible answer å, which is potentially logically consistent to the question. The generated answer à, will not be returned until the status of its condition group G. is verified by the reasoning module (discussed below).
The decoding module is analogous to FiD (Izacard & Grave 2020), i.c. token embeddings from different conditions (which are encoded separately) are concatenated for decoding. Different from Izacard & Grave (2020) which applied to independently retrieved passages for open-domain QA, the decoding module in T-Reasoner is used on coherent content, i.e. conditions from the same passage. The contextual information in the expanded condition s; helps connect conditions that are separately encoded. The decoding module takes token embeddings for all conditions hi.1..... h computed from Eq. to generate answer spans. The generation task is trained with teacher forcing, We do not write out the teacher foreing decoding loss ldecode here. Please refer to the T5 paper (Raffel ct al. 2019) for more information. If questions have multiple logically consistent answers, i.c. A > we randomly select an answer a; E Å as the label train the decoding module.
à Decode (b.........
(2)
We consider two different types of answers: "Yes"/"No" free-form answers. In the first case, we simply let the model generate a special a special token [YESNO] and consider the reasoning result from the reasoning module (see next) as the answer, i.e. the answer is "Yes" if the condition group is satisfied (or partially satisfied) or "No" if contradicted. Since some datasets only contain "Yes"/"No" questions, we can then safely discard the decoding module for these datasets. In the second case, i.c. answers are free-form text spans, we will return generated spans as answers only their condition groups have been verified as satisfied or partially satisfied by the reasoning module. If the condition group is contradicted, we will mark the question as not answerable.
Reasoning Module The reasoning module combines the local relationship between conditions from their embeddings , S,, and performs a logical reasoning process to decide the reasoning result for a condition group G; for the generated answer à, and to identify unsatisfied conditions C.
The input to the reasoning module is a list of vectors, $1,..., S, for conditions cI,..., Cn. that are output by the entailment module (Fq. [. We use another Transformer model as reasoner, because
5

---

Published as a conference paper at ICLR 2023
Transformers have the self attention mechanism which allows conditions ........ Sn ) to attend to each other, so the reasoning result a condition group can be summarized. This is crucial because, for example, if one of the conditions in disjunction group is satisfied, the condition group will be automatically satisfied regardless the status of other conditions the same group. We prepend a trainable vector to the list of condition embeddings to summarize the reasoning result.
The output vectors So, S will be used predict the status of the condition group and the unsatisfied conditions for the generated answer. The first vector So will be used to predict the reasoning result of the condition group. If the condition group is partially satisfied, we use the rest of vectors, , Sn, to identify unsatisfied conditions. We compute loss on both reasoning and unsatisfied condition predictions. Let ll, and be the one-hot labels for the two tasks.
Sp, $1, Sn Reason(So, -.,-...S.
Treason softmax_crose caaimax__cooss_eetpppp (W/ So, Tr) connd =
W 1o)
As discussed above ($3.)) the reasoning results of condition groups have three possible outcomes: "satisfied", "partially satisfied", and "contradicted". We merge the first two into one label "satisfied", and differentiate them by whether unsatisfied conditions exist, i.e. r E {satisfied, contradicted} and its one-hot label II,
1
Labels for conditions are "entailed", "contradicted", "not mentioned", "implied", and "unsatisfied", i.e. To E {0, 13 The first three labels are as they are named. The label "implied" means a condition implied by other conditions the condition group. For example, one of the conditions a disjunction group has been satisfied, the rest of conditions are "implied". The class "unsatisfied" means is an unsatisfied condition which must be returned together with the predicted answer. The labels may not apply to all datasets, c.g. ConditionalQA (Sun et al. (02 only annotates two labels "unsatisfied" vs. others, we will make changes to the loss function accordingly.
Loss Function We jointly train the entailment module and reasoning module. The final loss function is the sum of the answer loss Tressnn and the condition entailment loss loond- If the answers contain text spans, we jointly train the decoding module ddecoee as well.
( bresson + Loand
= Leason + loood + ddcoode
3.3 FINETUNE PRETRAINED CHECKPOINTS
The entailment module and decoding module (if adopted) load pretrained LM checkpoints, e.g. T5 (Raffel et al. 2019) and BART (Lewis et al., (019). The pretrained parameters are loaded for the entailment module and then finetuned for downstream tasks. The reasoning module is randomly initialized and jointly trained with other modules. The number of Transformer layers in the reasoning module is a pyper-paramettr. We choose the number of layers / = / = 4. Please see $4.1 ablation study on the number of Transformer layers for the reasoning task. The decoding module is also finetuned. If decoding module needed, we will initialize the entailment and decoding module from the same pretrained checkpoint.
4 EXPERIMENTS
We experiment with T-Reasoner on synthetic dataset, CondNLI, and two benchmark QA datasets, ConditionalQA (Sun et al. 2021a and ShARC (Sacidi et al. 2018), for scenario-based QA task.
4.1 CONDNLI
Dataset The synthetic CondNLI dataset is derived from an existing Natural Language Inference (NLI) dataset, MultiNLI (Williams et al.] 2018). An original NLI example contains a premise and a hypothesis, and a label indicating whether the premise entailed or contradicted by the hypothesis. We treat premises in NLI examples as conditions and hypotheses as facts provided in user scenarios.
"Some tasks have an additional class "irrelevant" because questions in the dataset not relevant the provided passages, i.e. T, E {0, 1

---

Published as a conference paper at ICLR 2023
Context: If all:
Ans (acc)
Conds (F1)
"Aged 59 1/2 or older".
TS (w/ FiD)
85.6
80.4
"Employed for two years"
ETC T-Reasoner 95.0
91.4
82.7
} then "Get at least S60 a week".
91.3
If any:
not "Has two children".
Table 2: Experiment results on the synthetic
"Has applied before."
CondNLI dataset in answer accuracy (Ans) and con- ditions Fl (Conds).
] then "Waive the upplication fees".
Question: "Eligible for $60 a week" correct?
6
8
10
15
20
Scenario: ["65 years old". "Rejected last year"]
T5 (w/ FiD)
85.6
85.2
85.3
82.2
74.1
Answer: Yes. | ("Employed for two years")
T- T-Reasoner
95.0
94.3
94.5
92.8
91.7
Table I: An example in COndNLL The answer is "Yes"
Table 3: Experiment results in generalizing to conditions at inference time.
with unsatisfied conditions [ ["Employed for two years") "].
An example is shown in Table[] The example contains four conditions, among which "Aged 59 1/2 or older" and "Employed for two years" belong condition group under a logical reasoning type "all", indicating that both conditions have to be satisfied in order to "Get at least $60 a week". The answer statement, e.g. "Get least $60 a week", also comes from NLI examples. We treat the premise of an NLI example as an answer statement and the corresponding hypothesis as the question, e.g. is "Eligible for $60 week" correct? In addition to the condition group and the answer statement that is relevant to the question, we add a few more condition groups as distractors to make the constructed dataset more challenging. Please Appendix
A for more information in dataset construction.
Baselines Previous work (Clark et al. 20206) showed that pretrained Transformer- based Language Models, e.g. RoBERTa (Liu et al.]]]]]111 2019)), have the ability to reason over multiple conditions answer a reasoning question in the deductive reasoning setting, e.g. "if A and then C" with facts on both conditions A and B provided. However, examples in CondNLI are usually longer and won't 't fit into RoBERTa's memory. Equivalently, we experiment with two other language models, T5 (Raffel et al. 2019) (with the FiD strategy (Izacard & Grave 2020) to adapt to longer input) and ETC (Ainslie et al. 2020), on the CondNI. d In ETC, we use the global tokens to predict unsatisfied conditions. In T5, To simplify the generation task, we assign an id to each condition and let FiD) generate unsatisfied condition ids. We also compare T-Reasoner with T5 on inputs that contains more conditions to test their generalization ability.
Results The experiment results are shown in Table 2 We measure both the accuracy of label prediction and the F1 of unsatisfied conditions. The results show that T-Reasoner performs significantly better than pretrained LMs, T5 and ETC, in both predicting rect answers (Ans) and unsatisfied conditions (Conds) on CondNLI. We additionally test T-Reasoner's 's abil- ity in generalizing to more conditions. We train T- Reasoner on templates with 6 conditions or fewer and test on the examples with more than 6 condi- tions. Figure 3 (Left) shows the change of perfor- mance in both label classification and unsatisfied con- dition prediction tasks as the number of conditions increase. We observe some decrease in performance in both tasks, but it is still reasonable with 20 con- ditions. Furthermore, we experiment with different numbers of layers in the reasoning module (Right). reasoning module needs least 3 layers for the reasoning task, especially for predicting unsatisfied conditions.
More conditions
Reasoner layers
100
50-
90-
80
80-
label acc (overal))
label (overal)) Gany
dany
75
75-
10
15
20
layers 6
conditions
Figure 3: Left: Generalization results of reason- ing over more conditions. Right: Results on the ablated model with different numbers of Trans- former layers in the reasoning module. We report both label accuracy and Fl of unsatisfied condi- tions. "any & all" indicates the subset of validation data in which conditions interact each other under the relationship of "any" and "all".
The Transformer-
4.2 SHARC
Dataset In the second experiment, we run T-Reasoner on real scenario-based QA dataset, ShARC (Saeidi et al. 2018), that has complex passages and many conditions. An example in ShARC contains
Exampless in CondNI. exceeds the limit of 512 tokens RoBERTa.
7

---

Published as a conference paper at ICLR 2023
Decision (micro/ macro)
Question (BLEU1 /4)
Decision (micro macro)
Question (BLEU1 4)
Condition (FI)
CM
61.9 / 68.9
54.4 34.4
T5 DISCERN
63.7 / 68.2
57.3 48.2
44.0
BERTQA
63.6 / 70.8
46.2 36.3
74.9 / 79.8
65.7 / 52.4
55.3
UcraNet
65.1/71.2
60.5 46.1
DGM
78.6 / 82.2
71.8/ / 60.2
57.8
Bison
66.9/7166
58.8/44, 44.3
T-Reasoner
79.8 / 83.5
71.7 / 60.4
69.2
E3
73.3
54.1/38.7
Table 5: Results the ShARC dataset (dev) compared to the baselines. The Condition (F1) numbers from open-
EMT
69.1/74.6
43.9
DISCERN
73.2/78.3
64.0/49. 49.1
sourced codes (Gao al. 2020b Ouyang al. 2020)
DGM
81.4/812
63.3 / 48.4
T-Reasoner
80.4/83.9
71.5/ 58.0
# conditions
1
2
3
4
Table 4:
Experimental results the ShARC datasct. Numbers for the baseline models (Sacidi
DGM
90.4
70.3
80.0
73.4
T- T-Reasoner
90.3
72.7
80.6
75.2
2018 Zhong & Zettlemoyer 2019 Verma et al.
al.
2020 Lawrence et al. 2019 Gao et al.
diff'
-0.1
2.4
0.6
1.8
2020a b) Ouyang et al. 2020) are horrowed from
Table 6: Ablated results on ShARC on answer accuracy
Ouyang (2020))
vs. number of conditions. Numbers of DGM (uyang et al. 2020 are obtained from open-sourced codes.
passage, a user question, and user scenario which is expressed in conversation history between a user and a machine. A model is expected to find an answer to the user's question, or raise a clarification question for the unsatisfied conditions. Answers in this dataset are restricted to one of the following labels: "yes", "no"" "inquire", and "irrelevant". The first three labels are equivalent to "satisfied", "contradicted", and "partially satisfied". "irrelvant" is a new label that should be predicted if the conversation history and the question are irrelevant to the provided passage. This task of predicting answers is called "Decision Making" in their original ShARC paper (Sacidi et al., 2018) and evaluated as micro and maero accuracy. In addition to the "Decision Making" task, they consider another task "Question Generation" " which is equivalent to predicting unsatisfied condition in "Reasoner@evaluuee evaluated with BLEU 1 and BLEU 4 scores. Compared to CondNLI, where conditions and their relationship are clearly mentioned in the context, conditions are embedded in the context in ShARC examples, e.g. Figure Please Appendix for more information in data preparation.
Baselines and Results We compare T-Reasoner to several strong baseline models, including the previous state-of-the-art models, DISCERN (Gao et al., (020b) and DGM (Ouyang et al. (0200). Different from the baseline models, which use pipeline systems to separately predict answer labels and unsatisfied conditions, T-Reasoner performs the two tasks jointly. The results are shown in Table T-Reasoner outperforms the previous baselines by points on the "Decision Making" task and
more than 8 points on the "Question Generation" task. T-Reasoner also significantly outperforms
other baseline models (Saeidi al. 2018 Zhong Zettlemoyer 2019
Verma al. 2020 Lawrence
et al.
2019
Gao et al. 2020a b Ouyang el al. 2020).
Condition Accuracy One problem with the ShARC Question Generation task is that only one of the unsatisfied conditions is annotated, even though multiple unsatisfied conditions exist. To further evaluate T-Reasoner's 's performance in predicting all unsatisfied conditions, we manually annotate the logical operations in 20 contexts that have more than one condition (857 data total) and use the annotated logical operations to find all unsatisfied conditions. We report the F1 of the
A Ablation:
predicted unsatisfied conditions (see Table 55. Compared to the baselines (Gao et al. 2020b Ouyang et al. )020), T-Reasoner improves the Fl by 11.4 points.
Ablation: Label Accuracy v.s. Conditions We additionally measure the accuracy versus the number of conditions in the context. Results in Thbl that the improvement in T-Reasoner's performance over the previous state-of-the-an model (DGM) mostly comes from questions that have more than one condition.
4.3 CONDITIONAL
Dataset In the third experiment, we run T-Reasonne on ConditionalQA (Sun et al. (021a), which contains longer context (documents), more conditions and more complex relationship between
UUnsatistied conditions are then paraphrased into questions, e.g. "Aged 59 1/2 or older" is paraphrased to "Are you aged 59 1/2 or older?" " Fach context ShARC has 32.9 data average.
8

---

Published as a conference paper at ICLR 2023
Yes No conds
Extractive
Conditional
Overall
EM/FI
EM/FI
w/ conds
EM F1
conds
EM
w/ conds
majority ETC
62.2/ 62.2
42.8/428
-/-
-/-
-/-
-/-
23.1/63.1
47.5
8.9 17.3
6.9 14.6
39.4 / 41.8
3,5/3,4
35.6/39.8
26.9/ 30.8
DocHopper
64.9/649
49.1/49.1
17.8 26.7
15.5/23.6
42.0 46.4
3.1/
40.6/ 45.2
31./ 36,0
36,0
TS w/ FiD
64.2/64..
48.0/48.0
2
23.5/33.4
45.2 / 49.7
4.7/5.8
44.4/
35.0/ 40.6
T-Reasoner
73.2/73..
54.7 54.7
34.4/ 04.4
30.3/43.1
51.6/ 56.0
12.5 14,4
57.2 63.5
/6.1/51.9
Table 7: Main results on ConditionalQA. The "EM/FI" columns reports the original EM/FI metrics which only evaluate answers. The "w/ conds" is the conditional EM/FI metric from ConditionalQAA See $4.3
Answer (w/ conds)
Conditions (P/R/FI)
TS w/ Fil T-Reasoner
3.2
9..3/2.6/2.7
10.6 / 12.2
40.4 34.4/ 34.4/40.4/ 37.8
T5 w/ FiD (conditional only)
6.8 7.4
13.8/63.0/ 21.3
Table 8: Ablated results on ConditionalQA in predicting unsatisfied conditions.
conditions. Furthermore, the ConditionalQA dataset contains mixture of "Yes"/NNo" questions and questions with free-form answers. Please see Appendix for details on data preparation.
Evaluation As introduced in ConditionalQA (Sun et al. 2021a), predictions are evaluated in two sets of metrics: EM/F1 and conditional EM/F1. EM/FT the traditional metrics that measures the accuracy of predicted answer spans. Conditional EM/F1 is a novel metric introduced by [Sun al. (202Ta), that jointly measures the accuracy of answer spans and unsatisfied conditions. Please refer to the ConditionalQA paper (Sun et al. 2021a) for more information. Briefly, the conditional EM/F1 is the product of the original answer EM/F1 and the F1 of the predicted unsatisfied conditions. The conditional EM/F1 is 1.0 if and only if the predicted answer span is correct and all unsatisfied conditions are found. If there's 's no unsatisfied condition, the model should predict empty set.
Baselines and Results We compare T-Reasoner with several strong baselines, including ETC (in pipeline) (Ainslie et al. (020), DocHopper (Sun et al.) 2021b), and T5 (with FiD) (Izacard & Grave 2020). The ETC pipeline first extracts possible answers from the context and then predict unsatisfied conditions independently. DocHopper is multi-hop retrieval system that iteratively retrieves evidence which contains answers and unsatisfied conditions. T5 (w/ FiD) is a encoder- decoder model. We train T5 (w/ FiD) to generate answers followed by a list of unsatisfied conditions ids. The experimental results are presented in Table 7 T-Reasoner significantly outperforms the baselines in predicting answers and jointly predicting answers and unsatisfied conditions relative improvement of 148% (Conditional) and 27.8% (Overall) in conditional F1 (F1 w/ conds).
Ablation: Condition Accuracy Since there's not metric that only measures the quality of predicted conditions, additionally report the F1 of the predicted unsatisfied conditions (Table 2 The best baseline models, T5 (w/ FiD), rarely predicts any conditions. Even though we train T5 (w/ FiD) only on the subset of questions that have conditional answers to force it predict unsatisfied conditions, its performance slightly improves but still much lower than T-Reasoner by 16.5 points in condition Fl.
5 CONCLUSION
We study the problem of scenario-based QA in which questions are accompanied by incomplete scenarios and models are asked to find answers that are consistent with the provided user scenario. Models are further asked identify unsatisfied conditions that are necessary for the predicted answers. We propose a system, T-Reasoner, that contains an entailment module to check whether a condition has been satisfied and a jointly trained reasoning module to verify the status of condition groups and predict unsatisfied conditions. T-Reasoner shows excellent reasoning ability, and can easily generalize to conditions a synthetic dataset CondNLI. Furthermore, T-Reasoner achieves state-of-the-art performance on two challenging scenario-based QA datasets ShARC (Saeidi et al. 2018) and ConditionalQA (Sun et al. 2021a

---

Published as a conference paper at ICLR 2023
ETHICAL STATEMENT
Experiments in this paper are performed on publicly available datasets for academic research purposes. No real user data used the experiments. Even though the proposed model could be applied to many real world problems to help users answer their questions, the accuracy of the proposed work is still limited and predictions may be misleading. Please carefully evaluate the performance before applying to real problems.
7 REPRODUCIBILITY STATEMENT
Datasets and codes will be released upon the acceptance of this paper, including all scripts for constructing the proposed synthetic dataset and the preprocessing script for the two benchmark QA datasets. Models are trained on public available data. All results are reproducible.
REFERENCES
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. ETC: encoding long and structured data in transformers. CoRR, 0es/2004.08483. 2020. URL
https://arxiv. org.bbs 20044.002... 08483
Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time-sensitive questions, 2021.
Kevin Clark, Minh- Luong, Quoc V. Le, and Christopher D. Manning, ELECTRA: pre-training text encoders discriminators rather than generators. CoRR, abs/2003.10555 2020a. URL
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language. CoRR, abs/2002.05867, 2020b.
05867
William W Cohen. Tensorlog: A differentiable deductive database. arXiv preprint 100.::3655.06523, 2016.
William W Cohen, Haitian Sun, Alex Hofer, and Matthew Siegler. Scalable neural methods for reasoning with a symbolic knowledge base. arXiv preprint arXiv:2002.06/15. 2020.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-aware language models as temporal knowledge bases, 2021.
Yifan Gao, Thien-Sheng Wu, Shafiq Joty, Caiming Xiong, Richard Socher, Irwin King, Michael Lyu, and Steven C.H. Hoi. Explicit memory tracker with coarse-to-fin reasoning for conversational machine reading. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 935-945, Online, July 2020a. Association for Computational Linguistics. doi: 10. URL URLhttps. https/////aaaaattooooooooo
1t1:3//w12222.ac--mmii
org 0000 -
Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq R. Joty, Steven C. H. Hoi, Caiming Xiong, Irwin King, and Michael R. Lyu. Discem: Discourse-aware entailment reasoning network for conver- sational machine reading. CoRR, abs/2010.01838, 2020b. URL
200 0 01 838
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. CoRR, /tlb//000.01282. 2020. URL htttts///aaxiiiiii abbs/ 2007 01282
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What's the answer right now?, 2022. URL htttts:///aaat http://awwww..og/aaee/es www./ba.//.......o..//.........e /2007 13332
Nora Kassner, Benno Krojer, and Hinrich Schütze. Are pretrained language models symbolic reasoners over knowledge? arXiv preprint arXiv:2006. 10413, 2020.
10

---

Published as a conference paper at ICLR 2023
Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. Attending to future tokens for bidirectional sequence generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pp. 1-10, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/1/D11--1
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequennceto--equuene pre-training for natural language generation, translation, and comprehension. CoRR, ttss//910.1346.. 2019, URL
http://arxiv. org hbbrgg/bbs//00
Jing Li, Aixin Sun, and Shafiq R Joty. Segbot: A generic neural text segmentation model with pointer network. In IJCAI, pp. 4166 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeullmoyer, and Veselin Stoyanov, Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. URLhttt://arxttt ottpe/iviirrraaooooo abs 1tt.//9977.11111
Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. Dialogue graph modeling for conversational machine reading. CoRR, abs/2011.14487, 2020. URL /ttp:::://ax.....///////////////
14827
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, htt//9900.1683, 2019. htRt: http:///wxxxvvv...aabee htap////bbss. 1910. 10683)
Hongyu Ren and Jure Leskovec. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems, 33: 13:19916-19726, 2020.
Hongyu Ren, Weihua Hu, and Jure Leskovee. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. arXiv preprint arXiv:2002. 05900, 2020.
Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. Rulebert: Teaching soft rules to pre-trained language models. arXiv preprint arXiv:2109, 13006, 2021.
Marzieh Saeidi, Max Bartolo, Patrick S. H. Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. Interpretation of natural language rules in conver- sational machine reading. CoRR, abs/1809, 01494, 2018. URL http: http://axxvv..oo org/ /
1809. 01494
Haitian Sun, Andrew Arnold, Tania Bedrax Weiss, Fernando Pereira, and William W Cohen. Faithful embeddings for knowledge base queries. Advances in Neural Information Processing Systems, 33: 2020.
Haitian Sun, William W. Cohen, and Ruslan Salakhutdinov. Conditionalqa: A complex reading comprehension dataset with conditional answers. CoRR, abs/2110.06884, 2021a. URL https
sarxiv.org//bs:
6684
Haitian Sun, William W. Cohen, and Ruslan Salakhutdinov. End-to-end multihop retrieval for compositional question answering over long documents. CoRR, abs/2106.00200, 2021b. URL
httts::/arxiv.com
http:///aaxii.ooor////////////// wwt./or//////oooo.../o.ooo..eeat /atp//22....00 00200
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Seap-of-thoughh Teaching pre- prained models to systematically reason over implicit knowledge. Advances in Neural Information Processing Systems, 33:20227-20233 2020.
Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Danish Contractor, Harshit Kumar, and Sachindra Joshi. Neural conversational QA: Learning to reason vs exploiting patterns. In Proceedings of the
2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7263- 7269, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 mmnlp-main.589. URL http:://aaaaaathohaaoa org /200 emn wp- in. 589
Adina Williams, Nikita Nangia, and Samuel Bowman. A
challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 112-1122. Association for Computational Linguisties, 2018. _
http:/ //cc web. wee..rgg/ae. wttp.oorggaattooogg//////oo

---

Construct Templates We use capital letter A, B. to represent conditions and lower-cased letters a, b... to represent the corresponding facts. We use another few letters Y.... represent the statements of conclusion, and lower-cased letters x, ..... to represent questions. Conditions are grouped together under a logical operator that specifies the relationship between conditions. For example, a logical operator "all" specifies that all conditions in the group must be satisfied order to Published as a conference paper at ICLR 2023
(Template) Context: If all [A, BJ. then U. If any [not C, DJ, then
Facts: a, not d.
Question: Is correct?
Label: Yes, if B
(Variables)
A: Aged 59 1/2 or older.
a: Tom is 65 years old.
B: Employed for two years.
b: NOT USED
Has two children
c: has two sons.
D: Has not applicd before.
not d: Rejected last year.
U: Get at least $60 a week
u: Eligible for $60 week.
V: Waive the application fees
V: NOT_USED
Table 9: An example of CondNLI Variables A, B.... and U, V, ... represent the conditions and premises. Variables a, ... represent the known facts. is the question. Each pair of variables, e.g. (A, a), instantiated with an NLI example.
Michael J. Q. Zhang and Eunsol Choi. Situatedqa: Incorporating extra-linguistic contexts into qa, 2021.
Victor Zhong and Luke Zettlemoyer. E3: Entailment-driven extracting and editing for conversational machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2310-2320, Florence. Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19 1223. UtL/nttpss//allaantoooooo.ooo
A CONDNLI DATASET CONSTRUCTION
We first construct templates for the CondNLI examples and then instantiate the variables in the template with real NLI examples.
make the condition group satisfied. Here, we consider four types of logical operations construet this synthetic dataset:
"all": all conditions under this logical type should be satisfied in order to make the answer true.
"any":
only requires one of the conditions under the logical type "any" to be satisfied. doesn't matter whether other conditions have been satisfied, contradicted, not mentioned in the question.
This is special case "all" "any" when there is only one condition. Conditions with the logical type "required" must be satisfied.
"required":
"optional": Conditions have the type "optional" if they are not relevant to the question.
We pair a condition group with a conclusion statement and get a logical statement "If all (A, B), then X". To challenge models' ability in identifying relevant conditions from context, add few distracting statement that leads to different conclusions, e.g. "If all (not C, D), then Y". An example of context template is shown in Table
Facts are constructed by randomly sampling subset from all possible facts {a, b, b....) A question is sampled from possible questions (x, V.... }. We then compute the answer (and unsatisfied conditions any) from the context, facts, and the question.
Generate Examples For templates with variables A, B, X, Y...,, a, b, x. y...., we instantiate the variables with NLJ examples to get the real data. We use the premises of original NI.I examples
12

---

Published as a conference paper at ICLR 2023
Answer labels
(yes / no negated yes / negated no
100 / 100 100 100 /00/
partially satisfied irrelevant)
Logical types
378 / 380 / 422/390
(any all required/ / optional)
# conditions scenario (4 ///566.
333 333//40// /////40/4 /71
Table 10: CondNLI statistics. The table shows the statistics of 550 randomly generated examples. The answer labels "negated yes" and "negated no" mean negation exist in rule templates. We set the minimum number of conditions in scenarios to 4 to avoid overly simple example. Each example may contain multiple logical types.
for conditions and conclusions, i.c. capital letter variables, and the hypothesis for question and facts, ie. lower-case variables. Note that sampling requires matching the entailment state of conditions, e.g. "not d" requires sampling from IIJI examples that are labeled as "contradict".
We restrict the number of conditions in the context to 6 and randomly generate 65 distinct templates During training, we randomly pick a template and instantiate it with NLI examples generate real training examples. This random generation process enables creating (almost) unlimited amount of training data. We randomly generate another 5000 examples for development and testing.
Quality Assessment Training and validation data in CondNLI are generated from NLI examples in the training and validation split of the MNLI dataset, respectively. This ensures that NLI examples used validation are not exposed at training time. We control the generation process to ensure that the automatically generated data are balanced in terms of answer labels, logical types of interacting conditions, and number of conditions included in scenarios. Results are shown in Table [0] We additionally require scenarios must have at least 4 conditions to avoid overly simple examples.
We additionally measure the Jaceard distance between premises and hypotheses of the NLI examples used in constructing the CondNLI dataset. The token-level Jaccard distance is 27.2. Even though token-level overlap exists, model still needs to understand the semantic relationship between premises and hypotheses to predict their entailment status.
B CONDITIONAL.A EXPERIMENT DETAILSS
An example in the ConditionalQA dataset provides a parsed web page as context. It also provides question, and a user scenario that is relevant to the context. We prepend the user scenario to the question as input to the model.
The context in ConditionalQA is provided as list of HTML elements. We treat each element at the leaf of the DOM tree condition c,, and prepend all its parents (from its direct parent the root) to get an expanded condition 8,.
Since need the decoding module generate answer spans, initialize the model with T5, i.e. we use parameters from the encoder to initialize the entailment module, and use decoder to initialize the decoding module. The reasoning module is randomly initialized.
C SHARC EXPERIMENT DETAILS
Different from ConditionalQA, where each sentence in the context treated as condition, conditions in the ShARC dataset shorter and are sometimes short phrases (sub-sentence). For example, the context "If you are a female Vietnam Veteran with a child who has a birth defect, you are eligible for contains two conditions, "If you are female Vietnam Veteran" and "with child who has birth defect" order to handle sub-sentence conditions, we follow the strategy proposed two
purpose reducing training complexity. The experiment in Figure 3 (left) shows the model's capability of generalizing to more conditions.
Restricting number conditions only
'It is arguable that this could be generally treated as condition, but it treated conditions with the logical operator "all" in the ShARC dataset.
13

---

Published as a conference paper at ICLR 2023
of the baseline models, DISCERN Gao et al. (2020b) and DGM Ouyang et al. (2020), that split sentence into EDUs (Elementary Discourse Units) using pretrained discourse segmentation model ei al. (2018). The discourse segmentation model returns a list of sub-sentences, each considered as condition.
While we could treat each condition independently we did previously for other datasets, the segmented EDUs are different in that they are not full sentences and may not retain their semantic meaning, Thus, we consider using the full context (usually less than 5 512 tokens) as the contextual information for condition c₁, i.e. the expanded condition s; includes the full context, but the condition is highlighted using the special tokens <CDT> and <\CDT>.
We do not need the decoding module for the ShARC dataset, so
safely discard it. We initialize the entailment module with ELECTRA (Clark et al. 2020a). The previous state-of-the-ar baselines (Ouyang et al. 2020 Gao al. 2020b) use ELECTRA to initialize their model. We use the same pretrained checkpoint to make a fair comparison.
For the question generation task, we use the same input in decision making, except that we replace the prefix "condition:" with "unsatisfied condition:" for "unsatisfied" conditions. We fine-tune T5 model for question generation.
D DATASET STATISTICS
Dataset statistics are shown in Table
Train
Dev
Test
ShARC
15581
1622
5866
ConditionalQA
2338
285
804
Table 11: Dataset statistics.
14


Reviews:
[{"review_id": "fewAWvzmEcH", "reviewer": "Reviewer_yAth", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a model to tackle scenario-based question-answering to predict the answer to a question along with unsatisfied conditions for the given user scenario. The proposed model comprises 3 components, an entailment module (to identify the condition), a reasoning module (that decides whether or not the conditions have been satisfied) and a decoding module (that outputs the answer spans for free-form questions). The proposed model outperforms the baselines on several datasets. ", "summary_of_the_review": "I think the authors propose an interesting model for scenario-based question-answering and the proposed model outperforms the baselines on multiple datasets. However, the paper would benefit from a more through error analysis.", "strength_and_weaknesses": "Strengths:\n- The proposed approach is quite interesting.\n- The model outperforms the baselines on multiple datasets.\n\n\nWeaknesses:\n- It would be interesting to add a baseline where the entailment module is trained and evaluated separately (possibly through silver training data generated by generating the training labels using any existing entailment model.\n- It would be good to conduct a qualitative analysis (with examples) and a thorough error analysis showing how each component performs, how errors propagating from one module affects other modules, etc.\n- The write-up needs improvements since there are several typos (e.g., special special, \"regardless the status\" --> \"regardless of the status\", \"These information\" --> \"this information\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YvKs8-cnhJ", "reviewer": "Reviewer_nCC1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for answering questions about scenarios -- questions for which there isn’t a fixed answer but is varied depending on additional conditions that are unstated in the text. \n\nMethod: Given a scenario, a question, and a set of conditions (extracted from an input text) that are to be considered when answering the question, the method involves the following steps:\n\n1. Use an \"entailment\" module to create entailment related representations of the information in the scenario against each of the conditions to be considered. Use this representation to make decisions about which conditions are satisfied, contradicted, or of unknown status. \n2. The representations of the tokens in the conditions are also used to then generate the answer using a (standard) decoding module. \n\nModel: The entailment module is a pretrained NLI encoder. The decoder is a pre-trained BART generator. The “reasoner” is a set of transformer layers that consume the aggregate representations of each condition from the entailment model. All components are fine-tuned end-to-end with three losses the specifically test for ability to produce correct answer, predict whether all conditions are satisfied, and predict for each individual condition whether it is satisfied, implied, contradicted, or not discussed etc.\n\nEvaluations: The paper evaluates the proposed model on three datasets, one derived from the multi-NLI dataset and two Scenario QA datasets. \n\nKey Contributions: The main contribution is in putting together the system with components that are tied to the steps involved in the process and a demonstration of the utility of this method on real datasets. ", "summary_of_the_review": "I think the paper presents a clear empirical advance using a simple but useful technique. This work would be a strong baseline for future methods to build and compare against.", "strength_and_weaknesses": "Strengths\n\nThe paper addresses an understudied and challenging problem and provides a non-trivial empirical advance.\n\nThe proposed methods, while straightforward, are well motivated. \n\nThe writing is clear for the most part. See some presentation suggestions below.\n\nWeaknesses\n\nI wouldn’t say what follow are necessarily reasons to reject but are something that can be addressed to strengthen the contributions of this paper: \n\nThe description of the paper doesn’t provide a clear articulation of the key challenge or insight that is being addressed. It simply provides a description of the system and a demonstration that it provides empirical gains.  \n\nIt would be useful to know if the specific design choices (e.g. having a summary vector for each condition and the additional transformer layers) are indeed necessary. I appreciate the information conveyed in Figure 3. It provides some evidence already. Will it be possible to add an experiment where the reasoner is removed completely? Instead, train a T5 model to do the two tasks directly. Also, It will be useful to analyze one baseline model to Figure 3 (left side). You have a similar analysis for the ShARC dataset in Table 5. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OR42yPFDxFb", "reviewer": "Reviewer_nxRZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper looks at QA when questions are asked in a given scenario, and they can be answered only if providing the model information about the scenario. In addition, such questions require a high level of reasoning, and thus the model should also be able to infer how conditions interact with each other and find correct answers that possibly satisfy all the conditions. To deal with these challenges, the paper introduces a new dataset derived from an existing one (i.e., MultiNLI), and the T-reasoner that contains an entailment module to check if conditions are satisfied by the scenario, a decoding module to identify eligible answers, and finally a reasoning module. Results on the synthetic dataset proposed in this paper show that it outperforms other SOTA models.", "summary_of_the_review": "Overall the paper clearly introduces the problem, and the modeling part is well supported with evidence. \nThe weakest part is the dataset introduced in this work, which is supposed to contribute to the paper. The dataset is derived from an existing dataset using some heuristics, but no data quality assessments are provided in the paper. For example, I'd be curious to see what the overlap between questions and scenario/constraints is, as it seems possible that overlapping plays an essential role in solving this dataset. I'd also highlight what the differences with other datasets are and why a new dataset is required to model this problem. ", "strength_and_weaknesses": "Strength:\n- The paper models a relevant scenario and shows some interesting advancements in an important direction for QA systems.\n- T-reasoner is a good attempt to jointly model the full process.\n\nWeaknesses:\n- Limitations about the synthetic dataset are missing (see my comments below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster