Title: Scenario-based Question Answering with Interacting Contextual Properties
Authors: Haitian Sun
William W. Cohen
Ruslan Salakhutdinov
Venue: ICLR 2023
Keywords: Question Answering

Abstract: In the scenario-based Question Answering (QA) task, models are asked to find answers that are appropriate to the user scenarios associated with the question and identify information that is missing from the scenarios but is necessary for the answers to hold. Scenarios commonly include multiple properties of users, such as age, employment status, and income level for the question “How much can I claim from this benefit”. The properties relevant to a potential answer are given in a document, which will state conditions necessary for the answer to hold. Documents also may specify how conditions interact with each other, e.g. with text like “one of the conditions below must apply”. Although understanding the relationship between conditions is crucial for solving this challenging QA task, limited work has been done so far in modeling this. In this paper, we propose the T-Reasoner model, which solves this problem with three jointly learned modules: an entailment module which checks whether a condition has been satisfied by the scenario, a decoding module which locates eligible answers from documents, and a reasoning module which infers the relationship between conditions and performs a reasoning step to determine the logically consistent answers and identify missing conditions. T-Reasoner outperforms strong baselines on a synthetic scenario-based QA dataset and achieves a new state-of-the-art on two scenario-based QA benchmarks, outperforming the prior best models by 3-10 points.

TL;DR: We proposed a model for scenario-based QA which requires reasoning over multiple contextual properties in user scenarios to find answers that are consistent with the scenarios and to identify necessary information which is missing from the scenarios.

Review discussion:
Strengths
The paper addresses an understudied and challenging problem and provides a non-trivial empirical advance.
The proposed methods, while straightforward, are well motivated.
The writing is clear for the most part. See some presentation suggestions below.
Weaknesses
I wouldn’t say what follow are necessarily reasons to reject but are something that can be addressed to strengthen the contributions of this paper:
The description of the paper doesn’t provide a clear articulation of the key challenge or insight that is being addressed. It simply provides a description of the system and a demonstration that it provides empirical gains.
It would be useful to know if the specific design choices (e.g. having a summary vector for each condition and the additional transformer layers) are indeed necessary. I appreciate the information conveyed in Figure 3. It provides some evidence already. Will it be possible to add an experiment where the reasoner is removed completely?
Instead, train a T5 model to do the two tasks directly.
Also, It will be useful to analyze one baseline model to Figure 3 (left side). You have a similar analysis for the ShARC dataset in Table 5.
The paper is mostly clearly written. The main ideas and the method are understandable from the text with some effort.
The work is of reasonable quality. The idea, even if somewhat straightforward, is will suited for the problem and represents a non-trivial advance for the specific problem.
NO.
Thank you so much for your valuable feedback. Please check out the new version of the paper, as labels of tables and figures have shifted.
Results of your proposed experiments with a T5 model (without a reasoning module) were shown in Table 2 for CondNLI and Table 7 & 8 for ConditionalQA (named as T5 w/ FiD).The only difference with the original T5 model is that we adopt the Fusion-in-Decoder (FiD) trick from Izacard et al. [1] because inputs to models are usually long (> 2000 tokens) and won’t fit into the GPU memory. Experiments on ConditionalQA show that the performance of T5 w/ FiD is 12.8 points lower than T-Reasoner (44.4 vs. 57.2) in predicting answers, and is much worse in predicting unsatisfied conditions, i.e. 2.7 in F1 of conditions compared to 37.8 from T-Reasoner, 35.1 points absolute drop in performance.  We will clarify the presentation to make it clear that this experiment can also be viewed as an ablation.
We additionally experiment with the original T5 (without FiD) on the ShARC dataset. Inputs from ShARC are relatively short, so they can fit into GPU memory. Results are added to Table 5. The performance of T5 is more than 10 points lower than T-Reasoner in predicting both answers and missing conditions.
We update our "Introduction" section to clearly reflect the challenges of the proposed task posed on existing pretrained LMs.
Specifically, we explain that the plain T5 model does not perform well as inputs become more complex, i.e. as the number of conditions increases and conditions become more complex. We conducted additional experiments (results presented in the new Table 3) to support this claim – accuracy of T5 drops by 11.5 points as the number of conditions increase from 6 to 20.
Furthermore, we experiment with reducing the semantic complexity of conditions in the CondNLI dataset by replacing them with capital letters, e.g. “if you satisfy all A, B, C, then X”. The performance of T5 increases from 85.6 to 98.8. This shows that plain Transformer models could solve simple reasoning problems, but becomes less effective as the complexity of inputs increases.
Therefore, additional modeling is required, e.g. by adopting a dedicated reasoning module in this paper, to explicitly model the logical interaction between conditions.
[1] Gautier Izacard, Edouard Grave “Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering”
Please let us know whether our response has clarified any of your concerns. If not, please let us know and we can further revise the paper or clarify any concerns.
Hi reviewer nCC1,
Could you please let us know whether our response has clarified any of your concerns? If not, please let us know and we can further revise the paper or clarify any concerns.
Thanks,
Authors
Strengths:
- The proposed approach is quite interesting.
- The model outperforms the baselines on multiple datasets.
Weaknesses:
- It would be interesting to add a baseline where the entailment module is trained and evaluated separately (possibly through silver training data generated by generating the training labels using any existing entailment model.
- It would be good to conduct a qualitative analysis (with examples) and a thorough error analysis showing how each component performs, how errors propagating from one module affects other modules, etc.
- The write-up needs improvements since there are several typos (e.g., special special, "regardless the status" --> "regardless of the status", "These information" --> "this information".
The code will be available so the results should reproducible if the authors specify all hyper-parameters used.
The proposed model is quite novel and improves over the baselines for
NO.
Thanks for your valuable feedback. We conducted the experiment you suggested to further analyze the behavior of our model. We experimented with the synthetic CondNLI dataset because this is the only dataset where intermediate labels for entailment are available. To be specific, we add a classification layer on the output vector of the entailment module, and supervise it with the original entailment labels from MNLI. The loss for this intermediate supervision is added to the final loss. We also measure the classification accuracy as an additional metric. Results show that the accuracy of the intermediate entailment task is 94.5, compared to 94.7 from a T5 deliberately finetuned on MNLI. The final accuracy of answers and unsatisfied conditions (F1) is 95.2 and 92.1, compared to 95.0 and 91.3 from T-Reasoner (Table 2). It shows that even though intermediate supervision with gold labels are provided for the entailment module, the performance only improves marginally.
We also run the suggested experiment on the ConditionalQA dataset. We use the T5 model finetuned on MNLI as a pre-learned entailment model to label the entailment status of conditions. We use the silver data to provide intermediate supervision for the entailment module (same as the CondNLI experiment discussed above). The accuracy of answers (EM) drops from 57.2 to 42.1, even worse than plain T5 w/ Fid without any intermediate supervision (44.4). We suspect that intermediate supervision with silver labels may hurt the model's performance if they are noisy.
Please let us know whether our response has clarified any of your concerns. If not, please let us know and we can further revise the paper or clarify any concerns.
Hi reviewer yAth,
Could you please let us know whether our response has clarified any of your concerns? If not, please let us know and we can further revise the paper or clarify any concerns.
Thanks,
Authors
Strength:
- The paper models a relevant scenario and shows some interesting advancements in an important direction for QA systems.
- T-reasoner is a good attempt to jointly model the full process.
Weaknesses:
- Limitations about the synthetic dataset are missing (see my comments below)
The paper is mostly understandable, and the results support the idea of the paper. Code and data will be available upon acceptance.
NO.
Thank you for your suggestion.
We have included more discussions about the new dataset CondNLI in Appendix A in the revised version of the paper, including a more comprehensive discussion of the construction process of the synthetic data and the statistics of the dataset (e.g. distribution of logical operations and labels).
To summarize, we enforce: (1) an equal distribution of positive and negative examples, (2) a uniform distribution of logical operations between conditions, e.g. “any”, and “all”, and (3) a balanced set of answers with or without conditions.
To clarify your question on overlap between questions and scenario/constraints, conditions and scenarios in CondNLI (originally from MNLI) are much more complex than the ones used in the example in Table 1, which was only used for illustration purposes. A real example from MNLI, for example, contains a premise “Policies that provide insurance over the insured's entire life and the proceeds (face amount) are paid only upon death of the insured” and a hypothesis “Whole life policies are a type of life insurance that only cover the insured person until retirement from the workforce”. The average token-level Jaccard distance between premises and hypotheses in MNLI is only 27.2.
Even though token-level overlap exists, a model still needs to understand the semantic relationship between premises and hypotheses to predict their entailment status.
Very limited resources are available in studying the task of reasoning with logically interacting conditions. We expect the new dataset, CondNLI, to be a simple and easy-to-use benchmark. Compared to other datasets used in this paper, ShARC and ConditionalQA,  CondNLI is fully controllable in many ways, e.g. the size of the dataset, the distribution of labels, the number of conditions, types of logical relationship between conditions, etc. One may also consider obtaining intermediate labels for training, e.g. entailment status of each condition (an ablated experiment suggested by reviewer yAth), which are not available in ShARC and ConditionalQA. We will open-source our codes and encourage people to use it creatively for their own projects.
Please let us know whether our response has clarified any of your concerns. If not, please let us know and we can further revise the paper or clarify any concerns.
Hi reviewer nxRZ,
Could you please let us know whether our response has clarified any of your concerns? If not, please let us know and we can further revise the paper or clarify any concerns.
Thanks,
Authors

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of scenario-based question answering, where the system needs to output possible answers based on a user’s scenario, as well as unsatisfied conditions. The paper proposed a sophisticated 3-staged approach to tackle this problem, outperforming several SOTA approaches on multiple datasets.

All the reviewers give a weak acceptance of this paper, think the problem is new/interesting, and appreciate the modeling approach for this challenging task.

Some reviewers questioned the quality of the constructed dataset and missing ablation studies. The authors did a good job clarifying these issues and adding new results in the response phase. Hence, I recommend acceptance of the paper. 

justification_for_why_not_higher_score: As one reviewer said, "The main contribution is in putting together the system with components that are tied to the steps involved in the process." I think the topic of the paper would be limited to only a subset of the ICLR audience. 
justification_for_why_not_lower_score: I think it is worth accepting the paper given the quality of the work and how the authors addressed reviewers' comments during the rebuttal. 

Decision: Accept: poster