Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper looks at QA when questions are asked in a given scenario, and they can be answered only if providing the model information about the scenario. In addition, such questions require a high level of reasoning, and thus the model should also be able to infer how conditions interact with each other and find correct answers that possibly satisfy all the conditions. To deal with these challenges, the paper introduces a new dataset derived from an existing one (i.e., MultiNLI), and the T-reasoner that contains an entailment module to check if conditions are satisfied by the scenario, a decoding module to identify eligible answers, and finally a reasoning module. Results on the synthetic dataset proposed in this paper show that it outperforms other SOTA models.
strength_and_weaknesses: Strength:
- The paper models a relevant scenario and shows some interesting advancements in an important direction for QA systems.
- T-reasoner is a good attempt to jointly model the full process.

Weaknesses:
- Limitations about the synthetic dataset are missing (see my comments below)
clarity,_quality,_novelty_and_reproducibility: The paper is mostly understandable, and the results support the idea of the paper. Code and data will be available upon acceptance.
summary_of_the_review: Overall the paper clearly introduces the problem, and the modeling part is well supported with evidence. 
The weakest part is the dataset introduced in this work, which is supposed to contribute to the paper. The dataset is derived from an existing dataset using some heuristics, but no data quality assessments are provided in the paper. For example, I'd be curious to see what the overlap between questions and scenario/constraints is, as it seems possible that overlapping plays an essential role in solving this dataset. I'd also highlight what the differences with other datasets are and why a new dataset is required to model this problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a model to tackle scenario-based question-answering to predict the answer to a question along with unsatisfied conditions for the given user scenario. The proposed model comprises 3 components, an entailment module (to identify the condition), a reasoning module (that decides whether or not the conditions have been satisfied) and a decoding module (that outputs the answer spans for free-form questions). The proposed model outperforms the baselines on several datasets. 
strength_and_weaknesses: Strengths:
- The proposed approach is quite interesting.
- The model outperforms the baselines on multiple datasets.


Weaknesses:
- It would be interesting to add a baseline where the entailment module is trained and evaluated separately (possibly through silver training data generated by generating the training labels using any existing entailment model.
- It would be good to conduct a qualitative analysis (with examples) and a thorough error analysis showing how each component performs, how errors propagating from one module affects other modules, etc.
- The write-up needs improvements since there are several typos (e.g., special special, "regardless the status" --> "regardless of the status", "These information" --> "this information".
clarity,_quality,_novelty_and_reproducibility: The code will be available so the results should reproducible if the authors specify all hyper-parameters used.
The proposed model is quite novel and improves over the baselines for 
summary_of_the_review: I think the authors propose an interesting model for scenario-based question-answering and the proposed model outperforms the baselines on multiple datasets. However, the paper would benefit from a more through error analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for answering questions about scenarios -- questions for which there isn’t a fixed answer but is varied depending on additional conditions that are unstated in the text. 

Method: Given a scenario, a question, and a set of conditions (extracted from an input text) that are to be considered when answering the question, the method involves the following steps:

1. Use an "entailment" module to create entailment related representations of the information in the scenario against each of the conditions to be considered. Use this representation to make decisions about which conditions are satisfied, contradicted, or of unknown status. 
2. The representations of the tokens in the conditions are also used to then generate the answer using a (standard) decoding module. 

Model: The entailment module is a pretrained NLI encoder. The decoder is a pre-trained BART generator. The “reasoner” is a set of transformer layers that consume the aggregate representations of each condition from the entailment model. All components are fine-tuned end-to-end with three losses the specifically test for ability to produce correct answer, predict whether all conditions are satisfied, and predict for each individual condition whether it is satisfied, implied, contradicted, or not discussed etc.

Evaluations: The paper evaluates the proposed model on three datasets, one derived from the multi-NLI dataset and two Scenario QA datasets. 

Key Contributions: The main contribution is in putting together the system with components that are tied to the steps involved in the process and a demonstration of the utility of this method on real datasets. 
strength_and_weaknesses: Strengths

The paper addresses an understudied and challenging problem and provides a non-trivial empirical advance.

The proposed methods, while straightforward, are well motivated. 

The writing is clear for the most part. See some presentation suggestions below.

Weaknesses

I wouldn’t say what follow are necessarily reasons to reject but are something that can be addressed to strengthen the contributions of this paper: 

The description of the paper doesn’t provide a clear articulation of the key challenge or insight that is being addressed. It simply provides a description of the system and a demonstration that it provides empirical gains.  

It would be useful to know if the specific design choices (e.g. having a summary vector for each condition and the additional transformer layers) are indeed necessary. I appreciate the information conveyed in Figure 3. It provides some evidence already. Will it be possible to add an experiment where the reasoner is removed completely? Instead, train a T5 model to do the two tasks directly. Also, It will be useful to analyze one baseline model to Figure 3 (left side). You have a similar analysis for the ShARC dataset in Table 5. 


clarity,_quality,_novelty_and_reproducibility: The paper is mostly clearly written. The main ideas and the method are understandable from the text with some effort. 

The work is of reasonable quality. The idea, even if somewhat straightforward, is will suited for the problem and represents a non-trivial advance for the specific problem.

summary_of_the_review: I think the paper presents a clear empirical advance using a simple but useful technique. This work would be a strong baseline for future methods to build and compare against.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of scenario-based question answering, where the system needs to output possible answers based on a user’s scenario, as well as unsatisfied conditions. The paper proposed a sophisticated 3-staged approach to tackle this problem, outperforming several SOTA approaches on multiple datasets.

All the reviewers give a weak acceptance of this paper, think the problem is new/interesting, and appreciate the modeling approach for this challenging task.

Some reviewers questioned the quality of the constructed dataset and missing ablation studies. The authors did a good job clarifying these issues and adding new results in the response phase. Hence, I recommend acceptance of the paper. 

justification_for_why_not_higher_score: As one reviewer said, "The main contribution is in putting together the system with components that are tied to the steps involved in the process." I think the topic of the paper would be limited to only a subset of the ICLR audience. 
justification_for_why_not_lower_score: I think it is worth accepting the paper given the quality of the work and how the authors addressed reviewers' comments during the rebuttal.
Decision: Accept: poster