Venue: NeurIPS 2022 Workshop TSRML

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary: In this paper, a markup-and-mask rationalization method is proposed for open-book question answering. For student model, to answer a question, an augmented passage with free text markup are fed into span selector to identify rationales and only those selected rationales are utilized in reading comprehension. The main difference between the teacher model and the distilled student model is that in the student model rationale acts as a bottleneck between the passage and the answer. Prompt learning is incorporated to generate silver annotations.
strengths: - The description of annotation generation is very detailed, also utilizing the combination of  in-context learning and end-task supervision to avoid the cost of gathering human annotations sounds like an interesting idea.

- Extensive experiments are provided and results are impressive, which show the usefulness of the framework. Also, current limitations are discussed in detail.

weaknesses: - The paper is not easy to follow because there is no smooth connection between different chunks. It’s better to write a pipeline figure to show how the whole framework works.

- One weakness of the paper is that it doesn't provide a math formula for their selective rationalization, which includes the usual sparsity constraint and coherent constraint. It is better to show how they optimize the objective function with hyper-parameters related to rationalization. 

overall_recommendation: This paper delivered an interesting idea with extensive experiments, but the writing flow is not smooth, sometimes hard to follow.
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary: The major contributions of the work are two-fold. Firstly, the work proposes a new type of extractive rationales, markup-and-mask, which are grounded by the original context and yet can be read alone. Secondly, the work proposes to use a prompted language model for obtaining silver training data, which is later used to train a student model. The student model is designed as a pipeline, making sure the answer is derived from the rationales. Experiments demonstrate the effectiveness of the proposed method and a list of comprehensive analyses also validates the source of improvement.
strengths: 1. The proposed design of markup-and-mask rationales is sound and novel. It's also proved to be useful as it increases the fraction of rationalizable answers.
2. The work adopts a prompted LM for obtaining silver training data for fine-tuning a student model, which is more efficient compared to human annotation. It also adopts some validations to ensure the quality of the data. The design of the student model is also effective in making the student model faithful to its generated rationales.
3. The experiments are comprehensive, providing different perspectives to understand how different parts of the whole system contribute to the final performance. 
weaknesses: 1. The work does not verify the training data for decontextualization as it does for rationale generation. Yet, the experiment shows that the student model fine-tuned in this way performs better than the teacher model. It would be helpful if the authors can have a discussion in the paper about what new supervision signals are brought during the process of the distillation.
2. One baseline that should be considered is to directly train the student model for generating rationales using the heuristic-based extracted rationales.


overall_recommendation: Overall I recommend accepting the paper given the novelty in terms of a new type of extractive rationales and an effective knowledge distillation paradigm. The results are all convincing given the thorough experiments.
review_title: A solid work proposing a new type of extractive rationales and an effective knowledge distillation paradigm

Meta-review: Following the unanimous recommendations from reviewers, the submission is accepted.
Decision: Accept