Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work aims at proposing a rationale extraction method that disentangles the rationale and non-rationale spans. Specifically, apart from predicting the label from the rationale span, the proposed method DARE adds another main objective that minimizes the mutual information (MI) between the representations of the rationale and non-rationale spans. Modified upon previous MI minimization work InfoNCE and CLUB, the authors propose a new method CLUB_NCE and use it in DARE. Experiments on two natural language datasets, BeerAdvocate and Legal Judgment Prediction show a performance advantage of DARE over the baselines.
strengths_and_weaknesses: The paper presents an overall novel and effective method for rationale extraction. The comparison between different MI minimization methods is aided by simulated experiments. The proposed method DARE is introduced rather clearly. Experiments on two natural language datasets show advantages of DARE over a few baseline methods.

The concerns that some readers might raise and complain include:
1- At line 88 and 111, a main argument for adding a regularizer that minimizes the MI between rationale and non-rationale representations, is that it can make the rationale more “comprehensive” and “rich” information. Can this argument be supported more intuitively with a discussion? (the connection between A and B’s disentanglement and A being “comprehensive”)

2- In Sec 2.3.2, CLUB_NCE is mainly CLUB with the approximation network p(y|x) replaced by the InfoNCE scoring function. While there are performance gains as shown in the experiments, the potential motivation and rationale behind this modification is not fully clear and can benefit from a further discussion.

3- The experiment presentation in Sec 3.2.2 needs improvements. The description and figure for the t-SNE experiment look confusing. How were the blue and red circles selected in a formal way? Can the distance between clusters be measured? The scale of 5(a) and 5(b) is different, making it harder for a comparison.

4- At line 295, the L1Out method was in previous tables but was not introduced until this stage. This should probably appear earlier in the text with more descriptions.

questions: Please see my concerns in the previous section. Also, RCNN is used as the selector and encoder architecture. Can pretrained transformer models be used instead? If so, would the re-embed operation in Sec 2.2.2 (line 101) lead to any issues?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to extraction rationales for text classification tasks. They propose to divide the sentence into two separate parts: rationale tokens and non-rationale tokens. They design a loss based on mutual information to encourage the embeddings of rationale tokens and embeddings of non-rationale tokens to be disentangled and expect more useful information can be squeezed into the rational part. Experiments and analysis are presented to support their design.
strengths_and_weaknesses: Strengths
- The writing is clear and easy to understand.
- The design is simple and seems to be effective.
- Comprehensive analysis and studies for the proposed model.

Weaknesses
- No loss to ensure that the information preserved by the non-rational embeddings are meaningful (see more details in the "Questions" section below). 
questions: Suggestions
- I feel like the reconstruction loss is needed. It takes both rationale embeddings ($v_{z_1}$ in Figure 2) and non-rationale embeddings ($v_{z_2}$ in Figure 2) as the input with the goal to reconstruct the original input. This loss encourages the model to encode as much information to $v_{z_1}$ and $v_{z_2}$ as possible while your MI loss makes $v_{z_1}$ and $v_{z_2}$ disentangled. Otherwise, without this reconstruction loss, the encoder can just learns to encode $v_{z_2}$ as random vectors. Those random $v_{z_2}$ can still have low mutual information with $v_{z_1}$, which is obviously not the expected situation. In this case $v_{z_2}$ encodes nothing and your approache will reduce the common selector + predictor approach. Although the current design somehow works and the above-mentioned case does not happen that much, I still think the reconstruction loss is needed.
- For Table 1, some of the improvements are small. Adding t-test results may help.

Questions:
- For Table 1 and Table 2, how many runs of experiments for the mean and std? 
- What would happen if you also add LM regularizer just like the original InfoCal?
limitations: The authors adequately addressed the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on the problem of rationale extraction for classification tasks. The proposed method addresses a hypothesized limitation in prior work, namely failing to fully exploit the original input to represent selected rationale tokens. The proposed method jointly optimizes the classification objective and a new objective which explicitly minimizes the mutual information between the representations of the selected rationales and the ones from the rest of the input to extract more information out of it.  Results on two rationale extraction tasks show improvements over existing rationale extraction methods. The developed mutual information objective is shown to be better than alternative ones on simulated experiments. 

strengths_and_weaknesses: **Strengths** 
- The narrative/structure of the paper was easy to follow and presentation was generally good. 
- Proposes an approach for rational extraction which combines ideas from existing selection methods and objectives to minimize mutual information between selected rationales and other parts of the input. 
- Presents results that show classification moderate improvements over previous rationale extraction methods on two  classification tasks.  

**Weaknesses** 
- Even though writing was fair in most places, some of the model design and evaluation choices lacked clarity and proper motivation/justification.  
- The proposed objective to minimize mutual information appears to be incremental and the comparison between alternatives incomplete (e.g. lack of comparison with plain InfoNCE on extraction task). 
- Several design choices are not well justified and the description lacks clarity in a few places (e.g. shared encoder/using trained InfoNCE in section 2.3, coherence regularizer, objective coefficients section 2.4). There are no ablation studies that demonstrate the source of improvement or contribution from each individual objective to the final performance. 
- The evaluation is lacking details for understanding the results and the logic behind the design choices. Also, the quality of the extracted rationales for the judgment dataset is based only on a handful of examples and a t-sne visualization. 
- The scope of the paper in the present form is limited to findings on two datasets and exploration beyond them would be required to be of interest to a broader audience. 
questions: The point that existing methods ignore information from the non-selected tokens in the abstract is not entirely clear at first. Since bidirectional representations are typically used to represent the tokens this statement creates confusion. It would be helpful to clarify early on that the disentanglement goal essentially aims to encourage dependence among tokens of the same type and independence for tokens of different types  (e.g. rationale/non-rationale).

Regarding the statement in lines 201-204: is the score reported with HardKuma using the paper's implementation or it is reported from prior work (line 189)? To understand the differences it is also important to know what is the number of parameters for each of them.

The evaluation is missing details on how the model and training objective hyper-parameter optimization was performed exactly. What values were examined for each hyper-parameter and what is their effect? Also, it is unclear how some design choices were made (shared encoder, MI minimization that depends on trained InfoNCE).   

In Table 2, was there a reason why an experiment using plain InfoNCE with DARE was not included? It'd be useful to report performance with this objective for completeness and for clearly demonstrating the added value of the selected one. 

The evaluation does not provide sufficient evidence about the quality of the extracted rationales for the second dataset since it is based on a handful of examples and t-sne. A systematic human evaluation study would be required to measure quality.  

In Eq. 6, could you elaborate on how is the coherence encouraged by this regularizer? Any examples of what happens if this is not enforced would be most helpful. 
limitations: Yes. There is a section in the Appendix. Limitations are not sufficiently investigated and discussed (see above).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies a method for rationale extraction for natural language processing (NLP), focusing on text classification tasks. The overall architecture involves a selector to select rationale tokens from an input text, a predictor to consume the selected rationale tokens to obtain a representation for predictions, and a guider to obtain a representation for the non-rationale tokens that will be regularized to minimize the mutual information (MI) with the rationale token-based representation. Compared to previous work, this paper argues that it is helpful to use information for non-rationale tokens to improve the selection of rationale tokens. In addition, to minimize mutual information between two representations, the paper presents a new approach (CLUB_NCE) that combines two existing methods of InfoNCE and CLUB. Experiments are conducted over two text classification datasets to demonstrate better prediction performance and more reasonable rationale for the method. The paper also provides a simulated experiments to compare the proposed MI minimization method with previous baselines and a visualization to demonstrate the disentangled representations for rationale and non-rationale tokens.
strengths_and_weaknesses: **Strength

+The paper considers an important problem of rationale extraction to improve interpretability for NLP models.

+The idea of using non-rationale information to improve rationale extraction is interesting and the introduced MI minimization method demonstrates good empirical results.

+The proposed method achieves competitive performance on two datasets that can provide useful baselines for future research.

**Weakness

-While interesting, the guider component (with non-rationale tokens) and the CLUB_NCE for MI minimization seem to be incremental extension of existing methods for rationale extraction and MI estimation. The method mostly involves modification and combination of closely related techniques (e.g., InfoNCE and CLUB).

-As indicated in the Appendix, the model only uses word2vec embeddings to encode texts and does not consider current pre-trained language models (e.g., BERT) that has significantly advanced different NLP tasks. The key question is whether the proposed method can still maintain its claimed benefits when applied with pre-trained language models. In addition, this important implementation information with word2vec and Bi-GRU should be presented in the main paper to make it easier to understand the methods.

-Although Section 3.3 provides empirical experiments to compare different MI estimation methods, it is helpful to better motivate for the designed MI method CLUB_NCE. In particular, Section 2.3.2 mainly describes the implementation of the method. Without a discussion on limitations of previous methods and motivation to address the limitations, it is difficult to understand the benefits of the designed method.

-The performance of the proposed method is very close and sometime worse than the closely related baseline InfoCal(HK), making the effectiveness of the method less convincing. It might be helpful to provide significant scores to strengthen the results.

-Due to the coherence regularization, the selected rationale tokens tend to form consecutive sequences, causing remaining non-selected tokens to involve nonconsecutive words that might lead to unnatural texts. As such, the learned representations for rationale and non-rationale tokens might be quite different, making the MI minimization less reasonable. As such, is it better to obtain non-rationale representation by subtracting rationale representation from the representation for the entire input text to avoid unnatural inputs? It might be helpful to explore this approach to better justify the design choice of the model in this work. 

questions: Please consider the weakness presented above.
limitations: The paper includes a discussion on broader impact and limitations in the appendix. However, the limitation discussion is limited to only selection of the l_r parameter. It might be helpful to further consider the limitations as described in the weakness section.
review_title: null

Meta-review: metareview: This paper presents an improved method for improving rationale extraction and model explainability. The key insight is to disentangle rationale from non-rationale tokens in the input. Reviewers raised some fairly minor questions regarding the notion of disentanglement, the proposed loss, and experiments (e.g, comparison with related models). The authors responded to the queries and included more discussion and results. The paper is solid, the results look good, and the added discussion will clarify the raised concerns. 
confidence: Certain
Decision: Accept