Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces Scaffold-Maximizing Training, a process for learning better “explanations” for learned models by optimizing the effect the produced explanations have on downstream “simulatability.” There’s a lot of terminology and nuance to the arguments in this paper, but in reading through it, my understanding is as follows:

- Given a trained model T (off the shelf, black-box, e.g., an image/text classifier), we want to be able to generate good explanations of why it predicted a certain output.
- How do we know what a “good explanation” is? The field hasn’t yet converged to a single metric, but one plausible measure that seems to be well-adopted and well-motivated is the *simulability* of the trained model T (or teacher model); in other words, how well can we use the explanations to train a student model (lower capacity than the teacher T) that “simulates” how the teacher model T performs on held-out data.
- The idea is that a student that perfectly simulates the teacher T both in terms of prediction, *and in terms of explanations w/ its own explanation model* is a sign that the original explanations for T are good!

In general, the paper builds up this narrative cleanly, ultimately arriving at a bi-level optimization problem for learning the student model parameters (as well as the student model’s explanation model), followed by the teacher model’s explanation model as the outer loop. Experiments on image classification, text classification, and multilingual regression (translation quality estimation) show that explainers trained via SMaT lead to higher simulability scores and “plausibility” (human likeness of explanations) than methods that just statically condition on some external explanation process, or that don’t use explanations at all.
strengths_and_weaknesses: The strengths of this paper are in it’s clarity and originality; I strongly believe that each part of the paper is well-motivated, and seeks to build to an answer of the question — what makes better explanations? The experiments are comprehensive and the results while not overwhelmingly convincing, do make enough of a point that the SMaT protocol is effective.

However, the biggest weakness of the paper is the assumption driving the main optimization objective of the framework — the inner-level optimization not only optimizes the simulability score of the learned student model relative to the teacher, but the “explanation regularization loss” between the explanations produced under the student and the explanations produced under the teacher. It seems to me that this is an incredibly strong assumption, that doesn’t immediately fall out of the logic from the paper — if the goal is to produce better explanations via simulability, and optimize the explanations for better simulability, then this sort of “explanation matching” loss is not necessary?

EDIT: Rebuttal cleared this up!
questions: - Following from the last paragraph of weaknesses — what happens if you remove the explanation regularization from the objective for all of the static and learned models (basically ablate the student explainer model altogether). At this point, I don’t know if the better simulability is actually a result of the bi-level optimization procedure (learning better explanations) or just learning to regress explanations with a student model...

limitations: This paper is fairly clear about its limitations, acknowledging that the “explanation” methods used in this work are mostly saliency driven (attention maps). Like the paper suggests, I’d be interested in seeing applications of SMaT to other explainability approaches!
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a bi-level optimization procedure to directly optimize the simulability objective of explanability. In order for the optimization to work well, the paper proposes a new parametrized attention layer that is a weighted sum of the attention heads from Transformer models. This new layer (SMaT) works very well under this objective, showing high simulability (expected) and plausibility rating over other saliency-based methods.
strengths_and_weaknesses: Overall comment:

I'm generally impressed by this paper. Explainability work has seen a promising set of development in the last 3 years focusing specifically on the "why" and "so what" question of explainability -- why do we want explainability and what should explainability do (functionally). I'm excited to see that this paper takes a step in that direction focusing on pushing the boundary on these new properties of explanations. However, though generally well-written, I'm a bit concerned with a few choices the authors took. First of all, the explainers being differentiable. This significantly limits the type of explainers we can use, especially the game-theoretic ones. Attention is not explanation [1] because it does not faithfully represent the output of the model (and does not conform to the axioms of explainability [2]). The authors did use Integrated Gradients -- which mitigates this part of my criticism. Second concern, since the learned attention is the "winning" explainer model, I wonder if this is because **attention is the superior explanation method** or because **attention explainer is easier to optimize under the bi-level optimization objective the paper proposed** (aka., any method that can maximize this objective will be a good explanation method, regardless on whether they satisfy other criteria of explainability such as completeness). This is probably a very hard question to address or to design experiments to falsify,  but would like to hear authors' thoughts on this.

Strengths:
- The paper is clear and well-written
- The experiment is very thorough (including all three major ML application fields)
- Plausibility is investigated through either human annotations or user studies
- The intention to directly optimize 

Weaknesses:
- Some philosophical questions asked above.
- Could compare to some other interpretability baselines like shapley values (but this is not my main concern, and nor should we require every single paper to run experiment on every method that the reviewer happens to know)

[1] Jain, S., & Wallace, B. C. (2019). Attention is not explanation. arXiv preprint arXiv:1902.10186.
[2] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.
[3] Lei, T., Barzilay, R., & Jaakkola, T. (2016). Rationalizing neural predictions. arXiv preprint arXiv:1606.04155.
questions: - Can you comment on what is the student model? Is it only constrained in a way that it has less parameters than the teacher, but otherwise receive the same input and have the same architecture as the teacher? Can your work be viewed as a type of a model-compression/knowledge distillation? 
- There is a type of work that optimizes through reinforce to hard-select words that shows to have high precision at selecting words that are also selected from human rationale [3]. This also referred to as "hard attention" -- have you guys considered showing a baseline of this method for AUC on MovieReviews? Is it better than SMaT or worse?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors introduce SMaT (Scaffold-Maximizing Training),  a framework to directly optimize the explanations produced on a teacher model to improve the distillation of a student model simulating the teacher model (and creating an explainer for the student model too). The authors limit the architecture of the models to attention based transformers, and conduct experiments on three tasks (nlp, translation, image classification). Using SMaT resulted in students which more accurately simulate the teacher and produces more plausible explanations.
strengths_and_weaknesses: The paper presents an interesting approach to learning explainers in such a way as to more accurately distill a teacher to a student model. The authors demonstrate their results over three very different tasks. The authors point show when other explainability approaches (Gradient L2 for example), results in improved simulability (often when more data is made available). The paper is well written and the experiment is well designed. 
questions: - Would SMaT work in the case of a born-again network? In all the examples the student is limited to a smaller set of data.
limitations: The authors do not address this matter.
review_title: null

Meta-review: metareview: This paper proposed a novel “Scaffold-Maximizing Training” framework to optimize model explanations, by leveraging a student model simulating the teacher model using the teacher’s explanations. The idea of constructing explanations such that student models can better simulate the teacher is interesting and appears to be an original contribution.

All reviewers agree that this paper made a solid contribution to the explainability of ML models, the experiments are comprehensive and thorough, and the paper is well written. Overall the paper is well-rounded with clear justifications of what each algorithmic component does, and convincing empirical support (the proposed algorithm has been demonstrated to be effective on a variety of tasks). After viewing the authors’ feedback, there is a broad consensus on accepting the paper. 

confidence: Certain
Decision: Accept