Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an architecture which jointly learns to classify as well as provide an explanation behind the classification. The explanation is generated from a separate part of the model which doesn’t affect the model performance much as the classification performance is achieved by a backbone network. The explanation generation part takes inputs from the intermediate layers of the backbone network which helps to produce better and more meaningful explanations. The results are promising on experiments shown on MNIST, FMNIST, CIFAR10 and QuickDraw datasets.
main_review: Positives:

+ The overall idea of integrating explanations while learning to classify is important and timely, especially considering the spate of post-hoc explanation methods available today.
+ The use of an exclusive backbone network for classification helps maintain good classification performance.
+ Post-hoc explanation generation techniques can be applied in the framework.
+ The results show that the method performs better in terms of classification performance and explanation generation as well compared to other baseline methods.

Concerns:

- One of the primary concerns is whether the method is applicable to complex large-scale datasets, on which post-hoc attribution methods show their results (e.g. ImageNet). The paper does not show results on large-scale datasets, limiting the understanding of its applicability. As much as the proposed idea is interesting and new, to a certain extent, it attempts to improve over a couple of earlier methods in this direction such as SENN (Ref [4] in the paper). Showing it works on large-scale datasets would be one important improvement to show, especially considering the pieces used in the method are known before.

- The contribution of every attribute is given by the coefficients w which are given by the matrix W (considering all the classes) and trained as a simple fully connected layer. Once the model is trained, these contributions are fixed and are used as it is during inference. This is different for SENN, as the contributions are given by a relevance function there (realized by a neural network) and hence produces different contributions for different inputs. How do you view this difference? Do you think the same contributions for every input across all data points is justified?

- The interpreter selects outputs from one or more intermediate layers from the backbone network specified by the function f. The layers chosen for different architectures are shown in Fig.1 and Fig.2 of supplementary section. Are these choices arbitrary? If not, how were the layers chosen? I’m curious to know if this choice affects the quality of the interpretation. Also, if this is non-trivial and empirical, this makes the framework difficult to scale to newer datasets/models.

- In L56, it is said that ‘FLINT can be specialized to post-hoc interpretability if a pre-trained deep neural network is available.’ As far as I understood, the post-hoc interpretability is generated from the backbone network given by f, which is part of the whole network. Then, what is meant by availability of a pre-trained deep neural network here, and how is this achieved?

- The final loss function is an aggregation of many different loss functions which makes this network highly dynamical in nature. One needs to take proper care while selecting the coefficients and the number of epochs where every loss function was introduced. This was pointed out in Algorithm 1, but how were these epoch numbers arrived at? How does one choose these on a dataset?

- Table 1 compares the accuracy numbers with SENN and the proposed method uses a similar architecture as SENN for MNIST. However, for CIFAR10, SENN used smaller architectures for their results, which were not considered why. Was there any reason for this? Would this be fair comparison? I’m convinced with the explanations provided for the training time differences with SENN though.

- In L106-108, the paper talks about other methods which use a known dictionary of concepts (such as concept bottleneck models). Was there a particular reason not to compare with such methods? While it may be understandable, it may be nice for the paper to explicitly state why the baselines used in the experiments were chosen (and other such models were not).

limitations_and_societal_impact: Not in the main paper, as suggested in paper submission guidelines (provided in the supplementary)
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces FLINT, a framework that jointly trains a classifier with an “interpreter”, an interpretable version of the model. The interpreter is a linear classifier over a set of attributes (each associated with a “concept”), which are derived from hidden states of the neural network. The interpreter is trained with three objectives: (1) matching the output of the original classifier, (2) a sparsity regularizer, and (3) an anto-encoding objective to reconstruct the input. Each attribute in the interpreter can be visualized by finding input that maximizes the attribute (with some regularization), which allows the interpreter to produce global interpretation for a class and local interpretation for an example. The method is tested on four image classification tasks. It has higher accuracy than two previous interpretable models (SENN and PrototypeDNN) and higher fidelity than two post hoc interpretation methods (LIME and VIBI). According to a human survey, the visualizations are human interpretable in most cases.
main_review: Strength:
The framework is novel, intuitive, and flexible. It allows us to provide an interpretable version of any neural classifier. The experiments are extensive, and the results seem good. The model can retain high accuracy while producing high-fidelity interpretation. 

Weakness:
My biggest concern is that the framework introduces extra complexity to model training. For example, the paper acknowledges that optimizing the full objective is hard, so they need a schedule to gradually increase the complexity of the objective (Section 3.1). This schedule may need to be changed for a new model. The framework also introduces several hyperparameters: the weight of different objectives and the number of attributes. These hyperparameters need to be tuned separately for different tasks/datasets. 

In summary, I am leaning towards accepting the paper. However, I am not very familiar with the latest interpretation methods and how to evaluate them, so I might miss something.
limitations_and_societal_impact: I do not have any suggestions for this part.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes FLINT, a new learning schema in which a neural net and a gray-box model are learned jointly by minimizing a composite loss: a loss on the labels + a set of losses controlling the concepts acquired by the gray-box models +
a distillation loss that encourages the gray-box models to behave similarly to the neural net.  The concepts in the gray-box model are built from intermediate neurons of the neural net to encourage faithfulness.  The authors also describe a visualization pipeline and a post-hoc explanation protocol.
main_review: The paper is generally well written;  all concepts are explained clearly.  The text is structured just fine.  I do not have any major gripes with the presentation itself.

One issue is that the message is unfocused: it is not exactly clear what the authors want to say with this paper.  Apparently it is only about "going beyond SENNs", but I think that there are some interesting and deeper issues implicit in this work that the authors do not address:

    1.  There is an unstated benefit of gray-box models like SENNs that the authors overlook:  SENNs are not only interpretable by design but also *faithful* by design.  This is no longer the case in FLINT: there is no guarantee that the g network is 100% faithful to the f network.  This also explains why SENNs underperform in Table 1: they trade off performance for interpretability AND faithfulness.  This actually highlights a limitation of FLINT and that this should be explicitly discussed in the paper.

    2.  This also means that, implicitly at lesat, the authors suggest that it is fine to give up on faithfulness -- that faithfulenss is not always fundamental.  I don't think that it is reasonable to do so in high-stakes applications.  In order to defend this point, the authors neglect to clarify which applications their method is best suited for.

The core idea -- namely that of training two twin networks based on (roughly speaking) the same base intermediate layers but with different aims -- is sensible.  Re-using the intermediate layer to tie the two models is an interesting choice and is reminiscent of multi-task learning solutions.  Again, this introduces a performance-faithfulness trade-off that should be addressed.  Some modeling choices (like combining the L1 norm with entropy) are quite empirical, but on the bright side they seem to perform well in practice.

The experimental validation is a bit limited, but the choice of competitors and research questions is reasonable.  The results seem to favor the proposed method.


As for the score, I am a bit on the fence.  All in all, the contribution is sensible and the experiments are reasonable, but the paper is somewhat lacking in terms of focus, significance and depth.
limitations_and_societal_impact: The limitations are barely discussed in the introduction; there is a section on limitations and negative impact in the supplementary.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes FLINT a Framework to Learn with Interpretation to solve Supervised Learning with Interpretation task.
FLINT involves training 3 models on a preditor model an interpreter and a decoder. This framework provides both global and local interpretations.

The interpreter network takes as an input the output of different hidden layers from the predictor network then computes an attribute dictionary of functions phi g(x)= softmax(Wphi(x)). Different phi corresponds to different concepts  phi_j(x)= Psi_j (f_I(x)) where Psi_j is shallow network and f_I(x) is the output of hidden state I.  So The model parameters of g depend on the parameters of Psi and Weights. Given interpreter with parameter and input x the contribution of phi_j = phi_j(x)*w_j,y_hat/max|ph_i*w_i_yhat| where y_hat the class output.

Global Interpretation is a set of class-attribute pairs (c, ph_j) such that their global relevance is greater than some threshold.
Local interpretation for a sample x provided is the set of attribute functions ph_j  with a local relevance score greater than some threshold.

FLINT has multiple loss functions, first to ensure that the interpreter output is similar to that of the model output cross-entropy is used. An additional loss is used to ensure diversity of ph_j and ensure input fidelity the input is reconstructed from phi using a decoder network which gives us the final loss.

FlINT was evaluated on 4 different image datasets, it was compared with other self explain networks SENN and PrototypeDNN to terms of accuracy and saliency methods LIME and VIBI in terms of fidelity.
main_review: Strength:

- Although self-explaining networks have been previously proposed FLINT framework itself is original and novel.
- The paper is technically sound and the claims are well supported.
- The paper organized and is well written.
- The relation to previous work is clearly outlined.
- The empirical results are quite impressive epically that FLINT_g  produces accuracy similar to that of FLINT_f.
- The paper included a subject evaluation showing that the concepts learned by FLINT are understandable by humans.
- The proposed method is significant, the constraints used to impose Fidelity to Output, Conciseness and Diversity of Interpretations, and Fidelity to Input can be adapted to many methods.



Weakness:

- It is unclear how one can choose the hidden layer to give input to the interpreter.
- Comparison was done on very simple datasets, it is unclear if this will generalize to larger more complicated datasets.
- Baseline models used have very low accuracy compared to state-of-the-art accuracy.
- Empirical evaluation is somewhat limited.

Questions and Comments:

- Accuracy reported on CIFAR10 as an example has baseline accuracy 84.7 and FLINT was able to maintain this however state of the art accuracy is around 98% if that model was used as a baseline or other high performing network will FLINT be able to maintain such accuracy?
- The paper only used very simple datasets which are generally easy to train, how will this approach work on larger datasets like image net will the proposed optimization still work.
- It would be interesting to see how  FLINT compares to other self-explaining networks such as INVASE [1] or FRESH[2].



[1] Yoon, Jinsung, James Jordon, and Mihaela van der Schaar. "INVASE: Instance-wise variable selection using neural networks." International Conference on Learning Representations. 2018.
[2] Jain, Sarthak, et al. "Learning to faithfully rationalize by construction." arXiv preprint arXiv:2005.00115 (2020).


Overall, I believe the proposed method is great and I really enjoyed reviewing the paper. 
limitations_and_societal_impact: Yes, in the supplementary.
review_title: null

Meta-review: All the reviewers agree that the paper has some nice and interesting ideas and it is indeed quite promising that the proposed method gives minimal accuracy loss while providing interpretability.

Having said that, a majority of them felt that a more thorough analysis of the method (e.g., ablation studies) and evaluation on at least one dataset on the scale of ImageNet is necessary to properly evaluate the utility of the proposed method. This is particularly so because of the large amount of existing work in this area and the limited novelty of the proposed method. I am sharing some of the key concerns and points that arose during the discussion period.

- As a problem on which there is earlier work (SENN, PrototypeDNN), restricting to small-scale datasets seemed limiting. The new results on CIFAR100 were nice to see, but results on one larger complex dataset (TinyImagenet, CUB, ImageNet) would have made it better to see the usefulness of the work (considering the method by itself has no new components, it's an aggregation of existing pieces).

- The paper seems to be missing analysis (ablation studies in the case of this work) where the results are shown with choice of different layers, or choice of different epochs where loss functions are introduced. These are important decisions in the framework, and how they impact the final result would have been nice to see (even if some of the choices did not yield strong results, and the method worked only for certain choices).

[Update] The authors' response dated 2nd September indeed satisfactorily addresses the two main issues of (i) evaluation on a larger dataset and (ii) analysis of the effect of different choices in the proposed framework. Since the reviewer/AC discussion had concluded by 30th August, this last authors' response was not noticed at the time of the original metareview. Given that all the reviewers agreed that the paper has interesting results, and that the two main criticisms have been satisfactorily addressed, I recommend acceptance of the paper.
Decision: Accept (Poster)