Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents the SchemaNet, an inference paradigm based on the cognitive concept of schema for inducing explainability of  trained image classification Vision Transformers. It first builds a codebook of by k-mean clustering running on the collection of visual tokens extracted from a probe dataset. With the codebook, it builds the ingredient relation graph (IR-Graph) at the instance level using a Feat2Graph module, and at the category level. It then exploits graph convolution network for computing the graph similarity of instance and category IR-Graphs for explainable classification inference. In experiments, the proposed method is tested using DEIT on CIFAR10/100 and Caltech-101 with promising results obtained. 
strength_and_weaknesses: Strength:

1) The proposed method leverages the ensemble-level information (the codebook and the category IR-Graphs), as a post-hoc processing step to address explainable image classification. 
2) The proposed method is able to explain the image classification results in terms of graph visualizations. 
3) The proposed method obtains promising results on three datasets (CIFAR10/100 and Caltech 101). 

Weaknesses:
 
1) This paper aims to address explainable image classification. There are a few aspects that could be addressed to consolidate the proposed method.  It will be better to show the inference results of misclassified examples to check if the inferred IR-Graphs can indeed provide insights of why they are misclassified.  It will also be interesting to compare with the recent methods explaining ViT models (e.g., Chefer  et al, Transformer interpretability beyond attention visualization, CVPR21). It will also be important to see if the proposed inference method is potential more robust to adversarial attacks as a sanity-check of the targeted explainability.  
  
2) Since the foundation of the proposed method is built on the codebook computed via k-mean clustering running on the collection of visual tokens extracted by a feature backbone (DEIT) from a probe dataset, it will be better to show the learned mean image patches to see if they make sense visually.  Individual image patches are shown in the IR-Atlas (e.g. Fig.5). How are they selected to represent each codebook index?
It will also lead to the potential scalability concerns.  The proposed method is tested on relatively small datasets, rather than the ImageNet-1K. It will be interesting to see ImageNet results if possible.  How large will the codebook need to be, as well as the training and inference cost  of the SchemaNet?

3) The proposed graph similarity computation via GCN is not clearly explained. Vertex embeddings are used and trained by GCNs. It seems that the backbone only provide initialization of the IR-Graph structures. The resulting explainablity may thus have a gap with the backbone. Will it be possible to use the visual tokens as features for the vertex in the IR-Graphs?  
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well written, and addresses an interesting and important problem. The proposed method makes sense intuitively based on the inspiration of the cognitive concept schema. Source code is provided but not checked by the reviewer. 
summary_of_the_review: Overall, the proposed method is an interesting and promising approach. The reviewer would like to see the rebuttal on the aforementioned weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a new network architecture, called a SchemaNet, that turns an image classification problem into a graph matching problem. In particular, a SchemaNet consists of: (1) a pre-trained visual transformer backbone, that extracts image features from an input image; (2) a Feat2Graph module, that turns deep image features of an image into an instance IR-Graph, where vertices represent concepts and edges represent relations between concepts; (3) a set of IR-Graphs (called "IR-Atlas"), each of which represents prototypical concepts and their relations for each class; and (4) a graph neural network that computes the similarity between the instance IR-Graph and each of the class IR-Graphs in the IR-Atlas. The final prediction is based on the latter similarity -- the class whose IR-Graph has the highest similarity with the instance IR-Graph is used as the predicted class of the input. The authors compared their SchemaNet with a number of baseline models, including the base transformer, bag-of-visual-words over deep image features, and BagNet, on CIFAR-10, CIFAR-100, and Caltech-101, and found that their SchemaNet achieved competitive classification accuracy as the baseline models.
strength_and_weaknesses: Strengths:

- Novelty: The proposed SchemaNet is novel -- the idea of turning image classification into a graph matching problem has not been explored before.
- Clarity: The paper is in general clearly written.

Weaknesses:

- Motivation: I suppose that the motivation for turning image classification into a graph matching problem is to build a model with a more interpretable reasoning process. However, the paper did not discuss how interpretable the visual words and the class IR-Graphs are. In particular, what are the meanings of the visual words? Do the learned class IR-Graphs make sense? The paper would be significantly better, if the authors can include interpretations of some class IR-Graphs. Without such interpretations, it would be difficult to see why we want to turn an image classification problem into a graph matching problem.
- A related issue: Since the authors used a graph convolutional neural network (GCN) to match an instance IR-Graph with each of the class IR-Graphs, and a GCN is not interpretable in general, how would the authors explain why an instance IR-Graph (representing a particular input image) is similar to a class IR-Graph? Again, without interpretability, I fail to see why we want to turn an image classification problem into a graph matching problem.
clarity,_quality,_novelty_and_reproducibility: This work is clearly written, is of a reasonable quality, and is novel. I believe that the work has no issue with reproducibility.
summary_of_the_review: Based on the strengths and the weaknesses discussed above, I believe that this work has value, but needs improvement in explaining how to interpret the learned class IR-Graphs and how an instance IR-Graph is similar to a particular class IR-Graph. Without such interpretability, it is difficult to see the motivation of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies interpretability for image classification. The authors propose a schema-based architecture (SchemaNet) based on vision transformers, which first extract features from pre-trained backbone and form a so-called ingredient relation graph (IR-Graph) with a feature to graph (Feat2Graph) module. The IR-Graph is matched with a set of imaginations (so-called IR-Atlas) with a matching network for making predictions. To test the SchemaNet, the authors conduct experiments on CIFAR10/100, and Caltech101, in comparison to the baseline without SchemaNet, bag-of-visual-words (BoVM) representation, BagNet, and show that SchemaNet performs better or on par with the baseline. Further, the authors also show some visualization of the learned IR-Graph and IR-Atlas.
strength_and_weaknesses: + The proposed method of learning graphs to make the deep neural network more interpretable is interesting and novel. The idea makes a difference as compared to existing works in XAI. 

- Although the proposed method brings some new aspects to XAI, the comparison in experiments is weak. The authors mainly compare the proposed approach to a couple of relatively simple methods such as the bag-of-word representations. However, there are quite some other existing representative methods such as class activation map (CAM) [a], and ABN [b], and [c] for vision transformer. Some of these existing methods [a,b] also contain learnable interpretability modules in the deep neural networks and provides interpretable predictions for image classification. 

- The evaluation and analysis of the interpretability is also relatively unclear and weak. Most of the experiments are conducted to show the accuracy on classification. However, there is not much analysis about the model interpretability. The only visualization in Figure 5 to shown the interpretability is unclear. For instance, what does the different colours refer to? How does these interpretability compared to human interpretability? How does these interpretability contribute to the model predictions? It is suggest to follow existing literature in XAI [c] to give both quantitative and qualitative evaluation about the interpretability. 

[a] Learning Deep Features for Discriminative Localization, CVPR2016
[b] Attention Branch Network: Learning of Attention Mechanism for Visual Explanation, CVPR2019
[c] Transformer Interpretability Beyond Attention Visualization, CVPR2021
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is overall clear. However, there are some parts that can be improved as detailed below.

First, terminologies are not so clear. The authors invent quite some new terminology such as "feature ingredients", "imagination of all categories", which are very unconventional words in the machine learning literature. This brings some difficulties to follow the description about the method. It is suggested to replace these words or explain why using them. 

Second, the concept of interpretability is not clearly analyzed or illustrated. For instance, in Figure 1 and Figure 5, the authors use quite some different colours to illustrate the learned interpretability. However, it is not clearly explained what different colours means and how the model provide good interpretability. 

Quality & Novelty.  
Although the proposed method brings some technical novelty to existing XAI literature. The evaluation of the proposed method is weak. 

First, evaluation compared to related representative methods [a,b] are not given. 

Second, modern evaluation metrics (such as perturbation test) used to evaluate the model interpretability is not used, please refer to [c] for more context. 

Third, the visualization about model interpretability is not clearly explained. 

Overall, the insufficiency in evaluation makes the overall technical novelty weak.

[a] Learning Deep Features for Discriminative Localization, CVPR2016
[b] Attention Branch Network: Learning of Attention Mechanism for Visual Explanation, CVPR2019
[c] Transformer Interpretability Beyond Attention Visualization, CVPR2021

Reproducibility: 
The authors provide some code to reproduce their method. 

summary_of_the_review: The paper studies interpretabiltiy for image classification and propose a new method with learnable graphs. Experiments are given to showcase is classification performance in CIFAR and Caltech. However, the comparison to existing related methods is not comprehensive. The analysis and evaluation about the model interpretabiltiy is also relatively unclear and weak as compared to existing works.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
Summary: This paper proposes a new machine model from the perspective of human cognition. This is very good. However, the system in this paper is too complex and is not concise and elegant enough. Its usefulness (especially in real environments or large datasets) remains to be further verified.

strength_and_weaknesses: 

(Positive) The authors think their method is an analogy with the human reasoning mechanism via impression matching. Such motives are very reasonable. I like it.

(Negative) The conceptual clustering shown in the lower left corner of Figure 2 is very interesting. It is the basis for the soundness of the method proposed in this article. I hope the authors give more examples (if possible, show all of them), especially in large resolutions.

(Positive) I am happy that, instead of relying on deep features, the desired outputs from schema rely on visual word relationships.

(Positive) Thanks to the authors, Figures 1 and 2 are very clear and easy to understand.

(Negative) In my opinion, the authors should validate the effectiveness of their method on more standard data such as ImageNet.

(Negative) Authors should compare their machine-learned knowledge with human knowledge and make it clear whether their machine-learned knowledge is more advanced than human knowledge. Is it worthwhile for humans to learn this independent knowledge from these machines?

(Negative) The method proposed in this paper is an overly complex engineering system, and there are many factors to be analyzed. The definition of Feat2Vertex (Equation (3) is necessary for workability but not elegant. There should be a tradeoff between the similarity at the semantic level as well as the adjacency relationship at the spatial level in Feat2Edge, which is also not elegant.

(Negative) The authors assign a dG dimensional random vector x ∈ R 174 dG to each vertex before feeding the graph to the GCN. Does this mean that there is randomness in every inference? What do you think of this randomness?

(Negative) In order to avoid the high complexity of fully connected graphs for IR-Atlas, this paper averages each instance graph. This involves engineering. Please analyze the impact of hyperparameters such as thresholds.


clarity,_quality,_novelty_and_reproducibility: 
The quality and clarity are median. The originality is good.

summary_of_the_review: 

See "Summary Of The Paper." I think this paper is a borderline paper, leaning toward being accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents the SchemaNet, a new post-hoc explainability technique for vision models based on the cognitive concept of schema. The main experiments are performed on CIFAR10/100 and Caltech-101.

The reviewers valued the interest of the SchemaNet and the obtained results. The reviewers made some comments and suggestions to improve the paper (e.g. comparison with other recent explainability methods; adding experiments on adversarial attacks; need for clarifications on the graph similarity computation via GCN; extending the explanation on how to interpret the learned class IR-Graphs, and what is the relation between an instance IR-Graph and a particular class IR-Graph; need for a deeper analysis of the obtained interpretability). The comments and suggestions were addressed by the authors in a satisfactory manner. Overall, the evaluations of this paper are positive and the reviewers agreed on accepting the paper.

In the final version of the paper authors need to address this last comment of one of the reviewers: "I would suggest to add the comparison of visualization to the other existing XAI methods (CAM, ViT-LRP), which can help highlighting the uniqueness of this method. Other approaches (CAM, ViT-LRP) mostly give attention based explanations, but this paper presents a new type of explanation. I find this part is novel but not sufficiently highlighted, and discussed".

justification_for_why_not_higher_score: The weak aspects pointed out by the authors and this final suggestion on adding more comparisons suggest the paper would need some more improvements to be accepted as spotlight.
justification_for_why_not_lower_score: This paper presents an interesting new explainability approach and all the reviewers agree on the value of the contributions.
Decision: Accept: poster