Schema Inference for Interpretable Image Classification
Authors: Haofei Zhang
Mengqi Xue
Xiaokang Liu
Kaixuan Chen
Jie Song
Mingli Song
Venue: ICLR 2023 poster (2023)

Abstract: In this paper, we study a novel inference paradigm, termed as schema inference, that learns to deductively infer the explainable predictions by rebuilding the prior deep neural network (DNN) forwarding scheme, guided by the prevalent philosophical cognitive concept of schema. We strive to reformulate the conventional model inference pipeline into a graph matching policy that associates the extracted visual concepts of an image with the pre-computed scene impression, by analogy with human reasoning mechanism via impression matching. To this end, we devise an elaborated architecture, termed as SchemaNet, as a dedicated instantiation of the proposed schema inference concept, that models both the visual semantics of input instances and the learned abstract imaginations of target categories as topological relational graphs. Meanwhile, to capture and leverage the compositional contributions of visual semantics in a global view, we also introduce a universal Feat2Graph scheme in SchemaNet to establish the relational graphs that contain abundant interaction information. Both the theoretical analysis and the experimental results on several benchmarks demonstrate that the proposed schema inference achieves encouraging performance and meanwhile yields a clear picture of the deductive process leading to the predictions. Our code is available at https://github.com/zhfeing/SchemaNet-PyTorch.

Published as a conference paper at ICLR 2023
SCHEMA INFERENCE FOR INTERPRETABLE IMAGE CLASSIFICATION
Haofei Zhang, Mengqi Xue, Xiaokang Liu, Kaixuan Chen & Jie Song!
Zhejjang University
(hacfei zhang, mqxue, yi jiyah,, chenkx, sjie) edu.ccccccucc
Mingli Song
Shanghai Institute for Advanced Study, Zhejiang University brooksongez ju. enu.en
ABSTRACT
In this paper, we study a novel inference paradigm, termed as schema inference, that learns deductively infer the explainable predictions by rebuilding the prior deep neural network (DNN) forwarding scheme, guided by the prevalent philo- sophical cognitive concept of schema. We strive to reformulate the conventional model inference pipeline into graph matching policy that associates the extracted visual concepts of an image with the pre-computed scene impression, by analogy with human reasoning mechanism via impression matching. To this end, devise an elaborated architecture, termed as SchemaNet, as a dedicated instantiation of the proposed schema inference concept, that models both the visual semanties of input instances and the learned abstract imaginations of target categories as topological relational graphs. Meanwhile, to capture and leverage the compo- sitional contributions of visual semantics in global view, we also introduce a universal Fear2Graph scheme SchemaNer to establish the relational graphs that contain abundant interaction information. Both the theoretical analysis and the experimental results on several benchmarks demonstrate that the proposed schema inference achieves encouraging performance and meanwhile yields a clear pic- ture of the deductive process leading to the predictions. Our code is available at https: thhb.....tttgtt com/ /tt.ingffgggg////////e//e///gg/s Saweeeaaee-PPPToreee
INTRODUCTION
"Now this representation of a general procedure of the imagination for providing a concept with its image is what call the schema for this concept"." "
Immanuel Kant
Deep neural networks (DNNs) have demonstrated the increasingly prevailing capabilities in visual representations compared to conventional hand-crafted features. Take the visual recognition task as an example. The canonical deep learning (DL) scheme for image recognition is to yield an effective visual representation from a stack of non-linear layers along with a fully-connected (FC) classifier at the end (He et al., 2016; Dosovitskiy et al., 2021; Tolstikhin et al., 2021; Yang et al., 2022a), where specifically the inner-produce similaritics are computed with each category embedding as typically required to
the prediction. Despite the great success of DL, existing deep networks simultaneously perceive low-level patterns as well as high-level semantics to make predictions (Zeiler & Fergus, 2014; Krizhevsky al., 2017). As such, both the procedure of computing visual represen- tations and the learned aategory-specifee embeddings are opaque to humans, leading to challenges in security-matter scenarios, such autonomous driving and healthcare applications.
" "qqual contribution.
"Corresponding author.
In Critique Pure Reason (A 1140/8180).

---

Published as a conference paper at ICLR 2023
Rrarhph
Local structure of #198
Local structure of +150
6SS
RES
RiM
TISR OHI0
UFF
AIB
Matcher
Skephout
18.60 (#196)
2.65 (#150)
Cauegury IR-Graph
Beqreseni
Ingrrelienn Vaszlicataas
$107
Categocy
1044
1.50
G150
211
AIDE
NLES
ALZS
EES
Figure I: An example showing how an instance IR-Graph is matched to the class imagination. The vertices of FR-Graphs represent visual semantics, and the edges indicate the vertex interactions. The graph matcher captures the similarity between the local structures of joint vertices (e.g., vertex #198 and vertex #1 #150) by aggregating information from their neighbors as class evidences. The final prediction is defined as the sum of all evidence.
Unlike prior works that merely obtain the targets in the black-box manner, here strive to devise an innovative and generalized DNN inference paradigm by reformulating the traditional one-shot forwarding scheme into interpretable DNN reasoning framework, resembling what occurs in deductive human reasoning. Towards this end, inspired by the schema in Kant's philosophy that describes human cognition as the procedure of associating an image of abstract concepts with the specific sense impression, we propose to formulate DNN inference into an interactive matching pro- cedure between the local visual semantics of an input instance and the abstract category imagination, which is termed as schema inference in this paper, leading to the accomplishment of interpretable deductive inference based on visual semantics interactions the micro-level.
To elaborate the achievement of the proposed concept schema inference, we take here image classification, the most basic task in computer vision, as an example to explain our technical details. At a high level, the devised schema inference scheme leverages pre-trained DNN to extract feature ingredients which are, in fact, the semantics represented by a cluster of deep feature vectors from specific local region in the image domain. Furthermore, the obtained feature ingredients organized into an ingredient relation graph (IR-Graph) for the sake of modeling their interactions that characterized by the similarity the semantic- at the spatial-level. We then implement the category-specific imagination as an ingredient relation atlas (IR-Atlas) for all target categories induced from observed data samples. As a final step, the graph similarity between an instance-level IR- IR-Graph and the category-leved IR-Atlas is computed as the measurement for yielding the target predictions. As such, instead of relying on deep features, the desired outputs from schema inference contribute only from the relationship of visual words, as shown in Figure
well as the adjacency relationship
More specifically, our dedicated schema-based architecture, termed as SchemaNet, is based on vision Transformers (ViTs) (Dosovitskiy al., 2021; Touvron et al., 2021), which are nowadays the most prevalent vision backbones. To effectively obtain the feature ingredients. we collect the intermediate features of the backbone from probe data samples clustered by k- k-means algorithm. IR-Graphs are established through customized Feat2Graph module that transfers the discretized ingredients array to graph vertices, and meanwhile builds the connections, which indicates the ingredient interactions relying on the self-attention mechanism (Vaswani et al., 2017) and the spatial adjacency. Eventually, graph similarities are evaluated via a shallow graph convolutional network (GCN).
Our work relates to several existing methods that mine semantic-rie -rchh visual words from DNN backbones for self-explanation (Brendel & Bethge, 2019; Chen et al., 2019; Nauta et al., 2021; Xue al., 2022b; Yang et al., 2022b). Particularly, Bag (Brendel & Bethge, 2019) uses DNN as visual
2

---

Published as a conference paper at ICLR 2023
Back propagation
Trainable
0.081
xN
Feat2Graph
0.15%
Matcher
0.510
10.7 00.36
Input
y
Backbone
Instance IR-Graph
Prediction
IR-Atlas
Figure 2: The overall pipeline of our proposed SchemaNet. Firstly, intermediate feature X from the backbone is fed into the Feat2Graph module. The converted IR-Graph is then matched to set of imaginations, iee, category-level IR-Atlas induced from observed data, for making the prediction.
word extractor that is analogous to traditional ang-of-visuullwordss (BoVW) representation (Yang al., 2007) with SIFT features (Lowe, 2004). Moreover, Chen et al. (2019) develop ProtoPNer framework that evaluates the similarity scores between image parts and learned class-specific prototypes for inference. Despite their encouraging performance, all these existing methods suffer from the lack of considerations the compositional contributions of visual semanties, which is, however, already proved critical for both human and DNN inference (Deng et al., 2022), and are not applicable to schema inference in consequence. We discuss more related literature in Appendix A.
In sum, our contribution is an innovative schema inference paradigm that reformulates the existing black-box network forwarding into a deductive DNN reasoning procedure, allowing for clear insight into how the class evidences are gathered and deductively lead to the target predictions. This is achieved by building the dedicated IR- IR-Graph and IR-Atlas with the extracted feature ingredients and then performing graph matching between IR-Graph and IR-Atlas to derive the desired predictions. We also provide a theoretical analysis to explain the interpretability of our schema inference results. Experimental results CIFAR-1O/100 Caltech- 101, and ImageNet demonstrate that the proposed schema inference yields results superior to the statc-of-the- interpretable approaches. Further, demonstrate that by transferring to unseen tasks without fine-tuning the matcher, the learned kedwledge each class is, in fact, stored IR-Atlas rather than the matcher, thereby exhibiting the high interpretability of the proposed method.
SCHEMANET
In this section, we introduce our proposed SchemaNet in detail. The overall procedure illustrated in Figure 2, including a Feat2Graph module that converts deep features to instance IR- TR-Graphs, learnable IR-Atlas, and graph matcher for making predictions. The main idea is to model an input image
instance-level graph in which nodes are local semantics captured by the DNN backbone and edges represent their interactions. Mcanwhile, category-Level graph is maintained as the imagination for each class driven by training samples. Finally, by measuring the similarity between the instance and category graphs, we are able to interpret how predictions are made.
2.1 PRELIMINARY
We first give a brief review of the ViT architecture. The simplest VIT implementation for image classification is proposed by Dosovitskiy et al. (2021), which treats an image as sequence of independent 16 x 16 patches. The embeddings of all patches are then directly fed to Transformer- based network (Vaswani et al., 2017), i.e., stack of multiple encoder layers with the same structure: a multi-head self afffatttentin mechanism (MHSA) followed by a multilayer perceptron (MLP) with residual connections. Additionally, they append class token (CLS) to the input sequence for classification alone rather than the average pooled feature. Following this work, Touvron et al. (2021) propose DeiT appending another distillation token (DIST) learn soft decision targets from teacher
3

---

Published as a conference paper at ICLR 2023
model. This paper will mainly focus on DeiT backbones due to the relatively simple yet efficient network architecture.
Let X [(++(xdd denotes the input feature (a sequence of visual tokens and auxillary tokens, e.g., CLS and DIST, each of which a d-dimensional embedding vector) to a MHSA module. The output can be simplified as MXSA(X) = Σx= V XW,, where V E [R(++()(+++)) denotes the self-attention matrix normalized by row-wise softmax of head h {I...., H. and Wi, E Rd is learnable projection matrix. As the MLP module processes each token individually, the visual token interactions take place only in the MHSA modules, facilitating the extraction of their relationships from the perspective of ViTs.
For convenience, we define some notations related to the self-attention matrix. Let vi be the average attention matrix of all heads and be the symmetric attention matrix: V (v + 1/2. We
partition V into four submatrices:
VT VA
(I)
VV
where T E T is the attention to the auxiliary tokens, V E Raxx represents the relationships of visual tokens, and T R Finally, let &cLS R represent the attention values the CLS token extracted from 4
2.2 FEAT2GRAPH
As shown Figure 2, given pre-trained backbone and an input image, the intermediate feature X is converted to an IR-Graph G for inference by the Feat2Graph module which includes three steps: (1) discretizing X into a sequence feature ingredients with specific semantics; (2) mapping the ingredients to weighted vertices of IR- IR-Graph; (3) assigning weighted edge to each vertex pair indicating their interaction. We start by defining IR- Grarr and IR-Atlas mathematically.
Definition (R-Graph). An IR- PRarr is an undirected fully-conneeee graph = (E, V), in which vertiex set V is the set of feature ingredients for an instance specific calegory, and edge set E encodes the interactions of vertex pairs.
In particular, indicate the vertex importance (for instance, "bird head" should contribute the prediction of bird more than "sky" in human cognition), non-negative weight E R_ assigned to each vertex. Let A E R
VI {M be the collection of all vertex weights. Besides, the interaction between vertex i and j is quantified by a non-negative weight ei.j E R,, With a slight abuse of as the weighted adjacency matrix in the following sections.
+
notation, we define E R |V|x|V|
The IR-Atlas is defined to represent the imagination of all categories:
Definition 2 (IR-Atlas). An IR-Atlas & of C classes is a of category-level IR-Graphs, in which element G (Ee, V.) for class has learnable vertex weights Åg and edge weights E...
The main purpose of feature discretization is to mine the most common patterns appearing in the dataset, each of which corresponds to a human understandable semantic, named visual word, such as "car wheel" or "bird head". However, local regions with the same semantic may show differently because of scaling, rotation, or even distortion. In our approach, we utilize the DNN backbone extract relatively uniform feature instead of the traditional SIFT feature utilized in (Yang et al., 2007). To be specific, a visual vocabulary constructed by k-means clustering running the collection X of visual tokens extracted from the probe d Further, a deep feature X (CLS and DIST are removed) is discretized to a sequence of feature ingredients by replacing each element x with the index of the closest visual word
Discretization.
= (w> J.M (w; E TRd)) of size M is
Ingredient(x)) arg min ||æ will2-
se(1....MM)
Strictly, the ingredient is referred to as the index of visual word w. With the entire ingredient set MI = M,). the discretized sequence is denoted as X = (ø .ei....,ãs in), where T E M is computed from Equation (2). The detailed settings of M are presented in Appendix E.
For probe dataset with D instances, the collection X has D visual tokens (ignoring CLS and DIST).
4

---

Published as a conference paper at ICLR 2023
Veat2Vertex. After we have computed the discretized feature X, the vertices of this feature the unique ingredients: V Unique(X). The importance of cach vertex is measured from two criterions: the contribution to the DNN's final prediction and the appearance statistically. Formally, for vertex M. the importance is defined
X 01 a1 CLS + 0:2
bag = 01
Σ (E=()|X)
1/ CLS + a21=(([(x)
where [ is the set of all the appeared position of ingredient in X, vi V is the attention between the i-th visual token and the CLS token, and 0122 2 0 are learnable weights balancing the two terms. When on = 0, equivalent to BoVW representation.
Feat2Edge. With the self-attention matrix V extracted from Equation (1), the interactions between vertices can be defined and computed with efficiency. For any two different vertices u, V, the edge weight the comprehensive consideration of the similarity in the view of VIT and spatial adjacency. The first term is the average attention between all repeated pairs 17 [7, X]:
alin
Σ
(u, x
where [1, X]] is the Cartesian product of E(u and E)) and V f is the attention between the visual tokens at and j positions. Furthermore, we define the adjacency as
adj
E
(u, x) (1)))
c+ |Pos(i) Pos(j))l2*
where function Pos(-) returns the original 2D coordinates of the input visual token with regard the patch array after the patch embedding module of ViTs. Eventually, the interaction between vertices and the weighted sum of the two components with learnable $1.2:
Cu,o = 31 e attn + B2
Equations (4) and (5) are both invariant when exchanging vertex and v, so our IR-Graph is equivalent to undirected graph.
is worth noting that only semantics and their relationships are preserved IR-Graphs for further inference rather than deep features.
2.3 MATCHER
After converting an input image to RR-rrahh, the matcher finds the most similar category-level graph in IR-Atlas. The overall procedure is shown in Figure 3, which is com- posed of GCN module and similarity computing mod- ule (Sim) that generates the final prediction. Detailed settings of the matcher
G
F C
F
F
Appendix E.3.
For feeding IR-Graphs to the GCN module, we assign each ingredient m MI (i.e., graph vertices) with trainable embedding vector, each of which is initialized from an de dd-dimensional random vector drawn independently from multivariate Gaussian distribution (0, Taa).
GraphConv
GCN
GraphCCov
In the GCN module, we adopt GraphConv (Morris al., 2019) with slight modifications for weighted edges. Let
WAP
F E RIV xdo be the input feature of all vertices to a GraphConv layer, the output is computed
Sim
GraphConv(F) Norm (a ((lac + E)FW)), (7)
y/
where a denotes a non-linear activation function, Norm denotes feature normalization, and W xddaxda is
Figure 3: Illustration of the matcher.
5

---

Published as a conference paper at ICLR 2023
learnable projection matrix. After passing through total La layers of GraphConv. weighted average pooling (WAP) layer summarizes the vertex embeddings p(La) weighted by A, yielding the graph representation AF(Le), Now we have for G and Z E _xx (Z1,..., &c) for all graphs in G. By computing the inner-product similarities, the final prediction logits is defined as y = -²T..
Prediction interpretation. Given instance graph G and category graph G. now analyze how the prediction is made based on their vertices and edges. We start with analyzing the similarity score s = 2z² where 2 = AF(La) and & = Af(L)) are weighted sum of the vertex embeddings respectively. For the sake of further discussion, let f! denotes the embedding vector of vertex output from the /-th GraphCony layer, and A denotes its weight. The original vertex embedding of v is defined as f which is identical for the same ingredient in any two graphs. Besides, let c V n be the set of shared vertices in G and G. The computation of can be expanded
Σ
P(Le) ((t)))
(8)
(L.))VVVV
Since directly analyzing Equation (8) is hard, show approximation and proof it in Appendix B. Theorem 1. For a shallow GCN module, Equation (8) can be approximated by
Eisdosh plba)
6EΦ
Particularly, LG and 01 our method is equivalent BoVW with a linear classifier.
Further, as delineated in Corollary 1, the interpretability of the graph matcher with shallow GCN can be stated as: (1) the final prediction score for a category the summation of all present class evidence (shared vertices ø in (b) represented by the vertex weights; (2) the shared neighbors (local structure) connected to the shared vertex & also contribute to the final prediction.
2.4 TRAINING SCHEMANET
Except for training SchemaNet by cross-entropy loss Leg between the predictions and ground truths, we further constrain the complexity of IR-Atlas. More formally, the complexity is defined for both edges and vertex weights of all graphs in IR-Atlas as
C
C. T
Σ H(Ad), C. CI
Σ H H(B...). ueV
(10)
where function H(x) computes the entropy of the input vector æ E R normalized by its sum
components, and Ee... denotes the weighted edges connected to vertex in G. of class The final optimization goal is
L LCe + YoC, reCe.
(11)
where 7. and Ye are hyperparameters. The overall training procedure shown Appendix
Sparsification. Each graph in IR-Atlas is initialized a fully-connected graph with random vertex and edge weights. However, this requires (CCVV()) memory space for storage and training the whole set of edges. To alleviate this issue, we initialize IR-Atlas by averaging the instance IR-Graphs for each class and remove the edges connected to vertices whose weights are below a given threshold 5, 0.01 from the -ppecffc graph. Such procedure will not only dramatically decrease the learnable parameters but also boost the final performance, as shown Table 1.
EXPERIMENTS
IMPLEMENTATION
Datasets. We evaluate our method on CIFAR-10/100 (Krizhevsky et al., 2009), Caltech-101 (Li et al., 2022), and ImageNet (Deng et al., 2009). Particularly, Caltech- 101 has around 9k images, and manually split it into training set with 7.4k images and test with 1.3k images.

---

Published as a conference paper at ICLR 2023
Table I:
Comparison results CIFAR-10/100 and Caltech-10 10cc with different backbones. We report the top-1 accuracy, number of learnable parameters, and FLOPs. The results in gray color are merely for listing the accuracy of ViTs rather than for making comparisons.
CIFAR-10
CIFAR-100
Caltech-101
Backbone
Method
Acc
#param.
FLOPs
Acc
#param.
FLOPs
Acc
#param.
FLOPs
BoVW-SIFT
20.20
Base
96.69
5.53M
1.27G
82.88
5.54M
1.27G
92.53
5.54M
1.27G
Backbone-FC
94.58
1.93K
1.06G
76.74
19.3K
1.06G
76.71
19.5K
1.06G
BoVW-Deep
94.95
10.3K
1.06G
75.45
103K
1.06G
60.82
104K
1.06G
DeiT-Tiny
BagNet
70.27
5.73M
1.10G
42.90
5.84M
1.10G
72.54
5.85M
1.10G
SchemaNet
95.92
396K
1.19G
77.11
105M
2.40G
83.24
106M
2.40G
SchemaNet-Ing Init
95.96
234K
1.06G
78.45
573K
1.06G
87.57
564K
1.06G
97.77
21.7M
4.63G
87.65
21.7M
4.63G
94.96
21.7M
4.63G
Backbone-FC
96.62
3,85K
3.87G
82.44
38.5K
3.87G
86.94
38.9K
3.87G
BoVW-Deep
10.3K
3,89G
77.39
103K
3.89G
65.54
104K
3.89G
DeïT-Small
BagNet
22.1M
3,95G
50.49
22.2M
3,95G
78.13
22.2M
3,95G
SchemaNet
97.42
396K
4.00G
82.21
105M
5.21G
84.66
5.21G
SchemaNet-Init
97.35
235K
3.89G
82,46
591K
3.89C
90.09
589K
3.89G
Buse
98.41
85.8M
17.6G
89.17
85,9M
17.6G
95.83
85.9M
17.6G
Backbone- FC
97.04
7.69K
14.7G
81.66
76.9K
14.7G
88.20
77.7K
14.7G
BoVW-Deep
95.50
10.3K
14.7G
72.23
103K
14.7G
66.17
104K
14.7G
DeiT-Base
BagNet
90.71
86.6M
14.9G
67.84
86.8M
14.9G
86.55
86.8M
14.9G
SchemaNet
97.26
396K
14.8G
79.26
105M
16.1G
81.12
106M
16.1G
SchemaNer-
97.07
235K
14.7G
79.36
606K
14.7G
90.72
593K
14.7G
Selection of the hyperparameters. Several hyperparameters are involved in our method, including
An in Equation (1 for adjusting the sparsity of IR-Atlas. We set A, = 0.5 and X = 0. as the default value for the following evaluation and the sensitive analyses are presented in Appendix H. The initial values of learnable 012 and $1,2 are set to 0.5, and we show their learning curves in Figure
3.2 EXPERIMENTAL RESULTS
We evaluate our method and comparison baselines with the following settings:
Bo VW-SIF::: the traditional BoVW approach that utilizing SIFT feature for constructing the visual vocabulary (Yang et al., 2007).
Base: the base ViT model directly trained on the benchmark datasets with initial weights
obtained from the official repository in (Touvron et al., 2021), which is our backbone.
Backbone-FC: the frozen backbone with an FC layer. The intermediate features extracted from the frozen backbone are then fed to a global average pooling layer followed by a linear classifier, similar to the standard CNN protocol.
Bo-W-Deep: the BoVW approach with extracted visual vocabulary.
BagNet: the implementation of BagNet (Brendel Bethge, 2019) with VIT backbone which is constructed following our "Base" setting.
SchemaNet: our proposed SchemaNet without initialization.
SchemaNet-Init: our proposed SchemaNet initialized by the average instance IR-Graphs.
The comparison results are shown in Table 1, where we demonstrate the top-1 accuracy, the number of learnable parameters, as well as the FL.OPs for each setting. It is noticeable that the proposed SchemaNet consistently outperforms the baseline methods. Specifically, for the backbone of DeiT- Tiny, ours achieves significant improvement (i.e., about 25.7% and 35.3% absolute gain on CIFAR- 10 and CIFAR-100, respectively) over BagNet. With larger backbone such as DeiT-Small and DeiT- Base, though the performance gap slightly shrinks, our SchemaNet with initializations still yields results superior to BagNet, by an average absolute gain of 12.7%. The reason that, unlike BagNet that uses summation to obtain the similarity map as the BoVW representation (without feature discretization), "BoVW-Deep" is implemented with the discretized visual words. As such, both of
7

---

Published as a conference paper at ICLR 2023
100
80
BoVW-Doop
BoVW-Deep
BagNett
BegNet
Ours!
Apc (%6
Ourv
50
BoVe-Deep
Acc
40
BoVW-Doop
BogNott
20
BagNet-
Ourst
Drop Rate (56)
Drop Rate (%)
(a) Accuracy curves on CIFAR-10.
(b) Accuracy curves on CIFAR-10K
Figure Accuracy decay curves of perturbation tests. represents positive/negative perturbation, accordingly.
The curve name ending with " "4"/-""
the approaches do not consider the visual word interactions, leading to the inferior performance especially when the size of the visual vocabulary expands on CIFAR-100 and Caltech-
Moreover, compared with "Backbone-Fi FC that merely relies on intermediate deep features, the proposed SchemaNet also demonstrates encouraging results, achieving about a 1.4% absolute gain. Furthermore, when initialization and redundant vertices are removed as indicated in Section 3.1, our "SchemaNet-Init" with an even less Iearnable number of parameters, still leads to a higher accuracy. The proposed method also delivers gratifying performance, par with the prevalent deep ViT especially on CIFAR-10, but gives a clear insight on the reasoning procedure.
We present experimental results in Appendix F, including the comparison results Ima- geNet (Appendix F.1), employing other attribution methods (Appendix F.2), attacking with adver- sarial images (Appendix F.3), the extendability analysis of our SchemaNet (Appendix F.4), and the inference cost all components (Appendix F.5).
3.3 EVALUATION OF THE INTERPRETABILITY
To figure how interpretability contributes the model prediction, we adopt the positive and negative perturbation tests presented in (Chefer et al., 2021) to give both quantitative and qualitative evaluations on the interpretability. For a fair comparison, the attention values to the CL.S token gCLS are extracted as the pixel relevance for BoWW-Deep, BagNet, and SchemaNet-Init with the DeiT-Tiny backbone. During the testing stage, we gradually drop the pixels in relevance descending/ascending order (for positive and negative perturbation, respectively) and measure the top-1 accuracy of the models. We plot the accuracy curves in Figure 4, showing that: (1) in the positive perturbation test, method behaves similarly to BoVW while outperforming BagNet; (2) in the negative perturbation test, our method achieves better performance by a large margin (the wree--uddr-thhe..o about 5.71 % higher than BagNet CIFAR-10, and 11.23% higher than BagNet CIFAR-100),
of ours is
3.4 VISUALIZATION
In Figure 5, we show the examples including instance IR- IR-Graphs and learned IR-Atlas with DeiT- Tiny backbone trained on Caltech- 101 dataset (for visualization purposes, the number of ingredients is to 256, which are shown entirely in Figure 11). Each row shows several instances of specific class. The first column includes the category graphs and excerpts of appeared ingredients Figure 11 for quicker reference. We further render the vertices, i.e., ingredients, in the category graph with different colors, and the appeared ingredients in the instance images and graphs are colored uniformally corresponding to the category graph.
We interpret the visualization from three perspectives: the interpretability of the ingredients, the interpretability of the edges, and the consistency between instance and category graphs. (1) Thanks to the powerful representation learning capability of DNNs, the extracted ingredients are able to represent explicit semantics (such "fuselage", "bird legs". etc.) with robustness. Besides, the
8

---

Published as a conference paper at ICLR 2023
IR-Atlas
Instance IR- IR-Graphs
Figure 5: Examples the learned IR-Atlas and instance IR-Graphs randomly sampled from three categories ("panda", "flamingo", and "airplanes") ") on Caltech- 101. Please zoom in for better view.
learned weight of each vertex is also consistent with human intuition the object ingredients get higher weights than the background ones. (2) The learned edge weights tend to connect object parts the adjacent background and other object parts while ignoring the background connections, which helps distinguish fine -raig categories with the perception of the surroundings, (3) As the category graphs are driven by the instances, they learn capture and model the general character of instances and eventually form the abstract imaginations of all categories. More examples are in Appendix
4 CONCLUSION AND OUTLOOK
In this paper, propose novel inference paradigm, named schema inference, guided by Kant's philosophy towards resembling human deductive reasoning of associating the abstract concept image with the specific sense impression. To this end, reformulate the traditional DNN inference into a graph matching scheme by evaluating the similarity between instance-level IR RR-Graph and category-leve level imagination in deductive manner. Specifically, the graph vertices are visual semantics represented by common feature vectors from DNN's intermediate layer. Besides, the edges indicate the vertex interactions characterized by semantic similarity and spatial adjacency, which facilitate capturing the compositional contributions to the predictions. Theoretical analysis and experimental results several benchmarks demonstrate the superiority and interpretability of schema inference. In future work, we will implement schema inference to more complicated vision tasks, such as visual question answering, that enables linking the visual semantics the phrases in human language. achieving more powerful yet interpretable reasoning paradigm.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China (61976186, U20B2066. 62106220), Ningbo Natural Science Foundation (2021J189), the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study (Grant No. SN-ZJU-SIAS-001) Open Research Projects Zhejiang Lab (NO. 2019KD0AD01/018) and the Fundamental Research Funds for the Central Universities (2011FZZX001-23)
REFERENCES
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 281. PMLR, 09-15 19-15 Jun 2019. URL www.tttps//omm U
m.r..presse / v97 anconal 1am.lmml.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXivv1607.064500 2016.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non non-linear classifier decisions by layer- layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Wieland Brendel and Matthias Bethge. Approximating CNNs with hag-of-local-feanree models works surprisingly well on imagenet. In International Conference on Learning Representations, 2019.
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alexander G Hauptmann. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 782-791, 2021.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Huiqi Deng, Qihan Ren, Hao Zhang, and Quanshi Zhang. Discovering and explaining the representa- tion bottleneck of dnns. In International Conference on Learning Representations, 2022. URL https: /ttps////peeeiiieeeeeeeeeeeeeeee
net/ forum? did iicl: gmd FHJ.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16458-16468. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Zunlei Feng, Jiacong Hu, Sai Wu, Xiaotian Yu, Jie Song. and Mingli Song. Model doctor: A simple gradient aggregation strategy for diagnosing and treating enn classifiers. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 616-624, 2022.
10

---

Published as a conference paper at ICLR 2023
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10705-10714, 2019.
Yunhao Ge, Yao Xiao, Zhi Xu, Meng Zheng, Srikrishna Karanam, Terrence Chen, Laurent Itti, and Ziyan Wu. A peek into the reasoning of neural networks: Interpreting with structural visual con- cepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2195-2204, 2021.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6928 6938, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-o--disti generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021.
Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Mapping images to scene graphs with permutation invariint structured prediction. Advances in Neural Information Processing Systems, 31, 2018.
Adrian Hoffmann. Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it? shortcomings of latent space prototype interpretability in deep networks. CoRR, abs/2105.02968, 2021. URL www.thtttttto...tm / orgg 210.. 02968.
Yongeheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Amalgamating knowledge from heterogeneous graph neural networks. In CVPR, 2021a.
Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. agta-aggreeator: Learning aggregate for 1-bit graph neural networks. In ICCV, 2021b.
Yongcheng Jing, Yining Mao, Yiding Yang, Yibing Zhan, Mingli Song, Xinchao Wang, and Dacheng Tao. Learning graph neural networks for image style transfer. In ECCV, 2022.
Thomas Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.0290) 2016.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
Yael Konforti, Alon Shpigler, Boaz Lerner, and Aharon Bar-Hillel. Inference graphs for cnn interpretation. In European Conference on Computer Vision, pp. 69-84. Springer, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):8-99, 2017.
Fei-Fei Li, Marco Andreeto, Mare' Aurelio Ranzato, and Pietro Perona. Caltech 101, 2022. URL https:/ htapps://daa./ /aata..... ca callted wecc...hh... edu/ rwwords/ /0086.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 3538-3545, 2018.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In NeurIPS, 2022.
11

---

Published as a conference paper at ICLR 2023
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
David G Lowe. Distinctive image features from computer vision, 60(2):91-100, 2004.
keypoints. International journal of
Wenjie Luo, Yujia Li. Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder. Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Par- tern Recognition, 15:211-222, 2017. ISSN 0031-3203. doi: htmm///ceenne/attiie/ppppp.
2016.11.008. URL https: /wwww sciencedirect. com/ S0031320316303588
Christopher Morris, Martin Ritzert, Matthias Fey, William I. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4602-4609, 2019.
Meike Nauta, Ron van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14933-14943, June 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, bighhpeeffrmance deep learning library. Advances in neural information processing systems, 32, 2019.
MDJ Peters. Extending explainability of generative classifiers with prototypical parts. Master's thesis, University of Twente, 2022.
Dawid Rymarczyk, Aneta Kaczyńska, Jarostaw Kraus, Adam Pardyl, and Bartosz Zieliński. Protomil: Multiple instance learning with prototypical parts for fine-grained interpretability. arXiv preprint arXiv:2108. 10612, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Suprosanna Shit, Rajat Koner, Bastian Wittmann, Johannes Paetzold, Ivan Ezhov, Hongwei Li, Jiazhen Pan, Sahand Sharifzadch, Georgios Kaissis, Volker Tresp, et al. Relationformer: A unified framework for image-to- -raphh generation. arXiv preprint arXiv:2203.10222. 2022.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference machine learning, pp. 3145- 3153. PMLR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. In Workshop International Conference on Learning Representations, 2014.
Erik Strumbelj and Igor Kononenko, An efficient explanation of individual classifications using game theory. The Journal of Machine Learning Research, 11:1-18, 2010.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021.
12

---

Published as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Suvidha Tripathi, Satish Kumar Singh, and Hwee Kuan Lee. Bag of visual words (bovw) with deep features patch classification model for limited dataset of breast tumours, 2022. URL
https: / wwwopppprpeepeeeeeeeoe oot//orattoeo wtt.g//telee/e/eer.eor..e.eeeeee /wtp//////PPPP0/P//
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Lauren J Wong and Sean McPherson. Explainable neural network-based modulation classification via concept bottleneck models. In 2021 IEEE I Ith Annual Computing and Communication Workshop and Conference (CCWC), pp. 0191-0196. IEEE, 2021.
Mengqi Xue, Qihan Huang, Haofei Zhang. Lechao Cheng, Jie Song, Minghui Wu, and Mingli Song. Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208. 10431, 2022a.
Mengqi Xue, Haofei Zhang, Qihan Huang, Jie Song, and Mingli Song. Learn decision trees with deep visual primitives. Journal of Visual Communication and Image Representation, 89: 103682, 2022b. ISSN 1047-3203. doi: //www www.ww.ceeen..d...occ rect. com/
URL https:
Jun Yang, Yu-Gang Jiang, Alexander G Hauptmann, and Chong-Wah Ngo. Evaluating bag-of-visual- words representations scene classification. In Proceedings of the international workshop on Workshop on multimedia information retrieval, pp. 197-206, 2007.
Xingyi Yang, Zhou Daquan, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. In NeurlPS, 2022a.
Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In ECCV, 2022b.
Yiding Yang, Xinchao Wang, Mingli Song, Junsong Yuan, and Dacheng Tao. Spagan: Shortest path graph attention network. In IJCAI, 2019.
Yiding Yang, Zunlei Feng. Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. NeurIPS, 2020a.
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In CVPR, 2020b.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.0700 30.01111111,, 2023.
Alireza Zareian, Zhecan Wang. Haoxuan You, and Shih-Fu Chang. Learning visual commonsense for robust scene graph generation. In European Conference on Computer Vision, pp. 642-657. Springer, 2020.
Matco Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini. Michelangelo Diligenti, Zohrch Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209. 200.000009000066 2022.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European conference computer vision, pp. 818-833. Springer, 2014.
Quanshi Zhang, Xin Wang, Ruiming Cao, Ying Nian Wu, Feng Shi, and Song- Chun Zhu. Extraction of an explanatory graph to interpret a cnn. IEEE transactions on pattern analysis and machine intelligence, 43(1 33336333877 2020a.
13

---

Published as a conference paper at ICLR 2023
Quanshi Zhang, Xin Wang. Ying Nian Wu, Huilin Zhou, and Song- Chun Zhu. Interpretable cnns for object classification. IEEE transactions on pattern analysis and machine intelligence, 43(10): 3416-3431 2020b.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Cheekong Lee. Protgnn: Towards self- explaining graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9127-9135, 2022.
Lingxiao Zhao and Leman Akoglu. Pairoorm Tackling oversmoothing in gnns. In International Conference Learning Representations, ICLR, 2020.
Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. Learning to generate graph from natural language supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1823 1823-1834, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza. Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In International Conference on Learning Representations, 2017. URL https: : / /pwp/eveeeewweep/////p/p. wtt:teeuut/aaeeeeeeoo BJ5UeU9xX.
14

---

Published as a conference paper at ICLR 2023
A
ADDITIONAL RELATED WORK
In this section, we introduce literature that is related our method.
A.1 VISUAL CONCEPT ANALYSIS
Visual concept analysis aims interpret the behavior of pre-trained deep models by extracting and tracking intuitive visual concepts during the DNN's inference procedure. Previous works use masks (Zhang et al., 2020b), explanatory graphs (Zhang al., 2020a) and probabilistic models (Kon- forti et al., 2020) to interpret the internal layers of convolutional neural networks (CNNs) (Liu et al., 2022: Yu et al., 2023). Further, Deng et al. (2022) propose to capture and analyze the interaction of visual concepts contributing cooperatively to the prediction results for CNNs. However, they use Shapley values to compute the interaction, which computationally expensive. Consequently, we adopt vision Transformers (ViT') (Dosovitskiy et al., 2021; Touvron et al., 2021) as the DNN back- bone of our method instead of conventional CNNs thanks to the multi-head self-attention (MHSA) mechanism ( (Vaswani et al., 2017) that explicitly encodes the interactions of visual tokens in the similarity level.
More recently, VRX (Ge et al., 2021) is proposed to graphical model to interpret the prediction of a pre-trained DNN, in which edges are constructed to represent the spatial relation of visual semantics. They further utilize a GCN module to make a prediction with the input structural concept graphs by training to mimic DNN's prediction. Nevertheless, their approach requires training a graph for every input sample, which is extremely expensive for applying to large-scale datasets. Besides, VRX does not construct the class imagination that carries the category-specific knowledge, meaning that the GCN predictor must learn to memorize and distinguish all the class-specific graphs, which impairs the overall interpretability.. Our proposed schema inference, however, explicitly creates an interactive matching procedure between the instances and category imagination. Moreover, our GNN model only captures the local structure for gathering class evidence, which is easier to accomplish.
A.2 VISUAL CONCEPT LEARNING
Visual concept learning refers to a range of methods that mine semantic-rich visual words from a DNN, which are further utilized make interpretable predictions towards a self-explanatory model in reality. Concept bottleneck models (CBMs) (Koh et al., 2020; Zarlenga et al., 2022; Deng et al., 2022; Wong& & McPherson, 2021) link the neurons with cuman--nterpretabbe semanties explicitly, encouraging the trustworthiness of DNNs. Human interventions of learned bottleneck layers can fix the misclassified concepts to improve the model performance. Another line of visual concept learning, port-prototype-basee based methods, collectively makes predictions on target tasks with DNN and semantic-rich prototypes, which can be divided into two schools according to their prototypes. (I) Sis-of-Visual-Wonr Word (Bo approaches (Brendel & Bethge, 2019; Gidaris et al., 2020; Tripathi et al., 2022) obtain prototypes with semantics through passively clustering the hand-crafted features (Yang et al., 2007) or deep features into a set of discrete embeddings (visual words) that relate specific visual semantics. The prediction procedure of these approaches is analog to the BoW model in natural language processing (NLP), in that interpretable representation is constructed based on the statistics of the occurrence of the visual words, and then fed to an interpretable classifier, such as the linear classifier or decision trees. (2) In contrast, prot-protoy networks (ProtoPNets) (Chen et al., 2019) and the following works (Nauta et al., 2021; Xue et al., 2022a; Zhang et al., 2022; Peters, 2022; Rymarczyk al., 2021) jointly train DNN (as the feature extractor) and parameterized prototypes. Then decisions are made based on the linear combination of similarity scores between prototypes and feature vectors at all the spatial locations. Despite the relatively high performance, as discussed in (Brendel & Bethge, 2019; Hoffmann et al., 2021), the similarity of the learned prototypes and deep features in the embedding space may be significantly different in the input space. As such, we only compare those approaches in which prototypes level of interpretability. To the best of our knowledge, none of the existing works have explored a deductive inference paradigm based on the interaction of visual semanties.
generated based clustering for the same
15

---

Published as a conference paper at ICLR 2023
A.3 SCENE GRAPH GENERATION
A scene graph can be generated from an input image to excavate the collection of objects, at- tributes, and relationships in whole scene. Typical graph-based representations and learning algorithms (Herzig et al., 2018; Chang et al., 2021; Shit et al., 2022; Zhong et al., 2021; Zareian al., 2020) adopt graph neural networks (GNNs) (Kipf Welling, 2016; Jing et al., 202 Yang et al., 2020b: Jing et al., 2021b; Yang et al., 2020a; Jing et al., 2022; Yang et al., 2019) to model the relationships between objects in a scene. Similar to scene graphs, our proposed SchemaNet also utilizes GNNs to represent the relationships between ingredients related to local semantics. However, the nodes in GNNs SchemaNet represent more fine-grained representations, i.e., local semantics of objects, compared to those relating to the whole objects in scene graphs. Moreover, in SchemaNet, the trained GNN is responsible for estimating the similarity between an instance IR-Graph and category IR- IR-Graphs and making classification based on the similarity scores.
A.4 FEATURE ATTRIBUTION FOR EXPLAINABLE AI
In the "Feat2Vertex" module, we extract the attention to the CLS token as one important component of the ingredient importance. Currently, plenty of approaches have been proposed for indicating local relevance (forming saliency maps) to the DNN's prediction, termed feature attribution methods. Most existing works can be roughly divided into three classes: perturbation-basee decomposition- approaches. Perturbation-base approaches (Strumbelj & Kononenko, 2010; Zeiler & Fergus, 2014; Ancona al., 2019; Zintgraf et al., 2017) compute attribution by evaluating output differences via removing or altering input features. Gradient-based approaches (Simonyan al., 2014; Shrikumar et al., 2017; Sundararajan et al., 2017; Selvaraju et al., 2017; Feng al., 2022) compute gradients w.E.t the input feature through backpropagation. Decomposition- based approaches (Bach al., 2015; Montavon al., 2017; Chefer et al., 2021) propagate the final prediction to the input following the Deep Taylor Decomposition (Montavon et al., 2017). Besides, CAM (Zhou et al., 2016) and ABN (Fukui et al., 2019) provide interpretable predictions with learnable attribution module. As most previous methods focus CNNs, Chefer al. (2021) propose ViT-LRP tailored for vision Transformers. However, most of the methods mentioned above are designed to generate a saliency map for particular class, making them inefficient in implementation due to the ((C) computational complexity or dependency on gradients.
gradient-based, and
B PROOF OF THEOREM
We first restate the theorem:
Theorem (Theorem restated). For a shallow GCN module, Equation (8) can be approximated by
Σ
Saaofflo p(laa)"
(12)
DE
Particularly, if LG = 0 and 01 = 0, our method is equivalent to BoVW with a linear classifier.
For further discussion, we first prove the following lemma:
Lemma 1. For random vectors 9 Rdcc drawn independently from multivariate Gaussian distribution NO, I), we have
Es [SWTWg"] W =
(13)
E, [fW W { ||W 12. where || Fl is the matrix Frobenius norm, and W E de xdc is a projection matrix.
Proof. To begin with, we expand term fW Wg as
da dg de ΣΣ>
fag:Wr,
(14)
Therefore, the expectation
dg do do ΣΣ
Ei [[W]] Wg "]
Σ
Wr Wr Wr, [fa] Byg [gc] = 0,
(15)
16

---

Published as a conference paper at ICLR 2023
175
32
150
32
400
64
64
125
96
96
300
100
128
128
200
192
192
100
224
224
32
64
128
160
192 224
32
64
128 160
192
224
(a) Raw vertex embeddings.
(b) Last layer embeddings.
Figure 6: Vertex embedding similarities between vertex v in the instance IR-Graph and vertex in the category IR-Atlas, which are extracted from the original vertex embeddings and output from the last GraphConv layer. We show the average similarity over 1000 samples on the Caltech- 101 dataset with visual vocabulary size of 256.
while
da
dc
de
E, [fW
[FWT
W
Ey
ffftt
1=1
6==
1=1
E
r.s Js
2
(16)
1=1
de
da
||
r5a
Particularly, if W is an identity matrix
E, [fW W = dg. Further, if the components of matrix W are drawn i.i.d. from Gaussian distribution (0, 1) and are independent with f.
(17)
Byw [FW [ =fc do.
(18)
Now proof Theorem
Proof. We start with simple case that the depth of GCN module is zero.
As G and & share the same original vertex embeddings, we have
Σ
10) (O)
Σ
(19)
(L.))EVXV
According to Lemma 1, the expectation of the same vertex term E [ff de is more significant than different vertices terms X( _!0)' for u + v when and & are constrained with sparsity.
In Figure 6(a), we visualize the vertex similarities f! ((0) showing that the similarities between different vertices are significantly lower than that of the same vertex.
17

---

Published as a conference paper at ICLR 2023
Consequently, Equation (8) in the case of LG 0 can be simplified
Σ A 1
(20)
where 2- V n V is the set of shared vertices of and G.
With an extra assumption that 01 = 0, As therefore represents the count of visual word ø. Combing the learnable term و ( as wa. the prediction of the matcher is
= AWF.
(21)
where Wie E R is the learnable projection matrix a linear classifier.
Now, we prove the whole theorem. For different vertices E V and E V, the summation term can be rewritten as the aggregation form with feature output from the
Su.0 = Anda a(L.)) fe e(Taa)
T
(LG - 1)-th layer:
8u Audo
Σ
eunteus jlbo - w((t))((ta) f
(22)
(o)))N(())xN((()))
where (u(u) represent the neighbors of vertex in Graph & and with itself, and C the self loop.
With the assumption that f( and f/ f((0-1) are whitened random vectors (we say random vector x E Rd is whitened if the components are zero mean and independent from each other, which can be achieved by normalization), the summation term fl f(LC-1)) with vertex i will be significantly larger than those with different vertices. We further illustrate the vertex similarities output from the final layer in Figure 6(b), showing that f(lta) s((ca) is conspicuous for vertices u =
Corollary 1. Consequently, Su, is significant if (u(u) and Na(u) have shared vertices, relative large edge weight product &u, and vertex weight product > which means and have similar local structure, particularly in the case that and u are the same vertex.
In conclusion, Equation (8) can be simplified the summation of the joint vertices & V VV the instance and category graph G and G.
C TRAINING ALGORITHM
The training algorithm of our proposed SchemaNet shown Algorithm with initializing IR-Atlas and sparsification.
D EFFECTIVE RECEPTIVE FIELD OF VITS
In this section, we discuss the effective receptive field (ERF) of ViTs, which is crucial as the visual token in the intermediate layers of ViT's may relate to other positions due to MHSA, affecting the interpretability of the ingredients (visual word semantics). Although Luo et al. (2016) propose to measure the ERF for CNNs, it cannot be directly implemented to Transformer- section, we propose relatively simple yet effective approach to measuring the ERF of ViTs.
models. In this
Definition (Transformer ERF). Supposed that visual sequence E R is fed into a VIT backbone with N layers, is randomly chosen anchor token in X, and E Rd is the random vector drawn from Gaussian distribution ( I) normalizing to the unit vector. The Transformer ERF is the average Euclidean distance between the closest token y in the 2D token array and I. so that when y is disturbed to ÿ = y + e the output change of token x, is less than a predefindd threshold 8, > 0.
18

---

Published as a conference paper at ICLR 2023
Algorithm 1 SchemaNet optimizer with initialization and sparsification.
Input: D {2 y/ va)]? ) the training dataset with D samples; 9 { GG | initial IR-Atlas; Backbone(-): the ViT backbone; Matcher(s; ): the graph matcher; L2 the visual vocabulary; O the of all trainable parameters; d;: the sparsification threshold.
1: procedure INITIALIZZATIO((D,, Qt)
2:
Sample a subset C as the probe dataset.
3: , Sc t- Ø, 9 +- Ø
4:
for (x. y) E D do
5:
G FEATTGGRAPH(BCcconn((()) !t)
6:
S, + S, UG
7:
end for
8: for I...., C do
9: end for
t AVERAGE(S.)
> compute average graph for each category
11: return &
end procedure procedure TRAINING(S, D, Ω2) for (z, v) E D do
14:
15:
for G.j E É do
Removing redundant edges
16:
X < 6 or Й
< 6, then
17:
&.j t NIL
18:
end end for
19:
20:
G - FEAAI2GRPPP(Ccccooo((e)) Q
21:
+ MatcherrG,, 9)
Compute the final loss and gradient Ve w.r. parameters θ. Update parameters θ with AdamW optimizer.
23:
end for
25:
end procedure
26: G INITIALIZATION(] ?)
27:
TRAINING(G, D. Q!)
48
48
1.0 .88
10
-36
-36
-36-
0.8
0.5
12
0.5
12
12
24
24
04
0,4
48
18
36
24
-12
12
24
36
46
18
-66
-24
-12
12
24
36
18
48
-36
-24
-12
12
36
48
DeiT-Tiny
(h) DeiT-Small
(c) DeiT-Baan
Figure 7: The heat maps of the receptive field of anchor token its neighborhood region. Averaged visualization results of 64 random images of three ViT' (DeīT-Ti, DeïT-S, and DeiT--) from Caltech- 101 dataset are demonstrated. The full image size is 224 x 224, and we crop a small region with size of 96 x 96 with the anchor token as the center for better visualization.
Figure 7 shows the visualization heatmap w.r.f. the change of the output anchor with different VIT backbones adopted in our method. As we can observe, for an input image size of 224 x 224, the output tokens with relatively significant changes mainly distributed in the circle with radius of 25. Therefore, in our method, all the ingredients are in charge 50 x 50 patches corresponding to the input image.
19

---

Published as a conference paper at ICLR 2023
E EXPERIMENTAL DETAILS
E. VISUAL VOCABULARY
Thanks the relatively uniform representation of visual seman- ties generated by DNNs, our visual vocabulary size M can be set to around 10 C' with competitive performance. Table shows the default choice of M on different datasets. Results with different vocabulary sizes are presented in Appendix G. Furthermore, all the intermediate features are extracted from the 9-th layer the backbone ViTs, including DeiT-Tiny, DeiT- Small, and DeiT-Base.
Table 2: Visual vocabulary size.
Dataset
Size M
CIFAR-I0
128
CIFAR-100 Caltech- 101
1024
1024
E.2 IR-GRAPH
Before an instance-level graph G and a category-level graph G are fed to the graph matcher, their vertex weights and edge weights are normalized follows. The vertex weights in A are divided by the sum of A
A
Anorm
(23)
E A
The adjacency matrix E' is divided by the row-wise summation:
Σ e
Enorm
(24)
Σ qiiL
Finally, the symmetric adjacency matrix Exym is defined as
Esym 2
(Enor + E
(25)
E.3 GCN SETTINGS
Now describe the detailed settings of the GCN module in the graph matcher. We adopt Lay- enNorm (Ba et al., 2016) as Norm(-) function and rectified linear unit (ReLU) as the activation in Equation (7). Besides, the embedding dimension de is set to 256 for all the experiments.
E.4 TRAINING DETAILS
The matcher and IR-Atlas in our method are optimized by AdamW (Loshchilov & Hutter, 2019) with learning rate 10-3 3 weight decay of 5 x 10-4 and cosine annealing as the learning rate decay schedule. We implement our method with Pytorch (Paszke et al., 2019) and train all the settings for 50 epochs with the batch size 64 on one NVIDIA Tesla A A00 GPU. All input images are resized to 224 x 224 pixels before feeding to our SchemaNet. We adopt ResNet-style data augmentation strategies: random-sized cropping and random horizontal flipping.
F ADDITIONAL RESULTS
This section contains additional experimental results, highlighting the efficiency, robustness, and extendability of our proposed schema inference.
F.1 RESULTS ON IMAGENET
We further implement SchemaNet on mini- Aani-ImaeeNet (Vinyals et al., 2016) and ImageNet- (Deng al., 2009). We adopt DeiT-Small as the backbone, and the visual vocabulary size is set to 1024 for mini- Mmii--mmaeNNe and 8000 for ImageNet- lk (due to the memory constraint). Particularly, as implementing 1000 fully-connected category-level IR-Graphs is expensive, we keep at most 500
20

---

Published as a conference paper at ICLR 2023
valid vertices for each class while all other vertices are pruned during the training process based on the initialized vertex weights. All other settings are identical to the experiments on Caltech- 101. The results presented in Table 3, drawing consistent conclusions as the main results in Table 1.
Table 3: Comparison results (top-1 accuracy) on ImageNet-Ik IK and mini-lmageNet.
Dataset
Base
Backbone-FC
BoVW-Deep
BagNet
SchemaNet
SchemaNet-Init
mini- ImageNet 95.35
90.04
90.24 58.95
85.64
89.40
91.02 74.05
ImageNet-Ik
79.90
69.23
68.92
F.2 COMPARISON RESULTS USING ATTRIBUTION METHODS
Instead of using raw attention extracted from the backbone in Equation (3), we further employ feature attribution methods for computing the ingredient importance. Specifically, we compare the results using raw attention, CAM (Zhou et al., 2016), and Transformer-LRP (Chefer et al., 2021) in Table 4. We can observe that using raw attention is superior to the attribution methods in terms of accuracy and rumning time. Such results can be explained that: (1) the attribution methods compute saliency map for each class (particularly, ViT-LRP conduct C-times backpropagation, consuming enormous time), which is significantly slower than implementation; (2) further, the backbone predicts soft probability distributions other than one-hot targets, the computed saliency maps for similar categories will be highlighted some extent, impairing the graph matcher.
Table 4: Comparison results (top-l accuracy and running time) of using different attribution methods on CIFAR-10 and CIFAR- datasets.
Raw Attention
CAM
ViT-LRP
Dataset
Acc
Time
Acc
Time
Acc
Time
CIFAR-10
95.96
0.8h
93.58
3.6h
91.28
53.8h
CIFAR- 100
78.45
3.7h
77.55
15.3h
F.3 ADVERSARIAL ATTACKS
To analyze the robustness of our proposed SchemaNet, we evaluate the pre-trained SchemaNet (from ImageNet- Ik described in Appendix F.l) on two popular adversarial benchmarks: ImageNet-A (Djo- longa et al., 2021) and ImageNet-R (Hendrycks et al., 2021). The comparison results are shown in Table 5, revealing that our method is more robust than the baselines.
Table 5: Adversarial attack results (top-1 accuracy) on ImageNet-A and ImageNet-R datasets.
Dataset
Backbone-FC
BoWW-Deep
BagNet
SchemaNet-Init
ImageNet-A 5.41 31.21
7.36
5.53
13.05
ImageNet-R
24.17
30.93
33.46
F.4 EXTENDABILITY
We evaluate the extendability of our method by extending trained SchemaNet to unseen tasks, while keeping the original framework frozen. As such, only the catcgory-level FR-Graphs for the new tasks optimized and inserted into the original IR-Atlas.
Specifically, the tasks are disjointly drawn from Caltech-101 dataset. The "Base" task has 21 classes, and task to 4 has 20 classes. The backbone model, i.e., DeiT-Tiny, is trained on the "Base" task and then is kept unchanged for the new tasks.
The experimental results are presented in Table 6, revealing limited performance degradation for incoming new tasks. Such results show that the graph matcher is only responsible for evaluating the graph similarity, while the category knowledge is stored in the imagination, i.e., IR-Atlas.
21

---

Published as a conference paper at ICLR 2023
Table 6: Top-1 accuracy results when extending SchemaNet unseen tasks. The accuracies evaluated on the corresponding task, and the average accuracy over all the tasks is given as well.
Average
Base
Base
97.57
97.57
Task
+Task
94.68
95.95
93.69
Task 2
Task
93.36
95.14
93.06
92.11
Task 3
Task
93.63
95.55
93.06
92.47
93.75
Task 91.49
+ +Task
92.37
95.14
92.74
91.76
90.42
80.0
78.5
78.5
75.0
78.1
78.3
70.0
77.7
78.1
63.0
77.3
77.9
76.9
77.7
06.5
77.5
256
512
1024
2048
2.
5
00.25
0.50
0.75
1,00
M
LG
(a) SchemaNet accuracy with ferent visual vocabulary size M.
(b) SchemaNet accuracy when us- ing GCN with various depth Lg.
(c) Sensitivity analysis of hyperpa- rameters involved in Equation (11).
Figure 8: Ablation study and sensitivity analysis hyperparameters of our proposed SchemaNet on CIFAR-100 1-0 backed with DeiT--Tiny.
F,5 FEATZGRAPH EVALUATION
We here present a performance bottleneck our method, i.e., the Feat2Graph module, in which feature ingredient array is converted to IR-Graph without benefiting from GPU parallel acceleration. Specifically, as duplicated visual words may exist after the feature discretization, which happens from time time, implement this module in C++ achieve better random access performance. The experiments are conducted on one NVIDIA Tesla A100 GPU platform with AMD EPYC 7742 64-Core Processor.
In Table 7, we show the average running time of each component with an input batch size of 64, We can see that without acceleration in parallel, the running time for "Feat2Edge" is significantly longer than others (about 1.75 ms per image). However, the inference time is still acceptable for real-time applications.
Table 7: Time costing (ms) the components in SchemaNet with input batch size 64.
Dataset
Backbone
Feat2Edge Feat2Vertex
Matcher
CIFAR-10
4.89
7.00
19.8
1.95
CIFAR-100
4.31
9,50
101.7
1.30
Caltech-101
4.12
8.12
111.7
1.42
G ABLATION STUDY
G.1 VISUAL VOCABULARY SIZE
Figure 8(a) shows SchemaNet accuracy when using different sizes of visual vocabulary. We can observe that with relatively small size, e.g., M 256 CIFAR-100 the performance is still competitive (about 7% absolute degradation). Besides, for the case of M 2048, however, the performance suffers from low generalizability of the vocabulary.
22

---

Published as a conference paper at ICLR 2023
1.50
1
1.25
2
4
1.00
Weights
0.75
0.50
0.25
0.00
30
35
50
Epochs
Figure 9: Learning curves of 0112 and $1,2 with DeiT-Tiny backbone Caltech-101 dataset.
G.2 NUMBER OF GRAPHCONV LAYERS
The effect of using different depths of GCN is illustrated in Figure 8(b). As LG increases to 5, the accuracy on CIFAR-I00 decreases rapidly, yielding that deep GCN matcher impairs the graph matching performance, which has been explored due to the over-smoothing caused by many convolutional layers (Zhao & Akoglu, 2020; Li al., 2018) 8). In our method, however, the graph matcher with only two layers of GraphConv is proven to be adequate for high performance and interpretability.
G.3 ABLATION OF WRIGHT COMPONENTS
The effect of XCLS and Aue defined in Equa- tion (3), and e and adi defined in Equation (6) shown in Table 8. By setting the correspond-
Table 8: Ablation study of the components defining
the vertex and edge weights with DeiT-Tiny as the backbone DNN.
ing weight (01.2 and 31,2) to zero once at time,
Accuracy
able analyze the components individually. In general, removing any term will lead to varying degrees of performance degra- dation. More significantly, when removing the term A bag the accuracy decreases by 6.03% on CIFAR- 100, revealing that the statistics of the ingredients help filter noisy vertices.
Settings
CIFAR- 10
CIFAR- 100
Learnable a1,2, $1.2
95,96
78.20
Fixed a₁ = 0, 022 =
95.88
75.91
Fixed an 1, 022
95.68
72.17
Fixed 8n
95.85
77.31
Fixed 81 = I, 1,83 0
95.89
77.84
Fixed 01,2 11,.2 0.5
95.92
75.62
The learning curves of the learnable weights are shown Figure 9, from which we can observe that the model tend to adopt relatively larger 02 (for visual word count) while keep XLL term for filtering noisy and background ingredients.
H SENSITIVITY ANALYSIS OF HYPERPARAMETERS
H.1 SPARSIFICATION THRESHOLD
Table 9 presents the SchemaNet-Init top-1 accuracy trained with different sparsification threshold 6,. We observe that appropriate value of 6,, e.g., 0.01, will boost the model performance.
Table 9: Sensitivity analysis of sparsification threshold &,.
d,
0.001
0.01
0.02
0.05
Ace
77.27
77.96
78.45
78.43
77.51
76.92
23

---

Published as a conference paper at ICLR 2023
(a)
Instance IR-Graph
Ingredients
B24
Rooster
A1B4 0.25
+100
0.20
1140
Y17
1240
9.3 (M103) 9.5 (#240)
18.8
AIP
A36
129
103
100
Tbis
9.9
cloa
46
10T
1FT
345
33
Category: Ibis
1.6 1.2 14.3 (EUIa)) 2.0 (L137) 17.9
(b)
Instance IR-Graph
146
Ingredients
PLT
A6G Q31
Liama
+150
265
22:0
3.4 (#17)
6.6 (#46)
(4.9 (#240)
CLRS
240
204
46
214
2344
204
234
248
740
234
M4
F6
240
2:0
249
240
340
200
Panda
637
240
240
340
THE
204
A213
240
Category: Panda
(#17) 15.8 (H204) 3.0 (#234) 21.3
Figure 10: Illustration of misclassified samples. Better view in color.
H.2
LOSS WEIGHTS
Figure 8(c) shows the sensitivity analysis of A and de in Equation (11) that constrains the complexity of AR-as..s We evaluate the top-1 accuracy performance on the CIFAR- 100 dataset with DeiT-Tiny as the backbone. The curve of X shows that neither dense nor extreme sparse IR-Atlas impairs the performance, while method is more robust when changing Xe
I MORE VISUALIZATIONS
I.1 VISUALIZZTION AND ANALYSIS OF MISCLASSIFIED EXAMPLES
Figure shows two misclassified examples along with the classification evidence. In Figure 10(a), the object is misclassified to a fine-grand fine-grained category "rooster" because of a noise ingredient #240, which should be #103. Unfortunately, ingredient #240 is crucial vertex the rooster's IR-Graph, contributing about 9.5 absolute gains in the logit, leading to misclassification. Figure 10(b) shows more complicated example. We can observe that instead of discretizing the appeared human face to the "face" ingredients, the backbone provides features closer #17, which is more similar to the animal's face. Moreover, some part of the object's body assigned to #240 rather than the panda's 's body (#204). Thus, it creates remarkable pattern, iee., the interaction between #17 and #240, which
24

---

Published as a conference paper at ICLR 2023
the crucial local structure in the llama's graph. As result, #240 and its local structure contribute about 14.9 gains in the logit. Although our schema inference framework capable of revealing why an image is misclassified by highlighting the key points, in future work, must explore a more compatible feature extractor that can generate more robust local features.
1.2 VISUALIZATION OF INGREDIENTS
In Figure 11, we visualize the whole set of ingredients on Caltech- 101 for visualizations Figure and Figures 12 to 14. We extract 256 ingredients on Ca0tech-101 for better view. For each cluster center generated from k-means clustering, we select the top-40 tokens that are closest to and show the corresponding image patch (50 x 50 in pixel, delineated in Appendix D). We can observe that image patches of the same cluster share the same semantics.
1.3 VISUALIZATION OF IR-ATLAS AND INSTANCE IR- RR-GRAPHS
We provide more visualization examples on Caltech- 101 dataset, shown in Figures 12 14.
25

---

Published as a conference paper at ICLR 2023
15c
05c
DEC
BEL LEC
SLL
Tiec
STC
tec
CEL
stc
DOC
201
5OC
CBC
TOT
822
Figure II: Ingredient visualization on Caltech- 101 datasee Please zoom in for better view
P2
retc
517
212
60
100
4RT
561
CEL
761
06
LBE
SRL
CRT
TSt
A/1
S2T
141
49C
951
Tet
AST
25t
151
Est
aSt
BEL
IST
26

---

Published as a conference paper at ICLR 2023
IR-Atlas
Instance IR-Graphs
Figure 12: Examples of the learned AR-Atlas and instance IR- IR-Graphs randomly sampled from five categories ("bass", "bonsai", "emu", "cuphoniumm" " and "grand piano") on Caltech-101.
27

---

Published as a conference paper at ICLR 2023
IR-Atlas
Instance IR-Grs
Figure 13: Examples of the learned IR-Atlas and instance IR- Grarrph randomly sampled from five categories ("hawksbill", "ibis", "kangaroo", "leopards", and "Ilama") on Caltech- 101.
28

---

Published as a conference paper at ICLR 2023
IR-Atlas
Instance IR-Graphs
Figure 14: Examples of the leameed IR-Atlas and instance IR- IR-Graphs randomly sampled from five
categories ("okapi", "rooster", "sunflower", "water lilly", and "wild cat") on Caltech-101 101.
29


Reviews:
[{"review_id": "gXnAFuvXqU", "reviewer": "Reviewer_zHVw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies interpretability for image classification. The authors propose a schema-based architecture (SchemaNet) based on vision transformers, which first extract features from pre-trained backbone and form a so-called ingredient relation graph (IR-Graph) with a feature to graph (Feat2Graph) module. The IR-Graph is matched with a set of imaginations (so-called IR-Atlas) with a matching network for making predictions. To test the SchemaNet, the authors conduct experiments on CIFAR10/100, and Caltech101, in comparison to the baseline without SchemaNet, bag-of-visual-words (BoVM) representation, BagNet, and show that SchemaNet performs better or on par with the baseline. Further, the authors also show some visualization of the learned IR-Graph and IR-Atlas.", "summary_of_the_review": "The paper studies interpretabiltiy for image classification and propose a new method with learnable graphs. Experiments are given to showcase is classification performance in CIFAR and Caltech. However, the comparison to existing related methods is not comprehensive. The analysis and evaluation about the model interpretabiltiy is also relatively unclear and weak as compared to existing works. ", "strength_and_weaknesses": "+ The proposed method of learning graphs to make the deep neural network more interpretable is interesting and novel. The idea makes a difference as compared to existing works in XAI. \n\n- Although the proposed method brings some new aspects to XAI, the comparison in experiments is weak. The authors mainly compare the proposed approach to a couple of relatively simple methods such as the bag-of-word representations. However, there are quite some other existing representative methods such as class activation map (CAM) [a], and ABN [b], and [c] for vision transformer. Some of these existing methods [a,b] also contain learnable interpretability modules in the deep neural networks and provides interpretable predictions for image classification. \n\n- The evaluation and analysis of the interpretability is also relatively unclear and weak. Most of the experiments are conducted to show the accuracy on classification. However, there is not much analysis about the model interpretability. The only visualization in Figure 5 to shown the interpretability is unclear. For instance, what does the different colours refer to? How does these interpretability compared to human interpretability? How does these interpretability contribute to the model predictions? It is suggest to follow existing literature in XAI [c] to give both quantitative and qualitative evaluation about the interpretability. \n\n[a] Learning Deep Features for Discriminative Localization, CVPR2016\n[b] Attention Branch Network: Learning of Attention Mechanism for Visual Explanation, CVPR2019\n[c] Transformer Interpretability Beyond Attention Visualization, CVPR2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Eg1nel6Ref_", "reviewer": "Reviewer_BvuD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the SchemaNet, an inference paradigm based on the cognitive concept of schema for inducing explainability of  trained image classification Vision Transformers. It first builds a codebook of by k-mean clustering running on the collection of visual tokens extracted from a probe dataset. With the codebook, it builds the ingredient relation graph (IR-Graph) at the instance level using a Feat2Graph module, and at the category level. It then exploits graph convolution network for computing the graph similarity of instance and category IR-Graphs for explainable classification inference. In experiments, the proposed method is tested using DEIT on CIFAR10/100 and Caltech-101 with promising results obtained. ", "summary_of_the_review": "Overall, the proposed method is an interesting and promising approach. The reviewer would like to see the rebuttal on the aforementioned weaknesses. ", "strength_and_weaknesses": "Strength:\n\n1) The proposed method leverages the ensemble-level information (the codebook and the category IR-Graphs), as a post-hoc processing step to address explainable image classification. \n2) The proposed method is able to explain the image classification results in terms of graph visualizations. \n3) The proposed method obtains promising results on three datasets (CIFAR10/100 and Caltech 101). \n\nWeaknesses:\n \n1) This paper aims to address explainable image classification. There are a few aspects that could be addressed to consolidate the proposed method.  It will be better to show the inference results of misclassified examples to check if the inferred IR-Graphs can indeed provide insights of why they are misclassified.  It will also be interesting to compare with the recent methods explaining ViT models (e.g., Chefer  et al, Transformer interpretability beyond attention visualization, CVPR21). It will also be important to see if the proposed inference method is potential more robust to adversarial attacks as a sanity-check of the targeted explainability.  \n  \n2) Since the foundation of the proposed method is built on the codebook computed via k-mean clustering running on the collection of visual tokens extracted by a feature backbone (DEIT) from a probe dataset, it will be better to show the learned mean image patches to see if they make sense visually.  Individual image patches are shown in the IR-Atlas (e.g. Fig.5). How are they selected to represent each codebook index?\nIt will also lead to the potential scalability concerns.  The proposed method is tested on relatively small datasets, rather than the ImageNet-1K. It will be interesting to see ImageNet results if possible.  How large will the codebook need to be, as well as the training and inference cost  of the SchemaNet?\n\n3) The proposed graph similarity computation via GCN is not clearly explained. Vertex embeddings are used and trained by GCNs. It seems that the backbone only provide initialization of the IR-Graph structures. The resulting explainablity may thus have a gap with the backbone. Will it be possible to use the visual tokens as features for the vertex in the IR-Graphs?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5mJOJb12CF", "reviewer": "Reviewer_S8GX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nSummary: This paper proposes a new machine model from the perspective of human cognition. This is very good. However, the system in this paper is too complex and is not concise and elegant enough. Its usefulness (especially in real environments or large datasets) remains to be further verified.\n", "summary_of_the_review": "\n\nSee \"Summary Of The Paper.\" I think this paper is a borderline paper, leaning toward being accepted.\n\n", "strength_and_weaknesses": "\n\n(Positive) The authors think their method is an analogy with the human reasoning mechanism via impression matching. Such motives are very reasonable. I like it.\n\n(Negative) The conceptual clustering shown in the lower left corner of Figure 2 is very interesting. It is the basis for the soundness of the method proposed in this article. I hope the authors give more examples (if possible, show all of them), especially in large resolutions.\n\n(Positive) I am happy that, instead of relying on deep features, the desired outputs from schema rely on visual word relationships.\n\n(Positive) Thanks to the authors, Figures 1 and 2 are very clear and easy to understand.\n\n(Negative) In my opinion, the authors should validate the effectiveness of their method on more standard data such as ImageNet.\n\n(Negative) Authors should compare their machine-learned knowledge with human knowledge and make it clear whether their machine-learned knowledge is more advanced than human knowledge. Is it worthwhile for humans to learn this independent knowledge from these machines?\n\n(Negative) The method proposed in this paper is an overly complex engineering system, and there are many factors to be analyzed. The definition of Feat2Vertex (Equation (3) is necessary for workability but not elegant. There should be a tradeoff between the similarity at the semantic level as well as the adjacency relationship at the spatial level in Feat2Edge, which is also not elegant.\n\n(Negative) The authors assign a dG dimensional random vector x ∈ R 174 dG to each vertex before feeding the graph to the GCN. Does this mean that there is randomness in every inference? What do you think of this randomness?\n\n(Negative) In order to avoid the high complexity of fully connected graphs for IR-Atlas, this paper averages each instance graph. This involves engineering. Please analyze the impact of hyperparameters such as thresholds.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4i0lSyOQDL", "reviewer": "Reviewer_bkvM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a new network architecture, called a SchemaNet, that turns an image classification problem into a graph matching problem. In particular, a SchemaNet consists of: (1) a pre-trained visual transformer backbone, that extracts image features from an input image; (2) a Feat2Graph module, that turns deep image features of an image into an instance IR-Graph, where vertices represent concepts and edges represent relations between concepts; (3) a set of IR-Graphs (called \"IR-Atlas\"), each of which represents prototypical concepts and their relations for each class; and (4) a graph neural network that computes the similarity between the instance IR-Graph and each of the class IR-Graphs in the IR-Atlas. The final prediction is based on the latter similarity -- the class whose IR-Graph has the highest similarity with the instance IR-Graph is used as the predicted class of the input. The authors compared their SchemaNet with a number of baseline models, including the base transformer, bag-of-visual-words over deep image features, and BagNet, on CIFAR-10, CIFAR-100, and Caltech-101, and found that their SchemaNet achieved competitive classification accuracy as the baseline models.", "summary_of_the_review": "Based on the strengths and the weaknesses discussed above, I believe that this work has value, but needs improvement in explaining how to interpret the learned class IR-Graphs and how an instance IR-Graph is similar to a particular class IR-Graph. Without such interpretability, it is difficult to see the motivation of this work.", "strength_and_weaknesses": "Strengths:\n\n- Novelty: The proposed SchemaNet is novel -- the idea of turning image classification into a graph matching problem has not been explored before.\n- Clarity: The paper is in general clearly written.\n\nWeaknesses:\n\n- Motivation: I suppose that the motivation for turning image classification into a graph matching problem is to build a model with a more interpretable reasoning process. However, the paper did not discuss how interpretable the visual words and the class IR-Graphs are. In particular, what are the meanings of the visual words? Do the learned class IR-Graphs make sense? The paper would be significantly better, if the authors can include interpretations of some class IR-Graphs. Without such interpretations, it would be difficult to see why we want to turn an image classification problem into a graph matching problem.\n- A related issue: Since the authors used a graph convolutional neural network (GCN) to match an instance IR-Graph with each of the class IR-Graphs, and a GCN is not interpretable in general, how would the authors explain why an instance IR-Graph (representing a particular input image) is similar to a class IR-Graph? Again, without interpretability, I fail to see why we want to turn an image classification problem into a graph matching problem.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster