Formal Conceptual Views in Neural Networks
Authors: Johannes Hirth
Tom Hanika
Venue: Submitted to ICLR 2023 (2023)
Keywords: Conceptual Scaling
Explainable AI
Global Explanation
Formal Concept Analysis
Lattices

Abstract:  Explaining neural network models is a challenging task that remains unsolved in its entirety to this day. This is especially true for high dimensional and complex data. With the present work, we introduce two notions for conceptual views of a neural network, specifically a many-valued and a symbolic view. Both provide novel analysis methods to enable a human AI analyst to grasp deeper insights into the knowledge that is captured by the neurons of a network.  We test the conceptual expressivity of our novel views through different experiments on the ImageNet and Fruit-360 data sets. Furthermore, we show to which extent the views allow to quantify the conceptual similarity of different learning architectures. Finally, we demonstrate how conceptual views can be applied for abductive learning of human comprehensible rules from neurons.  In summary, with our work, we contribute to the most relevant task of globally explaining neural networks models.

Under review as a conference paper at ICLR 2023
FORMAL CONCEPTUAL VIEWS IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Explaining neural network models is challenging task that remains unsolved in its entirety to this day. This is especially true for high dimensional and complex data. With the present work, we introduce two notions for conceptual views of a neural network, specifically a many-valued and a symbolic view. Both provide novel analysis methods to enable human Al analyst to grasp deeper insights into the knowledge that is captured by the neurons of a network. We test the conceptual expressivity of our novel views through different experiments and Fruit-360 data sets. Furthermore, we show to which extent the views allow to quantify the conceptual similarity of different learning architectures. Finally, we demonstrate how conceptual views can be applied for abductive learning of human comprehensible rules from neurons. In summary, with our work, we contribute the most relevant task of globally explaining neural networks models.
the ImageNet
INTRODUCTION
Neural networks (NN) are known for their great performance in solving learning problems. However, these excellent results are almost always achieved at the price of human explainability. This problem is addressed in research and practice from different standpoints. There are calls to refrain from using NN for important problems and to rely on explainable methods, even if they give worse results in terms of accuracy (Rudin 2019). The second major direction is to develop methods for explaining NN models. Such explanations can be classified as local explanations, i.e., why a particular data point was treated in a specifie manner (Ribeiro al. (016), and global explanations, i.e., approaches for explaining the whole NN model. The latter can be achieved, e.g., by mapping the NN to an explainable surrogate. A common approach for locally explaining NN models is highlight activation some hidden layer (Fong& & Vedaldi 2018) or, if possible, project this inversely. For flat data, e.g., images, this is viable approach since an essential explanatory component, the human, can be integrated into the process. This is not the case for high-dimensions or complex data. Global approaches are more difficult, in particular for high-dimension, and therefore less frequent. typical idea is to find an (explainable) surrogate for a NN, e.g., symbolic regression (Alaa & van der Schaar (019)).
We answer to the still growing interest for global explanations procedures for NN models by intro- ducing a novel intermediate space, called (symbolic) conceptual views. We demonstrate how NN models can represented in these views and how surrogate training, e.g., with decision trees, can profit from this. We further demonstrate how to compare NN models, e.g., when derived from diverse architectures, using Gromov- Wasssrrsein (Mémoli, 2011) distance within the views. Moreover, we demonstrate how symbolic conceptual views can be used to represent NN models with formal concept lattices (Ganter & Wille 1999) and profit from its human-centered approach for explainable data analysis. Finally, we show by application of subgroup discovery how numan-compprhensiile propositional statements can be derived from NN models with the use of background knowledge. This allows us extract global rules in form of propositional statements using the neurons of the NN,
2 RELATED WORK
Several approaches aim to provide insights or explanations into neural networks. Many of them highlight parts of the input that were relevant for a particular prediction (Ribeiro et al., 2016). so called local explanations. Those however, rely on the user's capability to comprehend input data representations. Hence, this approach is infeasible for higher dimensional learning problems. To overcome this limitation, the SOTA is to interpret models using symbolic concepts, approach of

---

Under review as a conference paper at ICLR 2023
neuro >symboli AI (Sarker et al. 2022). For example, Mao et al. (0119) ,. Asai & Fukumaga (0188)
FnnnFong & Vedaldi (018) introduce methods which classify the imputs of a model to pre-defined concepts. Hence, they require manually created input representations for all pre-defined concepts, in contrast of extracting them automatically. Particularly successful is TCAV (Kim et al. 2018). 8),
which predicts the importance of user-defined concepts. The above complemented by methods that automatically detect concepts for a given set of input/output pairs through identifying similar patterns of input samples at a given layer, e.g., ACE (Chorbani et al. 2019). So far these methods do detect only particularly outstanding concepts. Recent works try to estimate to which extent detected set of concepts is capable to approximate the model (Yeh et al.] [020). This approach, however, emphasizes classification performance and not explainability. i.e., concepts that are important for explanations may be omitted. This is in general true for surrogate based procedures that were not designed towards human comprehensibility (Alaa & van der Schaar (019). Moreover, recent study shows that the translation of initial layers does often correlate with random layers or gradient detectors in the input (Adebayo et al.] 2018). The most crucial downside of the automatic detection methods above is that although they provide symbolic concepts, these do not have to be interpretable. The overall principle of our approach is based on the fact that substantial portion of the input data is aggregated and represented in the last hidden layer (Clark et al. 2019 Korbar et al. 2017).
global interpretation of the NN needs a decoding into human comprehensible symbolic view. A mathematical method for human comprehensible conceptualizations in the language of algebra is formal concept analysis (FCA) (Ganter & Wille 1999 Wille 1982). In particular its well-claborated conceptual scaling theory (Ganter & Wille 1989) provides an extensive tool-set to analyze NNs.
This tool-set set enables both the translation of neural representations into symbolic space and further on the translation of this space into a human explainable space (Hanika & Hirth 2022 2021).
THE VIEWS OF NEURAL NETWORKS
We introduce in the following two notions of conceptual view of a neural network, in detail a many- valued and symbolic view. Both provide novel methods to enable human Al analyst to grasp deeper insights into the knowledge that is captured by the neurons. In addition to that the symbolic view facilitates the application of abductive learning procedures. This results in rules that allow to explain a NN by means of human comprehensible terminology, as well as, in terms of the neurons.
Let N be the set of neurons of the last hidden layer of NN. We interpret NNs as a function that maps input objects g E G, that are represented g (u1..... Um)) E R" to outputs in [0, 1|(c for classes C. The parameter m specifies the number of input features (see Figure I]. Naturally, we can interpret each neuron E N as a function by itself from the input layer up the activation of n, i.e., R + R. The output neurons can be characterized analogously by map e R + RR With Wi, we address the weights connecting the output neuron &; E C with hidden neuron ny N.
Definition (Many-Valued Valued Conceptual View) Let NN be a neural network, C its output classes and N = { } the neurons of the last hidden layer. We define the many valued conceptual view as V (O, W), where O E G| with value at (i,j) equal to the activation n. (g))),, called Object View, and W E [[X with value at (i,5) equal to the weight W..j. called Class View.
To give a short motivation: With the object view O we want to study the activation of the neurons N given an object g. Complementary, with the class view W. investigate the relation of the neurons N to the outputs e E C by their corresponding weights wxy. For example, we refer the reader to Figure [ which depicts the object and class view (right) of the network (left). In this, we find that ng (o,) is greater than n, (o:). from which we infer that the relation of o, to ny is greater than nı. We want employ the just introduced views comprehend the complete classification that is captured by a NN model. We can represent any object g as a row in the object view matrix, i.e., O(g)= == (n1 (g)............... ..... (g)))). Analogously, we can represent any class as a row in the class view matrix, i.c., W(c)) = (wi,1...., W.,)). The outputs of the NN for class & follow from the term O(g) W(c.) + b, where b bias. This can be rewritten as ((g) (c:) cos(O(g), ((+)) + where cos(O(g) ((a)) is the cosine value of the angle between O(g) and W(c).. Thus, to understand the inner representation of the classes C within the NN, it may be reasonable to grasp the objects and classes in the same space and classify objects using similarity measures. Using this approach can introduce an object-class distance map
2

---

Under review as a conference paper at ICLR 2023
dy C (d()))))((((())) W(c)), where sensible choice for is cosine similarity the Euclidean distance. We will investigate both in Section 4.1 Hence, using dy and similar distance maps for G x G and C x C, on can derive pseudo metric space (GUC, dy). From this representation of G and C' one can infer simple classification map, e-g., by applying I-NN classification.
Similarity of neural networks Conceptual views enable direct comparison of NNs. One can employ the Gromov-Wass Wasserstein distance (Mémoli 201T), as experimentally demonstrated in Sec- tion We contrast our results with a baseline of model fidelity. We may note two important facts. First, our approach for similarity is comparable to the recent idea of relating neural networks to particular kernel spaces (Shankar et al. 2020, Lee et al. 2019). This enables us to study how objects are hierarchically clustered such space. We may stress that our notion does not consider how objects are mapped into this (kernel) space, but rather investigates the space itself. Second, the used GW distance is invariant with respect to permutations of the many-valued conceptual views.
3.1 SYMBOLIC VIEW
The next step is to scale the vanyevalued conceptual view into a symbolic space. We employ conceptual scaling (Ganter & Wille 1989) from formal concept analysis (FCA), where data is represented in a formal context K- (G, M, I). There, G' is set of objects, M set of attributes and I _ G x M is an incidence relation, where (g. m) E I indicates that g has attribute m. From arise two derivation operators (-) PPG)) - P(M), with A' {m E M Vgg A : (g,m) E T}, and analogously (.)² P(M) + P(G), with B {o E G Vm E B: (g. m) E /}. Applying many-valued conceptual scaling to many-valued data set yields a formal context. this first attempt, we decided for dichotomic scaling, using thresholds for the object view åg and the class view dy (see Figure IT As final remark before we introduce the symbolic conceptual view on NN we want to point out simple but powerful observation. The to be employed relational structure is invariant with respect to row- or column permutations in the related many-valued conceptual view (Definition D
Definition 2 (Symbolic Conceptual View) Let V = (O, W) the many-valued conceptual view of NN and let 60, by be threshold values. We define the symbolic conceptual view Va= (Op, Wn) by
Op = N,,, with nz) Io : n2 > dσ and (Symbolic Object View)
(g) E Io : n2 (g S 60
W3 = NU, Iw), with (c,, n)) E Iw : Ws.j > dw and
(Symbolic Class View)
(c, Iw : Wi.i S dw
We introduced with N == (n n N a set of artificial symbols and used them as defined above.
This definition allows for constructing human comprehensible explanations given a background ontology, e.g., in form of human annotations of the objects or classes. We exemplify that in Figure using the formal context Sy that employs interpretable features Sm,..... Smy- We provide more details in Section Suitable threshold values dy, 6g depend on the architecture of the to be analyzed NN model. For example, if the activation function is ReLu, the neuron's co-domain positive. Thus it becomes difficult to determine a reasonable á for negative symbols N. as studied in Section 4.3
Neural Network
MV Cone. View
Symbolic Cone.
View
Hidke
Oeria
my
M1.
6
No
79
8W
91
6g
Say
Say
3my
4
Mglos]
Figure 1: A simplified neural network drawing (left), its many-valued conceptual view (middle) and its symbolic conceptual view (right).
3

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTAL STUDY
We support theoretical modeling of conceptual views, in particular the symbolic conceptual view, through an experimental study using common and well known data sets and NN models. First, evaluate the suitability of the in Section + introduced pseudo metric space by classification task. Second, we show how one may compare many-valued conceptual views (different NN models). Third, we demonstrate how to derive human comprehensible representation for NN model, that The code and trained models are available on GitHub
we can employ for explanations in Section 5
4.1 VALUED CONCEPTUAL VIEWS ON IMAGENET
We demonstrate that many-valued conceptual views are capable of capturing large share of a NN model. For this, we all twenty-four NN models from tensorflow that are trained the ImageNet (Deng et al. (2009) data set. The object view is calculated using the test set, i.c., 100k images, of ImageNer used in the ILSVRC (Russakovsky et al., 2015) challenge. In Appendix, Table we compiled basic statistics on these networks and our views. Although we report in columns two and four mean values and their standard deviation, we may stress that we not consider the individual values to be normal distributed.
Evaluate ImageNet Views To evaluate the quality of our views, we compare a one-nearest-neighhoo
(I-NN) classifier on the in Section introduced pseudo matric space (G U N, dy) directly with the NN classification function all 100,000 test images. In detail, we use model fidelity, i.c., count the instances where the 1-NN outputs the same class label as the NN and normalize this number by We differentiate in experiments
the cardinality of the test set. The results are depicted in Table [ between using cosine similarity and Euclidean distance within dy-
Observations We find that the view model is capable achieving high fidelity (see Table The MobilNetV1 model is the only exception. Moreover, we can state that using the Euclidean distance is superior to the cosine similarity in all instances. This is in particular true for the ResNet models, where the difference is up 0.6. Furthermore, for the EfficientNets we notice that there is an almost monotone relation between the number of neurons N (last hidden layer) and the fidelity. All this together suggests that the many-valued conceptual view is meaningful and that classification functions that are based on the resulting psuedo metric space can be used as surrogates for the NN model.
4.2 SIMILARITY OF NEURAL NETWORKS
Based on vanu-valued conceptual view, we can derive for all NN models a pseudo metric space as introduced in Section Hence, given the theory about metric spaces there are different approaches for comparing them. For example, one could compute the distance. However, due to the vast number of data points, any direct computation of the GH distance is infeasible. A different approach, which is still costly, but can be performed for subset of the data, the Wasserseennn (Mémoli (20T)) distance. In Figure (right) we depict the individual distances for all considered models with respect to the class and object view. We employed ten percent of the test data set and applied a uniform probability measure on the data points, i.c., a
(Mémoli (011))
-
Table 1: The fidelity between a NN model their MV conceptual view using I-NN classifier. Find all twenty-four results in Appendix, Table
Model
Cosine Euclidean
VGG16
0.945
0.841
DenseNet201
0.972
0.728
MobilNetV
0.575
0.449
ResNet1
0.999
0.314
EffBÖ
0.944
0.933
B1B77
0.985
0.979
4

---

Under review as a conference paper at ICLR 2023
Figure The similarity of twenty-four neural networks trained on the ImageNet data set. The base-line (left) is pair-wise fidelity between the employed models compared to a similarity using Crosov-Wasserstein distance on class view (right). Find the object view in Appendix, Figure 6
normalized counting measure. We compare our analysis to baseline that is derived from the fidelity measure (Figure 2 left).
Observations From the pairwise fidelity diagram (Figure left), we can infer that almost all models are very distinct with the exception of VGGI6, VGG19, ResNet50 and the EfficientNet instances. In addition we find that the later models become more similar with increasing number of neurons. The similarity plot for the views are different from the fidelity plot. We can visually identify clusters models. These clusters often correspond with similar networks architectures. For example in the object view we observe two clusters. In the class view this clustering is finer.
4.3 SYMBOLIC CONCEPTUAL VIEW
In this section we study thoroughly the activation functions and the number of neurons for a reasonable determination of threshold values in order compute meaningful symbolic conceptual views.
Parameter Study To evaluate the influence of the choice of the activation function as well as the number of neurons, trained one NN architecture several times on the Fruits-360 (Mureşan & Oltean 2017) data set. The used data set contains 67,692 images of 131 types of fruits or vegetables. additional 22,688 images. We train the architecture from the Fruits-360
The test set contains Jusing using all procedure parameters from Mureşan & Oltean (017) and modified the last two hidden layers. For the (last) hidden layer N we vary the size 2dh between 24 and 2 with powers of two. For the layer before that we follow the common approach for smooth decrease in dimension, i.e., we chose 21 10+-1 with 4 < < 10 dependent on the last hidden layer. For activation functions, we studied the impact of ReLu, Linear, Swish and Tanh in all layers. For each parameter setting we trained ten models and computed their respective conceptual views for the test data set. their distributions in Figure 3 Statistics on the quality of the computed views can be seen in Table We tested these distributions against normalization of the column vectors in the views and can report that the reported results are invariant.
experimen
Observations In general, we observe that the distributions for the object views differ quite largely among the different activation functions. From these we found that Tanh causes the most notable seperation of positive and negative values. We depicted the results for Tanh Figure 3 and for all other activation functions in Figure 7 Furthermore, we find that splitting with óo 0 seems to be meaningful for all examples with respect to separation and symmetry. This split into two set of almost equal size. The same is true for &y. Apart from these values, we experimented in this and all following experiments with different approaches to determine thresholds, such mean values, median values, median per neuron, well as kernel-density estimation for bi-variate Gaussians. However, the split at was favorable with respect to the achieved model fidelity. We report these scores for Tanh (Table 2 and for all activation functions in Appendix, Table 7 We conclude from our
httpps://gittubb.com//orree haap:////riiitiitttteeeeee imagama -Dtasset
5

---

Under review as a conference paper at ICLR 2023
W
Figure 3: The value distributions for the object (0) and class (W) view for ten runs using the Fruits- 360 data set and the Tanh activation function. The last hidden layer of size 24 (first column) to 2 (last column). The reader may find the plot for all activation functions in Appendix, Figure 7
ablation study that the of Tanh is suggested as well d0 dy O. We acknowledge that the used data set might influence this choice (Bellido & Fiesler 1993).
Symbolic Conceptual Views of ImageNet Based on the just found parameters we computed the symbolic conceptual views for the ImageNet models. We report the results in Table We found that the classes are uniquely represented (class separation equals 1). thus a perfection classification procedure is theoretically possible using the symbolic view. However, we observe that a direct application of 1-NN procedure using the binary vectors that arise from the symbolic view does result in very poor classification performance for ReLu. In contrast to that, the Swish based models achieved from mediocre to good results. Especially the larger (EfficientNet) NN models resulted in better symbolic views. A reason for the unfavorable results with ReLu might esteem from its positive co-domain, which hinders the construction of negated attributes N in our approach. Some selected distributions be found in Appendix, Figure
Symbolic Conceptual Views on Fruit-360 The twenty-four models in ImageNet employ ReLu and Swish activation functions only. Thus, want to complement experimental study s.c. views with results for using Tanh, which we conduct again on the Fruits-366 data set. Hence, we trained five models, namely the base-line model from Mureşan & Oltean 2017). VGG16, ResNet50, IncV3, and EПBO, the latter initialized with the original ImageNet weights. With exception of the baseline, we added each model three dense layers (including dropout layers with p=0.2) on top, that are sized 1024, 256, and 32. To all models we also added an additional layer of size 16. This reduction in size added in order to enable human explainability. The baseline model as well all added layers employ the Tanh activation. The output (prediction) layer is a dense layer using softmax activation without bias. We used sparse categorical crossentropy as the loss function. All other relevant parameters for reproducing our results are drawn from the published baseline model.
The results in Table 4 show that all models have high accuracy on the test data set, while the four transfer learned models outperform the baseline. We want to stress that our predictive results are only
Table 2: Results for the ablation study on the influence of the activation function and number of neurons on the quality of the (Symbolic) object/class view. We measure the quality in terms of fidelity of 1-NN to the original model, see V-Fid and SV-Fid for the symbolic case. Find the full table in Appendix, Table
2
2
26
2
2
29
Tanh
da = 0 Split
49.7/50.3
49.7/50.3
49.8/50.2
49.9/50.1
50.0/50/0
49.9/50.1
dw Split
49.9/50.1
49.8/50.2
49.8/50.2
50.0/50.0
49.9/50.1
50.0/50.0
Model Acc
90.5 0.8
94.3+ 0.5
94.7+ 0.5
94.9+ 0.4
95.0 0.4
94.8 0.3
V-Fid
98.3+ 0.5
99.5 0.1
99.7+ 0.0
99.7+ 0.0
99.8 0.0
99.8 0.0
SV-Fid
94.3= 1.4
97.4 0.4
97.7+
97.6 0.1
97.8 0.2
97.6 0.2

---

Under review as a conference paper at ICLR 2023
Table 3: The fidelity between six NN models and their symbolic conceptual view using 1-NN for
classification. Find all values in Appendix, Table g
Euclidean
Cos
Class Sep
Activation
VGGI6
0.552
0.552
1.0
ReLu
DenseNer201
0.000
0.000
1.0
ReLuu
MobilNetV1
0.036
0.036
1.0
ReLu
ResNet152V2
0.000
0.000
1.0
ReLu
ENBO
0.758
0.758
1.0
Swish
E1B7
0.957
0.957
1.0
Swish
used demonstrate that the model did fit the classification problem. We find that both, Euclidean and cosine based I-NN did perform well on the MV as well as the symbolic conceptual view spaces. In detail, could not find significant difference between the representations. Moreover, we cannot identify sigmificant differences in the classification performance with respect to the NN model. We also observed that an udditionally trained decision tree classifier was unable to learn within the MV conceptual view representation. However, the same procedure applied to the symbolic view was capable of producing competitive classification results, surrogate for the NN with very high fidelity.
Symbolic Conceptual Views through FCA Each algebraic relation corresponds to exactly one natural order structuring of the data involved by means of the underlying Galois connections. Through FCA one can reveal this order by means of formal concepts, i.e., all (A, B) E P(G) x P(M) such that A' = B and B' = A. The sets A and B are called extent and intent, respectively. The reader may notice strong resemblance to maximal bi-cliques. The set of all formal concepts of context K denoted by B (K) and its elements are ordered by inclusion C on the extent sets. The resulting order structure B(T = (2 (K), S) constitutes complete lattice that can be visualized as line diagram. Hence, applied to our analysis of NNs one may visualize the conceptual hierarchy learned by the network and size serves as an upper bound for the number of learned concepts.
Observations We computed the concept lattices for Base, ResNet, VGG16, Inc and EftBO and find, that their sizes vary between 126487 (VGG16) and 134100 (IncV33 for |N| 16, and between 3,498,829 (VGG16) and 3,803.799 (ResNet50). If we restrict our computation to N, i.e., omitting the artificially introduced negations N, we find the concept lattice sizes decrease by one magniude. In detail, between 5200 (VGG16) and 6573 (EffBO) for |N] 16, and 150884 (EfBB) and 1981 152 (IneV3) for | = 32. We compiled all values Appendix, Table 9 Overall, we observe that all are similar in size and large and therefore cannot be visualized using a line diagram. We note that formal concepts are composed of combinations of features. The minimum number of encoded features is present in so called meet-irreducible (MI) elements, i.e., concepts where the extent of (A, B) cannot be written as an intersection of extents from ((K) \ {(A, B)}. Hence, the number MI elements is bound by the number of attributes in the context, which serves lower bound the concepts captured by the NN model.
Independent of the size does this translation to the realm of FCA enable the application of vari- ous knowledge-basee -aaed methods, such as Description logic or Subgroup discovery, as investigated in Section 5 Within FCA we might consider to analyze cuts of the lattice, in particular those where we suspect problems in the representation. As we discovered in previous experiments, that Apple Red, Pink Lady, Plum and Cherry are indistinguishable by some symbolic representations (Table 4. one might want to "zoom" into those. We did this statistically with Figure 4 (top) and structurally with Figure [ (bottom). In the former on can identify formal concept based similarities among the selected fruits and the number of their shared concepts. In the latter figure the reader can infer the hierarchical dependencies between the different fruits (objects) indicated in color.
ABDUCTIVE LEARNING OF PARTIAL EXPLANATIONS
Symbolic conceptual views enable the application of various logical methods to derive human- comprehensible (partial) explanations. We draw from this correspondence and construct a formal
In this experiment we also studied the influence of omitting the layer of size 16.
7

---

Under review as a conference paper at ICLR 2023
Base
VGGI6
ResNet
Inc
80
PV LURY
83
SDI
100
Pot
Prod Toyds
Figure 4: FCA results for Apple Red, Pink Lady, Plum and Cherry using views with positive attributes. The columns labels of each heatmap displays the number form. concepts. The number within a cell is the number of shared concepts between the related row/column fruits. Heat indicates its fraction. Bottom: Drder-structured representation the selected fruits. Formal concepts containing Apple Pink Lady and Apple Red I are highlighted in Orange and Cherry is highlighted in blue.
Top:
context C = (C,Sx, It), where SM = {Sm...... Sma } is set of human-interpretable features that are known about the classes C. i.e., background knowledge.
Definition (Symbolic Interpretation) Given the s.c. view Vo (Op WD) of a NN, background knowledge C, and similarity relation on the classes P(C). Then is the formal context S SM. R) with (n, Sm.) R : {m] {Sm) the symbolic interpretation of the NN with respect to C and m.
We require ~ to be reflexive and symmetric but not necessarily transitive. The task for symbolic interpreting NN to deduce infer using background knowledge, which is done in the next section. Given symbolic interpretation for a NN we are able to express neurons in terms of the human-interpretable features Sas by applying the incidence relation S, i.e., for n E N one can compute {n I Furthermore, if is additionally equipped with propositional logic L(SM, {V,A, -}) then FCA (Hanika & Hirth 2022) also provides the means for expressing neurons in terms of propositional statements.
Evaluate the Symbolic View Interpretation We want to motivate how symbolic interpretations be used interpret neuron in terms of huuman-comprehensible. features SM and vice versa. To demonstrate both cases, we analyze the symbolic view of the Fruit-360. The attributes SM we use visual features (1), such shapes colors, and used the Scientific classification taxonomy (2)
8

---

Under review as a conference paper at ICLR 2023
Orange Color SubGroups
Visual SubGroups for 13
0.8
SubGroup
SubGroup Complement
Complement
1.00
Target Share
0,541667
Share
0,444444
o.P
405405
0.50
0.4
0.2
0.0425532
010600748
0J0573077
0,00
0.0
Brown
Stained
Unilorm Orange
1649
BATAS
^ Orange
Star
A Stttrr
Apple Taxon SubGroups
Taxon SubGroups
0.8
SubGroup Complement
SubGroup Complement
1.00
Target Shar
0.6
0.5
Target Share
0.375
0. 0.380952
0.50
0.425
0.2
0.3338322
044437783
0464545
0.00
0.0
Sapindales A SSaiinacceee
4.5
A -Eggplant A. -Lime
Figure 5: Exemplary results of the subgroup detection. Width indicates subgroup-size.
published in Wikipedia for each fruit/vegetable. We combined the German and English Wikipedia articles in order to derive a data set complete as possible. We infer the similarity relation in our experiment using subgroup detection (Herrera et al. 2010), as implemented in pysubgroup.
We depict four example results in Figure 5 where the taxons are given in the respective diagram titles, e.g., Apple, Orange. Diagrams on left depict subgroups terms of neurons, and, vice versa, on the right in terms of interpretable features. For both sides we find on the abscissa propositional statement combining the respective features.
From the high share (see ordinate) of the respective subgroups can infer that the propositional statements using the neurons or Sa features describe the taxons from adequately up to very good. In particular for the latter case (right) concrete statements: first, fruits that are not brown and not stained and not orange and not star shaped will neuron 13 from N. Second, if the neuron 13 and 14 and 9 used by the NN this implies that the fruit is orange with confidence about 0.54. Using this method one can infer the similarity relation and provide an explanation framework.
that the subgroups are pure, yet, not complete. To give two
LIMITATIONS AND CONCLUSION
The presented approach is novel and different former ideas with respect to three properties: first, we do not employ further hardly explainable methods, such as autoencoders. Second, our method is global by design. Third, conceptual views, as introduced in our work, do not require pre-defined concepts and their related input representations. We accomplished this by decoding both, the weights of all output neurons and the activations of the last hidden layer using symbolic conceptual views.
Our approach is limited by the necessary existence of multiple outputs. However, there are common approaches for splitting single outputs. Yet, a more significant limitation concerns the restriction to non-recursive architectures. Adapting our approach to such settings is probably possible, but requires substantial adjustment to the definition of the views. Finally, our method requires for human-comppehennsibe explanations the existence of domain-specific background knowledge.
Apart from this envision that the presented link of NN models FCA using symbolic conceptual views allows for both the explainability of NNs as well as increasing the performance NN surrogate learning procedures. Therefore, this beneficial research line should be further investigated and tested.
*See for htw.a//nwtttaae//eaa..aaaaaaaao.
http:///wewwwwiiiaeaaaae htwp///..oro///ioo /Applep in the right box.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), NeurlPS, pp. 9525-9536, 2018.
Ahmed M. Alaa and Mihaela van der Schaar. Demystifying black-box models with symbolic metamodels. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), NeurIPS, pp. 11301-1131] 2019.
Masataro Asai and Alex Fukunaga. Classical planning in deep latent space: Bridging the subsymbolic- symbolic boundary. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), AAAI, pp. 6094-6101.
AAAI Press, 2018.
I. Bellido and E. Fiesler. Do backpropagation trained neural networks have normal weight distribu- tions? In Stan Gielen and Bert Kappen (eds.), ICANN '93, pp. 772-775, London, 1993, Springer London. ISBN 978-1-4471-- 1-44--2-222066666
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. In Tal Linzen, Grzegorz Chrupala, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACI. Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL.,, BlackboxNI. ACI. 2019, Florence, Italy, August 1, 2019, pp. 276-286. Association for Computational Linguistics, 2019, doi: N0.18653/11/11910
Jia Deng, R. Socher, Li Fei-Fei, Wei Dong, Kai Li, and Li-Jia Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Puttern Recognition CVPR), volume 00, pp.
06 2009. doi: 10. D0.109//VR..200052000
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In CVPR, pp. 8730-8738. IEEE Computer Society, 2018.
B. Ganter and R. Wille. Conceptual scaling, In F. Roberts (ed.), Applications of combinatorics and graph theory to the biological and social sciences, pp. 139-167. Springer- Verlag, 1989.
B. Ganter and R. Wille. Formal Concept Analysis: Mathematical Foundations. Springer- Verlar, Berlin, 1999.
Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. Towards automatic concept-based explanations. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ahh Buc, Emily B. Fox, and Roman Garnett (eds.), NeurlPS, pp. 9273-9282, 2019.
Tom Hanika and Johannes Hirth. Quantifying the conceptual error in dimensionality reduction. In Tanya Braun, Marcel Gehrke, Tom Hanika, and Nathalie Hernandez (eds.), Graph-Based Representation and Reasoning- 26th International Conference on Conceptual Structures, ICCS 2021, Virtual Event, September 20-22, 2021, Proceedings. volume 12879 of Lecture Notes in Computer Science, pp. 105-118. Springer, 2021. doi: 10. 0101//99788--00-6695-- 8.
Tom Hanika and Johannes Hirth. On the lattice of conceptual measurements, Information Sciences, 10tp://100.001......22....0.0000
613:453 468, 2022. ISSN 0020-0255. doi:
Franciso Herrera, Cristóbal Carmona, Pedro González, and María del Jesus. An overview on subgroup discovery: foundations and applications. Knowledge and Information Systems, pp. 1-31, 2010. ISSN 0219-1377. doi: 10. 10.1007/101115-00-0053
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In Jennifer Dy and Andreas Krause (eds.), P Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. PMLR, 2018
Bruno Korbar, Andrea M. Olofson, Allen P. Miraflor, Catherine M. Nicka, Matthew A. Suriawinata, Lorenzo Torresani, Arief A. Suriawinata, and Saeed Hassanpour, Looking under the hood: Deep neural network visualization interpret whole-slide image analysis outcomes for colorectal polyps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
10

---

Under review as a conference paper at ICLR 2023
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buch E. Fox, and R. Garnett (eds.), ). Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. The neuro- symbolic concept leamer: Interpreting scenes, words, and sentences from natural supervision. In ICLR. OpenReview.net, 2019.
Horea Mureşan and Mihai Oltean. Fruit recognition from images using deep learning. arXiv preprint arXiv:1712.00588. 2017.
Facundo Mémoli.
Gromov-wasserster distances and the metrie approach to object matching. Foundations of Computational Mathematics, 11(4):417-477 2011.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), KDD, pp. 1 1135-1144. ACM, 2016.
ISBN 978-1-4503-4232-2.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215) 2019. ISSN 25225839.
doi: 10.1038/s42256- -01-0000--0--008--
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-255 252, 2015. doi: 10.1007/s1 126070000166--1
Md Kamruzzaman Sarker, Lu Zhou, Aaron Eberhart, and Pascal Hitzler. Neuro-symbolic artificial intelligence. Al Communications, 34(3):197-209) March 2022. doi: 10.3233/ai--20008.
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Jonathan Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht. Neural kernels without tangents. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 8614-8623, PMLR, 13-18 Jul 2020.
R. Wille. Restructuring lattice theory: An approach based hierarchies of concepts. In Ivan Rival (ed.), Ordered Sets: Proc. of the NATO Advanced Study Institute, pp. 445 Dordrecht, 1982. Springer. ISBN 978-98 777-94-00--7798-3
Chih- Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations deep neural networks. Advances in Neural 2020.
Information Processing Systems,
A APPENDIX
In this appendix we provide more figures and table data for our experimental study.

---

Under review as a conference paper at ICLR 2023
Table 4: Five neural networks (using Tanh activation) and their (symbolic) conceptual views were captured by different surrogates (decision tree, I-NN). We report their fidelity and accuracy. In symbolic conceptual view: IncV3 was unable to distinguish Cherry and Plum: EffBO was unable to distinguish Apple Red and Apple Pink Lady
Model
Model
DTree
Euclidean
Cos
ACC
ACC
Fid
ACC
Fid
ACC
Fid
Baseline
0.936
0.017
0.017
0.935
0.989
0.935
0.988
VGG16
0.988
0.017
0.018
0.988
0.998
0.988
0.997
ResNet50
0.989
0.018
0.018
0.989
0.998
0.989
0.997
IncV3
0.983
0.013
0.013
0.983
0.999
0.984
0.999
0.984
0.007
0.007
0.984
0.998
0.983
0.984
Symbolic
Class Sep
Baseline
0.936
0.857
0.879
0.927
0.964
0.927
0.964
1.0
VGG16
0.988
0.972
0.977
0.988
0.994
0.988
0.994
1.0
ResNet50
0.989
0.952
0.957
0.988
0.996
0.988
0.996
1.0
IncV3
0.983
0.975
0.988
0.984
0.997
0.984
0.997
0,992
0.984
0.938
0.934
0.984
0.996
0.984
0.996
0.992
Table 5; The fidelity between twentyfour neural networks and their object/class view using 1-NN for classification.
Model
Euclidean
Cosine
Model
Euclidean
Cosine
VGG16
0.945
0.841
ResNet101V2
0.995
0,466
VGG19
0,942
0.842
ResNet152V2
0.999
0.314
IncV3
0.990
0.753
IncResNetV2
0.999
0.983
DenseNet121
0.978
0.737
XCeption
0.977
0.792
DenseNet169
0.989
0.843
EffBO
0.944
0.933
DenseNer201
0.972
0.728
EffB
0.960
0.946
MobilNetV1
0.575
0.449
EfBB2
0.969
0.957
MobilNetV2
0.947
0.925
E
0,974
0.961
NasNetMobile
0.935
0.808
B1BB
0.981
0.972
NasNetLarge
0.880
0.831
EffB5
0.979
0.972
ResNet50
0.954
0.800
EfBB6
0.982
0.976
ResNet50v2
0.995
0.734
BffB7
0.985
0.979
12

---

Under review as a conference paper at ICLR 2023
Table 6: The average weights wij object values (g) and the number of neurons | and activation function f of the last hidden layer tensorflow imagenet models.
Model
W values
Bias
C values
N
VGG16
-3.359e-07 0.008
0.19
0.679 + 1.514
4096
ReLu
VGG19
-6.707e-07 + 0.008
-1.287e-05 + 0.192
0.613 + 1.402
4096
ReLu
IncV3
-38808e-05 + 0.034
-0.0099 + 0..38
6.025 + 15.13
2048
ReLu
DenseNet121
2.139e-08 0.049
-1.014c-07 0.012
1.731 4.603
1024
ReLu
DenseNet 169
1.456e-08 = 0.039
-1.038c-07
1.675 5.529
1664
ReLu
NenseNer201
1.019e-08 0.036
-1.178e-07 1.011
1.146 4.166
1920
ReLu
Ne
-0,0001 + 0.081
-0.005 + 0.744
0.435 + 0.838
1024
ReLu
MobilNerV2
-3.138e-05 0.041
0.0002 + 0.319
0.358 0.747
1280
ReLu
NasNetLarge
-20800-00 + 0.026
4.424e-05 + 0.040
0.198 0.533
4032
ReLu
NasNetMobile
-3.336e-07 0.039
0.0001 + 0.066
0.382 4.389
1056
ReLu
ResNet50
3.774c-07 = 0.033
-.881c-08 = 0.009
0.546 0.871
2048
ReLu
RcsNet101V2
6.668e-06 0.027
0.0016 = 0,292
39.97 + 167.8
2048
ReLu
ResNer152V2
1.038e-05 + 0.026
0.0016 + 0.287
94.08 + 187.4
2048
ReLu
ResNet50V2
8.014e-07 + 0.028
0.0011 0.292
19.91 + 74.65
2048
ReLu
IncResNetV2
-3.060e-05 0.037
-0.0012 0.230
106.8 + 124.9
1536
ReLu
XCeption
-3.246e-06 + 0.055
0.0008 + 0.281
2.974 + 13.41
2048
ReLu
EпвО
-7.495e-05 0.068
-5.143e-05 + 0.058
0.065 0.321
1280
Swish
EffB1
-5.647e-05 = 0.063
4.343e-05 = 0.045
0.056 0.313
1280
Swish
EfB2
-7.152e-05 0.059
-4.13e-005 0.054
0.019 0.260
1408
Swish
Eff33
-6.323e-05 + 0.054
-3.547e-05 + 0.046
0.010 + 0.252
1536
Swish
EnB4
-3.106e-05 0.050
-3.138e-05 0.057
-0.039 + 0.194
1792
Swish
EABS
-2.043e-05 = 0.049
= 2.738c-05=1000
-0.036 0.170
2048
Swish
EпВ6
-8.656e-06 0.046
-2.691c-05 + 0.071
-0.043 0.135
2304
Swish
EfB7
-9.562e-06 = 0.041
-..41e-05 + 0.060
-0.041 + 0.136
2560
Swish
13

---

Under review as a conference paper at ICLR 2023
Figure 6: The similarity of twenty-four neural networks trained the ImageNet data set. The base-line (left) is pair-wise fidality between the employed models compared to a similarity using Gromov-Wane Wasserstein distance the object (top right) and class (bottom right) view.
14

---

Under review as a conference paper at ICLR 2023
Swish
W
ReLu
W
Linear
W
TanH
W
Figure 7: These are the value distributions for the object (0) and class (W) view for ten runs using the Fruits-360 data set and the swish, relu, linear and tanh activation functions. The last hidden layer of size 24 (first column) 29 (last column).
15

---

Under review as a conference paper at ICLR 2023
Table 7: Results for the ablation study on the influence of the activation function and number of neurons on the quality of the (Symbolic) object/class view. The measure the quality in terms of fidelity of nearest neighbor classification the original model, see V-Fid and SV-Fid for the symbolic case.
2
2
2
2'
2
2
Swish
da 0 Split
57.1/62.9
52.8/47.2
49.2/50.8
44.5/55.5
45.6/54.4
45.0/55.0
6w = 0 Split
65.4/44.6
65.0/35.0
62.0/38.0
60.1/39.9
57.2/43.8
56.5/43.5
Model Acc
93.51
94.5
0.3
95.1 0.3
95.4 0.4
95.1 0.5
V-Fid
99.51 0.4
99.9 0.0
99.9 0,0
99.9 0.0
99.9 0.0
99.9 0.0
SV-Fid
77.1 9.2
88.8 1.3
89.6 1.6
88.8 1.4
88.4 0.9
86.7 1.6
ReLu
dg Split
55.1/44.9
55.1/44.9
54.7/45.3
53.3/46.7 7
55.3/44.7
54.1/45.9
by 0 Split
66.3/33.7
66.7/33.2
64.0/36.0
61.6/38.4
58.9/41.1
58.2/41.
Model Ace
93.7 0.4
94.5 0.5
94.9 0.5
94.9 0.5
95.0 0.4
94.8 0.6
V-Fid
99.7 0.0
99.8 0.0
99.9 0.0
99.9 0.0
99.9 0.0
99.9 0.0
SV-Fid
79.9 3.7
89.0 1.2
90.0 1.2
89.5 1.1
89.0 1.5
88.2 1.5
2
2"
2"
2'
2
2
Linear
on 0 Split
49.8/50.2
49.6/50.4
49.5/50.5
49.9/50.
49.9/50.
49.9/50.1
ow Split
49.7/50.3
49.3/50.7
49.7/50.3
50.0/50.0
50.0/50.0
50.0/50.0
Model Acc
85.3= 0.6
88.8 0.7
89.9 1.0
92.0 0.5
91.8 0.9
91.5 0.9
V-Fid
99.9 0.0
99.9 0.0
99.9 0.0
99.9 0.0
99.9 0.0
99.9 0.0
SV-Fid
55.5= 1.9
64.7 1.4
68.3 2.6
74.8 1.7
78.2 2.1
81.5 1.1
Tanh
dn = 0 Split
49.7/50.3
49.7/50.3
49.8/50.2
49.9/50.1
50.0/50/0
49.9/50.1
dw 0 Split
49.9/50.1
49.8/50.2
49.8/50.2
50.0/50.0
49.9/50.1
50.0/50.0
Model Acc
90.5 0.8
94.31 0.5
94.7
94.9 0.4
95.0 0.4
94.8 0.3
V-Fid
98.3= 0.5
99.5 0.1
9.0
99.7 0.0
99.8 0.0
99.8 0.0
SV-Fid
94.3= 1.4
97.4 0.4
97.6 9.
97.8 0.2
97.6 0.2
E
VGG16
600000
20000
520000
Count
400000
107
300000
10000
101
100000
204
ResNet50
ResNet150V2
107
167
Count
207
104
104
303
Figure 8: The value distributions for the object/class view for the EnB7, VGG16, ResNet50 and Resnet150v3 Model.
16

---

Under review as a conference paper at ICLR 2023
Table 8: The fidelity hetween twenty-four neural networks and their symbolic object/class view using nearest neighbor and cosine similarity for classification.
INN
Cos
Class Sep
Activation
VGGT6
0.552
0.552
ReLu
VGG19
0.5672
0.5672
1.0
ReLu
Inc
0.000
0.000
1.0
ReLu
1enseNet1
0.000
0.000
1.0
ReLu
DenseNet169
0.001
0.001
1.0
ReLu
DenseNet201
0.000
0.000
1.0
ReLu
MobilNetVV
0.036
0.036
1.0
ReLu
MobilNetV2
0.305
0.305
1.0
ReLu
NasNetMobile
0.004
0.004
1.0
ReLu
NasNetLarge
0.009
0.009
1.0
ReLu
ResNet50)
0.007
0.000
1.0
ReL.u
ResNet50V2
0.001
0.003
ReLu
INN
Cos
Class Sep
Activation
ResNet101V2
0.012
0.012
1.0
ReLu
52V2
0.000
0.000
1.0
ReLu
IncResNetV2
0.000
0.000
1.0
ReLu
XCeption
0.220
0.220
1.0
ReLu
Е
0.758
0.758
1.0
Swish
EfiB1
0.813
0.813
1.0
Swish
ElB2
0.869
0.869
1.0
Swish
EПВ3
0.898
0.898
1.0
Swish
E1B4
0.929
0.929
1.0
Swish
EffB5
0.935
0.935
1.0
Swish
0.951
0.951
1.0
Swish
ENB7
0.957
0.957
1.0
Swish
Table The size of the concept lattices of the symbolic views Table (see N = 16) and for 32
neuron (see |N| 32), for all (see All column) and only positive attributes (see Pos column).
Model
[
16
N
32
All
Pos
All
Pos
Base
130969
6517
3192044
155416
ResNet50
133130
5872
3803799
165009
VGG16
126487
5200
3498829
193516
IncV3
134100
5670
3782226
198152
EffBO
132403
6573
3767964
150884
17


Reviews:
[{"review_id": "ZemoD9JNL0b", "reviewer": "Reviewer_PGXP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper looks to interpret neural networks by examining the representations and the weights. The paper does a discretization of both of these and indexes them via a look up to create what it calls the symbolic conceptual view of a neural network. This conceptual view is then evaluated on various tasks. Experiments show that the conceptual view can approximate a neural network well, be used to compute similarity between networks and do “abductive learning” which as far as I can tell is about interpreting a trained neural network by giving human understandable forms to the symbolic conceptual view.\n", "summary_of_the_review": "The paper is very hard to read, and understand. From what I can understand the technical novelty is very limited and uninteresting. ", "strength_and_weaknesses": "+ The paper reports nice ablations and experimental results comparing different design choices\n+ The paper tackles an important problem of neural network interpretability, which is as relevant as ever and of great practical interest\n+ The paper is looking at a number of tasks ranging from comparing two neural networks for similarity, interpreting what a trained network has learned, and quality of function approximation with the discrete representation\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Jozill1abk", "reviewer": "Reviewer_xUGn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides two views to globally explain the learned knowledge of NN to humans. Specifically, the \"many-valued\" view explains each decision as a combination of object representation and classification, which focuses on the last hidden layer. The symbolic view can be used to represent NN models with formal concept lattices. This paper uses two main experiments to support the claim.", "summary_of_the_review": "See above", "strength_and_weaknesses": "**Strength**\n\n1 This paper shows two new views to help humans understand the learned knowledge of neural networks, which is interesting.\n\n2 The paper conducted many experiments to support its claim.\n\n**Weakness**\n\n1 The motivation and application of the proposed method need to be clarified. Why do we need a Formal conceptual view to understand the knowledge of Neural Networks? The author uses some abstract words and sentences, e.g., \"grasp deeper insights into the knowledge that is captured,\" which is hard to understand the motivation and application. In addition, it is hard to understand the goal of each step in the method section.\n\n2) Why is only the last hidden layer important for the Many-valued Conceptual View? From a network dissection perspective, the last several layers may have a similar function: they are sensitive to high-level concept understanding. Why are the rest layers ignored? Also, how to extend the explanation to convolutional layers (kernels), transformer-based layers or neural networks trained with self-supervised learning (contrastive loss)? The author only explains the vanilla MLP model.\n\n3 Use a large number of notations while some of them lack explanation. For instance, what is the meaning of it? What is the meaning of | |?\n\n4 Many typos and grammar errors which obstacles the understanding. For instance, Definition1: C its..., N = {} the..., Section 3.1 \"As a final remark before we introduce the symbolic conceptual view on NN we want to point out a simple but powerful observation.\" no comma at all. Section 4. \"First, we evaluate the suitability of the in Section 3 introduced pseudo metric space by a classification task. \"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CoL5lzJtP9o", "reviewer": "Reviewer_r4Ld", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper covers a novel approach to understanding neural networks in terms of an \"object view\" (the activations of the network given the input representation of an object) and the \"class view\" (the set of weights between the neurons and the class outputs----I think).  In addition the paper introduces an intermediate representation using some known attributes called the conceptual representation.  The paper carries out a lot of empirical experiments using pretrained trained models on ImageNet and networks that are trained on Fruit-360.", "summary_of_the_review": "For me, the paper was too full of poorly explained ideas.  It lacked a precision in writing but also an appreciation of the reader.  I accept that novel ideas are difficult to communicate, but for me the authors did not try to make their work understandable.  It is very difficult to make a judgement on what the authors did.  I have some scepticism about the utility of their approach, but ultimately I found it hard to care because the paper didn't seem to want to explain itself.", "strength_and_weaknesses": "The strength is that a new approach is proposed that is potentially interest.\n\nI am afraid the weakness, for me at least, is that the paper is almost incomprehensible.  The paper introduces so many new concepts without really explaining what any of them mean or why they are relevant.  There are a huge number of statements that are incomprehensible to me.  Partly this is linguistic (some statements are difficult to parse), but much more there is no real effort to explain what the author means.  It comes across to me as a stream of consciousness rather than a well argued and justified thesis.  I am prepared to accept that this may be because the authors are much more knowledgable and clever that this reviewer, however, there is a necessity to explain ideas in a way that is comprehensible to a general readership.  I am afraid this paper fails this test in my opinion.  For this work to be publishable would, in my view, require a complete rewrite where all the ideas were fully explained.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A7g65hFqmZM", "reviewer": "Reviewer_i9L7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two methods for deriving ‘conceptual views’ of neural networks, a ‘many-valued’ view and a ’symbolic’ view, that allow for human-interpretable analyses into the knowledge contained in the network. Each view is constructed using the combination of a neural network (trained on a classification task) and a set of datapoints, and is a concatenation of an object-based view that corresponds to the activations of the last hidden layer for each datapoint, and a class view which corresponds to the weights from the last hidden layer to the output neurons. Via experiments using a variety of networks trained on Imagenet and Fruits-360 respectively, it is shown how one or both of the conceptual views can (i) be used to define a similarity metric between different neural networks, (ii) be used to represent a network with a formal concept lattice, which depicts a hierarchical clustering of features, and (iii) be used to derive human-interpretable propositional statements when combined with background knowledge of the dataset using subgroup discovery.", "summary_of_the_review": "Overall, the paper does a good job of motivating and describing the conceptual views it proposes for neural network intepretability, and a range of applications of the views are described and demonstrated with experiments. The key weakness is that the general applicability of the methods is questionable, for instance it only seems to be defined for classification networks, and the human interpretability relies on the existence of domain-specific background knowledge.", "strength_and_weaknesses": "Strengths\n- The paper tackles a challenging problem, that of neural network interpretability, which is of considerable interest to the ML community.\n- The derivations of the concept views are clearly explained and easy to follow.\n-  A key strength of the paper is that the conceptual views (in particular the symbolic view) are shown to enable several different types of analyses as summarised above: eg a similarity metric between different networks, decomposition into a formal concept lattice, and derivation of human-interpretable propositional statements when combined with background knowledge.\n- A number of limitations of the methods are clearly outlined in the conclusion.\n\nQuestions / opportunities for improvement\n- It is unclear to what extent the methods of the paper would apply outside of networks trained on classification tasks, e.g. on regression or even a stochastic policy network for reinforcement learning tasks. Could the framework easily be extended beyond classification tasks / would it even make sense?\n- The symbolic view relies on a user-defined threshold that needs to be defined for the object view and class view - it’s not clear how the effectiveness/usefulness of the symbolic view depends on this choice, or what is a good way to make this choice in advance - it seems a bit arbitrary.\n- In section 4.2, it would be nice to have more discussion of how we can interpret the clustering results - e.g. what could explain the difference between the clusters in the object view and the class view?\n- In section 4.2, what is the rationale for shrinking the penultimate hidden layer as you grow the last hidden layer?\n- For the similarity metric between networks, it would be good to have at least a high level description of what the Gromov-Wasserstein distance is for the reader who is not aware of it.\n- As acknowledged by the authors, the method for deriving human-interpretable explanations from the conceptual views requires in addition the existence of domain-specific background knowledge, which limits the general applicability of the method.\n\nMinor\n- Section 4, Line 3  and in 4.1 “the in Section 3 introduced pseudo metric space” -> “the pseudo metric space introduced in Section 3”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject