Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a joint Gaussian mixture model (JGMM)-based post-hoc explanation method, which applies inter-layer deep features in a probabilistic model. The JGMM can explain deep features and inter-layer deep feature relationship on the latent component variables in its GMMs. The JGMM is applicable to any black-box DNN model for providing explanations for higher/lower level features and the black-box model between them. 

The proposed method is interesting and technically sound. It can obtain different forms of model explanations with same trends. However, the writing of this paper is poor which should be improved by the authors. In addition, the complexity of the JGMM should be discussed.

strength_and_weaknesses: Strength:
1. The authors proposed a novel viewpoint of DNN model explanation by mixture models, which introduces the correlation between the features of two layers into the probabilistic model instead of independently modeling in some other methods. The joint modeling can explain the learned representations of higher/lower-level features as well as the black-box model between them.
2. The JGMM is applicable to any black-box DNN model as a post-hoc interpreter.
3. It can produce explanations in various image classification tasks, which has been shown in the experiments. The visualizations can clearly show their superiority. Various forms of model explanations can be efficiently produced from the framework. The consistency among different explanations is also demonstrated.

Weaknesses:
1. The writing of this paper is not good enough. Some sentences are hard to understand, e.g., “a probabilistic model jointly models inter-layer deep features and produces faithful and consistent post-hoc explanations” in abstract, “The deep feature maps of DCNN usually has spatial dimensions” in section 3.3.
2. The position of variable Q is not clarified clearly. How to understand it under a probabilistic view? Besides, it is hard to understand how it works. Eq (22) shows its updating function, are there any references that can support it?
3. The paper has many typos, e.g., \hat{q} in eq (4) is undefined, a variable i in eq (9) is lost, many variables are unclearly defined. The authors should carefully check their paper.
4. How to select the component number of the two GMM? How to select which two layers are used for modeling the JGMM? Do different values affect the results? Are there any ablation studies for clarifying the points?
5. The definitions of global and local explanations should be clarified.
6. The comparisons of complexity and parameter number between JGMM and others should be discussed in the paper.


clarity,_quality,_novelty_and_reproducibility: As mentioned in the Weaknesses section, the paper is not clarified clearly, including methodology and experiments, although the proposed method is novel and technically sound. There are many typos and mistakes, and the total quality of the paper should be improved. The reproducibility is therefore not good. 
summary_of_the_review: As described above, the authors should carefully check and revise their paper. The whole paper should be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a simple joint Gaussian mixture model to generate post-hoc model explanations for DCNN. Essentially the authors use two GMMs to model the higher and lower features Y and X, with a projection matrix Q connecting the component probabilities from these two GMMs. Experiments demonstrated the proposed method’s capability of generating local and global model explanations. 
strength_and_weaknesses: Strength 

+ Generating model explanations is an important and interesting problem for the ML community.  
+ The proposed model can generate a wide range of explanation types including prototypes and counterfactuals. 
+ The definitions of prototypes, criticisms, etc. make sense to me, though (1) it is unclear how the definition of counterfactuals in Eq (7) is consistent with the standard definition of counterfactuals in Pearl’s do-calculus and (2) since Eq (9) is incomplete, it is unclear why the definition is the reverse of counterfactuals. 

Weaknesses

- The proposed model is not principled; that is, it does not seem to be a rigorous probabilistic graphical model (PGM) formulation any more. 
- A lot of details on Equation (2) and (3) are missing/unclear in the main paper. 
- Equation (4) is confusing and potentially problematic. 
- Given that one of the focus in evaluation is to generate counterfactual explanations, some important baselines are missing, e.g., [3]. 
- The baselines (decision trees, logistic regression, etc.) used for evaluating proxy model accuracy are weak. The authors did not even include the 2016 method LIME as a baseline.

clarity,_quality,_novelty_and_reproducibility: The authors mention that most existing ‘DCNN explaining methods are single-purpose systems’. Note that a proxy model like LIME can provide local explanations, but it is also able to provide counterfactuals if needed (one simply needs to use gradient-based methods on the locally linear proxy model to obtain such counterfactuals). 

The PGM in Figure 2 is a bit confusing, especially the lower plate. It does not look like a typical PGM, or a factor graph representation of a probabilistic model. For example, I assume that Sigma and mu should both directly point to x? 

Empirically, I can see what the authors mean to design by using the approximate posterior $q_k$ as the prior for $z_n$. However, the model is not principled; that is, it does not seem to be a rigorous probabilistic graphical model (PGM) formulation any more. In this case, it might be better not to describe the model as a PGM, but directly introduce the method as a computational graph. The authors should also revise Sec. 3.1 such that the description on prior and posterior is more rigorous. According to Eq (1), it seems Q is just a global parameter matrix that is related to both GMMs. In this case, the claim that the prior of z depends on the posterior of w is rather confusing. 

The authors refer to having only access to intermediate layers as a ‘black-box’ setting. This is somewhat inaccurate. In a black-box setting, one only has access to input and output of the model, but not the intermediate features. To me, this is more like the gray-box setting used in [1]. 

Another one of my major concerns is the limited technical merit. The proposed method is a direct application of GMM on the features X and Y. The only difference is that, instead of having one GMM modeling X, the authors propose to use two GMMs, one for X and one for Y, and connect them with a projection matrix Q. The idea of joint modeling is not new either [2]. 

The learning algorithm also follows the typical EM algorithm for learning GMMs. 

There are also places in the derivation that may not be correct. For example, the set of equations above Eq (2) is a bit confusing. Why is $[g_x]_i$ a Gaussian distribution parameterized by ($\hat{\mu}_i, \hat{\Sigma}_i$)? Shouldn’t it be one entry of the vector $g_x$? What parameters are $\Theta$ and $\lambda$? In other words, what posterior distributions are $\Theta$ and $\lambda$ associated with? What is the iterative algorithm to estimate these parameters. For example, how is $Q$ estimated? Without these important details, it is difficult to evaluate the correctness of the paper’s method part. The authors need to move more details from the Appendix to the main paper. The main paper in its current form still needs some work. 

Equation (4) is confusing and potentially problematic. The authors mention that this refers to directly sampling from a multivariate Gaussian distribution $p(x| w = j)$. However, $w$ is the component ID for the second GMM for $y$, while $x$ is model by the first GMM. From Sec. 3.1, if one uses $q_j$, the sampling is in fact also from a GMM, not from a Gaussian distribution. 

It is unclear how the definition of counterfactuals in Eq (7) is consistent with the standard definition of counterfactuals in Pearl’s do-calculus. It is also unclear how $\alpha_i$ in Eq (8) is related to $p(z=i|x)$ in Eq (7). 

The extension of the proposed method to DCNN is unclear. Specifically, the authors proposed to train $|R|$ matrices of $Q$ between S and T, but from the text it is unclear which $|R|$ positions to choose. 

Eq (9) is incomplete. It is unclear why the definition is the reverse of counterfactuals. 

Given that one of the focus in evaluation is to generate counterfactual explanations, some important baselines are missing, e.g., [3]. 

The baselines (decision trees, logistic regression, etc.) used for evaluating proxy model accuracy are weak. The authors did not even include the 2016 method LIME as a baseline. More recent baselines should be included. Otherwise it is difficult to evaluate whether the proposed method is actually doing well in terms of proxy model accuracy. 

In Figure 5, for the influential example case, it is hard to know what shape of red feather and beaks are typical. It would be helpful to provide more ‘typical’ images rather than only one of them. 

[1] Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate. AAAI 2022. 

[2] Speaker Adaptive Joint Training of Gaussian Mixture Models and Bottleneck Features. ASRU 2022.

[3] CounteRGAN: Generating Counterfactuals for Real-Time Recourse and Interpretability using Residual GANs. UAI 2022. 

Minor: 

There should be a ‘space’ before every bracket. 

summary_of_the_review: This paper proposed a simple joint Gaussian mixture model to generate post-hoc model explanations for DCNN. Essentially the authors use two GMMs to model the higher and lower features Y and X, with a projection matrix Q connecting the component probabilities from these two GMMs. 

Overall I feel the problem the paper explores, i.e., to develop a probabilistic model that is capable of making versatile explanations, is interesting. One of my major concerns is the limited technical merit of the proposed method. Besides, the chosen baselines are severely out of date, especially for the global part (Table 1). Another major concerns is that there are various inaccuracies and missing details that make one question the correctness and rigor of the paper. I feel these are difficult, if not impossible, to address in a single round of conference revision at ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a probabilistic joint Gaussian mixture model (JGMM) that could simultaneously produce various forms of explanations for DCNN, including proxy models, prototypes, criticisms, influential examples, counter-factual examples and semi-factual examples. In this way, the compatibility and consistency among different explanations could be guaranteed. Experiments were conducted to demonstrate the effectiveness of the proposed JGMM framework.
strength_and_weaknesses: [Strength]
1. This paper pointed out a potential problem in Explainable AI, that is, different types of explanations were produced from different explanation methods/systems, which may result in incompatibility and inconsistency between these explanations. To address the problem, the authors proposed a probabilistic JGMM framework that can simultaneously produce various types of explanations. 
2. Experiments verified the effectiveness of the proposed JGMM framework.
3. This paper is clearly written, and easy to understand.

[Weakness]
1. This paper was motivated by a potential problem in XAI, i.e., different types of explanations may be incompatible and inconsistent with each other. However, no theoretical or empirical evidences were provided to support such a claim. It would be great if the authors can give an example to show how different types of explanations will conflict with each other in a specific task. 
2. More experiments are encouraged to demonstrate the effectiveness of the proposed JGMM framework.
a)	For the faithfulness evaluation in Table 1, the authors only compared JGMM with four traditional proxy methods. However, there are many other interpretable models with better performance. A convincing demonstration should include a comparison with SOTA interpretable proxy models.
b)	For many types of explanations like prototypes and criticisms, The authors only showed explanation results produced by the proposed JGMM. There lacks a comparison between the proposed JGMM and corresponding baseline methods. It would be nice to see such a comparison, to show whether the proposed JGMM produced better explanation results or not.
c)	Some additional experiments were needed to further verify the consistency among different types of explanations produced by the proposed JGMM. 
3. The proxy model accuracy in Table 1 used “middle features” for prediction, why not use “low features”? If the proxy model accuracy using “low features” was poor (i.e., low features may not be faithful), were other types of explanations based on “low features” faithful?

Minors 

4. In experiments, the authors adopted “high feature”, “middle feature”, and “low feature”. What is the exact definition of “middle feature”? 

5. In Table 2, the evaluation metric “Counter-factual effectiveness” is not clearly defined in the paper.

6. In Equation (2), the summation in the denominator should be from k=1 to k=K_y, rather than K_x.

7. In Section 2, it is not appropriate to refer to LIME as a proxy model.

8. It would be more clear to give a table to list hyper-parameters used in the proposed JGMM. 

clarity,_quality,_novelty_and_reproducibility: Clarity: good; Quality: good; Novelty: good;
For the reproducibility, there is still room for improvement. More experimental details should be clarified.
summary_of_the_review: This paper is interesting and has significant contributions. It pointed out a potential problem in Explainable AI, i.e., different types of explanations were produced from different explanation methods/systems, which may result in incompatibility and inconsistency between these explanations. To address the problem, the authors proposed a probabilistic JGMM framework that can simultaneously produce various types of explanations. Experiments verified the effectiveness of the proposed JGMM framework. Besides, this paper is clearly written, and easy to understand. Nevertheless, I still have some concerns about this paper. Please refer to the weakness in my review. It would be great if my main concerns can be well addressed. 
In summary, I would suggest to accept this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A new a probabilistic model, named joint Gaussian mixture model(JGMM), is proposed for post-hoc explanations of deep neural networks. By jointly modeling the latent features of lower and higher layers with a joint GMM, JGMM, trained with a post-hoc EM algorithm, is capable of delivering both global explanations (like Prototypes, Criticisms, and Influentials) and local explanations (like counterfactual and semifactual examples). Experiments on common networks and datasets are conducted to demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strength.
(1) The paper is well-written with clear logic.
(2) The presented method is sound.
(3) The empirical experiments are convincing.

Weaknesses.
(1) The limit of the proposed JGMM is not fully demonstrated, e.g., under what circumstances will JGMM not work well?
(2) Section 3.3 is believed crucial for improving the practicability of the JGMM. However, this section is not easy to follow.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written. The presented method is believed novel. The reproducibility is likely OK as no code is available during the review.
summary_of_the_review: Explanations of deep neural networks are essential. The presented JGMM might serve as an explaining framework that produces versatile, consistent, faithful, and understandable explanations.

Notations should be revised carefully. For example, those used in the figures are inconsistent with those in the text. In Eq (1), the function G() is not defined.

I am quite concerned about using the GMM to model lower features (like the input images). Is GMM expressive enough? Or how does Section 3.3 help with that?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a joint GMM model for creating post-hoc explanations for high- and low-level features of a DNN. Better understanding of deep learning models is a very important topic, and it's great that the authors focused on this area. The reviewers, however, pointed out important issues with the model and manuscript: the PGM is not well formulated, setting is incorrectly specified (black-box vs gray-box), the hyperparameter choices not well explained, and the manuscript contains many typos. The paper would improve with a careful reevaluation of these areas.
justification_for_why_not_higher_score: The reviews for the paper are mixed. Particularly concerning are reviewers comments that the probabilistic graphical model is not well formulated, and that important baselines are omitted in the experiments section. The reviewers also noted a higher number of grammatical mistakes than average. With the suggested improvements, I believe that the paper will be stronger.
Decision: Reject