Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new framework for measuring trustworthiness of a model including new proposed metrics that can evaluate various models comparingly. The framework is applicable to text and image domains and is validated through experiments on datasets from each domain accordingly. The results showed that a model with higher accuracy might not be trustworthy according to the proposed metrics and framework introduced in the paper.
strength_and_weaknesses: **Strengths:**
1. The problem considered in this paper is interesting.
2. The framework is applicable to text and image domain.
3. Paper is well-written and easy to follow.
4. The collected datasets in this paper can be useful to the community.

**Weaknesses:**
1. The approach needs ground truth explanation collection for each task which I am not sure how feasible might be.
2. In some cases this ground truth collection might not be easy in cases where expert knowledge is required. What do authors think about these types of overheads. Please discuss.
3. The paper is not technically rigor and am not sure about the extend of novelty introduced in this work.
4. Authors (and ultimately the metric and framework proposed in this work) assume that the explanation methods are themselves complete and reliable such that the measures are being build upon these explanation methods. More discussion and justification needs on this issue.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. In terms of writing and some performed experiments, the paper can be considered as somewhat high quality; however, in terms of technical rigor and proposed framework and metric more discussions need to take place. The authors mentioned that data and code will be available for reproducibility but at the moment no such material is realsed.
summary_of_the_review: Overall, the problem considered is interesting, but the paper has some flaws in terms of lacking some more in depth discussions along with technicality.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors provide a quantitative measure to check the correspondence of feature importance as detected by a model with human perception of a feature's importance, which is incorporated in a framework to quantify model trustworthiness. 
strength_and_weaknesses: + The paper is well motivated and well written
+ Recent work seems to have been discussed
+ The measures are simple but intuitive
+ Experiments are conducted on real datasets

- It would be nice if the measures can be incorporated to produce both trustworthy and effective clusters
clarity,_quality,_novelty_and_reproducibility: All of the above are satisfactory
summary_of_the_review: I enjoyed reading the paper because it addressed the problem with clarity, and due to the simplicity of the defined measures
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a quantitative measure of trustworthiness of a black-box model trained on text or image datasets. This measure requires a set of ground-truth explanations to compare them against explanations provided by an explainable artificial intelligence (XAI) method applied to a black-model model. After introducing their measure, the authors test their measure on a set of datasets. 
strength_and_weaknesses: Strength

It is quite important to come up with a measure for trustworthiness of trained models. The authors have motivated their study nicely, and presented their measures in a clear manner.

Weaknesses

Both measures are straightforward. This could be an advantage, but this also causes them to overly simplify the concept of providing explanations, or for that matter, measuring trust. The ground-truth explanations for images are obtained from experts. There could be a few concerns there: First, creating a binary matrix for an image is a tedious task that is prone to errors. Second, when looking at an image, the humans look at the pattern that are caused by a collection of pixels not just single points (e.g., a skin lesion where the doctors look at the symmetry or the size of the lesion). When presented with same images that are transformed (e.g., rotated) the ground-truth would completely change.

A few questions for the authors:

1. SHAP also returns negative weights. In that case, do you use the absolute values of these weights as the feature importance?

2. As authors clearly stated on page 6, "[...] it is not the presence but the absence that needs to be considered [...]" This is very important, and I believe cannot be addressed with the current measures. Am I right?

3. Both measures are obtained by averaging over all samples in the evaluation set. Could the sample variance or the outliers would be of more interest than averages?
clarity,_quality,_novelty_and_reproducibility: Paper is clearly written but neither the quality nor the originality suffice for publication in ICLR. Since the authors have not provided the codes, the results are not reproducible.
summary_of_the_review: The authors have tried to provide a solution to a very difficult problem. Both proposed measures, in their current states, are only ad-hoc solutions.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This work develops a framework to evaluate the trustworthiness of a model using XAI tools and TWP, TWR that are developed in this work. Then, it illustrates an application of their framework to several real-world settings.   
strength_and_weaknesses: Strengths: 

-- Interesting and important topic. As rightly pointed out having XAI methods are not sufficient to argue whether a model is trustworthy or not. 
-- This work is not about evaluating XAI models, it is about the evaluation of a trained ML model. This adds a new dimension to XAI literature. 

Weaknesses:

-- there are critical logical flaws in the arguments and models. 
-- the models developed in this work do not solve the problem raised in the motivation. 
clarity,_quality,_novelty_and_reproducibility: While the topic raised in this work is important and interesting, the proposed model suffers from critical drawbacks and logical flaws. 

The main goal of this work is to develop an algorithm and a set of metrics to evaluate the trustworthiness of a model (independent of XAI's trustworthiness). This work uses XAI tools (SHAP and LIME) as an intermediate step for trustworthiness evaluation. It is not obvious that a model with low TWP and TWR is indeed untrustworthy or that the XAI methods produced unacceptable explanations, i.e. the  XAI is untrustworthy. 

To address this drawback, this work argues that: "The objective in our experiments is to compare the predictive models in terms of their trustworthiness. We do not intend to evaluate the effectiveness of the XAI methods themselves. Therefore, fair comparisons are only to be made between predictive models where the explanations are also obtained with the same XAI methodology, e.g., comparing the TWP or TWR values computed with different explanation methodologies is not fair." 
Now, suppose using SHAP, one finds model A has AUC = 0.9 while TWP and TWR are below 0.6, and model B has AUC = 0.6 while TWP and TWR are above 0.9. Should we trust the second model more than the first model? Suppose there is a model with AUC = 0.99 (MNIST) but TWP and TWR are below 0.6 while for another model with AUC = 0.95, TWP and TWR are 0.8, is this guaranteed that the second model is more trustworthy than the first model? An XAI method can easily fail in producing a good explanation and then using the proposed method one might conclude that a model is untrustworthy which is incorrect. This is actually not a hypothetical situation, the authors already have shown this in Table 2. 

The authors correctly mentioned that the trustworthiness of a model should be independent of the XAI method, but the current method cannot distinguish between an untrustworthy model and when the XAI methods are untrustworthy (not the actual model).

It is also sensitive to annotation. Another important problem is with human annotation. The regions annotated by humans as the regions of interest might not actually be the most optimal regions to differentiate between two classes (for example in distinguishing between 3 and 8). A model that looks for two connected closed loops to distinguish between 3 and 8, is a good model, a trustworthy model, and can achieve the goal of distinguishing between 3 and 8. There is no need for a trustworthy model to find the pixel level difference between 3 and 8. Hence, the human annotation that is used as a reference point is subjective, while the trustworthiness of a model should not be subjective.  
summary_of_the_review: This work has important logical flaws and I cannot recommend this work for publication even after a major review.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The authors proposed a general framework to quantify the trustworthiness of a black-box model on image or text datasets.

Strength:
Paper is well-written. This is an important topic to measure the trustworthiness of the models. 

Weakness:
There are major concerns in the rigorousness of the work, overly simplifying the concept of measuring trust, the sensitivity/feasibility of the framework for getting annotations, etc.  

justification_for_why_not_higher_score: There are major concerns in the rigorousness of the work, overly simplifying the concept of measuring trust, the sensitivity/feasibility of the framework for getting annotations, etc.
Decision: Reject