Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of "global counterfactual explanations," which extend methods from local counterfactual explanations (of individual data instances) to draw global insights about a model. They propose a new framework, which they call GLOBE-CE, which is based on feature translations of vectors of variable length, and provide a mathematical analysis of the implications of this for categorical features. They also improve the efficiency of an existing one (ARES), and compare the two with real world datasets and user studies.
strength_and_weaknesses: Strengths: The key idea behind GLOBE-CE -- that if groups of instances are explained together by a single set of counterfactuals based on a translation in the input space -- is clever, and, to my knowledge, novel. The experiments seem to show that GLOBE-CE provides a good tradeoff between reliability and efficiency. The paper is very clearly written. The insights about categorical features are also interesting.

Weaknesses:
- There are, in my opinion, too many contributions in this paper for me to fully understand all of them without going through the appendix (and related work) in some detail. Between all the contributions, there appears to have not been room for exposition, definitions, and evidence. For instance, the concept of "recourse cost" is never precisely defined, though identifying bias in it appears to be the phenomenon of study that the methods proposed are meant to uncover. Another example is that the explanation of how AReS works is not really sufficient to follow the paper's contributions to it; I'd have to go read the original paper to understand how it works, and without that intuition, the statement that "analyses suggest that AReS is highly dependent of the ground set V" isn't very convincing.
- The takeaway from the small user study is not obvious to me; measuring the effectiveness of the tool deserves a more in-depth evaluation than is provided here. (However, in a paper introducing this method I would not consider such a user study necessary.)
- I also honestly think the title is somewhat misleading; I thought this paper would quantify reliability and efficiency and either empirically or theoretically examine those tradeoffs. Instead, the paper consists of several contributions vaguely connected by the theme of tackling reliability and efficiency.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written (although again, in my opinion, skims over too many details for me to get a super strong picture of what's happening). I don't know the related literature to assess the degree of novelty, but it appears to be novel. The quality of what I was able to fully understand was strong, but the rigor / reproducibility were difficult to assess when most details were delegated to the appendix.
summary_of_the_review: I think there's some strong, interesting, and important work in this paper, but the missing context that seems to have been expected of the reader made it difficult to evaluate. With more pages devoted to the details, I believe this would be a very good paper. But I am not thrilled about its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 1. The efficiency of Actionable Recourse Summaries (AReS) is improved, and a method of Global & Efficient Counterfactual Explanations (GLOBE-CE) is designed to address the stability issues flexibly.
2. The authors provide a mathematical analysis of categorical feature translations and use it in their presented method.
3. Experiments on real-world datasets and user studies verify the proposed method's speed, reliability, and interpretability improvements.
strength_and_weaknesses: Strengths:
1. This paper attempts to balance efficiency and reliability for global explanation methods of model behavior, which is a valuable perspective in machine learning practice.
2. As the authors claim, it is the first to mathematically address the direct addition of translation vectors to one-hot encodings in the context of Counterfactual Explanations.
3. Overall, the experiments seem well-designed, and the results are promising. Moreover, the improvements of the proposed framework are decent over other methods and consistent in various datasets and under different settings.
 
Weaknesses:
1. Since the framework is built on many machine-learning models, it would be too hard to reproduce this work because it misses quite a few details, such as how to adapt to various models via different hyper-settings.
2. Categorical Translations have their limitation, such as omitting the rules of ’If A then B or C‘. Unavoidably, based on Categorical Translations, the presented framework may fail in similar cases. 
clarity,_quality,_novelty_and_reproducibility: 1. Clarity: Overall, The paper is well-written and easy to follow.
2. Quality & Novelty: This paper addresses an important challenge in global explanation frameworks, and makes a good attempt to provide mathematical insights. The main idea is well-motivated and intuitively useful.
3. Reproducibility: it would be too hard to reproduce this work because the main body and the appendix only provide insufficient details. I look forward to the authors' open-source implementation if this paper gets accepted.

summary_of_the_review: My pre-rebuttal rating is '6: marginally above the acceptance threshold'. I really appreciate the problem that this paper addresses and their effort in balancing reliability and efficiency. My concerns are mainly about the limitations caused by Categorical Translations and reproducibility. Please see the detailed concerns in the 'Weaknesses'. I would be happy to read the responses from the authors and willing to increase the score if my concerns are well-addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a framework for global counterfactual explanations while considering the bias that can arise across different subgroups and the need to allow for different translations. The paper also provides an analysis of categorical feature translations. Emprirical results support the theoretical analysis.
strength_and_weaknesses: Strengths: 

1. Well-motivated idea with clear illustration, especially Figure 2.


Weakness:
1. Clarity on how the mean cost for the translation vectors is estimated in Figure 1 (right-most figure) is needed.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written, adding more information in figures 1,2, and 3 would be helpful.
summary_of_the_review: I vote for accepting this paper considering the proposed framework for global counterfactual explanations.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: I spent an hour struggling to read this and, truthfully, I'm still not sure what it's about. 
strength_and_weaknesses: A caveat for the rest of this review: I am not an expert in explanation. As such, much of the jargon was unfamiliar to me. It's possible that an expert in this area would have been able to make more sense of the paper than I did. That being said: this paper is very unclear to a reader who is just a general expert in machine learning. Indeed, I actually still can't figure out what, precisely, this paper is about.

Some illustrative examples of clarity problems:
1. as far as I can tell, "global counterfactual explanation" is not defined

2. it's not clear what data setup the authors have in mind. I guess these are explanations for predictive models. In section 3.1, the authors write "x in X_sub". In context, I guess this notation is meant to be input features. What space do these features live in? Immediately after, the paper uses x + k*\delta, where \delta is a vector. That would seem to imply that x is a vector of real numbers. Does this method then only work for data where the features are vectors of real numbers? 

Similarly: what does subgroup mean in this context? What is a "successful" counterfactual? 

These confusions all occur in a single sentence. I believe such things could be clarified pretty easily; the issue is that they are not currently.

3. The takeaway messages from Figures 1 and 3 are totally unclear by looking at the figures. The captions do not explain the purpose of these figures.

4. I can't find any result in the paper that appears to correspond to the claim, "Global Counterfactual Explanations are reliable or efficient, but not both"
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: I was not able to understand this paper well enough to comment on its technical merits. I believe that this is fundamentally an issue of lack of clarity. This paper would need significant improvements to be published.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes various theoretical and technological aspects related to global counterfactual explanation proposing various methods trying to empirically prove the statement "Global Counterfactual Explanations Are Reliable Or Efficient, But Not Both". In particular, it proposes a novel global and efficient counterfactual explainer GLOBE-CE, and improves an existing global counterfactual explainer named Actionable Recourse Summaries (AReS). 
strength_and_weaknesses: + A novel counterfactual explanation method is proposed
+ An existing counterfactual explanation method is improved
+ An interesting and limitedly addressed research direction is studied

- The content is not organized in the best possible way, and the narrative is difficult to follow and chaotic in various passages.
- The Fast AReS extension is not clearly explained. 
- The experiments are performed on a limited number of datasets and evaluated with a limited and not updated number of measures.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper should be re-structured and reorganized. The overall history and concepts become clear only after several readings.

Quality: The theoretical presentation seems sound. The experimental part lacks statistical significance.
 
Novelty: the idea of Global CF is limitedly addressed in the literature, but the methods proposed/extended are not disruptive in terms of novelty.

Reproducibility: code is not publicly available, but many details are provided to replicate the experiments.
summary_of_the_review: The content of the paper might be interesting, but before reaching the qualified level for this conference, the paper should be re-structured and better presented. Experiments should include additional evaluation measures such as those used in Mothilal, R. K., Sharma, A., & Tan, C. (2020, January). Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 607-617) or Guidotti, R. (2022). Counterfactual explanations and how to find them: literature review and benchmarking. Data Mining and Knowledge Discovery, 1-55 or Mazzine R, Martens D (2021) A framework and benchmarking study for counterfactual generating methods on tabular data. Appl Sci 11(16):7274. Also, a comparison against a baseline "dummy" global counterfactual explainers combining in a naive way local counterfactual explainers should be reported.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The papers tackle the problem of "global counterfactual explanations," which extend methods from local counterfactual explanations (of individual data instances) to draw global insights about a model. All the reviewers agree that the paper contains some interesting ideas, however, all of them also brought up that the presentation/structure of the paper impairs both the understanding of the contribution as well as the reproducibility and the issue is significant enough for several of the reviewers not to be in support of accepting the paper. I encourage the authors to do a major revision of their paper in light of the reviews and resubmit to another top tier venue.
justification_for_why_not_higher_score: The presentation issues are significant enough for a major revision & resubmission.
Decision: Reject