reviews: 
Comments: The paper titled "EXAGREE: Towards Explanation Agreement in Explainable Machine Learning" addresses the challenge of explanation disagreement in machine learning. Explanation disagreement, where model explanations diverge based on methods, models, or stakeholder expectations, hampers trust and transparency in high-stakes environments. The authors propose a framework called EXplanation AGREEment (EXAGREE) that utilizes a Rashomon set—multiple models with similarly good predictive performance—to align explanations with diverse stakeholder expectations. By optimizing within this set, EXAGREE identifies Stakeholder-Aligned Explanation Models (SAEMs) that reduce disagreement while preserving predictive accuracy.

The authors formalize four types of explanation disagreement: stakeholder, model, explanation method, and ground truth disagreements. EXAGREE addresses these by introducing a two-stage process: Rashomon set sampling, followed by SAEM identification. Empirical analyses demonstrate that EXAGREE reduces explanation disagreements and improves fairness across datasets, positioning it as a potentially valuable tool for practitioners aiming to enhance trust and fairness in machine learning applications​.
Strengths: - The paper addresses a critical challenge faced by researchers and practitioners: how to proceed when even explainable AI tools disagree on feature importance. Additionally, it incorporates model and stakeholder rankings, making the approach quite comprehensive.
- The paper tackles its proposed problem by integrating methodologies from several different areas, including the XAI literature as well as general AI methods for optimization challenges.
- The insight to make the process end-to-end differentiable is both creative and practically useful.
In the appendix, the authors demonstrate the impact of the choice of $\epsilon$ on the Rashomon set, which serves as a valuable methodological sensitivity analysis.
- The empirical results test their methods across a variety of settings: 6 OpenXAI datasets, both synthetic and empirical; and 2 pre-trained models (logistic regression and artificial neural networks).
Weaknesses: The paper integrates several methods and techniques to address the explainability disagreement issue; however, it can feel somewhat dry and lacks the depth and technical details that would enable readers to fully appreciate the contributions and identify strengths and weaknesses. Technical jargon used to describe the methodology needs precise definitions, mathematical arguments should be clearly defined and explained, and additional background information would offer useful entry points for readers. Most of the following points align with this suggestion. This lack of precise definitions also contributes to my limited confidence in the recommendations, as it made it challenging to fully assess the work's potential impact.
- The loss functions for $L_{sparsity}$ and $L_{diversity}$ are not defined clearly in the paper or the appendix.
- Precise mathematical definitions of what a mask is and how it is derived are essential for readers to understand the methodology in depth, as this concept is central to the approach.
- Consider adding a sentence or footnote to define core terms in your algorithm, as these abstract concepts can vary in meaning:
    - "attribution set", "model representations", "model characterizations", "end-to-end optimization".
    - For instance, in the sentence "Training a Differentiable Mask-based Model to Attribution Network (DMAN) that maps feature attributions from model characterizations for use in the next stage," it would be helpful to clarify precisely what “model characterizations” entail.
    - Also, in Equation 4, where $f_{DMAN}^*$ is defined as the optimal surrogate model in the Rashomon set that describes feature attributions, it appears to be the loss between $f_{DMAN}$ and a set comprising ${\text{masks}, \text{attributions}}$. Minimizing the output to such abstractly defined elements would benefit from more clarity.
- It would be helpful to include insights or references for the result in row 196. Why is faithfulness proportional to agreement? Is this a theoretical result, an empirical finding from the paper, or something else?
- Figure 1 provides few entry points for readers and doesn’t seem to aid in understanding at its current placement. Consider either removing it or adding more descriptive captions to clarify each step (similar to Figure 2, which includes more context). Suggestions include captions for the rankings, lightbulb, etc. Additionally, why are stakeholders grouped together in the first half but not in the second half?
- Figure 2 is clear, but it seems to appear too early in the paper. Moving it to the end of Section 3 might make it more helpful, as readers would have more context to interpret it.
- Is the fairness improvement an explicit objective of the EXAGREE model? If so, please explain the rationale and mechanism. If it’s an outcome of the empirical analysis, please clarify this in the paper, as empirical results may not generalize across all applications.
- In row 276, in the explanation of the "Ranking Supervision and Correlation Metric," it would be beneficial to provide more context and motivation for this metric and how it fits in the big picture of your methodology before defining it.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 1