reviews: 
Comments: The paper proposes to address the problem of explanation disagreement across different models (of same performance on a dataset, called a Rashomon set), and across different explanation techniques. The authors say that disagreement between explanations hurts reliability of the model if and when deployed in high-stakes environment. They propose EXAGREE where they talk about 4 avenues of such disagreement, and propose an approach that **apparently** addresses the problem.
Strengths: The problem that the paper focuses on is interesting and the paper does a good job at disentangling the four scenarios of explanation disagreement. 

Originality and Significance (of the problem): High
Quality and Clarity: Poor
Weaknesses: The paper has a lot of weakness in my opinion and I delineate them as follows. You do not need to address all of them in the rebuttal, I have marked points that are major weakness, you can focus on them. 

1. Starting with Figure 1 on Page 1 -- totally unclear. How does the right side of the figure indicate agreement while the left does not? (this figure is not crucial to understand the paper, but I am just making a point of why the paper is so unclearly written and you do not need to address this point in the rebuttal, there are many more important problems. )

2. [Major] In Section 2 (Preliminaries), there are several mistakes in the formulation:

**a)** M* is not defined in Eq. 1. I assume what you mean is the optimal model by M* (the model with the best performance on the data). 

**b)** a_1, a_2,...,a_p are not defined as features.  

**c)** In Equation 1, I think the sign should be >= instead of <= (the loss of the other models will be higher than M* (assuming M* refers to optimal model)

**d)** Why is $\epsilon$ multiplied by L(M*), usually similar performing models are supposed to be an absolute threshold, not relative to the loss of the original model. 

3. [Most Major] In Section 2.1, you mention 4 axes of explanation disagreement. I am with you on the model and method disagreement, but what does stakeholder and ground truth disagreement mean? 

**a)** In Stakeholder disagreement you mention "Different stakeholders in S might prefer different rankings". Why would that ever happen? Do these stakeholders want something that is not real and want fake explanations that aligns with their mental expectations of what a model should care about? What is an example of such stakeholders wanting something different from reality? Your current example of a data scientist wanting statistical significance and domain expert valuing different features **does not** answer this. Does wanting statistical significance and valuing different features amount to wanting different ranking of features (this also assumes that these rankings won't be same in the first place)? And even if they want different rankings, should a technique be optimized to suit their demands and provide fake explanations? 

**b)** In ground truth disagreement you mention "ground truth interpretations (i think you mean explanations) from interpretable models can conflict with post-hoc explanations" -- yeah this is obvious, but does it matter? If I have an interpretable model, why do I care what a post-hoc technique says, I will never ever use that for such a model? Why would one do that? 

**c)** In the primary objective of the paper you write: "identify a well-performing model that minimizes disagreement (or maximizes agreement) between model explanations and stakeholder expectations. " -- where stakeholder expectation is either something we should not give them because it is fake or something I do not understand. Your rebuttal will help me understand if it is the latter (a solid convincing example is required). 

5. Section 2.3, I think the title of that subsection should be "Evaluation Metrics" and not "Evaluation Matrices". Anyway, in that section you mention two metrics: faithfulness assessment and fairness assessment. In faithful assessment you give a further breakdown in several metrics -- however, most of these metrics seems highly correlated, for e.g., the attribution values directly affect the ranking, so FA, RA, RC, SA, SRA, PRA should have extreme correlation (when you consider the attribution values with its sign). Can you measure this please? If they are highly correlated, then that is effectively one metric and not six. If they are not correlated, can you explain why that is not the case (except the case where you consider the absolute values, its obvious then). 

6. You have used the word "mask" in line 256, without defining, what does it mean? Also in line 259 what does the phrase "bridging the gap between models in Rashomon set and their feature attributions mean" mean? each model in the Rashomon set (or any model for that matter) has a feature attribution (produced by any technique), what is the gap between them -- they don't even lie in the same space, one is in the weight space and other is in the explanation space (they have different dimensionality). 

7. [Major] You have this complicated (and not at all well explained) pipeline of computing SAEM which is basically the models that have the highest agreement with stakeholder requirements. Please tell me why do you need this pipeline? You have the models that are trained on that dataset, you can use that to compute the set of the models that are best performing (according to some threshold $\epsilon$), and then compute the explanations of these models using whatever method you like, and then just give those models to the stakeholders with whom they have the highest agreement. What is the job of the models like DMAN, you are just using that to predict the model that will have the highest agreement (is that a correct understanding of the model's job?), but you don't need that you have the ground truth and you even say this in lines 268-269. 

8. [Major] The above were the problems with the problem setup, now coming to the experiment section. 
**a)** In table 1 you mention k= 0.25, what does k stand for? You have used k and l in Section 2.1 without ever defining them. For the same reason, I do not comprehend what does the k in caption of Figure 3 stand for. 

**b)** Why is SAEM technique bolded in Tables 1 and 2. The standard practice in ML papers to the bold the best performing method, but your technique, and this is misleading. 

**c)** If SAEM selects the models with the highest agreement with the stakeholder, why is it not the best for LR in Table 1 and 2? (especially in Table 2, it is pretty behind techniques that are not optimized for agreement, which alludes to the case that explanation disagreement might not even show up in actual experiments?)  

**d**) how is FIS_LR computed? Is this not just the feature importance and since you are using ground-truth explanations from pre-trained LR, this should have 100% agreement and should be the best 8 number of times (currently in Table 1 it is best 0 times)?
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 1