Title: Performative Validity of Recourse Explanations
Venue: NeurIPS 2025
Accepted: yes
Keywords: recourse

explainable machine learning

XAI

performativity

causality

distribution shift

robustness
Abstract: When applicants get rejected by a high-stakes algorithmic decision system, recourse explanations provide actionable suggestions for applicants on how to change their input features to get a positive evaluation. A crucial yet overlooked phenomenon is that recourse explanations are *performative*: When many applicants act according to their recommendations, their collective behavior may shift the data distribution and, once the model is refitted, also the decision boundary. Consequently, the recourse algorithm may render its own recommendations *invalid*, such that applicants who make the effort of implementing their recommendations may be rejected again when they reapply.
In this work, we formally characterize the conditions under which recourse explanations remain valid under their own performative effects. In particular, we prove that recourse actions may become invalid if they are influenced by or if they intervene on non-causal variables. Based on this analysis, we caution against the use of standard counterfactual explanations and causal recourse methods, and instead advocate for recourse methods that recommend actions exclusively on causal variables.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the terms performative effect of recourse explanations and performative validity. \nThey then investigate whether recourse explanations remain valid under their performative effects. They theoretically prove that recourse becomes performatively invalid only if the implemented action \ncarries predictive information about the post-recourse outcome.\nLastly, they empirically study the performative validity of three recourse methods: counterfactual explanations, causal recourse, and improvement-focused causal recourse.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The authors introduce a new perspective on recourse generation and connect the performative prediction to the CFE/recourse generation. \n- The problem is interesting and offers useful insights to the community. \n- The theory is well-written and easy to comprehend. Also, Figure 2 is intuitive.  \n\n**Weaknesses**\n\n- While certain features, such as a master's degree, may seem more costly to manipulate compared to others, like GitHub activity, this does not guarantee that acting on them truly reflects genuine qualification.  In the current landscape of generative AI tools, \n(e.g., ChatGPT and Gemini, among others), and cheating rings, among other shortcuts, even ostensibly robust signals (e.g., an academic degree) are manipulable (effect variables).  The feature-level recourse methods (whether CE, CR, or ICR) that suggest how much a user should change each feature (\"Translating CEs into Actions\" G4 in the appendix) are vulnerable to performative invalidity in practice. \nUltimately, decision-makers only see the (new) revealed features, not the underlying process by which users reach there.\n\n- I presume there are cases where the variable \"a\" is an effect variable in model 1 but is no longer an effect variable in model 2. Conversely, the variable \"a\" may not be an effect variable in model 1 but becomes an effect variable in model 2. For example, a model in the year 2012 that used Github activity versus a model in 2025 that used Github activity. How robust are the structural causal model generation processes to these sorts of changes, and how does that affect the performative validity?\n\n- I need some clarification. If \"neither the decision model nor the recourse algorithm is influenced by information about GitHub activity to arrive at their outputs,\" why is GitHub activity considered a feature in the model training? Why is it appearing in the SCM?\n\n- Experimental details are sparse, and lots of key information is missing. \n   - Could the authors include more details on the datasets (size, features, cleaning, etc.) and the generated SCMs?  \n   - From the experimental setting (G.2), the authors say they use logistic regression and decision-tree models. However, it's unclear how accurate they are on the datasets, how the models change after retraining, and so on. It's also unclear to me which model they use on which dataset and when.\n   - How did the authors compute L^p in the ICR case?\n\n\n- From the experimental setup, the authors mention that all settings include one cause and one effect variable. I am unsure how scalable the methods are. \n\n- Since the authors' work is more aligned with the robust CFE generation approaches, I am curious about the empirical comparative analysis between the robust CFE generation methods (https://www.ijcai.org/proceedings/2024/0894.pdf) and the authors' proposed perspective. \n\n- Since theoretical findings show that performative validity depends on the recourse method, would action-based, not feature-level, recourse methods, as those authors use, and would sequence-based recourse methods, and not the one-shot recourse methods authors use, do as well as the ICR at a computationally cheaper rate?\n\n\n**Miscellaneous**\n\n- When characterizing the invalidity of the recourse, the authors frame the problem from the user's perspective. That is, the issue seems to mainly affect users (unless there is a computed true positive minus false positive rate or true negative minus false negative rate). Therefore, I am curious, what incentives do the decision-makers have to choose an expensive and difficult method like ICR (e.g., gathering causal knowledge is not trivial and is costly) over CE methods? \n- Synthetically generated noise is a feasible way to test the model changes, but it does not sufficiently capture the complexities of change in data distributions and model drift.\n- Authors look at invalidity from the perspective of the recourse not making the user qualified anymore. How about if h(x')=1 but f(x')=0? What are the implications from the casual perspective and long-term effects on both the user and the decision-maker?\n- I am curious about the fairness implications of the suggested setup. \n\n\n**Minor**\n- Typos in the paper, e.g., decison instead of decision\n\nQUESTIONS:\n- While certain features, such as a master's degree, may seem more costly to manipulate compared to others, like GitHub activity, this does not guarantee that acting on them truly reflects genuine qualification.  In the current landscape of generative AI tools, \n(e.g., ChatGPT and Gemini, among others), and cheating rings, among other shortcuts, even ostensibly robust signals (e.g., an academic degree) are manipulable (effect variables).  The feature-level recourse methods (whether CE, CR, or ICR) that suggest how much a user should change each feature (\"Translating CEs into Actions\" G4 in the appendix) are vulnerable to performative invalidity in practice. \nUltimately, decision-makers only see the (new) revealed features, not the underlying process by which users reach there.\n\n- I presume there are cases where the variable \"a\" is an effect variable in model 1 but is no longer an effect variable in model 2. Conversely, the variable \"a\" may not be an effect variable in model 1 but becomes an effect variable in model 2. For example, a model in the year 2012 that used Github activity versus a model in 2025 that used Github activity. How robust are the structural causal model generation processes to these sorts of changes, and how does that affect the performative validity?\n\n- I need some clarification. If \"neither the decision model nor the recourse algorithm is influenced by information about GitHub activity to arrive at their outputs,\" why is GitHub activity considered a feature in the model training? Why is it appearing in the SCM?\n\n- Experimental details are sparse, and lots of key information is missing. \n   - Could the authors include more details on the datasets (size, features, cleaning, etc.) and the generated SCMs?  \n   - From the experimental setting (G.2), the authors say they use logistic regression and decision-tree models. However, it's unclear how accurate they are on the datasets, how the models change after retraining, and so on. It's also unclear to me which model they use on which dataset and when.\n   - How did the authors compute L^p in the ICR case?\n\n\n- From the experimental setup, the authors mention that all settings include one cause and one effect variable. I am unsure how scalable the methods are. \n\n- Since the authors' work is more aligned with the robust CFE generation approaches, I am curious about the empirical comparative analysis between the robust CFE generation methods (https://www.ijcai.org/proceedings/2024/0894.pdf) and the authors' proposed perspective. \n\n- Since theoretical findings show that performative validity depends on the recourse method, would action-based, not feature-level, recourse methods, as those authors use, and would sequence-based recourse methods, and not the one-shot recourse methods authors use, do as well as the ICR at a computationally cheaper rate?\n\nLIMITATIONS:\nNo, there is no limitations section in the main paper or appendix. \nIt would be good to comprehensively address both the theoretical and empirical limitations of their work.\nPlease see the weakness section and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates performative effects, a phenomenon where widespread adoption of recourse recommendations shifts the data distribution, potentially rendering the recommendations ineffective after the model is retrained. To address this, the authors introduce the notion of *performative validity* and formally characterize the conditions under which it can be preserved. They show that performative validity fails when recourse actions are either influenced by effect variables or directly intervene on them. Empirical evaluations support the theoretical findings, confirming that avoiding effect variables helps maintain performative validity.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  - Originality: The paper’s focus on performative effects and its formalization of performative validity is novel. This perspective has received limited attention in prior work, and the study is likely to inspire follow-up research in this emerging area.\n  - Quality: The finding that *improvement-focused causal recourse (ICR)* consistently maintains performative validity across diverse data-generating processes—such as resampling and additive noise—is theoretically and empirically compelling. To the best of my knowledge, this is the first paper to establish the superiority of ICR over CE and CR from the perspective of performative validity.\n  - Clarity: The paper explains related work, prerequisite concepts, and proposed methods with clarity and depth. Examples (e.g., Figure 1) and causal diagrams (e.g., Figure 2) effectively illustrate key ideas. Furthermore, thoughtful presentation choices, including text color and layout, enhance readability. The citations also appropriately cover the foundational work needed to situate this study’s contribution.\n- Weaknesses\n  - Significance: The study's practical impact could be better emphasized. While research on algorithmic recourse has gained substantial attention in academic circles, its societal deployment remains limited. Moreover, it is unclear which real-world domains performative effects occur in and how frequently. Some discussion or empirical justification on this point—possibly based on the experimental results in Q1—would be valuable. Although the GPA prediction task offers a semi-realistic use case, assumptions such as the frequency of model retraining may limit its generalizability.\n  - Practical scope: The reliance on causal knowledge narrows the proposed approach's applicability. As the authors acknowledge in Section 6, extending the framework to settings with incomplete or imperfect causal understanding is an essential and promising direction for future work.\n\nQUESTIONS:\nAre there any reliable sources that show how often performative effects occur in the real world? I'm ready to raise the Significance score if they're a frequent issue.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this paper is to address the problem of reliable algorithmic recourse (counterfactual explanations) specifically taking into account how recourses can change the distribution which in turn changes the models. Specifically: when many applicants act according to the recourse recommendations, their collective behavior may shift the data distribution >> shifts the new model trained on the updated data >> changes the model boundary >> makes the recourses invalid (since they are typically the closest points on the decision boundary). The paper investigates this crucial problem through a causal lens, showing that recourses may become invalid if they are influenced by  non-causal variables, and hence, advocates for recourses that specifically focus on causal variables.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nThis is a very interesting perspective on the challenge of reliable algorithmic recourse. Prior works have looked into this problem, aiming to provide a robust algorithmic recourse (or, robust counterfactual explanations) that finds recourses that would remain valid even under model changes. This work takes a different perspective on this problem, bringing in several novel elements:\n1. noting that data distributions change when people act upon the recourses, potentially leading to certain model changes (rather than a random model change)\n2. bringing in a causal perspective into the problem, showing how recourses that alter non-causal (spurious) features/variables can change the data distribution in a way that new models might no longer depend on that feature, thus making the recourses invalid\n\nProposition 4.1 is quite a surprising result - particularly the if and ONLY if part of the result. Some more clarifications on the assumptions will be needed. After resolving some comments/questions/clarifications, this can be quite a powerful claim.\n\nThey have also included limited empirical results to validate the theoretical findings and included some thoughtful and clever toy examples (more clarifications needed though).\n\n**Weaknesses: (includes questions)**\n\nW1. The results are proven based on certain worldviews/assumptions on HOW the data distribution changes once a recourse is provided, what part of the population receives and acts upon the recourse, consequently leading to the data distribution change. It would be nice to highlight these assumptions/worldviews upfront (not just mathematically, but physically what they mean) for better interpretability of the mathematical results and the conclusions derived from them. This is quite important in interpreting the results because this is quite a surprising claim. \n\n-- Q1: For the post-recourse distribution $P(X_p,Y_p)$, is it assumed that every point in the data distribution acts upon the recourse? Would this be a reasonable assumption in practice? What would be its physical meaning? \nDoes it have provision for:\n (i) only few individuals actually applying and receiving recourse and acting upon it? For instance, not all candidates might actually apply/interview/receive feedback. There might be several candidates who stay the same, and only a handful of them implement the recourse. (I am not sure if low $\\alpha$ alone is sufficient to model this sparsity? Low $\\alpha$ would still place a non-zero mass on *all* altered datapoints, but here only a handful of altered datapoints should have non-zero mass?) \n(ii) entirely new candidates can get added to the pool who have not applied before.\n\n-- Q2: The updated data distribution is assumed to be a mixture of $P(X,Y)$ and $P(X_p,Y_p)$ weighted by $\\alpha$ and $1-\\alpha$. Would this be a reasonable assumption in practice and what is its physical meaning? \n-- Q3: What happens when $\\alpha$ tends towards 0 or 1?\n\nW2. Similarly, there are also implicit assumptions being made on how the new model is being trained once the data changes which need to be made explicit. \n-- Q4: Are you assuming that the new model does not consider the data imbalance in the new data distribution? Depending on the $\\alpha$, there might be severe data imbalance in the new data distribution. What if the new model is trained after fixing the data imbalance, e.g., by reweighting?\n\nW3. A clear example will be very helpful. Though Examples 4.3 and 4.8 is provided, they lack a lot of details and have some notational typos (I also checked the details in Appendix E and F, but still have questions):\n\nIt says L is 1 when Y>0. Is L the true label or the predicted label?  \nIs L_hat/h a deterministic function of the features Xc and XE?\nWhat is beta?\nCould it be clarified what is the initial model h (perhaps provide a concrete model as an example, e.g., a linear classifier)? Then, for this model h and a chosen alpha, what is the hm? Then, for this example, could you show how does this statement mathematically play out: \"Then, the examination score becomes less predictive of the patient’s risk.\"\nAfter acting upon the recourse, are you assuming true label L will change if one acts upon Xc but not if one acts on Xe?\n\nW4. There are typos in the proofs in the Appendix, and also some details are missing in several places. For instance, Y and L have been used interchangeably in places. Some writing errors too.\n\nW5. While the mathematical results seem interesting, I am not fully clear on the final conclusions derived. \n\n-- Q4: What happens if the recourses are about the causal feature, the population updates that causal feature, and why wouldn't those recourses get invalidated?\n\nQUESTIONS:\nI have included several questions along with the Weaknesses.\nIt would be great to get clarifications for these questions.\nWith some more clarifications, this can be quite a powerful result.\n\nLIMITATIONS:\nSome discussion is included.\nGiven the nature of this work, I would encourage more elaborate discussion on the physical meaning of their assumptions, and the role that they would play on the conclusions derived.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the performative aspect of algorithmic recourse: when we provide an individual with a “recourse action” this very action may shift the model’s future behavior, invalidating the effectiveness of the recourse when the model is retrained. The authors  formalize the performative validity, a criteria to quantify when recourse remains valid under distributional shifts induced by the recourse itself, i.e. when users take a suggestion from a system and improve themselves, then the model retrained on new data points will not reject the users that would originally be accepted. The framework used are Structural Causal Models and they show that two mechanisms can induce such invalidity: actions that are influenced by effect variables and actions that intervene on effect variables, Finally,  the concepts are used to show that Counterfactual Explanations and Causal Recourse can lead to invalidity. While the Improvement-Focused Causal Recourse can satisfy this property.\nThe Formal Condition for Validity is that the recommended action $A$ is conditionally independent of the true post-recourse label $L^p$ given the post-recourse features $X^p$. This reduces the problem to a statistical independence condition: $A\\sqcup L^{p}|X^{p}$. The theorems provide a clear, graph-based diagnostic for when a recourse method is at risk of invalidating itself.\nSome experiments on SCM with various structural model show that ICR preserves the validity, while it is not the case for CE and CR.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths and weaknesses. \n\nThe paper is well written with a good balance between necessary and exhaustive information (such as discussion of the state of the art, or proofs) between main paper and supplemental. \n- Various examples are presented and discussed to illustrate the concept and properties of interests. \n- the main theorems are clearly stated and well motivated and explained, and we can apply the methodology.\n\n\nWeaknesses:\n- The impact of retraining the model is not discussed on experiments or theoretically. In the paper, it is not clear what is the real impact, and it seems very useful to link the mixture model (1) (and alpha), with the number of rejected recourses or previously accepted observations. Could you clarify? What is the appropriate concept to anticipate the shift / amplitude of the drop in performance of the recourses.\n- Assumption 4.6 seems to be strong : is it possible de relax it? Do we have equivalence with this assumption in theorem 4.7. \nHow can we use it in practice?\n- I wonder about the significance or interests about this work: although I found the scientific question interesting, I am not fully convinced by the initial examples given in HR, and I feel thai it is a good illustration but not a real problem in practice. Do you have any alternatives?\n\nQUESTIONS:\n- can you give more practical / real life examples \n- can we show mathematically that CE and CRa are not valid ? \n- Do we need to know the causal graphs in advance to apply the results proposed in the paper? This seems very difficult in real world. I am also wandering what would be the difficulties if we don't assume assumption 4.4 and 4.6? }\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}