Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper establishes a causal framing to characterize the effects of distribution shift on a trained model's fairness. A statistical test building from various factors of conditional independence, enabled through the assumed causal structure, is used to understand what factors of a dataset are unduly affected by the distribution shift. This knowledge illuminates the possible reasons a model's fairness qualities are not maintained after transfer which in turn motivate possible mitigation strategies. The proposed statistical test is then used to analyze model performance on two separate real-world datasets.

# UPDATE: After discussion period

I'm grateful to the authors for their patience to address my misunderstandings and willingness to improve the writing of the paper. After re-assessing the paper, I am happy to improve my score and recommend acceptance of this paper.
strengths_and_weaknesses: It's clear that a lot of work has been done in pursuing the research underlying this paper. There are thorough empirical investigations as well as substantial scholarship to properly anchor the work. The statistical test, built around predicting the binary indicator source of the dataset, appears to be a helpful tool. It unfortunately isn't given enough space to be sufficiently introduced or analyzed. I will try, to the best of my ability, to outline where I feel that the paper is strong and provide my view on its weaknesses with some recommendations for improvement along the lines recommended in this form ("originality, quality, clarity, and significance"). All recommended literature beyond those referenced in this paper will be included at the bottom of this section.

**To be up front, it is unclear what the intended contributions of this work are meant to be.** The causal framing of datasets and the effects of environment (e.g. source and target dataset) has been established previously (see Adragna, et al '20; Madras, et al 2019; Creager, et al '20, '21, _some of these citations were missing surprisingly_). The statistical test has some merit as a potential contribution but is not presented convincingly as such, nor is it used to any extensive effect in this paper. Finally, the mitigation approaches outlined in the Appendix are all established in the literature and are discussed, _even in this paper_, to not be realistic or particularly helpful. In this regard, it is unclear how to assess the paper's originality. Perhaps the extension of this lines of previous inquiry to real-world medical datasets is the major contribution? This is a noteworthy effort that is to be commended but the tenor of the findings only serve to reinforce well understood complexities with medical data (e.g. the type of distribution shift is compound and possibly too complex to adequately or feasibly mitigate).

Another issue along the lines of originality here is that the ML-pipeline remedies (lines 293-315) to address fairness challenges (among other issues raised by distribution shift) are almost entirely pulled from Chen, et al (2021) without proper attribution. At first read this section seemed like an odd departure from the narrative and form of the preceding sections of the paper. After re-reading the paper a second time, it comes across as a mea culpa trying to mask technical deficiencies in the work by redirecting the focus away from the results.

**The framing of "fairness transfer" is odd**. While the fairness of trained models is an important characteristic to monitor and assess, it strikes me as an odd framing to place model performance in as focused on "problems" with the data. The issue at hand are artifacts of the model training and its generalization capacity. Issues of fairness, while being spurred by characteristics of the data (especially under transfer), largely arise through the model overfitting to "degenerate" or local modes in the global data distribution. Placing this entire blame on the data doesn't seem appropriate. Perhaps, it might be better to frame this challenge as ways the datasets are insufficient to ensure equitable model performance across distribution shift? I acknowledge that the term "fairness transfer" is present in the literature, of which we can't control. But, there is some improvement possible to the way this literature frames and addresses these problems. I really enjoyed the attempts this paper makes to 1) identify the challenges to maintaining model fairness that arise due to distribution shift (re-affirming Nestor, et al 2019), 2) evaluate the effects of distribution shift on cross-sectional model performance, 3) propose possible mitigation strategies and 4) recommend possible mediation strategies to the ML pipeline when facing these types of challenges. I just wish that the narrative and structure used in this paper made the actual challenges more apparent without resorting to shorthand that isn't wholly indicative.

**The "diagnosis" of the distribution shifts in the real-world datasets is underwhelming.** At worst, the analysis described in the paper can be viewed as elementary data exploration. It was surprising that after introducing Algorithm 1, that it wouldn't be used as readily for testing the independence of each variable in the dataset. Simple distributional analyses of the data and its subgroups seem to dominate the assertion (and pretty well understood fact a priori) that the type of shift is compound. The mention of the statistical tests is a footnote and seems to be largely kept quite vague. I suppose that I was hoping for a bit more tangible evidence whether through tables or charts about how the statistical tests were used and how they were assessed to be meaningful or not (especially since they rely on prediction models internally, how accurate or useful were those by the way?!). Parenthetically stating that something was significant or not based on the p-value doesn't necessarily provide enough information about the validity of the underlying statistical test. It largely feels that the analysis of distribution shifts could be does by estimating the simple disease prevalence among subgroups or otherwise measuring the homogeneity of the dataset using something like optimal transport dataset distance (OTDD, Alvarez-Melis and Fusi, 2021). 

**The procedure and execution of the proposed statistical test is unclear.** 
As far as I understand the statistical test proposed in Algorithm 1 is built around learning predictive models of the data source, conditioned on the causal parents of the dataset. The procedures in setting up and evaluating these models is not made clear, nor are the underlying effects of possibly incorrect models on the overall ability of the statistical test to adequately reject the null hypothesis. As the core technical contribution (as far as I can tell), it is surprising that the mechanisms and analysis of this algorithm are not featured in any substantive manner. Additionally, it's unclear whether Algorithm 1 is used on each variable in the BCN individually or all at once. 

Beyond this, it is difficult to adequately gauge the validity or helpfulness of the proposed statistical tool with the datasets used in the paper. Here, the design of synthetic examples where a subset of the variables are affected by the shift (rather than *all* of them) would be very insightful. It would be helpful to see use cases where the proposed test does work well and as designed/intended. Right now, going from idea to implementation in complex settings, the presumed impact of the approach is significantly lacking.

Additional points where the clarity of the paper could be improved:
- The use of notation in Algorithm 1 is terse and not introduced anywhere. It's difficult to follow and understand. This is made harder by compounding the use of 'S' to be both the sample from the datasets and the source/target indicator.
- It would be instructive to outline the BCN basic structure or concepts using a figure early on. This could be particularly useful in the design and execution of synthetic examples where the causal graph is modified accordingly.
- It's unclear what the training/testing procedure is to assess model fairness before and after the dataset shift. Fairness properties of a model depend significantly on the training mechanism of the model. The absence of any discussion along these lines is concerning.
- In the EHR experiments, only 5 comorbidities were accounted for in the target population. How were these chosen? Why only those 5?


**Re: EHR experiments** It is stated that many hospitals do not have specialized ICU departments. This assertion is not supported by citation and seems speculative and misleading. This is made more apparent when the experiment utilizes MIMIC data that is derived from a hospital that *does* have specialized ICU departments. The poor model performance between the two subsets of data is not at all surprising since the subsets of the data cover very disparate patient populations. The design of these experiments is not very intellectually honest without proper description or justification. It can be understood if the subpopulations were designed to be an apparent and extreme example of distribution shift but without stating this, it appears that the authors are trying to construct a dramatic example to garner more significance. The conclusions made in lines 216-221 are exceedingly obvious. Of course the type of ICU a patient is admitted to affects the underlying performance of a model trained on a separate population. There are basic factors due to underlying health conditions, whether the procedure was elective or urgent, hosptial procedures, etc. that confound the presentation of data for use in predictive settings. By transferring between subpopulations designed to be as far apart as possible along some of these confounding dimensions (especially when these things are well understood within MIMIC--CSICU vs. MICU for example), there are some significant concerns about the intentions of the authors in using these experiments. Again, if this was clearly set out and justified from the outset it would be more more acceptable (yet, still inappropriate) that these empirical claims were made in this paper.

#### _**Recommended Literature**_

Madras, D., Creager, E., Pitassi, T., & Zemel, R. (2019, January). Fairness through causal awareness: Learning causal latent-variable models for biased data. In Proceedings of the conference on fairness, accountability, and transparency (pp. 349-358).

Creager, E., Madras, D., Pitassi, T., & Zemel, R. (2020, November). Causal modeling for fairness in dynamical systems. In International Conference on Machine Learning (pp. 2185-2195). PMLR.

Alvarez-Melis, D., & Fusi, N. (2020). Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33, 21428-21439.



questions: Unfortunately, I am not sure whether the authors will be able to sufficiently address the major concerns that I raised in my review above that would allow me to raise my score without re-submission. There are core issues to the framing and narrative of the paper necessitating a full revision as well as some additional experimental analysis that would need to be re-reviewed before acceptance for publication. This process is not supported by the reviewing timeline for NeurIPS.

I do have some questions though that may help me understand whether I should reconsider my assessment and if I misunderstood the paper in any way. 

In the authors view, what are the major intended contributions of this paper (see first major points in the section above)?

How does this paper differentiate from Creager, et al (2020 ICML)? Is it only in the use of real-world medical datasets and the corresponding simple prediction tasks?

How sensitive is Algorithm 1 to the quality/accuracy of the models used internal to the statistical test?

What efforts were taken in the construction of the experimental subpopulations to create a fair comparison of model performance?
limitations: The major limitations of this paper have been listed by the authors in the final paragraphs of the paper. They have clearly outlined major challenges from a data perspective that limit the feasibility of mitigation strategies as well as overall performance assurances with regards to transfer when facing distribution shift. 

However, it is unclear what kinds of assumptions and limitations that exist for the development and execution of the statistical tests in Alg. 1. I can guess that the data needs to admit some form of conditional independence estimation (which might be a fairly strong assumption on its own) as well as be able to adequately estimate the dataset source (predicting $P(S|pa(U))$).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an approach to detect distribution shifts between two environments or settings that could lead to the failure of transfer of fairness properties across the settings. The approach adopts the joint causal framework to identify the direct effects of the environment variable on other variables at hand. The paper also suggests some mitigation strategies related to post-deployment, data collection, and outcome definition in case distribution shifts make it challenging to transfer fairness properties. Experiments on two real-world health tasks: 1) predicting skin conditions in dermatology and 2) clinical risk prediction from EHR highlight the merits of the proposed approach. 
strengths_and_weaknesses: Strengths:
1. The paper is generally well written with a clear description of the proposed approach and the empirical details.
2. The work touches on an important aspect of the transfer of fairness notions, especially in healthcare where there is not enough clarity on which of the fairness notions are more suitable. The topic is of importance to the community and the said approach is one such mitigation/investigative tool. There is potential for further advancement in this area that is touched upon here and it is thus significant. 

Weakness:
1. One potential weakness as illustrated is the dependence on sample size which is known to affect conditional independence testing. 
2. There are some parallel approaches that touch upon identifying how shifts relate to model performance in the healthcare setting; it would be useful to discuss the current approach with respect to such methods (more on the approaches below).
questions: 1. [1] also discuss detecting shifts across environments using a causal discovery approach (FCI) that also adopts conditional independence tests. This is another line of work closely related to this paper and would be helpful to situate the current approach with regards to this work/discuss the differences.

2. In the dermatology task, what is the dimension of X considered? How would the dimensionality affect the results?

3. In figure 2d, does the X-axis represent the plots for the source and target? It's a little unclear as to what the number of images signifies here.

4. Reference missing on line 174.

5. Check if the references for path-specific effects (line 329) are appropriate. [2] is related to this.



References
[1] Singh, Harvineet, Vishwali Mhasawade, and Rumi Chunara. "Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database." PLOS Digital Health 1.4 (2022): e0000023.
[2] Mhasawade, Vishwali, and Rumi Chunara. "Causal multi-level fairness." Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021.

limitations: The authors have discussed some limitations of the work.

Some other concern is the lack of complete data, especially when considering multiple environments/settings in healthcare such as social determinants of health. This is pertinent since social determinants are known to shift across environments and would thus lead to failure of fairness transfer. 

One more concern is when considering multiple sources or multiple target environments. As the number of environments increases the chances of shift would also increase.  While the robust training could be done to incorporate multiple source environments,  assessing the performance across multiple target environments is crucial to understand under what kind of settings is the model fair.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors deal with the problem of diagnosing fairness transfer under distribution shifts. For this they use a causal framework and statistical hypothesis testing to identify the type of distribution shift. Their work relies on having the labels of sample of out-of-distribution data to correctly identify the type of shift. 
strengths_and_weaknesses: The paper deals with a very relevant topic.

The formalization of the problem is quite clear, simple, and suits very well the task at hand.  The paper is well written and easy to understand. Even if it has a long appendix, the main body is self-contained and does not rely on the appendix. 

The paper structure is not the typical one, with a related work section on page 7 and a discussion section that spans for over a page.
In the discussion section, there is a set of remedies. I am unsure if a set of remedies can be included in a scientific paper. 


Line 101 -- do Rabanser [53] use bootstrapping techniques to do dimensionality reduction?


questions: A synthetic experiment can be switching the labels of the test set. In this way one can have a pure concept shift, and if done only with protected attribute subgroups it can affect fairness. Would the method propose on this paper diagnose correctly this type of shift?

A strategy used in order to post-process the prediction to enforce fairness metric is Alabdulmohsin[4], the authors observe that fairness mitigating on source data leads to no improvement or worsening on test data. I wonder if simpler methods such as [https://www.nature.com/articles/s42256-021-00396-x] have the same results. 
limitations: A common underpinning of much work in ML fairness is that the metric selection is not justified from a socio-technichal perspective. In this work the authors acknowledge that the chosen metrics do not consider some social factors. 

The authors also acknowledge that the solution proposed depends on the number/quality of samples and that it should be perceived as to diagnose fairness transfer failures.
review_title: null

Meta-review: metareview: This is a compelling work characterizing some forms of model (non-)robustness to drift through a causal lens, with a focus on performance metrics including group-level fairness.  The methodological novelty to the work is a method for discovering structure for that drift, then using that structure to (i) estimate impact on metrics and (ii) mitigate those impacts.  That method requires a domain expert to provide a rough causal graph for the application at hand, which is a light negative; yet, the work then presents two in-depth studies in the healthcare space to argue that this is a surmountable requirement, at least in some settings.  This is a tough space to operate in, and I appreciated this very deep dive into two "real" cases -- as did the reviewers.
confidence: Certain
Decision: Accept