Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, the authors first show that under some assumptions (smoothness and convexity on the), enforcing individual fairness (IF) on the source distribution can potentially improve the performance of ML models on the target distribution, and then propose a training mechanism to add an IF regularizer into the training process to achieve individual fairness in the target distribution. The authors also show that domain adaptation algorithms that align the feature distributions in the source and target domain can be used to improve IF. 
strengths_and_weaknesses: The idea that enforcing individual fairness can help with distribution shift tasks is a natural idea and using regularization is also an intuitive way of achieving such an objective. The paper also provides a bridge between individual fairness and domain adaption literature.


The major weakness of this paper is that it's not clear why the main results are insightful, in particular, theorem 2.4 and theorem 2.7. It seems to simply say that the target loss can be bounded as a source loss plus the regularizer's penalties, which seems natural, perhaps even obvious. I wish the bound could be more concrete and interpretable. One way of fixing this problem is to add a corollary for theorem 2.4 for specific regularizers, which will help convince the reader that this bound is non-trivial for classes of regularizers they may use.

Another problem of this paper is the meaning and the interpretation of their bounds. For example, in theorem 2I would like to see $n_s$ and $n_t$ exposed in the bound in Theorem 2.4 because those variables describe the size of the input, thus the bound would be more insightful if the authors could explain why the bound is proportional to $n_s$ and inversely proportional to $n_t$. 

questions: Line 81, what does "the model class is mis-specified" mean? What is the function space $\mathcal{F}$ here? 
More to the point, if "mis-specification" means that you can only train a simple model on the source data, say linear, but the target data is quadratic, then I disagree that this is the difficulty with domain adaptation, even if you can train an optimal quadratic model on the source, the distribution shift will likely make it to be a much worse model on the data, even though this model is in the target hypothesis class. 




limitations: The example of wedding dress provided in section 2 feels uncomfortable, it almost seems to associate the development stage of a country with its brides wearing white dresses for weddings. You can avoid this by simply talking about a specific image dataset that contains photos of European-style wedding gowns, and note that it lacks examples of wedding dresses from a specific and contrasting culture.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper shows that satisfying individual fairness (IF) and representation alignment methods can complement each other. Enforcing IF is shown to mitigate algorithmic biases caused by covariate shift as long as the regression function satisfies IF. Conversely, IF can be enforced by aligning the distributions of the features under a factor model. Experiments verify the theoretical results.
strengths_and_weaknesses: Strengths
* Shows interesting connections between IF and domain adaptation methods.
* Provides theoretical evidence for the above connections.
* Empirical results on various datasets show that IF can be used for domain adaptation and vice versa.

Weaknesses
* Overall the compatibility between IF and domain adaptation seems to only hold in restrictive settings. IF is similar to domain adaptation only for covariate shifts, but not necessarily for other data shifts (label and concept shifts). Fundamentally, IF means similar samples get similar predictions, so it obviously cannot address all data shifts in general. Also domain adaptation is only useful to IF for specific factor model structures. I think the paper would be more interesting if IF techniques are used to improve domain adaptation technique (stated as future work), but the current contribution focuses on their similarities only. It could be that using IF along with domain adaptation does not help domain adaptation at all and vice versa, but I do not see any discussion about whether the two complement, subsume, or possibly have a negative effect on each other. 
* Several assumptions are made to make the theory work without much justification, and it is not clear how realistic they are. There should be more convincing explanations and possibly empirical evidence.
  * Assumption 2.1: this seems to assume that the penalty of the regression function $f_0$ is always small ($\leq \delta$). Wouldn't this result in trivial regression and penalty functions?
  * Assumption 2.2: why should we believe that R is strongly convex? 
  * Assumption 2.3: why is it reasonable that L is both strongly convex and strongly smooth?
  * Assumption 2.6: why is it reasonable to assume that R satisfies strong convexity? 
* In the experiments, the assumptions are not verified on the real datasets. There are only end results for IF and domain adaptation.

======

The author response addresses my concerns, so I am increasing my score.

questions: * Does IF improve domain adaptation when used together (and vice versa)?
* How realistic are the assumptions, and do they actually hold on the benchmark datasets?
limitations: * The authors say the limitations are stated in the paper, but it is not clear where exactly.
* There does not seem to be negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work mainly focuses on individual fairness problem under covariate shift. The authors find that individual fairness and domain adaptation methods can benefit from each other as below.

1. The regularizers for enforcing individual fairness can help models adapt to new domains. The authors consider the transduction and inductive settings. Under the standard smoothness and convexity assumptions on the regularizer and the loss function, the authors give a bound on the target-domain risk of models that are regularized by individual fairness.
2. On a synthetic setting where the source and target domain have the opposite sensitive attribute, the authors show that a domain invariant transformation appeals to individual fairness. 

The authors also validate their theory on two textual datasets.
strengths_and_weaknesses: **Strengths**:  
In general, the presented conclusions are intriguing and theoretically grounded. I checked the proof sketch but didn’t touch the proof details. The paper is quite clearly written. I didn’t find any problem in understanding the assumptions and the theories. The paper flow is lucid. The scope of the studied problem may have significant impacts on both the individual fairness and domain adaptation fields.

**Weaknesses**:  
I would suggest the authors to significantly improve the structure of this paper. There should be a preliminary section for better understanding the problem settings as well as some key concepts. In particular, the settings of domain adaptation need to be further explained. I would also suggest the authors to complement a separate section to introduce prior works on algorithmic fairness under domain shift.

A large body of recent works on algorithmic fairness under distribution shift is missing, including:

> [1] H. Singh, R. Singh, V. Mhasawade, and R. Chunara, “Fairness violations and
mitigation under covariate shift,” in Proceedings of the 2021 ACM Conference
on Fairness, Accountability, and Transparency, 2021, pp. 3–13.  
> [2] J. Schrouff, N. Harris, O. Koyejo, I. Alabdulmohsin, E. Schnider, K. OpsahlOng, A. Brown, S. Roy, D. Mincu, C. Chen et al., “Maintaining fairness across distribution shift: do we have viable solutions for real-world applications?” arXiv preprint arXiv:2202.01034, 2022.  
> [3] A. Rezaei, A. Liu, O. Memarrast, and B.D. Ziebart, Robust Fairness under Covariate Shift. AAAI, 2021.  
> [4] Y. Chen, R.P. Raab, J. Wang, and Y. Liu, “Fairness Transferability Subject to Bounded Distribution Shift.” arXiv preprint arXiv:2206.00129, 2022.  
questions: 1. Line 89: the ML model does not learn this function because the function is not in the model class. Why is the function not in the model class? If I am applying a deep neural networks for classification, it is likely the class of neural networks includes an approximate function that can recognize the non-western brides.  
2. Line 94: the accuracy of the predictions on the unlabeled samples from the target domain in the training data. This is confusing. The unlabeled samples in the target domain is not used for training, right? Do you mean test data?  
3. What is the implication of the transformation $T$ in Equation (2.9)?  
4. The bound in Theorem 2.9 is independent of $epsilon$. That said, the bounds still hold for very large $epsilon$. Isn’t it vacuous?  
5. What is the individual fairness metric in Table 1? Is it evaluated in the source domain or the target domain?  
6. How do you interpret the condition $\Phi b = 0$ in Theorem 3.1? Is it realistic?  

------
After rebuttal:
The authors have addressed my comments. I would increase my score from 6 to 7 accordingly.
limitations: I did not foresee any negative societal impact in this work.
review_title: null

Meta-review: metareview: The initial reviews were divergent. During the rebuttal and discussion phase, however, many of the raised concerns are addressed propertly, leading slightly towards accept. While some of the issues are not checked yet for whether to address them, I believe the authors' response answer them adequately. Hence I recommend the acceptance of this paper. 
confidence: Less certain
Decision: Accept