Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed an approach to improve the fairness of VFL models without the requirement of storing the fairness-sensitive features in a central server. To improve the fairness of VFL models, an additional adversarial loss is designed to remove the bias of unified feature representation. The experiments result show the effectiveness of the proposed fairness algorithms for VFL.
strengths_and_weaknesses: Strengths:
1.) The paper is well-written and easy to understand the concepts and algorithms.
2). The research question is in-time and the fariness related research in VFL setting is less explored.
3). The experiment's result seems promising.

Weaknesses:
1). It would be better if the experiments can be conducted in a different types of datasets, e.g. image dataset celebA. 

[1] Batch Label Inference and Replacement Attacks in Black-Boxed Vertical Federated Learning
[2] Deep Leakage from Gradients
 
questions: 1). The authors claim that their approach can protect the exchange of information between the local model, aggregation models and task-specific models. The protection is based on removing the useful information from embedding. However, much research shows that the exchange of information may disclose the information on the original data, e.g. [1,2]. I am wonderring if the proposed training apporach can defende this attacks.
limitations: The variety of the dataset could be improved.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel vertical federated learning framework (FairVFL) to achieve fair VFL models while keeping information private during communication. In order to encode a fair unified representation for the task platform, the prosed method employs adversarial learning at the fairness-insensitive platform side to prevent the fairness-sensitive bias. Moreover, instead of directly sending the unified representation to the fairness-insensitive platforms, the server sends a protected representation encoded by a mapper trained by a novel contrastive learning method.
strengths_and_weaknesses: Strengths:
- The paper is well-written and organized. The method is clearly explained and well-motivated.
- The results are very convincing. The proposed method can have a slightly higher performance on fairness with a reasonable amount of drop in main task performance, compared to the methods under centralized training. Also, plenty of experiments on different models show the method is model-agnostic.
- The ablation study on adversarial learning and contrastive adversarial learning helps shed light on the power of the two methods in the framework.

Weaknesses:
- Multiple times of communications might be a problem in federated learning.
- Also, the local adversarial training might also slow down the overall training and increase the computational burden for both servers and platforms.
- Two hyperparameters, $\lambda$ and $\gamma$, seem to be very important. It might be hard for the server to choose the correct values for a new task.

questions: - How consistent are $\lambda$ and $\gamma$ cross datasets?
limitations: Yes, the authors mention the limitations in the appendix. As the authors mention, the method requires more rounds of communications than the baseline VFL. Meanwhile, it also requires more computations than the baseline.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method to learn fair representations in the VFL setting. By fairness, the authors mean that the representations generated by FairVFL should not reflect sensitive attributes, such as age/gender. The authors achieve this goal via a contrastive adversarial learning on the sensitive attributes. The idea of this adversarial training is that, the discriminator aims to learn sensitive attributes, while the feature learner aims to fool the discriminator. Moreover, in order to conceal private information, the adversarial learning also aims to conceal all other information except for the sensitive attribute. Experiments on two real-world datasets show the effectiveness of FairVFL. 
strengths_and_weaknesses: **I am not very familiar with the fairness issue in machine learning. Please correct if I make misunderstandings regarding fair ML.**

Strengths: 
1. **Well-motivated problem.** Trustworthy machine learning is a good topic to explore. The authors combine trustworthy machine learning with VFL, which is a good attempt. 
2. Evaluations that verify the privacy protection (Fig. 3b) is a plus. It evaluates how well the proposed method protects attribute privacy, which is a focus in FL. 
3. **Presentation is OK**. Despite the very complex gradient flow (in my opinion), the authors manage to explain the overall workflow to me. 

Weaknesses: 
1. **Lacking a baseline**. The authors take plain VFL as the only baseline in the feature-partition scenario. Although there are other fair ML methods compared (e.g. FairGO, FairSM), they cannot be applied in the VFL setting (as far as I can infer from the paper). Therefore, an intuitive question to ask is that, are there simpler VFL+fairness baselines that can be applied to the problem, such as slightly modifying FairSM, FairGO? If there is such a baseline, the improvements made by the authors' designs would be clearer. 

2. **Definition of fairness**. The authors state in Sec. 3.1, line 114 that, "the counterfactual fairness requires that the model prediction for a sample should always hold if we only change fairness-sensitive attributes". However, in the evaluation, the authors use the metric "Gender F1", namely how well an attacker can guess the sensitive attribute, to measure fairness. I am a little confused about the metric. Specifically, as I can tell from Section 3.2, the unified representation $\mathbf{s}$ is computed from $\mathbf{s}_i^l$, which are all computed from fairness-insensitive features. In other words, sensitive attributes are not involved in the forward computation for the task $t$. It seems that in this sense, we have no chance to verify the 'counterfactual fairness', as we cannot even change the fairness-sensitive attributes. Thus, how the metrics (gender F1) are related to the notion of 'counterfactual fairness' should be better elaborated. Maybe the Gender F1 is more related to notions like conditional independence as far as I know. 

3. **Should discuss limitations and tradeoffs on efficiency and stability**. The overall workflow of FairVFL is rather complex (even though the authors manage to explain it to me). As far as I am concerned, there are many points in the workflow of FairVFL that contain computation dependencies and may lead to stragglers (e.g. the iterative adversarial learning of $A$, $D$), which may significantly compromise the efficiency. Thus, I would like to see an efficiency comparison between naive VFL and FairVFL to see the tradeoffs between fairness and efficiency. Also, adversarial training tend to be very unstable. The authors should discuss how to mitigate the instability and generate consistent results. 

questions: Q1-3: See Weaknesses 1 2 3. 

Q4: In Figure 4(b), it seems that in some cases the fairness-F1 scores can go below random. For example, in Figure 4(b) left, gender-F1 (binary classification) can be <0.5. This seems counter-intuitive, as the best fairness should be achieved at gender-F1=0.5.

I thus have a question, when gender-F1<0.5 (e.g. 0.45), can the attacker simply flip the prediction and get gender-F1>0.5 (e.g. 0.55)? What does it imply on fairness? 

Q5: The authors state that "Intuitively, highly relevant data samples are likely to share the same fairness-sensitive feature" in line 178. I wonder after the model is trained well, and $\mathbf{s}$ does not contain fairness-related information, does this assumption still hold? If not, how will it influence the model training? 
limitations: I do not see any ethical concerns.
review_title: null

Meta-review: metareview: This paper presents a fair vertical federated learning framework (FairVFL), by learning a set of unified fair representations of data/features distributed across decentralized platforms, i.e., these representations do not reflect sensitive attributes such as age/gender.  This is accomplished by having platforms learn local data representations from fairness-insensitive features, which are then aggregated at a central server; this is then sent (after a contrastive adversarial learning method removes private information) to platforms with fairness-sensitive features to remove bias by using adversarial learning. 

All reviewers found the method sound and the experimental results convincing -- two real-world datasets were used to show that the proposed method can effectively improve model fairness while preserving user privacy.

Some concerns were raised over the communication and computational overhead of the proposed method, but none of the reviewers considered it serious enough as to merit rejection. 

One note of caution on the Accept recommendation: this is a paper that did not receive a review with confidence level above 3.  Usually there is at least one review at the level of 4. 
confidence: Less certain
Decision: Accept