FairVFL: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning
Authors: Tao Qi
Fangzhao Wu
Chuhan Wu
Lingjuan Lyu
Tong Xu
Hao Liao
Zhongliang Yang
Yongfeng Huang
Xing Xie
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vertical Federated Learning
Fair Representation Learning
Adversarial Learning
Contrastive Learning

Abstract: Vertical federated learning (VFL) is a privacy-preserving machine learning paradigm that can learn models from features distributed on different platforms in a privacy-preserving way. Since in real-world applications the data may contain bias on fairness-sensitive features (e.g., gender), VFL models may inherit bias from training data and become unfair for some user groups. However, existing fair machine learning methods usually rely on the centralized storage of fairness-sensitive features to achieve model fairness, which are usually inapplicable in federated scenarios. In this paper, we propose a fair vertical federated learning framework (FairVFL), which can improve the fairness of VFL models. The core idea of FairVFL is to learn unified and fair representations of samples based on the decentralized feature fields in a privacy-preserving way. Specifically, each platform with fairness-insensitive features first learns local data representations from local features. Then, these local representations are uploaded to a server and aggregated into a unified representation for the target task. In order to learn a fair unified representation, we send it to each platform storing fairness-sensitive features and apply adversarial learning to remove bias from the unified representation inherited from the biased data. Moreover, for protecting user privacy, we further propose a contrastive adversarial learning method to remove private information from the unified representation in server before sending it to the platforms keeping fairness-sensitive features. Experiments on three real-world datasets validate that our method can effectively improve model fairness with user privacy well-protected.

Fai: A Fair Vertical Federated Learning Framework with Contrastive Adversarial Learning
Tao Qi Tsinghua University Beijing, China
Fangzhao Wu' Microsoft Research Asia Beijing China
Chuhan Wu
Tsinghua University Beijing, China wuchuhan com
taoqi qt@gmail.com
wufangzhaoQgmiil.cmm
Lingjuan Lyu
Tong Xu USTC
Hao Liao
Sony AI Tokyo, Japan Lingjuan. Lalaa..vvooon..ooo com
Shenzhen University Shenzhen, China haoliao0szu. edu. cn
Hefei, China tongxulustc. edu. cn
Zhongliang Yang
Yongfeng Huang Tsinghua University
Xing Xie
Tsinghua University
Microsoft Research Asia Beijing China
Beijing, China
& Zhongguancun Laboratory yfhuang@tsinghon edu,
cn
xingx@microsoot..cm
Abstract
Vertical federated learning (VFL) is privacy-preeerving machine learning paradigm that can learn models from features distributed on different platforms privacy-preserving way. Since in real-world applications the data may contain bias on faimnes--enssttve features (c.g., gender), VFL models may inherit bias from training data and become unfair for some user groups. However, existing fair machine learning methods usually rely on the centralized storage of fairness- sensitive features achieve model fairness, which are usually inapplicable federated scenarios. In this paper, we propose a fair vertical federated learning framework (FairVFL), which can improve the fairness of VFL models. The core idea of FairVFL is to learn unified and fair representations of samples based on the decentralized feature fields in privacy-preserve with fairness-insensitivy features first learns local data representations from local features. Then, these local representations are uploaded to server and aggregated into unified representation for the target task. In order to learn fair unified repre- sentation, we send to each platform storing Vairness-sensitin features and apply adversarial learning remove bias from the unified representation inherited from the biased data. Moreover, for protecting user privacy, we further propose a con- trastive adversarial learning method to remove private information from the unified representation in server before sending it to the platforms keeping fairness-sensitive sensitive features. Experiments three real-world datasets validate that method can effectively improve model fairness with user privacy well-protected.
way. Specifically, each platform
INTRODUCTION
In recent years, the explosion of data volume has enhanced the performance of machine learning (ML) models on many tasks, e.g., personalized recommendation and search 50 2 many weal-world
TTee corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

scenarios, feature fields of the same sample may be decentralized on different platforms 48 29 For instance, video watching behaviors and search engine behaviors of the same user usually spread across different platforms [42] To learn models from various input features. existing machine learning methods usually rely on centralized storage of different feature fields 31 [1] However, the data on
different platforms is usually privacy-sensitive, and cannot be centralized due to privacy concerns and risks 47 36) Thus, vertical federated learning (VFL), which can utilize decentralized features for unified model learning in -preeeeirgn way, has become more and more important [53 g
Existing VFL methods usually communicate intermediate results instead of raw data across platforms to enable models utilizzng decentralized feature fields [53] D]. For example, Yang et al. [ proposed a vertical federated framework for logistic regression. They first calculated local results based on local feature fields, and then shared enerypted local results with server to make predictions and learn parameters. Che et al. 7 proposed vertical federated multi-view learning framework to integrate decentralized medical data. They locally encoded medical feature fields kept by multiple hospitals into representation vectors and uploaded them to the server for diagnosis prediction. However, since real-world data usually encodes bias on fairness-sensitive features (e.g.. genders and ages)
277 [5]]
VFL models may inherit bias from data and become unfair to some user groups 45
Due to the importance of fairness in machine learning, fair ML methods have attracted increasing attention recent years 45
51 For example, Zafar et al. [5]] proposed a fairness-awar aware optimization framework to improve the model faimess, which can constrain the difference between prediction distributions for different user groups. Li al. [266 proposed a fair recommendation framework, which quantifies the recommendation unfairness different user groups as a penalty for item re-ranking. general, most of the existing fair MI. methods rely on the centralized storage of fairness-sensitvve features to achieve model fairness. However, in vertical federated learning, feature fields are decentralized on different platforms 13) making it difficult to apply existing fair ML methods to improve the fairness of VFL models. Thus, how to design fair VFL framework that can protect user privacy and meanwhile improve model fairness, is a rarely studied problem.
In this paper. we propose fair vertical federated learning framework (named FairVFL), which can improve the fairness of VFL models in privacy- preserving way. In FairVFL, we partition data into two groups based on their faireess sensitivities, i.c., chirmess-insensiine for the target task and fairness-sensitio model predictions. Along this line, the core of FairVFL is to learn unified and fair representations of samples based on their features distributed across different platforms with user privacy well-protected. Specifically, platforms with -ineesssiisnsstiive features first learn local data representations, which are then uploaded to servers to build unified representations. Since unified representations encode rich information of raw data, they can be used for the target task without access to the raw data. Besides, unified representations may inherit bias towards sensitive features (e.g., gender stereotype) from implicit feature correlations in biased data and hurt model fairness 45 46). In order to further learn fair unified representations, we apply adversarial learning to unified representations to remove bias encoded in them. Due to the decentralized storage of feature fields, unified representations are sent to platforms with fairness-sensitive features to learn adversarial gradients. Besides, since unified representations may still encode some private information of raw data 40 34 333, directly sharing them among platforms may incur privacy problems. Thus, we further use contrastive adversarial learning to remove private information from the unified representations before sending them to other platforms. Extensive experiments on three real-world datasets show that our FairVFL approach can effectively improve the fairness of VFL models and meanwhile the privacy is well protected.
features that can be used features (e.g., genders) that should be causally irrelevant to
2 RELATED WORK
2.1 Vertical Federated Learning
Vertical federated learning is a privacy-preserving paradigm for training ML models from feature fields that are kept by different platforms that share the same data ID space, and has been widely studied in recent years 43 [4 g. For example, Wu et al. 43 proposed a ad CTR prediction framework with vertical federated learning. They proposed to coordinate local platforms to learn behavior representations from local features and share the behavior representations with the server to model user interest in ads. However, since real-world data usually encode bias on VFL models may inherit bias from data and make
harness-sensittee features (c.g., genders) 45 46

---

unfair decisions for some user groups (e.g,, minority users). To improve model faimess in VFL, Liu al. 28 formulated the model performance gap for different user groups as a faimess regularization and incorporated it into the federated optimization objective. However, their method needs to share the data grouping information with the server, which may also leak user privacy on fairness-sensitive features (e.g., gender). Different from these methods, we propose unified framework that can improve the fairess of VFL models which applies the adversarial learning to learn fair models and employ novel contrastive adversarial learning method to protect user privacy.
2.2 Fair Machine Learning
With the increasing impacts of ML techniques on our society, the fairness of machine learning models has attracted substantial attention 45
Existing fair machine learning methods are usually designed to learn unbiased models with respect to various fairness-sensitive features (e.g., genders). Some of them applied pre-processing techniques
22
26 46
(e.g., resampling) or post-processing techniques [266 3 22 [ (e.g., re-ranking) to improve model fairness. However, these methods usually independently optimized model accuracy and fairness, which may only achieve a sub-optimal To better improve model fairness, many works have been proposed to jointly optimize model accuracy and fairness during model training [55 27 [6]. However, most of the existing fair ML methods rely on the centralized storage of feature fields, making them difficult to be applied in vertical federated learning. Different from these methods, we propose a fair vertical federated learning framework, which applies adversarial learning to improve the fairness of VFL models based on decentralized feature fields.
rrae-ooff between accuracy and faimess 45 46.
3 Methodology
3.1 Problem Definition of Fai
In VFL, different feature fields of the same samples are decentralized on multiple platforms. From a fairness perspective, these data can be partitioned into two groups, i.e., faimess-insensitive features and fairness-sensitvv features. The former can be taken by models as the inputs while the latter are expected to be causally irrelevant to the model predictions. Without loss of generality, we assume features, and we denote the platform keeping the i-th one feature fields are decentralized
that there are types of fairess--sensthee
with type dī as pe P Besides, we assume that on n platforms, and we denote the i-th platform as P. Moreover, we assume that there is task platform pe keeping labels of samples on the target task. Following existing works 43 50 we assume that there is a wouth- server pa for information aggregation.
Next, we present a formal definition of privacy protection and fairness. The privacy constraint requires that the private information in the local data of platform should not be disclosed to another platform. Thus, the raw data of local platform cannot be disclosed and the intermediate results shared with other platforms should be carefully protected. Besides, in this paper, we define model fairness from a casual view [27] The counterfactual fairness requires that the model prediction for sample should always hold we only change the fairness-sensitive features. According to the theoretical analysis in Li et al. (27). eliminating fairness-sensitiv feature information in the data representation that is used for the model prediction can achieve casual fairness. We remark that even though fairness-sensitive features are not taken the inputs, the model may also inherit bias related to sensitive features by The goal of fair vertical federated learning is to achieve a good trade-off among performance, privacy, and fairness, i.e., improving the fairness of VFL models with minimal performance drop in a
mining inherent shortcuts from data. 45 27 4d
way.
3.2 Federated Model Serving
The main framework of FairVFL to utilize decentralized feature fields is to encode them into a unified representation (Fig. D. The ML model in FairVFL is partitioned into three parts, i.e., multiple local models (Mili I, ..... n.,, an aggregation model M and task-specific model M where M, is the i-th local model maintained by the platform Ph. In FairVFL, local models are used to encode
"To simplify the presentation, we use different symbols to denote platforms storing different feature fields. We remark that platform store multiple sensitive feature fields in practice.
3

---

Falmness-Senstivve Feature Platform Pf
Task Platform P
Faireess Sensitive Feature Platform Fi
B4 807
84
Bias Discriminator Df
Taskespecific Model M
Bias Discriminator L11
800
Server
Sampler
A4!
ag
@
01+1
Sampler
Mapper.dk
Mapper A
$
1
047 88
Contrastive
05
Discriminator D
Discriminator
DE
acf
Aggregation Model 1
ac 60%
800
2A,
Mhiss
Li
$
representation
the 1-th local
sj
representation
the f-th local
sn
representation
the w-thh local
Local Model M)
Local Model
Local Model M!
Q
Q Q
Fairness-Insensitee Feature Platform P.
biinnsss-nsenniiivee Feature Platform P)
Platform P
6 task loss
4 discrimination loss
4 adversarial loss
di: contrastive discrimination loss
2 contrastive adversarial loss
unilled representation
contrastive negative sample
@ protected representation ① workflow
Figure 1: Framework of our fair vertical federated learning method (FairVFL.).
local Hairness-insensine features into local representations. We remark that FairVFL is general framework and the architecture of a specific local model can be adjusted based on the input data. For example, the local models for encoding textual search logs can be implemented by transformers (39 or PLMS TO.. When the task platform ps needs to serve a target sample s, _t will first distribute the ID of the sample to local model M encodes inneesssisseesstivve features of the target sample stored in P. and builds a local representation s/. Thus, we can obtain multiple local representations {s/li = 1...., n2). which encode different
platforms {1 = 1, nJ. In the i-th platform Ph the
feature fields the target sample. Following Wu al. [3 these local representations are then uploaded to the server nd for task prediction.
The aggregation model M is maintained by the server p It is used to aggregate information in local representations into a unified representation s. Specifically, different feature fields usually have inherent relatedness, mining which can enhance the model for the target task [
4e! Thus, first
apply multi-head self-attention network 33) to capture relatedness among local representations, where the contextual representation of s is denoted 6, Then we apply attention network contextual representations to model their relative importance for representing raw data and build the unified representation s Since can encode various information decentralized fairness-inen inneesstivis feature fields, we further upload the unified representation instead of raw data to the task platform ps to provide information for the target task. (We apply LDP protect before sending it to p*.)
The task-specific model M is maintained by the task-specific platform P is used to utilize information in the unified representation y is the model prediction. The architecture of the task-specific model M is also adjusted based on the target task (e.g., risk prediction). For instance, for the classification task, network to implement M In this way, decentralized fairness-insensittee feature fields of target sample be utilized FairVFL. for the target task without the disclosure of raw data.
make prediction for the target task: ÿ = (s), where
employ dense
3.3 Fair Representation Learning in FairVFL
As introduced in Sec. 3.2, the task prediction for each sample is computed based on their unified representation s. Thus, the core of FairVFL to achieve counterfactual faimess is to learn fair
The aggregation model can be also implemented by other pooling networks, like dense network.

---

and unified representation s that does not encode fairesss fairness-sensitive sensitive bias. Although the faimesss- sensitive features are not model inputs, the model may still encode bias related them by mining inherent shortcuts to these features from data. Thus, we apply adversarial learning to prevent unified representations from encoding fairness-sensitive bias. Vanilla adversarial learning techniques usually rely on centralized storage of input data and labels of fairnes--sensitivee attributes to learn adversarial gradients for model training. However, in vertical federated scenarios, feature fields are decentralized on different platforms, which brings challenges to the application of existing methods. An intuitive solution sharing the unified representation with platforms that store observed fairness- sensitive features ( 1, ..., m) to learn corresponding adversarial gradients. However, unified representations usually encode some private information, and directly sharing them across different platforms may lead to user privacy leakage. To tackle this challenge, we propose contrastive adversarial learning technique to protect user privacy by modifying unified representations.
C
As shown in Fig. 2 the core idea of the contrastive ad- versarial learning to learn a protected representation that only encodes information sensitive feature. The protected representation can be further shared with the platform keeping the cor- responding sensitive feature to learn adversarial gra- dients without leaking user privacy. Specifically, for each fairness- sensitive feature d;, we first map uni- fied representation into protected representation a;: a; = (s). where A; is a mapper based on MI.P and is the preimage of a, a; is designed only retain information on the fairness-Sensitive feature dh and eliminate other user privacy. To achieve this goal, we propose the contrastive adversarial learning method make the preimage of a; indistinguishable among unified representations with the same feature dř. Given representations of samples in the same training batch ((a),s') 1, 2,.,, F), we first rank these samples based on the relevance between their protected repre- sentations a¹ and a;: rj are the protected and unified representation of the j-th sample, r2 is the relevance score, and E is the batch size Intuitively, highly relevant data samples are likely to share the same chirness-sensitie feature di Thus, from the top E, samples ranked by relevance scores, we can randomly select unified representation s; that is highly likely to be the same with in dž, where E, is 8, is used as the negative sample in contrastive learning to learn contrastive discriminator D to classify preimage of a;:
a2i
84,
certain fairness-
Contrastive Loss
Contrastive
Discriminator
Matching
Matching
batch sampler
Random Selector
s/i
Ranking
Mapper A
unified representation of the current sample
a/, where a: and s
unified representation of the j-th sample in batch
contrastive negative sample
index of the in-bath sample ranked at the j-th position
The detailed workflow of con- trastive adversarial learning.
Figure 2:
(x,(D (xp(D)((a,s)) + exp(D)
C
log
ZEQ
where D-(() is implemented by an MLP network, Ω is the training set and x is data sample in Ω. Optimal discriminator Dę is learned by minimizing CP with fixed representations s, 8, a;
D arg min C.
Finally, the optimal discriminator Dr is used to learn contrastive adversarial loss L and the contrastive on a; with the fixed optimal discriminator DÇ. s and s
adversarial gradients da,
ac;
1
(x.(
5 =
log
(3)
Ω
(a. + Dx
PEN
'The relevance for the current data sample is masked
5

---

The contrastive adversarial gradient aci on is further back-propagated to the mapper A;:
ac;
Oci da, da dA,"'
a. A:(s),
0A;
where negative contrastive adversarial gradient ãλ. A, is used for model updating, and rep- resentations and s; are fixed and not tuned by the contrastive adversarial gradient. In this way, contrastive adversarial gradients can enforce the mapper A, to protect user private information in a.-
act
Since user privacy can be reduced from a;, we further share it with the platform P. keeping label y of the fairness-sensiiiee feature de to learn corresponding adversarial gradients. The platform P. first employs bias discriminator D to predict the label y; of chirness-sensttvee feature de from the protected representation a, and obtain bias discrimination loss L follows:
1 !
Σx, v. logyi OR JEN
V (a))
(5)
where D is implemented by MLP network. Moreover, besides private information, contrastive adversarial learning may remove bias information from a; and hurt fair representation learning. Thus, minimize bias discrimination loss L to optimize both bias discriminator D2 and mapper A₁:
D A = arg min D arg min Y. log (A:(s)) D.A:
D..
where De is the optimal bias discriminator, A, the i-th optimal mapper, and the unified representa- tion is fixed. Finally, for the i-th faimnes- feature di we can obtain the adversarial loss 2 Oc on unified representation s:
and corresponding adversarial gradients
8s
y logy?
Y = D)(((s))
BEN
where the optimal bias discriminator and mapper are fixed when optimizing the unified representation 9CT
The negative adversarial gradient as is further used to reduce fairess--ensitivy bias encoded in the unified representation based on task label y and predicted label ý. The task loss function also adjusted based on the target task. Finally, we can formulate the overall loss L as:
achieve counterfactual faimesss Besides, we can obtain the task loss L
C C=
Σ A.
Σ CCC
(8)
where L' L and L represents the training objective for model performance, faimeess and privacy respectively, X, is the weight of loss C9 and 7/2 is the weight of loss C?.
3.4 Federated Model Training
Next, we will introduce how we train model on decentralized fairness-sensitivee and insensitive feature fields. First, for the task loss L' the task platform pd can directly calculate gradient &ct on task model and gradient distributed to the server p for the further model updating. Second, it is usually difficult to obtain the optimal discriminator in adversarial learning. Following Goodfellow et al. [6.. we optimize the discriminator for single step based on discrimination gradients before executing adversarial learning. Specifically, for the contrastive adversarial learning, the sever first learns contrastive discrimination dC" gradient aD; contrastive discriminator D update it. Then the server utilizes updated D to
aM
ag. Di on unified representation s. グ.M is used to update M and 2s
ag!
learn contrastive adversarial gradient 8A. on mapper A, to update it. Third, for adversarial learning acd
on each fairness-sensitive feature di platform P. first calculates bias discrimination gradient 3
on the bias discriminator D and act da. on protected representation The former is used to optimize
D. The latter is distributed to the server l to calculate gradient on A;: ØA.
Oc: act da; which DA.

---

is used to optimize A. Next, the platform P. calculates the adversarial gradient 02i act
2a a;- 8a.
is further distributed to pa to calculate adversarial gradients dci on the unified representation s based on gradient back propagation:
An. Until now, the server p can further calculate 2 ac
ds
Da; ds
overall gradients 0c ac -i=1 A on the unified representation s. The server further oc ac äs and the gradient ac ax on each 8M ds aM=
aci ös
ds Dc
ös
calculates gradient 8M on the aggregation model:
local representation s/: ac 0c Finally, gradients on local representations are distributed to
as 28 05. corresponding platforms (P 1....... n) to optimize local models {Mili = 1,,, ..., n.). The detailed process of a training round in FairVFL is in Algorithm (Supplementary).
3.5 Discussion on Privacy and Efficiency
Privacy: In FairVFL, each data platform locally keeps its private data and never shares it with the outside, which can protect user privacy to some extent. Besides, for the vertical federated model learning and inference, the intermediate model results, i.e., data representations and model gradients, need to be exchanged across platforms, which may lead to potential privacy leakage. To tackle this challenge, we propose a contrastive adversarial learning method to protect private information in data representations, which can compare data representations with the same attribute and remove the suribute-irrelatee information in them. To further protect private information in gradients, we can apply differential privacy techniques perturb exchanged gradients as standard VFL methods (4
Efficiency: FairVFL. needs multiple times of communications in a single training round, which may arouse concerns on communication efficiency. Fortunately, the extra communication costs are usually small. This is because in the extra communication rounds FairVFL only needs to exchange protected representations and their gradients, and the extra communication cost is (AmEEf), where is the number of Sainness-eensttive platforms, H denotes the dimensionality of protected representations, and E denotes the batch size. Since M, E, and B is usually small in practice, the corresponding cost is usually minor. For the computation efficiency. compared with the standard VFL framework, FairVFL only increases the computation cost of the server and platforms with sensitive attributes. They need to extra compute the gradients on the attribute discrininators and contrastive discriminators. Thus, the computation complexities of FairVFL. and standard VFL are Odd D and O(D*) respectively, where and D denote the parameter sizes of the discriminators and the main task model. In widely used settings, the discriminator is usually implemented by a small FFNN network, whose parameters are usually much less than the main task model. Thus, the computation efficiency of FairVFL is still comparable with standard VFL methods.
4 Experiment
4.1 Datasets and Experimental Settings
We evaluate model performance and fairness on three real-world datasets. The first one is ADULT [24] which is widely used public dataset for fair ML 152
20
35
25
It contains rich feature fields (e.g..
education and social relation) for income prediction. 20,000 randomly selected data samples are used to construct training and validation dataset, and 10,000 randomly selected data samples are used to we treat gender and age as fairness-sensiee features and utilize other feature fields for income prediction. Besides, we randomly partition 12 types of feature fields in ADULT into three platforms to simulate the VFL scenarios [1 The second one NEWS, which is constructed by user logs (i.e., news clicks, search, and web-page browsing logs) on Microsoft News and Bing Search platforms during 6 weeks (June 23th July 20th, 2019) for news recommendation. These three types of data are decentralized on three platforms under VFL settings. We randomly select 100,000 news impressions in the first three weeks to construct the training and validation set and select 100,000 news impressions the last week to construct the test set. Besides, we also treat gender and age as faimess-sensitive attributes in NEWS. The third one is CelebA (0), which is a public face attributes dataset. To simulate the VFL setting, we use the raw images and a part of the raw attributes (i.c., the "Straight Hair" attribute and "Wavy Hair") as the input features. The raw images are stored in a data platform and other input attributes are stored in another platform.
construct test dataset. Following existing works 45 do
FairVFL is feature partition agnostic and does not rely on specific partition strategy.
7

---

Table I: Model performance and faimess on ADULT and NEWS.
Training
ADULT
NEWS
Strateyy
Model
Accurncy
Incomn Prodiction
PT
Gindder P1
Model Faimess
Age FT
Model
News Keormneehiion
ATC
MXXX TO
Gender F1
Model Fairness
Age Fl
MLP
82.15:00:6 18.1210.66
78.27=1.60
47.17
NAML
61.01100:13
30.80:00133
70.05:0.21
20.0113.08
CenTraii
TahNet
82.23=+1.02
79.07=11.79
4..12:11.24
LSTUR
64.68-00.33 30.97.0.200
70.45=0.37
20.00-10.39
Autolas
12.3
78.4911.50
79.22.0.84
48.99-1.225
NRMS
64.24.20.88
30.78.0.111
20.25..0.4
21.0720.81
MLP
17.9711.34
18.491122 25
50.92=6115
35092
NAML
60.73-00.15
10.10:00116
54.0895.99
15,931145
FairGo
TahNet
74.40
75.3341.49
50.28..5.32
15.63-3.144
LSTUR
61.03-00.24
28.31
53.57433.88
15.74
Autolas
7..8911.50
77
17.31
50.5924441
15.73
NRMS
61.49
28.83
53.81122:94
16.45
MLP
17.57=
18
50.66:26:43
15.55::3:10
NAML
60.59
28. 08.15:00.16
54,13555.38
15.74
FairSM
TabNet
74.44+1.66
15.07-1.39 1.39
50.61+4444
15.98+3.14
LSTUR
61.11
28.33-+0.34
15.16-+4.75
15.24-1.8
Auoolas
76.30+2.56
86.8842005
50.534-5.02
15.50+3.10
NRMS
61.78+0.31
28.95-+0.22
15.91+1.99
MLP
17.391142
78.08E1 24
15.75:4.62
NAKIL
28.21 10017
S4.7522
1566ET9U
FairRec
TabNet
71.8911.76
75 15.6511661
50.35
15.61
LSTUR
60.99
28.31 20.447
53.36.1.33
15.65
Autolai
76.60:11.11
77.1711
50.43:7001
15.83-33.96
NRMS
61.41:0.16
0.:76:
$3.60:12.89
16.26:12.39
MLP
81.4722.14
77 1.5
79.05
1..2
XAML
63.93.0.45
30.75.0.44
69. 69.72200.48
20.05
VFL
TabNet
81.77-1.72
78.0911..35
78.75:0.77
1..36.1..32
LSTUR
60.39
30.85.00.19
70.07:0.337
19.9221.63
Autolat
81.65=1.52
12102:1117
79.07=142
47.96=11.51
NRMS
64.38-00.13
30,93:20.11
70.67=0.25
21.41:0.66
MLP 76.74
77 17.8712..4
50.31
45.30
NAML
60.41
27.95.10.18
53.38114.40
15.5
FairVEL
TahNet
75.51=00.69
76.06+-0600
50.72=-5.72
15.48=2.46
LSTUR
60.98 -0.28
28.25-00.36
53.51+33.1
15,23=0.94
Autolas
16.19:0.99
86.8640.85
50.53a2.448
15.22-2..93
NRMS
60.4
28.81
53.3342.35 35
15.98
The target task is to classify whether the input sample has the attribute "Smiling". We also treat gender (the "Male" attribute) as the target fairness- fairness-sensitive senssiive attribute.
In FairVFL, the local representations and unified representations are 400-dimensional. The protected representations for gender and age are 32- and di-dimensional, respectively. The weights of con- trastive adversarial loss for different sensitive features are set to 0.25. The weights of adversarial loss for gender and age are set to & and & on NEWS, respectively. Besides, the weights of adversarial loss for gender and age are set to 1e2 and 1el on ADULT, respectively. We exploit Adam algorithm [ for model optimization with le-4 learning rate. The size of the mini-batch for model training is set to 32. Besides, for both age and gender, we set E; to 5 for simplification. We also use the dropout tech- nique [88] with a 0.2 drop probability to alleviate model overfitting. based on the validation dataset. Codes are available in wetph:tt////e htpp::///ithbbb com comm/taoq19//Fair///
are selected
4.2 Performance and Fairness Evaluation
Remark that we can achieve counterfactual fairness by reducing fairness-sensitive features encoded in data representations that used for the model predictions (27). Thus, following existing works 12 4 46 27], we utilize the ensemble of attackers to predict fairnes--ennsttvee features from unified representations s to evaluate the independence of faimess-sensitive features and data representations to verify model fairness. If attackers can distinguish fairness-sensitive bias from unified representations then we can say the model is unfair. Thus, lower F1 score on the fairness- sensitive feature classification task means better model fairness. Besides, we use accuracy and Fl score evaluate the classification task on ADULT and CelebA. Moreover, following these works [1 4I]. we use AUC and nDCG@ @ 10 to evaluate the news recommendation task on NEWS. Each experiment is repeated five times and we report the average performance.
We compare our FairVFL method with several recent representative baseline methods for fair representation learn- ing: (1) FairGo [6): user-centric fair representation learning framework that utilizes adversarial learning reduce fairness-sensitive bias information in rep- resentations; (2) FairSM 27 unified framework to achieve counterfactual fair- ness on representations: (3) FairRec 45]:
Table 2: Model performance and fairness on CelebA.
Main Task
Fairness
Method
Accuracy
F1
GenderF11
CenTrain
0.9180=000013
0.9180=F0:014
0.8706=0.0069
FairCio
0.9096+0.0086
0.9095+00.088
0.5159=0.05 70
FairSM
0.003 0.9027:10.0242
0.5259
FairRee
0.9070:10.0094 0.9067110.0097
0.5241
VFL
0.9192-10.0020 0.9174110.0021
0.8727-0:0066
FLi
0.9045=20.0130
0.9042=:0.035
0.5143=0.0778
an adversarial representation decomposi- tion framework that can decompose bias and blas-insensitive information. These methods rely on centralized storage of features to learn fair ML models. Besides, both FairVFL and these three meth- ods are general frameworks and can be applied to basic models to improve their fairness. addition, baseline methods also include standard vertical federated representation learning method (VFL) without fairness criterion 7 Thus, to compare their effectiveness, we combine them with several basic ML models on the three datasets. Basic models for ADULT are implemented by three SOTA
8

---

70.0
40.0
15
PaiVFL w/O CAL
FeirVEL wio CAL FairVFL
57.5
30.0
12
FaIVFL
20
Random
Social Relation
Random
Gender
15
Agj
FarVFL
10-
32.5-
10.0
adverissal loss
gender adverisial loss
Gender
Age
Education prediction
Social relation prediction
(a) Ablation study adversarial learning.
(b) Ablation study contrastive adversarial learning.
Figure 3: Ablation study adversarial learning and contrastive adversarial learning. CAL denotes contrastive adversarial learning and Random denotes predicting private data randomly.
models for modeling structural features, including MLP [7. TabNer | and AutoInt 37
Basic
models for NEWS are implemented by SOTA news recommendation models, including NAML [1]
LSTUR 1 and NRMS [44] Moreover, the VFL model for CelebA is based on ResNet-10 10 [9 Due to space limitations, detailed settings of these basic models are summarized in the Supplementary.
Results on ADULT and NEWS are summarized in Table 1 and results on CelebA are summarized in Table from which we have several observations. First, basic models based on vertical federated training (e.g., MIFI+VFI) can achieve similar performance with the same basic models based on centralized training (c.g., MLP). These results show that the vertical federated learning technique can effectively enable ML models to utilize decentralized feature fields for target tasks without the disclose of raw data. Second, models without the protection of fair ML methods (c.g., MLP and MLP+VFLL biased by tairness-sensitve features. This is because data may encode bias sensitive features like genders. Models may inherit bias from data and become unfair to some user groups Third, after applying our Fair method basic models, the fairness of these models effectively improves. This is because our Fair method applies the adversarial learning on bias discrimination to unified representations, which effectively reduce bias encoded in them and improve model fairness. Fourth, compared with models protected by centralized fair machine learning methods (i.e., FairGo, FairSM and FairRec), models protected by FairVFL can achieve similar performance and fairness. These results verify that our FairVFL method can effectively improve the fairness of VFL models.
4.3 Ablation Study
In this section, we first evaluate the effectiveness of adversarial learning improving model fairness. Due to space limitations, we only show results on ADULT in the following sections. Results are shown in Fig. from which we have several observations. First, after removing adversarial learning on gender discrimination, model fairness on user gender seriously drops. This is because users of the same gender usually have similar behavior patterns and there may exist gender bias data. Models may inherit gender bias from implicit feature correlations in data and become unfair to some user groups. To tackle this challenge, FairVFL applies adversarial learning on bias discrimination to unified representations to reduce gender bias encoded in them. Second, removing adversarial learning for age discrimination also seriously hurts model fairness on user age, which further verifies the effectiveness of adversarial learning for improving model fairness.
Next, verify the effectiveness of contrastive adversarial learning privacy protection. Since unified representations include private information of fairness-insensitive features on local platforms, communicating them across platforms may arouse privacy concerns. Motivated by membership inference attack methods, we evaluate the privacy protection by inferring the input features of unified representation from corresponding protected representations {a,li 1, ..., m} that are shared across platforms { Pro 1, .... m Specifically, based on the ADULT dataset, we infer the value of input features from each a, via 5 attackers and average the results. A lower inference accuracy (the Fl value) means better privacy protection. We remark that for faireess evaluation, we distinguish bias on fairness-sensitte features from unified representations instead of the shared representations, and these sensitive features are also not the input of representation learning. We only show results on two representative feature fields (i.e., education, and social relation) Fig. 3 due space limitations. First, without the protection of contrastive adversarial learning, shared representations very informative

---

1.5
24
77
35
Fair/LL Random
FaiNFL Random
FaiNFL
FarVEL
Educetion F1
10-
Social Relation
Gender F1
88
29
Age
23
80
17
0.1
0.5
10
0.5
0.1
0.5
10
0.1
0.5
10
Ye
(a) Influence CAL weight y. on privacy protection.
(b) Influence CAL weight y, on model fairness.
Figure 4: Influence of contrastive adversarial learning weights on privacy protection and model fairness, where contrastive adversarial loss weights for different sensitive features are set to Ye-
for inferring private user information. These results verify that directly sharing unified representations across platforms may leak user privacy. Second, contrastive adversarial learning effectively protect user privacy in protected representations. This is because contrastive adversarial learning enforces protected representations be indistinguishable for predicting its preimage from unified representations of the same faimness-ennstivve features, which can effectively reduce bias-irrelevant personal information in shared representations.
4.4 Hyper-Paramett Parameter Analysis
Next, we evaluate the influence of the weights of contrastive adversarial learning {fili 1, 2.., m) on both privacy protection and model fairness. To simplify the analysis, we set them to the same value (denoted 7c). The influence of % on privacy protection and model fairness is shown in Fig. 4 from which we have two observations. First, with the increase of "e. FairVFL. can achieve better privacy protection ability. This is intuitive since larger Te makes contrastive adversarial gradients more important to update the mappers, making them can reduce user privacy in unified representations more effectively. Second, larger Tee also hurts model fairness more seriously. This is because larger % makes mappers pay more attention to reducing user private information, which may hurt their abilities on retaining information of fairness-sensitive bias information. Thus, the protected representations may contain less bias information and hurt the effectiveness of adversarial learning in improving model fairness. Fortunately, when Ye is small (e.g., 0.25), FairVFL is effective in privacy protection since the performance of user privacy inference is close to random. Besides, FairVFL can achieve satisfied model fairness (about 0.5 and 0.15 Fl value for gender and age prediction) when % small (e.g., 0.25). This indicates that FairVFL. can achieve effective privacy protection ability without hurting model fairness and we set % 0.25 FairVFL.
CONCLUSION
In this paper, we propose fair vertical federated learning framework (named FairVFL), which can improve the faimess of VFL models in a privacy srivacy-preeervngg way. The core of FairVFL is to learn fair and unified representations to encode samples based on their decentralized feature fields. Each platform storing
features first learns local representations from their local features. Then these local representations are uploaded to a server to form a unified representation used for the target task. In order to further learn fair and unified representations, fairnes--sensttvve features and apply adversarial learning to reduce bias encoded in them. Besides, to protect shared representations from leaking user privacy, we propose a contrastive adversarial learning method to reduce user private information in them before sharing them with other platforms. Experimental results on three datasets show that FairVFL comparable with centralized baseline fair ML methods in terms of both performance and fairness, meanwhile effectively protects user privacy.
send them to platforms storing
Acknowledgments
This work was supported by the National Key R&D Program of China 2021ZD01 13902, the National Natural Science Foundation of China under Grant numbers 82090053, U1936126, and Tsinghua- Toyota Joint Research Fund 20213930033.
10

---

References
[1] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xic. 2019. Neural news recommendation with long-and and short-term user representations. In ACL. 336-345.
[2] Sercan O Anık and Tomas Pfister. 2020. Tabnet: AAAI, 6679-6687.
Attentive interpretable tabular learning. In
[3] Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. 2019. Fairness in recommendation ranking through pairwise comparisons. In KDD. 2212-2220.
[4] Avishek Bose and William Hamilton. 2019. Compositional fairness constraints for graph embeddings. In ICML. 715-724.
[5] Robin Burke, Nasim Sonboli, and Aldo Prdonez-Gaugen 2018. Balanced neighborhoods for multi-sided fairesss in recommendation. In ACM FAceT. 202-214.
[6] Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush Varshney. 2017. Optimized pre-processing for discrimination prevention. In NIPS. 1995-4004
[7] Sicong Che, Hao Peng, Lichao Sun, Yong Chen, and Lifang He. 2021. Federated Multi-View Learning for Private Medical Data Integration and Analysis. TIST (2021).
[8] Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. 2021. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems (2021).
[9] Yong Cheng, Yang Liu, Tianjian Chen, and Qiang Yang. 2020. Federated learning for privacy- preserving AI. Commun. ACM (2020), 33-36.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In NAACL. 4171-4186.
[11] Vijay Prakash Dwivedi, Deepak Kumar Singh, Saurav Jha, et al. 2017. Gender Classification of Blog Authors: With Feature Engineering and Deep Learning using LSTM Networks. In ICoAC. 142-148.
[12] Yanai Elazar and Yoav Goldberg. 2018. Adversarial Removal of Demographic Attributes from Text Data. In EMNLP, 11-21.
[13] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. 2015. Certifying and removing disparate impact. In KDD. 259-268.
[14] Fangcheng Fu, Yingxia Shao, Lele Yu, Jiawei Jiang, Huanran Xue, Yangyu Tao, and Bin Cui. 2021. VF2Boost: Very Fast Vertical Federated Gradient Boosting for Cooss-Enterprise Learning. In SIGMOD. 563-576.
[15] Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, et al. 2021. Towards Long-term Fairness in Recommen- dation. In WSDDM 445-453
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM (2020), 149.144.
[17] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting Deep Learning Models for Tabular Data. arXiv preprint 17X: (2021).
[18] Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In NIPS. 3315-3323.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770-778.

---

[20] Lingxiao Huang and Nisheeth Vishnoi. 2019. Stable and fair classification. In ICML. 2879- 2890.
[21] Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. 2019. Federated Topic Modeling. In CIKM. 1071-1080.
[22] Aria Khademi, Sanghack Lee, David Foley, and Vasant Honavar. 2019. Fairness in algorithmic decision making: An excursion through the lens of causality. In WWW. 2907-2914.
[23] Diederik Kingma and Jimmy Ba. 2015. Adam: Method for Stochastic Optimization. In ICLR.
[24] Ron Kohavi et al. 1996. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.. In KDD. 202-207.
[25] Joshua K Lee, Yuheng Bu, Deepta Rajan, Prasanna Sattigeri, Rameswar Panda, Subhro Das, and Gregory W Wornell. 2021. Fair Selective Classification via Sufficiency. In ICML. 6076-6086.
[26] Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021. User-oriented Fairness in Recommendation. In WWW. 624 632.
[27] Yunqi Li, Hanxiong Chen, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. 2021. Towards Personalized Fairness based on Causal Notion. In SIGIR. 1054-1063.
[28] Changxin Liu, Zirui Zhou, Yang Shi, Jian Pei, Lingyang Chu, and Yong Zhang. 2021. Achieving Model Fairness in Vertical Federated Learning. arXiv preprint arXiv:2109.08344 (2021).
[29] Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2020. Federated learning for vision-and-language grounding problems. In AAAI. 11572-11579.
[30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015, Deep learning face attributes in the wild. In ICCV. 3730-3738.
[31] Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. 2021. Feature inference attack on model predictions in vertical federated learning. In ICDE. 181-192.
[32] Binh Thanh Luong, Salvatore Ruggieri, and Franco Turini. 2011. 1. k-NN as an implementation of situation testing for discrimination discovery and prevention. KDD. 502-510.
[33] Lingjuan Lyu, Xuanli He, and Yitong Li. 2020. Differentially private representation for nlp: Formal guarantee and an empirical study on privacy and fairness. EMNLP. 2355-2365.
[34] Lingjuan Lyu, Yitong Li, Xuanli He, and Tong Xiao. 2020. Towards differentially private text representations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1813-1816.
[35] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2018. Learning adversarially fair and transferable representations. In /CML. 3384-3393.
[36] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Coimuurication-ffficeeeee Learning of Deep Networks from Decentralized Data. In AISTATS. 1273-1282.
[37] Weiping Song. Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019, Autoint: Automatic feature interaction learning via self-ateentive neural networks. In CIKM. 1161-1170.
[38] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR (2014), 1929-1958
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. 6000-6010.
12

---

[40] Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, and Zhiyuan Zong. 2020. Privacy leakage of real-world vertical federated learning. arXiv preprint arXiv:2011.09290 (2020).
[41] Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang. and Xing Xie. 2019. Neural news recommendation with attentive multi-view learning. In IJCAI.
[42] Chuhan Wu, Fangzhao Wu, Mingxiao An, Tao Qi, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019. Neural news recommendation with heterogeneous user behavior. In EMNLP. 4876 4876-4885.
[43] Chuhan Wu, Fangzhao Wu, Tao Di, Yongfeng Huang, and Xing Xie. 2020. FedCTR: Fed- erated Native Ad CTR Prediction with Multi-Platform User Behavior Data. arXiv preprint arXiv:2007. 12135 (2020).
[44] Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019. Neural news recommendation with multi-head self- EMNLP. 6390-6395.
[45] Chuhan Wu, Fangzhao Wu, Xiting Wang. Yongfeng Huang, and Xing Xie. 2021. FairRec.Fairness-aaar News Recommendation with Decomposed Adversarial Learning, In AAAI, 4462-4469.
[46] Le Wu, Lei Chen, Pengyang Shao, Richang Hong, Xiting Wang, and Meng Wang. 2021. Learning Fair Representations for Recommendation: Graph-based Perspective. In WWW. 2198-2208.
[47] Yuncheng Wu, Shaofeng Cai, Xiaokui Xiao, Gang Chen, and Beng Chin Ooi. 2020. Privacy preserving vertical federated learning for tree-based models. In VLDB. 2090-2103.
[48] Kai Yang, Tao Fan, Tianjian Chen, Yuanming Shi, and Qiang Yang. 2019. A quasi-newton method based vertical federated learning framework for logistic regression. arXiv preprint (2019).
[49] Shengwen Yang, Bing Ren, Xuhui Zhou, and Liping Liu. 2019. Parallel distributed logis- tie regression for vertical federated learning without third-party coordinator. arXiv preprint arXiv:1911.0982. (2019).
[50] Jing Yao, Zhicheng Dou, and Ji-Rong Wen. 2021. FedPS: A Privacy Protection Enhanced Personalized Search Framework. In WWW. 3757-3766.
[51] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodrigues and Krishna P Gummadi. 2019, Fairness constraints: A flexible approach for fair classification. JMLR (2019), 2737-2778.
[52] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In JCML. 325-333.
[53] Qingsong Zhang, Bin Gu, Cheng Deng, and Heng Huang. 2021. Secure Bilevel Asynchronous Vertical Federated Learning with Backward Updating. In AAAI. 10896-10904.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes] See Supplementary Information.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work focuses on privacy protection and model fairness.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results....
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
13

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] Section 4.1
and Supplementary Information.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and Supplementary Information.
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Section 4.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Supplementary Information.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.
(b) Did you mention the license of the assets? [Yes] See Supplementary Information.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See Section 4.1.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] See Supplementary Information.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Supplementary Information.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wFp48j0qsY0", "reviewer": "Reviewer_Tmvi", "summary": "This paper proposes a novel vertical federated learning framework (FairVFL) to achieve fair VFL models while keeping information private during communication. In order to encode a fair unified representation for the task platform, the prosed method employs adversarial learning at the fairness-insensitive platform side to prevent the fairness-sensitive bias. Moreover, instead of directly sending the unified representation to the fairness-insensitive platforms, the server sends a protected representation encoded by a mapper trained by a novel contrastive learning method.", "questions": "- How consistent are $\\lambda$ and $\\gamma$ cross datasets?", "limitations": "Yes, the authors mention the limitations in the appendix. As the authors mention, the method requires more rounds of communications than the baseline VFL. Meanwhile, it also requires more computations than the baseline.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and organized. The method is clearly explained and well-motivated.\n- The results are very convincing. The proposed method can have a slightly higher performance on fairness with a reasonable amount of drop in main task performance, compared to the methods under centralized training. Also, plenty of experiments on different models show the method is model-agnostic.\n- The ablation study on adversarial learning and contrastive adversarial learning helps shed light on the power of the two methods in the framework.\n\nWeaknesses:\n- Multiple times of communications might be a problem in federated learning.\n- Also, the local adversarial training might also slow down the overall training and increase the computational burden for both servers and platforms.\n- Two hyperparameters, $\\lambda$ and $\\gamma$, seem to be very important. It might be hard for the server to choose the correct values for a new task.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lLXATqNtUL", "reviewer": "Reviewer_rSwq", "summary": "This paper proposes a method to learn fair representations in the VFL setting. By fairness, the authors mean that the representations generated by FairVFL should not reflect sensitive attributes, such as age/gender. The authors achieve this goal via a contrastive adversarial learning on the sensitive attributes. The idea of this adversarial training is that, the discriminator aims to learn sensitive attributes, while the feature learner aims to fool the discriminator. Moreover, in order to conceal private information, the adversarial learning also aims to conceal all other information except for the sensitive attribute. Experiments on two real-world datasets show the effectiveness of FairVFL. ", "questions": "Q1-3: See Weaknesses 1 2 3. \n\nQ4: In Figure 4(b), it seems that in some cases the fairness-F1 scores can go below random. For example, in Figure 4(b) left, gender-F1 (binary classification) can be <0.5. This seems counter-intuitive, as the best fairness should be achieved at gender-F1=0.5.\n\nI thus have a question, when gender-F1<0.5 (e.g. 0.45), can the attacker simply flip the prediction and get gender-F1>0.5 (e.g. 0.55)? What does it imply on fairness? \n\nQ5: The authors state that \"Intuitively, highly relevant data samples are likely to share the same fairness-sensitive feature\" in line 178. I wonder after the model is trained well, and $\\mathbf{s}$ does not contain fairness-related information, does this assumption still hold? If not, how will it influence the model training? ", "limitations": "I do not see any ethical concerns. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**I am not very familiar with the fairness issue in machine learning. Please correct if I make misunderstandings regarding fair ML.**\n\nStrengths: \n1. **Well-motivated problem.** Trustworthy machine learning is a good topic to explore. The authors combine trustworthy machine learning with VFL, which is a good attempt. \n2. Evaluations that verify the privacy protection (Fig. 3b) is a plus. It evaluates how well the proposed method protects attribute privacy, which is a focus in FL. \n3. **Presentation is OK**. Despite the very complex gradient flow (in my opinion), the authors manage to explain the overall workflow to me. \n\nWeaknesses: \n1. **Lacking a baseline**. The authors take plain VFL as the only baseline in the feature-partition scenario. Although there are other fair ML methods compared (e.g. FairGO, FairSM), they cannot be applied in the VFL setting (as far as I can infer from the paper). Therefore, an intuitive question to ask is that, are there simpler VFL+fairness baselines that can be applied to the problem, such as slightly modifying FairSM, FairGO? If there is such a baseline, the improvements made by the authors' designs would be clearer. \n\n2. **Definition of fairness**. The authors state in Sec. 3.1, line 114 that, \"the counterfactual fairness requires that the model prediction for a sample should always hold if we only change fairness-sensitive attributes\". However, in the evaluation, the authors use the metric \"Gender F1\", namely how well an attacker can guess the sensitive attribute, to measure fairness. I am a little confused about the metric. Specifically, as I can tell from Section 3.2, the unified representation $\\mathbf{s}$ is computed from $\\mathbf{s}_i^l$, which are all computed from fairness-insensitive features. In other words, sensitive attributes are not involved in the forward computation for the task $t$. It seems that in this sense, we have no chance to verify the 'counterfactual fairness', as we cannot even change the fairness-sensitive attributes. Thus, how the metrics (gender F1) are related to the notion of 'counterfactual fairness' should be better elaborated. Maybe the Gender F1 is more related to notions like conditional independence as far as I know. \n\n3. **Should discuss limitations and tradeoffs on efficiency and stability**. The overall workflow of FairVFL is rather complex (even though the authors manage to explain it to me). As far as I am concerned, there are many points in the workflow of FairVFL that contain computation dependencies and may lead to stragglers (e.g. the iterative adversarial learning of $A$, $D$), which may significantly compromise the efficiency. Thus, I would like to see an efficiency comparison between naive VFL and FairVFL to see the tradeoffs between fairness and efficiency. Also, adversarial training tend to be very unstable. The authors should discuss how to mitigate the instability and generate consistent results. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "HQ-DeicBLuu", "reviewer": "Reviewer_ajz9", "summary": "This paper proposed an approach to improve the fairness of VFL models without the requirement of storing the fairness-sensitive features in a central server. To improve the fairness of VFL models, an additional adversarial loss is designed to remove the bias of unified feature representation. The experiments result show the effectiveness of the proposed fairness algorithms for VFL.", "questions": "1). The authors claim that their approach can protect the exchange of information between the local model, aggregation models and task-specific models. The protection is based on removing the useful information from embedding. However, much research shows that the exchange of information may disclose the information on the original data, e.g. [1,2]. I am wonderring if the proposed training apporach can defende this attacks.", "limitations": "The variety of the dataset could be improved.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.) The paper is well-written and easy to understand the concepts and algorithms.\n2). The research question is in-time and the fariness related research in VFL setting is less explored.\n3). The experiment's result seems promising.\n\nWeaknesses:\n1). It would be better if the experiments can be conducted in a different types of datasets, e.g. image dataset celebA. \n\n[1] Batch Label Inference and Replacement Attacks in Black-Boxed Vertical Federated Learning\n[2] Deep Leakage from Gradients\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept