Accelerated Vertical Federated Adversarial Learning through Decoupling Layer-Wise Dependencies
Authors: Tianxing Man
Yu Bai
Ganyu Wang
Jinjie Fang
Haoran Fang
Bin Gu
Yi Chang
Venue: NeurIPS 2025 poster ()
Keywords: Vertical Federated Learning
Adversarial Training
Adversarial Sample
Robustness
Primary area: deep_learning

Abstract: Vertical Federated Learning (VFL) enables participants to collaboratively train models on aligned samples while keeping their heterogeneous features private and distributed.
Despite their utility, VFL models remain vulnerable to adversarial attacks during inference. 
Adversarial Training (AT), which generates adversarial examples at each training iteration, stands as the most effective defense for improving model robustness. 
However, applying AT in VFL settings (VFAL) faces significant computational efficiency challenges, as the distributed training framework necessitates iterative propagations across participants.
To this end, we propose **_DecVFAL_** framework, which substantially accelerates **_VFAL_** training through a dual-level ***Dec***oupling mechanism applied during adversarial sample generation.
Specifically, we first decouple the bottom modules of clients (directly responsible for adversarial updates) from the remaining networks, enabling efficient _lazy sequential propagations_ that reduce communication frequency through delayed gradients. 
We further introduce _decoupled parallel backpropagation_ to accelerate delayed gradient computation by eliminating idle waiting through parallel processing across modules.
Additionally, we are the first to establish convergence analysis for VFAL, rigorously characterizing how our decoupling mechanism interacts with existing VFL dynamics, and prove that _DecVFAL_ achieves an $\mathcal{O}(1/\sqrt{K})$ convergence rate matching that of standard VFLs.
Experimental results show that _DecVFAL_ ensures competitive robustness while significantly achieving about $3\sim10\times$ speed up.

Accelerated Vertical Federated Adversarial Learning through Decoupling Layer- Dependencies
Tianxing Man
Yu Bai
Ganyu Wang*
Jinjie Fang"
Haoran Fang*
Bin Gu
Yi Chang
Chang2.3/4;
1,3,4
1 School of Artificial Intelligence, Jilin University, China
Western University, Canada
3 International Center of Future Science, Jilin University, China
EEgineering Research Center of Knowledge-Drive Human-Machine Intelligence, MOE, China {mantx, gubin, yichang) @jl.. edu. en {gvang382) (err:88222www..oom ca (baiy23, fangjj24, fanghr24) @mails jlu. edu. on
Abstract
Vertical Federated Learning (VFL) enables participants to collaboratively train models on aligned samples while keeping their heterogeneous features private and distributed. Despite their utility, VFL models remain vulnerable to adversarial attacks during inference. Adversarial Training (AT), which generates adversarial ex- amples at each training iteration, stands as the most effective defense for improving model robustness. However, applying AT in VFL settings (VFAL) faces significant computational efficiency challenges, as the distributed training framework necessi- tates iterative propagations across participants. To this end, we propose Dec VFAL framework, which substantially accelerates VFAL training through a dual-level Decoupling mechanism applied during adversarial sample generation. Specifically, we first decouple the bottom modules of clients (directly responsible for adver- sarial updates) from the remaining networks, enabling efficient lazy sequential propagations that reduce communication frequency through delayed gradients. We further introduce decoupled parallel backpropagation accelerate delayed gradient computation by eliminating idle waiting through parallel processing across modules. Additionally. rigorously characterizing how our decoupling mechanism interacts with existing VFL dynamics, and prove that DeCVFAL achieves an ((I/VK) convergence rate matching that of standard VFLs. Experimental results show that DecVFAL ensures competitive robustness while significantly achieving about 3 ~ 10x speed up.
the first to establish convergence analysis for VFAL,
1 Introduction
Federated learning (FL) enables collaborative training of deep learning models among distributed participants without sharing data [51]. Most FL research considers Horizontal Federated Learning (HFL), which assumes distributed clients possess data with identical features but varying sample spaces [80] Vertical Federated Learning (VFL) assumes distributed clients share the same samples but have different features 44 [66). VFL model comprises a server-maintained top model and client-side bottom models that map local data features embeddings. During inference, each client computes the local embeddings and uploads to the server through a communication channel for prediction 44)) Due its advantages facilitating data collaboration across multiple industries.
"Equal Contribution. t Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VFL has gained increasing attention in various domains such as recommendation systems
[3 75
finance [46] 77. healthcare [58] 6 and emerging applications 60 [9]
Threat model. Machine Learning (MI.) models have demonstrated vulnerability to adversarial attacks, carefully crafted inputs designed induce misclassification during inference. Distributed deployment inherently exposes VFL model to additional security threats [24] [6]. Adversarial attacks in VFL can manifest in multiple forms during inference: via third-party adversary intercepting and
altering embeddings during client-server communication [16., or through malicious or colluding clients perturbing local features of raw data [22 533.
The details of the threat model are presented
in Appendix A.3 These diverse attacks underscore the unique security challenges in VFL systems, motivating the urgent need to address the VFL robustness problem.
Extensive research has been conducted on de- fenses against adversarial attacks, with Adver- sarial Training (AT) emerging as the most em- pirically robust approach date [61]. AT is min-max robust training method that minimizes the worst-case training loss at generated adver- sarial examples [49]. The objective function
M2
M1
8
2
0.0
of AT is defined in Eq. (3.I). The deployment of AT in the FL, termed Federated Adversar- ial Learning (FAL), has garnered attention 44) In HFI. where participants maintain complete model copies [40), studies integrate AT into clients' local training 440 [4] 1 83 (77). How- ever, for applying AT in VFL settings (VFAL), models are partitioned across server and clients, requiring multiple communications for adver- sarial sample generation- aspect that has re-
0
-0"
-0
00
Time
Figure Comparison of computation efficiency for adversarial sample updates: VFL with PGD (up) versus Dec VFAL (down). M2, M3 are the layer-wise modules of VFL model. DecVFAL updates examples x times in approximately the same time as performing 2 PGD updates.
ceived limited research attention.
Observation. As shown in Figure up. inherent sequential dependencies across layers cause modules to remain idle until receiving necessary information (embeddings or gradients) from adjacent modules. Assuming M3 is server model and M1, .M2 are the modules of client models, VFAL can exhibit more substantial idle waiting due frequent server-client communications. Consequently. the training time for VFAL significantly exceeds that of regular VFLs. To illustrate (Figure VIAL using PGD-20 (Projected Gradient Descent) requires about 20 times more computational cost than regular VFL due to 20 iterations needed to generate adversarial example. Taking into account this, natural question arises: In light the intensive adversarial sample generation and inherent sequential dependencies cross participants, how robust performance?
20),
accelerate VFAL training while maintaining
To tackle the computational efficiency challenge in training robust VFL models, we propose Dec V- FAL. an accelerated VFAL framework through novel Decoupled backpropagation incorporating a dual-level decoupled mechanism for adversarial sample generation (Figure (b)). DeAL first decouples the bottom module fM, (responsible for adversarial updates) from the remaining network, enabling lazy sequential backpropagation. This approach fixes the partial derivatives (ō while utilizing current gradients at the bottom module (V,/M,) ) to perform multiple sequential sample updates without requiring frequent client-server communications, Simultaneously, Dec VFAL intro- duces decoupled parallel backpropagation to update partial derivatives in the remaining modules asynchronously using module-wist delayed gradients, eliminating idle waiting times and enabling truly parallel computation across all modules. Furthermore, by formulating AT as a dynamical system 39] we provide the convergence analysis for VFAL, accounting for multi-source approximation gradients from decoupling mechanisms and VFL architecture.
Contributions. In summary, the contributions of our paper are:
We propose DecVTAL, a novel dual-level decoupling framework that comprehensively addresses VFAL's computational bottlenecks. DecVFAL RecVFALrreduces communication complexity from r (es M x N) to M rounds while eliminating idle waiting through asynchronous parallel processing, achieving substantial speedup without sacrificing robustness.

---

We provide the first convergence analysis for VFAL, overcoming the challenge of han- dling multi-source approximation gradients from both decoupling mechanisms and VFL framework. We rigorously prove that DecVFAL achieves O (11/VK) convergence rate matching standard VFL.
Experimental evaluation multiple datasets demonstrates that VTAL achieves 3 10x speedup compared to existing VTAL methods while maintaining competitive robust accuracy, making robust VFL training practically feasible for real-worke deployment.
(<)
(b)
P4
Q,
plC-At
MM.0
Mut Pus
Deceapid Purakl Indigapggainn
Supsrntd Medgpogpaation
Tiins
32 Timo
AdverortatSurpp Updass
Mudii Updatia
Adromiriaa Sumpl Eydani
Meddl Eydating
Figure 2: The conceptual comparison between VFAL with PGD (a) and DecVFAL (b): VeAL accesses the data M x N times with only M server-client communications, each backpropagation is accelerated through the decoupling mechanism. In contrast, PGD necessitates full backpropagations to perform adversarial sample updates. (ŌM, represents the error gradient dyM, for simplicity).
Related Works
Accelerating Adversarial Training. PGD-based AT methods 44 demonstrate superior defensive capabilities but present significant computational challenges, Acceleration approaches include FreeAT [55]. which combines model and adversarial updates in single backward pass; FreeLB 82 which accumulates gradients before parameter updates; and YOPO [76). which restricts updates to the first layer. Alternative FGSM-based acceleration methods like ATAS [70] and FastAT [255 offer computational advantages but struggle to achieve optimal robustness due to their reliance on single gradient steps. Importantly, these existing solutions target centralized training paradigms and fail to address the unique computational efficiency challenges in cross vross-participant VFAL..
Decouple Training. Decoupled training has emerged as effective methods for improving computa- tional efficiency by addressing the inherently sequential propagation in neural networks. ADMM [59] decomposes optimization into parallel subproblems. Synthetic Gradients
28
enable asynchronous
updates through gradient prediction. Delayed Gradient Method (2 26 81 temporal shifts in gradient
computation for parallel processing and Lifted Machines 20 36 transform network architecture
to enhance parallelization. Notably, these techniques have yet to be applied to adversarial example
generation, which the most computationally intensive component AT.
3 Preliminaries
Notations.
A standard synchronous VFL framework comprising one central server and C clients [4]. We consider classifier implemented as T-layer deep neural network f(Q; where repre- sents the input features and θ denotes the set of trainable parameters. The training dataset denoted as {( yi \ -- with S representing the total number of samples. Each sample is composed of features from different clients, specifically To = (To, ( (C))- The classifier comprises client mod- and server model fa parameterized by vb. The classifier function expressed as (0, = fa(tb La (n) wo...))......................... S(o (oo))) We give the list of notations in Table B.I
els [f(1).... f(cy) parameterized by [0.
3

---

Adversarial Training. AT optimizes the parameters ② of a neural network f on adversarially perturbed imputs, thereby increasing robustness against test-time attacks 49 This process involves solving minimax problem on the cross-entropy loss:
min max Se
C(((O, 1k);
(3.1)
where ((:3)) is the loss function, 7/s represents the adversarial perturbation, bounded by a non- negative scalar value that constrains the perturbation's 's magnitude.
Definition (Vertical Federated Adversarial Learning). Building upon the standard VFL models and the minimax problem in AT, a T-layer neural network f is defined recursively as: It fe (1 tt), I.....T T, where 2; are the output of the t-th layer, O are the parameters of layer ft. O denotes the concatenation of (0:) VFAL addresses the following problem:
min max Inalls Se
Σ ((TT.BB))
ΣΣ
Ro,, 11)1,),
(3.2)
subject to F1i = f1(01, m)), 21,1 = fr(Oc,, (t-1),)
where 2y,,, is the output of the last layer fr: ZT, f(O: + )= (rr: fo (+r-1i... f1(O1: To. to.
7.).. )), R. a potential regularization term for layer ft. Assume t. is the number of client model layers, for to < t S T, O, = 2, are the server model parameters; for 0 < t to. θ, = (0.,1)...... (c,(c)) are the client model parameters. T/s
(.t.).........c.C represents adversarial perturbations on sample i.
Backward Locking. In VFAL's distributed setting, T T-layer neural network is partitioned into MM modules. The partial derivatives computation in module fM,. depends on enror gradients from module MML+1. This creates "lock" preventing modules from updating until receiving backward results from dependent modules. As shown 2 (a), each adversarial example update in VFAL requires sequential gradient propagation from output to input layer, significantly increasing computational overhead due to cross pooss-participant communication.
4 Methods
To address the training efficiency bottleneck, DecVFAL introduces dual-level decoupled mechanism that utilizes module-wise staleness to unlether the dependencies across layers inherent in VFAL.. As shown in Figure (b), DecVFAL utilizes delayed gradients to eliminate backward locking, enabling module-wise asynchronous backpropagation. It restricts perturbation update propagations to the bottom model to reduce full propagations and utilizes gradients from disparate iterations to achieve parallel backward computation.
4.1 Lazy Sequential Backpropagation
key observation in VFAL that the adversarial perturbation directly coupled with the bottom module of the network. This insight allows us to decouple the bottom module fM, and the remaining modules fMg. and 2M, is the output of bottom module. The VFAL classifier can be rewritten as: f(O; + T = fNi, (0 fM, (0M, 3o + 1))- PGD-based AT (PGD-r) involves r sweeps of forward and backward propagation to generate an adversarial example, resulting in extensive computational cost. To mitigate this, we introduce a "lazy" backpropagation mechanism by freezing the partial derivatives dWMn of the remaining modules slack variable pM;
fuū (0yi, ITM;). ). where fMū, = f fMa
(4.1) where PM; obtained after each full backpropagation. The perturbation 7 is updated using PMM; and N-step gradient ascent, while keeping the network parameters θ fixed (lines 7-11 in Algorithm []. As shown in Figure [ DecVFAL accesses the data M x N times for each adversarial example generation while only requiring M Tull forward and backward propagation, where M <r.
PM4 =V Vi,, (Elfra, (fM, (9M, :20 v)), VaM, (FoM, (fM, (OM : ni))
This frozen slack variable introduces an oracle error in adversary updating, resulting in a delayed gradient.
Inspired by the optimal control theory BB 39 54 and under Assumptions in (B.2). we bound costate difference bottom module Lemma
Lemma Bound the costate/gradient difference the bottom module. There exists a constant G' dependent on T and K such that for all n E (O...., N.). J. m. E (O....,M) .).. ). and i Sl.
PM
N G'a,, (Mx N 11), P.
(4.2)
3 This is inspired by YOPO 76). but extends beyond first-layer limitations to support multi layer modules, offering greater flexibility of network partitioning in VFAL settings.

---

where G² + (K" T(T 1)K2T-2 + TKK2), PM is the delayed gradient to the first module Mi iteration m. is the delay module Mi raised from parallel backpropugation.
4.2 Decoupled Parallel Backpropagation
We decouple backpropagation across the entire network using delayed gradients, enabling parailel updates of the partial derivatives in the remaining modules. The forward pass is performed from module module Mg. In backward pass, all modules except the last store delayed error gradients to perform the backward computation without locking. The module fM₂, keeps the stale error gradient 64m-Th
7 MM MM.
BeMy While lazy backpropagation in the bottom modules, backward computation in the remaining modules fM, is:
8Lm-Th Óx,
8L , (LMe-I)
(4.3)
601-1 ó
Each module receives a gradient from dependent module subsequent computation. The gradients across modules exhibit varying time delays. Moving from module to Mr the corresponding time delays decrease from Mx to 0, delay 0 indicates up-to-date gradients. This mechanism breaks the backward locking constraint, enabling decoupled parallel backpropagation.
For private guaranted We implement DecVFAL within VFL-CZOFO, hybrid cascaded VFL framework [62 with Zeroth Order Optimization (ZOO) and compression, and analyze the errors caused by multi-source approximate gradients due to existing VFL techniques and DecVFAL in Lemma 2
Lemma 2. Bound the gradient n. Under hybrid cascaded VFL, gradient V A respect to yj involves
estimation gradient V ,, ZOO (Appendix A.5) and compression gradient A (Appendix A.6 Under the Assumption ! and Lemma 3
at the i-th sample and k-th iteration, the pseudo-parte peruto-pattial derivative for y satisfies
the following inequality ip. argmiin
V A (n) mn dr, 0.) we define G = KG', az Lnn then:
E|| E
E
$ p(X)
2 Tomn
M8-1
20 Lany
(MRN 1)² (M)N-1)2 1)²(3) 1 2l.yy )
HHCe** 1222d
x2))
(4.4)
Algorithm 1: Dec algorithm
Input:
Learning rates ung us: Train set (X, Y).
Output: Model parameters θ ((2)) (2))...... ((c) )
Initialization: Clients and Server initialize model parameters p()) 02))..... A vh.
2 while not convergent do Randomly select sample
for m 1 to M do m < f(zo + 17 m)";
for k = to Mr in parallel do
Decoupled Parallel Backpropagation
if then
for = 0 N-/ do
Lazy Sequential Propagation
xM, + fM, (Zu + m.);
Updates adversarial perturbation: 17 m 17 ta,PM, VafMii
Backward computation with delayed gradient:
5m
6x,
52
(LMz L.M,l::
6z 8z
S
for each client do
Update client model parameters: gct + 0%) as veL(f((2 (10 m)):
Update server model parameters: t+1 ah as LF + ))m,)))
'According 62 due the introduction of ZOO, Dec satisties differential privacy guarantee.
5

---

4.3 Algorithm
Algorithm Timplements VecL framework within the VFL setting where clients and server collaboratively train robust model. In each iteration, after randomly selecting sample x; the framework performs M full propagations where all modules across clients and server operate in parallel. For each module Mr. partial derivatives are computed using delayed gradients / /ōzz with delay 7 = MM My, Simultaneously. the bottom module (lypically at client side) performs N adversarial perturbation updates through lazy sequential propagation: my 7) n7m + arPM. Vn,f,, using fixed gradient pM, without requiring additional client-server communications. After adversarial example generation, clients update their parameters ((c) and the server updates its parameter y,, completing one training iteration.
4.4 Acceleration of DecVFAL
DecVFAL, accelerates VFAI. training through our dual-Ievel decoupling mechanism. The lazy sequential back- propagation enables M N adversarial sample updates while requiring only M full propagations, empirically achieving comparable robustness PGD-r when M x slightly larger than r. For a standard full propaga- tion taking time T, our decoupled parallel backpropagation reduces this to Mg
T_ by eliminating idle waiting.
Unlike prior research [27 26 that uses delayed gradients solely for model training (limiting acceleration to Toor + Thack) our method enables concurrent forward and backward propagation during adversarial sample
MM
generation since parameters remain constant. Idcally, this further reduces computation time T
Mxx
5 Convergence Analysis
Assumptions. The formal definition and detailed discussion of the assumptions are in Appendix We make several crucial assumptions: the functions fi, Sex L, and R2 K --pipschitz continuous æ, uniformly with respect to and th the gradient of adversarial loss function. V. (1), v,, 0). satisfies Lipschitz conditions (Assumption [y; the adversarial loss function A,,,, th, 0) possesses an unbiased gradient (Assumption ( and characterized by bounded Hcssian matrices Hę and Ho (Assumption 3}. as well as bounded cooodccood gradients Qu and Qo (Assumption 4 Ac(1, vt, 0) exhibits z-strong concavity with respect 17 (Assumption 5
B.2
Theorem 1. Under Assumptions 5 if the step sizes satisfy Qig < 1/Loy,
min fa Oig
max (ckp, ). aMd < co, with v/ = argmaxx, Ai(n, v. 0). A = R (17 - & ,00) inl& R (1) vtt .0))).
am
L, max L, Lo, Le, Linn Lon). Eo {1 Leaus. J. and E {1 + Lpaass ). then:
K-1
Σ
E
[IVR (n 00 [1 I Iz Ep E. + E..
(5.1)
K
where x(M, = D(X) L2g (1
MN+1
[M - 1p slnmm and:
E.g9
Trnn
I 2A
(firstooreer optimization convergence term).
amK
2L. (o +og)
I1
(stochastic gradient descent term).
Omm
Lo
(M, am (Ev 35//1) |1]
E. =1²²
(zeroth-order error term),
40m
am
SH]CT(M, N) 3
E. EaM am
25 + +
(compression error term),
z2
30 3" (M,,
Ep
(2Ea + 3 (decoupled backpropagation error term).
am
Corollary 1. we choose ag and as as VE. == J. 0 =
= +x we derive the sublinear
convergence rate:
E [ (n 0t) [8] 01 NMx M
(5.2)
The bias term O NMMK) reveals direct trade-off between model modularity and performance. the number of modules Mr increases, the bias grows proportionally, which aligns with our empirical observations in Table
Remark 1.
For fixed Mx. the bias term exhibits properties analogous to those identified in (54) (M, N) is monotonically decreasing with respect M and convex with respect N. This finctional behavior implies an
Remark 2.

---

optinal strategy: maximizing M subject to communication budget, while carefully calibrating N to the critical point where marginal returns diminish. Our ablation studies (Figure 5 validate these theoretical findings.
Proof Sketch. Our proof begins by recasting the original min-max optimization problem Hamiltonian
system (Appendix (.44). The convergence analysis hinges on three distinct types of approximate gradients:
delayed gradient (Lemma [ and Lemma 2), compression gradient (Lemma and estimated gradient (Lemma)) B) We establish the global convergence of the framework by proving that the loss function L(n, 4) 0) is L-smooth (Assumption[] []. Through a comprehensive analysis of the M loop, N loop, and outer loop dynamics, we prove the asymplotic convergence of model parameters (Theorem The complete convergence analysis of DecVFAL, with detailed proofs, is presented in Appendix
6 Experiments
We extensively evaluated DecVFAL against diverse baseline VFAL methods. Ablation studies were conducted to analyze component contributions. Detailed procedures are available in Appendix [C] and the code is accessible htttps://github. http:///hhuu..coo//rrollllaaaaae
6.1 Experiment Setups
Real-world VFL datasets are proprictary and not publicly accessible. Therefore, utilized three public datasets instead for our main experiments: MNIST B4 CIFAR-10 [I]. large scale dataset CIFAR- 100 B1, and Tiny-ImageNet [3] (Appendix(C.6)) These datasets were vertically partitioned among all participants, with each client retaining a portion of features for each sample, while the server exclusively held the labels. Detailed information about the dataset partitioning can be found in Section C.]
Datasets.
Baselines. We deploy the baseline algorithms and Dec in hybrid cascaded VFL framework, syn- chronous VFL-CZOFO (62) The implemented AT algorithms include PGD-r [49), and the accelerated AT methods: FreeAT-r (55), FreeLB-r [82), and YOPO-m-n (76). Additionally, we integrated data parallelism. model parallelism, and asynchronous mechanisms with PGD, resulting in DP-PGD, MP-PGD, and Asy-PGD, respectively. We also compare with FGSM-based accelerated AT methods, ATAS 77), FastAT
25
Adversarial attack. Following the threat model in VFL (Appendixx A.3), we employ various adversarial attack
methods including FGSM 12 PGD-r [9] and CW [4] We also simulate scenarios where malicious clients
cannot directly obtain gradients and implement black-box altacks: CERTIFY (CER) [. zero-order-based FGSM (ZO-FGSM) and PGD (ZO-PGD) [. Additionally, Considering the case the third-party adversary. we employ adversarial attacks that corrupt embeddings using different corrupted client selection methods: Thompson Sampling with Empirical Maximum Reward (E-TS) 16 and All Corruption Pattern (ALL).
Training procedures. We deployed a VFL setup with one server and two clients for the experiments. For the Lnd and the server employed single-layer perceptron,
MNIST dataset, each client used two-layer perceptron with the model partitioned into three modules and trained using batch size 32. On CIFAR- 10, while the server maintained a single-layer perceptron, each client utilized R 1esNet-18@ddvidee divided into two modules (first layer and remaining layers), creating a three-module structure across participants with hatch size 80. For CIFAR-100 we used ResNeXt-50 on client sides, similarly partitioned into three modules. Additional experiments with ResNet- 18 MNIST followed the same partitioning scheme CIFAR- 10. All models were trained convergence using Adam optimizer with fixed learning rate 0.0001 for fair comparison. Detailed parameter settings and hardware specifications for the training procedures are summarized in Appendix C.3 and Table [3
6.2 Evaluation on computational efficiency
For each dataset, we trained models to converge and plotted training and testing curves in Figures 3 and DccVFAL achieved better test accuracy than other baseline algorithms significantly less time on MNIST. Due to setting close parameters specify the number of full propagations [TT for CIFARIO. 10, DecVFAL. achieved convergence speed comparable to FreeAT and FreeL.B, while delivering better robustness, as shown in Table 2
Figure 3:
MNIST Figure 4: CIFAR-10
training-testing curves training-testing curves
$ FFor different model architectures, we conduct experiments with ResNet on MNIST (Appendix C.5 'Residual connections naturally align with dynamical systems theory [9 68), where entire residual blocks function as single dynamical units.
7

---

6.3 Evaluation on Robustness
For the MNIST dataset, VeAL demonstrates the most optimal trade-off between computational efficiency and model robustness. As shown in T DecVFAL achieves the best robust performance while requiring only 1/10 of the training time per epoch compared to standard PGD adversarial training. Similarly for CIFAR-I IO, the results in Table show that DecVFAL achieves comparable robust performance under most adversarial attacks while requiring only 1/3 of the training time per epoch compared PGD. These results consistently demonstrate the effectiveness VecL in balancing computational efficiency with model robustness.
Table 1: Results of MNIST Robust Training
Training Methods
Clean Accuracy
White- Box Adv.
Atk
Black-Box Adv. Atk
Third Adv.
Train Time
FGSM
PGD
CW
CFR
70-FGSM
70-PGD
AL..
E-TS
(s/epoch)
None
96.46
47.75
8.58
56.75
56.37
55.97
60.44
36.01
40.27
106.86
PGD
92.31
74.90
57.85
90.09
88.92
87.30
83.37
36.71
41.23
3484.57
FreeAT
92.68
67.29
41.33
85.11
84.13
83.51
80.18
19.01
20.82
853.64
FreeLB
93.77
57.18
18.76
85,30
82.47
82.30
79.73
65.33
71.11
3459.81
YOPO
96.13
86.36
73.52
92.49
91.63
91.17
88.06
79.81
84.84
629.43
DP-PGD
93.28
78.64
60.97
88.40
86.60
86.49
82.84
$1.72
56.68
3451.44
MP-PGD
93.11
75.23
48.98
78.82
76.65
76,28
76.19
48.11
54.67
3423.91
Asy-PGD
91.25
72.40
50.41
84.53
82.55
82.10
79.50
38.42
42.99
3724.47
NoLazy
90.17
72.58
55.53
89.76
85.32
77.03
50.96
26.66
26.05
317.78
DecL
96.30
91.62
77.68
92.84
91.91
92.13
89.21
92.20
94.53
355.16
Table 2: Results of CIFAR- 10 Robust Training
Training Methods
Clean Accuracy
White-Bot Adv.
Atk
Black- Box Adv. Atk
Third Adv.
Train Time
FGSM
PGD
CW
CER
70-FGSM
ZO-PGD
ALL
E-TS
(s/epoch)
None
83.93
53.32
55.42
62.59
50.39
52.38
55.58
76.06
78.93
70.03
PGD
78.00
59.08
68.47
76.73
70.00
70.32
70.56
69.54
72.67
296.35
FreeAT
80.09
63.63
61.93
77.01
68.99
70.99
71.85
71.44
74.86
252.11
FreeLB
81,58
52.09
54.91
63.70
53.91
56.92
59.17
76.30
78.70
301.43
YOPO
75.34
58.80
68.11
74.68
70.10
69.97
69.96
64.38
69.05
297.45
DP-PGD
75.47
59.37
68.24
74.56
69.79
69.74
70.04
66.19
69.42
331.93
MP-PGD
74.92
59.38
68.14
74.30
69.92
69.53
69.90
64.70
68.66
334.48
Asy-PGD
73.32
57.00
66.61
72.48
67.56
67.93
67.83
63.36
67.83
331.45
DecVFAL
81.83
63.69
68.59
74.72
71.31
71.05
72.07
74.93
77.75
98.99
6.4 Experiments on Real-World Datasets
To further validate the generality of our framework in practical federated appli-
Table 3: Results of robust training on real-world datasets
cations, we conducted experiments on
Methods
COVID-19
Credit Fruad
Speedup
two real-world datasets: COVID-19 Im-
Clean
PGD
AA
Clean
PGD
AA
vs PGD
age Data Collection [50] and Credit Card
Fraud Delection B These datasets
2.86
3.38
42.19
1.00x
widely used healthcare and financial anomaly detection scenarios, where data are naturally vertically partitioned across organizations. We simulated two clients, each holding disjoint feature subsets of the same samples. methods shared identical hyperparameters as in previous experiments. Table summarizzs sumeerammmriz the robustness and efficiency performance on both datasets, Across both settings. DecVFAL consistently delivers the best robustness-efficiennyy efffciincc off. On the COVID-19 dataset, achieves a 3.72 x speedup relative to standard PGD while maintaining comparable robust accuracy under multiple attacks. On the Credit dataset, DecL achieves 2.35x speedup with strong adversarial robustness, outperforming prior acceleration methods.
FreeAT
40.36
5.72
2.54x 11333x
YOPO
77.90
66.49
58.51
90.10
32.29
22.39
3.39 /2000x
DEAL
88.22
44.75
89.58
47.13
47.39
3.72x /2.35x
6.5 Experiments on large dataset
8

---

Table 4: Results of CIFAR-100 Robust Training
We further evaluate the robustness results of Vee FAL CIFAR-100, which is more difficult dataset with more classes, As shown in Table 4 DecL achieves the highest robust accuracy against all types of adversarial attacks, while reducing training time only 1/3 of standard PGD. Although FreeAT and FreeLB perform slightly better on clean data, they show significant performance degradation under ad- versarial attacks. This degradation occurs primarily because their simultaneous updates of model parame- ters and adversarial samples lead to overfitting, high- lighting the advantage of DecL
Training Motheds
Robust Accuracy (%)
Train Time
Clean
FGSM
PGD
20: PD-PGD
ETS
(s/spoch)
Nowe
53.55
16.89
9
27.07
32.51
123.65
PGD
51 31.99
39.46
79.15
45.40
27.82
1106
FreeAT
54,06
36.30
35.17
47.54
43.32
946.35
FreeLB
54.35
15.30
8.19
21.91
45.12
935.29
YOPO
51.91
37.44
36.00
46.02
31,65
587.70
DP-PGD
35.02
37.72
1096.77
MP-PGD
48.48
35.99
35.52
41.70
16.19
1150.38
Asy-PGD
51.19
39.83
39.36
45.91
25.00
1115.73
DecV
54.33
43.43
43.24
49.60
446.36
6.6 Comparison with FGSM-based Methods
We also compared with com- putationally efficient FGSM- based methods (FastAT and ATAS). Table 5 presents the results across different pertur- bation budgets. DccVFAL-6- 2 consistently achieves supe- rior robustness while requir- ing only 1/3 of PGD's training
Table 5: Comparison with FGSM-based methods
Method
Time (h)
E
4/255
8/255
12/255
PGD-10
AA
PGD-10
AA
PGD-10
AA
PGD-10
8.23
68.47
63.08
44.61
34.55
34.59
16.29
FasATT
1.45
63.61
57.69
40.41
30.62
30.58
14.20
ATAS
1.85
66.23
61.08
42.56
32.46
32.85
15,30
VELL--6
2.75
68.59
62.34
46.27
36.48
36.14
18.72
VEAL-3--
1.45
66.19
60.58
42.96
31.76
31.98
15.11
time. Notably, DecVFAL-3-3 matches FastAT's 's training efficiency while delivering better robustness, demon- strating DecVFAL balances computational efficiency and model robustness through its configurable parameters.
6.7 Ablation Study
Impact of the number of clients C. We conducted additional experiments on the MNIST dataset by varying the number of clients C' among 3. 5, and 7. DecVFAL consistently achieved superior robustness and enhanced computational efficiency across all client configurations compared baseline methods. Additionally, DecL and baselines against third party attacks in 7-client setting, including corruption pattern selection and malicious client scenarios (see Appendix (67]. VFAL. still demonstrated better robust performance.
tested
Table 6: Results for different number of clients C'
No. Clients
Training Methods
Clean Accuracy
White-Box Adv.
Atk
Black- Book-BBx.dd Adv. Atk
Third Adv.
Train Time (s/epoch)
PGD
FGSM
CW
CER
Z0-FGSM
Z0-PGD
ALL
3
PGD
98.05
64.56
82.83
96.02
96.46
93.98
94.72
89.56
1015.80
PGD
96.78
69.50
84.51
93.00
95.20
92.36
93.08
78.62
1145.63
7
PGD
96.18
63.86
79.96
90.96
93.30
90.37
94.12
69.36
1158.11
3
DecVFAL
98.67
80.82
89,50
97.28
97.90
96.03
96.97
93.83
88.29
DecVFAL
98.30
76.52
87.39
97.34
97.54
93.85
95.73
91.87
92.93
DecVFAL
76.57
87.17
83.90
96.21
93.23
90.80
81.21
94.83
Impact of the number of modules MM. To verify the impact of the number of modules MM.. we con- ducted additional experiments on the MNIST dataset. Fach client employed ResNet- model, which was partitioned into varying numbers of modules: 2, 3, 4. 5. and 6. All configurations were trained for an identical number of epochs. As indicated by Remark I increasing the number of modules leads larger errors in the gradient n, which in turn negatively im- pacts the algorithm's 's accuracy. This is demonstrated by the results shown in Table
Table 7: Results of the diverse number of modules
Robust Accuracy (%)
Split Positions for Modules
Clean
FGSM
PGD
[: 1: 18 19]
98.90
48.79
57.49
[: 11199 18 : 19)
98.71
45.88
56.55
[: 11:9:
13 18 19]
98.58
44,32
53.42
111550
11:5::9110
13
19]
98.69
47.09
45.49
1: ::15: 13:
98.22
40.63
Impact of lazy propagation. We evaluated the effectiveness of lazy propagation on the MNIST dataset using DecVTAL framework (M=5, N=10). By setting N=1, we implemented variant without lazy propagation, denoted as "Nolazy". As shown in Table T Nolazy only updates each sample 5 times to generate adversarial examples. While the decoupled parallel propagation still provides some training time advantage, the overall model performance cannot be guaranteed under this configuration.
Limitation of the setting of M and N. We conducted experiments on the MNIST dataset to explore the dependence on parameters M and N. Fillsteee the change in accuracy with fixed M = 3, 10, respectively, while varying N. The performance exhibits mon-monotoiic relationship with N, as analyzed in

---

Remark 2
Initially, performance improves as N increases, however. beyond certain threshold, it deteriorates significantly. This pattem emphasizes the critical importance of optimal N selection to maintain model accuracy.
Impact of split position. We conducted additional experiments on MNIST to evaluate the effect of different split positions. The server model still kept as single-layer perceptron, while each client utilized a ResNet- model that is split into two modules at various positions. The results in Table demonstrate that Dee VecV performs well across various split positions compared to PGD. However, as more layers are included in the bottom module duriig lazy sequential backpropagation, the computational load increases, leading to longer training time.
Table Results different split positions
Split Positions
Robust Accuracy (%)
Train Time
[ Mi
Ma
Ma]
Clean
FGSM
PGD
(slopoch)
E1:
18
19]
98.90
48.79
57.49
107.54
M-10 10, varying N
M- varying N
[ [::
18:
19
96.77
43.03
42.98
226.76
M-3, varying
[ 9:
19
96.75
41.33
49.77
318.12
Me
13
18
19]
98.83
39.73
43.46
431.14
17 18:18:10
19)
98.43
36.36
45.88
538.65
Figure 5: Result of fixed M and varying
PGD
98.48
32,53
41.93
575.45
Evaluation under attacks involving corruption pattern selection. To further assess our framework's resilience more complex attack scenarios, we conducted experiments on the MNIST dataset using seven clients. Specifically, we evaluated VecLFAL and baseline methods against attacks involving corruption pattern selection. In this setup, adversaries could selectively corrupt client data or communications. The server model remained a single-layer perceptron. We implemented various corruption patterns, including E-TS, RC, and FC. As shown in Table the results demonstrated that even under these challenging conditions, DeAL maintained superior performance compared to baseline methods.
Table 9: Results of evaluation under attacks with various corruption patterns.
Corrupted clients: 1/7
Training
White-Bos Box Adv. Atk
Black- Box Adv. Atk
Third Adversary Atk
Methods
PGD
FGSM
CW
CER
Z0-FGSM
Z0-PGD
E-TS
PC
RC
PGD
92,238
94.01
93.85
94.091
94.03
93.399
88.842
88.922
88.982
DecVFAL
95.613
96.575
96,795
96.605
96,585
96.044
93.048
93.87
93.219
Corrupted clients: 3/7
Training
White Box Adv. Atk
Black Box Adv. Adk
Third Advensary Ark
Metheds
PGD
FGSM
CW
CER
ZO-FGSM
ZO-PGD
E-TS
RC
PGD
79.888
87.099
93.359
94. 94.101
92.819
92.758
77.364
78.105
77.754
DecVFAL
86.569
92,949
96.044
96.404
95.543
94.922
84,816
85,577
84.685
Cormpted clients: 5/7
Training
White-Box Adv.
Atk
Black-Box Adv. Atk
Third Adversary Atk
Methods
PGD
FGSM
CW
CER
ZO-FGSM
ZO-PGD
E-TS
RC
PGD
64.724
80.689
91.526
93.279
90.935
91.587
69.03
68.53
69.111
DecVFAL
78.235
87.31
91.987
96.044
93.049
94.121
75.972
76.062
76.322
7 Conclusions
This paper presented VFAL, framework that significantly accelerates VFAL while maintaining robustness. VeCL incorporates a dual-level decoupled mechanism to enable lazy sequential decoupled parallel backpropagation for adversarial example generation. This approach maintains the same theoretical convergence rate O(1/VK) as regular VFLs while providing acceleration each iteration, resulting in significant empirical improvements with 3-10 fold speedup across various datasets, Comprehensive experiments demonstrate DecVFAL's effectiveness across various neural architectures and VFL. configurratis.
Acknowledgement
Dr. Yi Chang was supported by the National Key R&D Program of China under Grant No. 2023YFF0005400, and the National Natural Science Foundation of China through grants No. U2341229 and No. 62076138.
10

---

References
[1] Bhagoji, A.N., Chakraborty, S., Mittal, P., Calo, S.: Analyzing federated learning through adversarial lens. In: International conference on machine learning. pp. 634-643. PMLR (2019)
[2] Cai, D., Fan, T. Kang, Y., Fan, I.., Xu, M., Wang. S., Yang. Q. Accelerating vertical federated learning.
IEEE Transactions on Big Data p. 1-10 (2024). https://do..org//00 1ttp://00.11///tttaaaaaaaa 2022.3 01022.192299, http:
//dx.doi. org/0100000. 1109/TEDATA...... 2022. 3192898
[3] Carcillo, F., Le Borgne, Y.A., Caclen, O., Kessaci, Y., Oblé, F., Bontempi, G.: Combining unsupervised and supervised learning in credit card fraud detection. Information sciences 557, 317-331 (2021)
[4] Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 ieee symposium security and privacy (sp). pp. 39-57. Ieee (2017)
[5] Castiglia, TJJ, Das, A., Wang, S., Patterson, S.: Compressed-vflt Communication--fficicenn learning with vertically partitioned data. In: International Conference on Machine Learning. pp. 2738-2766. PMLR (2022)
[6] Cha, D., Sung, M., Park, Y.R., et al.: Implementing vertical federated learning using autoencoders: Practical application, generalizability. and utility study. JMIR medical informatics (6)) e26598 (2021)
[7] Chen, C., Zhou, J., Wang, L., Wu, X., Fang, W., Tan, J., Wang. L., Liu, A.X., Wang, H., Hong, C.: When homomorphic eneryption marries secret sharing: Secure large-scale sparse logistic regression and applications in risk control. In: Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery Data Mining- pp. 2652-2662 (2021)
[8] Chen, P.Y., Zhang, HI, Sharma, Y., Yi. J., Hsieh, C.J.: Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In: Proceedings of the 10th ACM workshop on artificial intelligence and security. pp. 15-26 (2017)
[9] Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. Advances in neural information processing systems (2018)
[10] Chen, W., Ma. G., Fan, T. Kang, Y., Xu, Q. Yang. Q.: Secureboostt: A high performance gradient boosting tree framework for large scale vertical federated learning. arXiv preprint arXiv:2110.10921 (2021)
[11] Cohen, J., Rosenfeld, E., Kolter, Z.: Certified adversarial robustness via randomized smoothing, In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 1310-1320. PMLR (09-15 Jun 2019), https: httppe//prooeedingg .mlr, ppwps/v97/c///// httpv///oooo/ press//17/ccheen19. html
[12] Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with ensemble of diverse parameter- free attacks. In: International conference on machine Icarning. pp. 2206-221111 2216. PMLR (2020)
[13] Cui, J., Chen, C., Lyu, I.., Yang, C.,, Li, W.: Exploiting data sparsity in secure cross-platform social recommendation. Advances in Neural Information Processing Systems 34, 10524-10534 (2021)
[14] Deng, Y., Kamani, M.M., Mahdavi, M.: Distributionally robust federated averaging. Advances in neural information processing systems 33, 15111 15122 (2020)
[15] Devlin, J., Chang, M.W., Lee. K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1.Xi:18810.04800 (2019)
[16] Duanyi. Y.. Li. S., Ye. X., Liu, J.: Constructing adversarial examples for vertical federated learning: Optimal client corruption through multi-armed bandit. In: The Twelfth International Conference on Learning Representations (2023)
[17] Fu, C., Zhang, X., Ji, S., Chen, J., Wu, J., Guo, S., Zhou, J., Liu, A.X., Wang. T: Label inference attacks against vertical federated learning. In: 31st USENIX security symposium (USENIX Security 22). pp- 1397-1414 (2022)
[18] Gao, X., Jiang, B., Zhang. S.: On the inlommatioonadaative variants of the admm: an iteration complexity perspective. Journal of Scientific Computing 76. 327-363 363 (2018)
[19] Ge. N., Li. G., Zhang, L., Liu, Y.: Failure prediction in production line based on federated learning: an empirical study. Journal of Intelligent Manufacturing 33(8). 2277-2294 (2022)
[20] Gu, F., Askari, A.. El Ghaoui, L.: Fenchel lifted networks: A lagrange relaxation of neural network training.
In: International Conference on Artificial Intelligence and Statistics. pp. 3362-33623333 13711 PMLR (2020)

---

[21] Haddadpour. F,, Mahdavi, M.: On the convergence of local descent methods in federated learning. arXiv preprint 1rXiv:1910.1442) (2019)
[22] Huang. F., Gao, S., Pei, J., Huang, H.: Nonconvex zeroth stochastic admm methods with lower function query complexity. arXiv preprint arXiv:1907.1346 (2019)
[23] Huang, F., Gao, S., Pci, J., Huang, H.: Accelerated zeroth-order momentum methods from mini minimax optimization. arXiv preprint arXiv:2008.00 0000 3 (2020)
[24] Huang, W. Ye, M., Shi, Z., Wan, G., Li. H., Du B., Yang. Q:: Federated learning for generalization, robustness, fairness: A survey and benchmark. IEFE Transactions on Pattern Analysis and Machine Intelligence (2024)
[25] Huang, Z., Fan, Y., Liu, C., Zhang, W. Zhang. Y., Salzmann, M., Süsstrunnk, S., Wang, J.: Fast adversarial training with adaptive step size. IEEE Transactions on Image Processing 32. 6102-61 14 (2023)
[26] Huo, Z.. Gu, B., Huang, H.: Training neural networks using features replay. Advances in Neural Information Processing Systems 31 (2018)
[27] Huo, Z., B., Huang. H., al.: Decoupled parallel backpropagation with convergence guarantee. In: International Conference on Machine Learning. pp. 2098-22 2106. PMLR (2018)
[28] Jaderberg. M., Czarnecki, W.M., Osindero, S., Vinyals, O., Graves, A., Silver, D., Kavukcuoglu. K.: Decoupled neural interfaces using synthetic gradients. In: International conference on machine learning. pp. 1627-1635, PMLR (2017)
[29] Jin, X., Chen, P.Y., Hsu, C.Y., Yu, C.M., Chen, T:: Cafe: Catastrophic data leakage in vertical foderated learning. Advances in Neural Information Processing Systems 34, 994-1006 (2021)
[30] Khan, A., Thij. M., Wilbik, A.: Communication-effcceenn vertical federated learning. Algorithms 15(8). 273 (2022)
[31] Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech, rep., University of Toronto (2009)
[32] Kurakin, A., Goodfellow, L., Bengio, S.: Adversarial machine learning at scale. arXiv preprint arXiv:661.001336 (2016)
[33] Lc, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), 3 (2015)
[34] LeCun, Y., Bottou, I.., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Proceedings of the IEEE. vol. 86, pp. 2278-2324. IEEE (1998)
[35] LeCun, Y., Touresky, D., Hinton, G., Sejnowski, T.: A theoretical framework for prepppropagttion In: Proceedings of the 1988 connectionist models summer school. vol. 1, pp. 21-28 (1988)
[36] Li, J., Fang, C., Lin, Z.: Lifted proximal operator machines. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 4181 44888 (2019)
[37] Li, M., Chen, Y., Wang. Y., Pan, Y:: Efficient asynchronous vertical federated learning via gradient predic- tion and double-end sparse compression. In: 2020 16th international conference on control, automation, roboties and vision (ICARCV), pp. 291-296 IEEE (2020)
[38] Li. Q. Chen, L., Tai, C., Weinan, E.: Maximum principle based algorithms for deep learning. Journal of Machine Learning Research 18(165), 1-29 (2018)
[39] Li, Q., Hao, S.: An optimal control approach to deep learning and applications discrete-weight neural networks. In: International Conference on Machine Learning. pp. 2985-2994. PMLR (2018)
[40] Li, X., Song, Z., Yang, J.: Federated adversarial learning: framework with convergence analysis. In: International Conference on Machine Learning. pp. 19932-19959. PMLR (2023)
[41] Liu, J., Xie, C., Kemthapadi, K., Koyejo, S., Li. B.: Rvfr: Robust vertical federated learning via feature subspace recovery. In: NeurIPS Workshop New Frontiers in Federated Learning: Privacy, Fairness, Robustness, Personalization and Data Ownership (2021)
[42] Liu, S., Kailkhura, B., Chen, P.Y., Ting, P., Chang. S., Amini, L.: Zeroth-ordes stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems 31 (2018)
12

---

[43] Liu, Y.. Kang, Y.. Xing, C., Chen, T., Yang. Q.: A secure federated transfer learning framework. IEEE Intelligent Systems 35(4), 70-82 (2020)
[44] Liu, Y., Kang, Y. Zou, T. Pu, Y., He, Y., Ye, X., Ouyang. Y., Zhang. Y.Q. Yang, Q.: Vertical federated learning: Concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering (2024)
[45] Liu, Y., Zou, T., Kang. Y., Liu, W., He, Y., Yi, Z., Yang, Q.: Batch label inference and replacement attacks in black-boxed vertical federated learning. arXiv preprint 11XX:::2122.054440 (2021)
[46] Long, G., Tan, Y., Jiang, J.. Zhang, C.: Federated learning for open banking. In: Federated learning: privacy and incentive, pp. 254. Springer (2020)
[47] Lundberg, S.M., Lee, S.I.: A unified approach interpreting model predictions. Advances in neural information processing systems 30 (2017)
[48] Luo, X., Wu, Y., Xiao, X., Ooi, B.C.: Feature inference attack on model predictions in vertical federated learning. In: 2021 IEEE 37th International Conference on Data Engineering (ICDE). pp. 181-192. IEEE (2021)
[49] Madry, A., Makelov, A., Schmidt, L.., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint XXv:11006.06088 (2017)
[50] Maguolo, G., Nanni, L.: A critic evaluation of methods covid- 19 automatic detection from x-ray images. Information fusion 76, 1-7 (2021)
[51] McMahan, H.B., Moore, E., Ramage, D., y Arcas, B.A.: Federated learning of deep networks using model averaging. arXiv preprint arXiv: 160.00260 2, 2 (2016)
[52] Pang, Q., Yuan, Y., Wang, S., Zheng, W.: Adi: Adversarial dominating inputs in vertical federated learning systems. Xiv preprint wtxixx2221.027777 (2022)
[53] Qiu, P., Zhang, X., Ji, S., Du, T., Pu, Y., Zhou, J., Wang. T:: Your labels are selling you out: Relation leaks in vertical federated learning. IEEE Transactions on Dependable and Secure Computing (2022)
[54] Seidman, J.H., Fazlyab, M., Preciado, V.M., Pappas, G.J.: Robust deep learning as optimal control: Insights and convergence guarantees. Proceedings of Machine Learning Research (2020)
[55] Shafahi, A., Najibi, M., Ghiasi, M.A., Xu, Z., Dickerson, J., Studer, C., Davis, L.S., Taylor, G., Goldstein, T.: Adversarial training for free! Advances in neural information processing systems 32 (2019)
[56] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(52), 1-11 (2017)
[57] Sokolov, A., Hitschler. J., Ohta, M., Riezler, S.: Sparse stochastic zeroth- order optimization with an application to bandit structured prediction. arXiv preprint HtXiv:1806.0445 (2018)
[58] Song, Y., Xie, Y., Zhang, H., Liang, Y., Ye, X., Yang, A., Ouyang, Y.: Federated learning application on telccommunication joint healthcare recommendation. In: 2021 IEEE 21 International Conference on Communication Technology (ICCT). pp. 1443 1448. IEEE (2021)
[59] Taylor, G., Burmeister, R., Xu, Z., Singh, B., Patel, A.. Goldstein, T.: Training neural networks without
gradients: A scalable admm approach. In: International conference machine learning, pp. 2731-2731.
PMLR (2016)
[60] Teimoori, Z., Yassine, A., Hossain, M.S.: A secure cloudlet-based charging station recommendation for electric vehicles empowered by federated learning. IEEE Transactions on Industrial Informatics 18(9), 6464 6473 (2022)
[61] Tramèr, Bonch, D., Kurakin, A., Goodfellow, I., Papernot, N., McDaniel, Ensemble adversarial training: Attacks and defenses. In: 6th International Conference on Learning Representations, ICLR 2018 C018-Conference Track Proceedings (2018)
[62] Wang, G., Gu, B., Zhang, Q. X., Wang, B., Ling, C.X.: A unified solution for privacy and communi- cation efficiency in vertical federated learning. Advances in Neural Information Processing Systems 36 (2024)
[63] Wang, G., Zhang, Q. Xiang, L., Wang. B.. Gu, B., Ling, C.: Secure and fast asynchronous vertical federated learning via cascaded hybrid optimization. arXiv preprint 160777 (2023)
13

---

[64] Wang. G.: Interpret federated learning with shapley values. arXiv preprint arXiv: (2019)
[65] Wang, Y., Ma, X., Bailey, J. Yi, Zhou, B., Gu, Q.: On the convergence and robustness of advensarial training. arXiv preprint arXiv:2112.0830 (2021)
[66] Wei, K., Li, J., Ma, C., Ding, M., Wei, S., Wu, F,, Chen, G., Ranbaduge, T.: Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309 (2022)
[67] Weinan, E., Han, J., Li, Q.: A mean-field optimal control formulation of deep learning. arXiv preprint 1rXiv:1807.00 80.0000033000000000 (2018)
[68] Weinan, E.: A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics 1(5), 1-11 (2017)
[69] Weng, H., Zhang, J., Xue, F, Wei, T., Ji, S., Zong, Z.: Privacy leakage of real-world vertical federated learning. arXiv preprint arXiv:2011.088 1.09290 (2020)
[70] Wong, E., Rice, L.,, Kolter, J.Z.: Fast better than free: Revisiting adversarial training. arXiv preprint #rXivv2001.0399 (2020)
[71] Wu, Z., Li, Q. He, B.: Practical vertical federated learning with unsupervised representation learning. IEEE Transactions on Big Data (2022)
[72] Xie, L.,, Liu, J., Lu, S., Chang, T.H., Shi, Q: An efficient learning framework for federated xgboost using secret sharing and distributed optimization. ACM Transactions on Intelligent Systems and Technology (TIST) 13(5), 1-28 (2022)
[73] Xu, W., Fan, H., Li. K., Yang, K.: Efficient batch homomorphic encryption for vertically federated xgboost. arXiv preprint 1xXiv:2112.0420 (2021)
[74] Yang, S., Ren, B., Zhou, X., Liu, L.: Parallel distributed logistic regression for vertical federated learning without third-party coordinator. arXiv preprint arXiv: 1. 09824 (2019)
[75] Yuan, H., Ma, C., Zhao. Z., Xu, X., Wang, Z.: A approach based on personal data cloud and federated learning. In: 2022 IEEE International Conference on Web Services (ICWS). pp. 322-330. IERE (2022)
oriented service recommendation
[76] Zhang, D., Zhang. Lu, Y., Zhu, Z., Dong, B.: You only propagate once: Accelerating adversarial training via maximal principle. Advances neural information processing systems (2019)
177| Zhang, G., Lu, S., Zhang, Y., Chen, X., Chen, P.Y., Fan, Q., Martic, L., Horesh, L.,, Hong, M., Liu, S.: Distributed adversarial training to robustify deep neural networks at scale. In: Uncertainty artificial intelligence.
PMLR (2022)
[78] Zhang, J., Guo, S., Qu, Z., Zeng, D., Wang, H., Liu, Q., Zomaya, A. A.Y.: Adaptive vertical federated learning on unhalanced features. IEEE Transactions on Parallel and Distributed Systems 33(12), 4006-4018 (2022)
[79] Zhang, Q.. Gu, B., Dang, Z., Deng, C., Huang, H.: Desirable companion for vertical federated learning: New zeroth-order gradient based algorithm. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. pp. 2598-2607 (2021)
[80] Zhao, J., Zhu, X., Wang, J., Xiao, J.: Efficient client contribution evaluation for horizontal federated learn- ing. In: ICASSP 2021-02222 -2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 3060-3064. IEEE (2021)
[81] Zhao, X., Zhang, H., Huo, 7.,, Gu, B.: Accelerated on-device forward neural network training with module-wise descending asynchronism. Advances in Neural Information Processing Systems 36 (2024)
[82] Zhu, C., Cheng. Y., Gan, Z., Sun, S., Goldstein, T., Liu, J.: Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint 1rXiv:1909.11766 (2019)
[83] Zizzo, G., Rawat, A., Sinn, M., Buesser, B.: Fat: Federated adversarial training. arXiv preprint arXiv:2012.01791 (2020)
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly articulate our three key contributions: (1) proposing DecVFAL framework with dual-level decoupling mechanism, (2) establishing the first convergence analysis for VFAL, and (3) demonstrating empirical results showing 3-10x speedup while maintaining robustness. These claims are supported by theoretical analysis in Section 5 and experimental results in Scction 6.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 5 (Convergence Analysis), discuss the limitations through Remark 1, which reveals direct trade-off between model modularity and performance. also discuss in Appendix B.2 the assumptions that bound our theoretical guarantees. Additionally, Section 6.6 (Ablation Study) empirically shows the limitations of our method when increasing the number of modules.
Theory assumptions and proofs
Question: For each theoretical result. does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our theoretical results presented in Section 5 with assumptions clearly stated. Lemmas 1-7 and Theorem presented with complete pronfs Appendix All assumptions (B.2) are formally defined and referenced in statements theorems, and proofs follow a step-hy-step approach with proper cross-referencing.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: Section 6 and Appendix C provide comprehensive details experimental setups, including dataset descriptions (C.1), adversarial attack configurations (C.2), hyperparameters (C.3), hardware specifications (C.4), and model architectures. Tables 9-11 specific hyperparameters used for training and evaluation, enabling reproducibility.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
code is available at the anonymous repository http://lanomymoou..oppe... weweeeeeeeneee/eeeeLLA....
Our
mentioned in Section 6. We use pub- licly accessible datasets (MNIST, CIFAR-I0, CIFAR-100, Imn-ImmaeNNe) and provide detailed instructions on the implementation of DecVFAL both the main paper and supplementary matcrials.
6. Experimental setting/details
Question: Does the paper specify all training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Section I and Appendix C provide comprehensive experiment details. We specify model architectures, training procedures, optimizer configurations (Adam with learning rate 0.0001). data partitioning methods, and evaluation metrics. Tables 9 and 10 in Appendix C list specific hyperparameters adversarial training and attacks.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
15

---

Justification: While report comprehensive results across multiple datasets and attack methods in Tables 1-6, do not explicitly include error bars confidence intervals. Our experiments were with fixed random seeds reproducibility, and the consistent performance improvements across multiple datasets and settings support our claims.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Table 11 in Appendix C.4 provides detailed hardware specifications for each experiment, including CPU and GPU configurations. Training time per epoch reported for each method in Tables 1-3, providing clear metrics on computational efficiency.
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://///errrpps.ccccblbll///// cosu
Answer: [Yes]
Justification: Our research focuses on improving computational efficiency and robustness of privacy- preserving machine learning frameworks (VFL), which aligns with the ethical principles of privacy protection and enhanced security. We use public benehmark datasets and do not collect any personally identifiable information.
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]
The paper primarily focuses on the technical aspects of accelerating robust VFL. While we acknowledge that our work has positive implications for privacy-preserne and secure machine learning deployments, we did not include dedicated discussion of broader socictal impacts. Future versions will include this discussion.
Justification:
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focuses on efficiency improvements for vertical federated learning and does not release high-risk models datasets. We use standard benchmark datasets (MNIST, CIFAR- 10, CIFAR- 100. Siny-ImageNee and method aims enhance security against adversarial attacks rather than create risks.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code. data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the original papers for all datasets (LeCun et al. for MNIST, Krizhevsky for CIFAR, Le & Yang for Any-ImageNet and adversarial attack methods used in our experiments.
All datasets used publicly available for research purposes and use them within their intended use.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our anonymous code repository
includes documentation for the DecVFAL framework and implementation details. Our framework builds upon established VFL architectures with novel decoupling mechanisms, and all components documented in both the paper and repository.
14. Crowdsourcing and research with human subjects
16

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our research not involve crowdsoureing or human subjects. All experiments are computational, using public benchmark datasets and evaluating algorithmic performance.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our research does not involve human subjects require IRB approval. Our work focuses algorithmic improvements to vertical federated learning systems using public benchmark datasets.
16.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs it important, original, non-standard component the core methods this research? Note if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research does not use LLMss component of our methods. Our work focuses on adversarial robustness in vertical federated learning with image classification datasets, and does involve language models any capacity.
17

---

Appendal Contents
Backgrooae
Vertical Federated Learning
19
Vertical Federated Adversarial Learning
A
Threat Model
Adversarial Training as a Dynamical System
20
Zeroth Order Optimization
Communicatn Compression
Convergence Analysis
B
Notations
Assumptions
Proposition
Definition
B.
Lemma
8.6
Theorem
Experiment Details
Dataset Details
Adversarial Attack
Eyyerparamett...
Environment
Performance across Various NN Architeccture.
Experiments on Tiny Iww.mmaaeee
18

---

A Background
A.1 Vertical Federated Learning
VFL encompasses range of architoctural designs tailored for collaborative machine learning across multiple parties. These architectures, distinguished by data and parameter distribution. as well as the trainability of the server model, include Aggregated Vertical Federated Learning (aggV FL) 17 45, where client parties contribute intermediate results aggregated through a non-trainable function in the server party; Aggregated Vertical Federated Learning with Central Features (aggV F L.), similar agg but incorporating its own features; Split Vertical Federated Learning (splitV 17 41 featuring a trainable server model pro- cesses intermediate results from passive parties: and Split Vertical Federated Learning without Local Features (splitV F Lo). where the server party doesn't It provide any features to the model but relies solely on intermediate results from client parties.
Because VFL is collaboration system that requires parties exchange gradient model level information, it has been of great research interest study communication efficiency, and data privacy protection. Various strategies are adopted to heighten communication efficiency, typically involving reducing the cost coordination and compressing the data transmitted between parties, such as multiple client updates (78), asynchromous coordination 37. one-shot communication [7]]. and data compression [S] [37] In terms of data privacy protection, VFL relies on culting-edge technologies like Homomorphic Encryption (HE) (74,, Multi-Party Computation (MPC) and Differential Privacy (DP) 62 to preserve data privacy.
A.2 Vertical Federated Adversarial Learning
Emerging research has investigated the distinct challenges posed by adversarial attacks in the context VFL 24 Due to the distributed nature, VFL struggles to ensure client trustworthiness and thus renders highly susceptible to adversarial perturbations, underscoring the pressing need for enhanced VFL model robustness (24). this is particularly evident in neural network models. Prior works have proposed that adversaries (third-pant -aritd client party) can generate adversarial samples by introducing manipulated perturbations to raw data or embeddings in the corrupted clients, aiming to mislead the inference VPI. models [48] 69 [3 17]. However, existing VFI. defense mechanisms hased on cryptographic 45] and non-eryptographic 41 only concentrate on mitigating inference attacks and backdoor attacks while neglecting adversarial attacks.
A.3 Threat Model
In the context of VFL, we focus untargeted adversarial attacks, constructed during the inference phase. The adversary's 's objective corrupt samples whose original prediction is yuu causing the server model to output 9 * yu. We categorize these adversarial attacks into two primary scenarios:
Malicious (colluding) clients. In this scenario, we consider the presence of malicious (colluding) clients acting as adversary. During the attack, all malicious clients (one or more) collaboratively and simultancously generate the adversarial feature partition. The attacks are further classified based on the level of knowledge these clients possess:
White-box adversarial attack. Under relaxed protocol, clients have access to the server model f. and the output of all clients Fı., This protocol could occur when the client needs to make interpretable decisions based on the server model's parameters [48 64 47 This implies the malicious clients have the necessary information calculate the partial gradient to the features.
Black-box adversarial attack. Under basic VFL. protocol, all participants keep their private data (e.g., labels and features), as well as the server model f. and client models (f()) )o local during inference. Clients only receive the final prediction results and cannot directly obtain the gradient, thus necessitating the use of black-box methods to approximate it.
Third party adversary. We also consider an adversary third party in VFL inference, who can access, replay, and manipulate messages on the communication channel between endpoints, where embeddings and predictions are exchanged. Third-party adversaries usually cannot achieve access to top model parameters, thus this scenario generally falls under the black- box attack category. Due to resource constraints, previous work assumed that the adversary can corrupt at most C. < C' clients [16].
19

---

A.4 Adversarial Training as Dynamical System
With the link between optimal control and deep learning [39], research recast neural networks as dynamical systems and formulated the robust optimization problem an optimal control problem [34
T-1
min By
max 75
Σ L(TTA M6)
Re O.)
(A.1)
subject to
1. fa Oel, i= 1.,,..,, S, t= 1- 11 fo + ni, Oo): i=
where 24 E IR: represents the states (i.c., the input the f-th layer), f Rd x e + Rd the state transition map, @' are the trainable control parameters, θ denotes the concatenation of (0) Jusisr--,, and the initial conditions Maximum principle, proved in (76). we define the Hamiltonians: Ho(x, p, 8, 7) = fo(x + n, 0) (2 0) and H, (r, p. 0) = p" fa( Rs(x, 0). then there exists optimal costate trajectory pt, satisfied:
provided by the inputs to the network, 20,;. According to the two-player Pontryagin
: ( 03)
zg = Io +
(A.2)
p: V. H. (z Pit1, 0':)) pr VV y)
(A.3)
where 0* : 0 . is the solution of the problem
A.1
Due to the compositional structure, feed-forward deep neural networks can be viewed as dynamical systems. This approach has been recently explored in several papers, which leverage this interpretation propose new
training algorithms [68 38 67 76].
According to equation A. the two-player Pontryagin Maximum principle, proved in 76). gives necessary
conditions for an optimal setting of the parameters 0", perturbations ,....., 775 and corresponding trajectories {xi,e). Define the Hamiltonians
H.(x,p,0) : f((2, 8) (x( 0). t T-
(A.4)
Ho(æ, p. 8,n) == fo(r + n, 0) Ro(x, 0)
The two-player maximum principle says in this case that 4, ft, and RS are twice continuously diffcrentiable, with respect to æ, uniformly bounded in æ and t along with their partial derivatives, and the image sets [fe 0)10 [R")) } and (R, 0) E R } are convex for all and i, then there exists optimal costate trajectory pt such that the following dynamics satisfied
11 (a (,), xi.- V HoHa,,, Pi,,, 0i.n" n.)
(A.5)
= V.Hi 07), Pr.: = - V. V
and the following Hamiltonian condition for all 0g R. and E X
Helsi.s. Pi 0c)
He Helci..Pi1.s. t = 1'...,1-11
(A.6)
Σ Ho pi+ dc, m S
Ho( P4 0
Σ
These necessary optimality conditions can be used to design an iterative algorithm of the following form. For each data point S1.
Compute state costate trajectories {I.,e) and {pt,) from the dynamics, keeping and m fixed:
x 21, = V (x 1 0c) (n)
(n) V H, (()) Op.
1
2. PL V.Hi(s!? P... 01) (/) (z (m) v1)
3. Minimize the Hamiltonian Ho, 1, 0g. 71) with respect
4. Maximize the of Hamiltonians D He (Ire, i pt + 1, with respect to 6, for all
20

---

As was noticed as early as [35]. it can be seen from the chain rule that the backward costate dynamics are equivalent to backpropagation through the network. With this interpretation, the gradient of the total loss for the i-th data point with respect to the adversary 77 can he written V (200 n, fo) Pi. (n) For a fixed value
of loo performing gradient descent on Ho find worst-case adversarial perturbation can be expressed as the following updates, where a > O a step size:
(++1) "i qi
Foo 3 0o) Pi.
(A.7)
An important observation made in 76 is that the adversary is only present in the first layer Hamiltonian condition and this function can be minimized by computing gradients only with respect to the first layer of the network. More explicitly, instead using Pi,1. as in the updates above, we could instead use Po. and the updates
qi
(++1)
aV + ",,,, 0o) P6.1
(A.8)
This removes the need to do a full backpropagatin to recompute the costate Př, (v) cost of now being an approximate gradient.
for every update of 1); (6) at the
A.5 Zeroth Order Optimization
have been developed to effectively solve many ML problems for which obtaining explicit gradient expressions is difficult or infeasible. Such problems include structure prediction tasks, where explicit gradients are challenging to derive [57]. as well as bandit and black-box learning problems [56] 42, where obtaining explicit gradients is feasible. Specifically, ZOO relies solely function values for optimization, eschewing the need for explicit gradients.
ZOO methods 23 22
Formally, given a function f(z) with input z, the gradient (x(r) can be estimated using Z00. One common approach is to sample random perturbations u within the domain of f and evaluate the function shifts. The ZO gradient estimator V fx= is given by:
Vs(²)
1 Σ [f(z pa,) - f()) " 9
(A.9)
where ja serves as a scaling factor for the random perturbation, while uy represents the j-th random perturbation sampled from a distribution p across the domain of f. The parameter denotes the number of random samples employed for estimation. Normalizing the perturbation by " ensures the estimator's 's unbiasedness. The expectation of the Zeroth Order (ZO) gradient estimator yields an unbiased estimate of the true gradient, expressed as [=)] = fz()), provided that the samples uy are drawn from distribution with mean of zem..
The application of 7000 to VFL has been discussed, highlighting its specific properties such as model applicahility 79, privacy security concerns 442, and considerations regarding communication cost and computational efficiency [62].
A.6
Communication Compression
Compression is pivotal technique in VFL that aims to mitigate communication overhead by reducing the volume of data transmitted among participating parties. In the context of neural network-based VFI. algorithms, high-dimensional input vectors are inherently mapped onto lower-dimensional representations, which serve natural compression purpose. However, further enhance communication efficiency, specialized dimensionality reduction techniques are often integrated. Several VFL frameworks have been proposed to incorporate compres- sion techniques: AVFL (2) leverages PCA to compress the data before transmission, reducing the communication load. CE-VFL 30) employs both PCA and autoencoders to learn latent representations from raw data, which are then used for model training. SecureBoost+ [10] and eHJE-SecureBoost [73] encode encrypted gradients into a compact form, minimizing the number cryptographic operations and the data transmission size. C- C-L 5 introduces an arbitrary compression scheme to VFL, offering a thcorctical analysis of how compression parameters impact the overall system efficiency.
Compression techniques play a critical role in VFL by enabling more efficient data transmission without compromising the integrity of the learning process. The selection of an appropriate compression method is contingent upon the specific requirements of the VFL scenario, including the sensitivity of the data, the computational resources available, and the desired balance between communication efficiency and model performance.
21

---

B Convergence analysis
B.1 Notations
Notations
Delmitions
Network
The number samples
Neural octwork model
Model Parameters
Tnput sample and corresponding label
The mini-batch with size
Expectation
A (1.
Iteration index for purameter updating
Vertical Fedorated Learning
The number of clicnts
Jan Soanf...- (i)
Client models
(9 (egernee
Client model parameters
model
Server model parumeters
C.
Loss function
= Un Liss Ree
The complete VFL model
00
Leaming rate for server model pammeters
0
Leaming rate
parameters
Adversarial Training
Adversarial Loss Function
Ga(n.o).9)
B Deen A:(h,
$ A.
(m), e4, 0:) argmax, A.(g, s).
Adversarial perturbation
Projection operator
Learning
adversarial sample
lieration index for adversarial sample updates
(ZeAth SeAPh moyy
The sample with features from all clients
(u.m Rure Buess
The adversurial perturbation for sample from all clients
Optimal Control Formulation of Deep Leaminn
N1
Hamiltonian function for layer
= PuccnPP.cci.--PPeee
Costates at layer
Number layers neural network
0.1....T-11
Layer indices
1r
State transition map for layer
States at layer
6'
Trainable parameters for layer
Decoupled parallel Buckpropugution
Mx
numher
layers
The number of client model's layens
f-
(/. fa.....a........
The VFI, model from layer-wice wise view
== (2..,,
Client model parameters from layer-wise view
22.
The output of clicuts
Client model network excluding first layes/module
Lazy Sequential Backpropagation
M
Number of iterations for propagations
N
Number of iterations for propagations in bottom mod-
Ra
Regularizer for layer
Fihs
The VFL model excluding the first layer/module
The stateloutput of sample at layer iteration
Pia
The costate/gradient of sample layer
Zoteth Order Gradient Estimation
Smoothing parameter
Random vector
Query budget for gradient estimation
Loss difference
Estimation Gradient from ZOO
Compressor
CC V.4g, 0.
Compressor compressing information bits
Compression Gradicut
Table 10: Table of Notations
22

---

B.2 Assumptions
Assumption 1. Lipschitz Gradient: There exists constant K > 0 such that for all t E I...... le, T. the functions ft. fo. and Ru are K- K-Lipschitz uniformly in and th For all each sample i E L 1...... the function V Ac v. 0). (n) v. 0). (nA.(n, v, 0) satisfy the following Lipschitz conditions:
(I v', v,,A((n, v. 0)1 L 1|
1/1.
(B.1)
VIV (n, 20, - Vn,Ai(n v, 0]1| Ln ||0
011
(B.2)
|I w 01 V..Ai (1) v, 0)l L Iln
1/11-
(B.3)
IVv.Ai(n; (1) 0,0") V..Ai(n, vs 0)1 / Les ||0" 811
(B.4)
VVAA 45, 0) - VoA.(1) 1, 0)| 2enlli m/1- IV w',0) VoAi(n, 10 0) Toolli -
(B.5)
(B.6)
Assumption 2. Unbiased Gradient and Bounded Variance: There exists ox > and on > 0, the stochas-
tic gradients are unbiased, i.e. V.V v) Rv v. 0), E, V. V.R(n, v, 0). 1,.., B and satisfy:
E Vo 0,0) VoR(n, v, 0)1| o.. ElIV&GB(n v, 0) VeR(n, vs 0)1s² o².
(B.7)
(B.8)
Assumption 1 are the basic assumptions for solving the non-convex optimization problem with stochastic gradient descent (1).
Assumption 3. Bounded Hessian: The Hessian for As ib, 0) is bounded, ie.there exist positive constants He and He for Adn v. 0)., vp and O, the following inequalities holds:
A. v. 0) 15 Ha.
(B.9)
V 1o.2o...1 [0.0.. A:(ms v.. 0) Hg.
(B.10)
Assumption 4. Bounded Block-coordinate Gradient: The gradient of all the participants' local output w.r.t. their local input is bounded, i.e. for, all i E I...., S there exist positive constants Qs and Qo satisfies the following inequalities:
IIV As (m,, 1. 0, Qu. (mm, ||VoA:(mm 0,0) Qo.
(B.11)
(B.12)
Assumption B are the fundamental assumptions for bounding the compression loss. Compression introduces errors into the imput of the loss function: therefore, with bounded Hessian, can determine the maximum effect of these errors on the loss. Additionally, bounding the block-coordinated gradient is common practice VFL analysis. This approach helps constrain the entire model's gradient when the gradients of other parts have been bounded 62 [5]
Assumption 5. z-Strongly Concave: If function A: 1h, 0) is z-strongly concave for n, then for all sb and O, the following inequalities satisfy:
d, 0)|-
(B.13)
Assumption made in previous results on convergence of robust training 65 and is justified through the reformulation of robust training distributionally robust optimization. It helps bound the delayed gradient of n).
B.3 Proposition
Proposition 1. Under Assumption [ and Assumption
5 the loss function R(n', w/, 0) is Lm-oth for v.
La La-smooth for 6, and the following inequality holds for all V. O, and 8:;
Lo 10/ 0|1
R R V. 0') R(, (, S (VoR (VR(m. 0' 0)
2
(V. 0)0 v w) + L 1| ~1|
(B.14)
2
where Tay = Tapy + Loggknng and Tug = Log + Lo,,uue This assumption is consistent with Proposition I in (54). This can help us to connect the N-loop and M-loop-
Proposition 2. The classical prak--popagation-bbae based gradient descent algorithm can be viewed an algorithu attempting to solve the PMP (7). The costate processes Pt and the gradient V A(n, v. 8) satisfy the following equation:
pt - Aln v.00
(B.15)
23

---

B.4 Definition
Definition 1. Compression Error (forward message) Considering sample we can define the compression error of )) Ca,, c E 2.2..... C. i.e. Ca, = C( Dt.... We denote the expected norm of the error from the client at global iteration synchronized, the error from all clients is e (ef,e, c2.i ..., Co,). Then, the expected norm of the error from all clients:
Ex, Elle,,ll ? and Ex max. Ee,t. Since all client operations
Bl|e.1| Ell(et, S2 e
Σ Ellea.11?
S C5
(B.16)
B.5 Lemma
Lemma 3. Zeroth-Order Optimization. For arbitrary f in problem (P), the following conditions hold:
fx()) is continuously differentiable, its gradient is Lyp Lipschitz continuous with Ly L:
f = Ex)((()))
(B.17)
where u is drawn from the uniform distribution over the unit Euclidean sphere, fx(x) = B(f(x + ju)) is the smooth approximation of f.
2) For any x. R. the following inequalities satisfy:
L2y²d
(z)
(B.18)
Proof of this lemma is provided in 42 18
Lemma 4. Bound the costate difference at the bottom module. There exists constant G' dependent on T and K such that for all {O...., N). m E {O.....
...MMM M). and (1,...,,8:
pm-- M ||
G'a,, (MrN-1)
(B.19)
Where G' =TKT+I(K" +T(T 1)K2T-2 TKTT), m is the iteration index of full propagation.
This lemma bounds difference the costates of the first module in the adversary's 's N-loop. fix the data point i, and for case of notation drop the dependence of state variables on the index i, while also suppressing the notational dependence on @ for all functions, as θ is fixed during the updates for the adversary We define ı, and p. the state and costate trajectories generated from the initial condition up + ry. We additionally define öpf : pf and dal == zi s/,, E the iteration index example updates. We first prove bounds on ||pé|| and |
Proof:
Applying Assumption we have:
|p < | V w)lls K.
(B.20)
V H. P 0c)1l
1ppt Vall (...(.2....1111H1 B.)
K Kllpé+1 I
< K K² + ... TT
s k I -T+
(B.21)
Next, from Assumption 1. we have || [ (Io + 177) fi (zo + + K|mn" n/11. By induction, we
have:
|| K n''11
(B.22)
To bound Ip2n, phu, we first note that 11 || ||
v K|180411. We write:
24

---

||óp1
VV (og pb+1) H.(((s.
IV Hel (i+1)
V. H. peta) + -V
KVi++1 V2fe(x.)
v=f((xx)) + (fet1 pit1 Vaf((a))) + -
-T--
K110211+ ||+
1 Σ (K* 1-1 -1 + k>]loz(11|
(B.23)
Applying (8.22 we have:
| 11. (K" + (T(T-))R2T- +TK** 1ll -1
(B.24)
updates with the form:
77 [+ = = of an Vnfm, + n/' AM,) T PMA
(B.25)
Applying Assumption ( and [B.21). we have:
11 I K kT+TTa 1).
(B.26)
Finally, substituting with B.26 gives the desired result:
Iia.,. - P'r, Gla,(e- 1);
(B.27)
where G' =KK + +T( - 1)K-r T
Then, We are going to bound
PM m- m.N
PM1
PM PMI
PM1 PM11 +MMi - PMM
|P
m.D PMaall
+ 1l
m.0 P.
G'a,(Ti N) + G' (N
an [(T + 1)NV -1]
a'a,, [Mx - 1],
(B.28)
where, (a) is obtained using the triangle inequality, (b) is obtained using (B.77, for each M M-looo, the adversary updated N times. Proof completes.
Lemma 5. Bound Compression Error. Under Assumption 3 f and Definition 1
the norm of the difference
between the loss function value with and without compression error is bounded:
E V An di 0) V.A.(n v. 1 H&CE** E| A
(B.29)
di VoAi(,, b. = Q3
(B.30)
E 2. A. 1h Art,, v. I| 133C.*
(B.31)
The proof of this lemma proceeds same to Lemma D.4 in
Lemma 6. Bound the gradient for n. Due to the communication between the clients and the server involved in the update process of adversarial examples, the gradient V respect to 1j involves estimation gradient V A from ZOO and compression gradient V v. Under the Assumption I and Lemma 3 at the i-th sample and k-th iteration, the pseudo-partial partiol derivative for satisfies the following inequality:
V (n Mer,, 0.)l
f I...,MM
argmin
E E| (,h, s p(X) (1 (1- MN+1
2G2 Lnn
[MKN 1/3 (2(2 2Lmm
1
Lnn
p2
x HBCE** + 4 + k²2
(B.32)
where G KG', Os T and E X.
25

---

Proof:
According the chain rule, note that A,A.(fs: vis, 0z) can be split follows:
E V 1/2 0.)
|| E nf V A. tha 0:)11
El| y B V Alú. vs. 0:)11
(B.33)
For (a): we view the clients' networks as independent model. From Proposition we can get the following: 1 | -| V.r. An, m. dh,, 0.)Il& K, where 1,2, M denotes M M-loop index, 1,2, N denotes N Nolo index.
(B.34)
According to the Lemma 8 in [54, drop the dependence of all functions @ and the data point index i for the proof. The N-llop of the adversary's updates can be written (B.25). Recall that the true gradient of A(1 m.x
A(n v,, (x + 11) M.
(B.35)
We will bound the maximum difference of the update vector to the true gradient over the iterations of the adversary's updates. In this sense, the adversary's updates be viewed as standard gradient method with an inexact gradient oracle. write
, (x 7) PM,," v(,(i
|1= -I= (a
n)
PM, p Vnfm, (x 7) PM
Slppf 1 pi m.
VIl (x (x)
SKG'a, [MMxN- 1]
-Gan [Mx - 1].
(B.36)
We now appeal to inexact oracle convergence result in [5]. we define a (8, p. L) oracle as returning a vector g(²") such that the following inequality holds:
Given concave function fx') and point x'.
5 ||> = f(x') f()) (g(*"), - x) ||
x 21
(B.37)
It can be shown that if we have an approximate gradient bound of the form (B.36). and A is Luw -mooth [Assumption[] and z-strongly concave 1) (Assumption 5). then the updates for the adversary are created by -oracle, where
(8, 26, 2
== c [MxN 11? (2
(B.38)
2
Letting Qn < 1/Tny and applying Theorem 2 (max, A(r) A(f)) from the Lyn smoothness of A in gives
15]. along with the inequality ||VA(1))
IV AV,A(A)0)1' 011 < L2n 1111 -n-1p(1
MN+1
2G
[MKN- 1MKN-11( (2+ -1
Lmn
Lm
2Lm
(1
MN-1
2G Iap
[M - 112 (2 (2+ 2T
< D(
(B.39)
Topn
where 17 is the true solution to the inner maximization problem. Since we initialize 17 X. we have that 1|7000 21 ((2)). We can get:
EllV Ft... 12' D( (1-
MN+1
2G Lup
[MKN 11² (2+ 2mm -21mm
+
(B.40)
Lipp
For (b): we use Lemma 3 and Assumption
E V xe, Aiee, wi, 0:)11
3 2.. A (fn. sh,, 0.) - V... Ailfi, vi. 00)0 Ailn, (1. Vx., Ai(h,,
+ 3R (ft,, 1/s, 0:)11
33H 2d K2).
(B.41)
Substituting (a) and (b) completes the proof.
26

---

Lemma 7. Connecting Gradients. Under the Assumption inequality be obtained:
7 Assumption 5
and Lemma 2
the following
L ct
"..Ss(""",,"00" 0.011 V o 1h Vo9a(n" M.
(B.42)
2²
Ldn 5t
(B v VoGa(n" .v. 0)
(B.43)
where c* = 3(H d2 + (M,,
TM,,N) D(X)Lim (1- Lon
MN+1
2aa bm
M 113 (: 2imn )
Proof:
Under Assumption Assumption 5 and Lemma
2
for server model parameters v). we can get:
B V 0, - V. .,,0)
P.I ehs 0.) V V. 0:)1
B
SEB
L
Ellii 1l
B
L
B A.tr 44, 0.)11
B²²
i88
5*
(B.44)
22
Similar to proof & in B.44), client model parameters we get:
L3n 5t
B V VoGa(f.
(B.45)
x²
B.6 Theorem
Theorem Bound the Global Update Round.
When the parameters are updated with the perturbations:
i or argmin
V (n.""" 1he,, 0.)
(B.46)
The gradient of îJj is bounded:
E| BJIV, 2i 0.11
(B.47)
where ç = 3(H3CE* 12.30 (M,
(MM,N)
L(x) (1- (1- T.mm
MN+1
2G Lng
M 11} ( (e+ )
The global iterates satisfy:
E
2 8*)) 11 (n"
K
2A
2oo
+
am
Om
RM
2(1 + Lyjas) HBC + 3(1 Leam QHH3C+ CH,
(2(1 + LyarM 3
3(1 + Laass Lznn
am
z²
3as (1 Locen) L2
3 (M(M,N)L22
o do L Lyars
(T ol aaraaanal + LoaM Lā,
1am
20,
10
30M K² (M,
[2(1 Lyar) + 3(1 Loau Lon
(B.48)
amz?
Proof:
For the gradient respect vi, there exists compression error, but no estimation enror: VeGe 1/th 0*) (L/B) Eies Ai wh. 08), where of is the output of the adversary's inner problem iteration of and
27

---

Vogolfi ,²*0)) satisfy the following equations:
i argmin
V A. (11i "00"",,ss, 0.)
(B.49)
M
12mm
++*1
ap (ii ,²,00)
(B.50)
For the gradient respect to 8, there exist compression error and estimation error: (o v 0*) (1/B) Eies Vod. wi, 0%), the Vog NAA ,²)) satisfy the following equation:
@++1 0 - as Voln("i 0²)
(B.51)
Furthermore, (1 k and VaR(n)** *A 4* 0*) are true full gradients. We begin with the inequality for the I-smoothhess of VR(m"" .itt 0²), and apply Proposition k 0, 1, 1,... K is the iteration indice, we can get:
8h) )*) and Voga(n"** * 0h) are true stochastic gradients, R(n w 0h)
R
A+1
, 26 h41 : gg R (n 1* ,0t)
V R (1" t 1tt 04) wett2 - w²) + Ls
2 1 25 ?
VoR (nrih, (n""t, (44) ghtt 8¹) + Le
lah+2 g%|
(B.52)
For (a):
( (7)tt 5k w/b 0²) k6- 16t)
Lu | 2t4
2
(VoR (1rmh 16 8h) -as VOS wh, ,01)) + L
Vo 10 00/
2
--ap(1- - 2
Tu )11 V (nriE WA
112 Tua
11 Des wer - vu (1 we (x) 112
2
+1 Iw (VoR (1)*h 2 06
V.R (11th
1 00)
Vobuini ,09)
=-av(1 Lucky
)/
V (n 1ot
11a Lyak
11
(F"" ,*, VuR (prie 1566 .0)) 112
2
2
+ +av - )
(.RR (1' 6
V (1" w/th 0*) Voga(1 + wt (4.0))
+av )waw)
(VOR (1=** 2
V. VoGE(1 2/th 8h) Vo 00 80)))
<-ap(1- Luny
)/ V (n 6b. 0²) 11 +L V Galit s0, - V. *tt 0²)00>112
2
+ L.a V Vr
it * .0*) - V (v"tt,**.00
1..0000 .0*) 112
+ (1 L
V P
11 * xt ,0²)
(1)* V..R
* 0*) voboln w²t 00,
+ Ow
(1 Lo )| R
(7)-h *
11a
(1 La V G=()"* ,00 - v Ga o
2
2
a V R 17 2 "² 8¹)
1
av
(1 Low )|l V (" ,*, V.Ge(1 "atii"""""", ,",0">'""
2
2
+ L V
GB(n" + */,0""
0² - V.R (2 ,²) 1|2
+ (av(1-Laan
R (n=* wt 0²)) , (n) w/, 0*) VeSa(n"* wt ,',,)))))))))
..)).
(B.53)
Note that E V (19 * 0*0]] V.R ( 2 ,0²), where the expectation is taken over the randomness
of the mini-batch sampling. We can get:
E V R (1) t 3 00)) R.R ,,, 00) VoSu(1i** ,6t o.
(B.54)
28

---

Then, can get:
V R (7 2t 00)) 168-11 154) + 4
2 Nett $19
E
de LIVOR (1t 10, 80)) 11 4t'
(1 LooD) [
wh 00) Vuga(n . |1
2
2
+ LEa
[1V Go(nn * ,0.00) - R (1" x0tt00))))) 11]]
(B.55)
Under Assumption
we can get:
[IV ,0,0 VoR (n 25
,50 ,00) 10),, |8] s
(B.56)
Furthermore, under Lemma and Lemmaly 7 can get:
E
|I Gn' w 0*) VoGa(n 000)
s 2T
V Gis"" 2xt ,00
Vosis(f 10 0t ]1²] + 2F GiV VoGa(1) ", 000)
CH +2 Lon 5t
(B.57)
2²
Finally, can be obtained:
E V
R (17' 255 0*))
wh w²t) + 2 | - 15619)
Ly
au (1 + Legee Le (2H3 + L2n
s -ー 2
Ow
E | V 17"
11a]
+ Luagos. (B.58)
2²
For (b), similar to the proof for yb in B.55), for 0, we can get:
E VoRk (n7 wtt .0 gh-
8¹)
Le g
8h
||]
< 2 [IVoR (vt w 0*)) 112) +
a0 (1 LOC |IVoSis(i ,0000 VoSi(1)** . x+|1
2
LOaEE |IV : "0,,00 0hh VeR (n w.th 86 [1
+ +ap(1 - L VOR (11th wtt ot) VoR (n wt .00) VoGE(1) .6s .00.0/]
(B.59)
Under Assumption 2 we can get:
nablas R (17* wht 0t) VOR (n=0t wt. 000 VoGi(1n" . : =0
(B.60)
E VoG it 5* .00 - VoR (niie ut 00)) 112] ag.
(B.61)
Furthermore, under Lemma
Lemma 5 and Lemma
we can get:
| ( wt:00 0h) VoGu(n"* w* 8* [P]
< 3E
|iv w: )hh Vo wh 8* )1
+ 3E
| ,0',0" - Vogo(T) "." [00
33
(1" 0h) Volu(n" " w "4,8"''3 (1)
3 L2 ol
12 +3 + Lbn LBn 5t
(B.62)
z2
Finally, can be obtained:
V (n 10. (0)) .0st-1 -154) +
La lostts - 1t1p]
E2
2
< d0 ||VoR (17-he ,50 0000)00
1P]]
as
(1 + (3 2dd + 30 + و Lbn
Loajo
(B.63)
29

---

Substituting a) and b), we can get:
17' R+1 .1tt11 -10tt1,,00 ,0it1) - R (orut .05
)0)
< E
FRR R
(7 25. 84)) whrz - 25t)) + 4x
| 15/4
2
1
VoR
n7 .
166 8th) ghhz 8e)
Lg
phht1 - 84 11/]
2
+2 Lin -
+ ) Lyo
<
4ei E
ViveR
(7) 13. 100 .00) 11 + csi
(1 + Lya ((H
2
2 E
|IVoR (nt w²t 84) [1
+
08
2
(1 - Lo00
L²n²d
4 + 30 +3 +
Lbn -th
+ Loo
z²
(B.64)
Since s and are updated synchronously in the outer loop, we take amm min {aw, ag), and combine the gradient:
R (ntttt1 10tt1,00t - R (n wp ",p,, ]
+2 .5
Om B [IVR (n""""0", 0000000)))))00) |1|] + av (1 Laa))(H
22 + Lua
+ 09 (1+ + (3 12220 3QUH3CE* +3 Lbn
Lon 5t ) Loago0
(B.65)
Summing these inequalities from to take & mar (E*), and then (= max
(5*):
k=0....
Cmm
VR
( 1/th ,b, Bh) 112)
L2n
E
R
11) 2 24 00)) R (nr (uttt.ght))
-r (1 (2H +2
Loago
K
x2
13 12
Len
a0
Lo00)
+ +30 +3
LOaÃoo
2
32
12
E
R (17 uu v. 80) R (2 (,, (x,,
av
LLa +2
Loa
2
22
22
12 5 Loo
a6 (1 Load) (3
+3Q3113C2 +3
(B.66)
2
1
2
Then, define A = R (1)-00 00,0%) infa (R (n" wt 0*1) and ax max (av, as):
30

---

|VR m7 wh 06) 119)
k=0
2A
ax + thpaaw (2H +2 22
Len (
2
amk
am
2²
am
ao(1 + Locke) Lu +30 +3
+
2Lau
Om
1
2²
Om 2
2A
a0 + LoaM))
(2H Lon
amk
am
am
anr(1 Lacer))
L2d2d
Lim
20
(3
HQGHCCE +3
Om
2²
Om
2A +
21pasio
2L
= ( amk
2 Loar) H + + Loau Low (QHHCE
Om
Om
Om
31 + Loorr))2
aut
[2(1 + Lgan) + 3(1 L
4am
am
= 2A
2
2
RM 2 L.am)) +/
Loau
amnK
Oum
am
11 30m + LL
QMT
[2(1 + Tyanr L + 3(1 Loo
dam
Om
2A =(-
2Lvadro
2La
Cmnk
Oim
aM
2(1 LNseer) )H3 + 3(1 LaerM) QuH (M,
(2(1 + L 3(1 + Loa
2²
30 + LLd
( am + )
(T(M,N)L' & + Loce Lin
40m
20m=2
40m
30MM Ka (M,,M, N)
[2(1 + L + 3(1 + Loa M L2a1]
(B.67)
am
Corollary
According to Theorem If choose as and ar as O0 ((+t))))
=(-L),, KT
0
0
VK
T= 0
derive the sublinear convergence rate:
K-1
VR (1" "0, (0t,00))1 s 0( ) 0
NMK M
(B.68)
K R=0
Proof:
E
( wt, ,00) 113]
0 ((2A) + (2 + 2Loo²)
2(1 + LUAM + LaaN) (M,
(2(1 + L 3(1 + Loars) Lon
2²
(1 + LOaM) (M, & (a + CMM )Li
+ (T(M, 2 am (1 LoCKss Lbn
Aam
2amz²
1am
3K 7(M, N)
2(1 + LOOCM) Lin, + 3(1 Laan)
x2
NMR
O
+ 0
(B.69)
M
31

---

Experiment Details
C.1 Dataset details
Our experiments constricted public datasets MNIST and CIFARIO:
benchmark dataset for image classification, comprising 60,000 examples for training and 10,000 examples for testing.
MNIST
CIFARIO B Another public dataset for image classification that consists of 60,000 images
categorized into 10 classes.
CIFARI00 :J: Another public dataset for image classification that consists of 60.000 images
categorized into 100 classes.
Tiny-ImageNet :33 A scaled-down version of ImageNet that contains 200 classes with 500 training images, 50 validation images, and 50 test images per class. Each image has resolution of 64x64 pixels, providing a more challenging henchmark than MNIST and CIFAR datasets.
To simulate the VFL. scenario, allocated distinct features each party based on the described methodology prior works 53 17. We partition the last dimension of features according to the feature proportion of each client. We use masking to ensure that each client receives distinct features.
C.2 Adversarial attack
validate robustness. we employed suite of adversarial attack methods. FGSM fast, non-iterative attack PGD-r iteratively perturbs input data using gradient information to maximize the model's 's loss [49]; and
32];
CW uscs custom loss function to ensure minimal perturbations while achieving misclassification 4. CERTIFY (CER) generates adversarial perturbations with Gaussian noise [II]. For black-box attacks, we combined adversarial methods with zeroth-order optimization: FGSM (ZO-FGSM) and PGD (ZO-PGD) [8). We also considered scenarios involving a third-party adversary, corrupting embeddings using different client selection strategies, including Thompson Sampling with Empirical Maximum Reward (E-TS) [6] and All Corruption
Patterns (ALL).
C.3 Hyperparameters
For the parameter updates of both the server and client models, we have adopted the Adam optimizer with a uniform learning rate of ag os = 0.0001.
Moreover, We follow the hyperparameters choices of 55 76
for training.
Table 11: Hyperparameters for Adv. Training
Client Model
batch
Z0O
Compress
Adv.
DecVIAL
PGD
FreeAT
FreeLB
Dataset
size
type
MNIST
MLP
32
100
0.05
scale
0.02
0.002
10
40
8
40
CIFARIO
ResNet- 18
80
200
0.5
scale
8/255
1/255
10
CIFARI0D
ResNeXi.S
80
1000
0.5
scale
3/255
0.4/255
3
10
Table 12: Hyperparameters for Attack
Client
Z0-FGSM
ZO-PGD
ALL & B-TS
Datasct
Model
MNIST
MLP
100 0.05
16/255
24/255
4/255
100
0.32
0.5
128/255
64/255
96/255
12/255
10
96/255
12/255
CIFARIO
R8
200 0.05
0.01
10
8/255
1/255
100
128/255
0.8
64/255
32/255
40
32/255
2/255
10
32/255
64/255
MNIST
R8
100
0.05
96/255
40
64/255
2/255
100
0.8
0.5
204/255
64/255
40
153/255
16/255
40
128/255
16/255
CIFAR100
ReaNcXtt50
0.05
1/255
1/255
0.2/255
100
3/255
0.8
3/255
16/255
10
3/255
0.4/255
16/255
2/255
C.4 Environment
In our experiments, we utilized the following software environment: PyTorc version 2.2.1, CUDA version 12.1.
and Python version 3.1.. The hardware specifications are detailed in Table
32

---

Table 13: Hardware Specifications
Experiment Description
CPU
GPU
MNIST Robust Training
AMD EPYC 755IP
A4000*1
CIFAR-10 Robust Training
AMD EPYC 7452
4090*4
(IFAR-100 Robust Training
AMD EPYC 7452
4090*4
Performance across various NN architectures
Intel E5-2683 v4
4090*
Impact of split position
AMD EPYC 7J13
4090*4
Impact of the number modules
AMD EPYC 7J13
4090*4
Impact the number the clients
Intel Platinum 8336C
4
Limitation of the setting of M and N
Intel Fold 6430
4090*8
C.5 Performance across various NN architectures.
We expanded our experiments by incorporating ResNer18 on the MNIST dataset, introducing different architectural context for evaluating our framework. Similar as experiments in CIFAR-10, 10, the entire model is partitioned into three modules: the first layers of the client models, the remaining layers of the client models, and the server's single-layer perceptron. As shown in Table DecVFAL achieves the best robust performance while requiring only one-seventh of the training time per epach for PGD adversarial training.
Table 14: Results of MNIST Robust Training with Resnet- 18
Training Methods
Clean
White-Box Adv. Atk
Black-Box Adv. Atk
Third Adv.
Train Time
Accuracy
FGSM
PGD
CW
CER
ZO-FGSM
Z0-PGD
ALI
E-TS
(s/epoch)
None
98.66
56.56
12.26
20.99
68.37
48.67
71.43
47.96
75.71
90.86
PGD
98.23
84.73
74.74
21.16
83.98
38,21
83.69
44.56
72.49
1180.74
FreeAT
98.44
79.47
82.20
52.33
90.82
94.84
89.04
46.52
70.15
332.63
FreeLB
98.82
70,81
40.68
31.53
80.05
36.51
81.27
65.91
87.07
1419.10
YOPO
98.72
83.11
82.13
21.30
87.44
54.13
87.33
71.05
88.98
240.62
DP
98.63
80.20
66.63
29.80
80.63
42.68
81.55
44.77
70.31
1175.02
MP
98.12
81.38
74.47
36.31
86.42
53.64
84.84
50.38
74.10
1181,69
Asy-PGD
98.05
79.42
76.27
27.57
85.93
42.63
84.71
57.95
80.16
1167.09
DecVFAL
98.98
89.00
83.20
50.80
93.91
90.95
91.17
70.42
84.14
167.89
C.6
Experiments on Timy-ImageNet
evaluate the scalability of DecVFAL to more complex datasets, we conducted experiments on Tiny-ImageNet using similar AT methods. Table TS presents the robustness performance and training time across different perturbation budgets. DecVFAL-5 2-2 consistently outperforms FastAT and FreeAT, achieving robust accuracy comparable
better than ATAS while requiring less training time. Notably, DecVFAL's performance close standard PGD-5 with approximately half the training time. These results demonstrate maintains its computational advantages and robustness henefits when applied to more challenging datasets with higher resolution images and increased class complexity, further confirming its practicality for real-world deployments.
DecVFAL
Table 15: Experimental results Tiny- Imageeet
1/255
2/255
5 4/255
Method
Time (h)
PGD-5
AA
PGD-5
AA
PGD-5
AA
PGD-5
8.03
34.17
32.89
28.39
24.14
16.48
11.47
FreeAT-4
7.53
29.45
27.78
21.67
16.66
9.27
4.97
FastAT
4.64
24.04
22.42
19.45
15.27
10.52
6.18
ATAS
5.22
32.83
31.57
27.17
22.24
16.25
10.25
DecVFAL-3-2
4.30
33.41
31.85
27.95
23.11
16.74
10.93
33


Reviews:
[{"review_id": "tkGf9ZVkLn", "reviewer": "Reviewer_ikWZ", "summary": "DecVFAL accelerates adversarial training in VFL via dual-level decoupling (lazy sequential + parallel backpropagation), achieving 3–10× speedup with rigorous convergence guarantees and competitive robustness.", "questions": "1. While experiments on public datasets (MNIST/CIFAR) demonstrate efficiency, how does DecVFAL perform on industry VFL benchmarks (e.g., financial fraud detection or healthcare datasets with heterogeneous feature distributions)?  \n\n2. Corollary 1 identifies a bias term \\( \\mathcal{O}(N\\mathcal{M}_K/M) \\) that degrades performance with increasing modules  \\mathcal{M}_K . What practical strategies can mitigate this?  \n\n3. DecVFAL assumes synchronous client-server updates (Algorithm 1). How would asynchronous settings (common in cross-device VFL [36]) affect convergence?", "limitations": "See questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:​​\n\n​​1. Rigorous Technical Foundation​​: Proposes a mathematically grounded dual-decoupling mechanism (lazy sequential + parallel backpropagation) with convergence guarantees (Theorem 1, Corollary 1).\n2. ​​Comprehensive Evaluation​​: Validated across 4 datasets (MNIST, CIFAR-10/100, Tiny-ImageNet), 8 baselines (PGD, FreeAT, YOPO), and 10 attack methods (FGSM, PGD, CW, etc.), demonstrating ​​3–10× speedup​​ without compromising robustness (Tables 1–3).\n3. ​​Theoretical-Experimental Alignment​​: Convergence analysis explicitly accounts for VFL-specific challenges (e.g., gradient delays, compression errors) and aligns with empirical results.\n\nWeaknesses:\n\n​​1. Limited Real-World Data​​: Experiments use partitioned public datasets (Sec. 6.1). Testing on industry VFL benchmarks (e.g., finance/healthcare) would strengthen practicality.\n2. Bias Term Trade-Off: Convergence rate includes \\( \\mathcal{O}(N\\mathcal{M}_K/M) \\) (Corollary 1), implying performance degradation with many modules. Not empirically explored beyond Table 5. \n3. Decoupling Inspiration: Builds on prior work (e.g., Synthetic Gradients [27], ADMM [57]), though VFAL-specific extensions are non-trivial.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments and constructive feedback, which have been invaluable in improving the quality and clarity of our manuscript. Below, we provide detailed responses to the identified weaknesses and questions.\n\n**W1&Q1-Real-World Dataset:**\n\nThank you for the recognition of this limitation. To address your concern, we added **2 real-world datasets: COVID-19 Image Data Collection [1] and Credit Card Fraud Detection [5]** to further validate our work. **VFL Setup:** we set 2 clients which each uses ResNet-18 for COVID-19 and MLP for Credit, one server employs 1-layer classifier. DecVFAL achieves **3.72× speedup** and **2.35× speedup** respectively compared to standard PGD, while maintaining superior robustness.\n\n\nTable 1: Results of Robust Training on **COVID-19 dataset [1]**\n\n| Methods  | clean  | FGSM  | PGD   | AA    | Time (min) &darr; | Speedup (vs PGD) |\n|---------|-------|-------|-------|-------|------------|------------------|\n| None   | **98.19** | 0.36  | 0.00  | 0.00  | -       | -            |\n| PGD     |  69.20|  67.02|  **67.39**|  **66.30** | 29.64    | 1.00×            |\n| FreeAT  | 77.53 | 49.27 | 50.54 | 40.76  |    11.65   | 2.54×            |\n| YOPO  |  77.90 | 66.30 | 66.49 | 58.51  |   8.73    |3.39×            |\n| DecVFAL |  88.22 |  **67.39**|  **62.86**|  44.75|  7.97     | **3.72×**            |\n\nTable 2: Results of Robust Training on **Credit dataset [5]**\n\n| Methods  | clean | FGSM  | PGD   | AA    | Time (min) &darr;| Speedup (vs PGD) |\n|---------|-------|-------|-------|-------|------------|------------------|\n| None   | **92.45** | 0.00  | 2.86  | 3.38  | 5.19       | -            |\n| PGD     | 88.28 | 40.63 | 42.19 | **49.48** | 36.02      | 1.00×            |\n| FreeAT  | 92.44 | 14.58 | 40.36 | 5.72  | 27.16      | 1.33×            |\n| YOPO | 90.10 | 54.68 | 32.29 | 22.39 | 17.98 | 2.00× |\n| DecVFAL | 89.58 | **68.22** | **47.13** | 47.39 | **15.32**      | **2.35×**            |\n\n**W2&Q2-Impact of the number of modules:**\n\n1. The trade-off between module number and accuracy is inherent to module decoupling and must be carefully balanced. Our **theoretical analysis (Remark 1)** and **empirical results (Table 5)** consistently demonstrate that increasing module numbers $\\mathcal{M}_K$ degrades performance. This outcome **is consistent with prior decoupled training work** [2-4].\n\n2. The practical strategies to mitigate this include (1) against excessive module partitioning, (2) align the execution time of all partitioned modules to maximize parallelization benefits.\n\n\n**W3-Novelty:**\n\n\nThank you for recognizing the non-trivial contributions of our framework. We want to emphasize the unique challenges our work addresses.\n\n1. **Technical Innovation:** While we build upon fundamental techniques, applying decoupling to communication-efficient VFL presents distinct challenges. Our approach uniquely combines decoupling with AT and VFL-specific methods like Zero-Order Optimization and compression [8]. The key innovation lies in **decoupling adversarial sample generation with frozen parameters** - a departure from prior work [2-4] that only decouples parameter updates. This enables true concurrent forward/backward propagation and addresses the computational bottleneck in adversarial sample generation, achieving substantial 3-10× acceleration for VFAL.\n\n2. **Theoretical Contribution:** The simultaneous presence of **multiple gradient approximations** creates analytical complexity not addressed in existing literature. Our framework must account for three interacting approximation sources: delayed gradients (from decoupling), compressed gradients (from embedding compression), and estimated gradients (from ZOO). We provide the first convergence analysis that captures how these approximation errors propagate and interact within VFAL systems, requiring careful treatment of their cumulative effects.\n\n\n**Q3-Asynchronous Settings Impact:**\n\nWe think that full asynchronous VFL updating presents significant convergence challenges. AT operates through **a minimax optimization** that alternates between adversarial sample generation and model parameter updates. This requires **freezing model parameters during adversarial sample generation**, followed by synchronized parameter updates using the generated samples. Asynchronous settings would **disrupt this minimax execution order**, potentially leading to scenarios where model parameters are updated before optimal adversarial samples are identified.\n\n\n\n[1] MAGUOLO, Gianluca; NANNI, Loris. A critic evaluation of methods for COVID-19 automatic detection from X-ray images. Information fusion, 2021, 76: 1-7.\n\n[2] JADERBERG, Max, et al. Decoupled neural interfaces using synthetic gradients. In: International conference on machine learning. PMLR, 2017. p. 1627-1635.\n\n[3] HUO, Zhouyuan, et al. Decoupled parallel backpropagation with convergence guarantee. In: International Conference on Machine Learning. PMLR, 2018. p. 2098-2106.\n\n[4] HUO, Zhouyuan; GU, Bin; HUANG, Heng. Training neural networks using features replay. Advances in Neural Information Processing Systems, 2018, 31.\n\n[5] CARCILLO, Fabrizio, et al. Combining unsupervised and supervised learning in credit card fraud detection. Information sciences, 2021, 557: 317-331.\n\n\n[6] CROCE, Francesco, et al. RobustBench: a standardized adversarial robustness benchmark. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).\n\n[7] LIU, Yang, et al. Vertical federated learning: Concepts, advances, and challenges. IEEE transactions on knowledge and data engineering, 2024, 36.7: 3615-3634.\n\n[8] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491."}, {"review_id": "q8LwuMsaha", "reviewer": "Reviewer_yT3W", "summary": "This paper introduces DecVFAL, a novel framework that addresses the computational efficiency challenges in VFAL through an innovative dual-level decoupling mechanism. Specifically, the framework first employs \"lazy sequential backpropagation\" to decouple the client bottom module from the rest of the network, reducing communication frequency between participants; second, it implements \"decoupled parallel backpropagation\" to enable asynchronous computation across modules, eliminating idle waiting time. The authors provide the first convergence analysis for VFAL, proving that DecVFAL achieves an O(1/√K) convergence rate matching standard VFL. Experimental results demonstrate that DecVFAL achieves 3-10x training acceleration compared to existing methods while maintaining model robustness.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper addresses a critical computational bottleneck in VFAL, which has significant implications for practical deployment. The proposed dual-level decoupling mechanism represents a novel approach that cleverly resolves the inherent sequential dependencies in VFAL.\n\n2. The paper provides the first convergence analysis for VFAL, considering the interaction of multi-source approximation gradients, decoupling mechanisms, and VFL architecture, which is a significant contribution to the theoretical foundation.\n\n3. The work includes extensive evaluations across multiple datasets, different model architectures, and various attack types, demonstrating the effectiveness and generalizability of the method. The detailed analysis of key parameters (M, N), number of modules, and split positions provides valuable insights into the method's design principles.\n\n## Limitations and Questions\n\n1. While the paper mentions differential privacy guarantees, there is limited analysis of whether the decoupling mechanism (particularly delayed gradients) introduces new privacy vulnerabilities. Could malicious participants extract additional information from these delayed gradients?\n\n2. The proposed decoupling strategies are primarily based on feed-forward network architectures. How would these strategies adapt to models with complex structures (e.g., Transformers or networks with dense skip connections), particularly when these structures are distributed across different participants?\n\n3.  While DecVFAL reduces the number of communication rounds, each round may transmit more information. How does this trade-off affect overall performance in bandwidth-constrained environments?\n\n4. The experiments indicate that M and N parameter choices significantly impact performance. Is it possible to develop adaptive strategies that dynamically adjust these parameters based on task characteristics, communication costs, and computational resources?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful comments and constructive criticisms. Your feedback has been invaluable in improving the quality and clarity of our manuscript. Below, we address the weaknesses and respond to your questions.\n\n**W1-Privacy Analysis:**\n\n1. **Delayed gradients do not introduce new privacy vulnerabilities.**\nDelayed gradients transmit same content as immediate gradients differing only in transmission timing. Malicious participants cannot extract additional private information from delayed gradients beyond what they could already obtain from immediate gradients.\n\n2. DecVFAL operates within the VFL-CZOFO framework [1], which provides differential privacy through zeroth-order optimization. The delayed gradients undergo **the same privacy-preserving transformations as standard gradients**, maintaining equivalent protection levels.\n\n3. While AT increases communication rounds and **may require a larger privacy budget** compared to standard VFL-CZOFO, we will provide detailed privacy analysis in the revised version to quantify this trade-off between robustness and privacy.\n\n**W2-Model with Complex Architectures:**\n\nDecoupling easily adapts to complex architectures by treating functional blocks as module boundaries [2-6].\nFor complex structures like Transformers and residual networks, we can treat entire functional blocks (including their attention or residual connections) as **single dynamical units**. This preserves the sequential flow while enabling parallel processing. Our ResNet-18 experiments illustrate this concept: residual blocks naturally work as complete dynamical units, maintaining both the residual connections and the dynamical systems framework (footnote 4, page 7).\n\n\n**W3-Transmitted Content:**\n\nDecVFAL and standard VFAL transmit **the same content** (i.e. forward propogation for embeddings and backward propagation for  gradients).DecVFAL achieves efficiency by reusing gradients to reduce the total number of communication rounds.\n\n**W4-The selection of M and N:**\n1. The optimal M and N values depend on many factors: dataset size and features, model structure, and training settings like learning rate and adversarial step size. Because of these complex interactions, developing an adaptive strategy is difficult.\n3. **Practical Selection Strategy.** Empirically, M and N can be selected by referencing the choice of r in PGD-r, where **M×N should be slightly larger than r**, and maximizing M within the communication budget typically achieves excellent performance (only M relates to server-client communication).\n\n\n\n[1] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491.\n\n[2] CHEN, Ricky TQ, et al. Neural ordinary differential equations. Advances in neural information processing systems, 2018, 31.\n\n[3] LI, Qianxiao, et al. Maximum principle based algorithms for deep learning. Journal of Machine Learning Research, 2018, 18.165: 1-29.\n\n[4] LI, Qianxiao; HAO, Shuji. An optimal control approach to deep learning and applications to discrete-weight neural networks. In: International Conference on Machine Learning. PMLR, 2018. p. 2985-2994.\n\n[5] HAN, Jiequn, et al. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 2019, 6.1: 1-41.\n\n[6] WEINAN, Ee. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 2017, 5.1: 1-11."}, {"review_id": "G2bLMJ1At7", "reviewer": "Reviewer_2ipB", "summary": "The paper proposes DecVFAL, a fast adversarial learning technique for vertical federated learning setting. To accelerate training:\n\n1.\tClient-side performs lazy refinement steps on the small bottom module ‘N’ times while keeping the upper-layer gradients frozen. \n2.\tUse delayed gradients concurrently to update the rest of the modules. \n\nAuthors provide theoretical guarantees for convergence and demonstrate 3-10x speed-up on various datasets.", "questions": "1.\tCan you briefly instate the significance of AL in VFL and why does it need acceleration in practical scenarios? Considering most of the applications (e.g., in hospitals) involves collaboration of a few parties and would have enough resources.  \n\n2.\tIn this VFL scenario, a separate server is utilized between parties and assumed to hold the labels. Is this a realistic in VFL? What other threats is this setup exposed to? How would the method adapt when parties have non overlapping user sets and rely on secure entity matching?", "limitations": "Yes some of them.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**: \nAlthough the technique comprises well-known tricks (gradient reuse, async updates), it shows good practical speed-up for training. Provides extensive comparisons with baselines with ablations on number of clients, modules, and other decoupling hyperparameters. \n\n**Weaknesses**: \n\n1.\tNovelty is marginal as technique largely reuses know ideas in this context. \n\n2.\tMost of the compared baselines are outdated (mostly from 2019) and some recent ones (e.g., FastAT, ATAS) are only lightly covered.\n \n3.\tNetworks that are used are small (e.g., 84 % clean accuracy on CIFAR 10). It is unclear if results hold for deeper models. \n\n4.\tDecVFAL sometimes yields higher robust accuracy than full PGD despite using weaker inner maximisers with small module in the client-side. It indeed entails strong clarification and justifications. Even the theoretical analysis suggest degradation in performance due to the added bias. \n\n5.\tThe results are inconsistent. For instance, why does PGD and FreeLB accuracy trend differs strongly in Figure 3 and Figure 4 for different datasets?\n\n6.\tThe results for impact of split position in Table 6 seems counter intuitive. Large bottom modules providing better gradient suggests increase in performance. \n\n7.\tExperiments max out at seven clients and assume perfectly aligned sample IDs. Larger client size or heterogeneous environments are untested.\n\n8.\tThe sensitivity of the parameters M and N should be carefully studied and discussed. \n\n9.\tThe analysis of the communication cost is missing. Using decoupled modules suggest substantial increase in communication of smashed layer’s embeddings and should be quantified.", "was_revised": true, "final_justification": "My doubts about the experimental settings are clear now and the additional results and discussion on my comments seems satisfactory. However, considering most of the gains are marginal for bigger datasets (almost 2-3x), I am increasing my score by 1 point.", "rebuttal": "We sincerely thank you for your insightful comments and constructive criticisms. Your feedback has been invaluable in improving the quality and clarity of our manuscript. Below, we address the weaknesses and respond to your questions.\n\n**W1-Novelty:**\n\nDue to space limitations, please refer to W3 of Reviewer ikWZ.\n\n**W2-Baselines:**\n\n1.  We selected sota AT acceleration methods (PGD, FreeAT, YOPO). They specifically target iteration complexity reduction, **directly addressing the primary bottleneck in VFL scenarios: communication overhead**. These baselines provide the most relevant comparison for communication-constrained environments like VFAL.\n\n2. While we recognize recent AT developments, **most focus on improving robustness** through sample optimization (DOM [1]) or loss function enhancements (MART [2]) rather than addressing the computational bottleneck that VFAL faces. These approaches tackle complementary but different challenges.\n\n3. Following your valuable suggestion, we enhanced our CIFAR-10 experiments to include recent methods like DOM [1] and MART [2] in Table 1. **Importantly, our decoupling framework is compatible with these optimization strategies**, allowing for combined benefits as shown in our \"DecVFAL-6-2+DOM\" results.\n\n**Table 1: Comparison with Recent AT Methods on CIFAR-10**\n\n|Method|Clean|4/255||8/255||12/255||Time&darr;|Speedup|\n|-|-|-|-|-|-|-|-|-|-|\n|||PGD|AA|PGD|AA|PGD|AA|(h)|(vsPGD)|\n|PGD|78.00|68.47|63.08|44.61|34.55|34.59|16.29|8.23|1.00×|\n|DOM[1]|75.06|63.74|62.26|44.26|36.59|36.87|21.29|13.26|0.62×|\n|MART[2]|80.98|64.91|**63.76**|43.22|32.46|31.75|14.28|7.85|1.05×|\n|DecVFAL-6-2|**81.83**|**68.59**|62.34|**46.27**|36.48|36.14|18.72|2.75|3.00×|\n|DecVFAL-6-2+DOM|74.31|66.22|61.44|46.24|**39.16**|**39.17**|**21.97**|3.47|2.37×|\n|DecVFAL-3-3|81.24|66.19|60.58|42.96|31.76|31.98|15.11|**1.45**|**5.68×**|\n\n\n**W3-Deeper model:**\n\n1. **Standard Benchmarks for Comparison.** Our experimental settings follow well-established protocols from VFL research [6-7] and adversarial training studies (e.g., RobustBench [4]). This ensures fair comparison with existing methods and enables reproducible results.\n\n2. **Deeper Architectures Included.** We have actually evaluated DecVFAL on several deeper models, including **ResNeXt-50 on CIFAR-100 and Tiny-ImageNet** (detailed in Section 6.1 and Appendix C.6). Since DecVFAL is designed to be architecture-agnostic, it naturally scales to modern deep networks without structural modifications.\n\n3. **Additional Experiments.** We are currently conducting experiments with deeper networks on CIFAR-10 to further validate our approach. These results will be available within one week.\n\n**W4-PGD Performance:**\n\nThe key insight is that DecVFAL performs significantly **more adversarial updates ($M\\times N$) compared to standard PGD ($r$ iterations)**, where typically $M\\times N > r$ in our experiments (e.g., 5$\\times$10=50 vs. 40 updates for MNIST, 6$\\times$2=12 vs. 10 updates for CIFAR10). While individual updates use weaker gradients, the substantially **higher update frequency provides more comprehensive adversarial space exploration**, compensating for per-update approximation errors.\n\n\n**W5-Performance differences:**\n\nThe primary reason for the accuracy differences is the **different adversarial sample generation settings**: **40 iterations** on MNIST versus **10 iterations** on CIFAR-10 to generate adversarial samples.\n1. **Slower Convergence for PGD.** The high iteration count combined with MNIST's **2-layer perceptron model** results in **slower convergence** compared to others.\n2. **Overfitting for FreeLB**:  FreeLB's **gradient accumulation mechanism** combined with 40 iterations causes **overfitting issues**, leading to poor generalization and **accuracy degradation**.\n\n\n**W6-Number of Layers in Bottom Module:**\n\nThis counter-intuitive result in Table 6 is caused by **error accumulation during backpropagation**. While larger bottom modules do provide better gradient approximations and reduce the initial delayed gradient error, **this error still persists** and becomes progressively **larger** as it propagates backward through the network layers. This **accumulation effect** ultimately leads to the unstable performance we observed.\n\n**W7-Client Size and Heterogeneous Environments:**\n\n1. **Experiments on More Clients.** We have conducted additional experiments on MNIST with **14 and 28 clients** as shown in Table 1 in W1 of Reviewer Ecn8. \n2. **About Heterogeneous Settings.** **Sample ID alignment is indeed a separate challenge in VFL** that involves entity resolution and record linkage [5]. Our work focuses on the complementary problem of speeding up adversarial training when participants already have aligned data, following standard VFL assumptions.\n\n\n**W8-The selection of M and N:**\n\nDue to space limitations, please refer to W4 of Reviewer yT3W.\n\n**W9-Communication:**\n\n1. Decoupling does **not introduce additional communication overhead**, as only the server module and the top-layer client modules require true server-client communication, while other inter-module communications are purely local computations that barely affect efficiency.\n2. **To fully address your concern, we have included detailed communication cost statistics below** to quantify the actual communication requirements and demonstrate the efficiency of our approach.\n\n**Table 2: Communication Cost**\n\n|Methods|Comm.Cost|(MB)&darr;||Speedup (vs PGD)|\n|-|-|-|-|-|\n||MNIST|CIFAR10|CIFAR100||\n|PGD|294.48|122.92|261.86|1.00×|\n|FreeLB|294.48|122.92|261.86|0.98×|\n|FreeAT|117.79|98.34|209.49|1.18×|\n|YOPO|147.24|**61.46**|**130.93**|0.99×|\n|DecVFAL|**73.62**|73.75|**130.93**|**3.00×**|\n\n**Q1-Why AT and Acceleration are needed:**\n\n1. **Why AT is essential in VFL.** VFL faces unique security challenges as detailed in Appendix A.3. These include **third-party attackers modifying embeddings** during communication and **malicious participants** corrupting local features. While we could use different defenses for each threat, **AT offers a unified solution** that builds inherent robustness against multiple attack types simultaneously.\n\n2. **The resource utilization challenge.** You're absolutely right that institutions like hospitals have substantial computational resources. However, this actually shifts the bottleneck to **how efficiently we use these resources**. In VFAL, generating each adversarial sample requires multiple rounds of **client-server communication for gradient updates**. This creates **significant idle time** where powerful computing resources sit unused.\n\n3. Our approach reduces communication rounds from r to M, allowing modules to **process computations in parallel during communication delays**. This means institutions can fully utilize their existing computational infrastructure rather than leaving it idle. Our experiments show **consistent 3-10× speedup**.\n\n**Q2-VFL setup:**\n\n1. The server-holding-labels setup is **a standard and realistic assumption** in VFL literature, as evidenced by numerous works [3,6,7], where one party naturally possesses the target labels. An example is fraud detection where a bank holds transaction labels (whether fraud) and transaction history feature while partnering with e-commerce platforms holding user shopping features.\n2. It is worth that 'server' and 'client' are **roles within the VFL framework** rather than fixed physical entities—in practical deployments, these roles can be **distributed across different participating organizations** based on their data ownership and computational capabilities, with the label-holding participant typically assuming the server role for coordination purposes.\n3. Our threat model considers multiple adversarial attack vectors including **third-party adversaries** intercepting embeddings and **malicious clients**, as detailed in Appendix A.3, though VFL involves other security threats like backdoor and inference attacks that fall outside our scope.\n4. For scenarios with non-overlapping user sets requiring secure entity alignment, this represents another research topic in VFL [3,5], and decoupling can be integrated with existing privacy-preserving entity resolution techniques since AT operates independently of the entity alignment phase.\n\n**Typo:**\n\nThank you for catching the typo. We will correct them and clarify DecVFAL-3-3 and DecVFAL-6-2 indicate the settings of M and N.\n\n[1] LIN, Runqi, YU, Chaojian, HAN, Bo and LIU, Tongliang. On the Over-Memorization During Natural, Robust and Catastrophic Overfitting. In: The Twelfth International Conference on Learning Representations. 2024.\n\n[2] WANG, Yisen, et al. Improving adversarial robustness requires revisiting misclassified examples. In: International conference on learning representations. 2020.\n\n[3] LIU, Yang, et al. Vertical federated learning: Concepts, advances, and challenges. IEEE transactions on knowledge and data engineering, 2024, 36.7: 3615-3634.\n\n[4] CROCE, Francesco, et al. RobustBench: a standardized adversarial robustness benchmark. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).\n\n[5] HUANG, Lingxiao, et al. Coresets for Vertical Federated Learning: Regularized Linear Regression and $ K $-Means Clustering. Advances in Neural Information Processing Systems, 2022, 35: 29566-29581.\n\n[6] WEI, Kang, et al. Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309, 2022.\n\n[7] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491."}, {"review_id": "Knu2K1S8xn", "reviewer": "Reviewer_Ecn8", "summary": "This paper proposes DecVFAL framework to accelerate vertical federated adversarial training. DecVFAL decouples the bottom modules of clients and backpropagation to eliminate idle waiting times. Experiments on four public benchmarks demonstrate that DecVFAL effectively balances computational efficiency with model robustness.", "questions": "1. Will deploying more complex models on the server side, such as multi-layer perceptron affect the results?\n2. Typo: \"single-layer perception\" in line 859", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths\n1. The paper is well-written and relatively easy to follow.\n2. There are extensive theoretical analysis about the convergence analysis.\n3. Well-designed ablation studies of the impacts of components and hyperparameters are conducted.\n\n- Weaknesses\n1. The problem setting seems niche with only 2 clients in main experiments. Even in ablation studies, the max number of clients is only 7.\n2. The model deployed in server side is kept as a single-layer perceptron. Analysis or justification for such choice could be helpful. \n3. The speedup is dependent on the split position. Training time increases with more layers included in bottom layers, which restricts the architecture.", "was_revised": true, "final_justification": "The rebuttal effectively addresses the main concerns, particularly on client scaling and server model analysis. Therefore, I maintain my score of 4.", "rebuttal": "Thank you for your insightful comments and for recognizing the contribution of our work. Your feedback has been invaluable in enhancing the quality and clarity of our manuscript. We reply to the weaknesses and questions.\n\n**W1-Client Scale:**\n\n**Experiments on More Clients.** Thank you for this valuable feedback. We have conducted additional experiments on MNIST with **14 and 28 clients** as shown in Table 1. These results demonstrate that DecVFAL maintains strong robustness and computational efficiency across different client configurations.\n\n**Table 1: Results with Varying Client Numbers (MNIST)**\n\n| No. Clients | Method | Clean | Robust  |Accuracy|| Time (h) &darr;| Speedup |\n|-------------|--------|---------------|-------------------|------------|---------|-|-|\n| | | | FGSM | PGD | AA |  | vs PGD |\n| 14 | PGD  | 96.79 | 86.33 | 85.16 | 78.31 | 3.09 | 1× |\n| | FreeAT  | **97.72** | 65.10 | 67.50 | 47.48 | 2.47 | 1.25× |\n| | YOPO  | 96.74 | 83.64 | 83.19 | 74.50 | 1.77 | 1.74× |\n| | DecVFAL  | 97.11 | **89.63** | **86.40** | **79.44** | **0.71** | **4.35×** |\n| 28 | PGD | 95.95 | 88.53 | 87.16 | 80.27 | 3.59 | 1× |\n| | FreeAT | **96.83** | 82.39 | 83.00 | 65.92 | 0.87 | 4.12× |\n| | YOPO | 95.93 | 89.09 | **88.86** | **83.75** | 0.83 | 4.32× |\n| | DecVFAL | 96.00 | **89.99** | 88.04 | 82.63 | **0.73** | **4.92×** |\n\n**W2&Q1-Server model:**\n\n1. It is worth that server and client represent functional roles rather than physical deployment constraints.\nThe single-layer perceptron for the server model is a **standard configuration in VFL research**, following established VFL frameworks where the server typically handles final classification while clients process feature-specific computations [1-3].\n\n\n2. Thank you for this insightful question. We tested DecVFAL (M=6, N=8)  on the MNIST dataset by deploying various server-side models: 1-layer perceptrons, 4-layer perceptrons, and 16-layer perceptrons. The experimental results reveal that as the server model becomes more complex (with more layers), it tends to slow down the entire framework's parallel efficiency. This happens because the server acts as a single module, and when it takes longer to process, **all other modules must wait for its output before proceeding**.\n\n**Table: Results with Different Server Model Complexities**\n\n| Server Model | Method | Clean | Robust | Accuracy|| Time &darr; |\n|--------------|--------|---------------|-------------------|------------|-|---------|\n| | | | FGSM | PGD | AA | (s/epoch) |\n| 1-Layer | DecVFAL | 98.10 | 90.40 | **91.93** | **89.58** | **9.866** |\n| 4-Layer  | DecVFAL | **98.35** | **90.57** | 91.70 | 88.74 | 9.986 |\n| 16-Layer | DecVFAL | 97.75 | 89.78 | 89.60 | 83.97 | 13.218 |\n\n\n\n**W3-Split Position:**\n\nWe acknowledge this important practical consideration. When one module contains significantly more layers, other modules must idle waiting for its output, reducing overall efficiency as demonstrated in Table 6. However, our framework provides flexibility to optimize split positions based on computational balance. We recommend treating server-client communication as a module boundary and ensuring each module's computation time is **balanced and slightly greater than communication delay**. This design enables modules to simultaneously send embeddings while receiving gradients, effectively **eliminating communication-induced idle time and achieving maximum acceleration benefits**.\n\n\n**Q2-Typo:**\n\nThank you for catching this typo. We will correct \"single-layer perception\" to \"single-layer perceptron\" in line 859.\n\n\n[1] LIU, Yang, et al. Vertical federated learning: Concepts, advances, and challenges. IEEE transactions on knowledge and data engineering, 2024, 36.7: 3615-3634.\n\n[2] WEI, Kang, et al. Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309, 2022.\n\n[3] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491."}]

Author rebuttal:

Meta-review: This paper proposes a novel method which accelerates the adversarial training process in Vertical Federated Learning (VFL) by introducing a decoupling strategy with lazy sequential back-propagation, achieving significant speedup in performance. It also provides rigorous convergence guarantees and competitive robustness.  One weakness is the marginal performance gain for bigger datasets, as pointed by reviewer 2ipB.

During the rebuttal process, the authors addressed most of the concerns from the reviewers, and no additional concerns is raised. After the AC-reviewer discussion, the reviewers reached a consensus that the paper can be accepted. 

Considering the consensus of reviewers and the nontrivial technical contribution of this work targeting a timely and important challenge,  efficiency bottleneck of VFL, I recommend the acceptance of this paper.
Decision: Accept (poster)