SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $L_{*} = \max_{m}\{L,L_{0},L_{m}\}$ ,  $\eta_0 = \eta_m = \eta$ ,  $\frac{1}{p_*} = \min_{m} p_m$ ,  $\mu_{*} = \max_{m}\{\mu_{m}\}$ ,  $d_{*} = \max_{m}\{d_{h_{m}}\}$ ,  $\mathbf{G}_{*} = \max_{m}\{\mathbf{G}_{0},\mathbf{G}_{m}\}$ ,  $H_{*} = \max_{m}\{H_{0},H_{m}\}$ ,  $\mathcal{E}$  and  $\Gamma$  are the upper bound for the square of the norm of the forward and backward compression error respectively.  $T$  is the total number of iterations (communication rounds).
Remark 5.3. If we choose  $\eta = \frac{1}{\sqrt{T}}$  and  $\mu_{*} = \frac{1}{\sqrt{T}}$ . Design the compression to make  $\mathcal{E} = \mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$  and  $\Gamma = \mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$  we can derive:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left\| \nabla f \left(w _ {0} ^ {t}, \mathbf {w} ^ {t}\right) \right\| ^ {2} = \mathcal {O} \left(\frac {d _ {h}}{\sqrt {T}}\right)
$$
where  $d_h = \max_m \{d_{h_m}\}$  is the maximum output layer size of the clients.
Remark 5.4. This theorem proves that we significantly relieve the slow convergence problem of ZOO-based VFL [41] from  $\mathcal{O}\left(\frac{d}{\sqrt{T}}\right)$  to  $\mathcal{O}\left(\frac{d_h}{\sqrt{T}}\right)$ .
6 Experiment
We conducted a systematic experiment on the SOTA and baseline VFL and our proposed framework. The primary objective of our study was to empirically validate the security measures of our approach and demonstrate its capability in reducing communication overhead. Furthermore, we conducted an ablation study to quantify the contributions of each component toward improving communication efficiency. Due to space limitations, extra experiment details, the CIFAR-10 experiments, extra experiments on less common datasets, and extra experiments on other aspects of our framework have been placed in appendix E. The source code for this project is available at the following URL: https://github.com/GanyuWang/VFL-CZ0F0.
6.1 Experiment Setups
Datasets The datasets were vertically partitioned among all participants in our experiments. Each client held a portion of the features of each sample, while the server held the corresponding labels. Both the server and the client kept the sample IDs during training. We utilized two datasets in our main experiments: MNIST [20] and CIFAR-10 [19]. For both datasets, we employed two clients in our experiments, with each client being responsible for half of the images<sup>8</sup> in the respective dataset. Therefore, we denoted these datasets as dist-MNIST and dist-CIFAR-10.
Models In our dist-MNIST experiment, we implemented a multilayer perception (MLP). Each client utilized a one-layer Fully Connected Layer (FCL), with an input size equal to the flattened local data input size, and an output size of 64. The server employed a two-layer FCL. The first layer concatenated all the outputs from all clients and produced embeddings with 128 neurons. The second layer outputs 10 neurons for prediction. ReLU [11] was used as the activation function for the server and clients. Practically, the server kept a table of the outputs from all clients for all of the samples, i.e.  $\tilde{\Phi}_i^t = [h_1(w_1^{t - \tau_{1,i}^t};x_{1,i}),\dots,w_M^{t - \tau_{M,i}^t}]$  was maintained by the server. The server updated the corresponding value in the table when it received  $h_m(w_m^{t - \tau_{m,i}^t};x_{m,i})$  from client  $m$  during each communication round. The clients' outputs from the table were then used for prediction. Regarding the dist-CIFAR-10 dataset, each client padded its half-image to full size and trained a ResNet-18 [15]. The server then summed the outputs from all the clients.
**Frameworks for Comparison** We compare our framework with other SOTA VFL frameworks including "split-learning" [34], "compressed-VFL" [5], "Syn-VFL-ZO", "VAFL" [6], "ZOO-VFL" [41]. All frameworks utilized identical base models and training procedures. While some frameworks also introduced alternative VFL settings, e.g. server and clients both processing the labels, we only focus on the VFL setting where the server stores the labels and multiple clients possess non-intersecting features.
Training Procedures For the experiment applying the split MLP model on dist-MNIST, a batch size of 64 was utilized, and the model was trained for 100 epochs. For the experiment applying the ResNet-18 on dist-CIFAR-10, a batch size of 128 was used, and the model was trained for 50 epochs. The learning rate  $\eta$  was chosen from  $[0.1, 0.01, 0.001, \dots]$ , and  $\mu$  for ZOO was chosen from  $[1, 0.1, 0.001, 0.0001, \dots]$ . We use a uniform scale compressor [2] on the forward and backward messages with different compression bits [8, 4, 2, 1]. The number of sampling  $q$  used for Avg-RandGradEst in the dist-MNIST experiment was set to [10, 100], while in the dist-CIFAR-10 experiment, it was set to either [100, 500, 1000]. The sampling distribution is  $\mathcal{U}(\mathcal{S}(0, 1))$ . We used a vanilla SGD with a fixed learning rate for all VFL frameworks to ensure a fair comparison. The reported test accuracy values in the tables have been obtained from five independent runs.
Details of the feature splitting can be found in Appendix E.1
9This is the synchronous version of ZOO-VFL, the algorithm is in Appendix E.1
Table 2: Sampling Times of Avg-RandGradEst and the Corresponding Differential Privacy Guarantee
q	σ2	ζ	δ	ε	δ'	ε̅	δ̅
10	1.61	20.1	10-6	2.8 × 10-3	10-6	85.9	9.4 × 10-2
100	0.15	7.3	10-6	3.3 × 10-3	10-6	93.6	9.4 × 10-2
6.2 Evaluation on the Privacy Protection
In this part, we evaluate the privacy protection of our method and Gaussian Mechanism. The major difference between the ZOO and Gaussian Mechanism in DP is that the variance of the stochastic gradient estimation (ZOO) is intrinsically controlled by the hyperparameter of the ZOO and the characteristic of the objective function, while in the Gaussian Mechanism, the variance is a parameter that directly controls the magnitude of the noise added.
We obtain the DP guarantee w.r.t. the sampling times of ZOO in the dist-MNIST experiment and report the result in Table 2. We estimate  $\sigma^2$  by freezing the model and repeatedly applying Eq.2 100 times, then calculating the variance of the estimated gradients. We obtain the maximum gradient norm  $\zeta$  by recording the norm throughout the
entire training process<sup>10</sup>. We set  $\delta = 10^{-6}$  and calculate  $\epsilon$  based on the given sampling times  $q$ . The iteration  $T$  was set to the total iteration of 50 epochs, which is 93800.
Figure 2: The Utility Comparing with the VAFL with Gaussian Mechanism
In the second part, we compared the utility of our framework with the VAFL[6] using the Gaussian Mechanism and gradient clipping [1] on the partial derivative w.r.t. the client's output with the corresponding  $(\epsilon, \delta)$ -DP. We set the gradient norm clipping bound to 0.1 And we plotted the training accuracy for each epoch in Figure 2. As illustrated in the figure, our scheme exhibits comparable utility to VAFL with the same privacy guarantee.
6.3 Evaluation of Training Efficiency and Communication Cost
In this part, we compare our framework with SOTA VFL in the effectiveness of training and communication efficiency. We apply the best tuning for all other frameworks with respect to test accuracy and communication efficiency. (The experiment for dist-CIFAR-10 is in Appendix E.3)
Figure 3-(a) demonstrates that our VFL-CZOFO framework has the fastest convergence rate comparable to VAFL, demonstrating the superiority of our cascade hybrid optimization method. It is worth noting that the pure-ZOO-based VFL suffers from a slower convergence compared to other frameworks, even with an MLP model. Figure 3-(b) plots the total communication cost against the training accuracy, where the crosses indicate the point where the curve reaches  $95\%$  training accuracy for MNIST. It shows that our VFL-CZOFO framework achieves the same level of convergence with much lower communication costs.
Table 3 report the important statistic about the training. As shown in the table, our framework significantly outperforms other communication-efficient frameworks in terms of communication cost.
(a) MNIST by epochs
(b) MNIST by comm. cost
Figure 3: Compare with other VFL Frameworks
Alternatively, one can also apply gradient clipping[1] to get a better privacy guarantee by bounding  $\zeta$ , however, we want to focus on the intrinsic privacy of our method, therefore only record the maximum norm.
Table 3: Evaluating the Test Accuracy and the Comm. Cost
Privacy	Test Accuracy	Cost(60%)	Cost (80%)	Cost(90%)	Cost (95%)	Cost (total)
Split learning [34]	✗	98.03 ± 0.07	175 MB	234 MB	937 MB	1758 MB	5859 MB
Compressed-VFL [5]	✗	98.17 ± 0.13	109 MB	330 MB	440 MB	1099 MB	3663 MB
VAFL [6]	✗	97.36 ± 0.14	123 MB	184 MB	307 MB	1106 MB	6144 MB
VAFL[6]+DP[1]	✓	95.81 ± 0.20	123 MB	184 MB	799 MB	1413 MB	6144 MB
Syn-ZOO-VFL	✓	84.51 ± 0.72	1406 MB	2520 MB	-	-	-
ZOO-VFL [41]	✓	85.62 ± 0.45	860 MB	1300 MB	-	-	-
Ours	✓	95.00 ± 0.21	16 MB	24 MB	39 MB	205 MB	790 MB
6.4 Ablation Study
Minimal Zeroth-Order Optimization In this section, we evaluate the impact of different ZOO parameters on convergence and communication costs. We did not apply compression in this section. We plot the training curve against epochs in Figure 4-(a), and against the logarithm base 10 of the total backward message size in Figure 4-(b). Table 4 reports the test accuracy and communication costs. (The experiment for dist-CIFAR-10 can be found in Appendix E.3.2 with a similar conclusion.)
Figure 4 illustrates how increasing the number of sampling  $q$  can enhance the convergence of our proposed method. When  $q$  is set to 100, the training curve of our method closely matches that of the VAFL method. Notably, the use of ZOO in VFL results in a significant reduction in communication cost, reducing the backward cost by a factor of 40 compared to VAFL.
(a) MNIST by epochs
Figure 4: Ablation Study on ZOO
(b) MNIST by comm. cost
Table 4: Ablation Study on ZOO
ZO Type	Test Accuracy	Backward Cost
FO	97.36 ± 0.14	3073 MB
AvgRand-100	95.30 ± 0.25	75 MB
AvgRand-10	93.33 ± 0.55	7.5 MB
AvgRand-1	79.66 ± 0.96	0.75 MB
Compression of the Communication In this section, we investigate the impact of compression on convergence and communication costs using Avg-RandGradEst with 100 samplings. We evaluate different compression schemes on either the forward (F) or backward (B) message. Figure 5 presents the training curve plotted against the epoch and the log10 of the communication cost of the message with compression. Table 5 displays the test accuracy and the total cost at the end of the 100-epoch training period. (corresponding dist-CIFAR-10 experiment can be found in Appendix E.3.2.)
The results show that increasing the compression rate reduces communication costs but leads to a decline in test accuracy. Notably, even when compressing the gradient information to 2 bits, there is no significant decrease in test accuracy, and the message size is significantly reduced to  $5.4\mathrm{MB}$ . Hence, a trade-off can be identified in practical application to achieve a balance point.
(a) MNIST by forward cost
Figure 5: Ablation Study on Compression
(b) MNIST by backward cost
Table 5: Ablation Study on Compression
Compression Type	Test Accuracy	Message Cost
No Compression	95.30 ± 0.25	3073 MB
F-8 bits	95.02 ± 0.17	769 MB
F-4 bits	93.35 ± 0.23	386 MB
F-2 bits	79.10 ± 0.41	194 MB
F-1 bit	30.20 ± 0.45	96 MB
B-8 bits	94.58 ± 0.21	19 MB
B-4 bits	94.50 ± 0.10	10 MB
B-2 bits	94.16 ± 0.16	5.4 MB
B-1 bit	90.67 ± 0.22	3.1 MB
While the utilization of ZOO improves communication efficiency and security, it comes at the expense of increased computational costs for the server. Specifically, the server needs to perform  $q$  extra forward propagations on its local model compared to other FOO-based VFL methods. A more comprehensive analysis and supplementary experiments can be found in Appendix E.2. However, it is important to note that the server typically has higher computational capabilities, which makes the associated costs manageable and acceptable.
We propose a solution in VFL that improves communication efficiency and privacy simultaneously. Our method has been theoretically proven to outperform ZOO-based VFL in terms of convergence, while also providing proof of the intrinsic differential privacy guarantees. Through our experiments, we demonstrate that our method substantially reduces the communication cost of VFL in comparison to the state-of-the-art communication-efficient VFL. Furthermore, our method achieves comparable utility to VFL models that offer the same level of privacy guarantee.
This work has been supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program. Bin Gu was partially supported by the National Natural Science Foundation of China under Grant 62076138.
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] William Ralph Bennett. Spectra of quantized signals. The Bell System Technical Journal, 27(3):446-472, 1948.
[3] Albert S Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22(2):507-560, 2022.
[4] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560-569. PMLR, 2018.
[5] Timothy J Castiglia, Anirban Das, Shiqiang Wang, and Stacy Patterson. Compressed-vfl: Communication-efficient learning with vertically partitioned data. In International Conference on Machine Learning, pages 2738-2766. PMLR, 2022.
[6] Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. Vafl: a method of vertical asynchronous federated learning. arXiv preprint arXiv:2007.06081, 2020.
[7] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211-407, 2014.
[8] Wenjing Fang, Derun Zhao, Jin Tan, Chaochao Chen, Chaofan Yu, Li Wang, Lei Wang, Jun Zhou, and Benyu Zhang. Large-scale secure xgb for vertical federated learning. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 443-452, 2021.
[9] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
[10] Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium (USENIX Security 22), Boston, MA, 2022.
[31] Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning, pages 15750-15769. PMLR, 2022.
[32] Thilina Ranbaduge and Ming Ding. Differentially private vertical federated learning. arXiv preprint arXiv:2211.06782, 2022.
[33] Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.01451, 2022.
[34] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564, 2018.
[35] Yujia Wang, Lu Lin, and Jinghui Chen. Communication-efficient adaptive federated learning. arXiv preprint arXiv:2205.02719, 2022.
[36] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454-3469, 2020.
[37] Kang Wei, Jun Li, Chuan Ma, Ming Ding, Sha Wei, Fan Wu, Guihai Chen, and Thilina Ranbaduge. Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309, 2022.
[38] Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, and Zhiyuan Zong. Privacy leakage of real-world vertical federated learning. arXiv preprint arXiv:2011.09290, 2020.
[39] Kai Yang, Tao Fan, Tianjian Chen, Yuanming Shi, and Qiang Yang. A quasi-newton method based vertical federated learning framework for logistic regression. arXiv preprint arXiv:1912.00513, 2019.
[40] Ram Zamir and Meir Feder. On lattice quantization noise. IEEE Transactions on Information Theory, 42(4):1152-1159, 1996.
[41] Qingsong Zhang, Bin Gu, Zhiyuan Dang, Cheng Deng, and Heng Huang. Desirable companion for vertical federated learning: New zeroth-order gradient based algorithm. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2598-2607, 2021.
[42] Qingsong Zhang, Bin Gu, Cheng Deng, Songxiang Gu, Liefeng Bo, Jian Pei, and Heng Huang. AsySQN: Faster vertical federated learning algorithms with better computation resource utilization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3917-3927, 2021.
[43] Qingsong Zhang, Bin Gu, Cheng Deng, and Heng Huang. Secure bilevel asynchronous vertical federated learning with backward updating. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10896-10904, 2021.
[44] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610, 2020.
[45] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.