Title: A Unified Solution for Privacy and Communication Efficiency in Vertical Federated Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vertical Federated Learning

Zeroth Order Optimization

Communication Efficiency

Privacy
Abstract: Vertical Federated Learning (VFL) is a collaborative machine learning paradigm that enables multiple participants to jointly train a model on their private data without sharing it.
To make VFL practical, privacy security and communication efficiency should both be satisfied. Recent research has shown that Zero-Order Optimization (ZOO) in VFL can effectively conceal the internal information of the model without adding costly privacy protective add-ons, making it a promising approach for privacy and efficiency.
However, there are still two key problems that have yet to be resolved. First, the convergence rate of ZOO-based VFL is significantly slower compared to gradient-based VFL, resulting in low efficiency in model training and more communication round, which hinders its application on large neural networks. Second, although ZOO-based VFL has demonstrated resistance to state-of-the-art (SOTA) attacks, its privacy guarantee lacks a theoretical explanation.
To address these challenges, we propose a novel cascaded hybrid optimization approach that employs a zeroth-order (ZO) gradient on the most critical output layer of the clients, with other parts utilizing the first-order (FO) gradient. This approach preserves the privacy protection of ZOO while significantly enhancing convergence.
Moreover, we theoretically prove that applying ZOO to the VFL is equivalent to adding Gaussian Mechanism to the gradient information, which offers an implicit differential privacy guarantee. 
Experimental results demonstrate that our proposed framework achieves similar utility as the Gaussian mechanism under the same privacy budget, while also having significantly lower communication costs compared with SOTA communication-efficient VFL frameworks.

A Unified Solution for Privacy and Communication Efficiency in Vertical Federated Learning

Abstract

Vertical Federated Learning (VFL) is a collaborative machine learning paradigm that enables multiple participants to jointly train a model on their private data without sharing it. To make VFL practical, privacy security and communication efficiency should both be satisfied. Recent research has shown that Zero-Order Optimization (ZOO) in VFL can effectively conceal the internal information of the model without adding costly privacy protective add-ons, making it a promising approach for privacy and efficiency. However, there are still two key problems that have yet to be resolved. First, the convergence rate of ZOO-based VFL is significantly slower compared to gradient-based VFL, resulting in low efficiency in model training and more communication round, which hinders its application on large neural networks. Second, although ZOO-based VFL has demonstrated resistance to state-of-the-art (SOTA) attacks, its privacy guarantee lacks a theoretical explanation. To address these challenges, we propose a novel cascaded hybrid optimization approach that employs a zeroth-order (ZO) gradient on the most critical output layer of the clients, with other parts utilizing the first-order (FO) gradient. This approach preserves the privacy protection of ZOO while significantly enhancing convergence. Moreover, we theoretically prove that applying ZOO to the VFL is equivalent to adding Gaussian Mechanism to the gradient information, which offers an implicit differential privacy guarantee. Experimental results demonstrate that our proposed framework achieves similar utility as the Gaussian mechanism under the same privacy budget, while also having significantly lower communication costs compared with SOTA communication-efficient VFL frameworks.

1 Introduction

Federated Learning [30, 22, 23, 18, 31] is an emerging technique that has raised wide attention recently. It has become one of the most important distributed learning frameworks for enabling multiple data holders to train a model collaboratively without sharing their local privacy data explicitly.

Server

iFO

izO

IFast

ISlow

IFast

Embedding

Partialderivative

Embeddings

ZOinformation

Embedding

ZOinformation

[画]

hm(x)

[画]

画

hm（x+μu)

[·]

[画]

hm(x)

[.·]

ohn

[m]

hm(x)

Privacy secure

Privacy secure

Risk of privacy leakage

Comm. efficient

Comm.efficient

Clients

IFO

ZO

Slow

Fast

Slow

FO

Fast

(a)FOO-basedVFL

(b)ZOO-based VFL

(c) VFL-CZOFO (Ours)

Our research focuses on VFL, where each client possesses all the data points, but only a nonintersecting subset of the features (vertically distributed). In VFL, all participants collaborate to train a single global model. The client trains a feature extraction model that maps its local data sample to embeddings, and the server collects the embeddings from all clients to make a prediction [21, 34, 6, 39, 16, 37, 14, 42, 43, 13].

Ensuring both privacy and computation-communication efficiency is crucial for practical VFL implementations. One common approach to achieve this is starting with a basic VFL [5, 6, 28] and subsequently applying privacy protection techniques, such as Secure Multiparty Computation (SMC) [8], Differential Privacy (DP) [32, 36], and Homomorphic Encryption (HE) [36, 32] to protect privacy. However, the addition of these protection techniques often increases computationcommunication costs. To avoid these costly protective add-ons, research attention has turned to Zero-Order Optimization (ZOO). Literature reports that ZOO-based VFL is able to conceal the internal information of the model (gradient/model parameter) making the SOTA privacy inference attack ineffective [41]. This finding highlights the potential of ZOO for efficient and secure VFL.

However, two critical problems remained. First, the convergence rate of ZOO-based VFL is significantly slower [41] compared to VFL frameworks optimized with gradient, resulting in a high number of communication rounds and increased communication costs. Furthermore, the convergence rate is negatively correlated to the size of the parameters optimized using ZOO [41, 27, 12], which hinders the application of ZOO-based VFL on large neural networks. Second, although ZO-based VFL has demonstrated resistance to SOTA attacks, its privacy guarantee has not been theoretically explained.

To solve the first problem, we reevaluate the advantages and disadvantages of applying first-order (FO) or zero-order (ZO) gradient methods for each component of the VFL, and find an optimal balance between efficiency and privacy protection. This analysis leads to a novel cascaded hybrid optimization framework that efficiently solves the convergence problems while fully preserving privacy protection. Specifically, our proposed framework (VFL-CZOFO) utilizes the ZO gradient for optimizing the most critical output layer of the clients, while other parts are optimized with the FO gradient 2 . To address the second problem, we theoretically prove that applying ZOO on VFL is equivalent to adding Gaussian Mechanism [7] to the gradient information and can provide a corresponding intrinsic ( ϵ, δ )-DP guarantee.

For a clear illustration, a schematic diagram of our proposed framework is presented in Figure 1-(c). The advantage of our framework is two-fold. First, it greatly alleviates the slow convergence problem of ZOO-based VFL (Figure 1-b), where our framework's convergence is solely limited by the output size of the clients, rather than the parameter size of the entire model. Second, compared with the FOO-based VFL (Figure 1-a), the connection layer of our framework used ZOO, allowing the backward messages to be solely losses, thus inheriting the privacy protection of the ZOO-based VFL.

The primary innovations of our paper are as follows: 1) Our framework employs a novel cascaded hybrid optimization method, in which different optimization methods are applied to different layers of the global model in each iteration, which significantly improves the convergence rate of ZOO-based VFL while preserving privacy. To the best of our knowledge, no prior research in VFL has proposed a similar method that cascaded a hybrid optimization and exploits its advantages. 2) We provide

a theoretical explanation of the intrinsic privacy guarantee of ZOO, based on ( ϵ, δ )-DP, which is fundamentally different from the common DP mechanism, where noise is added to items afterward.

In summary, the contributions of our paper are:

We propose the cascade hybrid optimization method in VFL, where ZOO is applied to the most essential part of the VFL, which provides intrinsic privacy protection and significantly improves the convergence rate compared with the ZOO-based VFL.

Theoretically analysis shows: 1) the intrinsic ( ϵ, δ )-differential privacy guarantee provided by ZOO within our framework, 2) the convergence of our framework and its superiority compared with the ZOO-based VFL, 3) the compatibility of communication compression for both forward and backward messages within our framework.

Extensive experiments show: 1) with the same ( ϵ, δ )-differential privacy guarantee, our method can achieve a similar utility as the Gaussian Mechanism, 2) Our method significantly reduces the communication cost compared with the SOTA communication-efficient VFL.

2 A Detailed Comparison with SOTA VFL Frameworks

Table 1 presents a comparison of our VFL framework with other SOTA VFL frameworks, including 'split-learning' [34], 'compressed-VFL' [5], 'Syn-ZOO-VFL" 3 , 'VAFL' [6], 'ZOO-VFL" [41]. In the table, Asyn/Syn are abbreviations for asynchronous and synchronous. The T is the iteration of the model. d and d h represent the size of the entire global model's parameter and the size of the client's output embeddings, respectively. The 'communication size per iteration' column summarizes the number of elements in the original messages before any post-processing, such as compression. B is the batch size. q is the sampling times for multiple point estimation of ZOO [27].

Compared to the FOO-based VFL [34, 5, 6] our framework has two distinct advantages. First, our framework leverages the intrinsic privacy protection of ZOO, which is a result of its stochastic characteristics and the concealment of internal information. Second, we leverage the advantage of ZOO to reduce the communication cost from the server to the client where the backward message consists of merely q elements, which is typically significantly smaller than the FO-based VFL with d h B elements. The key difference between our framework and the FOO-based framework is that our approach reduces communication costs while simultaneously ensuring privacy protection. In contrast, FOO-based VFL typically shares internal information (parameters/gradient) [6, 5] and then applies for extra privacy protection mechanism [36, 32].

Compared with the ZOO-based VFL [41], this framework and ours both leverage the privacy protection of ZOO in VFL. However, ZOO-based VFL suffers from a slow convergence rate of O ( d/ √ T ) which is constrained by the global model parameter size d . This hinders its effectiveness when dealing with 'larger' networks containing millions of parameters, causing higher communication costs due to increased communication rounds. To address this issue, we applied ZOO to the crucial connection layer between the server and the client. By doing so, we maintained the privacy protection offered by ZOO while significantly mitigating the slow convergence problem, reducing the constraint from the entire global model size d to merely the output size of the clients d h .

3 Method

Problem Definition In the VFL framework, there is one server and M clients 4 . The server holds the label y i for each sample i ∈ [ n ] , ( [ n ] ≜ { 1 , 2 · · · n } ), while each client holds a non-intersecting feature set for all the samples. Specifically, the features for sample i on client m are denoted as x m,i . Client m 's model, h m ( w m ; x m,i ) , is parameterized by w m and takes local feature x m,i as input, outputting the embedding. The server inputs the embeddings from all clients into its own model, which is parameterized by w 0 , and then calculates the losses for updating the parameters. The VFL framework can be viewed as solving a finite-sum problem in composite form:

where w = [ w 1 , · · · w M ] denotes the parameter for all clients, X denotes the entire feature set across all clients, and y denotes the labels for all samples. For notation brevity, we define the outputs from all clients as Φ i ( w ) = [ h 1 ( w 1 ; x 1 ,i ) , · · · , h M ( w M ; x M,i )] = [ h 1 ,i , . . . , h M,i ] . Therefore, the loss for the i -th sample 5 can be represented as f i ( w 0 , Φ) . L ( · ; y true ) is the loss function.

Cascaded and Minimal Application of ZOO on the Connection Layer Applying ZOO on VFL resolves the privacy leakage issue from the gradient [41], but simply applying ZOO to the entire VFL has a slow convergence problem, leading to a high number of communication rounds. Specifically, the major problem of ZOO is that with more dimensions on the gradient that need to be estimated, more variance will be introduced in the estimation of the gradient [27, 3]. This is especially problematic for machine learning scenarios where even relatively small models may have millions of parameters. To address the slow convergence problem of ZOO-based VFL, we precisely apply ZOO to the client's output layer, while the rest of the model utilizes the FO gradient. This approach significantly improves the convergence of the ZOO-based VFL, since only a small portion of the entire model uses ZOO.

ZOO[12] estimates the gradient of a function by sampling a random perturbation within the function's domain of definition and evaluating its shift on the value domain. To estimate the partial derivative with respect to the output layer of the client's model, we apply the average random gradient estimation (Avg-RandGradEst) [27, 25] which utilizes q i.i.d. samples { u j m,i } q j =1 on the objective function:

where f i ( h m,i + µ m u m,i ) is the abbreviation for f i ( w 0 , h 1 ,i , · · · h m,i + µ m u m,i · · · , h M,i ) , d h m is the size of the output layer of client m . ϕ ( d h m ) is a dimension-dependent factor [26] related to the choice of the distribution p of the random vector u j m,i . If p is the multivariate uniform distribution U ( S ( 0 , 1)) on a unit sphere centered at 0 with radius 1 , then ϕ ( d h m ) = d h m . If p is the multivariate normal distribution N ( 0 , I ) , then ϕ ( d h m ) = 1 . q is the sampling times. The clients cannot calculate δ j m,i by themselves because the loss function f i ( · ) is held by the server, therefore the server has to send the δ j m,i back to the clients 6 . To implement multiple-point estimation, the server must transmit multiple distinct values of δ j m,i to the client, with the number of values sent equaling the number of sampling points q used in the Avg-RandGradEst.

The theoretical difficulty in analyzing our framework is that different optimization methods are cascaded within a single model. Unlike most VFL research [34, 5, 28, 6, 41] which updates the global model with a unified optimization method, e.g. ∇ f i ( w m ) (VAFL [6]) or the ˆ ∇ f i ( w m ) (ZOOVFL [41]), we apply a cascaded hybrid gradient via chain rule, i.e. ˆ ∇ h m f i ( h m,i ) · ∇ w m h m,i , where ˆ ∇ h m f i ( h m,i ) is the stochastic gradient estimator of the partial derivative w.r.t. the output layer of the client, and ∇ w m h m ( w m ) is the local gradient of the client.

Compress the Communication Furthermore, we demonstrate that compression is compatible with our approach, and can be applied to all communications in the framework, including the forward message [5] and the backward message [35]. We define the compressor of the client as C m ( · ) , and the compressor of the server as C 0 ( · ) . In each communication round, the client m sends compressed embeddings instead of the raw message to the server, and the server replies with the compressed messages. We apply the uniform scale compressor [2] in the experiment. However, other compression schemes such as Top-K [24], sign-SGD [4], and lattice quantization [40] can also be applied to the different parts of our framework to further reduce the message size.

The theoretical challenge in proving the convergence with compression lies in that two errors (forward and backward) are introduced in the analysis. The first error affects the embeddings sent from the client to the server, where the error influences the server's model input, creating uncertainty in the loss value and gradients for updating the parameter. We bound this error by imposing assumptions on the curvature of the loss function, i.e. an upper bound for the norm of the Hessian matrix. The second error affects the message sent from the server to the clients, which then updates using the hybrid gradient with compression error. As a result, there are three different sources of uncertainty in the clients' parameter update: the two mentioned above, as well as the error caused by ZOO.

Algorithm In this section, we propose our asynchronous VFL 7 with ZOO on the connection layer and compression on the communication messages. Starting with the client, which randomly selects a sample i and sends its local model output to the server. Receiving the forward message from the client m , the server calculates the δ j m,i with Avg-RandGradEst (Eq. 2) and group δ j m,i with j = 1 , 2 , · · · , q into a vector ∆ m,i . The server compresses ∆ m,i and sends it back to the client. Then the server and client update their local model with the 'gradient'.

Algorithm 1 VFL-CZOFO

4 Security Analysis

Threat Model We discuss privacy under two scenarios: 'honest-but-curious' and 'honest-butcolluded'. In both scenarios, all participants are assumed to follow the protocol and perform their assigned tasks. Under the 'honest-but-curious' scenario, a curious client attempts to obtain private information from other participants using the information they have received. In the "honest-butcolluded" scenario, some participants collude to obtain private information from other participants. The attacker in this scenario can access all information and messages from the colluding participants. In the "honest-but-colluded" scenario, the attacker's capability is either equal to or stronger than in the "honest-but-curious" scenario. This is because the attacker can acquire more information from other participants when colluding, therefore increasing their ability to infer sensitive data.

Theorem 4.1. Differential Privacy Guarantee of Sharing the ZO Gradient: Under the 'honestbut-colluded" threat model where the attacker can access all information of all clients through the entire training process, including the client's dataset, model, and the ZO gradient received from the server. Under algorithm 1, with q being sufficiently large, ζ being the maximum l 2 -norm of the partial gradient w.r.t. any client's output through the entire training. If the following condition holds:

we can derive that sharing the stochastic gradient estimation from the server ensures (¯ ϵ, ¯ δ ) -DP for all ϵ, δ, δ ′ > 0 , where ¯ ϵ = √ 2 T ln(1 /δ ′ ) ϵ + Tϵ ( e ϵ -1) , ¯ δ = Tδ + δ ′ .

Proof sketch: The proof can be found in Appendix C.1. Firstly, we establish that each client's update can be modeled with an ideal sequence, where the unbiased parameter is updated with the gradient of the smoothed loss function f u ,i ( w t 0 , Φ i ) = E u [ f i ( w 0 , Φ( w )) + µ u ] . Since Avg-RandGradEst in Eq. 2 is an average of q independently and identically distributed samples from f i ( · ) , applying the central limit theorem, the estimated gradient is normally distributed for sufficiently large q . We can then apply the differential privacy guarantee for the Gaussian Mechanism to the estimated gradient. Finally, the accumulated privacy guarantee is derived from the advanced composition theorem.

Remark 4.2 . This theorem tells that under the worse case where a strong attacker has obtained all of the ZO information ∆ m,i from all client m ∈ [ M ] , of all sample i ∈ [ N ] , during the entire training process t ∈ [ T ] , the attacker cannot differ a single data point in the dataset, with ( ¯ ϵ, ¯ δ )-DP guarantee. Remark 4.3 . Since the ZO gradient is shared from the server to clients, this theorem mainly provides a privacy guarantee for the label on the server.

Remark 4.4 . For a weaker attacker, where the attacker can only access the information from fewer clients ('honest-but-colluded'), or only one client ('honest-but-curious'), the privacy guarantee either remains the same or is strengthened.

Additionally, in Appendix C.2, we provided a detailed discussion of the privacy protection of our approach at the framework level, plus a discussion on the SOTA inference attacks under 'honest-butcurious" [10, 33, 45, 45, 44, 17, 9] and 'honest-but-colluded"[10, 45, 29, 38] threat model.

5 Convergence Analysis

Assumption 5.1. The formal definition and detailed discussion of the assumptions are in Appendix B. We assume that there is a feasible optimal solution for f ( · ) , ∇ f i ( · ) is L -Lipschitz continuous, f i ( · ) has an unbiased gradient, bounded Hessian H m and bounded block-coordinate gradient G m . The activation of the clients is independent, and the client has a uniformly bounded delay τ .

Theorem 5.2. Under assumption 5.1, to solve the problem 1 with algorithm 1, the following inequality holds.

where L ∗ = max m { L, L 0 , L m } , η 0 = η m = η , 1 p ∗ = min m p m , µ ∗ = max m { µ m } , d ∗ = max m { d h m } , G ∗ = max m { G 0 , G m } , H ∗ = max m { H 0 , H m } , E and Γ are the upper bound for the square of the norm of the forward and backward compression error respectively. T is the total number of iterations (communication rounds).

Remark 5.3 . If we choose η = 1 √ T and µ ∗ = 1 √ T . Design the compression to make E = O ( 1 √ T ) and Γ = O ( 1 √ T ) we can derive:

where d h = max m { d h m } is the maximum output layer size of the clients.

Remark 5.4 . This theorem proves that we significantly relieve the slow convergence problem of ZOO-based VFL [41] from O ( d √ T ) to O ( d h √ T ) .

6 Experiment

We conducted a systematic experiment on the SOTA and baseline VFL and our proposed framework. The primary objective of our study was to empirically validate the security measures of our approach and demonstrate its capability in reducing communication overhead. Furthermore, we conducted an ablation study to quantify the contributions of each component toward improving communication efficiency. Due to space limitations, extra experiment details, the CIFAR-10 experiments, extra experiments on less common datasets, and extra experiments on other aspects of our framework have been placed in appendix E. The source code for this project is available at the following URL: https://github.com/GanyuWang/VFL-CZOFO .

6.1 Experiment Setups

Datasets The datasets were vertically partitioned among all participants in our experiments. Each client held a portion of the features of each sample, while the server held the corresponding labels. Both the server and the client kept the sample IDs during training. We utilized two datasets in our main experiments: MNIST [20] and CIFAR-10 [19]. For both datasets, we employed two clients in our experiments, with each client being responsible for half of the images 8 in the respective dataset. Therefore, we denoted these datasets as dist-MNIST and dist-CIFAR-10.

Models In our dist-MNIST experiment, we implemented a multilayer perception (MLP). Each client utilized a one-layer Fully Connected Layer (FCL), with an input size equal to the flattened local data input size, and an output size of 64. The server employed a two-layer FCL. The first layer concatenated all the outputs from all clients and produced embeddings with 128 neurons. The second layer outputs 10 neurons for prediction. ReLU [11] was used as the activation function for the server and clients. Practically, the server kept a table of the outputs from all clients for all of the samples,

corresponding value in the table when it received h m ( w t -τ m,i m ; x m,i ) from client m during each communication round. The clients' outputs from the table were then used for prediction. Regarding the dist-CIFAR-10 dataset, each client padded its half-image to full size and trained a ResNet-18 [15]. The server then summed the outputs from all the clients.

Frameworks for Comparison We compare our framework with other SOTA VFL frameworks including 'split-learning' [34], 'compressed-VFL' [5], 'Syn-VFL-ZO' 9 , 'VAFL' [6], 'ZOOVFL" [41]. All frameworks utilized identical base models and training procedures. While some frameworks also introduced alternative VFL settings, e.g. server and clients both processing the labels, we only focus on the VFL setting where the server stores the labels and multiple clients possess non-intersecting features.

Training Procedures For the experiment applying the split MLP model on dist-MNIST, a batch size of 64 was utilized, and the model was trained for 100 epochs. For the experiment applying the ResNet-18 on dist-CIFAR-10, a batch size of 128 was used, and the model was trained for 50 epochs. The learning rate η was chosen from [0 . 1 , 0 . 01 , 0 . 001 , · · · ] , and µ for ZOO was chosen from [1 , 0 . 1 , 0 . 001 , 0 . 0001 , · · · ] . We use a uniform scale compressor [2] on the forward and backward messages with different compression bits [8 , 4 , 2 , 1] . The number of sampling q used for Avg-RandGradEst in the dist-MNIST experiment was set to [10 , 100] , while in the dist-CIFAR-10 experiment, it was set to either [100 , 500 , 1000] . The sampling distribution is U ( S ( 0 , 1)) . We used a vanilla SGD with a fixed learning rate for all VFL frameworks to ensure a fair comparison. The reported test accuracy values in the tables have been obtained from five independent runs.

6.2 Evaluation on the Privacy Protection

In this part, we evaluate the privacy protection of our method and Gaussian Mechanism. The major difference between the ZOO and Gaussian Mechanism in DP is that the variance of the stochastic gradient estimation (ZOO) is intrinsically controlled by the hyperparameter of the ZOO and the characteristic of the objective function, while in the Gaussian Mechanism, the variance is a parameter that directly controls the magnitude of the noise added.

Figure 2: The Utility Comparing with the V AFL with Gaussian Mechanism We obtain the DP guarantee w.r.t. the sampling times of ZOO in the dist-MNIST experiment and report the result in Table 2. We estimate σ 2 by freezing the model and repeatedly applying Eq.2 100 times, then calculating the variance of the estimated gradients. We obtain the maximum gradient norm ζ by recording the norm throughout the 10 -6 0 10 20 30 40 50 Epoch

1.0

0.8

accuracy

0.6

0.4

Ours ∈ = 85.9

Ours E = 93.6

0.2

entire training process . We set δ = 10 and calculate ϵ based on the given sampling times q . The iteration T was set to the total iteration of 50 epochs, which is 93800 .

In the second part, we compared the utility of our framework with the V AFL[6] using the Gaussian Mechanism and gradient clipping [1] on the partial derivative w.r.t. the client's output with the corresponding ( ϵ, δ ) -DP. We set the gradient norm clipping bound to 0 . 1 And we plotted the training accuracy for each epoch in Figure 2. As illustrated in the figure, our scheme exhibits comparable utility to V AFL with the same privacy guarantee.

6.3 Evaluation of Training Efficiency and Communication Cost

In this part, we compare our framework with SOTA VFL in the effectiveness of training and communication efficiency. We apply the best tuning for all other frameworks with respect to test accuracy and communication efficiency. (The experiment for dist-CIFAR-10 is in Appendix E.3)

Figure 3-(a) demonstrates that our VFL-CZOFO framework has the fastest convergence rate comparable to V AFL, demonstrating the superiority of our cascade hybrid optimization method. It is worth noting that the pure-ZOObased VFL suffers from a slower convergence compared to other frameworks, even with an MLP model. Figure 3(b) plots the total communication cost against the training accuracy, where the crosses indicate the point where the curve reaches 95% training accuracy for MNIST. It shows that our VFL-CZOFO framework achieves the same level of convergence with much lower communication costs.

Table 3 report the important statistic about the training. As shown in the table, our framework significantly outperforms other communication-efficient frameworks in terms of communication cost.

1.0

XXRXRXXXXXXXXXX

0.8

Train accuracy

0.6

0.4

Split-Learning

C-VFL

Syn-ZOO-VFL

0.2

VAFL

ZOO-VFL

?

VFL-CZOFO

0

5

10

15

20

25

30

Epoch

0.8

Train accuracy

0.6

0.4

Split-Learning

C-VFL

Syn-ZOO-VFL

0.2

VAFL

ZOO-VFL

VFL-CZOFO

0

500

1000

1500

2000

2500

3000

Communication Cost (MB)

6.4 Ablation Study

Minimal Zeroth-Order Optimization In this section, we evaluate the impact of different ZOO parameters on convergence and communication costs. We did not apply compression in this section. We plot the training curve against epochs in Figure 4-(a), and against the logarithm base 10 of the total backward message size in Figure 4-(b). Table 4 reports the test accuracy and communication costs. (The experiment for dist-CIFAR-10 can be found in Appendix E.3.2 with a similar conclusion.)

Figure 4 illustrates how increasing the number of sampling q can enhance the convergence of our proposed method. When q is set to 100, the training curve of our method closely matches that of the VAFL method. Notably, the use of ZOO in VFL results in a significant reduction in communication cost, reducing the backward cost by a factor of 40 compared to VAFL.

1.0

0.8

Train accuracy

0.6

0.4

FO

RandGrad-100

0.2

RandGrad-10

RandGrad-1

0

10

20

30

40

50

Epoch

(a) MNIST by epochs

(b) MNIST by comm. cost

1.0

0.9

Train accuracy

0.8

0.7

0.6

0.5

FO

0.4

RandGrad-100

RandGrad-10

0.3

RandGrad-1

2

-1

0

1

2

3

log10 of the total backward message size in MB

Compression of the Communication In this section, we investigate the impact of compression on convergence and communication costs using Avg-RandGradEst with 100 samplings. We evaluate different compression schemes on either the forward (F) or backward (B) message. Figure 5 presents the training curve plotted against the epoch and the log10 of the communication cost of the message with compression. Table 5 displays the test accuracy and the total cost at the end of the 100-epoch training period. (corresponding dist-CIFAR-10 experiment can be found in Appendix E.3.2.)

The results show that increasing the compression rate reduces communication costs but leads to a decline in test accuracy. Notably, even when compressing the gradient information to 2 bits, there is no significant decrease in test accuracy, and the message size is significantly reduced to 5.4 MB. Hence, a trade-off can be identified in practical application to achieve a balance point.

1.0

1.0

0.9

0.9

accuracy

0.8

accuracy

0.8

0.7

0.7

0.5

0.6

8bit

Nocompression

0.6

0.5

8 bit

Nocompression

0.4

4 bit

2 bit

0.4

4 bit

2 bit

0.3

1 bit

0.3

1 bit

0

1

2

3

1

0

1

2

log10of total forward messagesizein MB

logl0 of total backward message size in MB

(a) MNIST by forward cost (b) MNIST by backward cost

7 Limitation

While the utilization of ZOO improves communication efficiency and security, it comes at the expense of increased computational costs for the server. Specifically, the server needs to perform q extra forward propagations on its local model compared to other FOO-based VFL methods. A more comprehensive analysis and supplementary experiments can be found in Appendix E.2. However, it is important to note that the server typically has higher computational capabilities, which makes the associated costs manageable and acceptable.

8 Conclusion

We propose a solution in VFL that improves communication efficiency and privacy simultaneously. Our method has been theoretically proven to outperform ZOO-based VFL in terms of convergence, while also providing proof of the intrinsic differential privacy guarantees. Through our experiments, we demonstrate that our method substantially reduces the communication cost of VFL in comparison to the state-of-the-art communication-efficient VFL. Furthermore, our method achieves comparable utility to VFL models that offer the same level of privacy guarantee.

Acknowledgments and Disclosure of Funding

This work has been supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program. Bin Gu was partially supported by the National Natural Science Foundation of China under Grant 62076138.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

William Ralph Bennett. Spectra of quantized signals. The Bell System Technical Journal , 27(3):446-472, 1948.

Albert S Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics , 22(2):507-560, 2022.

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning , pages 560-569. PMLR, 2018.

Timothy J Castiglia, Anirban Das, Shiqiang Wang, and Stacy Patterson. Compressed-vfl: Communicationefficient learning with vertically partitioned data. In International Conference on Machine Learning , pages 2738-2766. PMLR, 2022.

Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. Vafl: a method of vertical asynchronous federated learning. arXiv preprint arXiv:2007.06081 , 2020.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Wenjing Fang, Derun Zhao, Jin Tan, Chaochao Chen, Chaofan Yu, Li Wang, Lei Wang, Jun Zhou, and Benyu Zhang. Large-scale secure xgb for vertical federated learning. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 443-452, 2021.

Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , pages 1322-1333, 2015.

Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium (USENIX Security 22), Boston, MA , 2022.

Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological cybernetics , 20(3-4):121-136, 1975.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Bin Gu, Zhiyuan Dang, Xiang Li, and Heng Huang. Federated doubly stochastic kernel learning for vertically partitioned data. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2483-2493, 2020.

Bin Gu, An Xu, Zhouyuan Huo, Cheng Deng, and Heng Huang. Privacy-preserving asynchronous vertical federated learning algorithms for multiparty collaborative learning. IEEE transactions on neural networks and learning systems , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yaochen Hu, Di Niu, Jianming Yang, and Shengping Zhou. Fdml: A collaborative machine learning framework for distributed features. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2232-2240, 2019.

Xiao Jin, Pin-Yu Chen, Chia-Yi Hsu, Chia-Mu Yu, and Tianyi Chen. Cafe: Catastrophic data leakage in vertical federated learning. Advances in Neural Information Processing Systems , 34:994-1006, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450, 2020.

Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learning on non-IID features via local batch normalization. In International Conference on Learning Representations , 2021.

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887 , 2017.

Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications. In International Conference on Artificial Intelligence and Statistics , pages 288-297. PMLR, 2018.

Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine , 37(5):43-54, 2020.

Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems , 31, 2018.

Yang Liu, Yan Kang, Liping Li, Xinwei Zhang, Yong Cheng, Tianjian Chen, Mingyi Hong, and Qiang Yang. A communication efficient vertical federated learning framework. Scanning Electron Microsc Meet at , 2019.

Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. Feature inference attack on model predictions in vertical federated learning. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) , pages 181-192. IEEE, 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750-15769. PMLR, 2022.

Thilina Ranbaduge and Ming Ding. Differentially private vertical federated learning. arXiv preprint arXiv:2211.06782 , 2022.

Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.01451 , 2022.

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564 , 2018.

Yujia Wang, Lu Lin, and Jinghui Chen. Communication-efficient adaptive federated learning. arXiv preprint arXiv:2205.02719 , 2022.

Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security , 15:3454-3469, 2020.

Kang Wei, Jun Li, Chuan Ma, Ming Ding, Sha Wei, Fan Wu, Guihai Chen, and Thilina Ranbaduge. Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309 , 2022.

Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, and Zhiyuan Zong. Privacy leakage of real-world vertical federated learning. arXiv preprint arXiv:2011.09290 , 2020.

Kai Yang, Tao Fan, Tianjian Chen, Yuanming Shi, and Qiang Yang. A quasi-newton method based vertical federated learning framework for logistic regression. arXiv preprint arXiv:1912.00513 , 2019.

Ram Zamir and Meir Feder. On lattice quantization noise. IEEE Transactions on Information Theory , 42(4):1152-1159, 1996.

Qingsong Zhang, Bin Gu, Zhiyuan Dang, Cheng Deng, and Heng Huang. Desirable companion for vertical federated learning: New zeroth-order gradient based algorithm. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2598-2607, 2021.

Qingsong Zhang, Bin Gu, Cheng Deng, Songxiang Gu, Liefeng Bo, Jian Pei, and Heng Huang. AsySQN: Faster vertical federated learning algorithms with better computation resource utilization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 3917-3927, 2021.

Qingsong Zhang, Bin Gu, Cheng Deng, and Heng Huang. Secure bilevel asynchronous vertical federated learning with backward updating. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10896-10904, 2021.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610 , 2020.

Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems , 32, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research proposes a novel cascaded hybrid optimization approach named VFL-CFOZO for VFL that combines zeroth-order (ZO) gradient and first-order (FO) gradient optimization techniques. The critical output layer of the clients uses the ZO gradient, while other parts utilize the FO gradient. This approach enhances convergence while maintaining privacy protection. Experimental results demonstrate that VFL-CZOFO achieves similar utility as the Gaussian Mechanism in DP privacy preservation while significantly reducing communication costs compared to state-of-the-art communication-efficient VFL frameworks.\n\nSTRENGTHS:\n- A novel idea and good motivation. Based on existing research about FOO and ZOO frameworks in VFL, this research focuses on privacy in transmitted parameters and efficiency on other layers to load balance.\n\n- Complete analysis and proofs. Compared to the previous works in ZOO-VFL, such as [40], this research supplies clear proof of DP privacy. \n\n- Moreover, convergence proofs are also sufficient.\n\nWEAKNESSES:\n- Lack of scalability. In VFL, the dimension of the dataset often increases with the scale of clients, which means higher sample times should be applied to guarantee convergence. Thus, the current experiments with two clients are insufficient in this aspect. The evaluation of the correlations of client size and “enough” sample times should be added.\n\n- The potential risk of offline clients. The identical random sequence demands that the server and client maintain high synchronization. If a client drops for a while and reconnects with high delay, renegotiating the random seed may be better than using the original random ID because the completeness and timeliness of the information in an offline client are usually uncertain for the server.   \n\n- Some minor mistakes. In the explanation of Figure 4 in Section 6.4, the symbol q is miswritten as p. Inconsistent expression of the paper. ‘VAFL-ZOFO’ in Figure 5 of Appendix E.4 but ‘VAFL-CZOFO’ throughout the paper. Table 1 of Appendix A.2 shows that the output embeddings without delay formula should be Φ(w^t) instead of the w^t with the wide tide.\n\nQUESTIONS:\n1. Why not add VAFL+DP to Figure 3 in Section 6.3? This work also incorporates privacy and communication efficiency. Though the paper is compared for training accuracy in 6.2, it needs to be clarified and intuitive enough. \n\n2. Is there any basis or analysis of the application of the chain rule on a cascaded hybrid gradient? [40] indicates that the chain rule is inapplicable between two ZOE gradients due to extra variance. However, this research directly uses it between FO and ZO gradients.\n\nLIMITATIONS:\nAs mentioned in Section 7, the extra computational costs for the server do limit the scalability and utility of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper proposes a solution for privacy and communication efficient vertical FL training framework by utilizing zero-order-optimisation (ZOO). Since the convergence of ZOO-based VFL is significantly slower than standard gradient-based VFL, the paper proposes to only implement ZOO on the cut layer.

STRENGTHS:
The paper provides a strong motivation and a realistic setup of VFL. It also provides a detailed comparison with the SOTA VFL framework in Section 2 with convergence rate. The method of apply different optimization for different layer is very interesting.

WEAKNESSES:
- The method proposes to use a compressor for communication since the ZOO will generate extra communication cost, but didn’t analyse which compressing method is the best for the method. 

- It seems from Fig 2 that the method give lower performance compared to Gaussian.

QUESTIONS:
- Why choose the uniform scale compressor in particular for the experiment? Have you compared it with other compressors?

- It mentions that the paper proposes an asynchronous VFL, but how does asynchronous work? To finish the forward prop, the model will need to have all the intermediate output from all clients. I'm surprised to see that the synchronous version (Syn-VFL-ZO) perform much worse than asynchronous version (the paper version). Is there any reason why? Also, the synchronous version doesn't seem to use the Avg-RandGradEst.

LIMITATIONS:
- Only 2 clients for experiments. Especially in this case, ZOO can potentially lower the performance when the number of clients increase. 

- Only work on the image datasets, which each client holds half of the image. This particular experimental setup is not realistic in the real-life scenarios.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is aiming at solving two critical problems of Vertical Federated Learning (VFL), the convergence rate of ZOO-based VFL and the privacy guarantee of ZOO-based VFL. This study provides a simple solution of using different optimizations i.e., the first-order optimization (FOO) and zeroth-order optimization method (ZOO) in the VFL framework, and theoretically explained the privacy guarantee with differential privacy. Experiments are conducted with regard to differential privacy, training efficiency, and communication cost which showed significant improvement in communication cost compared with SOTA and baselines.\n\nSTRENGTHS:\nThis study discusses the training efficiency and privacy in applying ZOO to VFL, which is a significant problem in this area. The idea of cascaded different optimization is novel and interesting. The motivation of balancing the advantages of FOO&ZOO sounds. I am appreciated that it proposes a simple solution that is easy to understand and will inspire more following studies in this area. The solution is validated from different dimensions and thus I think the effectiveness of the solution is reliable. \n\nThe paper is also theoretically contributed. Theorem 4.1 explains the privacy of ZOO. Theorem 5.2 guarantees training efficiency. The experiments are solid from my perspective, comprehensively covering the essential aspects of the algorithm, including the privacy budget of DP, the learning curve, and the communication cost at each stage. The method has high performance with a substantial improvement in communication efficiency, making this work great potential of being a new SOTA baseline.\n\nWEAKNESSES:\nAccording to Table 1, solely applying ZOO on the client layer only reduce the backward message size from $d_h B$ to $q$ compared with FOO-based VFL. However, the forward communication size was not improved. I agree with the improvement of your work in improving the convergence rate, however, a large reduction in the total communication cost could be due to the compression. Could you illustrate more on the contribution of compression and ZOO in reducing the total communication cost? \n\nAs you mention in #261-#263, the fundamental difference between ZOO and Gaussian Mechanism is that your privacy budgets ($\\epsilon, \\delta$) are implicitly controlled by the parameter of ZOO. Will this limit the application of your solution when a certain degree of privacy budget is required?\n\nQUESTIONS:\nIt is less clear to me whether the reduction of total communication cost is from “applying ZOO” or “applying compression”. The current ablation study did not tell the ratio of the reduction of the communication cost between “applying ZOO” and “applying compression”.\n\nHow your approach solves the problem if a certain privacy budget is required? Since the privacy budget of your framework is not directly controlled by the magnitude of the noise.\n\nLIMITATIONS:\nYes, the limitation was discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pioneering Zero-Order Optimization (ZOO)-based VFL algorithm that effectively ensures privacy preservation while significantly enhancing communication efficiency. Regarding privacy, the paper demonstrates theoretically that ZOO can inherently offer $(\\epsilon,\\delta)$-differential privacy, providing a strong foundation for understanding the privacy preservation achieved by ZOO. Concerning communication efficiency, the method ingeniously combines first-order and zero-order gradient optimization, resulting in remarkable improvements in training and communication efficiency. Moreover, the paper rigorously proves the convergence of the proposed algorithm. Extensive experiments are conducted, further affirming the superiority of the method.\n\nSTRENGTHS:\n1. The paper presents a novel VFL method that applies different optimization methods to different layers of the global model in each iteration, significantly improving the convergence rate of ZOO-based VFL while preserving privacy.\n2. The theoretical proof that ZOO can inherently offer $(\\epsilon,\\delta)$-differential privacy provides a strong foundation for understanding the privacy preservation achieved by ZOO in VFL.\n3. This paper conducts extensive experiments, offering concrete evidence of its effectiveness.\n\nWEAKNESSES:\n1. The experiments are currently conducted with only two clients, but further experiments with varying numbers of clients are necessary. \n2. The article divides experiments on different datasets into multiple chapters for presentation, which hinders a comprehensive display of the model. \n3. The computation cost of the server is extremely high, and it escalates with an increase in the number of clients due to the average random gradient estimation for each client.\n\nQUESTIONS:\n1. Does the number of clients significantly affect the computational efficiency?\n2. In Appendix E.4 and E.5, Figure 5 and Figure 6 present VFL-ZOFO, which is different from VFL-CZOFO. Do they have any differences?\n\nLIMITATIONS:\n1. The computation cost of the server is extremely high.\n2. This method is not suitable for many client situations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a hybrid Federated Learning (FL) framework, named VFL-CZOFO, which aims to provide intrinsic privacy protection while also significantly improving the convergence rate when compared to existing ZOO-based frameworks.\n\nSTRENGTHS:\n1. Faster Convergence: The paper demonstrates that VFL-CZOFO achieves faster convergence compared to other ZOO-based frameworks.\n\n2. Theoretical Solidity: The paper is theoretically sound, with a proof of convergence, and guarantees $(\\epsilon, \\delta)$-DP (differential privacy).\n\nWEAKNESSES:\nThe major concern lies in the experimental performance of VFL-CZOFO. \n\n1. The experiments only involve 2 clients, which are considered too small for a conclusive evaluation. The proposed algorithm's performance in a large-scale federated learning system remains unclear. \n\n2. The usage of only 2 datasets and lack of clarity regarding heterogeneity raise concerns about the model's generalizability and applicability in diverse scenarios.\n\n3. The choice of epsilon (around 90) for privacy evaluation seems excessively high, raising doubts about the actual privacy protection offered. Additionally, the training doesn’t seem to converge well at epoch 50 in Fig. 2, potentially affecting the total privacy budget.\n\n4. Table 3 shows that the proposed method sacrifices accuracy compared to the FOO-based method while improving communication cost. This trade-off is questionable, as accuracy is generally more critical in most cases.\n\nQUESTIONS:\nVFL-CZOFO is designed for better privacy protection while reducing communication costs. In the case where VFL-CZOFO guarantees a significantly smaller privacy budget than VAFL+Gaussian, what will the accuracy be?\n\nLIMITATIONS:\nThe paper introduces a framework that provides intrinsic privacy protection while also improving communication costs. However, there are weaknesses and questions listed above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}