CoPriv: Network/Protocol Co-Optimization for Communication-Efficient Private Inference
Authors: Wenxuan Zeng
Meng Li
Haichuan Yang
Wen-jie Lu
Runsheng Wang
Ru Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Private Inference
Network/Protocol Co-Optimization
Winograd Convolution
Structural Re-parameterization

Abstract: Deep neural network (DNN) inference based on secure 2-party computation (2PC) can offer cryptographically-secure privacy protection but suffers from orders of magnitude latency overhead due to enormous communication. Previous works heavily rely on a proxy metric of ReLU counts to approximate the communication overhead and focus on reducing the ReLUs to improve the communication efficiency. However, we observe these works achieve limited communication reduction for state-of-the-art (SOTA) 2PC protocols due to the ignorance of other linear and non-linear operations, which now contribute to the majority of communication. In this work, we present CoPriv, a framework that jointly optimizes the 2PC inference protocol and the DNN architecture. CoPriv features a new 2PC protocol for convolution based on Winograd transformation and develops DNN-aware optimization to significantly reduce the inference communication. CoPriv further develops a 2PC-aware network optimization algorithm that is compatible with the proposed protocol and simultaneously reduces the communication for all the linear and non-linear operations. We compare CoPriv with the SOTA 2PC protocol, CrypTFlow2, and demonstrate 2.1× communication reduction for both ResNet-18 and ResNet-32 on CIFAR-100. We also compare CoPriv with SOTA network optimization methods, including SNL, MetaPruning, etc. CoPriv achieves 9.98× and 3.88× online and total communication reduction with a higher accuracy compare to SNL, respectively. CoPriv also achieves 3.87× online communication reduction with more than 3% higher accuracy compared to MetaPruning.

CoPriv: Network/Protocol Co-Optimization for Communication-Efficcett Private Inference
Wenxuan Zeng Peking University zwx. andy@stu, pku. edu. en
Meng Li" Peking University meng 11 @pku. edu on
Haichuan Yang Meta Al chuan@metaa com
Wen-jie jie Lu Ant Group juhou. www.wligrannt....ommmm com
Runsheng Wang Peking University r. wang@pku edu. cn
Ru Huang
Peking University
ruhuang@pku, edu.
Abstract
Deep neural network (DNN) inference based on secure 2-party computation (2PC) offer cryptographicalllysseent
privacy protection but suffers from orders of magnitude latency overhead due to enormous communication. Previous works heavily rely proxy metric of ReLU counts approximate the communication overhead and focus on reducing the ReLUs to improve the communication effi- ciency. However, observe these works achieve limited communication reduction for state-of-thed tfrte-ff-hhe-rt (SOTA) 2PC protocols due to the ignorance of other linear and non-linear operations, which now contribute the majority of communication. In this work, we present CoPriv, framework that jointly optimizes the 2PC inference protocol and the DNN architecture. CoPriv features a new 2PC protocol for convo- lution based on Winograd transformation and develops DNN-aware optimization significantly reduce the inference communication. CoPriv further develops a 2PC-aware network optimization algorithm that is compatible with the proposed protocol and simultaneously reduces the communication for all the linear and non- linear operations. We compare CoPriv with the SOTA 2PC protocol, CrypTFlow2, and demonstrate 2. communication reduction for both ResNet-18 and ResNer- 32 on CIFAR-100 100. We also compare CoPriv with SOTA network optimization methods, including SNI., MetaPruning, etc. CoPriv achieves 9.98 and 3.88 online and total communication reduction with higher accuracy compared SNL, respectively. CoPriv also achieves 3.87 x online communication reduction with more than 3% higher accuracy compared to MetaPruning.
Introduction
Deep learning has been applied to increasingly sensitive and private data and tasks, for which privacy emerges one of the major concerns. To alleviate the privacy concerns when deploying the deep neural network (DNN) models, secure two-party computation (2PC) based DNN inference is proposed and enables cryptographicallysston privacy guarantee 39
the server owns a private model and the client private data. The server is willing to provide the machine learning service (MLaaS) but does not want to give it out directly. The client wants to apply the model on the private data without revealing as well. Secure 2PC frameworks can fulfill both parties' requirements: two parties can learn the inference results but nothing else beyond what can be derived from the results.
Secure 2PC helps solve the following dilemma
28
"Corresponinng author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The privacy protection of secure 2aC-ba inference is achieved at the cost of high communication complexity due to the massive interaction between the server and the client [31 48
+
25
This
leads to orders of magnitude latency gap compared to the regular inference on plaintext
[31
43
2PC-based inference usually has two stages, including pre-processing stage that generates the input independent helper data and online stage to process client's 's actual query [3]
48 43
25
24
Because the helper data is independent of client's query, previous works [48] 25 assume the pre- processing stage is offline and thus, focus optimizing the communication of the input dependent
stage. Specifically, 48 19 26
6 7 32 observe ReLU accounts for significant communication and ReLU count is widely used as proxy metric for the inference efficiency. Hence, the problem of improving inference efficiency is usually formulated as optimizing networks to have as few ReLUs possible. For example, DeepReduce [26] and SNL M7] achieve more than 4x and 16x ReLU reduction with less than 5% accuracy degradation on the CIFAR-10 dataset.
However, such assumption may no longer be valid for state-of-the-art (SOTA) 2PC protocols. We profile the communication of ResNet-18 with dif- ferent ReLU optimization algorithms, in-
7:11
Gazelle
Conoolation
Francttonn
wilh GC
Reductio
ReLU Comu. (MB)
cluding DeepReduce [26), SENet (32),
1114
Comen
and SNL 7 and observe very limited communication reduction as shown
Try with
Figure [b). This is because, hand, the communication efficiency of ReLU has been drastically improved over the past few years from garble cir- cuit (GC) to VOLE oblivious transfer (OT) as shown in TigureTaal a) [28 43 24), and ReLU only accounts for 40% and 1% of the online and total communication, respectively. On the other hand, recent studies suggest for MLaaS, it is more im- portant consider inference request arrival rates rather than studying individual inference isolation 18 In this case, the pre-processing communication, which is significantly higher than the online communication as shown Figure bb), cannot be ignored and may often be incurred online as there may not be sufficient downtime for the server to hide its latency [8]
10-1
Cheeral
VOLBED
01
[28]
[43]
[24]
DeepReDuce
SENeet (b)
SNL
(a)
Figure 1: (a) Communication for ReLU communication
has reduced by 131x from [28] to [24] (b) existing net-
work optimizations suffers from limited reduction on- line and total communication.
Therefore, improve the efficiency of 2PC-based inference, in this paper, we argue that both pre-processing and online communication are important and propose CoPriv to jointly optimize the 2PC protocol and the DNN architecture. CoPriv first optimizes the inference protocol for the widely used 3-by-3 convolutions based on the Winograd transformation and proposes series of DNN-aware protocol optimization. The optimized protocol achieves more than 2. communication reduction for ResNet models [20] without accuracy impact. For lightweight mobile networks, e.g., MobileNetV2 (66), we propose differentiable ReLU pruning and re- re-parameterization algorithm in CoPriv. Different from existing methods [26] 7 32), CoPriv optimizes both linear and non-linear operations to simultaneously improve both the pre-processing and online communication efficiency.
With extensive experiments, we demonstrate CoPriv drastically reduces the communication of 2PC-based inference. The proposed optimized protocol with Winograd transformation reduces the convolution communication by 2.1% compared to prior-art CrypTFlow2 [43 for both the baseline ResNet models. With joint protocol and network co-optimization, CoPriv outperforms SOTA ReLU- optimized methods and pruning methods. Our study motivates the community to directly use the communication the efficiency metric to guide the protocol and network optimization.
2 Preliminaries
2.1 Threat Model
CoPriv focuses on efficient private DNN inference involving two parites, i.e., Alice (server) and Bob (client). The server holds the model with private weights and the client holds private inputs. At the end of the protocol execution, the client leams the inference results without revealing any

---

Table 1: Comparison with prior-at methods.
Table 2: Notations used in the paper.
Method
Procoeal Opt.
NetworKOpt
Notatin
Mcuning
Cony. Trancation
ReLL
H,
Height width ourput feature map
BJ 29- SEWWESSS 524
C,K
Inpul and oulput channel number
12 Mit
A, B.G
Winograd transformation matriices Size of output tile and convolation filter
Number per input
"Cafre (oun)
X
Security parameter that messures the attack hardness
information to the server. Consistent with previous works [39
48
43
42
26
32
adopt an
honest-but-curiee but-curiouu security model in which both parties follow the protocol but also try to learn more from the information than allowed. Meanwhile, following [39
[3] 42], we assume trusted third party exists so that the helper data needs to be generated by the client and the server.
28
48
2.2 Arithmetic Secret Sharing
CoPriv leverages the 2PC framework based on arithmetic secret sharing (ArSS). Specifically, an l-bit value x is shared additively in the integer ring Zzz as the sum of two values, e.g., (a)) and (a) can be reconstructed as (x), + (a)) mod 2 In the 2PC framework, x is secretly shared with the server holding (z). and the client holding (.))..
ArSS supports both addition and multiplication the secret shares. Addition be conducted locally while multiplication requires helper data, which are independent of the secret shares and is The communication cost of single multiplication of secret shares is O(((A + I))- When multiplications share the same multiplier, instead of simply repeating the multiplication protocol for & times, (8 31 43] proposes a batch optimization algorithm that only requires O + t()) communication. enabling much more efficient batched multiplications.
generated through communication 4 42
Most 2PC frameworks use fixed-point arithmetic. The multiplication of two fixed-point values of 1-bit precision results in a fixed- fixed-point value of 2/-bit precision. Hence, in order to perform subsequent arithmetics, truncation is required to scale the value down to l-bit precision. Truncation usually executes after the convolutions and before the ReLUs. It requires complex communication and sometimes leads to even more online communication compared to ReLU as shown in Figure [ We refer interested readers to 43
for more details.
2.3 Winograd Convolution
We first summarize all the notations used in the paper in Table Then, consider ID convolution with m outputs and filter size of r, denoted as F(m,7) r). With regular convolution, F(m, r) requires multiplications between the input and the filter. With the Winograd algorithm, F(m,r) can be
computed differently. Consider the example of F(2,3) 3) below:
X
F1
22
W
[wo
W1
W22]
Y
[yo
V1]
Wo
21
02
mo
+ m1
+ m2
3/0
W1
22
23
m1
m2
m3
3/1
where mp, m1, m2, my are computed as
ma = (Fa x2 Wo
mi ( + Z2) - + w1 + w2)/2 m2 (x2 21) w1 + w22/22
my = (x11 x3) w2
With the Winograd algorithm, the number of multiplications is reduced from 6 in the regular con- volution to 4. Similar Winograd transformation can be applied to 2D convolutions by nesting algorithm with itself [34] The 2D Winograd transformation F(m x m, r x r), where the output tile size is m x m, the filter size is x r, and the input tile size n, where m r 1, can be formulated as follows,
Y
& = = AT[GGG o BBBBAA
(1)
where @ denotes the regular convolution and @ denotes element-wist matrix multiplication (EWMM). A, B, and G are transformation matrices that are independent of W and X and can be computed based on m and [3
3

---

With the Winograd algorithm, the multiplication of such a 2D convolution can be reduced from m ?? to n², ie.. (m + 1)2 the cost of performing more additions which can be computed locally in private inference scenario. While the reduction of multiplication increases with m, m = 2 and More details about
m are most widely used for the better inference precision [34 49
15
Winograd convolution are available in Appendix D
2.4 Related Works
To improve the efficiency of private inference, existing works can be categorized into three classes,
including protocol optimization 8 38
43
42
29
4 39], network optimization [26] 7
19
40) 37 6
32 27 51 35|, and joint optimization 48
25
In Table I we compare CoPriv with these works
qualitatively and as can be observed, CoPriv leverages both protocol and network optimization and can simultaneously reduce the online and total communication induced by convolutions, truncations, and ReLUs through network optimization. We leave more detailed review of existing works to Appendix A and more detailed comparison in terms of their techniques to Appendix
3 Motivation
In this section, we analyze the origin of the limited communication reduction of existing ReLU- optimized networks and discuss our key observations that motivates CoPriv.
the total communication is dominated by convolutions while the online com- munication cost of truncations and ReLUs are comparable. As shown in Figure [ the main operations that incurs high communication costs are convolutions, truncations, and ReLUs. With CrypTFlow2, over 95% communication is in the pre-processing stage generated by convolutions, while the truncations and ReLUs requires similar communication. This observation contradicts to the assumption of previous works 6 19
Observation 1:
on one hand, ReLU no longer dominates the online communication and simply reducing ReLU counts leads to diminishing return for online communication reduction; on the other hand, pre-processing communication cannot be ignored as not only consumes a lot of power in the server and client but also can slow down the online stage. As pointed out by a recent system analysis [18], for a pratical server that accepts inference request under certain arrival rates, the pre-processing stage is often incurred online as there is insufficient downtime to hide its latency. Therefore, we argue that both the total and the online communication are crucial and need to be reduced to enable more practical 2PC-based inference.
Observation 2: the communication cost scaling of convolution, truncation and ReLU is different with the network dimensions. The communication cost of different operations scales differently with the network dimensions. For a convolution, the major operations are multiplications and additions. While additions are computed fully locally, multiplications requires extra communication to generate the helper data. Hence, the communication cost for a convolution scales linearly with the number of multiplications and the complexity is O(CKr²(X + HW)) following (25). Both truncations and ReLUs are element-wise and hence, their communication cost scales linearly with the input feature size, i.c., O(HWC), C). Therefore, for networks with a large number of channels, the pre- pre-processing communication becomes even more dominating.
Since the communication is dominated by the multiplications in convolution layers, our intuition is to reduce the mudriplications by conducting more local additions for better communication efficiency. Meanwhile, as in Figure 2 MobileNetV2 are much more communication efficient compared to ResNets with a similar accuracy, and hence, becomes the focus of our paper.
Observation 3: ReLU pruning and network repaaaamterizaatinn can simultaneously reduce the online and total communication of all operations. Existing network optimizations either linearize the ReLUs selectively 226 7 40 linearization usually achieves a better accuracy but achieves limited communication reduction. In contrast, channel -wise or layer-wiss pruning can simultaneously reduce the communication for all the operators but usually suffers from larger accuracy degradation
37 6 32 or prune the whole channel or layer [36]. Selective ReLU
(26). We observe for an inverted residual block in MobileNetV2, as shown in Figure 3 if both of the two ReL.Us are removed, the block can be merged into a dense convolution after training. During training, this preserves the benefits of prratetrratttiizaaaion shown in RepVGG [14 to protect network accuracy, while during

---

Comm. (GB)
ResNa-50 75.39)
MobilcNetV21.. 1.4 (.4
1d Cone AKLU
1d Cory
10P
MolokeNaV--..00 (Ax--72.06))
3x DWY Core Trunc ReCU
DW Oote Truro
3x0 Coie- Truno
bit Cone Truro
Ie Coey Trune
10
Convolution
Truncation
ReLU
Figure 3: Illustration ReLU pruning and network re- ee-parameterization for communication reduction. DW means depth-wist and x Conv also called point-wise (PW) convolution.
Figure 2: Comparison of communi-
cation breakdown between ResNet-50 and MobileNetV2 on ImageNet.
Comum (GB)
Onlline
Pie proe.
ReoNet-18
29
MobilcNetV2
NetworkkProtocul Co-optimization Section 4.2 (step 1)
Network Optimization
Section 4.1
Section 4.2 (step 2)
Original Architecture
Winograd Transformation
Differentiable Proning
Network
Optimized Architecture
Protocol Optimization
Ro-parameterizaonn
Comm,
Online Pre-proe
Comm. (UI))
Prespene
Comm. (UW)
Oulise
Pro
Comn. (GB)
Oaline
ReaNe-18
60
RoNa--18
29
ResNet-11 18
29
ResNer-13 18
0,5
29
MobileNaV2
MobilNNeV2
MoiileNerV2
Figure 4: Overview CoPriv and the communication cost after each optimization step. The example of the communication cost is measured on the CIFAR- 100 dataset.
inference. the dense convolution is more compatible with optimized protocol and can reduce communication of truncations as well.
CoPriv: A New Paradigm Towards Efficient Private Inference
In this section, present CoPriv, pretorkl-netwerk co-optimization framework for communication efficient 2PC-based inference. The overview of CoPriv is shown Figure 4 Given the original network, first leverage the Winograd transformation with DNN-aware optimization to reduce the communication when computing 3-by-3 - convolutions without any accuracy impact (Section[4.) (.11) The optimized transformation enables reduce the total communication of ResNet-18 by 2.1 x. However, the communication reduction of MobileNetV2 is limited as the Winograd transformation can only be applied to the 3-by-3 depth-wise convolution, which only contributes to small portion of the total communication. Hence, for MobileNetV2, we propose an automatic and differentiable ReLU pruning algorithm (Section 4.2). By carefully designing the pruming pattern, CoPriv enables further network re- ce-paraetterization of the pruned networks. which reduces the online and total communication by 5x and 5 in the example, respectively, combining with our optimized convolution protocol.
4.1 Winograd Transformation for Protocol Optimization
We propose to leverage the Winograd transformation to reduce the communication when computing convolution. Following Eq. T for each 2D output tile, because A, B, G known publicly, the filter transformation GWG' G and feature transformation T B can be computed locally on the server and client [17]. Then, EWMM is performed together by the server and the client through communication while the final output transformation [:] can be computed locally as well.
For each output tile, as (m + 1)² multiplications are needed and there is no shared multiplier for batch optimization described in Section 2.2 the communication cost is O(X(m + 1)2) - Consider there are in total T [H/m] x [W/m tiles per output channel, C' input channels, and K output channels, the communication complexity of convolution layer now becomes O(TTC (m+r-1)²). To reduce the enormous communication, present the following optimizations.
For simplicity, omit the bit precision for all the downstream analysis.
5

---

Aggregation
FWMIN-GENN
nist
Thak
Ovige
Coesokaion
Figure 5: Winograd transformation procedure with tile aggregation.
Table 3: Communication complexity of different convolution types and the measured communication
for ResNet-18 18 block with 14x x26 feature dimension.
Conv. Type
Comm. Complexity
ResNet Block Comm. (GB)
Regular Cony.
O'CKF" HW)
23,74 (1x)
FW RWM -aaed Winograd Conv.
0 (m 1)2)
256 (10.78x
Tile Aggregation (GEMM-based)
(m 17CT(A + K)
10.45 (2.27x
min[[m 1r 1)CCTA K),
+ DNN-aware Sender Selection
(m+r- -1)CC((++)
7.76 (3.06x 4)
Communication reduction with tile aggregation While the Winograd transformation helps to reduce the total number of multiplications, in Table we observe the communication for a ResNet block actually increases. This is because the baseline protocol can leverage the batch optimization mentioned in Section Scction[..]jt reduce the communication complexity. To further optimize the Winograd- based convolution protocol adapt private inference, observe the following opportunities for the batch optimization: first, each transformed input tile needs to multiply all the K filters; secondly, each transformed filter needs to multiply all the transformed input tiles for given channel. Hence, for the t-th tile in f-th output channel, we re-write Eq. as follows:
Yy A U V
where Uyc denotes the transformed filter correspnding the f-th output channel and c-th input channel and Ve, denotes the transformed input of the t-th tile in the c-th input channel. Then, consider each pixel location, denoted V), within the tile separately, yielding:
((,,)) A AT (() 1 S KSSSKK K &
We illustrate this transformation procedure in Figure 5 Re-writing the EWMM into general matrix multiplication (GEMM) enables us to fully leverage the batch optimization and reduce the communi- cation complexity shown in Table 3 We can reduce the communication of ResNet block by 2.27x.
DNN-aware adaptive convolution protocol When executing the convolution protocol, both the server and the client can initiate the protocol. Because the input feature map and the filter are of different dimensions, we observe the selection of protocol intializer impacts the communication round as well as the communication complexity. While CrypTFlow2 always selects the server to initialize the protocol, we propose DNN architecture-aware convolution protocol to choose between the server and the client adaptively based on the layer dimensions to minimize the communication complexity. As shown in Table 3 the communication of the example ResNet block can be further reduced by 1.35x x with the adaptive protocol.
4.2 Differentiable ReLU Pruning and Network Re-Parametterzation
We now describe our network optimization algorithm to further reduce both pre-processing and online communication for MobileNetV2. The core idea is to simultaneously remove the two ReLUs

---

within an inverted residual block together, after which the entire block can be merged into dense convolution layer to be further optimized with our Winograd- maaograd-based protocol as Figure 3 For the example in Figure 4 both the pre-processing and online communication for MobileNetV? can be reduced by 5x. To achieve the above goal, the remaining questions to answer include: 1) which ReLUS to remove and how to minimize the accuracy degradation for the pruned networks, and 2)
how to re- re-parameterize the inverted residual block to guarantee functional correctness. Hence, we propose the following two-step algorithm.
Step communication-avaal differentiable ReLUU pruning identify "unimportant" activation functions, CoPriv leverages differentiable pruning algorithm. Specifically, CoPriv assigns an architecture parameter a(0 < a S 1) to measure the importance of each ReLU. During pruning, the forward function of a ReLU now becomes ReLU(()) + (1 - a) z. CoPriv jointly learns the model weights and the architecture parameters a. Specifically, given a sparsity constraint s, we propose the one-level optimization formulated as follows:
min LCF + Leomm 8.4. ||a||o 8, 8,a
(2)
where LCe is the task-specific cross entropy loss and Loomm is the communication-awwrt regulariza- tion to focus the pruning on communication-henn blocks and is defined as
Comm Σ ac (Comm, = Comm, = 0)),
where Comm,(a; 1) is the communication to compute the i-th layer when a, 1. We also show the effectiveness of communication-aww -warre regularization in the experimental results. To ensure the network after ReLU pruning can be merged, two ReLUs within the same block share the same a. During training, in the forward process, only the top-s a's are activated (i.e., a 1) while the remainings are set to O. In the backpropagation process, a's are updated via straight-through estimation (STE) [4.
Once the pruning finishes, the least important ReLUs are removed and we perform further finetuning to improve the network accuracy. Specifically, in CoPriv, we leverage knowledge distillation (KD) 21 to guide the finetuning of the pruned network.
Step 2: network re-parameterrzatiion The removal of ReL.Us makes the inverted residual block linear, and thus can further merged together into a single dense convolution. Motivated by the previous work of structural re-parameterization [16] 14 13 I0 [2]. we describe the detailed re-parameterization algorithm in Appendix C The re-parameterized convolution has the same number of input and output channels as the first and last point-wise convolution, respectively. Its stride equals the stride of the depth- wise convolution.
Experiments
5.1 Experimental Setup
We adopt CypTFlow2 protocol for the 2PC-based inference, and we measure the communication
and latency under LAN setting [43] with 377 MBps bandwidth and 0.3ms echo latency. For
Winograd, we implement F(2 x 2,3 x 3) and F(4 x 4,3 3 x 3) transformation for convolution with stride of and ((2 2, 3,37 3) transformation when stride is 2 [23]. We apply CoPriv to MobileNetV2 with different width multipliers on CIFAR-100 (30] and ImageNet [9] datasets. The details of our experimental setups including the private inference framework, the implementation of the Winograd based convolution protocol and the training details are available in Appendix
5.2 Biero-Benchhmrk on the Convolution Protocol with Winograd Transformation
We first benchmark the communication reduction of the proposed convolution protocol based on Winograd transformation. The results on ResNet-18 and ResNet-32 are shown in Figure a) and (b). As can be observed, the proposed protocol consistently reduces the communication to compute the convolutions. While the degree of communication reduction depends on the layer dimensions, average, 2.1 x reduction is achieved for both ResNet- 18 and R2sNet-32. We also
7

---

Convolation Comm. (M18)
Beweine CoypiFoo2
4090
+Wavgnd
IM
Itasatee Etysiilood
1290
(32,8,9)
(:12,
Block Idx
Block Idx
Feature Dimension
(a) ResNet-18
(b) ResNet-32
(c) Influence of Feature Dimension
Figure 6: Communication of the convolution protocol with Winograd transformation (a) ResNet-18 and (b) ResNet-32 on CIFAR-100; (c) comparison between F(2 x 2,3 x 3) and F(4 x 4, 3) with
different feature dimensions.
Top-1 Acc. (%)
Top-1 Acc. (%)
70
70
SNL (Mix2-077)
SNL (Mbx2-w0.75)
68
Cabri-w0.75
68
CaPriv-W0.75 (ouns)
SUNet
SENG
66
DepfieDDac
66
DeepleeDcce
SNI (Re4Ne=18)
SNL (Rechee 18)
Online Comm. (GB)
Total Comm. (GB)
Figure 7: Comparison with efficient ReL.U-pptimzze methods on CIFAR-100 dataset.
benchmark the Winograd-based protocol with output tile sizes of 2 and 4, i.e., F(2 2,3 x 3) and F(4 x 4,3 x 3). Figure shows that when the feature resolution is small, small tile size achieves more communication reduction while larger resolution prefers larger tile size. Hence, for the downstream experiments, 9 dataset, respectively.
(2 x 3) for CIFAR-100 and [(4 4, 3 x 3) for ImageNet
5.3 Benchmark with ReLU-Optimized Networks on CIFAR-100
benchmark CoPriv with different network optimization algorithms that focus on reducing ReLU counts, including DeepReDuce (26). SENet (32) and SNL 17.. on the CIFAR-100 dataset. As DeepReDuce, SENet and SNL use ResNet- 18 as the baseline model, we also train SNL algorithm on the MobileNerV2-ww.77 for comprehensive comparison.
Results and analysis From Figure 7 we make the following observations: 1) previous methods, in- cluding DeepReDuce, SENet, SNL, do reduce the pre-processing communication and their online communication reduction quickly saturates with notable accuracy degradation; 2) CoPriv achieves SOTA Pareto front of accuracy and online/total communication. Specifically, CoPriv outperforms SNL on ResNet- 18 with 4.3% higher accuracy as well as 1.55% x and 19.5x online and total communi- cation reduction, respectively. When compared with SNL. on MobileNetV2 2-- 75, CoPriv achieves 1.67% higher accuracy with 1 1.66x and 2.44x online and total communication reduction, respectively.
5.4 Benchmark ImageNet
We also benchmark CoPriv on the ImageNet dataset with the following baselines: lightweight mobile networks like No (2 with different capacities, Rel.U-optimized networks, including SNL 7 and SENee (2), and SOTA pruning methods, including uniform pruning and MetaPruning [36] We train both our CoPriv and SNL the MobileNetV2-w1.. with self distillation.
Results and analysis From the results shown in Figure we observe that 1) compared with SNL on MobileNetV2-w CoPriv achieves 1.4% higher accuracy with 9,98x and 3.88x online and total communication reduction, respectively; 2) CoPriv outperforms 1.8% higher accuracy compared

---

Top-1 Ace, (%)
70
SCC 0002-5-0
SNT (Miniin
SMC Oitior
65
Cahinota
Liakee Franeg
Viel
Mix
Mesthanng
Hoathskk
Racke 58at
Seddee Mard
Randil Vin
12
25
100
125
150
175
200
100
200
300
400
500
600
Online Comm. (GB)
Total Comm. (GB)
LAN Latency (6)
Figure 8: Comparison with SOTA efficient ConvNets, ReLU-optimized methods, channel-wise pruning MetaPruning [36] and uniform pruning on ImageNet dataset. We mark the four points of from left to right, respectively.
CoPriv as
Table 4: Ablation study of our proposed optimizations in CoPriv on CIFAR-100. Two sparsity constraints are on the two sides of '/", respectively.
Model
Online Communication (GB)
Total Communication (GB)
Baseline SNL (ResNct-18)
0.507
61.45
+Winograd
0.507
33.84
Baseline SNL (MobileNetV2)
0.530
7.710
+Winograd
0.530
7.132
MobrueNaV2+Prruiing
0.606 7..582
7.806 7.761
0.331 0.260
4.197 3,546
+Winograd (CoPriv)
0.331 0.260
3.154 2.213
with MobileNerV3--Small while achieving 9.41 and 1.67 online and total communication reduction; 3) CoPriv demonstrates its strong scalability for communication optimization. Compared to the baseline MobileNetV2-w CoPriv achieves 2.52x and 1.9x online and total communication reduction, respectively without compromising the accuracy; 4) when compared with SOTA pruning methods, for a high communication budget, CoPriv-w1.44. achieves 1.6% higher accuracy with 2.28 x and 1.08 x online and total communication reduction, compared with MetaPruning- 1.0g: for low communication budget, compared with MetaPruning-O.33 0.335x CoPriv-w1.A4 w1.4-D achieves 5.5% higher accuracy with 3.87 online communication reduction.
5.5 Ablation Study
Effectiveness of different optimizations in CoPriv To understand how different optimizations help improve communication efficiency, we add our proposed protocol optimizations step by step on both SNL and CoPriv, and present the results in Table 4 According to the results, we find that 1) our Winograd-based convolution protocol consistently reduces the total communication for both networks; 2) when directly applying the Winograd transformation on MobileNetV2, less than 10% communication reduction is achieved as Winograd transformation only helps the depth-wise convolution, which accounts for only a small portion pre-processing communication; 3) compared with SNL, CoPriv achieves higher accuracy with much lower communication. The findings indicate that all our optimizations are indispensable for improving the communication efficiency.
Block-wise communication comparison and visualization We compare and visualize the block- wise communication reduction of CoPriv on MobileNetVV2ww.77 on the CIFAR- dataset. From Figure 9 it is clear that our adaptive convolution protocol is effective for communication optimization, and different layers benefit from CoPriv differently. More specifically, for the total communication. block 4 benefits more from the Winograd transformation over the network re- parameterization, while block 16 benefits from both the re-parameterization and the adaptive convolution protocol. Both pruning and re- e-parameterization are important for online communication as they remove the communication ReLU and truncation, respectively. The results demonstrate the importance of all the proposed optimization techniques.

---

10
Total Comm. (MB)
10F
Gerraio
160
Online Comm. (MB)
120
100
80
60
20
10
14
17
18
Block
Figure 9: Block-wise visualization of online and total communication with different techniques on CoPriv-w0.75 on CIFAR-100.
5.6 Effectiveness of Communication-Aware Regularization
To show the effectiveness and importance of introducing communication-awse regularization Ceomm formulated Eq. we compare our pruning method with Leomm and pruning methods of SNL 7 and SENet (32 without Loomm in Figure TO As we can observe, Lcomm indeed helps to focus the pruning on the later layers, which incur more communication cost, and penalizes the costly blocks (e.g.. block # 16 and # 18). In contrast, SNL does not take communication cost into consideration and SENee (ReLU sensitivity based) focuses on pruning early layers with more RcLUs, both of which incur huge communication overhead.
Comm. Bottleneck
Total Comm. (MB)
1000
Original MBV2
Pruning Method CoPriv (w/ (_comm)
800
Pruning Method SNL (w/o comm)
600
Pruning Method SENet (w/o I. comm)
400
200
2
3
5
6
8
9
10
11
12
13
14
15
16
17
19 panallis
18
Block Idx
coutly blocks
Figure 10: Comparison of different pruning method and influence of Ccomm during the search in each block (ReLU budget is set to 60% in this example).
Conclusion
In this work, we propose network/protocol co-optimization framework, CoPriv, that simultancously optimizes the pre-processing and online communication for the 2PC-based private inference. Co- Priv features an optimized convolution protocol based on Winograd transformation and leverages series of DNN-aware protocol optimization to improve the efficiency the pre-processing stage. We also propose differentiable communication-awwe aware ReLU pruning algorithm with network re- parameterization to further optimize both the online and pre-processing communication induced by ReI.U, truncation and convolution. With extensive experiments, CoPriv consistently reduces both online and total communication without compromising the accuracy compared with different prior-art efficient networks and network optimization algorithms.
10

---

Acknowledgement
This work was supported in part by the NSFC (62125401) and the 111 Project (B (B18001).
References
[1] Syed Asad Alam, Andrew Anderson, Barbara Barabasz, and David Gregg. Winograd convo- lution for deep neural networks: Efficient point selection. ACM Transactions on Embedded Computing Systems, 21(6);1-28, 2022.
[2] Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputingg 5(4- 514-5):185-199 1993.
[3] Barbara Barabasz, Andrew Anderson, Kirk M Soodhalter, and David Gregg. Error analysis and improving the accuracy of winograd convolution for deep neural networks. ACM Transactions on Mathematical Software (TOMS), 46(4):1-33, 2020.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 2013.
[5] Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers in handwriting recognition. Suvisoft, 2006.
[6] Minsu Cho, Zahra Ghodsi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Sphynx: A deep neural network design for private inference. IEEE Security & Privacy, 20(5):22-34, 2022.
[7] Minsu Cho, Ameya Joshi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde. Selective network linearization for efficient private inference. In International Conference on Machine Learning, pages 3947-3961. PMLR, 2022.
[8] Daniel Demmler, Thomas Schneider, and Michael Zohner. Aby-a framework for efficient mixed-protocod secure two-party computation. In NDSS, 2015.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[10] Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmipnet: Hicrarchical vision mlp with re-parameterized locality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 578-587, 2022.
[11] Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1911-1920, 2019.
[12] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding, Resrep: Lossless cnn pruning via decoupling remembering and forgetting, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4510-4520, 2021.
[13] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building convolution as inception-like unit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10886-10899 2021.
[14] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13733-13742, 2021.
[15] Javier Fernandez-Marquess, Paul Whatmough, Andrew Mundy, and Matteww Mattina. Searching for winograd-award quantized networks. Proceedings of Machine Learning and Systems, 2: 2:14-29, 2020.
11

---

[16] Yonggan Fu, Haichuan Yang, Jiayi Yuan, Meng Li, Cheng Wan, Raghuraman Krishnamoorthi Vikas Chandra, and Yingyan Lin. Depthshrinker: new compression paradigm towards boosting real-hardware efficiency of compact neural networks. In International Conference on Machine Learning, pages 6849-6886 6862. PMLR, 2022.
[17] Vinod Ganesan, Anwesh Bhattacharya, Pratyush Kumar, Divya Gupta, Rahul Sharma, and Nishanth Chandran. Efficient ml models for practical secure inference. arXiv preprint arXiv:2209.044/ 1, 2022.
[18] Karthik Garimella, Zahra Ghodsi, Nandan Kumar Jha, Siddharth Garg, and Brandon Reagen. Characterizing and optimizing end-to-end systems for private inference. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ISPLOS),APLLCE 2023, page 194104, New York, NY, USA, 2023. Association for Computing Machinery.
[19] Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg. Cryptonas: Private inference relu budget. Advances in Neural Information Processing Systems, 33:16611-1697,, 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[21] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint Tttb::5503.053/// 2015.
[22] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3, In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[23] Chengcheng Huang, Xiaoxiao Dong, Zhao Li, Tengteng Song, Zhenguo Liu, and Lele Dong. Efficient stride 2 winograd convolution method using unified transformation matrices on fpga. In 2021 International Conference on Field-P Preerd-rogree Technology (ICFPT), pages 1-9. IEEE, 2021.
[24] Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding. Cheetah: Lean and fast secure (Two-Party) deep neural network inference. In 31st USENIX Security Symposium (USENIX Security 22), pages 809-826, 2022.
[25] Siam Umar Hussain, Mojan Javaheripi, Mohammad Samragh, and Farinaz Koushanfar. Coinn: Crypto/ml codesign for oblivious inference via neural networks. In Proceedings of the 202 ACM SIGSAC Conference on Computer and Communications Security, pages
2021.
[26] Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen. Deepreduce: Relu reduction for fast private inference. In International Conference on Machine Learning, pages 4839 4849. PMLR, 2021.
[27] Nandan Kumar Jha and Brandon Reagen. Deepreshape: Redesigning neural networks for efficient private inference, arXiv preprint a3Xiv:2304.1 10593, 2023.
[28] Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. GAZELLE: A low latency framework for secure neural network inference. In 27th {USENIX) Security Symposium ((USENIX) Security 18), pages 1651-1669 2018
[29] Brian Knott, Shobha Venkataraman, Awni Hanmun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-party computation meets machine learning. Advances in Neural Information Processing Systems, $4:4961-4973 2021.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

[31] Nishant Kumar, Mayank Rathee, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow: Secure tensorflow inference. IEEE Symposium on Security and Privacy (SP), pages 336-353. IEEE, 2020.
[32] Souvik Kundu, Shunlin Lu, Yuke Zhang, Jacqueline Liu, and Peter A Beerel. Learning to linearize deep neural networks for secure and efficient private inference. arXiv preprint arXiv:2301.09233 2023.
[33] Souvik Kundu, Yuke Zhang, Dake Chen, and Peter A Beerel. Making models shallow again: Jointly learning to reduce non-linearity and depth for latency-efficient private inference. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4684-4688, 2023.
[34] Andrew Lavin and Scott Gray, Fast algorithms for convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4013-4021, 2016.
[35] Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P Xing, and Hao Zhang. Mpcformer: fast, performant and private transformer inference with mpc. arXiv preprint arX1v:2211.01452, 2022.
[36] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo. Xin Yang, Kwang- Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference computer vision, pages 3296-3305, 2019.
[37] Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. Safenet: A secure, accurate and fast neural network inference. In International Conference on Learning Representations, 2021.
[38] Payman Mohassel and Peter Rindal. Aby3: A mixed protocol framework for machine learning. In ACM SIGSAC conference on computer and communications security, pages 35-52, 2018.
[39] Payman Mohassel and Yupeng Zhang. Secureml: system for scalable privacy-preserving machine learning. In 20/7 IEEE symposium security and privacy (SP), pages 19-38. IEEE, 2017.
[40] Hongwu Peng, Shanglin Zhou, Yukui Luo, Shijin Duan, Nuo Xu, Ran Ran, Shaoyi Huang, Chenghong Wang, Tong Geng, Ang Li, et al. Polympenet: Towards relu-free neural architecture search in two-party computation based private inference. arXiv preprint arXiv:2209.094) 09444, 2022.
[41] Deevashwer Rathee, Anwesh Bhattacharya, Rahul Sharma, Divya Gupta, Nishanth Chandran, and Aseem Rastogi. Secfloat: Accurate floating-poin meets secure 2-party computation. In 2022 IEEE Symposium on Security and Privacy (SP), pages 576-595, IEEE, 2022.
[42] Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nishanth Chandran, and Aseem Rastogi. Simn: A math library for secure mn inference. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1003-1020. IEEE, 2021.
[43] Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow2: Practical 2-party secure inference. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 225-342, 2020.
[44] Brandon Reagen, Woo-Seok Choi, Yeongil Ko, Vincent T Lee, Hsien-Hsin S Lee, Gu- Yeon Wei, and David Brooks. Cheetah: Optimizing and accelerating homomorphic encryption for private inference. In IEEE International Symposium on High-PPrformance Computer Architecture (HPCA),, pages 26-39. IEEE, 2021.
[45] Conrad Sanderson and Ryan Curtin. Armadillo: a template-bart c++ library for linear algebra. Journal of Open Source Software, 1(2):26, 2016.
[46] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
13

---

[47] Liyan Shen, Ye Dong, Binxing Fang, Jinqiao Shi, Xuebin Wang, Shengli Pan, and Ruisheng Shi. Abnn2: secure two-party urbitrary-bitwidii quantized neural network predictions. In Proceedings of the 59th ACM/IEEE Design Automalion Conference, pages 361-366, 2022.
[48] Wenting Zheng Srinivasan, PMRL Akshayaram, and Popa Raluca Ada. Delphi: A cryptographic inference service for neural networks. In Proc. 29th USENIX Secur. Symp, pages 2505-2522 2019.
[49] Kevin Vincent, Kevin Stephano, Michael Frumkin, Boris Ginsburg, and Julien Demouth. On improving the numerical stability of winograd convolutions. 2017.
[50] Juan Yepez and Seok-Bum Ko. Stride 2 1-d, 2-d, and 3-d winograd for convolutional neural networks. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 28(4):853-863, 2020.
[51] Wenxuan Zeng. Meng Li, Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpevit: Searching for accurate and efficient mpe-friendly vision transformer with heterogeneous attention. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5052-5063, 2023.
14

---

A Related Works
Private inference has been a promising solution to protect both data and model privacy during deep learning inference. In recent years, there has been an increasing amount of literature on efficient private inference. According to the optimization technique, these works can be categorized into three types, i.e., 1) protocol optimization; 2) network optimization; and 3) joint optimization.
In protocol optimization, ABY [ provides highly efficient conversion between arithmetic sharing, boolean sharing and Yao's sharing, and construct mixed protocols. As an extension, ABY3 [38] switches back and forth between three secret sharing schemes using three-party computation (3PC). CypTFlow2 [43] proposes a new protocol for secure and comparison and division which enables effecient non-linear operations such as ReLU SiRNN 42] further proposes 2PC protocols for
bitwidth extension, mixed-precision linear and non-linear operations. CrypTen [29] proposes a
software framework that provides a flexible machine learning focused API. More recently, SecFloat 41 proposes the crypto-friee rrypt--riendly precise functionalities to build a library for 32-bit single-precision floating-pont point operations and math functions. These works lack consideration for neural network architecture and has limited communication reduction.
In network optimization, DeepReDuce 26 proposes manually remove ReLUs with a three-step optimization pipline. SNI. 7 proposes Rel-aee optimization that leverages gradient- NAS to selectively linearize subset of ReLUs. CryptoNAS [19] uses ReLU budget as a proxy and leverages NAS to tailor ReLUs. PolyMPCNet [40] and SAFENet (37) replace ReLUs with MPC-friendly polynomial, while Sphynx (6) proposes an MPC-friendly ReLU-efficient micro-search space. SENet (32) innovatively measures the ReLU importance via layer pruning sensitivity and automatically optimize the network to meet the target ReLU budget. ShepReShape [26) finds that wider networks are more ReLUUefficient than the deeper ones and designs ReL ReLf-efficient baseline networks with with FLOPP-Rel.UAAcuraay balance. For Transeermer-based models, MPCFormer 35 proposes to simplify Softmax by replacing exponential with more MPC-friendly quadratic operation. MPCVIT [5T proposes to mix both high accuracy and high efficiency attention variants to accelerate private Vision Transformer (ViT) inference. Network optimization mainly focuses on ReLU reduction which dominates the online communication, but total communication including convolution and truncation cannot be optimized.
Unluckily, only using either protocol or network optimization just leads to limited efficiency improve- ment. Delphi [4]] jointly optimizes cryptographic protocols and network by gradually replacing ReLU with quadratic approximation. COINN (25) simultaneously optimizes quantized network and protocols with ciphertext-awarc quantization and automated bitwidth configuration. Recently, [17] proposes to use Winograd convolution for reducing the number of multiplications and design the efficient convolution operation to reduce the communication cost. However, it does not take private inference into consideration for Winograd algorithm, and still suffers tremendous communication overhead. In this work, we jointly optimize the network and protocol and fully consider their coupling properties.
B Details of Experimental Setup
protocol for private inference. We leverage the Athos [43] tool chain to convert both input and weight into fixed-point with the bit-width 41 and scale 12. We measure the communication and latency under a LAN setting 43 with 377 MBps bandwidth and 0.3ms echo latency. All of our experiments are evaluated on the Intel Xeon Gold 5220R CPU @ 2.20GHz.
Private inference framework CoPriv is built based on CypTTFlow2 43
Implementation of Winograd- based convolution protocol The convolution protocol with Wino- grad transformation and optimization implemented in C++ with Eigen and Armadillo matrix calculation library (45] in the CrypTFlow2 [43 framework. We implement F(2 x 2,3 x 3) and F(4 x 4, x 3) transformation for convolution with stride of 1 and F(2 x 2,3 x 3) transformation when stride is 2 [3]. For CIFAR- 100 dataset, we use F(2 x 2,3 x 3) transformation as the image resolution is small and for ImageNet dataset, we use 4, We only apply 22 xx3) for stride of 2 on ImageNet dataset. When evaluating CoPriv, we determine the optimal sender according to the analysis in Table 3 before inference. Winograd implementation enables us to measure the communication cost and latency of each convolution module.
15

---

Imput
Inpu:
Isput
Soc5
3x3
Fase AV
Fase Cors
Su5
ON
ON
Figure 11: An example of re-paramelerization for batch normalization and convolution filter.
Networks and datasets We apply proposed CoPriv to the widely used lightweight mobile network MobileNetV2 [46) with different width multipliers, e.g., 0.75, 1.0 and 1.4 to trade off the model accuracy and efficiency. We evaluate the top-1 accuracy and online and total communication on both CIFAR-100 and ImageNet ( dataset.
Differentiable pruning and finetuning setups We first search and prune redundant ReLUs for 10 epochs and then finetune the pruned network for 180 epochs with stochastic gradient descent (SGD) optimizer 2] cosine learning scheduler and initial learning rate of 0.1. During finetuning stage, train our proposed CoPriv with knowledge distillation to boost its performance.
C Network Re-Parameterization Algorithm
Network (or structural) re- p--aammtterizationn is useful technique proposed by RepVGG [14, and is extended to [1] 10, 13
16 12 The core idea re- pe-armmtterization is to decouple the training-time time architecture (with high performance and low efficiency) and inference-tine network architecture (with equivalent performance and high efficiency). Re- Re-parameterization is realized by converting one architecture to another via equivalently merging parameters together. Therefore, during inference time, the network architecture is not only efficient but also has the same high performance as the training-time architecture. Figure 11 is simple example of e-paarameterization.
In this work, we can also leverage this technique to merge adjacent convolutions together after ReLU removal. For the network ce-parameterization mentioned in Section 4.2) here we provide the following detailed algorithm to equivalently merge the inverted residual block into a single dense convolution shown in Figure With the help of network re- parameterization, we further optimize the total communication including convolution and truncation.
Algorithm 1: Network Re-parameterization for Inverted Residual Block
Input
:An inverted residual block with weights W1x1, Waxa, and W The number of input and output channels Nos, Nout. The size of re- parameterized weights r.
Output :Regular convolution with re-parameterized weights W,,
W, torch. forch.eye(C) (N..)..
W W,. wt..usquueeze (2) u2squeeze(2);
W, torch. mn www.aaffnnnnninnnnmmmmmmmmm pad pad= (= 1 ))
W torch nn Barch.m..fumctional..com/// (W Wix1)
W march.nn..uune =orcd.m..fuucttiaallccommmm (Wr, Waxs. padding=
W, forch.an.functioonaie
W7 Wix1);
7 Wrea torch zeros (Nout, No.. N,.,
for E 10....,No
do
L W (i, 1r/2], ||] 1;
10 W, W, Wres:
11 return W.:
D Details of Winograd Convolution
D.1 Comparison between Regular Convolution and Winograd Convolution
To help readers better understand the multiplication reduction of Winograd convolution, we demon- strate regular convolution and Winograd convolution in Figure I2 Given input 1x and
16

---

Input Layer
Input Layer
Winograd Domain
Sliding Window
Transform
Elemmnnts
4 Windows in Total
Tronsform Output
Tile in Total
Per Window
9 #MUL
16 #MUL
Convolution
16 Elements Per Tile
EWMM
STEP3
STEF4
Outpur Tile of Layer
Transform
Ouiput Tile
of L
Filter
STEP'S
Layeri
Layer
(a) Regular convolution with 9x4-36 #MUL.
(b) Winograd convolution with 16x1-16 #MUL in F(2x2, 3x3).
Figure 12: Comparison between (a) regular convolution and (b) Winograd convolution.
a filter E R3x3 regular convolution requires
times multiplications (implemented using GEMM with im2col algorithm (5] while F(2 x 2,3 x 3) Winograd transformation only times multiplications (EWMM), which achieves 2.25x reduction. Moreover,
requires x 1 F(4 x 4, x 3) with larger tile size, i.c., can further achieve 4x multiplication reduction. The improvement gets benefit from the Winograd's ' ability to convert im2col to EWMM and calculate the whole tile in Winograd domain at once.
D.2 Details of Input Tiling and Padding
Given a large 2D input I R where >> m + r the core technique for ensuring the equivalence of regular convolution and Winograd convolution is input tiling and padding. The output size r + 1, the input tile size n m + and the total tile number T per channel computed as
1 1
1
n m -
where [.] denotes taking the upper bound value. For each tile, Winograd convolution is individually performed and results an output tile with m X m size. After the tiles are computed with Winograd convolution, the output tiles are concatenated together to form the final output.
For some input size, the input cannot covered by tiles. For instance, when leveraging (22 x33) on the input I E 7x7, the rightmost and bottom pixels cannot be divided into complete tile. To solve this problem, we pad these positions with enable the tiles totally cover the whole input. The correctness and equivalence can be proved with Eq. Also, 17 shows the overhead caused by padding negligible.
D.3 Support for Stride of 2 Winograd Convolution
Conventional Winograd convolution only supports stride = convolution filter. However, in recent efficient neural networks, e.g., MobileNetV2, EfficientNet has several stride of 2 layers reduce the feature map size by half. To enable extreme optimization for efficient networks, we introduce F(2 2, 3 3) for stride of 2 Winograd convolution for private inference.
There are various methods to construct stride of Winograd kernel such as dividing input and convolution filter into different groups [50]. However, it is not a simple way to implement stride of Winograd kernel.
is an extremely convenient method using unified transformation matrices.
Based on 23 even positions of input and filter are computed by F(2, while odd positions are computed by regular convolution. Transformation matrices are derived as follows and can be computed using Eq. [
11
0
0
1
0
0
0
0
1
0
BT_
0
0
0 1
1
A
0
0
0
0
1
0
0
0
0
-1
1
0
0
1
17

---

Correctness analysis. Here, we take ID algorithm as an example to prove the correctness
Winograd convolution for stride of The algorithm can be nested with itself to obtain a 2D algorithm
Given input X and filter F as
Tu
T1
you
2g0 21
X
12
F
y1
Y = X
33
L/22
34
First, we calculate regular convolution with stride of 2 using im2col algorithm S as
yo
270
22
zoll + 2/11 + 2/3/22
Y1
= 33
31
2/2
2/4
#220 + 2:32/1 + 2/4/22
L/2
thus, 20 2080 + 2191 + 228/2 and 21
2230 + #38/1 Z4Y/2.
Then, we calculate Winograd convolution for stride of as
== [(F)) (B x)).
and then
1
0 07
1
0
1
0
07
370
1
1
1
0
0
yo
31
Y22
1
IC
1
1
2/1
0
0
1
0
}}
L1/2_
T3
0
1
0
0
-1
0
and further simplify the calculation as
yo
X2
wozo 02
1 1 0
0
1/1
21
1 1
11000
0
T1V1
Y22
0
[(
yo
+
1/2
42
#23/ + X2
1/1
23
T3311
1/2
32
243/2 2282
therefore, the convolution result
Folo + #1// + T23/
Y22
Yy.
F2f0 + E3l/1 + 2442
D.4 Transformation Matrices for Winograd Convolution
We provide the transformation matrices A, B, G for F(2 x 2,3 x 3) and F(4 x 4,3 x Winograd transformation based polynomial Chinese remainder theorem (CRT) or Lagrange interpolation
34
For F(2 x 2,3 3), we have
0
0
1
0
1/2
1/2
1/2
1
0
B
G
A
0
-1
1/2
-
1/2
1/2
0
-1
-1
0
1
-1
0
0
For F(4 x 4,3 3), we have
18

---

4
0
-5
0
1 1
0
1/4
0
0
0
4
-4
1
0
1/6
-1/6
1/6
B
O
4
-4
4
-1
1
0
G=
-
1/6
1/6
-
- 1
0
-2
-1
2
1
1/24
1/12
1/6
0 2
-1
-2
1
0
1/24
-1/12
1/6
0
0
1
0
1
0
AT
0
1
-1
2
-2
0
0
4
0
-1
8
1
The correctness analysis is the same with Section D.3
E
Comparison with More Related Work
33 is recent work that aims optimizing the network architecture for latency-efficient private inference. Both CoPriv and [33] aim to reduce the number ReLUs and the depth of the whole network in order to improve the efficiency. However, they have totally different motivations and methods. 1) Motivations: [33] still regards ReLU as the main latency bottleneck and removes convolution layers reduce computation. In contrast, CoPriv fuses neighboring convolution layers to better leverage our Winograd-basee The difference in motivation leads to different criterion when selecting convolutions to remove. 33 selects convolutions based on ReLU sensitivity [32) while CoPriv simultancously considers both accuracy and communication cost. Methods: 33 first determines which convolutions to
optimization for communication reduction of all operators.
remove based on ReLU sensitivity [32] and then, uses the gated branching method for training. In contrast, CoPriv simultaneously train the architecture parameters with the model weights, and the equivalent re- parameterization is conducted post training. enabling us to leverage the benefits of over- parameterization during training shown in RepVGG [14. Quantitatively, we compare CoPriv with 33 in Table CoPriv achieves much lower online and total communication while achieving higher
accuracy compared
33
Table 5: Accuracy and communication comparison between CoPriv and 33
Method
Top-1 Acc. (%) Online Comm. (GB) Total Comm. (GB)
33
69.10
0.81
CoPtiv (ours)
70.58
0.43
5.14
F More Detailed Comparison with Prior-Art Methods
As supplement of Table I we provide a more detailed comparison with prior-art methods in terms of different techniques in Table 6
Table 6: Comparison with prior-art methods in of different techniques.
Network Optimization
Method
Protocal Optimization
Convolution
Truncation
ReLU
39 & 19 10 57
RcLU, Trunc, Cooy,
27
26
X Comy: (Online Offline)
ReLU count/semtiivity awarr aware NAS
13
ReLU count-awars NAS
Com: (Winograd based)
Factorized wise
X
Channel Reduction
Channel Reduction
Cl Channel Reduction
CoPPiv (ours)
CCoii: (Winogr Piiisid
RG-par aaniiinniiii
Communsttionniiiaa NA3
G Comparison of Convolution Protocol with HE-Based Method
We also compare CoPriv with SOTA homomorphic encryption (HE) based method, Cheetah 24 which uses HE-based protocol for convolution instead of oblivious transfer (OT) in CrypTFlow2. Chectah achieves lower communication compared to CrypTFlow2 the cost of more computation
19

---

overhead for both the server and client. Here, we compare their inference latency. Hence, we believe Cheetah and CrypTFlow2/CoPrvv have different applicable scenarios. For example, for less perfor- mant clients, Cheetah may not be applicable, while when the bandwidth is low, CrypTFFoo2/CoPrivv may not be the best choice. When comparing CoPriv with Cheetah, we observe a similar trend. As shown in T we find while CoPriv always outperforms CrypTTFlow2, for high bandwidth, CoPriv achieves lower latency compared to Cheetah while for low bandwidth, Cheetah incurs lower latency.
Table 7: Convolution latency (8) comparison with Cheetah for different blocks (expand ratio is set to 6) and bandwidths,
Bandwidth
384 MBps
24
44 MBps
24
9 MBBs 47
39
Dimension:
x 4
64)
Baseline OT (CrypTFlow2)
2.14
3.55
16.2
(Chectah)
1.35
1.54
2.99
CoPriv w/ Winograd (ours)
0.21
0.64
3.00
Dimension: (8 8 32)
Baseline OT (CrypTFlow2)
1.29
3.94
16.1
(Cheetah)
0.72
0.73
1.53
CoPriv w/ Winograd (ours)
0.17
0.45
2.16
20


Reviews:
[{"review_id": "c4YA5Cjqlp", "reviewer": "Reviewer_yVeD", "summary": "This paper presents CoPriv that jointly optimizes 2PC protocols and DNN architectures. It argues that SOTA 2PC protocols mainly focus on minimizing ReLU-based metric, which no longer contributes to the majority of communication. It proposes a new protocol for convolution with Winograd transformation and proposes a series of other DNN-aware optimizations. It shows communication reduction compared to SOTA protocols, CrypTFlow2, and other network optimization methods.", "questions": "The paper shows a consistent communication reduction, which the reviewer appreciated. However, it would be better if the author can provide some analysis on the end-to-end speedup using common setups.", "limitations": "The paper only analyzes the communication volume, where the effect on the communication time, inference speedup has not been well discussed. It can be the case that with higher bandwidth, this communication volume reduction will not be significant. However, the reviewer appreciates the communication volume reduction alone, and would not consider this as a major limitation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The observations in the motivation section are informative, well written and supported.\n(2) The Winograd based transformation with tile aggregation is well motivated and well illustrated (Figure 4). \n(3) The adaptive convolutional protocol is novel, insightful and effective.\n(4) The ablation study which adds optimizations step by step is conclusive and well-conducted.\n(5) The proposed methods consistently achieve communication reduction compared to SOTA.", "weaknesses": "The implication of end-to-end speedup is not well studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer yVeD for your thoughtful feedback!\n\n---\n\n**Q1:** The implication of end-to-end speedup is not well studied.\n\n**A1:** Thanks for the meaningful suggestion on the inference speedup! In our experiments, we compare the inference latency of MobileNetV3 (with different capacities), ReLU-optimized networks (including SNL and SENet), and SOTA pruning methods (including uniform pruning and MetaPruning) on the ImageNet dataset with a widely used LAN communication setting (i.e., 377 MBps bandwidth and 0.3ms echo latency) following CrypTFlow2. \nAs shown in Figure 8(c), on the ImageNet dataset, 1) our proposed CoPriv outperforms SNL with 6% higher accuracy and 1.4$\\times$ latency reduction, and 2) CoPriv outperforms SENet with 2.8% higher accuracy and 2.2$\\times$ latency reduction. These results demonstrate a consistent accuracy improvement with latency reduction.\n\nWhen the communication bandwidth becomes lower, e.g., in the wireless setting, we expect our latency reduction can be further improved.\nIn brief, we do agree with your valuable suggestions on the inference speedup, and we will add more results about the speedup in the wireless setting with a lower bandwidth in the revised version."}, {"review_id": "bGkgAlaI65", "reviewer": "Reviewer_vLvc", "summary": "The paper introduces CoPriv, a framework that optimizes the 2-party computation (2PC) inference protocol and the deep neural network (DNN) architecture to reduce communication overhead. CoPriv features a new 2PC protocol for convolution based on Winograd transformation and develops DNN-aware optimization to reduce inference communication. ", "questions": "Please refer to the weakness section.", "limitations": "While the authors acknowledge the limitation of Winograd convolution, stating that it can only be applied to 3x3 depth-wise convolution, they do not discuss the limitations of their entire work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors highlight a significant point that pruning ReLU may no longer be the most effective method for reducing computational and communication costs in private inference. This is due to the increasing prominence of linear and truncation operations. This insight is of considerable importance to the community.", "weaknesses": "1. Figure 1 could benefit from more specific details. The left figure should include specific numbers for each portion and the amount of communication cost reduction achieved by each technique. It's also unclear which private inference method is used in the left figure.\n\n2. The paper does not clearly explain why the DNN-aware adaptive convolution protocol can reduce communication costs. As a major contribution, it would be beneficial if the authors could provide a detailed explanation of how the selection of protocol initializer impacts communication costs and the criteria for selecting the initializer.\n\n3. The novelty of the proposed ReLU pruning method is not clear unless the authors can explain how L_{comm} affects the training results. It would be valuable to compare the proposed ReLU pruning method with DeepReduce/SNL/SENet in terms of which ReLUs remain in the network and final accuracy.\n\n4. It would be beneficial to differentiate the proposed re-parameterization method from the following work: \n\"Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference\" (https://openaccess.thecvf.com/content/CVPR2023W/ECV/papers/Kundu_Making_Models_Shallow_Again_Jointly_Learning_To_Reduce_Non-Linearity_and_CVPRW_2023_paper.pdf).\n\n5. From Figure 9, it appears that the authors do not apply ReLU pruning and re-parameterization to each block. The criteria for deciding which blocks are suitable for ReLU pruning and re-parameterization are not clear.\n\n6. In Table 4, the accuracy remains the same for MobieNetV2 with and without pruning+re-parameterization. Since pruning and re-parameterization typically lead to accuracy degradation, could the authors provide an explanation for these results?\n\n7. Given that Cheetah outperforms CrypTFlow2, it would be more valuable and informative to compare the proposed protocol optimization method with Cheetah rather than CrypTFlow2. This comparison could provide a more accurate assessment of the proposed method's performance relative to the current state-of-the-art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer vLvc for your thoughtful feedback!\n\n---\n\n**Q1:** Figure 1 could benefit from more details.\n\n**A1:** Thanks for your advice! We will improve this figure carefully to include more details. \nFor ReLU, [22] represents Gazelle which uses garbled circuit (GC), [33] represents CrypTFlow2 which uses IKNP OT, and [19] represents Cheetah which uses VOLE OT. We get these numbers directly from each paper.\n\n---\n\n**Q2:** Not clearly explain the DNN-aware adaptive convolution protocol.\n\n**A2:** As introduced in Section 4.1, in the Winograd-based convolution protocol, with tile aggregation, the server and client need to jointly run the OT-based matrix multiplication protocol for $(m+r-1)^2$ times. For each OT-based matrix multiplication, the server and the client hold the weight and activation of shape $(K, C)$ and $(C, T)$, respectively, where $K$, $C$, and $T$ denote the number of output channels, input channels, and # tiles, respectively, and are impacted by the DNN architecture. We observe the cost of the OT-based matrix multiplication depends on the OT initializer. More specifically, when the server initializes the OT, the round of communication and the communication of each round are $O(CK)$ and $O(\\lambda + T)$, respectively (the complexity is derived based on CrypTFlow2). In contrast, when the client initializes the OT, the round and communication of each round become $O(CT)$ and $O(\\lambda + K)$, respectively. As $K$, $C$, and $T$ are known before inference, our DNN-aware adaptive protocol will select the optimal OT initializer for each network and each layer to minimize the communication cost.\n\n---\n\n**Q3:** How $L_{comm}$ affects training.\n\n**A3:** The key insight of our ReLU pruning is 1) to directly use communication to guide the pruning instead of proxy metrics, e.g., ReLU count, and 2) to enforce two ReLUs within the same block to share $\\alpha$ so that they can be removed simultaneously to enable re-parameterization and Winograd-based optimization. In contrast, SENet/DeepReDuce/SNL just focus on reducing ReLU count and thus, can barely reduce the total communication. \n\nTo show the importance of introducing $L_{comm}$, we compare our pruning method w/ $L_{comm}$ and pruning methods of SNL/SENet w/o $L_{comm}$ in **Figure 2 in the rebuttal PDF**. As we see, $L_{comm}$ helps to focus the pruning on the later layers, which incur more communication cost, and penalizes the costly blocks (e.g., block #16/#18). In contrast, SENet focuses on pruning early layers with more ReLU counts.\n\n---\n\n**Q4:** Differentiate the proposed re-parameterization from CVPRW 2023.\n\n**A4**: Thanks for pointing out the valuable work. We will cite this paper in our revised version. We also make the following comparison:\n\n1) Similarity: the two papers both reduce the ReLUs and the network depth.\n\n2) Different motivations: the CVPRW paper still regards ReLU as the main latency bottleneck and removes convolution layers to reduce computation. In contrast, we fuse neighboring convolution layers to better leverage our Winograd-based optimization for communication reduction of all operators. The difference in motivation leads to different criterion when selecting convolutions to remove. The CVPRW paper selects convolutions based on ReLU sensitivity while we consider both accuracy and communication cost.\n\n3) Different methods: the CVPRW paper first determines which convolutions to remove based on ReLU sensitivity and then, uses the gated branching method for training. In contrast, we simultaneously train the architecture parameters with the model weights, and the re-parameterization is conducted **post training**, enabling us to leverage the benefits of over-parameterization shown in RepVGG (RepVGG: Making VGG-style ConvNets Great Again). \n\nAs shown in **Table 1 in the rebuttal PDF**, CoPriv achieves both better accuracy and lower communication compared to the CVPRW paper.\n\n---\n\n**Q5:** The criteria for ReLU pruning and re-param is not clear.\n\n**A5**: We prune ReLUs based on the architecture parameter $\\alpha$ of each block. $\\alpha$ is trained jointly with model parameters and considers both accuracy and communication cost. During searching, we gradually fix small $\\alpha$ to 0 until the required communication is achieved. After ReLU pruning, there are only three sequential convolutions in the inverted residual block. Then, we further re-parameterize the block into a single convolution.\n\n---\n\n**Q6:** Explain the accuracy degradation of pruning and re-param.\n\n**A6:** MobileNetV2 with pruning degradates the accuracy compared with original MobileNetV2, but here we directly report the accuracy of MobileNetV2 after pruning. In contrast, re-parameterization will not hurt the accuracy because it **equivalently** merges the convolutions within an inverted residual block into a single convolution **post training**. We show the detail of re-parameterization in Appendix C.\n\n---\n\n**Q7:** Compare the protocol optimization with Cheetah.\n\n**A7**: Thanks for the valuable advice. Cheetah uses HE-based protocol for convolution instead of OT in CrypTFlow2. Cheetah achieves lower communication compared to CrypTFlow2 at the cost of more computation overhead for both the server and client. Hence, we believe Cheetah and CrypTFlow2/CoPriv have different applicable scenarios. For example, for less performant clients, Cheetah may not be applicable while when the bandwidth is low, CrypTFlow2/CoPriv may not be the best choice.\n\nWhen comparing with Cheetah, we observe a similar trend. As in **Table 3 in the rebuttal PDF**, we find while CoPriv always outperforms CrypTFlow2, for high bandwidth, CoPriv achieves lower latency compared to Cheetah while for low bandwidth, Cheetah incurs lower latency. Hence, in our paper, we focus on comparison with CrypTFlow2 and other OT-based baselines.\n\n**Limitation:** There is still an efficiency gap between ciphertext and plaintext inference, and we leave more valuable work as our future work."}, {"review_id": "IJw5Fo0mbH", "reviewer": "Reviewer_AJR8", "summary": "The paper presents optimizations to secure two-party computation of convolutional network inference. There are optimizations for both linear and non-linear layers, resulting in an overall single-digit factor improvement.\n", "questions": "n/a", "limitations": "n/a", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The optimizations look interesting and are underlined well with benchmarks. I particularly appreciate the trade-off visualization in Figure 8. \n", "weaknesses": "Table 1 doesn't make sense to me. I don't think there is merit in pointing out that prior work hasn't optimized certain aspects because it might be that the work is efficient without extra effort. If anything, the table should contain numerical speed-ups.\n\nUsing the acronym ASS for arithmetic secret sharing might put off readers as it's identical to a vulgar word.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer AJR8 for your thoughtful feedback!\n\n---\n\n**Q1:** Table 1 should contain numerical speed-ups.\n\n**A1**: We thank the reviewer for the valuable feedback. As shown in Figure 1 in the paper, operations like ReLU, truncation, and convolution, are major contributors to the online or total communication. To reduce their communication, we hope to emphasize the importance of both protocol and network optimization, and all the components of the neural network should be fully considered, and we also compare our CoPriv with existing methods. Hence, we have a qualitative comparison in Table 1 and leave quantitative comparison in experimental results.\n\nWe do agree simple qualitative comparison may not be very useful. We augment the table in **Table 2 in the rebuttal PDF** following your suggestion to include numerical speed-ups in the table. We will think of better ways to make the comparisons as well.\n\n---\n\n**Q2:** Using the acronym ASS for arithmetic secret sharing might put off readers as it's identical to a vulgar word.\n\n**A2:** Thanks for your advice! We agree with your opinion, and we will consider a more appropriate acronym for arithmetic secret sharing like SS or ArSS in our revised version."}, {"review_id": "mmbKaahjdm", "reviewer": "Reviewer_P7t8", "summary": "This paper presents a framework that simultaneously optimizes the 2PC inference protocol and the neural network architecture to achieve a significant reduction in communication. The framework outperforms state-of-the-art (SOTA) approaches by achieving communication reduction.", "questions": "Please refer to the weaknesses section.", "limitations": "Please refer to the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand\n2. The paper highlights the current scenario where nonlinearity no longer dominates the communication overhead.", "weaknesses": "1. It would be beneficial to include more ablation experiments to demonstrate the individual contributions of the ReLU pruning and re-parameterization approaches.\n\n2. Providing the percentage or number of multiplication reduction resulting from the Winograd algorithm would provide additional insights into the overall communication reduction.\n\n3. Please ensure that the legends are appropriately positioned within the figures, specifically those that cover certain data points.\n\n4. Empirically comparing the proposed ReLU pruning method with prior methods would provide a deeper understanding of its effectiveness.\n\n5. The most recent work on the ReLU reduction is as follows, the author could consider including it in the introduction:\n\"S. Kundu, et al., Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference, CVPRW 2023.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer P7t8 for your thoughtful feedback!\n\n---\n**Q1:** Include more ablation experiments to demonstrate the individual contributions of ReLU pruning and re-parameterization.\n\n**A1:** Thanks for your suggestion! In Section 5.5, we perform an ablation study by adding our proposed techniques step by step. And we also add more ablation experiments with 60% and 30% ReLU remained shown in the table below. \nFor 30% ReLU, CoPriv achieves 1.4$\\times$ online communication reduction after pruning, 3.2$\\times$ and 3.6$\\times$ online/total communication reduction after re-parameterization compared with baseline MobileNetV2. All of these results indicate that our proposed optimizations are indispensable for improving the communication efficiency.\n\nModel (60% ReLU) | Online Comm. (GB) | Total Comm. (GB)\n------ | ------ | ------ \nBaseline MobileNetV2  | 0.82 | 8.00\n+Pruning  | 0.64 | 7.81 \n+Re-parameterization  | 0.43 |  5.62 \n\nModel (30% ReLU) | Online Comm. (GB) | Total Comm. (GB)\n------ | ------ | ------ \nBaseline MobileNetV2  | 0.82 | 8.00\n+Pruning  | 0.58 | 7.76 \n+Re-parameterization  | 0.26 |  2.21\n\n---\n\n**Q2:** Providing the percentage or number of multiplication reduction resulting from the Winograd algorithm would provide additional insights into the overall communication reduction.\n\n**A2:** For ResNet-18 and 32 with regular convolutions, Winograd algorithm reduces the number of multiplications by 2.25$\\times$ with $F(2\\times2, 3\\times3)$ transformation theoretically and empirically (shown in our micro-benchmark in Section 5.2), resulting $\\sim 2.1\\times$ communication reduction; \nAnd for our proposed CoPriv, we compute the number of multiplications of original MobileNetV2 and CoPriv with 40% ReLUs remained which is shown in **Figure 1 in the rebuttal PDF**. From this, we observe that the number of multiplication is significantly reduced to 68% with our Winograd algorithm, resulting a lower communication.\n\n---\n\n**Q3:** Ensure that the legends are appropriately positioned within the figures.\n\n**A3:** Thanks for your kind suggestion on our figures!  We will carefully improve our writing and figures in the revised version.\n\n---\n\n**Q4:** Empirically comparing the proposed ReLU pruning method with prior methods would provide a deeper understanding of its effectiveness.\n\n**A4:** The key insight of our ReLU pruning is 1) to directly use communication to guide the pruning instead of proxy metrics, e.g., ReLU count, and 2) to enforce two ReLUs within the same block to share $\\alpha$ so that the two ReLUs can be removed simultaneously to enable re-parameterization and Winograd-based optimization. In contrast, SENet/DeepReDuce/SNL just focus on reducing ReLU count and thus, can barely reduce the total communication. So, our focus is mainly on the optimization pattern rather than the pruning algorithm itself, and note that other pruning method can be easily plugged into our proposed framework.\n\nIn this paper, we have compared CoPriv with ReLU-optimized methods including SNL, SENet and DeepReDuce in Section 5.3 and 5.4. Among them, SNL and SENet propose NAS (SENet further uses pruning sensitivity to analyze ReLU importance) to prune ReLUs while DeepReDuce manually decides which ReLUs to be removed.\nOne big difference between our pruning method and prior methods is the $L_{comm}$, which means ReLUs in different layers are not equivalent, and makes our method communication-aware rather than ReLU count-aware. We discover that $L_{comm}$ helps us better trade-off the efficiency and accuracy.\n\nTo show the importance of introducing $L_{comm}$, we compare our pruning method w/ $L_{comm}$ and pruning methods of SNL/SENet w/o $L_{comm}$ in the **Figure 2 in the rebuttal PDF**. As we can see, $L_{comm}$ helps to focus the pruning on the later inverted residual blocks, which incur more communication overhead, and our method effectively penalizes the importance of the costly blocks (e.g., block #16 and #18), achieving significantly lower communication. By adjusting $L_{comm}$, CoPriv has a strong ability to better trade-off the efficiency and accuracy.\n\n---\n\n**Q5:** The author could consider including the most recent work (CVPRW 2023) on the ReLU reduction in the introduction.\n\n**A5:** Thanks for pointing out the valuable work. We will include this paper in our revised version. We also make the following comparison:\n\n1) Similarity: the two papers both reduce the ReLUs and the network depth.\n\n2) Motivations are different: the CVPRW paper still regards ReLU as the main latency bottleneck and removes convolution layers to reduce computation. In contrast, we fuse neighboring convolution layers to reduce truncations and better leverage our Winograd-based optimization for communication reduction. The difference in motivation leads to different criterion when selecting convolutions to remove. The CVPRW paper selects convolutions based on ReLU sensitivity while we consider both accuracy and communication cost.\n\n3) Methods are different: the CVPRW paper first determine which convolutions to remove and then, use the gated branching mechanism to train new convolutions. In contrast, we simultaneously train the architecture parameters $\\alpha$ with the model weights and the re-parameterization is conducted **post training**, enabling us to better leverage the benefits of over-parameterization as shown in the RepVGG paper (RepVGG: Making VGG-style ConvNets Great Again). \n\nAs shown in the table below, our method achieves both better accuracy and lower communication compared to the CVPRW paper.\n\nMethod | Top-1 Acc. (%) | Online Comm. (GB) | Total Comm. (GB)\n------  | ------ | ------ | ------ \nCVPRW 2023 | 69.10 | 0.81 | 46.5\nCoPriv (ours) | 70.58 | 0.43 | 5.14"}]

Author rebuttal:
We sincerely thank all the reviewers for the thoughtful feedback and helpful comments!

**Rebuttal One-page PDF:** we attach the one-page PDF here to include the figures and tables mentioned in the following responses. We give a brief introduction of these figures and tables below to provide convenience for quick review for the reviewers:

* Figure 1: Comparison of the number of multiplications in each block between original MobileNetV2 and CoPriv with Winograd transformation (based on the comments of Reviewer P7t8).
* Figure 2: Comparison of different pruning method and the influence of $L_{comm}$ during the search in each block (based on the comments of Reviewer vLvc and P7t8).
* Table 1: Accuracy and communication comparison between the CVPRW paper and our CoPriv (based on the comments of Reviewer vLvc and P7t8).
* Table 2: Comparison with prior-art methods with qualitative optimization level (based on the comments of Reviewer AJR8).
* Table 3: Latency comparison with Cheetah for different blocks and bandwidths (based on the comments of Reviewer vLvc).

Meta-review: This work proposes a method that jointly optimizes 2PC protocols and DNN architectures. It argues that other linear and non-linear operations are also important in addition to ReLU which was the focus of previous works. 

The final ratings are unanimously accept (6, 6, 7, 7). Reviewers recognize the novelty and clarity of the paper as well as the improvements over STOA. 

The authors addressed most of the concerns in the rebuttal. Please add the additional discussions and experiments in the final version.
Decision: Accept (poster)