SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenxuan Zeng
zwx.andy@stu.pku.edu.cn
Meng Li*
meng.li@pku.edu.cn
Haichuan Yang
haichuan@meta.com
Wen-jie Lu
juhou.lwj@antgroup.com
Runsheng Wang
r.wang@pku.edu.cn
Ru Huang
ruhuang@pku.edu.cn
Deep neural network (DNN) inference based on secure 2-party computation (2PC) can offer cryptographically-secure privacy protection but suffers from orders of magnitude latency overhead due to enormous communication. Previous works heavily rely on a proxy metric of ReLU counts to approximate the communication overhead and focus on reducing the ReLUs to improve the communication efficiency. However, we observe these works achieve limited communication reduction for state-of-the-art (SOTA) 2PC protocols due to the ignorance of other linear and non-linear operations, which now contribute to the majority of communication. In this work, we present CoPriv, a framework that jointly optimizes the 2PC inference protocol and the DNN architecture. CoPriv features a new 2PC protocol for convolution based on Winograd transformation and develops DNN-aware optimization to significantly reduce the inference communication. CoPriv further develops a 2PC-aware network optimization algorithm that is compatible with the proposed protocol and simultaneously reduces the communication for all the linear and nonlinear operations. We compare CoPriv with the SOTA 2PC protocol, CryptTFlow2, and demonstrate  $2.1 \times$  communication reduction for both ResNet-18 and ResNet32 on CIFAR-100. We also compare CoPriv with SOTA network optimization methods, including SNL, MetaPruning, etc. CoPriv achieves  $9.98 \times$  and  $3.88 \times$  online and total communication reduction with a higher accuracy compared to SNL, respectively. CoPriv also achieves  $3.87 \times$  online communication reduction with more than  $3\%$  higher accuracy compared to MetaPruning.
Deep learning has been applied to increasingly sensitive and private data and tasks, for which privacy emerges as one of the major concerns. To alleviate the privacy concerns when deploying the deep neural network (DNN) models, secure two-party computation (2PC) based DNN inference is proposed and enables cryptographically-strong privacy guarantee [39, 28, 48, 44, 25, 24, 47].
Secure 2PC helps solve the following dilemma [28, 48, 44]: the server owns a private model and the client owns private data. The server is willing to provide the machine learning as a service (MLaaS) but does not want to give it out directly. The client wants to apply the model on the private data without revealing it as well. Secure 2PC frameworks can fulfill both parties' requirements: two parties can learn the inference results but nothing else beyond what can be derived from the results.
The privacy protection of secure 2PC-based inference is achieved at the cost of high communication complexity due to the massive interaction between the server and the client [31, 48, 43, 25]. This leads to orders of magnitude latency gap compared to the regular inference on plaintext [31, 43]. A 2PC-based inference usually has two stages, including a pre-processing stage that generates the input independent helper data and an online stage to process client's actual query [31, 48, 43, 25, 24]. Because the helper data is independent of client's query, previous works [48, 25] assume the pre-processing stage is offline and thus, focus on optimizing the communication of the input dependent stage. Specifically, [48, 19, 26, 6, 7, 32] observe ReLU accounts for significant communication and ReLU count is widely used as a proxy metric for the inference efficiency. Hence, the problem of improving inference efficiency is usually formulated as optimizing networks to have as few ReLUs as possible. For example, DeepReduce [26] and SNL [7] achieve more than  $4\times$  and  $16\times$  ReLU reduction with less than  $5\%$  accuracy degradation on the CIFAR-10 dataset.
However, such assumption may no longer be valid for state-of-the-art (SOTA) 2PC protocols. We profile the communication of ResNet-18 with different ReLU optimization algorithms, including DeepReduce [26], SENet [32], and SNL [7] and observe very limited communication reduction as shown in Figure 1(b). This is because, on one hand, the communication efficiency of ReLU has been drastically improved over the past few years from garble circuit (GC) to VOLE oblivious transfer (OT) as shown in Figure 1(a) [28, 43, 24], and ReLU only accounts for  $40\%$  and  $1\%$  of the online and total communication, respectively. On the other hand, recent studies suggest for MLaaS, it is more im
important to consider inference request arrival rates rather than studying individual inference in isolation [18]. In this case, the pre-processing communication, which is significantly higher than the online communication as shown in Figure 1(b), cannot be ignored and may often be incurred online as there may not be sufficient downtime for the server to hide its latency [18].
Therefore, to improve the efficiency of 2PC-based inference, in this paper, we argue that both pre-processing and online communication are important and propose CoPriv to jointly optimize the 2PC protocol and the DNN architecture. CoPriv first optimizes the inference protocol for the widely used 3-by-3 convolutions based on the Winograd transformation and proposes a series of DNN-aware protocol optimization. The optimized protocol achieves more than  $2.1 \times$  communication reduction for ResNet models [20] without accuracy impact. For lightweight mobile networks, e.g., MobileNetV2 [46], we propose a differentiable ReLU pruning and re-parameterization algorithm in CoPriv. Different from existing methods [26, 7, 32], CoPriv optimizes both linear and non-linear operations to simultaneously improve both the pre-processing and online communication efficiency.
With extensive experiments, we demonstrate CoPriv drastically reduces the communication of 2PC-based inference. The proposed optimized protocol with Winograd transformation reduces the convolution communication by  $2.1 \times$  compared to prior-art CrypTFlow2 [43] for both the baseline ResNet models. With joint protocol and network co-optimization, CoPriv outperforms SOTA ReLU-optimized methods and pruning methods. Our study motivates the community to directly use the communication as the efficiency metric to guide the protocol and network optimization.
2.1 Threat Model
CoPriv focuses on efficient private DNN inference involving two parites, i.e., Alice (server) and Bob (client). The server holds the model with private weights and the client holds private inputs. At the end of the protocol execution, the client learns the inference results without revealing any
Figure 1: (a) Communication for ReLU communication has reduced by  $131 \times$  from [28] to [24]; (b) existing network optimizations suffers from limited reduction of online and total communication.
Table 1: Comparison with prior-art methods.
Method	Protocol Opt.	Network Opt.
Conv.	Truncation	ReLU
[43, 42, 39, 8, 38, 29, 41]	✓	X	X	X
[32, 7, 19, 40, 37, 6, 27, 26]	X	X	X	✓
[48, 25]	✓	X	X	✓
[17]	✓	✓	X	X
CoPriv (ours)	✓	✓	✓	✓
Table 2: Notations used in the paper.
Notation	Meaning
H,W	Height and width of output feature map
C,K	Input and output channel number
A,B,G	Winograd transformation matrices
m,r	Size of output tile and convolution filter
T	Number of tiles per input channel
λ	Security parameter that measures the attack hardness
information to the server. Consistent with previous works [39, 48, 43, 42, 26, 7, 32], we adopt an honest-but-curious security model in which both parties follow the protocol but also try to learn more from the information than allowed. Meanwhile, following [39, 28, 48, 43, 42], we assume no trusted third party exists so that the helper data needs to be generated by the client and the server.
2.2 Arithmetic Secret Sharing
CoPriv leverages the 2PC framework based on arithmetic secret sharing (ArSS). Specifically, an  $l$ -bit value  $x$  is shared additively in the integer ring  $\mathbb{Z}_{2^l}$  as the sum of two values, e.g.,  $\langle x\rangle_s$  and  $\langle x\rangle_c$ .  $x$  can be reconstructed as  $\langle x\rangle_s + \langle x\rangle_c \mod 2^l$ . In the 2PC framework,  $x$  is secretly shared with the server holding  $\langle x\rangle_s$  and the client holding  $\langle x\rangle_c$ .
ArSS supports both addition and multiplication on the secret shares. Addition can be conducted locally while multiplication requires helper data, which are independent of the secret shares and is generated through communication [43, 42]. The communication cost of a single multiplication of secret shares is  $O(l(\lambda + l))$ . When  $t$  multiplications share the same multiplier, instead of simply repeating the multiplication protocol for  $t$  times, [8, 31, 43] proposes a batch optimization algorithm that only requires  $O(l(\lambda + tl))$  communication, enabling much more efficient batched multiplications.
Most 2PC frameworks use fixed-point arithmetic. The multiplication of two fixed-point values of  $l$ -bit precision results in a fixed-point value of  $2l$ -bit precision. Hence, in order to perform subsequent arithmetics, a truncation is required to scale the value down to  $l$ -bit precision. Truncation usually executes after the convolutions and before the ReLUs. It requires complex communication and sometimes leads to even more online communication compared to ReLU as shown in Figure 1. We refer interested readers to [43] for more details.
2.3 Winograd Convolution
We first summarize all the notations used in the paper in Table 2. Then, consider a 1D convolution with  $m$  outputs and filter size of  $r$ , denoted as  $F(m,r)$ . With regular convolution,  $F(m,r)$  requires  $mr$  multiplications between the input and the filter. With the Winograd algorithm,  $F(m,r)$  can be computed differently. Consider the example of  $F(2,3)$  below:
$$
X = \left[ \begin{array}{c c c c} x _ {0} & x _ {1} & x _ {2} & x _ {4} \end{array} \right] ^ {\top} \quad W = \left[ \begin{array}{c c c c} w _ {0} & w _ {1} & w _ {2} \end{array} \right] ^ {\top} \quad Y = \left[ \begin{array}{c c} y _ {0} & y _ {1} \end{array} \right] ^ {\top}
$$
$$
\left[ \begin{array}{c c c} x _ {0} & x _ {1} & x _ {2} \\ x _ {1} & x _ {2} & x _ {3} \end{array} \right] \left[ \begin{array}{c} w _ {0} \\ w _ {1} \\ w _ {2} \end{array} \right] = \left[ \begin{array}{c} m _ {0} + m _ {1} + m _ {2} \\ m _ {1} - m _ {2} - m _ {3} \end{array} \right] = \left[ \begin{array}{c} y _ {0} \\ y _ {1} \end{array} \right]
$$
where  $m_0, m_1, m_2, m_3$  are computed as
$$
\begin{array}{l l} m _ {0} = (x _ {0} - x _ {2}) w _ {0} & m _ {1} = (x _ {1} + x _ {2}) (w _ {0} + w _ {1} + w _ {2}) / 2 \\ m _ {3} = (x _ {1} - x _ {3}) w _ {2} & m _ {2} = (x _ {2} - x _ {1}) (w _ {0} - w _ {1} + w _ {2}) / 2. \end{array}
$$
With the Winograd algorithm, the number of multiplications is reduced from 6 in the regular convolution to 4. Similar Winograd transformation can be applied to 2D convolutions by nesting 1D algorithm with itself [34]. The 2D Winograd transformation  $F(m \times m, r \times r)$ , where the output tile size is  $m \times m$ , the filter size is  $r \times r$ , and the input tile size is  $n \times n$ , where  $n = m + r - 1$ , can be formulated as follows,
$$
Y = W \circledast X = A ^ {\top} \left[ \left(G W G ^ {\top}\right) \odot \left(B ^ {\top} X B\right) \right] A, \tag {1}
$$
where  $\circledast$  denotes the regular convolution and  $\odot$  denotes element-wise matrix multiplication (EWMM).  $A$ ,  $B$ , and  $G$  are transformation matrices that are independent of  $W$  and  $X$  and can be computed based on  $m$  and  $r$  [34, 1].
With the Winograd algorithm, the multiplication of such a 2D convolution can be reduced from  $m^2 r^2$  to  $n^2$ , i.e.,  $(m + r - 1)^2$  at the cost of performing more additions which can be computed locally in private inference scenario. While the reduction of multiplication increases with  $m$ ,  $m = 2$  and  $m = 4$  are most widely used for the better inference precision [34, 49, 3, 15]. More details about Winograd convolution are available in Appendix D.
2.4 Related Works
To improve the efficiency of private inference, existing works can be categorized into three classes, including protocol optimization [8, 38, 43, 42, 29, 41, 39], network optimization [26, 7, 19, 40, 37, 6, 32, 27, 51, 35], and joint optimization [48, 25, 17]. In Table 1, we compare CoPriv with these works qualitatively and as can be observed, CoPriv leverages both protocol and network optimization and can simultaneously reduce the online and total communication induced by convolutions, truncations, and ReLUs through network optimization. We leave more detailed review of existing works to Appendix A and more detailed comparison in terms of their techniques to Appendix F.
In this section, we analyze the origin of the limited communication reduction of existing ReLU-optimized networks and discuss our key observations that motivates CoPriv.
Observation 1: the total communication is dominated by convolutions while the online communication cost of truncations and ReLUs are comparable. As shown in Figure 1, the main operations that incurs high communication costs are convolutions, truncations, and ReLUs. With CrypTFlow2, over  $95\%$  communication is in the pre-processing stage generated by convolutions, while the truncations and ReLUs requires similar communication. This observation contradicts to the assumption of previous works [7, 6, 19, 32]: on one hand, ReLU no longer dominates the online communication and simply reducing ReLU counts leads to diminishing return for online communication reduction; on the other hand, pre-processing communication cannot be ignored as it not only consumes a lot of power in the server and client but also can slow down the online stage. As pointed out by a recent system analysis [18], for a practical server that accepts inference request under certain arrival rates, the pre-processing stage is often incurred online as there is insufficient downtime to hide its latency. Therefore, we argue that both the total and the online communication are crucial and need to be reduced to enable more practical 2PC-based inference.
Observation 2: the communication cost scaling of convolution, truncation and ReLU is different with the network dimensions. The communication cost of different operations scales differently with the network dimensions. For a convolution, the major operations are multiplications and additions. While additions are computed fully locally, multiplications require extra communication to generate the helper data. Hence, the communication cost for a convolution scales linearly with the number of multiplications and the complexity is  $O(CKr^2(\lambda + HW))$  following [25]. Both truncations and ReLUs are element-wise and hence, their communication cost scales linearly with the input feature size, i.e.,  $O(HWC)$ . Therefore, for networks with a large number of channels, the pre-processing communication becomes even more dominating.
Since the communication is dominated by the multiplications in convolution layers, our intuition is to reduce the multiplications by conducting more local additions for better communication efficiency. Meanwhile, as in Figure 2, MobileNetV2 are much more communication efficient compared to ResNets with a similar accuracy, and hence, becomes the focus of our paper.
Observation 3: ReLU pruning and network re-parameterization can simultaneously reduce the online and total communication of all operations. Existing network optimizations either linearize the ReLUs selectively [26, 7, 40, 37, 6, 32] or prune the whole channel or layer [36]. Selective ReLU linearization usually achieves a better accuracy but achieves limited communication reduction. In contrast, channel-wise or layer-wise pruning can simultaneously reduce the communication for all the operators but usually suffers from a larger accuracy degradation [26]. We observe for an inverted residual block in MobileNetV2, as shown in Figure 3, if both of the two ReLUs are removed, the block can be merged into a dense convolution after training. During training, this preserves the benefits of over-parameterization shown in RepVGG [14] to protect network accuracy, while during
Figure 2: Comparison of communication breakdown between ResNet-50 and MobileNetV2 on ImageNet.
Figure 3: Illustration of ReLU pruning and network re-parameterization for communication reduction. DW means depth-wise and  $1 \times 1$  Conv is also called point-wise (PW) convolution.
Figure 4: Overview of CoPriv and the communication cost after each optimization step. The example of the communication cost is measured on the CIFAR-100 dataset.
inference, the dense convolution is more compatible with our optimized protocol and can reduce communication of truncations as well.
4 CoPriv: A New Paradigm Towards Efficient Private Inference
In this section, we present CoPriv, a protocol-network co-optimization framework for communication-efficient 2PC-based inference. The overview of CoPriv is shown in Figure 4. Given the original network, we first leverage the Winograd transformation with DNN-aware optimization to reduce the communication when computing 3-by-3 convolutions without any accuracy impact (Section 4.1). The optimized transformation enables to reduce the total communication of ResNet-18 by  $2.1 \times$ . However, the communication reduction of MobileNetV2 is limited as the Winograd transformation can only be applied to the 3-by-3 depth-wise convolution, which only contributes to a small portion of the total communication. Hence, for MobileNetV2, we propose an automatic and differentiable ReLU pruning algorithm (Section 4.2). By carefully designing the pruning pattern, CoPriv enables further network re-parameterization of the pruned networks, which reduces the online and total communication by  $5 \times$  and  $5 \times$  in the example, respectively, combining with our optimized convolution protocol.
4.1 Winograd Transformation for Protocol Optimization
We propose to leverage the Winograd transformation to reduce the communication when computing a convolution. Following Eq. 1, for each 2D output tile, because  $A, B, G$  are known publicly, the filter transformation  $GWG^{\top}$  and feature transformation  $B^{\top}XB$  can be computed locally on the server and client [17]. Then, EWMM is performed together by the server and the client through communication while the final output transformation  $A^{\top}[\cdot]A$  can be computed locally as well.
For each output tile, as  $(m + r - 1)^2$  multiplications are needed and there is no shared multiplier for batch optimization described in Section 2.2, the communication cost is  $O(\lambda (m + r - 1)^2)$ . Consider there are in total  $T = \lceil H / m\rceil \times \lceil W / m\rceil$  tiles per output channel,  $C$  input channels, and  $K$  output channels, the communication complexity of a convolution layer now becomes  $O(\lambda TCK(m + r - 1)^2)$ . To reduce the enormous communication, we present the following optimizations.
2For simplicity, we omit the bit precision  $l$  for all the downstream analysis.
Figure 5: Winograd transformation procedure with tile aggregation.
Table 3: Communication complexity of different convolution types and the measured communication for a ResNet-18 block with  $14 \times 14 \times 256$  feature dimension.
Conv. Type	Comm. Complexity	ResNet Block Comm. (GB)
Regular Conv.	O(CKr2(λ + HW))	23.74 (1×)
EWMM-based Winograd Conv.	O(λCKT(m + r - 1)2)	256 (10.78×↑)
+ Tile Aggregation (GEMM-based)	(m + r - 1)2CT(λ + K)	10.45 (2.27×↓)
+ DNN-aware Sender Selection	min[(m + r - 1)2CT(λ + K), (m + r - 1)2CK(λ + T)]	7.76 (3.06×↓)
Communication reduction with tile aggregation While the Winograd transformation helps to reduce the total number of multiplications, in Table 3, we observe the communication for a ResNet block actually increases. This is because the baseline protocol can leverage the batch optimization mentioned in Section 2.2 to reduce the communication complexity. To further optimize the Winograd-based convolution protocol to adapt to private inference, we observe the following opportunities for the batch optimization: first, each transformed input tile needs to multiply all the  $K$  filters; secondly, each transformed filter needs to multiply all the transformed input tiles for a given channel. Hence, for the  $t$ -th tile in  $f$ -th output channel, we re-write Eq. 1 as follows:
$$
Y _ {f, t} = A ^ {\top} \left[ \sum_ {c = 1} ^ {C} U _ {f, c} \odot V _ {c, t} \right] A \quad 1 \leq f \leq K, 1 \leq t \leq T,
$$
where  $U_{f,c}$  denotes the transformed filter corresponding to the  $f$ -th output channel and  $c$ -th input channel and  $V_{c,t}$  denotes the transformed input of the  $t$ -th tile in the  $c$ -th input channel. Then, consider each pixel location, denoted as  $(\xi, \nu)$ , within the tile separately, yielding:
$$
Y _ {f, t} ^ {(\xi , \nu)} = A ^ {\top} \left[ U _ {f, c} ^ {(\xi , \nu)} V _ {c, t} ^ {(\xi , \nu)} \right] A \quad 1 \leq f \leq K, 1 \leq t \leq T, 1 \leq \xi , \nu \leq m + r - 1
$$
We illustrate this transformation procedure in Figure 5. Re-writing the EWMM into general matrix multiplication (GEMM) enables us to fully leverage the batch optimization and reduce the communication complexity as shown in Table 3. We can now reduce the communication of a ResNet block by  $2.27 \times$
DNN-aware adaptive convolution protocol When executing the convolution protocol, both the server and the client can initiate the protocol. Because the input feature map and the filter are of different dimensions, we observe the selection of protocol initializer impacts the communication round as well as the communication complexity. While CypTFlow2 always selects the server to initialize the protocol, we propose DNN architecture-aware convolution protocol to choose between the server and the client adaptively based on the layer dimensions to minimize the communication complexity. As shown in Table 3, the communication of the example ResNet block can be further reduced by  $1.35 \times$  with the adaptive protocol.
4.2 Differentiable ReLU Pruning and Network Re-Parameterization
We now describe our network optimization algorithm to further reduce both pre-processing and online communication for MobileNetV2. The core idea is to simultaneously remove the two RLUs
within an inverted residual block together, after which the entire block can be merged into a dense convolution layer to be further optimized with our Winograd-based protocol as in Figure 3. For the example in Figure 4, both the pre-processing and online communication for MobileNetV2 can be reduced by  $5 \times$ . To achieve the above goal, the remaining questions to answer include: 1) which ReLUs to remove and how to minimize the accuracy degradation for the pruned networks, and 2) how to re-parameterize the inverted residual block to guarantee functional correctness. Hence, we propose the following two-step algorithm.
Step 1: communication-aware differentiable ReLU pruning To identify "unimportant" activation functions, CoPriv leverages a differentiable pruning algorithm. Specifically, CoPriv assigns an architecture parameter  $\alpha (0\leq \alpha \leq 1)$  to measure the importance of each ReLU. During pruning, the forward function of a ReLU now becomes  $\alpha \cdot \mathrm{ReLU}(x) + (1 - \alpha)\cdot x$  . CoPriv jointly learns the model weights  $\theta$  and the architecture parameters  $\alpha$  . Specifically, given a sparsity constraint  $s$  , we propose the one-level optimization formulated as follows:
$$
\min  _ {\theta , \alpha} \mathcal {L} _ {C E} + \mathcal {L} _ {c o m m} \quad \text {s . t .} \quad \| \alpha \| _ {0} \leq s, \tag {2}
$$
where  $\mathcal{L}_{CE}$  is the task-specific cross entropy loss and  $\mathcal{L}_{comm}$  is the communication-aware regularization to focus the pruning on communication-heavy blocks and is defined as
$$
\mathcal {L} _ {\text {c o m m}} = \sum_ {i} \alpha_ {i} (\operatorname {C o m m} _ {i} (\alpha_ {i} = 1) - \operatorname {C o m m} _ {i} (\alpha_ {i} = 0)),
$$