SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathrm{Comm}_i(\alpha_i = 1)$  is the communication to compute the  $i$ -th layer when  $\alpha_{i} = 1$ . We also show the effectiveness of communication-aware regularization in the experimental results. To ensure the network after ReLU pruning can be merged, two ReLUs within the same block share the same  $\alpha$ . During training, in the forward process, only the top- $s$ $\alpha$ 's are activated (i.e.,  $\alpha = 1$ ) while the remainings are set to 0. In the backpropagation process,  $\alpha$ 's are updated via straight-through estimation (STE) [4].
Once the pruning finishes, the least important RLUs are removed and we perform further finetuning to improve the network accuracy. Specifically, in CoPriv, we leverage knowledge distillation (KD) [21] to guide the finetuning of the pruned network.
Step 2: network re-parameterization The removal of ReLUs makes the inverted residual block linear, and thus can be further merged together into a single dense convolution. Motivated by the previous work of structural re-parameterization [16, 14, 13, 11, 10, 12], we describe the detailed re-parameterization algorithm in Appendix C. The re-parameterized convolution has the same number of input and output channels as the first and last point-wise convolution, respectively. Its stride equals to the stride of the depth-wise convolution.
We adopt CypTFlow2 [43] protocol for the 2PC-based inference, and we measure the communication and latency under a LAN setting [43] with 377 MBps bandwidth and 0.3ms echo latency. For Winograd, we implement  $F(2 \times 2, 3 \times 3)$  and  $F(4 \times 4, 3 \times 3)$  transformation for convolution with stride of 1 and  $F(2 \times 2, 3 \times 3)$  transformation when stride is 2 [23]. We apply CoPriv to MobileNetV2 with different width multipliers on CIFAR-100 [30] and ImageNet [9] datasets. The details of our experimental setups including the private inference framework, the implementation of the Winograd-based convolution protocol and the training details are available in Appendix B.
5.2 Micro-Benchmark on the Convolution Protocol with Winograd Transformation
We first benchmark the communication reduction of the proposed convolution protocol based on Winograd transformation. The results on ResNet-18 and ResNet-32 are shown in Figure 6(a) and (b). As can be observed, the proposed protocol consistently reduces the communication to compute the convolutions. While the degree of communication reduction depends on the layer dimensions, on average,  $2.1 \times$  reduction is achieved for both ResNet-18 and ResNet-32. We also
(a) ResNet-18
(b) ResNet-32
(c) Influence of Feature Dimension
Figure 6: Communication of the convolution protocol with Winograd transformation on (a) ResNet-18 and (b) ResNet-32 on CIFAR-100; (c) comparison between  $F(2 \times 2, 3 \times 3)$  and  $F(4 \times 4, 3 \times 3)$  with different feature dimensions.
Figure 7: Comparison with efficient ReLU-optimized methods on CIFAR-100 dataset.
benchmark the Winograd-based protocol with output tile sizes of 2 and 4, i.e.,  $F(2 \times 2, 3 \times 3)$  and  $F(4 \times 4, 3 \times 3)$ . Figure 6(c) shows that when the feature resolution is small, a small tile size achieves more communication reduction while larger resolution prefers a larger tile size. Hence, for the downstream experiments, we use  $F(2 \times 2, 3 \times 3)$  for CIFAR-100 and  $F(4 \times 4, 3 \times 3)$  for ImageNet [9] dataset, respectively.
5.3 Benchmark with ReLU-Optimized Networks on CIFAR-100
We benchmark CoPriv with different network optimization algorithms that focus on reducing ReLU counts, including DeepReDuce [26], SENet [32] and SNL [7], on the CIFAR-100 dataset. As DeepReDuce, SENet and SNL all use ResNet-18 as the baseline model, we also train SNL algorithm on the MobileNetV2-w0.75 for a comprehensive comparison.
Results and analysis From Figure 7, we make the following observations: 1) previous methods, including DeepReDuce, SENet, SNL, do not reduce the pre-processing communication and their online communication reduction quickly saturates with a notable accuracy degradation; 2) CoPriv achieves SOTA Pareto front of accuracy and online/total communication. Specifically, CoPriv outperforms SNL on ResNet-18 with  $4.3\%$  higher accuracy as well as  $1.55 \times$  and  $19.5 \times$  online and total communication reduction, respectively. When compared with SNL on MobileNetV2-w0.75, CoPriv achieves  $1.67\%$  higher accuracy with  $1.66 \times$  and  $2.44 \times$  online and total communication reduction, respectively.
5.4 Benchmark on ImageNet
We also benchmark CoPriv on the ImageNet dataset with the following baselines: lightweight mobile networks like MobileNetV3 [22] with different capacities, ReLU-optimized networks, including SNL [7] and SENet [32], and SOTA pruning methods, including uniform pruning and MetaPruning [36]. We train both our CoPriv and SNL on the MobileNetV2-w1.4 with self distillation.
Results and analysis From the results shown in Figure 8, we observe that 1) compared with SNL on MobileNetV2-w1.4, CoPriv achieves  $1.4\%$  higher accuracy with  $9.98 \times$  and  $3.88 \times$  online and total communication reduction, respectively; 2) CoPriv outperforms  $1.8\%$  higher accuracy compared
Figure 8: Comparison with SOTA efficient ConvNets, ReLU-optimized methods, channel-wise pruning MetaPruning [36] and uniform pruning on ImageNet dataset. We mark the four points of CoPriv as CoPriv-w1.4-A/B/C/D from left to right, respectively.
Table 4: Ablation study of our proposed optimizations in CoPriv on CIFAR-100. Two sparsity constraints are on the two sides of “/”, respectively.
Model	Online Communication (GB)	Total Communication (GB)
Baseline SNL (ResNet-18)	0.507	61.45
+Winograd	0.507	33.84
Baseline SNL (MobileNetV2)	0.530	7.710
+Winograd	0.530	7.132
MobileNetV2+Pruning	0.606 / 0.582	7.806 / 7.761
+Re-parameterization	0.331 / 0.260	4.197 / 3.546
+Winograd (CoPriv)	0.331 / 0.260	3.154 / 2.213
with MobileNetV3-Small-1.0 while achieving  $9.41 \times$  and  $1.67 \times$  online and total communication reduction; 3) CoPriv demonstrates its strong scalability for communication optimization. Compared to the baseline MobileNetV2-w1.4, CoPriv achieves  $2.52 \times$  and  $1.9 \times$  online and total communication reduction, respectively without compromising the accuracy; 4) when compared with SOTA pruning methods, for a high communication budget, CoPriv-w1.4-B achieves  $1.6\%$  higher accuracy with  $2.28 \times$  and  $1.08 \times$  online and total communication reduction, compared with MetaPruning-1.0  $\times$ ; for a low communication budget, compared with MetaPruning-0.35  $\times$ , CoPriv-w1.4-D achieves  $5.5\%$  higher accuracy with  $3.87 \times$  online communication reduction.
Effectiveness of different optimizations in CoPriv To understand how different optimizations help improve communication efficiency, we add our proposed protocol optimizations step by step on both SNL and CoPriv, and present the results in Table 4. According to the results, we find that 1) our Winograd-based convolution protocol consistently reduces the total communication for both networks; 2) when directly applying the Winograd transformation on MobileNetV2, less than  $10\%$  communication reduction is achieved as Winograd transformation only helps the depth-wise convolution, which accounts for only a small portion of pre-processing communication; 3) compared with SNL, CoPriv achieves higher accuracy with much lower communication. The findings indicate that all of our optimizations are indispensable for improving the communication efficiency.
Block-wise communication comparison and visualization We compare and visualize the blockwise communication reduction of CoPriv on MobileNetV2-w0.75 on the CIFAR-100 dataset. From Figure 9, it is clear that our adaptive convolution protocol is effective for communication optimization, and different layers benefit from CoPriv differently. More specifically, for the total communication, block # 4 benefits more from the Winograd transformation over the network re-parameterization, while block # 16 benefits from both the re-parameterization and the adaptive convolution protocol. Both pruning and re-parameterization are important for online communication as they remove the communication of ReLU and truncation, respectively. The results demonstrate the importance of all the proposed optimization techniques.
Figure 9: Block-wise visualization of online and total communication with different techniques on CoPriv-w0.75 on CIFAR-100.
5.6 Effectiveness of Communication-Aware Regularization
To show the effectiveness and importance of introducing communication-aware regularization  $\mathcal{L}_{comm}$  formulated in Eq. 2, we compare our pruning method with  $\mathcal{L}_{comm}$  and pruning methods of SNL [7] and SENet [32] without  $\mathcal{L}_{comm}$  in Figure 10. As we can observe,  $\mathcal{L}_{comm}$  indeed helps to focus the pruning on the later layers, which incur more communication cost, and penalizes the costly blocks (e.g., block # 16 and # 18). In contrast, SNL does not take communication cost into consideration and SENet (ReLU sensitivity based) focuses on pruning early layers with more ReLUs, both of which incur huge communication overhead.
Figure 10: Comparison of different pruning method and the influence of  $\mathcal{L}_{comm}$  during the search in each block (ReLU budget is set to  $60\%$  in this example).
In this work, we propose a network/protocol co-optimization framework, CoPriv, that simultaneously optimizes the pre-processing and online communication for the 2PC-based private inference. CoPriv features an optimized convolution protocol based on Winograd transformation and leverages a series of DNN-aware protocol optimization to improve the efficiency of the pre-processing stage. We also propose a differentiable communication-aware ReLU pruning algorithm with network reparameterization to further optimize both the online and pre-processing communication induced by ReLU, truncation and convolution. With extensive experiments, CoPriv consistently reduces both online and total communication without compromising the accuracy compared with different prior-art efficient networks and network optimization algorithms.
This work was supported in part by the NSFC (62125401) and the 111 Project (B18001).
[47] Liyan Shen, Ye Dong, Binxing Fang, Jinqiao Shi, Xuebin Wang, Shengli Pan, and Ruisheng Shi. Abnn2: secure two-party arbitrary-bitwidth quantized neural network predictions. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 361-366, 2022.
[48] Wenting Zheng Srinivasan, PMRL Akshayaram, and Popa Raluca Ada. Delphi: A cryptographic inference service for neural networks. In Proc. 29th USENIX Secur. Symp, pages 2505-2522, 2019.
[49] Kevin Vincent, Kevin Stephano, Michael Frumkin, Boris Ginsburg, and Julien Demouth. On improving the numerical stability of winograd convolutions. 2017.
[50] Juan Yepez and Seok-Bum Ko. Stride 2 1-d, 2-d, and 3-d winograd for convolutional neural networks. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 28(4):853-863, 2020.
[51] Wenxuan Zeng, Meng Li, Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpcvit: Searching for accurate and efficient mpc-friendly vision transformer with heterogeneous attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5052-5063, 2023.
Private inference has been a promising solution to protect both data and model privacy during deep learning inference. In recent years, there has been an increasing amount of literature on efficient private inference. According to the optimization technique, these works can be categorized into three types, i.e., 1) protocol optimization; 2) network optimization; and 3) joint optimization.
In protocol optimization, ABY [8] provides a highly efficient conversion between arithmetic sharing, boolean sharing and Yao's sharing, and construct mixed protocols. As an extension, ABY3 [38] switches back and forth between three secret sharing schemes using three-party computation (3PC). CypTFlow2 [43] proposes a new protocol for secure and comparison and division which enables efficient non-linear operations such as ReLU. SiRNN [42] further proposes 2PC protocols for bitwidth extension, mixed-precision linear and non-linear operations. CrypTen [29] proposes a software framework that provides a flexible machine learning focused API. More recently, SecFloat [41] proposes the crypto-friendly precise functionalities to build a library for 32-bit single-precision floating-point operations and math functions. These works lack consideration for neural network architecture and has limited communication reduction.
In network optimization, DeepReLU [26] proposes to manually remove ReLUs with a three-step optimization pipeline. SNL [7] proposes ReLU-aware optimization that leverages gradient-based NAS to selectively linearize a subset of ReLUs. CryptoNAS [19] uses ReLU budget as a proxy and leverages NAS to tailor ReLUs. PolyMPCNet [40] and SAFENet [37] replace ReLUs with MPC-friendly polynomial, while Sphynx [6] proposes an MPC-friendly ReLU-efficient micro-search space. SENet [32] innovatively measures the ReLU importance via layer pruning sensitivity and automatically optimize the network to meet the target ReLU budget. DeepReLU [26] finds that wider networks are more ReLU-efficient than the deeper ones and designs ReLU-efficient baseline networks with FLOPs-ReLU-Accuracy balance. For Transformer-based models, MPCFormer [35] proposes to simplify Softmax by replacing exponential with a more MPC-friendly quadratic operation. MPCViT [51] proposes to mix both high accuracy and high efficiency attention variants to accelerate private Vision Transformer (ViT) inference. Network optimization mainly focuses on ReLU reduction which dominates the online communication, but total communication including convolution and truncation cannot be optimized.
Unluckily, only using either protocol or network optimization just leads to limited efficiency improvement. Delphi [48] jointly optimizes cryptographic protocols and network by gradually replacing ReLU with quadratic approximation. COINN [25] simultaneously optimizes quantized network and protocols with ciphertext-aware quantization and automated bitwidth configuration. Recently, [17] proposes to use Winograd convolution for reducing the number of multiplications and design the efficient convolution operation to reduce the communication cost. However, it does not take private inference into consideration for Winograd algorithm, and still suffers tremendous communication overhead. In this work, we jointly optimize the network and protocol and fully consider their coupling properties.
B Details of Experimental Setup
Private inference framework CoPriv is built based on CypTFlow2 [43] protocol for private inference. We leverage the Athos [43] tool chain to convert both input and weight into fixed-point with the bit-width 41 and scale 12. We measure the communication and latency under a LAN setting [43] with 377 MBps bandwidth and 0.3ms echo latency. All of our experiments are evaluated on the Intel Xeon Gold 5220R CPU @ 2.20GHz.
Implementation of Winograd-based convolution protocol The convolution protocol with Winograd transformation and optimization is implemented in  $\mathrm{C + + }$  with Eigen and Armadillo matrix calculation library [45] in the CrypTFlow2 [43] framework. We implement  $F(2\times 2,3\times 3)$  and  $F(4\times 4,3\times 3)$  transformation for convolution with stride of 1 and  $F(2\times 2,3\times 3)$  transformation when stride is 2 [23]. For CIFAR-100 dataset, we use  $F(2\times 2,3\times 3)$  transformation as the image resolution is small and for ImageNet dataset, we use  $F(4\times 4,3\times 3)$ . We only apply  $F(2\times 2,3\times 3)$  for stride of 2 on ImageNet dataset. When evaluating CoPriv, we determine the optimal sender according to the analysis in Table 3 before inference. Winograd implementation enables us to measure the communication cost and latency of each convolution module.
Figure 11: An example of re-parameterization for batch normalization and convolution filter.
Networks and datasets We apply our proposed CoPriv to the widely used lightweight mobile network MobileNetV2 [46] with different width multipliers, e.g., 0.75, 1.0 and 1.4 to trade off the model accuracy and efficiency. We evaluate the top-1 accuracy and online and total communication on both CIFAR-100 and ImageNet [9] dataset.
Differentiable pruning and finetuning setups We first search and prune redundant RLUs for 10 epochs and then finetune the pruned network for 180 epochs with stochastic gradient descent (SGD) optimizer [2], cosine learning scheduler and initial learning rate of 0.1. During finetuning stage, we train our proposed CoPriv with knowledge distillation to boost its performance.
C Network Re-Parameterization Algorithm
Network (or structural) re-parameterization is a useful technique proposed by RepVGG [14], and is extended to [11, 10, 13, 16, 12]. The core idea of re-parameterization is to decouple the training-time architecture (with high performance and low efficiency) and inference-time network architecture (with equivalent performance and high efficiency). Re-parameterization is realized by converting one architecture to another via equivalently merging parameters together. Therefore, during inference time, the network architecture is not only efficient but also has the same high performance as the training-time architecture. Figure 11 is an simple example of re-parameterization.
In this work, we can also leverage this technique to merge adjacent convolutions together after ReLU removal. For the network re-parameterization mentioned in Section 4.2, here we provide the following detailed algorithm 1 to equivalently merge the inverted residual block into a single dense convolution as shown in Figure 3. With the help of network re-parameterization, we further optimize the total communication including convolution and truncation.
Algorithm 1: Network Re-parameterization for Inverted Residual Block
Input: An inverted residual block with weights  $W_{1 \times 1}$ ,  $W_{3 \times 3}$ , and  $W_{1 \times 1}'$ . The number of input and output channels  $N_{in}$ ,  $N_{out}$ . The size of re-parameterized weights  $r$ .  
Output: Regular convolution with re-parameterized weights  $W_r$ .  
 $W_r = \text{torch.eye}(N_{in})$ ;  
 $W_r = W_r$ .unsqueeze(2).unsqueeze(2);  
 $W_r = \text{torch(nn)}.functional_pad(W_r, pad = (\frac{r-1}{2}, \frac{r-1}{2}, \frac{r-1}{2}, \frac{r-1}{2}))$ ;  
 $W_r = \text{torch(nn)}.functional.conv2d(W_r, W_{1 \times 1})$ ;  
 $W_r = \text{torch(nn)}.functional.conv2d(W_r, W_{3 \times 3}, padding = \frac{r-1}{2})$ ;  
 $W_r = \text{torch(nn}.functional.conv2d(W_r, W_{1 \times 1}')$ ;  
 $W_{res} = \text{torch.zeros}(N_{out}, N_{in}, r, r)$ ;  
for  $i \in [0, \dots, N_{out} - 1]$  do  
 $\left\lfloor W_{res}[i, i, \lfloor r/2 \rfloor, \lfloor r/2 \rfloor] = 1; \right.$ $W_r = W_r + W_{res}$ ;  
return  $W_r$ ;
D Details of Winograd Convolution
D.1 Comparison between Regular Convolution and Winograd Convolution
To help readers better understand the multiplication reduction of Winograd convolution, we demonstrate regular convolution and Winograd convolution in Figure 12. Given an input  $I \in \mathbb{R}^{4 \times 4}$  and
(a) Regular convolution with  $9 \times 4 = 36$  #MUL.
(b) Winograd convolution with  $16 \times 1 = 16$  #MUL in  $\mathrm{F}(2 \times 2, 3 \times 3)$ .
Figure 12: Comparison between (a) regular convolution and (b) Winograd convolution.
a filter  $F \in \mathbb{R}^{3 \times 3}$ , regular convolution requires  $9 \times 4 = 36$  times multiplications (implemented using GEMM with im2col algorithm [5]) while  $F(2 \times 2, 3 \times 3)$  Winograd transformation only requires  $16 \times 1 = 16$  times multiplications (EWMM), which achieves  $2.25 \times$  reduction. Moreover,  $F(4 \times 4, 3 \times 3)$  with a larger tile size, i.e., 6 can further achieve  $4 \times$  multiplication reduction. The improvement gets benefit from the Winograd's ability to convert im2col to EWMM and calculate the whole tile in Winograd domain at once.
D.2 Details of Input Tiling and Padding
Given a large 2D input  $I \in \mathbb{R}^{l \times l}$ , where  $l > m + r - 1$ , the core technique for ensuring the equivalence of regular convolution and Winograd convolution is input tiling and padding. The output size  $l' = l - r + 1$ , the input tile size  $n = m + r - 1$  and the total tile number  $T$  per channel is computed as
$$
T = \lceil \frac {l ^ {\prime}}{n} \rceil^ {2} = \lceil \frac {l - r + 1}{m + r - 1} \rceil^ {2},
$$
where  $\lceil \cdot \rceil$  denotes taking the upper bound value. For each tile, Winograd convolution is individually performed and results an output tile with  $m\times m$  size. After all the tiles are computed with Winograd convolution, the output tiles are concatenated together to form the final output.
For some input size, the input cannot be covered by tiles. For instance, when leveraging  $F(2 \times 2, 3 \times 3)$  on the input  $I \in \mathbb{R}^{7 \times 7}$ , the rightmost and bottom pixels cannot be divided into a complete tile. To solve this problem, we pad these positions with 0 to enable the tiles totally cover the whole input. The correctness and equivalence can be proved with Eq. 1. Also, [17] shows the overhead caused by padding is negligible.
D.3 Support for Stride of 2 Winograd Convolution
Conventional Winograd convolution only supports stride  $s = 1$  convolution filter. However, in recent efficient neural networks, e.g., MobileNetV2, EfficientNet has several stride of 2 layers to reduce the feature map size by half. To enable extreme optimization for efficient networks, we introduce  $F(2 \times 2, 3 \times 3)$  for stride of 2 Winograd convolution for private inference.
There are various methods to construct stride of 2 Winograd kernel such as dividing input and convolution filter into different groups [50]. However, it is not a simple way to implement stride of 2 Winograd kernel. [23] is an extremely convenient method using unified transformation matrices.
Based on [23], even positions of input and filter are computed by  $F(2,2)$  while odd positions are computed by regular convolution. Transformation matrices are derived as follows and can be computed using Eq. 1:
$$
B ^ {\top} = \left[ \begin{array}{c c c c c} 1 & 0 & - 1 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & - 1 & 0 & 1 \end{array} \right], \quad G = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right], \quad A ^ {\top} = \left[ \begin{array}{c c c c c} 1 & 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 & 1 \end{array} \right].
$$
Correctness analysis. Here, we take a 1D algorithm as an example to prove the correctness Winograd convolution for stride of 2. The algorithm can be nested with itself to obtain a 2D algorithm [34].
Given input  $X$  and filter  $F$  as
$$
X = \left[ \begin{array}{c} x _ {0} \\ x _ {1} \\ x _ {2} \\ x _ {3} \\ x _ {4} \end{array} \right], \quad F = \left[ \begin{array}{c} y _ {0} \\ y _ {1} \\ y _ {2} \end{array} \right], \quad Y = X   \circledast   F = \left[ \begin{array}{c} z _ {0} \\ z _ {1} \end{array} \right].
$$
First, we calculate regular convolution with stride of 2 using im2col algorithm [5] as
$$
Y _ {1} = \left[ \begin{array}{c c c} x _ {0} & x _ {1} & x _ {2} \\ x _ {2} & x _ {3} & x _ {4} \end{array} \right] \cdot \left[ \begin{array}{c} y _ {0} \\ y _ {1} \\ y _ {2} \end{array} \right] = \left[ \begin{array}{c} x _ {0} y _ {0} + x _ {1} y _ {1} + x _ {2} y _ {2} \\ x _ {2} y _ {0} + x _ {3} y _ {1} + x _ {4} y _ {2} \end{array} \right],
$$
thus,  $z_0 = x_0y_0 + x_1y_1 + x_2y_2$  and  $z_{1} = x_{2}y_{0} + x_{3}y_{1} + x_{4}y_{2}$
Then, we calculate Winograd convolution for stride of 2 as
$$
Y = A ^ {\top} \cdot \left[ (G F) \odot (B ^ {\top} X) \right],
$$