SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
Y _ {2} = \left[\begin{array}{c c c c c}1&1&1&0&0\\0&0&1&1&1\end{array}\right] \cdot \left[\left(\begin{array}{c c c}1&0&0\\0&1&0\\1&0&1\\0&1&0\\0&0&1\end{array}\right)\right. \odot \left(\left[\begin{array}{c c c c c}1&0&0\\0&1&0\\1&0&1\\0&1&0\\0&0&1\end{array}\right]\right) \odot \left(\left[\begin{array}{c c c c c}1&0&- 1&0&0\\0&1&0&0&0\\0&0&1&0&0\\0&0&0&1&0\\0&0&- 1&0&1\end{array}\right]\right) \cdot \left[\begin{array}{c}x _ {0}\\x _ {1}\\x _ {2}\\x _ {3}\\x _ {4}\end{array}\right]\left. \right),
$$
and further simplify the calculation as
$$
Y _ {2} = \left[ \begin{array}{c c c c c} 1 & 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 & 1 \end{array} \right] \cdot [ (\left[ \begin{array}{c} y _ {0} \\ y _ {1} \\ y _ {0} + y _ {2} \\ y _ {1} \\ y _ {2} \end{array} \right]) \odot (\left[ \begin{array}{c} x _ {0} - x _ {2} \\ x _ {1} \\ x _ {2} \\ x _ {3} \\ x _ {4} - x _ {2} \end{array} \right]) ] = \left[ \begin{array}{c c c c c} 1 & 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 & 1 \end{array} \right] \cdot \left[ \begin{array}{c} x _ {0} y _ {0} - x _ {2} y _ {0} \\ x _ {1} y _ {1} \\ x _ {2} y _ {0} + x _ {2} y _ {2} \\ x _ {3} y _ {1} \\ x _ {4} y _ {2} - x _ {2} y _ {2} \end{array} \right],
$$
therefore, the convolution result is
$$
Y _ {2} = \left[ \begin{array}{c} x _ {0} y _ {0} + x _ {1} y _ {1} + x _ {2} y _ {2} \\ x _ {2} y _ {0} + x _ {3} y _ {1} + x _ {4} y _ {2} \end{array} \right] = Y _ {1}.
$$
D.4 Transformation Matrices for Winograd Convolution
We provide the transformation matrices  $A, B, G$  for  $F(2 \times 2, 3 \times 3)$  and  $F(4 \times 4, 3 \times 3)$  Winograd transformation based on polynomial Chinese remainder theorem (CRT) or Lagrange interpolation [34].
For  $F(2 \times 2, 3 \times 3)$ , we have
$$
B ^ {\top} = \left[ \begin{array}{c c c c} 1 & 0 & - 1 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & - 1 & 1 & 0 \\ 0 & 1 & 0 & - 1 \end{array} \right], \quad G = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ 1 / 2 & 1 / 2 & 1 / 2 \\ 1 / 2 & - 1 / 2 & 1 / 2 \\ 0 & 0 & 1 \end{array} \right], \quad A ^ {\top} = \left[ \begin{array}{c c c c} 1 & 1 & 1 & 0 \\ 0 & 1 & - 1 & - 1 \end{array} \right].
$$
For  $F(4\times 4,3\times 3)$  , we have
$$
\begin{array}{l} B ^ {\top} = \left[ \begin{array}{c c c c c c} 4 & 0 & - 5 & 0 & 1 & 0 \\ 0 & - 4 & - 4 & 1 & 1 & 0 \\ 0 & 4 & - 4 & - 1 & 1 & 0 \\ 0 & - 2 & - 1 & 2 & 1 & 0 \\ 0 & 2 & - 1 & - 2 & 1 & 0 \\ 0 & 4 & 0 & - 5 & 0 & 1 \end{array} \right], \quad G = \left[ \begin{array}{c c c} 1 / 4 & 0 & 0 \\ - 1 / 6 & - 1 / 6 & - 1 / 6 \\ - 1 / 6 & 1 / 6 & - 1 / 6 \\ 1 / 2 4 & 1 / 1 2 & 1 / 6 \\ 1 / 2 4 & - 1 / 1 2 & 1 / 6 \\ 0 & 0 & 1 \end{array} \right], \\ A ^ {\top} = \left[ \begin{array}{c c c c c c} 1 & 1 & 1 & 1 & 1 & 0 \\ 0 & 1 & - 1 & 2 & - 2 & 0 \\ 0 & 1 & 1 & 4 & 4 & 0 \\ 0 & 1 & - 1 & 8 & - 8 & 1 \end{array} \right]. \\ \end{array}
$$
The correctness analysis is the same with Section D.3.
E Comparison with More Related Work
[33] is a recent work that aims at optimizing the network architecture for latency-efficient private inference. Both CoPriv and [33] aim to reduce the number of ReLUs and the depth of the whole network in order to improve the efficiency. However, they have totally different motivations and methods. 1) Motivations: [33] still regards ReLU as the main latency bottleneck and removes convolution layers to reduce computation. In contrast, CoPriv fuses neighboring convolution layers to better leverage our Winograd-based optimization for communication reduction of all operators. The difference in motivation leads to different criterion when selecting convolutions to remove. [33] selects convolutions based on ReLU sensitivity [32] while CoPriv simultaneously considers both accuracy and communication cost. 2) Methods: [33] first determines which convolutions to remove based on ReLU sensitivity [32] and then, uses the gated branching method for training. In contrast, CoPriv simultaneously train the architecture parameters with the model weights, and the equivalent re-parameterization is conducted post training, enabling us to leverage the benefits of overparameterization during training shown in RepVGG [14]. Quantitatively, we compare CoPriv with [33] in Table 5. CoPriv achieves much lower online and total communication while achieving higher accuracy compared to [33].
Table 5: Accuracy and communication comparison between CoPriv and [33].
Method	Top-1 Acc. (%)	Online Comm. (GB)	Total Comm. (GB)
[33]	69.10	0.81	46.5
CoPriv (ours)	70.58	0.43	5.14
F More Detailed Comparison with Prior-Art Methods
As a supplement of Table 1, we provide a more detailed comparison with prior-art methods in terms of different techniques in Table 6.
Table 6: Comparison with prior-art methods in terms of different techniques.
Method	Protocol Optimization	Network Optimization
Convolution	Truncation	ReLU
[43, 42, 39, 8, 38, 29, 41]	ReLU, Trunc, Conv.	X	X	X
[32, 7, 19, 40, 37, 6, 27, 26]	X	X	X	ReLU count/sensitivity-aware NAS
[48, 25]	Conv. (Online to Offline)	X	X	ReLU count-aware NAS
[17]	Conv. (Winograd-based)	Factorized Point-wise Conv.	X	X
[36]	X	Channel Reduction	Channel Reduction	Channel Reduction
CoPriv (ours)	Conv. (Winograd-based)	Re-paramerization	Re-paramerization	Communication-aware NAS
G Comparison of Convolution Protocol with HE-Based Method
We also compare CoPriv with SOTA homomorphic encryption (HE) based method, Cheetah [24], which uses HE-based protocol for convolution instead of oblivious transfer (OT) in CryptTFlow2. Cheetah achieves lower communication compared to CryptTFlow2 at the cost of more computation
overhead for both the server and client. Here, we compare their inference latency. Hence, we believe Cheetah and CryptTFlow2/CoPriv have different applicable scenarios. For example, for less performant clients, Cheetah may not be applicable, while when the bandwidth is low, CryptTFlow2/CoPriv may not be the best choice. When comparing CoPriv with Cheetah, we observe a similar trend. As shown in Table 7, we find while CoPriv always outperforms CryptTFlow2, for high bandwidth, CoPriv achieves lower latency compared to Cheetah while for low bandwidth, Cheetah incurs lower latency.
Table 7: Convolution latency (s) comparison with Cheetah for different blocks (expand ratio is set to 6) and bandwidths.
Bandwidth	384 MBps [24]	44 MBps [24]	9 MBps [47, 39]
Dimension: (4 × 4 × 64)
Baseline OT (CrypTFlow2)	2.14	3.55	16.2
HE (Cheetah)	1.35	1.54	2.99
CoPriv w/ Winograd (ours)	0.21	0.64	3.00
Dimension: (8 × 8 × 32)
Baseline OT (CrypTFlow2)	1.29	3.94	16.1
HE (Cheetah)	0.72	0.73	1.53
CoPriv w/ Winograd (ours)	0.17	0.45	2.16