SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ran Ran
North Carolina State University rran@ncsu.edu
Nuo Xu
Lehigh University  
nux219@lehigh.edu
Tao Liu
Lawrence Technological University tliu3@ltu.edu
Wei Wang
Anonym, Inc.  
wei@anonymco.com
Gang Quan
Florida International University gaquan@fiu.edu
Wujie Wen
North Carolina State University  
wwen2@ncsu.edu
The marriage of Graph Convolutional Network (GCN) and Homomorphic Encryption (HE) enables the inference of graph data on the cloud with significantly enhanced client data privacy. However, the tremendous computation and memory overhead associated with HE operations challenges the practicality of HE-based GCN inference. GCN inference involves a sequence of expensive matrix-matrix multiplications, and we observe that directly applying the state-of-the-art HE-based secure matrix-matrix multiplication solutions to accelerate HE-GCN inference is far less efficient as it does not exploit the unique aggregation mechanism of two-dimension graph node-features in GCN layer computation. As a result, in this paper, we propose a novel HE-based ciphertext packing technique, i.e., Penguin, that can take advantage of the unique computation pattern during the HE-GCN inference to significantly reduce the computation and memory overhead associated with HE operations. Specifically, Penguin employs  $(i)$  an effective two-dimension parallel packing technique for feature ciphertext with optimal graph node partitioning and graph feature interleaving, and  $(ii)$  an interleaved assembly technique that can effectively make use of blank slots to merge ciphertexts after feature reduction and thus significantly reduce costly rotation operations. We perform detailed theoretical analysis to support our arguments. In the meantime, our experimental results also show that Penguin can achieve up to  $\sim 10\times$  speedup and around  $\sim 79\%$  reduction in computational memory overhead, significantly outperforming state-of-the-art solutions. To the best of our knowledge, this is the first work that can ensure the protection of both graph structure and features when accelerating HE-GCN inference on encrypted data. Our code is publicly available at https://github.com/ranran0523/Penguin.
Graph Convolution Neural Networks (GCNs) have recently demonstrated phenomenal performance for many privacy-sensitive applications such as social networks [34], cross-domain recommendation systems [35], and personal healthcare [18]. A popular solution for clients seeking to leverage these advanced GCN models is to utilize cloud-based inference services. However, clients often hesitate to share their graph data with the public cloud due to concerns about sensitive information, such as graph structure and node features that reveal personal social relationships and medical records. To address this privacy concern, one viable approach is to adopt the Homomorphic Encryption (HE) scheme [3, 6, 7]. By performing the entire inference computation on the cloud using encrypted data,
(b)
Figure 1: (a) Latency comparison of different HE operations under same encryption parameter and hardware environment; (b) Latency breakdown of linear/nonlinear HE operations in a typical GCN layer computation. (Detailed settings in Sec. 4.1). (c) Single optimization and wasted ciphertext slots have a negative effect on memory utilization and computation latency.
the privacy of client data is significantly enhanced. This enables privacy-preserving GCN inferences while ensuring that sensitive information remains confidential.
While the idea of embedding HE into GCN inference on graph data seems appealing, it faces several significant challenges: Firstly, similar to HE-based CNN inference on non-graph data (such as for using convolutional neural networks (CNN) [10, 4, 8, 23, 2, 19, 26, 17, 14]), the enhanced privacy would come at the cost of the tremendously escalated computational overhead associated with HE operations (e.g., ciphertext (ct) rotations/multiplications, additions), which could be orders of magnitude higher than the counterparts in the non-encrypted computation [10, 29, 16]. Secondly, existing solutions focusing on alleviating computation overhead of HE-based CNN inference may not be applicable or optimal to GCNs due to computing pattern differences between the CNN and GCN [21]. For example, a GCN layer's computation is dominated by the special consecutive matrix multiplications  $(A \cdot X \cdot W)$  for 2-dimensional feature-node aggregation-feature aggregation via multiplying a high dimensional feature matrix  $X$  with weight matrix  $W$ , followed by graph node aggregation with  $A$ , while a CNN layer's computation is bottlenecked by multi-channel 2D convolutions. Thirdly, simply treating the above critical matrix operations in HE-based GCN inference as a traditional encrypted matrix-matrix multiplication (MM) problem for speedup is sub-optimal because: 1) state-of-the-art (SOTA) HE-based MM acceleration often requires the matrix to satisfy some special properties, e.g. square matrix with size  $64 \times 64$  [15], while GCN matrices like feature matrix  $X$  are typically irregular depending on applications (i.e.  $2708 \times 1433$  in Cora dataset [32]); 2) SOTA solutions focus on a one-time MM without considering the consecutive MMs' result in the next GCN layer. This leads to inefficient ciphertext space utilization and unnecessary HE operations, which further translates into prolonged HE-GCN inference, as we shall show in Sec. 4.2.
To better understand the computation cost of HE operations that dominate the HE-GCN inference latency, we profile the latency of different HE operations using one GCN layer with 32 hidden units and the Cora dataset with 2708 graph nodes and 1433 (32) input (output) features per node. All HE operations are defined in Sec. 2. For generality, we assume both feature matrix and adjacency matrix are encrypted, which is a typical case in inductive learning (e.g. dynamic graph structure in link prediction) [22]. Without loss of generality, the same indexed features from different nodes are packed as a ciphertext (feature-wise packing) and the encrypted matrices are diagonal-encoded for MMs (detailed settings in Sec. 4.1). As Figure 1 (a) shows, first, the latency of ciphertext rotation and ciphertext multiplication (CMult) can be much higher than other operations like plaintext (pt) multiplication (PMult) or Addition, e.g.  $>20 \times$  Rotation v.s. PMult. Furthermore, about  $>99\%$  latency comes from the linear operations (mainly HE rotation and CMult due to the consecutive MMs), instead of the nonlinear operations (ReLU replaced by a square function) due to feature reduction in GCN (from 1433 input features to 32 output features). Meanwhile, for linear latency, Rotation and CMult dominate the latency (e.g.  $>96\%$  of total) as the size of the adjacency matrix could be quite large (Cora:  $2708 \times 2708$ ) in the GCN problem. Last, we profile the latencies of different ciphertext packing formats under the same evaluation setup as (b) in Figure 1 (c). From the profiling result in Figure 1 (c), either the node-wise packing format (e.g. 1 ciphertext contains one node's 1433 features) or the feature-wise packing format (e.g. 1 ciphertext contains the same indexed features from 2707 nodes) could not effectively perform the HE-GCN inference. With node-feature-joint
packing format (e.g. 1 ciphertext packs 32 features and 128 nodes) by our proposed Two-Dimension Parallel-Packing (see Sec. 3.2), the ciphertext size is fully exploited, and the total HE operation count reaches a minimum, leading to significantly reduced latency and memory cost. These results indicate that the key to accelerating the HE-based GCN inference is to significantly reduce the rotation and CMult operations with a GCN-dedicated ciphertext packing format.
To this end, we propose Penguin, a novel HE ciphertext packing framework dedicated to accelerating GCN inference with the consideration of encrypting both graph structure and features simultaneously (both adjacency matrix  $A$  and input feature matrix  $X$ ). The driving vision of Penguin is: feature ciphertext packing ( $X$ ) for efficient HE-based GCN inference needs to be designed in a manner that is aware of the unique GCN computation—both the left-side graph node aggregation  $AX$  and right-side feature aggregation ( $XW$ ), instead of optimization in one direction (either  $AX$  or  $XW$ ). In this way, the whole ciphertext space can be efficiently utilized with minimized slot waste, enabling the significant reduction of ciphertext number (memory overhead) as well as the expensive HE rotation and CMulti operations under the single instruction multiple data (SIMD) architecture. Our major contributions are three-fold: 1) We propose an efficient two-dimension parallel packing technique for ciphertext via optimal graph node partition and feature interleaving. By performing the feature-level aggregation first and formulating the HE computation overhead as a constrained optimization problem, we analytically obtain the best feature-node partition that can maximize the usage of ciphertext space and minimize the costly HE operations. Experimental results are well consistent with theoretical analysis. 2) We propose an interleaved assembling (IA) technique to efficiently merge ciphertexts with blank slots incurred by feature dimension reduction in the feature aggregation stage. This extra-level optimization further significantly reduces the number of ciphertexts and associated HE operations. 3) We comprehensively evaluate our proposed Penguin for CKKS-based GCN inference using Cora-based graph node classification, Citeseer-based link prediction, and Pubmed-based link prediction. Results show that our method achieves by up to about  $10 \times$  inference speedup and  $79\%$  memory overhead reduction, significantly outperforming the state-of-the-art solutions. To the best of our knowledge, this is the first work focusing on accelerating the HE-based private graph convolutional neural network inference on encrypted graph data, of which both the sensitive graph features and graph structure are protected.
CKKS Homomorphic Encryption Scheme. Homomorphic Encryption (HE) allows computations on encrypted data. HE has different categories according to the different computation types they support. The Leveled HE (LHE) schemes support a limited number of additions or multiplications while Fully HE (FHE) allows an arbitrary number of computations using a bootstrapping procedure that can effectively refresh the ciphertext and obtain a new ciphertext that encrypts the same value but has lower noise [9]. In this work, we focus on reducing the number of bottlenecked operations in CKKS—one of the promising LHEs, without considering the costly bootstrapping.
CKKS [6] is an LHE scheme and its security is based on the hardness of ring learning with errors (RLWE) problem. CKKS allows arithmetic operations on encrypted data over fixed-point numbers with predefined precision, which makes it an ideal candidate for performing machine learning tasks where most of the computations are approximate. The supported homomorphic operations include ciphertext addition  $Add \sim (ct_1 + ct_2)$ , ciphertext multiplication  $CMult \sim (ct_1 \times ct_2)$ , plaintext multiplication  $PMult \sim (ct \times pt)$ , ciphertext Rotation  $\rho(ct, k)$ . The rotation is to apply Galois automorphisms of the cyclotomic extension to the plaintext polynomials in encrypted form resulting in a cyclic shift of the slot vector. Among these four operations, Rotation and  $CMult$  are substantially slower ( $\sim 20 \times$  slower) than ciphertext-plaintext addition and multiplication as shown in our runtime performance of CKKS in Figure 1 due to the expensive key-switching operation [23].
Graph Convolution Neural Network. To extract the hidden graph features  $H$ , the 2-dimensional feature-node aggregation of a typical GCN layer can be often abstracted as [21]:
$$
H = \sigma \left(\tilde {D} _ {j} ^ {- \frac {1}{2}} \tilde {A} _ {j} \tilde {D} _ {j} ^ {- \frac {1}{2}} X W\right) \tag {1}
$$
Where  $X \in R^{N \times F}$  is the input feature matrix.  $W_{j} \in R^{F \times F'}$  represents weight parameters to transform the input features from an input dimension  $F$  to an output dimension  $F'$  (feature level aggregation).  $\tilde{D}_{j}$ ,  $\tilde{A}_{j}$  is the adjacency matrix with self-loop. The  $XW$  term is implemented by
Figure 2: Feature-Optimized Packing Ciphertext Computation Flow.
a fully-connected (FC) layer (node level aggregation) and then multiplied with the normalized adjacency matrix  $\tilde{D}_j^{-\frac{1}{2}}\tilde{A}_j\tilde{D}_j^{-\frac{1}{2}}$ . Finally, a non-linear activation function  $\sigma$  (e.g. ReLU) is applied to get one GCN layer's output feature matrix  $H$ . Throughout this work, we refer  $A$  as the normalized adjacency matrix since normalization could be absorbed in a pre-processing step.
Threat Model. We adopt a threat model setting consistent with prior works [10, 15, 4, 8, 19, 23, 28]. A client uploads private and sensitive data to the cloud for obtaining the online machine learning model prediction results. The cloud server is semi-honest (e.g. honest but curious). To ensure data privacy, the client encrypts their own data by HE and decrypts this inference result by their private key. In this work, we focus on encrypting both graph node features  $X$  and the normalized adjacency matrix  $A$ . The clients run the decoder of GAE [22] at their end because this step does not involve trained model parameters on the cloud server.
Overview. The GCN inference  $A \cdot X \cdot W$  can be separated into the two-dimension  $(A \cdot X$  on the nodes and  $X \cdot W$  on the features) aggregation on feature matrix  $X$ . When we perform HE matrix multiplication on the encrypted feature matrix (ciphertexts), it is inevitable that we need to perform HE rotation on the same ciphertext. Unfortunately, the rotation operation not only incurs high latency but also generates a huge number of ciphertext copies that consume a large amount of memory space. In this section, we propose a holistic solution set to systematically address these issues.
In order to effectively reduce the number of ciphertexts involved in HE computation, our design is built upon the feature-wise packing since multiplying  $W$  often leads to a lower feature dimension. However, for non-densely packed ciphertexts, feature-wise packing is further subject to the data alignment issue, resulting in extra rotations. To overcome this challenge, we propose the two-dimension parallel-packing. In addition, considering that the layer-wise feature number reduction would result in many wasted slots, we further propose the interleaved assembling to efficiently merge such ciphertexts.
3.1 Motivation of Feature-Oriented Ciphertext Packing
The major inference computation in GCN can be illustrated as  $A \cdot X \cdot W$ , where  $A \in R^{N \times N}$  is the normalized adjacency matrix used for node-wise aggregation,  $X \in R^{N \times F}$  is the input feature matrix, and  $W \in R^{F \times F'}$  is the weight matrix used for feature-wise aggregation. Apparently, we can choose  $A \cdot X$  or  $X \cdot W$  as the first step, which will not change the final product. However, considering that matrix  $X$  is encrypted as ciphertexts, the order of computation will affect the efficiency since the ciphertexts with fewer dimensions will reduce the required HE operations and copies of the ciphertexts. For example, if  $F' < F$ , we first perform  $X \cdot W$  to produce an intermediate product with fewer dimensions, i.e.,  $R^{N \times F'}$ . This will reduce the computational overhead and latency in the next step  $A \cdot X$ . On the contrary, if  $F < F'$ , we first perform  $A \cdot X$ .
We explore two ciphertext packing design options that could lead to minimized computational overhead of a single-dimension aggregation (either graph node or feature). One is the feature-wise packing, where one ciphertext only packs one feature data from different nodes. The number of ciphertexts is proportional to feature number. The other is the node-wise packing, where the number of ciphertexts is equal to the number of graph nodes. However, in this case, the number of graph
Figure 3: (a) The Two-Dimension Parallel-Packing. (b) The Interleaved Assembling.
nodes (or ciphertexts) does not change during inference, this inevitably results in too many wasted empty slots in the ciphertexts and thus would yield more HE rotations. As the example in Figure 2 shows, we assume the ciphertext packing size is 8, adjacency matrix  $A \in R^{4 \times 4}$ , weight matrix  $W \in R^{4 \times 1}$ , and feature matrix  $X \in R^{4 \times 4}$ . The 4 feature-wise packing ciphertexts can be reduced to 1 ciphertext after feature aggregation, which only needs 4 ciphertext-multiplication (CMult) in the next adjacency matrix multiplication. If using the node-wise packing, we still need 16 CMult in the next step. Therefore, we choose feature-wise packing in our design.
3.2 Two-Dimension Parallel-Packing
**Optimization Problem Formulation** Following the definition in the previous section, we assume that the ciphertext has  $M$  available slots for packing data and consider the following general case:  $M > N$  and  $M > F$ . As shown in Figure 2, after the feature-wise packing, the same features of different data from feature matrix  $X$  are encoded into the same ciphertext  $X = [ct_{1} \ldots ct_{F}]$  and then multiplied with weights  $w_{ij}$  to get matrix  $X' = XW = [ct_{1}', \ldots, ct_{F'}']$  in the same feature-wise packing format (see Eq.(2)).
$$
c t _ {j} ^ {\prime} = \sum_ {i = 1} ^ {F} c t _ {i} \otimes w _ {i j}, j \in F ^ {\prime} \tag {2}
$$
For node-wise aggregation, we need to perform diagonal-encoded matrix multiplication [12] on ciphertexts  $F'$  individually. However, due to  $M > N$ , we need to generate the corresponding ciphertext copies with each data aligned by rotating each ciphertext twice and scaling with the mask vector then summing (see Eq.(3)).
$$
c t _ {i j} ^ {\prime} = \rho \left(c t _ {j} ^ {\prime}, i\right) \otimes m s _ {i} + \rho \left(c t _ {j} ^ {\prime}, - (N - i)\right) \otimes m s _ {- (N - i)}, i \geq 1 \tag {3}
$$
After that, we multiply them with the corresponding diagonal-encoded vector  $d_{i}$  of adjacency matrix  $A$  and sum up them to get the node-wise aggregation result of  $A \cdot X'$  (see Eq.(4)).
$$
c t _ {j} ^ {\prime \prime} = \sum_ {i = 0} ^ {N - 1} \left(c t _ {i j} ^ {\prime}\right) \otimes d _ {i}, c t _ {0 j} ^ {\prime} = c t _ {j} ^ {\prime} \tag {4}
$$
In this process, we can find that data alignment issue for non-densely packed ciphertexts leads to extra rotations. We propose the two-dimension parallel-packing to solve it. Our idea is to leverage the matrix partition to fully pack data in all slots and amortize the HE computation cost.
Figure 3(a) shows our basic idea. We partition  $n$  (a power two number) graph nodes into a small block to fully utilize the size of the ciphertext and encode the feature matrix  $X$ . In each ciphertext, we actually pack  $n$  nodes corresponding to  $f$  different features together in an interleaved way. For feature-wise aggregation, we adopt the baby-step algorithm [19, 15] to get the different output features with good alignment. As shown in Figure 3 (a), the number of rotations  $2(f - 1)$  used here for each ciphertext depends on the number of different features  $f = M / n$  (we assume the output feature  $F' \geq f$ ). Then, we continue to rotate each ciphertext for  $n - 1$  times and perform the diagonal encoded matrix multiplication [12] for  $A \cdot X$ . The total complexity of rotation is:
$$
(n - 1) \cdot \left(N \cdot F ^ {\prime} / M\right) + (2 (f - 1)) \cdot \left(N / n \cdot \left(F ^ {\prime} / f\right)\right) \tag {5}
$$
Since  $M = n \cdot f$ , the total complexity is further equal to
$$
\left(N \cdot F ^ {\prime} / M\right) \cdot \left(\left(n - 1\right) + 2 (f - 1)\right) \tag {6}
$$
$$
O (n + 2 f) \geq O (2 \cdot \sqrt {2 n f / 2}) = O (2 \cdot \sqrt {M}) \tag {7}
$$
Where  $n = 2f$ , the total complexity of rotation reaches a minimum. Hence minimizing the number of rotations can be modeled as an optimization problem:
$$
\operatorname {a r g m i n} _ {(f, n)} \{(n - 1) + 2 (f - 1) \} s. t. \left\{ \begin{array}{l} M = n * f \\ f = 2 ^ {k}, k \in N ^ {+} \\ M > N, M > F \end{array} \right. \tag {8}
$$
Supporting Large Graph Eq. 8 assumes that the number of graph nodes should be smaller than that of ciphertext slots ( $M > N$ ), however, for scaled graph networks, it is possible that  $M \leq N$ . For example, the PubMed [32] contains 19717 nodes, which is far more than the 4096 ciphertext slots. To address this, our method can be scalable to such cases by splitting a large graph into several sub-graphs. Assuming we use feature-wise encoding and it requires 5 pts (each with 4096 slots) to pack 1 feature. Accordingly, each feature will have 4 pts with fully packed 4096 nodes and 1 ct with partially packed 3333 nodes. To solve the problem under the constraint- $M \leq N$ , we split  $N$  as:
$$
N = x \cdot M + R \tag {9}
$$
where  $x = N \mod M$ ,  $R = N\% M$ . Eq. 9 leads to one  $R \times F$  sub-block matrix and  $x$  of  $M \times F$  sub-block matrices. For the  $R \times F$  matrix, we refer to Eq. 7 to optimize the Rotation as  $N = R < M$ . For other  $M \times F$  sub-block matrices with  $N = M$ , we change the assumption from  $N < M$  to  $N = M$ . Then, again with our proposed Two-Dimension Parallel-Packed ct, the total complexity of rotations becomes:
$$
(M \cdot F ^ {\prime} / M) \cdot ((n - 1) + 2 (f - 1)) = F ^ {\prime} \cdot (n + 2 f - 3) \tag {10}
$$
The corner case  $f = 1$  is different from that in the discussion of optimization problem formulation. Because ct here is fully packed for  $n = M$  and does not have the data alignment issue. For  $n = M, F = 1$ , the total complexity of Rotation becomes:
$$
F ^ {\prime} \cdot (M - 1) \tag {11}
$$
Except for this corner case  $n = M$ ,  $f = 1$ , the total complexity reaches a minimum when  $n = 2f$ . We compare the previously proved minimum with the corner case  $n = M$ ,  $f = 1$  here, and get the difference of rotations complexity as follows:
$$
F ^ {\prime} \cdot ((M - 1) - (2 \sqrt {M} - 3)) = F ^ {\prime} \cdot (M - 2 \sqrt {M} - 4) = F ^ {\prime} \cdot (\sqrt {M} - 2) ^ {2} > 0 \tag {12}
$$
In general, since  $M$  is set as  $\geq 2^{11}$  to guarantee security level [17, 28], the above inequality 12 always holds. Thus, when considering a  $M \times F$  matrix, the proposed Two-Dimension Parallel-Packing can still reach the minimum at  $n = 2f$ .
3.3 Interleaved Assembling