SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In GCN inference, the reduction in feature size may result in wasted slots in two-dimensional parallel-packing. As the example shown in Figure 3(b), we optimally encode 32 different features into one ciphertext at the beginning. After the feature extraction layer with 16 hidden units, the previous dense encoded ciphertext will have half of the slots turn to blank. These blank slots in ciphertext bring higher memory overhead, especially given that the adjacency matrix  $A$  is also encrypted, resulting in more CMult operations thus computational overhead. We propose interleaved assembling to solve this issue. Figure 3(b) shows our idea. We rotate the  $ct_2$  that contains node 5-8's features by 1 slot and then add it with  $ct_1$  that contains node 1-4's features. After that, we have a new ciphertext  $ct'$  that contains 8 nodes with 1 feature. Meanwhile, we multiply the two mask vectors with the ciphertexts for sub-square matrix  $A_{11}$  (for node 1-4) and  $A_{22}$  (for node 5-8) and get an interleaved assembled ciphertext contains matrix  $A_{11}$  and  $A_{22}$ . Then, by rotating ciphertext  $ct'$  3 times and performing element-wise multiplication with new diagonal-encoded ciphertexts of the matrix  $A_{11}$  and  $A_{22}$ , we could get the results-ciphertext  $ct_1'$  of node 1-4 with the matrix  $A_{11}$  and node 5-8 with the matrix  $A_{22}$  simultaneously. In this way, the complexity of HE operations including both rotation and ciphertext multiplication can be reduced by half. After that, we repeat the steps to perform multiplication on  $ct'$  and ciphertext that contains  $A_{21}$  and  $A_{12}$  to get the  $ct_2'$ . Based on the formula  $ct'' = ct_1' + \rho(ct_2', 1)$ , we get the final result ciphertext  $ct''$  that multiplies matrix  $(A_{11}, A_{12}, A_{21}, A_{22})$ . By leveraging such an interleaved assembling, we could achieve  $\frac{f'}{f}$  times reduction of the total computational complexity, where  $f'$  is the number of features on the current ciphertext, and  $f$  is the number of features on the ciphertext before feature reduction.
Datasets. We adopt the Cora [32], Citeseer [11] and Pubmed [32] scientific publication datasets for graph learning. The Cora, Citeseer, and Pubmed contain 2708, 3327, and 19717 publication nodes divided into 7, 6, and 3 classes respectively. And each node consists of 1433, 3703, and 500 unique word features, respectively. To test the link prediction task [22],  $90\%$  of edges are removed and all node features are retained on all datasets.
Models. We train 3 Graph Auto-Encoder (GAE) models with 2 hidden layers and 2 activation layers on 3 different datasets, i.e., Cora, CiteSeer, and Pubmed. The three models follow the same GAE architecture in [22], and are implemented using the DGL library [33]. Table 1 lists the model architecture and pertinent encryption parameters for encrypting both adjacency matrix  $A$  and feature matrix  $X$ . We use  $x^{2}$  as the non-linear function [10] to replace the ReLU activation and apply the ADAM optimizer to train the model for 200 epochs using a learning rate of 0.01. The accuracy of each model (AUC in Link Prediction) is maintained at the original level.
Encryption parameters. For all tasks, we apply a scaling factor  $\Delta = 2^{30}$  to ensure the accuracy of the encrypted inference using CKKS. Each rescale consumes 30 bits of ciphertext modulus  $Q$ , and there are 6 times rescale and corresponding 6 levels across the whole network. Thus, we set  $Q = 218$  and the polynomial degree  $N = 2^{13}$  to guarantee a 128-bit security level. Additionally, the scale factor of mask plaintext used in comparison with E2DM [15] & uSCORE [13] is set to  $2^{15}$ .
Baseline designs. To better evaluate the proposed approach, we develop several baselines, including:
- Penguin-family. We implement several Penguin baselines by applying only our proposed two-dimension parallel-packing technique (see Sec. 3.2). We set up different pairs of features and nodes when optimizing the packing format. Table 2 lists the numbers of features/nodes selected. Here  $\text{Penguin}(f, n)$  denotes that  $f$  features and  $n$  nodes are used in the corresponding baseline design. Note that, the baseline designs with  $f = 1$  or  $n = 1$  are the extreme cases when only the feature-wise or node-wise packing method is used.
- Penguin+IA. We develop two Penguin+IA baselines by further applying the proposed Interleaved Assembling (IA) technique (see Sec. 3.3) to the Penguin-family.
- We also implement a set of state-of-the-art secure matrix matrix multiplication solutions, which include Gazelle [17], HElayers [1], E2DM [15] and uSCORE [13]. To ensure a fair comparison, we do not apply the activation pruning technique for multiplicative level reduction across all implementations [28], as this technique is orthogonal to our proposed solutions.
Measurements. We use inference latency as our main performance metric, which is averaged over 20 simulations. Besides, we record the Homomorphic Operation Count (HOC), including the number of rotations (Rotation), the number of ciphertext multiplications (CMult), etc. We also calibrate the numbers of ciphertexts and memory usage. A lower number of these metrics indicates better performance.
Environment. We conduct all experiments on a machine equipped with Threadripper 3975WX CPU using the single thread setting to test the inference latency and train these GAE models with 2 Nvidia 3090 GPUs. We use Microsoft SEAL version 3.7.2 [31] to implement the RNS-variant of CKKS [5] scheme.
Table 1: Model and encryption parameters.
Dataset	# Layers	Accuracy (AUC)	Encryption Parameters	Mult Level	Security Level
Hidden1	Hidden2	Activation	N	Q	P
Cora	0.974
Citeseer	32	16	x2	0.747	8192	218	30	6	128-bit
PubMed	0.858
Table 2: Ablation study of Two-Dimension Parallel-Packing and Interleaved Assembling.
Dataset	Packing-Format	HOC	# of Ciphertexts	Memory (GB)	Latency (s)	Speedup (×)
Rot	CMult	Others
Cora	Penguin(1433,1)	1048K	74K	282K	2708-2708-2708	2.38	7018.51	-
Penguin(1, 2708)	260K	130K	223K	1433-32-16	1.82	2475.78	2.83
Penguin(16,256)	9.7K	9.3K	157k	990-22-11	0.49	678.03	10.35
Penguin(32,128)	8.3K	124K	188K	990-22-22	0.65	871.15	8.06
Penguin(64,64)	13.5k	237k	365K	990-43-43	1.25	1650.28	4.25
Penguin(32,128)+IA	6.9K	9.3K	157K	990-22-11	0.49	660.67	10.62
Penguin(64,64)+IA	10.3K	9.3K	220k	989-22-11	0.49	693.13	10.13
CiteSeer	Penguin(3703,1)	1521K	1110K	3852K	3327-3327-3327	2.92	9240.10	-
Penguin(1, 3327)	319K	160K	385K	3703-32-16	3.08	3064.91	3.01
Penguin(16,256)	110K	130K	324K	3016-26-13	1.40	950.30	9.72
Penguin(32,128)	9.8K	173K	367K	3016-26-26	1.62	1225.05	7.54
Penguin(64,64)	16.3K	346K	734K	3016-52-52	2.51	2429.47	3.80
Penguin(32,128)+IA	7.4K	130K	324K	3016-26-13	1.39	928.10	9.96
Penguin(64,64)+IA	12K	130K	387K	3016-26-13	1.39	982.08	9.41
PubMed	Penguin(19717,1)	5974K	817K	12687K	19717-19717-19717	9.75	44586.03	-
Penguin(1, 500)	1106K	4732K	4897K	2500-160-80	17.3	37727.58	1.18
Penguin(16,256)	69K	4673K	4837K	2496-156-78	3.76	30906.28663	1.44
Penguin(32,128)	59K	6151K	6314K	2480-155-155	11.6	40474.11	1.10
Penguin(64,64)	117K	12222K	12547K	2472-309-309	42.9	80424.70	0.55
Penguin(32,128)+IA	49K	4633K	4794K	2480-155-78	3.76	30522.43	1.46
Penguin(64,64)+IA	73K	4633K	4954K	2472-155-78	3.75	30701.59	1.45
4.2.1 Two-Dimension Parallel-Packing
Table 2 presents our evaluation results of the proposed two-dimension parallel-packing and interleaved assembling approach. We find that the packing format  $Penguin(f, n = 1)$  performs the worst on the three datasets due to having the largest number of HOCs and no slot packing optimization. This results in significant latency and memory overhead. In particular, since PubMed contains more encrypted features (number of cts), the same design performs worse on PubMed than on the other two datasets. We use this  $Penguin(f, n = 1)$  as the baseline to compare the speed of other approaches.
Our results clearly show that our proposed two-dimension parallel-packing method can significantly reduce the HOCs (especially the number of rotations) and the number of ciphertexts. For example, on the Cora dataset, our Penguin(16, 256), Penguin(32, 128), and Penguin(64, 64) designs can reduce the number of rotations from  $1048\mathrm{K}$  to  $9.7\mathrm{K}$ ,  $8.3\mathrm{K}$ , and  $13.5\mathrm{K}$ , respectively, thus reducing memory usage by  $\sim 79\%$ ,  $\sim 76\%$ , and  $\sim 47\%$ , and reaching  $\sim 10.35\times$ ,  $\sim 8.06\times$ , and  $\sim 4.25\times$  speed up, respectively.
In particular, the results we observed are well consistent with the theoretical analysis. For example, with  $M = f * n = 4096$ ,  $n = 2f$ , we have the theoretical minimum  $f_{\min} = \sqrt{2048} \simeq 45$  (see Section 3.2). We can observe that the baseline Penguin(32, 128) with  $f = 32$ ,  $n = 128$  is very close to the theoretical minimum and achieves the best results among the three designs. Meanwhile, other HOCs besides Rotation may increase under the optimal packing and affect the overall latency. For example, Penguin(32, 128) yields more ciphertext multiplication (CMult) than Penguin(16, 256) due to wasted slots from feature reduction, which can be further optimized using the proposed Interleaved Assembling method.
Moreover, as our discussion of large graph in Section 3.2 indicates, all designs exhibit significantly worse performance in the PubMed dataset compared to the other two datasets. This is because the number of nodes in PubMed is significantly larger than the size of ciphertext (19717  $\gg$  4096), which means that it needs to be multiplied with a large  $19717 \times 19717$  adjacency matrix. Therefore, the number of CMult  $\gg$  number of Rot. However, our proposed packing technique can still improve the performance in such cases.
4.2.2 Interleaved Assembling
Table 2 also reports the evaluation results of incorporating the two-dimensional parallel packing and interleaved assembly methods. For example, in Cora, the number of rotations, the number of CMult, and the number of other HOCs in the Penguin(32, 128) + IA design are further reduced by 1.4K, 114.7K, and 31K, respectively, compared to the parallel-packing only Penguin(32, 128). This makes Penguin(32, 128) + IA the best design on all datasets, i.e., with the minimum memory
Table 3: Compare with the state-of-the-art.
Dataset	Method	Security Level	Latency (s)	Amortized Latency	Speedup (×)
Cora	Gazelle [17]	128-bit	3832.36	1.42	-
E2DM(64) [15]	98-bit	3150.74	1.16	1.22
HElayers [1]	128-bit	2102.47	0.78	1.82
uSCORE(32,128) [13]	98-bit	1727.12	0.64	2.22
Penguin(32,128)+IA	128-bit	660.57	0.24	5.92
Citeseer	Gazelle [17]	128-bit	4727.94	1.42	-
E2DM(64) [15]	98-bit	4561.15	1.37	1.04
HElayers [1]	128-bit	3044.58	0.92	1.54
uSCORE(32,128) [13]	98-bit	2377.50	0.72	1.97
Penguin(32,128)+IA	128-bit	928.10	0.28	5.07
Pubmed	Gazelle [17]	128-bit	158655.54	8.05	-
E2DM(64) [15]	98-bit	154530.49	7.84	1.03
HElayers [1]	128-bit	103283.56	5.24	1.54
uSCORE(32,128) [13]	98-bit	78843.49	4.00	2.01
Penguin(32,128)+IA	128-bit	30522.43	1.55	5.19
usage of 0.49GB, 1.39GB, and 3.76GB and a  $10.62 \times$ ,  $9.96 \times$ , and  $1.46 \times$  speedup on dataset Cora, Citeseer, and PubMed, respectively. These results illustrate that our proposed interleaved assembly can effectively reduce the wasted empty slots and save the number of ciphertexts selected in the computation, thus significantly improving the efficiency based on the SIMD.
4.2.3 Compare with State-of-the-art Solutions
In our evaluation, we conduct a comparative analysis of our best-designed model,  $Penguin(32,128)+IA$  with several state-of-the-art (SOTA) solutions, including Gazelle [17], HElayers [1], E2DM [15] and uSCORE [13]. All these SOTA solutions can speed up HE-GCN inference with the optimized matrix-matrix multiplication. The results are summarized in Table 3.
It is worth noting that our approach uses encryption parameters which offer a 128-bit security level. This security level is higher than that of the other two SOTA solutions, which can only guarantee a 98-bit security level and require more multiplicative levels to mask the plaintexts. For a fair comparison, we measure the amortized latency—the time required for link predictions of a single node. As reported in Table 3, our method achieves an amortized latency of 0.24s on Cora, which is  $5.92 \times$ $4.83 \times$ ,  $3.25 \times$  and  $2.67 \times$  faster than that of Gazelle, E2DM, HElayers and uSCORE, respectively. The similar improvement can be observed across the CiteSeer and PubMed.
The superiority of our method over other SOTA solutions can be attributed to two key factors:
The first reason is that all prior approaches are primarily designed to solve the single matrix-matrix multiplication problem instead of two-way matrix multiplications in GCN inference. In GCN inference, the computation pattern mainly consists of two computational components—the common FC layer for graph feature aggregation  $(X \cdot W)$  and the adjacency matrix multiplication for graph node aggregation  $(A \cdot X)$ . These two components represent distinct and orthogonal computation patterns for the input feature map. Consequently, optimizing only one of these directions leads to sub-optimal performance. In contrast, our proposed "Penguin", which incorporates two-way parallel encoding and an optimized sub-block matrix size, offers superior performance with theoretical guarantees.
The second compelling factor contributing to the excellence of our approach is the adaptability to address the issue of wasted slots of ciphertext, a capability notably absent in prior methods. In studies such as [15, 1, 13], traditional matrix-multiplication methods establish sub-block matrix partitions based on square or rectangular sub-block matrices determined by the input weight matrix size. However, these conventional methods face challenges in effectively managing the problem of wasted slots that arise after feature aggregation in Graph Convolutional Network (GCN) inference. As confirmed in Table 3, our two-way parallel packing method substantially improves HE-GCN inference by leveraging the Interleaved Assembling technique that can effectively minimize the wasted slots. This, in turn, leads to extra memory space and HOC reduction.
Putting all these together, our method clearly demonstrates its advantage over SOTA solutions in accelerating HE-GCN inference.
CryptoNets [10] is the first work that demonstrates the feasibility of building privacy-preserving machine learning (PPML) by HE. However, the long inference latency and the inflexible packing format make it hard to be applied to large-scale models and datasets. Another following work named SHE [25], translates the nonlinear ReLU and Max Pooling operations as Boolean operations to support the TFHE-based [7] PPML without modifying the pretrained models. There also exist many multi-party computation (MPC) solutions that combine the two-party computation protocols [36] with HE frameworks to achieve the low inference latency [30, 17, 26, 24, 27]. However, they suffer from high communication overhead incurred by data transfer between multiple parties. Recent studies such as LoLa [4], CHET [8], and HEAR [20] leverage the ciphertext packing technique to place multiple data in the same ciphertext so that HE operations can be conducted efficiently via single instruction multiple data (SMD) for accelerating HE-based CNN inference. These approaches are often not applicable or optimal to GCN inference due to the very different computation patterns between the GCN and CNN. CryptoGCN [28] is the first attempt to build HE-based PPML for GCNs. It packs the ciphertexts from individual node to relieve the adjacency matrix multiplication overhead. However, they assume the adjacency matrix as plaintext, which is not applicable to dynamic graph settings which require protecting both graph structure and features like our work.
Cheetah [14] aims to eliminate the need of costly rotation in HE-matrix multiplication. The basic idea is to replace the dot-sum operation with two polynomials multiplications. However, it requires communication between the client and server to perform ciphertext decryption and re-encryption after each layer computation, e.g. after a convolutional layer. In contrast, our approach operates within a "HE without-client-aid" setting. Here, the server only requires the client to encrypt and send data to the server once, after which the server can perform the whole computation of inference and only send the final encrypted result back to the client. It does not require the frequent communication between client and server like [14]. Therefore, our work focuses on HE computation optimization. In our view, our solution and [14]'s MPC+HE setting represent two distinct directions to realize PPML. Each approach is tailored for different private inference scenarios. Hence, conducting a direct comparison between our method and the approach in [14] could be challenging since they are designed for different purposes and provide distinct advantages.
Gazelle [17] is a hybrid matrix multiplication algorithm based on mixture of naive implementation and diagonal encoding method [12]. It is designed to handle the general matrix multiplication that occurs in FC layers, where the output dimension is smaller than the input dimension. Additionally, E2DM [15] and uSCORE [13] address encrypted matrix-matrix multiplication optimization by breaking down the problem into small square matrix multiplications and facilitating consecutive matrix multiplications. HElayers [1], on the other hand, leverages a tiles-tensor-based matrix-multiplication technique for multiplying two square matrices and consumes less multiplicative depth compared to [15]. While these general solutions have proven effective in various contexts, their efficiency is limited when applied to accelerate Homomorphic Encryption-based Graph Convolutional Network (HE-GCN) inference, as detailed in Section 4.2.
In this paper, we propose a two-dimension parallel packing technique for feature ciphertext by optimizing the feature matrix partition size and further propose an interleaved assembling technique to merge ciphertexts that have wasted slots from feature reduction in CKKS-based secure GCN inference. These techniques can better save ciphertext memory and effectively reduce the number of homomorphic operations required. Experimental results based on the GAEs for link prediction and 3 popular graph datasets show that our solution can speed up the latency of the secure GCN inference by  $10\times$  and reduce the memory requirement by more than  $79\%$  , greatly outperforming the state-of-the-art solutions.
We thank all anonymous reviewers for their constructive comments and suggestions on this work. This work is partially supported by the National Science Foundation (NSF) under Grants No. CNS-2348733 and CNS- 2349538.
[34] Le Wu, Peijie Sun, Richang Hong, Yanjie Fu, Xiting Wang, and Meng Wang. Socialgen: An efficient graph convolutional network based model for social recommendation. arXiv preprint arXiv:1811.02815, 2018.
[35] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys (CSUR), 2020.
[36] Andrew Chi-Chih Yao. How to generate and exchange secrets. In 27th Annual Symposium on Foundations of Computer Science (sfcs 1986), pages 162-167. IEEE, 1986.