SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$^{\star}$ Hongwu Peng $^{1}$ ,  ${}^{\star}$ Ran Ran $^{2}$ , Yukui Luo $^{3}$ , Jiahui Zhao $^{1}$ , Shaoyi Huang $^{1}$ , Kiran Thorat $^{1}$ , Tong Geng $^{4}$ , Chenghong Wang $^{5}$ , Xiaolin Xu $^{6}$ , Wujie Wen $^{2}$ , Caiwen Ding $^{1}$
<sup>1</sup>University of Connecticut, <sup>2</sup>North Carolina State University,
<sup>3</sup> University of Massachusetts Dartmouth, <sup>4</sup>University of Rochester,
$^{5}$ Indiana University Bloomington,  $^{6}$ Northeastern University
{hongwu.peng, jiahui.zhao, shaoyi.huang, kiran_gautam.thorat}@uconn.edu,
{rran, wwen2}@ncsu.edu, yluo2@umassd.edu, tgeng@ur.ROchester.edu
cw166@iu.edu, x.xu@northeastern.edu, caiwen.ding@uconn.edu
The growth of Graph Convolution Network (GCN) model sizes has revolutionized numerous applications, surpassing human performance in areas such as personal healthcare and financial systems. The deployment of GCNs in the cloud raises privacy concerns due to potential adversarial attacks on client data. To address security concerns, Privacy-Preserving Machine Learning (PPML) using Homomorphic Encryption (HE) secures sensitive client data. However, it introduces substantial computational overhead in practical applications. To tackle those challenges, we present LinGCN, a framework designed to reduce multiplication depth and optimize the performance of HE based GCN inference. LinGCN is structured around three key elements: (1) A differentiable structural linearization algorithm, complemented by a parameterized discrete indicator function, co-trained with model weights to meet the optimization goal. This strategy promotes fine-grained node-level non-linear location selection, resulting in a model with minimized multiplication depth. (2) A compact node-wise polynomial replacement policy with a second-order trainable activation function, steered towards superior convergence by a two-level distillation approach from an all-ReLU based teacher model. (3) an enhanced HE solution that enables finer-grained operator fusion for node-wise activation functions, further reducing multiplication level consumption in HE-based inference. Our experiments on the NTU-XVIEW skeleton joint dataset reveal that LinGCN excels in latency, accuracy, and scalability for homomorphically encrypted inference, outperforming solutions such as CryptoGCN. Remarkably, LinGCN achieves a  $14.2 \times$  latency speedup relative to CryptoGCN, while preserving an inference accuracy of  $\sim 75\%$  and notably reducing multiplication depth. Additionally, LinGCN proves scalable for larger models, delivering a substantial  $85.78\%$  accuracy with 6371s latency, a  $10.47\%$  accuracy improvement over CryptoGCN. The codes are shared on Github<sup>1</sup>.
Graph learning, a deep learning subset, aids real-time decision-making and impacts diverse applications like computer vision [1], traffic forecasting [2], action recognition [3, 4], recommendation systems [5], and drug discovery [6]. However, the growth of Graph Convolution Network (GCN)
<https://github.com/harveyp123/LinGCN-Neurips23>
model sizes [7, 8] introduces challenges in integrating encryption into graph-based machine learning services. Platforms such as Pinterest's PinSAGE [9] and Alibaba's AliGraph [10] operate on extensive user/item data graphs, increasing processing time. Furthermore, deploying GCN-based services in the cloud raises privacy concerns [11, 12] due to potential adversarial attacks on client data, such as gradient inversion attacks [13, 14]. Given the sensitive information in graph embeddings, it's crucial to implement lightweight, privacy-focused strategies for cloud-based GCN services.
Privacy-Preserving Machine Learning (PPML) with Homomorphic Encryption (HE) helps protect sensitive graph embeddings, allowing client data encryption before transmission and enabling direct data processing by the cloud server. However, this security comes with substantial computational overhead from HE operations such as rotations, multiplications, and additions. For instance, using the Cheon-Kim-Kim-Song (CKKS) [15], a Levelled HE (LHE) [16] scheme, the computational overhead escalates significantly (e.g.  $\sim 3\times$ ) with the total number of successive multiplications, which is directly related to the layers and non-linear implementations in deep GCNs.
Three research trends have emerged to lessen computational overhead and inference latency for HE inference. First, leveraging the graph sparsity, as in CryptoGCN [12], reduces multiplication level consumption. Second, model quantization, seen in TAPAS [17] and DiNN [18], binarizes activation and weight to 1 bit, leading to a  $3\% -6.2\%$  accuracy loss on smaller datasets and limited scalability. Third, nonlinear operation level reduction, as in CryptoNet [19] and Lola [20], substitutes low-degree polynomial operators like square for ReLU in CNN-based models. However, these methods struggle with performance due to absent encryption context of graph features, and the latter two are not suitable for GCN-based models.
To bridge the research gap, in this paper, we propose LinGCN, an end-to-end policy-guided framework for non-linear reduction and polynomial replacement. This framework is designed to optimize deep Spatial-Temporal Graph Convolution Network (STGCN)-based models for HE-based inference. We conduct extensive experiments on the NTU-XVIEW [21] skeleton joint [22, 23, 24] dataset and compare our approach with CryptoGCN [12]. The pareto frontier comparison of accuracy-latency between LinGCN and CryptoGCN [12] demonstrates
Figure 1: Frontier of LinGCN vs. CryptoGCN [12]
that LinGCN surpasses CryptoGCN [12] by over  $5\%$  accuracy under the same latency budget, and achieves a 14.2-fold latency reduction under the same accuracy constraint ( $\sim 75\%$  accuracy).
Our proposed framework, LinGCN, is principally influenced by two observations: the need to conserve levels in the CKKS scheme and the crucial role of synchronized linearization. In the CKKS scheme, deeper networks demand a larger polynomial degree, thereby increasing the latency of homomorphically encrypted operators. Non-linear layer pruning can mitigate this by reducing level consumption. However, unstructured non-linear pruning is insufficient; structural linearization becomes essential for effective level reduction, underscoring synchronized linearization's significance in optimizing CKKS performance. We summarize our contributions as follows:
1. We present a differentiable structural linearization algorithm, paired with a parameterized discrete indicator function, which guides the structural polarization process. This methodology is co-trained with model weights until the optimization objective is attained. Our approach facilitates fine-grained node-level selection of non-linear locations, ultimately yielding a model with reduced multiplication levels.
2. We introduce a node-wise polynomial replacement policy utilizing a second-order trainable polynomial activation function. This process is further steered by an all-ReLU based teacher model employing a two-level distillation approach, thus promoting superior convergence.
3. We have engineered a corresponding HE solution that facilitates finer-grained operator fusion for node-wise activation functions. This development further mitigates multiplication level consumption during our HE-based inference process.
CKKS Homomorphic Encryption Scheme. The CKKS scheme [15], based on the ring learning with errors (RLWE) problem, allows arithmetic operations on encrypted fixed-point numbers. It provides configurable precision via encryption noise as natural error  $e$  [25]. We denote the cyclotomic polynomial degree as  $N$  and the polynomial coefficient modulus as  $Q$ , both cryptographic parameters. A ciphertext  $ct$  encrypts a message  $m$  with noise  $e$  as  $ct = m + e$  (mod  $Q$ ). The CKKS scheme's security level [26], measured in bits, suggests  $2^{128}$  operations to break a  $\lambda = 128$  encryption. CKKS supports operations including ciphertext addition  $Add(ct_1, ct_2)$ , ciphertext multiplication  $CMult(ct_1, ct_2)$ , scalar multiplication  $PMult(ct, pt)$ , rotation  $Rot(ct, k)$  and rescaling  $Rescale(ct)$ . Scalar multiplication multiplies a ciphertext with plaintext, and rotation cyclically shifts the slot vector. For example,  $Rot(ct, k)$  transforms  $(v_0, \dots, v_{N/2-1})$  into  $(v_k, \dots, v_{N/2-1}, v_0, \dots, v_{k-1})$ . The level of a ciphertext  $(L)$ , the number of successive multiplications it can undergo without bootstrapping, is reduced by the Rescale operation after each multiplication. With the level-reduced target network, we do not consider using bootstrapping [27] in this work.
STGCN for Graph Dataset with Timing Sequence. The focus of this study revolves around the STGCN, a highly esteemed category of GCNs proposed by [4]. The STGCN model primarily employs two core operators, Spatial graph convolution (GCNConv) and Temporal convolution, designed to extract spatial and temporal information from input graph data, respectively. The Spatial convolution operation, as defined in Equation 1, precisely illustrates the process of extracting spatial information. In Equation 1, the variables  $A$ ,  $D$ ,  $X_{i}$ ,  $W_{i}$ , and  $X_{i,\mathrm{out}}$  correspond to the adjacent matrix, degree matrix, input feature, weight parameter, and output feature, respectively [4].
$$
\boldsymbol {X} _ {i, \text {o u t}} = \boldsymbol {D} ^ {- \frac {1}{2}} (\boldsymbol {A} + \boldsymbol {I}) \boldsymbol {D} ^ {- \frac {1}{2}} \boldsymbol {X} _ {i} \boldsymbol {W} _ {i} \tag {1}
$$
Related Work. CryptoNets [19] pioneered Privacy-Preserving Machine Learning (PPML) via Homomorphic Encryption (HE) but suffered from extended inference latency for large models. Subsequent studies, like SHE [28] which realizes realize the TFHE-based PPML, and those employing Multi-Party Computation (MPC) solutions[29, 30, 31], reduced latency but faced high communication overhead and prediction latency. Research like TAPAS [17] and XONN [32] compressed models into binary neural network formats, while studies like CryptoNAS [30], Sphynx [33], and SafeNet [34] used Neural Architecture Search (NAS) to find optimal architectures. Delphi [29], SNL [35], and DeepReduce [36] replaced ReLU functions with low order polynomial or linear functions to accelerate MPC-based Private Inference (PI). Nevertheless, these strategies proved suboptimal for homomorphic encrypted inference. Solutions like LoLa [20], CHET [37], and HEAR [38] sought to use ciphertext packing techniques but were not optimized for GCN-based models.
Threat Model. In line with other studies [12, 37, 39], we presume a semi-honest cloud-based machine learning service provider. The well-trained STGCN model's weight, including adjacency matrices, is encoded as plaintexts in the HE-inference process. Clients encrypt data via HE and upload it to the cloud for privacy-preserving inference service. The server completes encrypted inference without data decryption or accessing the client's private key. Finally, clients decrypt the returned encrypted inference results from the cloud using their private keys.
3 LinGCN Framework
We present two pivotal observations that serve as the foundation for our LinGCN.
CKKS Based Encrypted Multiplication. As described in section 2, the message  $m$  is scaled by a factor  $\Delta = 2^p$  before encoding, resulting in  $\Delta m$ . Any multiplicative operations square the ciphertext's scale, including inherent noise  $e$ . The Rescale operation reduces this scale back to  $\Delta$  by modswitch [15], reducing  $Q$  by  $p$  bits and lowering the level, as depicted in Figure 2. When  $Q \rightarrow q_o$  after  $L$  multiplications and rescaling, the ciphertext's multiplicative level reaches 0. At this stage, bootstrapping [12] is necessary for further multiplication, unless a higher  $Q$  is used to increase the multiplicative level  $L$ .
Observation 1: Saving Level is Important in CKKS Scheme. Within a STGCN model, the graph embedding is encrypted with an initial polynomial degree, denoted as  $N$ , and a coefficient modulus,
represented by  $Q$ . These factors are determined by the network's multiplicative depth. As the multiplication depth increases, the coefficient modulus must also increase, subsequently leading to a higher polynomial degree  $N$ . To maintain a security level above a certain threshold (for example, 128-bit) in the face of deeper multiplicative depths, a larger  $N$  is necessitated [37, 38, 12]. A larger  $N$  inherently results in increased complexity in each homomorphic encryption operation such as Rot and CMult.
These operations are integral to the computation of convolution (Conv) and graph convolution (GCN-Conv). We present a comparative analysis of operator latency with differing polynomial degrees  $N$  in Figure 2. Thus, it can be inferred that pruning certain operators to conserve levels not only diminishes the latency of the pruned operators but also reduces other operators' latency.
Observation 2: Synchronized Linearization Matters. Our study mainly targets reducing multiplication depth in standard STGCN layers, comprising an initial spatial GCNConv operation, two non-linear operators, and a temporal convolution. Under the CKKS base HE scheme, each operator's input needs synchronized multiplication depths, with any misalignment adjusted to the minimal depth. This is particularly vital for the GCNConv operator, while later node-wise separable operators enable structured non-linear reduction to boost computational efficiency.
Figure 2: Top: Rescale decreases the ciphertext level. Bottom: Higher polynomial degree leads to longer HE operator's latency.
Unstructured non-linear reduction strategies, common in MPC setups like AutoReP [40], PASNet [41] SNL [35], RRNet [42], DELPHI [29], and SAFENet [43], prove suboptimal for homomorphic encrypted inference. Our example in Figure 3(b) illustrates node-wise unstructured non-linear reduction in an STGCN layer, leading to unsynchronized remaining multiplication depth budgets for nodes after the layer. This approach is ineffective in reducing levels of consumption for homomorphic encrypted inference, emphasizing the necessity for a more structured non-linear
reduction approach. To address the limitations of unstructured non-linear reduction, we propose a structured approach, illustrated in Figure 3(c), where we apply an equal level of non-linear reduction to each target node, reducing the overall multiplication depth. In contrast to CryptoGCN's layer-wise pruning method [12], our scheme provides nodes with the flexibility to perform non-linear operations at preferred positions. This fine-grained, structural non-linear reduction may improve the non-linear reduced structure, potentially augmenting the efficiency and effectiveness of HE inference.
Figure 3: Unstructured vs. structural linearization. Unstructured one doesn't lead to effective level reduction.
3.2 Learning for Structural Linearization
Problem Formulation. Our methodology is aimed at the node-wise structural dropping of non-linear operators within a given  $L$ -layered STGCN model  $f_{W}$ , parameterized by  $W = \{W_{i}\}_{i=0}^{2L}$ . This model maps the input  $X_{0} \in R^{V \times C \times T}$  to the target  $Y \in R^{d}$ . Here,  $V$ ,  $C$ , and  $T$  represent the number of nodes, channels, and frames, respectively. The primary objective of our approach is to eliminate the non-linear operator (designated as  $\sigma_{n}$ ) in a structured, node-wise manner. The ultimate aim is to achieve a reduced multiplication level while minimizing any resultant drop in model accuracy. This goal serves to optimize the efficiency of homomorphic encrypted inference for STGCN model.
We introduce an indicator parameter, denoted as  $h$ , to mark the node-wise non-linear operator dropping. The following conditions apply: (1) If  $h_{i,k} = 0$ , the non-linear operator  $\sigma_n$  is substituted by the linear function  $f(x) = x$ ; (2) If  $h_{i,k} = 1$ , the operator  $\sigma_n$  is utilized. Here,  $h_{i,k}$  signifies
the  $k_{th}$  node of the  $i_{th}$  non-linear layer. The proposed node-wise non-linear indicator parameter  $h_{i,k}$  facilitates the expression of the  $i_{th}$  non-linear layer with partial linearization, as given by  $X_{i,k} = h_{i,k} \odot \sigma_n(Z_{(i-1),k}) + (1 - h_{i,k}) \odot Z_{(i-1),k}$ . Here,  $Z_{i-1}$  denotes the input of the  $i_{th}$  non-linear layer. Consequently, the issue of structural linearization can be formulated as follows:
$$
\underset {W, h} {\operatorname {a r g m i n}} \mathcal {L} = \underset {W, h} {\operatorname {a r g m i n}} \mathcal {L} _ {a c c} \left(f _ {W} \left(X _ {0}\right), Y\right) + \mu \cdot \sum_ {i = 1} ^ {2 L} \left\| h _ {i} \right\| _ {0} \tag {2}
$$
$$
\text {s u b j e c t} \forall j, k \in [ 1, V ], \left(h _ {2 i, j} + h _ {2 i + 1, j}\right) = \left(h _ {2 i, k} + h _ {2 i + 1, k}\right)
$$
In our formulation,  $||h_r||0$  represents the  $L_0$  norm or the count of remaining non-linear operators.  $\mathcal{L}$  acc signifies cross-entropy loss, while  $\mu$  is the  $L_{0}$  norm penalty term in the linearization process. The second term of Eq. 2 poses a challenge due to its non-differentiability, stemming from zero norm regularization and additional constraints on the discrete indicator parameter  $h_i$ . Hence, this issue becomes difficult to handle using traditional gradient-based methods.
Differentiable Structural Linearization. To handle the non-differentiable discrete indicator parameter, we introduce a trainable auxiliary parameter,  $h_w$ , to denote its relative importance. However, the transformation from  $h_w$  to the final indicator  $h$  must comply with the structural constraint in Eq.2, ruling out simple threshold checks used in SNL [35]. Structural pruning methods [44, 45], useful for weight or neuron pruning, are ill-suited for our problem due to its exponential complexity arising from 2 non-linear selections of 25 nodes, and its distinct nature from weight or neuron pruning.
To tackle this challenge, we propose a structural polarization forward process, detailed in Algorithm 1. The algorithm first ranks the relative importance of two auxiliary parameters  $(h_{w(2i,j)}$  and  $h_{w(2i + 1,j)})$  for every  $j_{th}$  node within the  $i_{th}$  STGCN layer, obtaining the higher and lower values and their respective indices. We then sum each  $i_{th}$  layer's higher auxiliary parameter value into  $s_h$ , and lower auxiliary parameter value into  $s_l$ . Next, we conduct a threshold check of  $s_h$  and assign the final polarization output of the indicator value into the corresponding locations of higher value indices, applying the same polarization for lower values. The proposed structural polarization enforces the exact constraint as given in Eq. 2, where each STGCN layer maintains a synchronized non-linear count across all nodes. The structural polarization is also
Algorithm 1 Structural Polarization.
Input:  $h_w$ : auxiliary parameter
Output:  $h$ : final indicator
1: for  $i = 0$  to  $L$  do
2:  $s_h, s_l = 0$  and  $ind_h, ind_l \gets \emptyset$
3: for  $j = 1$  to  $V$  do
4: if  $h_{w(2i,j)} > h_{w(2i + 1,j)}$  then
5:  $s_h + = h_{w(2i,j)}, s_l + = h_{w(2i+1,j)}$
6:  $ind_{h} \gets (2i,j), ind_{l} \gets (2i + 1,j)$
7: else
8:  $s_h + = h_{w(2i + 1,j)}, s_l + = h_{w(2i,j)}$
9:  $ind_{h} \gets (2i + 1, j)$ ,  $ind_{l} \gets (2i, j)$
10: end if
11: end for
12:  $h_{ind_h} = s_h > 0$  and  $h_{ind_l} = s_l > 0$
13: end for
capable of capturing the relative importance of the auxiliary parameter within each node and applying the proper polarization to every position, all with a complexity of only  $O(V)$ . Each STGCN layer retains the flexibility to choose all positions to conduct the non-linear operation, or just select one position for each node to conduct the non-linear operation, or opt not to perform the non-linear operation at all. This selection is based on the importance of their auxiliary parameters.
Despite its benefits, the structural binarization in Algorithm 1 is discontinuous and non-differentiable. To approximate its gradient, we use coarse gradients [46] via straight through estimation (STE), a good fit for updating  $h_w$  and reflecting the optimization objective in Eq.2. Among various methods such as Linear STE[47], ReLU and clip ReLU STE [48], we adopt Softplus STE [49] due to its smoothness and recoverability discovered in [49], to update  $h_w$  as follows:
$$
\frac {\partial \mathcal {L}}{\partial h _ {w (i , k)}} = \frac {\partial \mathcal {L} _ {a c c}}{\partial X _ {i , k}} \left(\sigma_ {n} \left(Z _ {i - 1}\right) - Z _ {i - 1}\right) \frac {\partial h _ {i , k}}{\partial h _ {w (i , k)}} + \mu \frac {\partial h _ {i , k}}{\partial h _ {w (i , k)}}, \quad \frac {\partial h _ {i , k}}{\partial h _ {w (i , k)}} = S o f t p l u s \left(h _ {w (i, k)}\right) \tag {3}
$$
The gradient of the auxiliary parameter, as displayed in Eq. 3, consists of two components: the gradient from the accuracy constraint and the gradient from the linearization ratio constraint. The gradient stemming from the accuracy constraint will be negative if it attempts to counteract the linearization process, thereby achieving a balance with the  $L_0$  penalty term  $\mu \cdot \sum_{i=1}^{2L} ||h_i||_0$ . The
Figure 4: Overview of single layer STGCN with non-linear reduction and computation.
entirety of the linearization process is recoverable, which allows for the exploration of a near-optimal balance between accuracy and linearization ratio via the back-propagation process.
3.3 Learnable Polynomial Replacement with Teacher Guidance
CryptoNet [19] and DELPHI [29] suggest using quadratic functions  $y = x^2$  as ReLU replacements for private inference. A second-order polynomial replacement [50]  $y = x^2 + x$  for ReLU was proposed but resulted in decreased performance. CryptoGCN [12] uses a layer-wise trainable polynomial function  $(ax^2 + bx + c)$  for the STGCN model, but it suffers from significant accuracy loss and scalability problems. To tackle these issues, we offer a two-tiered strategy: node-wise trainable polynomial replacement and a distillation technique to facilitate the convergence of replacement.
Node-wise Trainable Polynomial Replacement. In our HE setup (see Figure 4), each node can independently perform non-linear, convolution, and further non-linear operations without increasing the multiplication depth. We suggest using a node-wise trainable polynomial function as the nonlinear function (see Eq. 4). To prevent gradient explosion from quadratic terms, we include a small constant  $c$  to adjust the gradient scale of the  $w_{2}$  parameter [41].
$$
\sigma_ {n} (x) = c \cdot w _ {2} x ^ {2} + w _ {1} x + b \tag {4}
$$
Improved Convergence Through Distillation. Training a polynomial neural network from scratch often overfits, leading to accuracy loss [50, 12]. To counteract this, we recommend initially training a baseline ReLU model as the teacher, then transferring its knowledge to the polynomial student model to lessen accuracy degradation. The student model's parameters are initialized from the teacher's and polynomial function parameters start at  $(w_{2} = 0, w_{1} = 1, b = 0)$ . To aid knowledge transfer, we use KL-divergence loss [51] and peer-wise normalized feature map difference penalty. The loss function for polynomial replacement is as follows:
$$
\mathcal {L} _ {p} = (1 - \eta) \mathcal {L} _ {C E} \left(f _ {W, s} \left(X _ {0}\right), Y\right) + \eta \mathcal {L} _ {K L} \left(f _ {W, s} \left(X _ {0}\right), f _ {W, t} \left(X _ {0}\right)\right) + \frac {\varphi}{2} \sum_ {i = 1} ^ {L} M S E \left(\frac {X _ {i , s}}{\left| \left| X _ {i , s} \right| \right| _ {2}}, \frac {X _ {i , t}}{\left| \left| X _ {i , t} \right| \right| _ {2}}\right) \tag {5}
$$
In the above equation,  $\eta$  is employed to balance the significance between the CE loss and the KL divergence loss components, while  $\varphi$  acts as the penalty weight for the peer-wise feature map normalized  $L_{2}$  distance term, analogous to the approach in [52].
3.4 Put it Together