SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The comprehensive workflow of our proposed LinGCN framework is outlined in Algorithm 2. Initially, we construct the student model,  $M_S$ , utilizing the parameters of the teacher model,  $M_T$ , and subsequently initialize the indicator auxiliary parameter  $h_w$  for  $M_S$ . Following this,  $M_S$  undergoes training iterations involving updates for structural linearization. Upon
achieving convergence, the indicator function  $h$  is fixed, and the ReLU function in  $M_S$  is replaced with a polynomial function. Subsequently, we initialize  $w_{poly}$  and train the final model using a combination of CE loss and the two-level distillation loss as defined in Eq. 5.
The resulting model output will feature structurally pruned node-wise polynomial functions, rendering it primed for the final stage of homomorphic encryption-based inference.
Further Operator Fusion to Save Level. Figure 4 shows an example of the single-layer STGCN structure generated by our LinGCN framework. The STGCN utilizes both input nodes information and aggregates them to generate output node features. We adopt the same AMA format [12] for GCNCov computation, as such, we are able to fuse the plaintext weights  $c \cdot w_{2}$  of node-wise polynomial function into GCNCov layer and save one multiplication depth budget. Layer operators involve another temporal convolution layer and non-linear/linear layer, we can still fuse the plaintext weights  $c \cdot w_{2}$  from the second polynomial function into the convolution operator and save the multiplication depth budget. As such, the ReLU-reduced example shown in Figure 4 only consumes 3 multiplication level of ciphertext as op
posed to the original 4 multiplication depth budget. For the convolution operator, we adopt the same computation flow from [12, 38].
Algorithm 2 LinGCN Framework Workflow.
Input: Pretrain ReLU-based model  $M_T$ , linearization penalty  $\mu$  and optim.  $OP_L$ , polynomial replacement optim.  $OP_P$
Output: Level-reduced polynomial model
1: Copy  $M_S$  from  $M_T$
2: Initialize  $h_w$  for  $M_S$
3: for Structural linearization iterations do
4: Calculate  $\mathcal{L}$  via Eq. 2 and Algorithm 1
5: Update  $W$  and  $h_w$  through back propagation (Eq. 3) by minimizing  $\mathcal{L}$  using  $OP_L$
6: end for
7: Freeze  $h_w$  and  $h$
8: Replace ReLU in  $M_{s}$  with polynomial
9: Initialize  $w_{poly}$
10: for Polynomial replacement iterations do
11: Calculate  $\mathcal{L}_p$  via Eq. 5
12: Update  $W$  through back propagation by minimizing  $\mathcal{L}_p$  using  $OP_{P}$
13: end for
HE Parameter Setting. We performed experiments with two sets of encryption parameters: one without and one with non-linear reduction. We used a scaling factor  $\Delta = 2^{33}$  in both cases to ensure comparable accuracy. This scaling consumes 33 bits from the ciphertext modulus  $Q$  per level. Without non-linear reduction, a 3-layer STGCN network needs 14 levels, and a 6-layer network needs 27. For a security level of 128-bit, we set  $Q$  to 509 (932) and the polynomial degree  $N$  to  $2^{15}(2^{16})$  for 3 (6) layers STGCN network. In the non-linear reduction case, the total level count ranges from 14 to 9 for 3-layer, and 27 to 16 for 6-layer networks. With the reduced models, we can use smaller  $Q$  and  $N$  while maintaining 128-bit security. Specifically, we set  $Q$  to 410 to 344 and  $N$  to  $2^{14}$  for the 3-layer network (from 3 non-linear reduced), and  $Q$  to 767 to 569 and  $N$  to  $2^{15}$  for the 6-layer network (starting from 5 non-linear reduced).
Experiment Dataset. The NTU-RGB+D dataset [21], the largest available with 3D joint annotations for human action recognition, is utilized in our study. With 56,880 action clips in 60 classes, each is annotated with  $(X,Y,Z)$  coordinates of 25 joints (nodes) per subject. We chose the NTU-cross-View (NTU-XView) benchmark given its representativeness as a human skeleton joint dataset, containing 37,920 training and 18,960 evaluation clips. For a thorough evaluation, 256 frames were used from each clip, resulting in a  $2\times 3\times 256\times 25$  input tensor for two individuals each with 3 channels.
Baseline Model. Our experiment runs on 8*A100 GPU server, and uses the state-of-the-art STGCN architecture for human action recognition, which combines GCN and CNN. We tested three configurations: STGCN-3-128, STGCN-3-256, and STGCN-6-256, where the first number represents the layer count, and the second the last STGCN layer's channel count. These networks contain a stack of three or six STGCN layers, followed by a global average pooling layer and a fully-connected layer. They were trained for 80 epochs using SGD, with a mini-batch size of 64, a momentum of 0.9, weight decay of  $10^{-4}$ , and dropout 0.5. The initial learning rate (LR) was 0.1, with a decay factor of 0.1 at the 10th and 50th epochs. The baseline accuracy of models can also be found in Table 1.
LinGCN Algorithm Setting. In our LinGCN algorithm, we used the baseline model from Table 1 as the teacher model in Algorithm 2. For structural linearization, we used SGD optimizer with learning rate (LR) 0.01 for both weight and auxiliary parameters. The  $L_0$  norm penalty  $\mu$  was varied from 0.1
Table 1: All ReLU based model architecture and accuracy
Model	Layer-wise number of channel configuration	Accuracy (%)
STGCN-3-128	3-64-128-128	80.64
STGCN-3-256	3-128-256-256	82.80
STGCN-6-256	3-64-64-128-128-256-256	84.52
to 10 for desired linearized layers, over 25 epochs. Then, ReLU was replaced with a second-order polynomial in the polynomial replacement iterations. We set scaling factor  $c$  at 0.01, parameters  $\eta$  and  $\varphi$  at 0.2 and 200 respectively. This process took 90 epochs, with SGD optimizer and initial LR of 0.01, decaying by a factor of 0.1 at the 40th and 80th epochs.
Private Inference Setup. Our experiments used an AMD Ryzen Threadripper PRO 3975WX machine, single threaded. The RNS-variant of the CKKS scheme [53] was employed using Microsoft SEAL version 3.7.2 [54]. We adopted the Adjacency Matrix-Aware (AMA) data formatting from CryptoGCN [12] to pack input graph data and performed GCNConv layer inference via multiplying with sparse matrices  $A_{i}$  split from the adjacency matrix  $A$ . The temporal convolution operation was optimized using the effective convolution algorithm from [38, 12, 55]. The computation for the global average pooling layer and fully-connected layer also follows the SOTA [38, 12].
Table 2: STGCN-3-128 comparison
Model	STGCN-3-128
Methods	Non-linear layers	Test acc (%)	Latency (s)
LinGCN	6	77.55	1856.95
LinGCN	5	75.48	1663.13
LinGCN	4	76.33	1458.95
LinGCN	3	74.27	850.22
LinGCN	2	75.16	741.55
LinGCN	1	69.61	642.06
CryptoGCN	6	74.25	4273.89
CryptoGCN	5	73.12	1863.95
CryptoGCN	4	70.21	1856.36
Table 3: STGCN-3-256 comparison
Model	STGCN-3-256
Methods	Non-linear layers	Test acc (%)	Latency (s)
LinGCN	6	80.29	4632.05
LinGCN	5	79.07	4166.12
LinGCN	4	78.59	3699.49
LinGCN	3	76.41	2428.88
LinGCN	2	74.74	2143.46
LinGCN	1	71.98	1873.40
CryptoGCN	6	75.31	10580.41
CryptoGCN	5	73.78	4850.93
CryptoGCN	4	71.36	4831.93
4.2 Experiment Result and Comparison
LinGCN Outperforms CryptoGCN [12]. The proposed LinGCN algorithm demonstrates superior performance compared to CryptoGCN [12]. The STGCN-3-128 and STGCN-3-256 models, which are the basis of this comparison, share the same backbone as those evaluated in CryptoGCN [12]. Hence, we have undertaken a comprehensive cross-work comparison between the LinGCN framework and CryptoGCN [12]. The latter employs a heuristic algorithm for layer-wise activation layer pruning, which necessitates a sensitivity ranking across all layers. This approach, however, proves to be ineffective, leading to significant accuracy drops as the number of pruned activation layers increases. Comparative results between LinGCN and CryptoGCN [12] are provided in Table 2 and Table 3. In the context of LinGCN, the number of non-linear layers presented in the table represents the effective non-linear layers post-structural linearization. For the STGCN-3-128 and STGCN-3-256 models, LinGCN yielded an accuracy of  $77.55\%$  and  $80.28\%$  respectively for the baseline 6 non-linear layers model, exceeding CryptoGCN [12] by  $3.3\%$  and  $4.98\%$ . This outcome signifies the superiority of our proposed teacher-guided node-wise polynomial replacement policy over the naive layer-wise polynomial replacement provided in CryptoGCN [12].
Importantly, the LinGCN model for the 6 non-linear layers employs a more fine-grained node-wise polynomial function, which is fused into the preceding Conv or GCNConv layers. Thus, the encryption level required is lower than that of CryptoGCN [12], resulting in reduced latency. When considering non-linear layer reduction performance, LinGCN experiences only a 1.2 to  $1.7\%$  accuracy decline with a 2 effective non-linear layer reduction, whereas CryptoGCN [12] displays approximately  $4\%$  accuracy degradation when pruned for 2 non-linear layers. Remarkably, LinGCN achieves an accuracy improvement of more than  $6\%$  over CryptoGCN [12] for models with 4 nonlinear layers. Even in models with only 2 non-linear layers, LinGCN exhibits exceptional accuracy
(75.16% and 74.74%). While this accuracy aligns with the performance of 6 non-linear layer models in CryptoGCN [12], LinGCN significantly outperforms it in terms of private inference latency, providing more than a five-fold reduction compared to CryptoGCN [12].
Non-linear Layer Sensitivity. We conduct a sensitivity analysis on the STGCN-3-256 model to investigate the effect of non-linear layers. The results are displayed in Figure 5, wherein the  $i$  layers in the figure represent the remaining effective non-linear layers in the model, and every  $2i - 1$  and  $2i$  exhibit a total number of non-linear nodes summing up to an integer multiple of 25. During the automatic structural linearization process, the gradient propagation is orchestrated to
Figure 5: STGCN-3-256 structural linearization
strike a balance between two goals: (i) preserving the feature structural information at the deeper layers, and (ii) mitigating the over-smoothing effect [56] resulting from the deep linearized GCN layers. Consequently, the middle 4th layer's nonlinearity is most important within the STGCN model.
LinGCN is Scalable. Beyond the evaluation of the 3-layer model, we also apply our LinGCN to a 6-layer STGCN-6-256 model, which no prior work evaluated, maintaining the same settings to assess the trade-off between accuracy and latency. An intriguing finding is that the teacher-guided full polynomial replacement not only maintains model performance but also surpasses the accuracy of the teacher baseline by  $0.95\%$ , achieving an accuracy of  $85.47\%$ . This suggests that the STGCN model equipped with a smooth polynomial activation function may enhance model expressivity. Experiments reveal a significant redundancy in the non-linear layers of the STGCN-6-256 model, as it exhibits no
Table 4: LinGCN for STGCN-6-256 model.
Model	STGCN-6-256
Methods	Non-linear layers	Test acc (%)	Latency (s)
LinGCN	12	85.47	21171.80
LinGCN	11	86.24	19553.96
LinGCN	7	85.08	8186.35
LinGCN	5	83.64	7063.51
LinGCN	4	85.78	6371.39
LinGCN	3	84.28	5944.81
LinGCN	2	82.27	5456.12
LinGCN	1	75.93	4927.26
accuracy degradation up to the reduction of 8 non-linear layers. When the pruning extends to 10 non-linear layers, the accuracy experiences a slight dip of  $3.2\%$  compared to the non-reduction one.
Pareto Frontier of LinGCN. Figure 1 compares the Pareto frontier between LinGCN and CryptoGCN [12], showing LinGCN consistently surpassing CryptoGCN by at least  $4\%$  in accuracy across all latency ranges. As latency constraints ease (around 6000s), LinGCN's accuracy notably increases, improving by roughly  $10\%$  over CryptoGCN. Impressively, LinGCN maintains an accuracy of  $75.16\%$  while achieving a private inference latency of 742s, making it 14.2 times faster than CryptoGCN.
4.3 Abalation Studies
Non-linear Reduction & Replacement Sequence matters. Our LinGCN algorithm employs a sequence of non-linear reduction and polynomial replacement, proving more efficacious in terms of search space convergence compared to the inverse sequence of polynomial replacement followed by non-linear reduction. The reasoning behind this lies in the possibility that the polynomial replacement process may produce a model optimized for a given architecture, hence subsequent changes to the architecture, such as non-linear reduction, could lead to significant accuracy degradation. Keeping all other parameters constant and merely altering the replacement sequence, we present the accuracy evaluation results of STGCN-3-256 model in Figure 6a. As depicted in the figure, the sequence of polynomial replacement followed by non-linear reduction incurs an accuracy degradation exceeding  $2\%$  compared to the baseline sequence across all effective non-linear layer ranges.
LinGCN Outperforms Layer-wise Non-linear Reduction. A significant innovation in our LinGCN framework is the structural linearization process. This process allows nodes to determine their preferred positions for non-linearities within a single STGCN layer, as opposed to enforcing a rigorous constraint to excise the non-linearity across all nodes. This less restrictive approach yields a more detailed replacement outcome compared to the layer-wise non-linear reduction. To demonstrate the impact of this method, we kept all other parameters constant for of STGCN-3-256 model, altering
(b)
Figure 6: Ablation study for (a) Replacement sequence, (b) node-wise vs. layer-wise linearization. (c)  $\eta$ : KL divergence distillation parameter (d)  $\varphi$ : feature map distance distillation parameter.
only the structural linearization to layer-wise linearization, and subsequently evaluated the model's accuracy, as shown in Figure 6b. The results indicate that layer-wise linearization results in a  $1.5\%$  decrease in accuracy compared to structural linearization within the range of 2 to 5 effective non-linear layers. This demonstrates the effectiveness of the proposed structural linearization methodology.
Distillation Hyperparameter Study. In prior experiments, we kept  $\eta$  and  $\varphi$  at 0.2 and 200 respectively. To assess their influence during distillation, we tested  $\eta$  and  $\varphi$  across a range of values while distilling the STGCN-3-256 model with 6 non-linear layers. We test  $\eta \in [0.1, 0.2, 0.3, 0.4, 0.5]$  and  $\varphi \in [100, 200, 300, 400, 500]$ . Notably, we maintained  $\varphi = 200$  and  $\eta = 0.2$  for the respective  $\eta$  and  $\varphi$  ablations. The study, shown in Figure 6c and Figure 6d, confirms that  $\eta = 0.2$  and  $\varphi = 200$  yield optimal accuracy, and larger penalties might lower accuracy due to mismatch with the teacher model. This study thus justifies our selection of the  $\eta$  and  $\varphi$ .
LinGCN Generalize to Other Dataset. Without loss of generality, we extended our evaluation on Flickr [57] dataset, which is a representative node classification dataset widely used in GNN tasks. It consists of 89,250 nodes, 989,006 edges, and 500 feature dimensions. This dataset's task involves categorizing images based on descriptions and common online properties. For the security setting, we assume that
Table 5: LinGCN for Flickr dataset.
Num. of nonlinear layers in GCN layers	Accuracy (val/test, %)	Latency (s)
6	0.5281/0.5275	4290.93
2	0.5247/0.5266	2740.94
1	0.5269/0.5283	2525.80
node features are user-owned and the graph adjacency list is public. The Flickr dataset has a larger adjacent list but smaller feature dimension compared to the NTU-XVIEW dataset. We utilize three GCN layers with 256 hidden dimensions. Each GCN layer has similar structure as STGCN backbone architecture and has two linear and nonlinear layers. We conduct full-batch GCN training to obtain ReLU-based baseline model accuracies of  $0.5492 / 0.5521$  for validation/test dataset. We obtain the accuracy/latency tradeoff detailed in the Table 5. LinGCN framework substantially diminishes the number of effective nonlinear layers, which leads to 1.7 times speedup without much accuracy loss.
Our LinGCN optimizes HE-based GCN inference by reducing multiplication levels through a differentiable structural linearization algorithm and a compact node-wise polynomial replacement policy, both guided by a two-level distillation from an all-ReLU teacher model. Additionally, we improve HE solutions for GCN private inference, enabling finer operator fusion for node-wise activation functions. LinGCN outperforms CryptoGCN by  $5\%$  in accuracy and reducing latency by 14.2 times at  $\sim 75\%$  accuracy, LinGCN sets a new state-of-the-art performance for private STGCN model inference. In the future, we will expand the proposed algorithm on other neural networks.
This work was in part supported by the NSF Grants CNS-2348733, 2247892, 2247893, OAC-2319962, Semiconductor Research Corporation (SRC) Artificial Intelligence Hardware program, and UConn CACC center. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
A.1 HE encoding with AMA format
Prior to encoding input data into polynomials, it is necessary to map the four-dimensional tensor  $X \in R^{B \times C \times T \times J}$  to a one-dimensional vector in  $R^{N/2}$  using the AMA format, as proposed in [12]. This transformation allows for more efficient execution of STGCN forward-computation in the HE domain. Below, we present the definition of the Vec function employed to map tensor  $X$  to a vector in  $R^{N/2}$ :
$$
\begin{array}{l} V e c (X) = y _ {j} = \left(y _ {0, j}, \dots , y _ {i, j}, \dots , y _ {N / 2, j}\right) \in R ^ {N / 2} \\ s.t. y _ {i, j} = X \left(\left(i \mod T\right) \% B\right) \times \left(i \mod B \cdot T\right) \times \left(i \% T\right) \times j \tag{6} \\ j \in J \\ \end{array}
$$
Following the mapping process, the vectors  $y_{j}$  are encoded into polynomials with degree  $N$  and subsequently encrypted into ciphertext  $ct_{j}$ , as detailed in [53]. In this study, when  $N$  is set to  $2^{16}$ , all tensors, including intermediate tensors, can be encrypted and packed into 25 ciphertexts, which corresponds to the number of nodes. For cases where  $N = 2^{15}$  ( $2^{14}$ ) the number of ciphertexts is 50(100). By selecting an appropriate value for  $N$ , the encryption and packing processes can be optimized to maintain performance and efficiency.
A.2 HE Setting Details
In Table 6, we furnish comprehensive details regarding the HE inference parameters. Specifically, i-STGCN-3 denotes a 3-layer STGCN model with i effective non-linear layers, while i-STGCN-6 signifies a 6-layer STGCN model with i effective non-linear layers. In this context, N represents the polynomial degree, and Q corresponds to the coefficient modulus.
To guarantee computation precision utilizing a one-time rescale operation, we assign the scale factor  $p$  for both ciphertext and plaintext to  $2^{33}$ . This allocation results in a reduction of the current Q of ciphertext by  $p$  bits. This setup ensures that the overall performance and accuracy conform to the desired criteria while capitalizing on the security and resilience advantages conferred by HE.
Table 6: HE parameter settings in detail.
Model	Encryption Parameters	Mult Level
N	Q	p	q0
6-STGCN-3	32768	509	33	47	14
5-STGCN-3	32768	476	33	47	13
4-STGCN-3	32768	443	33	47	12
3-STGCN-3	16384	410	33	47	11
2-STGCN-3	16384	377	33	47	10
1-STGCN-3	16384	344	33	47	9
12-STGCN-6	65536	932	33	41	27
11-STGCN-6	65536	899	33	41	26
7-STGCN-6	32768	767	33	41	22
5-STGCN-6	32768	701	33	41	20
4-STGCN-6	32768	668	33	41	19
3-STGCN-6	32768	635	33	41	18
2-STGCN-6	32768	602	33	41	17
1-STGCN-6	32768	569	33	41	16
A.3 HE inference on GCNConv and Temporal-Conv Layer
Upon obtaining the AMA-packed ciphertexts  $ct_{j}$ , the adjacency matrix multiplication  $A \cdot X$  can be decomposed into a series of plaintext multiplications,  $PMult$ , in the HE domain. This decomposition accelerates HE-inference without necessitating rotations. Furthermore, the subsequent temporal convolution is performed solely on the temporal dimension  $T$ , utilizing  $1 \times 9$  kernels.
Table 7: Comparison of latency breakdown between the non-reduced model with optimized model.
Model	HE Operators latency (s)	Total Latency (s)	Speedup (×)
Rot	PMult	Add	CMult
6-STGCN-3-128	1336.25	378.25	99.65	37.45	1851.60	-
2-STGCN-3-128	392.21	266.13	68.90	14.31	741.55	2.50
6-STGCN-3-256	2641.09	1508.19	397.17	74.90	4621.36	-
2-STGCN-3-256	777.68	1062.21	274.96	28.63	2143.47	2.16
12-STGCN-6-256	18955.09	1545.09	396.23	275.39	21171.80	-
2-STGCN-6-256	4090.08	1006.79	244.19	115.05	5456.12	3.88
$$
c t _ {k} ^ {\prime} = A \cdot X = \sum_ {i = 1} ^ {m} c t _ {k} A _ {i} = \sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {J} P M u l t \left(c t _ {i _ {k}}, a _ {i _ {k} k}\right) \tag {7}
$$
The AMA-packed ciphertexts allow for natural temporal convolution by single-node ciphertext  $ct_{j}$ , facilitating independent computation. This approach results in a ReLU-reduction design through structural pruning of ReLUs. The primary constraint to consider in this context is ensuring that the level consumption of each ciphertext remains equal prior to the GCNConv layer (node aggregation).
A.4 Further Detail of Operator Fusion
During the HE-inference process, employing weight fusion conserves the multiplicative depth, consequently reducing the ciphertext level budget. For instance, batch normalization, defined by an affine transformation  $a^{\prime}x + b^{\prime}$ , and a polynomial activation function, defined by  $(ax + b)^2 + c$ , can be readily fused into the corresponding temporal convolution layer  $w x + b^{\prime \prime}$  with the function  $w(a(a^{\prime}x + b^{\prime}) + b) + b^{\prime \prime} = (w\cdot a\cdot a^{\prime})x + ab^{\prime} + wb + b^{\prime \prime}$ . As a result, three consecutive multiplications are consolidated into a single multiplication (pre-computing  $w\cdot a\cdot a^{\prime}$ ), thereby reducing the level consumption of ciphertext from 4 to 2 ( $1\times 9$  convolution, batch normalization, and polynomial activation).
Analogous to the temporal-convolutional layer, the same fusion strategies can be applied to the polynomial activation and batch normalization of the GCNConv layer. Utilizing AMA-packed ciphertexts, the node aggregation in GCNConv is translated as depicted in Equation 7, where each ciphertext carries out scalar multiplication with the plaintext of matrix elements  $a_{i_k,k}$ . Consequently, these plaintext matrix elements  $a_{i_k,k}$  are fused with the primary  $1 \times 1$  convolutional kernels to conserve the multiplicative level, reducing the total level consumption of the GCNConv layer from 4 to 2 ( $1 \times 1$  convolution, adjacency matrix multiplication, batch normalization, and polynomial activation).
A.5 Operator latency breakdown
Table 7 presents a comprehensive operator latency breakdown encompassing Rot, PMult, Add, and CMult operations. The designation i-STGCN-3-128 refers to an STGCN-3-128 model with i residual non-linear layers. As indicated in the table, the non-linear reduction contributes to a significant reduction in latency. By leveraging a smaller polynomial degree N, the overall latency experiences substantial improvement.
A.6 More Training Details and Insight
In this section, we present the training curves for the STGCN-3-256 model, which employs 6 to 1 effective second-order polynomial (non-linear) layers. Figures 7(a) through Figure 7(f) depict the training curve progression. During the training process, we utilized mixed-precision training for the polynomial model, which led to occasional instability in some iterations, as evidenced by spikes in the loss values. Nevertheless, the training process demonstrated rapid recovery following such loss spikes.
As demonstrated in training curve, a smaller number of second-order polynomial (non-linear) layers contribute to a more stable training process and facilitate smoother convergence. This finding