SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The key steps of the process can be summarized as follows: (i) all clients agree on and initialize a HE keypair, (ii) each client encrypts the local neighbor feature array and sends it to the server, and (iii) upon receiving all encrypted neighbor feature arrays from clients, the server performs secure neighbor feature aggregation  $\left[\left[\sum_{j\in \mathcal{N}_i}\boldsymbol {A}_{ij}\boldsymbol {x}_j\right]\right] = \sum_{k = 1}^{K}\left[\left[\sum_{j\in \mathcal{N}_i}\mathbb{I}_k(c(j))\cdot \boldsymbol {A}_{ij}\boldsymbol {x}_j\right]\right]$ , where  $[\cdot ,\cdot ]$  represents the encryption function. The server then distributes the aggregated neighbor feature array to each client, and (iv) upon receiving the aggregated neighbor feature array, each client decrypts it and moves on to the model training phase. We can also use HE for a secure model gradient aggregation function during the model's training rounds, which provides extra privacy guarantees.
Since model parameters are often floating point numbers and node features can be binary (e.g., one-hot indicators), a naive HE scheme would use CKKS (Cheon et al., 2017) for parameters and integer schemes such as BGV (Brakerski et al., 2014) for features. To avoid the resulting separate cryptographic setups, we adopt CKKS with a rounding procedure for integers and also propose an efficient HE file optimization, Boolean Packing, that packs arrays of boolean values into integers and optimizes the cryptographic communication overhead. The encrypted features then only require twice the communication cost of the raw data, compared to  $20\mathrm{x}$  overhead with general encryption.
Training Algorithm. Based on the insights in the previous section, we introduce the FedGCN training algorithm (details are provided in Appendix A):
- Pretraining Communication Round The algorithm requires communication between clients and the central server at the initial communication round.
1. Each client  $k$  sends its encrypted accumulations of local node features,  $[[\{\sum_{j \in \mathcal{N}_i} \mathbb{I}_k(c(j)) \cdot A_{ij} x_j\}_{i \in \mathcal{V}_k}],$  to the server.
2. The server then accumulates the neighbor features for each node  $i$ ,  $\mathbb{[}\sum_{j\in \mathcal{N}_i}\pmb {A}_{ij}\pmb {x}_j\mathbb{]} = \sum_{k = 1}^{K}\mathbb{[}\sum_{j\in \mathcal{N}_i}\mathbb{I}_k(c(j))\cdot \pmb {A}_{ij}\pmb {x}_j\mathbb{]}$ .
3. Each client receives and decrypts the feature aggregation of its one-hop,  $\mathbb{[}\{\sum_{j\in \mathcal{N}_i}\pmb {A}_{ij}\pmb {x}_j\}_{i\in \nu_k}\mathbb{]}$ , and if needed two-hop, neighbors  $\mathbb{[}\{\sum_{j\in \mathcal{N}_i}\pmb {A}_{ij}\pmb {x}_j\}_{i\in \mathcal{N}_{\nu_k}}\mathbb{]}$ .
- Federated Aggregation After communication, FedGCN uses the standard FedAvg algorithm McMahan et al. (2017) to train the models. Other federated learning methods, e.g., as proposed by Reddi et al. (2020); Fallah et al. (2020), can easily replace this procedure.
5 FedGCN Convergence and Communication Analysis
In this section, we theoretically analyze the convergence rate and communication cost of FedGCN for i.i.d. and non-i.i.d. data with 0-, 1-, and 2-hop communication.
We first give a formal definition of the i.i.d. and non-i.i.d. data distributions, using distribution-based label imbalance (Hsu et al., 2019; Li et al., 2022). Figure 3 visualizes eight example data distributions across clients. For simplicity, we assume the number of clients  $K$  exceeds the number of node label classes  $C$ , though Section 6's experiments support any number of clients. We also assume that each class contains the same number of nodes and that each client has the same number of nodes.
Definition 5.1. Each client  $k$ 's label distribution is defined as  $[p_1 \quad p_2 \quad \ldots \quad p_C] \in \mathbb{R}^C$ , where  $p_c$  denotes the fraction of nodes of class  $c$  at client  $k$  and  $\sum_{c} p_c = 1$ .
Definition 5.2. Clients' data distributions are i.i.d. when nodes are uniformly and randomly assigned to clients, i.e., each client's label distribution is  $[1 / C\ldots 1 / C]$ . Otherwise, they are non i.i.d.
Non-i.i.d. distributions include that of McMahan et al. (2017) in which  $p_i = 1 - p + \frac{p}{C}$  for some  $i = 1, 2, \dots, C$  and  $\frac{p}{C}$  otherwise, where  $p \in [0, 1]$  is a control parameter; or the Dirichlet distribution (Hsu et al., 2019)  $Dir(\beta / C)$ , where  $\beta \geq 0$  is a control parameter. With these distributions, each client has one dominant class. If  $p = 0$  or  $\beta \to \infty$ , all nodes at a client have the same class.
Figure 3: Example client data distributions. Different colors represent different node label classes. Client data is generated from the data distribution with different parameters  $p$  and  $\beta$  respectively. We use  $\| \pmb{x} \|$  to denote the  $\ell_2$  norm if  $\pmb{x}$  is a vector, and the Frobenius norm if  $\pmb{x}$  is a matrix. Given model parameters  $\pmb{w}$  and  $K$  clients, we define the local loss function  $f_k(\pmb{w})$  for each client  $k$ , and the global loss function  $f(\pmb{w}) = \sum_{k=1}^{K} f_k(\pmb{w})$ , which has minimum value  $f^*$ .
Assumption 5.3. ( $\lambda$ -Lipschitz Continuous Gradient) There exists a constant  $\lambda > 0$ , such that  $\| \nabla f_k(\boldsymbol{w}) - \nabla f_k(\boldsymbol{v}) \| \leq \lambda \| \boldsymbol{w} - \boldsymbol{v} \|$ ,  $\forall \boldsymbol{w}, \boldsymbol{v} \in \mathbb{R}^d$ , and  $k \in [K]$ .
Assumption 5.4. (Bounded Global Variability) There exists a constant  $\sigma_G \geq 0$ , such that the global variability of the local gradients of the cost function  $\| \nabla f_k(\boldsymbol{w}_t) - \nabla f(\boldsymbol{w}_t) \| \leq \sigma_G, \forall k \in [K], \forall t^3$
Assumption 5.5. (Bounded Gradient) There exists a constant  $G \geq 0$ , such that the local gradients of the cost function  $\| \nabla f_k(\boldsymbol{w}_t) \| \leq G, \forall k \in [K], \forall t$ .
Assumptions 5.3, 5.4 and 5.5 are standard in the federated learning literature (Li et al., 2019a; Yu et al., 2019; Yang et al., 2021). We consider a two-layer GCN, though our analysis can be extended to more layers. We work from Yu et al. (2019)'s convergence result for  $\mathrm{FedAvg}^4$  to find:
Theorem 5.6. (Convergence Rate for FedGCN) Under the above assumptions, while training with  $K$  clients,  $T$  global training rounds,  $E$  local updates per round, and a learning rate  $\eta \leq \frac{1}{\lambda}$ , we have
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \| \nabla f (\boldsymbol {w} _ {t - 1}) \| ^ {2} \right] \leq \frac {2}{\eta T} (f (\boldsymbol {w} _ {0}) - f ^ {*}) + \frac {\lambda}{K} \eta \| I _ {l o c a l} - I _ {g l o b} \| ^ {2} + 4 \eta^ {2} E ^ {2} G ^ {2} \lambda^ {2}, \tag {4}
$$
where  $f^{*}$  is the minimum value of  $f$ ,  $I_{local} = K\pmb{X}_k^T\pmb{A}_k^T\pmb{A}_k^T\pmb{A}_k\pmb{A}_k\pmb{X}_k$ ,  $I_{glob} = \pmb{X}^T\pmb{A}^T\pmb{A}^T\pmb{A}\pmb{A}\pmb{X}$ .
The convergence rate is thus bounded by the difference of the information provided by local and global graphs  $\| I_{local} - I_{glob}\|$ , which upper bounds the global variability  $\| \nabla f_k(\pmb{w}) - \nabla f(\pmb{w})\|$ . By 1- and 2-hop communication, the local graph  $A_{k}$  is closer to  $\pmb{A}$ , resulting in faster convergence.
Table 1 bounds  $\left\| I_{local} - I_{glob}\right\|$  for FedGCN trained on an SBM (stochastic block model) graph, in which we assume  $N$  nodes with  $C$  classes. Nodes in the same (different) class have an edge between them with probability  $\alpha (\mu \alpha)$ . Appendix D details the full SBM. Appendix H derives Table 1's
${}^{3}$  Clients with i.i.d. label distributions may still have global variability  ${\sigma }_{G} > 0$  due to having finite samples.
Our analysis applies to any federated training algorithm with bounded global variability (Yang et al., 2021).
Non-i.i.d.	i.i.d.
0-hop	(1 - 1/K4)N5/C5||B4|| + (1 - 1/C)5/2(1 - p)5	(1 - 1/K4)N5/C5||B4||
1-hop	(1 - 1/K4(1 + cαp + cμ)2)N5/C5||B4|| + (1 - 1/C)5/2(1 - p)5	(1 - 1/K4(1 + cα + cμ)2)N5/C5||B4||
2-hop	(1 - 1/K4(1 + cαp + cμ)6)N5/C5||B4|| + (1 - 1/C)5/2(1 - p)5	(1 - 1/K4(1 + cα + cμ)6)N5/C5||B4||
Table 1: Convergence rate bounds of FedGCN with the Stochastic Block Model and data distribution from McMahan et al. (2017) We define  $c_{\alpha} = \frac{(1 - \mu)\alpha N(K - 1)}{CK}$  and  $c_{\mu} = \frac{\mu\alpha N(K - 1)}{K}$  for the SBM;  $\alpha N$  is a constant,  $c_{\alpha} \gg c_{\mu}$  and  $c_{\mu} \simeq 0$ . More hops speed up the convergence from the order of 2 to 6 (highlighted as purple and green). Communication helps more when data is more i.i.d with factor  $c_{\alpha}p$  (highlighted as blue). Non-i.i.d. data implies a longer convergence time with factor  $(1 - p)^5$ .
results; the SBM structure allows us to develop intuitions about FedGCN's convergence with different hops, simply by knowing the node features and graph adjacency matrix (i.e., without knowing the model). Appendix F derives corresponding expressions for the required communication costs. We validate these results in experiments and Appendix G, and make the following key observations:
- Faster convergence with more communication hops: 1-hop and 2-hop communication accelerate the convergence with factors  $(1 + c_{\alpha}p + c_{\mu})^{2}$  and  $(1 + c_{\alpha}p + c_{\mu})^{6}$ , respectively. When the i.i.d control parameter  $p$  increases, the difference among no (0-hop), 1-hop, and 2-hop communication increases: communication helps more when data is more i.i.d.
- More hops are needed for cross-device FL: When the number of clients  $K$  increases, as in cross-device federated learning (FL), the convergence takes longer by a factor  $\frac{1}{K^4}$ , but 2-hop communication can recover more edges in other clients to speed up the training.
- One-hop is sufficient for cross-silo FL: If the number of clients  $K$  is small, approximation methods via one-hop communication can balance the convergence rate and communication overhead. We experimentally validate this intuition in the next section.
6 Experimental Validation
We experimentally show that FedGCN converges to a more accurate model with less communication compared to previously proposed algorithms. We further validate Section 5's theoretical observations.
We use the Cora (2708 nodes, 5429 edges), Citeseer (3327 nodes, 4732 edges), Ogbn-ArXiv (169343 nodes, 1166243 edges), and Ogbn-Products (2449029 nodes, 61859140 edges) datasets to predict document labels and Amazon product categories (Wang et al., 2020b; Hu et al., 2020).
Methods Compared. Centralized GCN assumes a single client has access to the entire graph. Distributed GCN (Scardapane et al., 2020) trains GCN in distributed clients, which requires communicating node features and hidden states of each layer. FedGCN (0-hop) (Section 4) is equivalent to federated training without communication (FedGraphnn) (Wang et al., 2020a; Zheng et al., 2021; He et al., 2021a). BDS-GCN (Wan et al., 2022) randomly samples cross-client edges in each global training round, while FedSage+ (Zhang et al., 2021) recovers missing neighbors by learning a linear predictor based on the node embedding, using cross-client information in each training round. It is thus an approximation of FedGCN (1-hop), which communicates the 1-hop neighbors' information across clients. FedGCN (2-hop) communicates the two-hop neighbors' information across clients.
We consider the Dirichlet label distribution with parameter  $\beta$ , as shown in Figure 3. For Cora and Citeseer, we use a 2-layer GCN with Kipf and Welling (2016)'s hyper-parameters. For Ogbn-Arxiv and Ogbn-Products, we respectively use a 3-layer GCN and a 2-layer GraphSage with Hu et al. (2020)'s hyper-parameters. We average our results over 10 experiment runs. Detailed experimental setups and extended results, including an evaluation of the HE overhead, are in Appendix E and G.
6.2 Effect of Cross-Client Communication
We first evaluate our methods under i.i.d.  $(\beta = 10000)$  and non-i.i.d.  $(\beta = 100,1)$  Dirichlet data distributions on the four datasets to illustrate FedGCN's performance relative to the centralized, BDS-GCN, and FedSage+ baselines under different levels of communication.
Cora, 10 clients	CiteSeer, 10 clients
Centralized GCN	0.8069±0.0065	0.6914±0.0051
β = 1	β = 100	β = 10000	β = 1	β = 100	β = 10000
FedGCN(0-hop)	0.6502±0.0127	0.5958±0.0176	0.5992±0.0226	0.617±0.0118	0.5841±0.0168	0.5841±0.0138
BDS-GCN	0.7598±0.0143	0.7467±0.0117	0.7479±0.018	0.6709±0.0184	0.6596±0.0128	0.6582±0.01
FedSage+	0.8026±0.0054	0.7942±0.0075	0.796±0.0075	0.6977±0.0097	0.6856±0.0121	0.688±0.0086
FedGCN(1-hop)	0.81±0.0066	0.8009±0.007	0.8009±0.0077	0.7006±0.0071	0.6891±0.0067	0.693±0.0069
FedGCN(2-hop)	0.8064±0.0043	0.8084±0.0051	0.8087±0.0061	0.6933±0.0067	0.6953±0.0069	0.6948±0.0032
Ogbn-Arxiv, 10 clients	Ogbn-Products, 5 clients
Centralized GCN	0.7±0.0082	0.7058±0.0008
β = 1	β = 100	β = 10000	β = 1	β = 100	β = 10000
FedGCN(0-hop)	0.5981±0.0094	0.5809±0.0017	0.5804±0.0015	0.6789±0.0031	0.658±0.0008	0.658±0.0008
BDS-GCN	0.6769±0.0086	0.6689±0.0024	0.6688±0.0015	0.6996±0.0019	0.6952±0.0012	0.6952±0.0009
FedSage+	0.7053±0.0073	0.6921±0.0014	0.6918±0.0024	0.7044±0.0017	0.7047±0.0009	0.7051±0.0006
FedGCN(1-hop)	0.7101±0.0078	0.6989±0.0038	0.7004±0.0031	0.7049±0.0016	0.7057±0.0003	0.7057±0.0004
FedGCN(2-hop)	0.712±0.0075	0.6972±0.0075	0.7017±0.0081	0.7053±0.002	0.7057±0.0009	0.7055±0.0006
Table 2: Test Accuracy on four datasets, for i.i.d.  $(\beta = 10000)$  and non-i.i.d.  $(\beta = 100,1)$  data. FedGCN (1-hop,2-hop) performs best on i.i.d. and non-i.i.d. data, and FedGCN (0-hop) has the most information loss. We assume FedSage  $^+$  's linear approximator perfectly learns neighbor information.
As shown in Table 2, FedGCN(1-, 2-hop) has higher test accuracy than FedSage+ and BDS-GCN, reaching the same test accuracy as centralized training in all settings. FedGCN(1-, 2-hop) is able to converge faster to reach the same accuracy with the optimal number of hops depending on the data distribution. In such cross-silo setting (10 clients), FedGCN(1-hop) achieves a good tradeoff between communication and model accuracy. FedGCN (0-hop) performs worst in the i.i.d. and non-i.i.d. settings, due to information loss from cross-client edges.
Why Non-i.i.d Has Better Accuracy in 0-hop? In Table 2 with 10 clients, 0-hop has better accuracy since non-i.i.d. has fewer cross-client edges (less information loss) than i.i.d as in theoretical analysis.
Figure 4: Test accuracy vs. communication cost until convergence of algorithms in the i.i.d., partial-i.i.d. and non-i.i.d. settings for the OGBN-ArXiv dataset. FedGCN uses orders of magnitude less communication (at least  $100 \times$ ) than BDS-GCN and FedSage+, while achieving higher test accuracy.
Communication Cost vs. Accuracy. Figure 4 shows the communication cost and test accuracy of different methods on the OGBN-ArXiv dataset. FedGCN (0-, 1-, and 2-hop) requires little communication with high accuracy, while Distributed GCN, BDS-GCN and FedSage+ require communication at every round, incurring over  $100 \times$  the communication cost of any of FedGCN's variants. FedGCN (0-hop) requires much less communication than 1- and 2-hop FedGCN, but has lower accuracy due to information loss, displaying a convergence-communication tradeoff. Both 1- and 2-hop FedGCN achieve similar accuracy as centralized GCN, indicating that a 1-hop approximation of cross-client edges is sufficient in practice to achieve an accurate model.
Figure 5: Test accuracy with the number of clients on the Cora dataset.
Cross-Silo and Cross-Device Training. As shown in Figure 5 (left), 1-hop and 2-hop communication achieve similar test accuracy in the cross-silo setting with few clients, though 1-hop communication has lower communication costs. However, in the cross-device setting with many clients (Figure 5 right), the 1-hop test accuracy drops with more clients, indicating that 2-hop communication may be necessary to maintain high model accuracy, as suggested by our theoretical analysis.
7 Conclusion & Future Directions
We propose FedGCN, a framework for federated training of graph convolutional networks for semi-supervised node classification. The FedGCN training algorithm is based on the insight that, although distributed GCN training typically ignores cross-client edges, these edges often contain information useful to the model, which can be sent in a single round of communication before training begins. FedGCN allows for different levels of communication to accommodate different privacy and overhead concerns, with more communication generally implying less information loss and faster convergence, and further integrates HE for privacy protection. We quantify FedGCN's convergence under different levels of communication and different degrees of non-i.i.d. data across clients and show that FedGCN achieves high accuracy on real datasets, with orders of magnitude less communication than previous algorithms.
Although the FedGCN can overcome the challenges mentioned above, it mainly works on training accumulation-based models like GCN and GraphSage. There are several open problems in federated graph learning that need to be explored.
Federated Training of Attention-based GNNs Attention-based GNNs like GAT (Graph attention network) require calculating the attention weights of edges during neighbor feature aggregation, where the attention weights are based on the node features on both sides of edges and attention parameters. The attention parameters are updated at every training iteration and cannot be simply fixed at the initial round. How to train attention-based GNNs in a federated way with high performance and privacy guarantees is an open challenge and promising direction.
Neighbor Node and Feature Selection to Optimize System Performance General federated graph learning optimizes the system by only sharing local models, without utilizing cross-device graph edge information, which leads to less accurate global models. On the other hand, communicating massive additional graph data among devices introduces communication overhead and potential privacy leakage. To save the communication cost without affecting the model performance, one can select key neighbors and neighbor features to reduce communication costs and remove redundant information. For privacy guarantee, if there is one neighbor node, it can be simply dropped to avoid private data communication. FedGCN can be extended by using selective communication in its pre-training communication round.
Integration with  $L$ -hop Linear GNN Approximation methods To speed up the local computation speed,  $L$ -hop Linear GNN Approximation methods use precomputation to reduce the training computations by running a simplified GCN  $(A^L X W$  in SGC Wu et al. (2019),  $[AXW, A^2 XW, \dots, A^L XW]$  in SIGN Frasca et al. (2020), and  $\Pi X W$  in PPRGo Bojchevski et al. (2020) where  $\Pi$  is the pre-computed personalized pagerank), but the communication cost is not reduced if we perform these methods alone. They are thus a complementary approach for efficient GNN training. FedGCN (2-hop, 1-hop) changes the model input  $(A$  and  $X)$  to reduce communication in the FL setting, but the GCN model itself is not simplified. FedGCN can incorporate these methods to speed up the local computation, especially in constrained edge devices.
This research was supported by the National Science Foundation under grant CNS-1909306, Cloudbank support through CNS-1751075, and the Lee-Stanziale Ohana Fellowship by the ECE department at Carnegie Mellon University. The authors would like to thank Jiayu Chang, Cynthia (Xinyi) Fan, and Shoba Arunasalam for helping with the coding.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 (2018).
Longfei Zheng, Jun Zhou, Chaochao Chen, Bingzhe Wu, Li Wang, and Benyu Zhang. 2021. ASFGNN: Automated separated-federated graph neural network. Peer-to-Peer Networking and Applications 14, 3 (2021), 1692-1704.
Zhixin Zhou and Arash A Amini. 2019. Analysis of spectral clustering algorithms for community detection: the general bipartite setting. The Journal of Machine Learning Research 20, 1 (2019), 1774-1820.