SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuhang Yao
Carnegie Mellon University  
yuhangya@andrew.cmu.edu
Weizhao Jin
University of Southern California weizhaoj@usc.edu
Srivatsan Ravi
University of Southern California sravi@isi.edu
Carlee Joe-Wong
Carnegie Mellon University  
cjoewong@andrew.cmu.edu
Methods for training models on graphs distributed across multiple clients have recently grown in popularity, due to the size of these graphs as well as regulations on keeping data where it is generated. However, the cross-client edges naturally exist among clients. Thus, distributed methods for training a model on a single graph incur either significant communication overhead between clients or a loss of available information to the training. We introduce the Federated Graph Convolutional Network (FedGCN) algorithm, which uses federated learning to train GCN models for semi-supervised node classification with fast convergence and little communication. Compared to prior methods that require extra communication among clients at each training round, FedGCN clients only communicate with the central server in one pre-training step, greatly reducing communication costs and allowing the use of homomorphic encryption to further enhance privacy. We theoretically analyze the tradeoff between FedGCN's convergence rate and communication cost under different data distributions. Experimental results show that our FedGCN algorithm achieves better model accuracy with  $51.7\%$  faster convergence on average and at least  $100 \times$  less communication compared to prior work<sup>1</sup>.
Graph convolutional networks (GCNs) have been widely used for applications ranging from fake news detection in social networks to anomaly detection in sensor networks (Benamira et al., 2019; Zhang et al., 2020). This data, however, can be too large to store on a single server, e.g., records of billions of users' website visits. Strict data protection regulations such as the General Data Protection Regulation (GDPR) in Europe and Payment Aggregators and Payment Gateways (PAPG) in India also require that private data only be stored in local clients. In non-graph settings, federated learning has recently shown promise for training models on data that is kept at multiple clients (Zhao et al., 2018; Yang et al., 2021). Some papers have proposed federated training of GCNs (He et al., 2021a; Zhang et al., 2021). Typically, these consider a framework in which each client has access to a subset of a large graph, and clients iteratively compute local updates to a semi-supervised model on their local subgraphs, which are occasionally aggregated at a central server. Figure 1(left) illustrates the federated node classification task of predicting unknown labels of local nodes in each client.
$^{1}$ Code in https://github.com/yh-yao/FedGCN
Figure 1: Federated GCN training schematic for node classification, with colors indicating the known labels of some nodes. Nodes in a graph (shown as circles) are distributed across clients, and dashed lines show cross-client edges between nodes at different clients. Arrows in the left figure indicate that each client can exchange updates with a central server during the training process to predict the unknown labels of the grey nodes in each client. At right, we show a graph with i.i.d (top) and non-i.i.d. (bottom) data distribution across clients, which affects the number of cross-client edges.
The main challenge of applying federated learning to GCN training tasks involving a single large graph is that cross-client edges exist among clients. In Figure 1, for example, we see that some edges will connect nodes in different clients. We refer to these as "cross-client edges".
Such cross-client edges typically are stored in both clients. Intuitively, this is due to the fact that edges are generated when nodes at clients interact with each other. Thus, the interaction record, though not personal node characteristics, is then naturally stored at both nodes, i.e., in both clients. For example, a graph may represent buying behaviors (edges) that exist between users (nodes) in two countries (clients). Users in one country want to buy items in another country. The records of these transactions between users in different countries (i.e., the cross-client edges) are then stored in both clients. Due to the General Data Protection Regulation, however, sensitive user information (node features including name, zip code, gender, birthday, credit card number, email address, etc.) cannot be stored in another country. Yet these cross-client edges cannot be ignored: including cross-country transactions (cross-client edges) is key for training models that detect international money laundering and fraud. Another example is online social applications like Facebook and LinkedIn. Users in different countries can build connections with each other (e.g., a person in the United States becoming Facebook friends with a person in China). The users in both the U.S. and China would then have a record of this friendship link, while the personal user information cannot be shared across countries.
However, GCNs require information about a node's neighbors to be aggregated in order to construct an embedding of each node that is used to accomplish tasks such as node classification and link prediction. Ignoring the information from neighbors located at another client, as in prior federated graph training algorithms (Wang et al., 2020a; He et al., 2021b), may then result in less accurate models due to loss of information from nodes at other clients.
Prior works on federated or distributed graph training reduce cross-client information loss by communicating information about nodes' neighbors at other clients in each training round (Scardapane et al., 2020; Wan et al., 2022; Zhang et al., 2021), which can introduce significant communication overhead and reveal private node information to other clients. We instead realize that the information needed to train a GCN only needs to be communicated once, before training. This insight allows us to further alleviate the privacy challenges of communicating node information between clients (Zhang et al., 2021). Specifically, we leverage Homomorphic Encryption (HE), which can preserve client privacy in federated learning but introduces significant overhead for each communication round; with only one communication round, this overhead is greatly reduced. Further, in practice each client may contain several node neighbors, e.g., clients might represent social network data in different countries, which cannot leave the country due to privacy regulations. Each client would then receive aggregated feature information about all of a node's neighbors in a different country, which itself can help preserve privacy through accumulation across multiple nodes. In the extreme case when nodes only have one cross-client neighbor, we can further integrate differential privacy techniques (Wei et al., 2020). We
propose the FedGCN algorithm for distributed GCN training based on these insights. FedGCN greatly reduces communication costs and speeds up convergence without information loss, compared with existing distributed settings (Scardapane et al., 2020; Wan et al., 2022; Zhang et al., 2021)
In some settings, we can further reduce FedGCN's required communication without compromising the trained model's accuracy. First, GCN models for node classification rely on the fact that nodes of the same class will have more edges connecting them, as shown in Figure 1(right). If nodes with each class tend to be concentrated at a single client, a version of the non-i.i.d. (non-independent and identically distributed) data often considered in federated learning, then ignoring cross-client edges discards little information, and FedGCN's communication round may be unnecessary. The model, however, may not converge, as federated learning may converge poorly when client data is non-i.i.d. (Zhao et al., 2018). Second, GCN models with multiple layers require accumulated information from nodes that are multiple hops away from each other, introducing greater communication overhead. However, such multi-hop information may not be needed in practice.
We analytically quantify the convergence rate of FedGCN with various degrees of communication, under both i.i.d. and non-i.i.d. client data. To the best of our knowledge, we are the first to analytically illustrate the resulting tradeoff between a fast convergence rate (which intuitively requires more information from cross-client edges) and low communication cost, which we do by considering a stochastic block model (Lei and Rinaldo, 2015; Keriven et al., 2020) of the graph topology. We thus quantify when FedGCN's communication significantly accelerates the GCN's convergence.
In summary, our work has the following contributions:
- We introduce FedGCN, an efficient framework for federated training of GCNs to solve node-level prediction tasks with limited communication and information loss, which also leverages Fully Homomorphic Encryption for enhanced privacy guarantees.
- We theoretically analyze the convergence rate and communication cost of FedGCN compared to prior methods, as well as its dependence on the data distribution. We can thus quantify the usefulness of communicating different amounts of cross-client information.
- Our experiments on both synthetic and real-world datasets demonstrate that FedGCN outperforms existing distributed GCN training methods in most cases with a fast convergence rate, higher accuracy, and orders-of-magnitude lower communication cost.
We outline related works in Section 2 before introducing the problem of node classification in graphs in Section 3. We then introduce FedGCN in Section 4 and analyze its performance theoretically (Section 5) and experimentally (Section 6) before concluding in Section 7.
Graph neural networks aim to learn representations of graph-structured data (Bronstein et al., 2017). GCNs (Kipf and Welling, 2016), GraphSage (Hamilton et al., 2017), and GAT (Veličković et al., 2017) perform well on node classification and link prediction. Several works provide a theoretical analysis of GNNs based on the Stochastic Block Model (Zhou and Amini, 2019; Lei and Rinaldo, 2015; Keriven et al., 2020). We similarly adopt the SBM to quantify FedGCN's performance.
Federated learning was first proposed in McMahan et al. (2017)'s widely adopted FedAvg algorithm, which allows clients to train a model via coordination with a central server while keeping training data at local clients. However, FedAvg may not converge if data from different clients is non-i.i.d. (Zhao et al., 2018; Li et al., 2019a; Yang et al., 2021). We show similar results for federated graph training.
Federated learning on graph neural networks is a topic of recent interest (He et al., 2021a). Unlike learning tasks in which multiple graphs each constitute a separate data sample and are distributed across clients (e.g., graph classification (Zhang et al., 2018), image classification (Li et al., 2019b), and link prediction (Yao et al., 2023), FedGCN instead considers semi-supervised tasks on a single large graph (e.g., for node classification). Existing methods for such tasks generally ignore the resulting cross-client edges (He et al., 2021a). Scardapane et al. (2020)'s distributed GNN proposes a training algorithm communicating the neighbor features and intermediate outputs of GNN layers among clients with expensive communication costs. BDS-GCN (Wan et al., 2022) proposes to sample cross-client neighbors. These methods may violate client privacy by revealing per-node information to other clients. FedSage+ (Zhang et al., 2021) recovers missing neighbors for the input graph based
on the node embedding, which requires fine-tuning a linear model of neighbor generation and may not fully recover the cross-client information. It is further vulnerable to the data reconstruction attack, compromising privacy.
All of the above works further require communication at every training round, while FedGCN enables the private recovery of cross-client neighbor information with a single, pre-training communication round that utilizes HE. We also provide theoretical bounds on FedGCN's convergence.
3 Federated Semi-Supervised Node Classification
In this section, we formalize the problem of node classification on a single graph and introduce the federated setting in which we aim to solve this problem.
We consider a graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V} = [N]$  is the set of  $N$  nodes and  $\mathcal{E}$  is the set of edges. The graph is equivalent to a weighted adjacency matrix  $\mathbf{A}\in \mathbb{R}^{N\times N}$ , where  $\mathbf{A}_{ij}$  indicates the weight of an edge from node  $i$  to node  $j$  (if the edge does not exist, the weight is zero). Every node  $i\in \mathcal{V}$  has a feature vector  $\boldsymbol{x}_i\in \mathbb{R}^d$ , where  $d$  represents the number of input features. Each node  $i$  in a subset  $\mathcal{V}^{train}\subset \mathcal{V}$  has a corresponding label  $y_{i}$  used during training. Semi-supervised node classification aims to assign labels to nodes in the remaining set  $\mathcal{V}\backslash \mathcal{V}^{train}$ , based on their feature vectors and edges to other nodes. We train a GCN model to do so.
GCNs (Kipf and Welling, 2016) consist of multiple convolutional layers, each of which constructs a node embedding by aggregating the features of its neighboring nodes. Typically, the node embedding matrix  $\pmb{H}^{(l)}$  for each layer  $l = 1, 2, \dots, L$  is initialized to  $\pmb{H}^{(0)} = \pmb{X}$ , the matrix of features for each node (i.e., each row of  $\pmb{X}$  corresponds to the features for one node), and follows the propagation rule  $\pmb{H}^{(l + 1)} = \phi (\pmb{A}\pmb{H}^{(l)}\pmb{W}^{(l)})$ . Here  $\pmb{W}^{(l)}$  are parameters to be learned,  $\pmb{A}$  is the weighted adjacency matrix, and  $\phi$  is an activation function. Typically,  $\phi$  is chosen as the softmax function in the last layer, so that the output can be interpreted as the probabilities of a node lying in each class, with ReLU activations in the preceding layers. The embedding of each node  $i \in \mathcal{V}$  at layer  $l + 1$  is then
$$
\boldsymbol {h} _ {i} ^ {(l + 1)} = \phi \left(\sum_ {j \in \mathcal {N} _ {i}} \boldsymbol {A} _ {i j} \boldsymbol {h} _ {j} ^ {(l)} \boldsymbol {W} ^ {(l)}\right), \tag {1}
$$
which can be computed from the previous layer's embedding  $\pmb{h}_j^{(l)}$  for each neighbor  $j$  and the weight  $A_{ij}$  on edges from node  $i$  to node  $j$ . For a GCN with  $L$  layers in this form, the output for node  $i$  will depend on neighbors up to  $L$  steps away (i.e., there exists a path of no more than  $L$  edges to node  $i$ ). We denote this set by  $\mathcal{N}_i^L$  (note that  $i \in \mathcal{N}_i^L$ ) and refer to these nodes as  $L$ -hop neighbors of  $i$ .
To solve the node classification problem in federated settings (Figure 1), we consider, as usual in federated learning, a central server with  $K$  clients. The graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  is separated across the  $K$  clients, each of which has a sub-graph  $\mathcal{G}_k = (\mathcal{V}_k,\mathcal{E}_k)$ . Here  $\bigcup_{k = 1}^{K}\mathcal{V}_k = \mathcal{V}$  and  $\mathcal{V}_i\cap \mathcal{V}_j = \emptyset$  for  $\forall i\neq j\in [K]$ , i.e., the nodes are disjointly partitioned across clients. The features of nodes in the set  $\mathcal{V}_k$  can then be represented as the matrix  $X_{k}$ . The cross-client edges of client  $k$ ,  $\mathcal{E}_k^c$ , for which the nodes connected by the edge are at different clients, are known to the client  $k$ . We use  $\mathcal{V}_k^{train}\subset \mathcal{V}_k$  to denote the set of training nodes with associated labels  $y_{i}$ . The task of federated semi-supervised node classification is then to assign labels to nodes in the remaining set  $\mathcal{V}_k\backslash \mathcal{V}_k^{train}$  for each client  $k$ .
As seen from (1), in order to find the embedding of the  $i$ -th node in the  $l$ -th layer, we need the previous layer's embedding  $\pmb{h}_j^{(l)}$  for all neighbors of node  $i$ . In the federated setting, however, some of these neighbors may be located at other clients, and thus their embeddings must be iteratively sent to the client that contains node  $i$  for each layer at every training round. He et al. (2021a) ignore these neighbors, considering only  $\mathcal{G}_k$  and  $\varepsilon_{k}$  in training the model, while Scardapane et al. (2020); Wan et al. (2022); Zhang et al. (2021) require such communication, which may lead to high overhead and privacy costs. FedGCN provides a communication-efficient method to account for these neighbors.
4 Federated Graph Convolutional Network
In order to overcome the challenges outlined in Section 3, we propose our Federated Graph Convolutional Network (FedGCN) algorithm. In this section, we first introduce our federated training method with communication at the initial step and then outline the corresponding training algorithm.
Federating Graph Convolutional Networks. In the federated learning setting, let  $c(i)$  denote the index of the client that contains node  $i$  and  $W_{c(i)}^{(l)}$  denote the weight matrix of the  $l$ -th GCN layer of client  $c(i)$ . The embedding of node  $i$  at layer  $l + 1$  is then  $h_i^{(l + 1)} = \phi \left(\sum_{j\in \mathcal{N}_i}A_{ij}h_j^{(l)}W_{c(i)}^{(l)}\right)$ .
Note that the weights  $W_{c(i)}^{(l)}$  may differ from client to client, due to the independent local training in federated learning. For example, we can then write the computation of a 2-layer federated GCN as  $\hat{\pmb{y}}_i = \phi \left(\sum_{j\in \mathcal{N}_i}\pmb {A}_{ij}\phi \left(\sum_{m\in \mathcal{N}_j}\pmb {A}_{jm}\pmb{x}_m^T\pmb{W}_{c(i)}^{(1)}\right)\pmb{W}_{c(i)}^{(2)}\right)$ . To evaluate this 2-layer model, it then suffices for the client  $k = c(i)$  to receive the message  $\sum_{m\in \mathcal{N}_j}\pmb {A}_{jm}\pmb{x}_m^T$ . We can write these messages as
$$
\sum_ {j \in \mathcal {N} _ {i}} \boldsymbol {A} _ {i j} \boldsymbol {x} _ {j}, \text {a n d} \left\{\sum_ {m \in \mathcal {N} _ {j}} \boldsymbol {A} _ {j m} \boldsymbol {x} _ {j} \right\} _ {j \in \mathcal {N} _ {i} / i}, \tag {2}
$$
which are the feature aggregations of 1- and 2-hop neighbors of node  $i$  respectively. This information does not change with the model training, as it simply depends on the (fixed) adjacency matrix  $\overline{A}$  and node features  $\pmb{x}$ . The client also naturally knows  $\{\pmb{A}_{ij}\}_{\forall j\in \mathcal{N}_i}$ , which is included in  $\mathcal{E}_k\bigcup \mathcal{E}_k^c$ .
One way to obtain the above information is to receive the following message from clients  $z$  that contain at least one two-hop neighbor of  $k$ :
$$
\sum_ {j \in \mathcal {N} _ {i}} \mathbb {I} _ {z} (c (j)) \boldsymbol {A} _ {i j} \boldsymbol {x} _ {j}, \text {a n d} \forall j \in \mathcal {N} _ {i}, \sum_ {m \in \mathcal {N} _ {j}} \mathbb {I} _ {z} (c (m)) \cdot \boldsymbol {A} _ {j m} \boldsymbol {x} _ {m}. \tag {3}
$$
Here the indicator  $\mathbb{I}_z(c(m))$  is 1 if  $z = c(m)$  and zero otherwise. More generally, for a  $L$ -layer GCN, each layer requires  $\forall j \in \mathcal{N}_i^L / \mathcal{N}_i^{L-1}, \sum_{m \in \mathcal{N}_j} \mathbb{I}_z(c(m)) \cdot \pmb{A}_{jm}\pmb{x}_m$ . Further,  $\mathcal{E}_i^{L-1}$ , i.e., the set of edges up to  $L-1$  hops away from node  $i$ , is needed for normalization of  $A$ .
To avoid the overhead of communicating between multiple pairs of clients, which can also induce privacy leakage when there is only one neighbor node in the client, we can instead send the aggregation of each client to the central server. In the 2-layer GCN example, the server then calculates the sum of neighbor features of node  $i$  as  $\sum_{j \in \mathcal{N}_i} A_{ij} x_j = \sum_{k=1}^{K} \sum_{j \in \mathcal{N}_i} \mathbb{I}_k(c(j)) \cdot A_{ij} x_j$ . The server can then send the required feature aggregation in (2) back to each client  $k$ . Thus, we only need to send the accumulated features of each node's (possibly multi-hop) neighbors, in order to evaluate the GCN model. If there are multiple neighbors stored in other clients, this accumulation serves to protect their individual privacy<sup>2</sup>. For the computation of all nodes  $\mathcal{V}_k$  stored in client  $k$  with an  $L$ -layer GCN, the client needs to receive  $\{\sum_{j \in \mathcal{N}_i} A_{ij} x_j\}_{i \in \mathcal{N}_{\mathcal{V}_k}^L}$ , where  $\mathcal{N}_{\mathcal{V}_k}^L$  is the set of  $L$ -hop neighbors of nodes  $\mathcal{V}_k$ .
FedGCN is based on the insight that GCNs require only the accumulated information of the  $L$ -hop neighbors of each node, which may be communicated in advance of the training. In practice, however, even this communication may be infeasible. If  $L$  is too large,  $L$ -hop neighbors may actually consist of the entire graph (social network graphs have diameters  $< 10$ ), which might introduce prohibitive storage and communication requirements. Thus, we design FedGCN to accommodate three types of communication approximations, according to the most appropriate choice for a given application:
- No communication (0-hop): If any communication is unacceptable, e.g., due to overhead, each client simply trains on  $\mathcal{G}_k$  and ignores cross-client edges, as in prior work.
- One-hop communication: If some communication is permissible, we may use the accumulation of feature information from nodes' 1-hop neighbors,  $\{\sum_{j\in \mathcal{N}_i}\pmb {A}_{ij}\pmb {x}_j\}_{i\in \mathcal{V}_k}$ , to approximate the GCN computation. 1-hop neighbors are unlikely to introduce significant memory or communication overhead as long as the graph is sparse, e.g. social networks.
- Two-hop communication: To further improve model performance, we can communicate the information from 2-hop neighbors,  $\{\sum_{j\in \mathcal{N}_i}\pmb {A}_{ij}\pmb {x}_j\}_{i\in \mathcal{N}_{\nu_k}}$  and perform the aggregation of  $L$  -layer GCNs. As shown in Figure 2, the 2-hop approximation does not decrease model accuracy in practice compared to  $L$  -hop communication for  $L$  -hop GCNs, up to  $L\leq 10$
In the extreme case when the node only has one neighbor stored in other clients, we can drop the neighbor, which likely has minimal effect on model performance, or add differential privacy to the communicated data.
Figure 2: Test Accuracy of GCNs with different numbers of layers, using centralized and FedGCN training on Cora (left) and Ogbn-Arxiv (right) datasets with 10 clients and non-i.i.d partition. Communicating 2-hop information in FedGCN is sufficient for up to 10-layer GCNs. In Ogbn-Arxiv, BatchNorm1d is added between GCN layers to ensure consistent performance (Hu et al., 2020).
Why Not  $L$ -hop Communication? Although FedGCN supports  $L$ -hop communication, communicating  $L$ -hop neighbors across clients requires knowledge of the  $L - 1$  hop neighborhood graph structures, which may incur privacy leakage. If  $L$  is large enough,  $L$ -hop neighbors may also cover the entire graph, incurring prohibitive communication and computation costs. Thus, in practice we restrict ourselves to 2-hop communication, which only requires knowledge of 1-hop neighbors. Indeed, Figure 2 shows that even on the Ogbn-Arxiv dataset, which has more than one million edges, adding more layers and  $k$ -hop communication does not increase model accuracy for  $L \geq 3$  and  $k \geq 2$ . Thus, it is reasonable to use 0, 1, or 2-hop communication with  $L \leq 3$ -layer GCNs in practice.
Secure Neighbor Feature Aggregation. To guarantee privacy during the aggregation process of accumulated features, we leverage Homomorphic Encryption (HE) to construct a secure neighbor feature aggregation function. HE (Brakerski et al., 2014; Cheon et al., 2017) allows a computing party to perform computation over ciphertext without decrypting it, thus preserving the plaintext data.