SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Overview of the graph datasets used in this study.
Avg. # nodes	Min. # nodes	Max. # nodes	Avg. # edges	Min. # edges	Max. # edges
MalNet-Tiny	1,410	5	4,994	2,860	4	20,096
MalNet-Large	47,838	3,374	541,571	225,474	20,597	3,278,318
TpuGraphs	38,444	299	615,019	62,475	380	1,058,278
We follow GraphGym [36] to represent design spaces of GNN as (message passing layer type, number of pre-process layers, number of message passing layers, number of post-process layers, activation, aggregation). Our code is implemented in PyTorch [22].
Implementation details for MalNet-Large. We consider three model variations for the MalNet-Large dataset. Please refer to their hyperparameters in Table 5. We use Adam optimizer [18] with the base learning rate of 0.01 for GCN and SAGE. For GraphGPS, we use AdamW optimizer [20] with the cosine scheduler and the base learning rate of 0.0005. We use L2 regularization with a weight decay of 1e-4. We train for 600 epochs until convergence. For Prediction Head Finetuning, we finetune for another 100 epochs. We limit the maximum segment size to 5,000 nodes, and use a keep probability  $p = 0.5$  if not otherwise specified. We train with CrossEntropy loss.
Table 5: Detailed GNN/Graph Transformer designs used in MalNet-Tiny and MalNet-Large.
model	GCN	SAGE	GraphGPS
message passing layer type	GCNConv	SAGEConv	GatedGCN+Performer
pre-process layer num.	1	1	0
message passing layer num.	2	2	5
post-process layer num.	1	1	3
hidden dimension	300	300	64
activation	PReLU	PReLU	ReLU
aggregation	mean	mean	mean
Implementation details for MalNet-Tiny. We use the same model architectures/training schedules as in the MalNet-Large dataset. The only difference is that as graphs in MalNet-Tiny have no more than 5000 nodes, so we limit maximum segment size to 500 here.
Implementation details for TpuGraphs. We only consider SAGE with configurations (SAGEConv, 0, 4, 3, 128, ReLU, sum) for the TpuGraphs dataset. We use Adam optimizer with the base learning rate of 0.0001. We train for 200,000 iterations until convergence. We by default limit the maximum segment size to 8,192 nodes, and use a keep probability  $p = 0.5$  if not otherwise specified. Since we care more about relative ranking than the absolute runtime, we use PairwiseHinge loss within a batch during training:
$$
\mathcal {L} (\widehat {y} _ {1}, \widehat {y} _ {2}) = \sum_ {i} \sum_ {j} \mathbb {I} [ y _ {i} > y _ {j} ] \cdot \max  (0, 1 - (\widehat {y} _ {i} - \widehat {y} _ {j}))
$$
Convergence analysis. To study the effect on convergence for the proposed framework GST and technique SED, we visualize training/test curve per epoch on TpuGraphs dataset in Figure 5, MalNet-Tiny dataset in Figure 6. We show that the convergence speed of various methods studied are quite similar. In addition, the convergence speed in terms of iterations is similar between Full Graph Training and the proposed GST. Due to certain implementation overheads, e.g., on-the-fly graph segment extraction, embedding table query, we didn't observe speed up in terms of wall-clock time for our current implementation yet. Nevertheless, the implementation can be optimized further to reduce the overhead, we leave it for future work.
Ablation study on partition algorithms. We incorporated an ablation study focusing on various graph partition algorithms. The test accuracy of GST+EFD using the SAGE backbone for 5 iterations is depicted in Table 6. Our findings indicate that the Random Edge-Cut algorithm delivers subpar results due to its inability to maintain the integrity of the subgraph structure. Some attentive readers might hypothesize that ignoring edges between segments could lead to a decrease in graph property prediction accuracy, leading us to further investigate Vertex-Cut partition algorithms. In theory, Vertex-Cut techniques, which distribute edges across different machines and duplicate nodes as necessary, are likely to result in less information loss compared to Edge-Cut methods. Our empirical findings show that all partition algorithms that manage to retain the local structure have quite similar performance levels. This suggests that edges connecting different segments do not have a significant impact on the final prediction accuracy.
The paper presents Graph Segment Training (GST), a framework for predicting properties of large graphs using a divide-and-conquer approach. This method addresses the challenge of memory
Figure 5: Accuracy curve on TpuGraphs dataset.
Figure 6: Accuracy curve on MalNet-Tiny dataset.
limitation during training and has the potential to bring significant impact across various domains, with notable efficiency and accuracy improvements. GST can be particularly beneficial to industries that need to manage and interpret large-scale graph data. In telecommunications, it can help optimize network infrastructure, while in cybersecurity, it could improve anomaly detection in network traffic. Furthermore, enabling the capability in large network analysis can contribute to more resilient infrastructure, enhancing the quality of life in many communities. As with any advancement in AI, there's a risk that the benefits of this technology will be unevenly distributed, potentially increasing economic disparity. Companies with access to large amounts of data and the computational resources to analyze it might reap disproportionate benefits, which could further exacerbate the digital divide.
While Graph Segment Training (GST) represents a significant advancement in graph property prediction, there are several potential limitations to this approach, as highlighted below:
Segmentation Limitations: The efficacy of the Graph Segment Training (GST) approach is influenced by the proficiency of the graph partitioning procedure. While our empirical findings indicate that various partitioning algorithms that maintain locality tend to produce satisfactory outcomes, partitions created through random edge-cut methods haven't demonstrated the same level of success.
Table 6: Test accuracy when combined with different partition algorithms.
MalNet-Tiny	MalNet-Large
Edge-Cut	Random	85.43±0.98	74.02±2.23
Edge-Cut	Louvain	88.95±0.67	89.16±0.85
Edge-Cut	METIS	89.24±0.53	89.78±0.68
Vertex-Cut	Random	88.12±1.17	87.69±1.51
Vertex-Cut	DBH [33]	88.79±0.74	89.28±0.93
Vertex-Cut	NE [39]	89.16±0.70	89.49±0.87
Historical Embedding Table: The use of a historical embedding table to obtain embeddings for non-sampled segments introduces additional implementation complexity and potential for errors. If not managed properly, it could lead to inefficient memory usage or even slower the training process.