SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For our evaluation process, we computed zero-shot transfer performance of the model on the test set, using the pretraining checkpoint at the 8,000 step of pretraining. The evaluation was conducted on 500 test tasks, with batch size of 1, measured on the downstream task of graph classification accuracy. To maintain consistency, we kept the number of shots and queries constant at 3 for all evaluation tasks. We sample 1-hop neighbourhoods for ConceptNet, FB15K-237 and NELL and 2-hop neighbourhoods for Wiki.
E Ablation on Table 4 for the PG-NM setting
In Table 4, we ablate on various configurations of the self-supervised objective PG-NM. As described in Section 3.2, PG-NM is composed of an attribute prediction loss, dropnode and zeronode augmentations, with the default setting of sampling 3 shots neighbor matching tasks. Our best setting, referred to
as simply "PG-NM", is also shown in Table 1 and comprises of attribute prediction, dropnode and zeronode augmentations, with the default setting of 3 shots.
The ablation results reveal that using all of these elements together results in the highest performance. Specifically, attribute prediction has the greatest impact on PG-NM's performance, as its removal results in an average  $7\%$  drop across all ways, as shown in the 'No-Attr' column.
Removing the dropnode and zeronode augmentations results in an average  $3\%$  drop across all ways, as shown in the No Aug' column. Removing both attribute prediction and augmentations results in performance that is similar to just removing attribute prediction alone, which is also roughly a  $7\%$  drop across all ways, as shown in the 'No Attr, Aug' column. Additionally, we found that decreasing the number of shots to 1 from the default setting of 3 resulted in an average  $3.5\%$  drop across all ways, as shown.
F Evaluation using different numbers of shots
We show evaluation using different numbers of shots, as shown in Figures 3, 5, and 6, 7.
Figure 5: In-context learning accuracy on FB15K-237 in a 20-ways setting wrt. the number of prompt examples (shots).
Figure 6: In-context learning accuracy on NELL in a 20-ways setting wrt. the number of prompt examples (shots).
G Scaling with Data Size
We explore how the model scales with more pretraining tasks. Note that we use the number of train steps as a proxy because the model sees more pretraining tasks as the training proceeds with almost no redundancy (0.20% for 10k tasks). The result on arXiv in a 5-ways setting is illustrated in Figure 4. It shows that the Contrastive baseline saturates quickly and its performance fluctuates given more
Figure 7: In-context learning accuracy on arXiv in a 5-ways setting wrt. the number of prompt examples (shots).
pretraining tasks. Instead, PRODIGY consistently shows an improvement in performance as more data is pretrained on.
We use one NVIDIA A100-SXM4-80GB GPU for all our experiments. One pretrain run of 10k steps takes 3 to 4 hours.
Our work aims to extend the success of in-context learning to graphs and start building toward graph foundation models. This would allow cost-effective and accurate predictions, especially in domains where labeled data is scarce and long tail such as network anomaly detection, rare disease diagnosis/treatment, supply chain disruption, and recommendations for new users. However, overreliance on prior knowledge from pretraining could also lead to increased social bias and unfair benefits to the dominate groups. To mitigate this, pretraining data should be diverse and well-balanced, and the pretrained models should be tested on downstream tasks over different groups and subdistributions.