Package: graphgym
GraphGymGraphGym is a platform for designing and evaluating Graph Neural Networks (GNN).
GraphGym is proposed inDesign Space for Graph Neural Networks,
Jiaxuan You, Rex Ying, Jure Leskovec,NeurIPS 2020 Spotlight.Please refer toPyGfor a tightly integrated version of GraphGym and PyG.Highlights1. Highly modularized pipeline for GNNData:Data loading, data splittingModel:Modularized GNN implementationTasks:Node / edge / graph level GNN tasksEvaluation:Accuracy, ROC AUC, ...2. Reproducible experiment configurationEach experiment isfully described by a configuration file3. Scalable experiment managementEasily launchthousands of GNN experiments in parallelAuto-generateexperiment analyses and figures across random seeds and experiments.4. Flexible user customizationEasilyregister your own modulesingraphgym/contrib/,
such as data loaders, GNN layers, loss functions, etc.NewsGraphGym 0.3.0 has been released. Now you may install stable version of GraphGym viapip install graphgym.GraphGym 0.2.0 has been released. Now GraphGym supports Pytorch Geometric backend, in addition to the default DeepSNAP backend.
You may try it out inrun_single_pyg.sh.cdrun
bashrun_single_pyg.shWhy GraphGym?TL;DR:GraphGym is great for GNN beginners, domain experts and GNN researchers.Scenario 1:You are a beginner to GNN, who wants to understand how GNN works.You probably have read many exciting papers on GNN, and try to write your own GNN implementation.
Using existing packages for GNN, you still have to code up the essential pipeline on your own.
GraphGym is a perfect place for your to start learningstandardized GNN implementation and evaluation.Figure 1: Modularized GNN implementation.Scenario 2:You want to apply GNN to your exciting applications.You probably know that there are hundreds of possible GNN models, and selecting the best model is notoriously hard.
Even worse, we have shown in ourpaperthat the best GNN designs for different tasks differ drastically.
GraphGym provides asimple interface to try out thousands of GNNs in paralleland understand the best designs for your specific task.
GraphGym also recommends a "go-to" GNN design space, after investigating 10 million GNN model-task combinations.Figure 2: A guideline for desirable GNN design choices.(Sampling from 10 million GNN model-task combinations.)Scenario 3:You are a GNN researcher, who wants to innovate GNN models / propose new GNN tasks.Say you have proposed a new GNN layerExampleConv.
GraphGym can help you convincingly argue thatExampleConvis better than sayGCNConv:
when randomly sample from 10 million possible model-task combinations, how oftenExampleConvwill outperformGCNConv,
when everything else is fixed (including the computational cost).
Moreover, GraphGym can help you easily do hyper-parameter search, andvisualizewhat design choices are better.
In sum, GraphGym can greatly facilitate your GNN research.Figure 3: Evaluation of a given GNN design dimension(BatchNorm here).InstallationRequirementsCPU or NVIDIA GPU, Linux, Python3PyTorch, various Python packages; Instructions for installing these dependencies are found below1. Python environment (Optional):We recommend using Conda package managercondacreate-ngraphgympython=3.7sourceactivategraphgym2. Pytorch:InstallPyTorch.
We have verified GraphGym under PyTorch 1.8.0, and GraphGym should work with PyTorch 1.4.0+. For example:# CUDA versions: cpu, cu92, cu101, cu102, cu101, cu111pipinstalltorch==1.8.0+cu101-fhttps://download.pytorch.org/whl/torch_stable.html3. Pytorch Geometric:InstallPyTorch Geometric,
follow their instructions. For example:# CUDA versions: cpu, cu92, cu101, cu102, cu101, cu111# TORCH versions: 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0CUDA=cu101TORCH=1.8.0
pipinstalltorch-scatter-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-sparse-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-cluster-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-spline-conv-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-geometric4. GraphGym and other dependencies:gitclonehttps://github.com/snap-stanford/GraphGymcdGraphGym
pipinstall-rrequirements.txt
pipinstall-e.# From latest verionpipinstallgraphgym# (Optional) From pypi stable version5. Test the installationRun a single experiment.Run a test GNN experiment using GraphGymrun_single.sh.
Configurations are specified inexample.yaml.
The experiment is about node classification on Cora dataset (random 80/20 train/val split).cdrun
bashrun_single.sh# run a single experimentRun a batch of experiments.Run a batch of GNN experiments using GraphGymrun_batch.sh.
Configurations are specified specified inexample.yaml(controls the basic architecture)
andexample.txt(controls how to do grid search).
The experiment examines 96 models in the recommended GNN design space, on 2 graph classification datasets.
Each experiment is repeated 3 times, and we set that 8 jobs can be concurrently run.
Depending on your infrastructure, finishing all the experiments may take a long time;
you can quit the experiment byCtrl-C(GraphGym will properly kill all the processes).cdrun
bashrun_batch.sh# run a batch of experiments(Optional) Run GraphGym with CPU backend.GraphGym supports cpu backend as well -- you only need to add one linedevice: cputo the.yamlfile. Here we provide an example.cdrun
bashrun_single_cpu.sh# run a single experiment using CPU backend(Optional) Run GraphGym with PyG backend.Run GraphGym with Pytorch Geometric (PyG) backendrun_single_pyg.shandrun_batch_pyg.sh,
instead of the default DeepSNAP backend.
The PyG backend follows the native PyG implementation, and is slightly more efficient than the DeepSNAP backend.
Currently the PyG backend only supports user-provided dataset splits, such as PyG native datasets or OGB datasets.cdrun
bashrun_single_pyg.sh# run a single experiment using PyG backendbashrun_batch_pyg.sh# run a batch of experiments using PyG backendGraphGym In-depth Usage1 Run a single GNN experimentA full example is specified inrun/run_single.sh.1.1 Specify a configuration file.In GraphGym, an experiment is fully specified by a.yamlfile.
Unspecified configurations in the.yamlfile will be populated by the default values ingraphgym/config.py.
For example, inrun/configs/example.yaml,
there are configurations on dataset, training, model, GNN, etc.
Concrete description for each configuration is described ingraphgym/config.py.1.2 Launch an experiment.For example, inrun/run_single.sh:pythonmain.py--cfgconfigs/example.yaml--repeat3You can specify the number of different random seeds to repeat via--repeat.1.3 Understand the results.Experimental results will be automatically saved in directoryrun/results/${CONFIG_NAME}/;
in the example above, it isrun/results/example/.
Results for different random seeds will be saved in different subdirectories, such asrun/results/example/2.
The aggregated results over all the random seeds areautomaticallygenerated intorun/results/example/agg,
including the mean and standard deviation_stdfor each metric.
Train/val/test results are further saved into subdirectories, such asrun/results/example/agg/val; here,stats.jsonstores the results after each epoch aggregated across random seeds,best.jsonstores the results atthe epoch with the highest validation accuracy.2 Run a batch of GNN experimentsA full example is specified inrun/run_batch.sh.2.1 Specify a base file.GraphGym supports running a batch of experiments.
To start, a user needs to select a base architecture--config.
The batch of experiments will be created by perturbing certain configurations of the base architecture.2.2 (Optional) Specify a base file for computational budget.Additionally, GraphGym allows a user to select a base architecture tocontrol the computational budgetfor the grid search,--config_budget.
The computational budget is currently measured by the number of trainable parameters; the control is achieved by auto-adjust
the hidden dimension size for GNN.
If no--config_budgetis provided, GraphGym will not control the computational budget.2.3 Specify a grid file.A grid file describes how to perturb the base file, in order to generate the batch of the experiments.
For example, the base file could specify an experiment of 3-layer GCN for Cora node classification.
Then, the grid file specifies how to perturb the experiment along different dimension, such as number of layers,
model architecture, dataset, level of task, etc.2.4 Generate config files for the batch of experiments,based on the information specified above.
For example, inrun/run_batch.sh:pythonconfigs_gen.py--configconfigs/${DIR}/${CONFIG}.yaml\--config_budgetconfigs/${DIR}/${CONFIG}.yaml\--gridgrids/${DIR}/${GRID}.txt\--out_dirconfigs2.5 Launch the batch of experiments.For example, inrun/run_batch.sh:bashparallel.shconfigs/${CONFIG}_grid_${GRID}$REPEAT$MAX_JOBS$SLEEPEach experiment will be repeated for$REPEATtimes.
We implemented a queue system to sequentially launch all the jobs, with$MAX_JOBSconcurrent jobs running at the same time.
In practice, our system works great when handling thousands of jobs.2.6 Understand the results.Experimental results will be automatically saved in directoryrun/results/${CONFIG_NAME}_grid_${GRID_NAME}/;
in the example above, it isrun/results/example_grid_example/.
After running each experiment, GraphGym additionally automatically averages across different models, saved inrun/results/example_grid_example/agg.
There,val.csvrepresents validation accuracy for each model configuration at thefinalepoch;val_best.csvrepresents the results at the epoch with the highest average validation error;val_best_epoch.csvrepresents the results at the epoch with the highest validation error, averaged over different random seeds.
When test set split is provided,test.csvrepresents test accuracy for each model configuration at thefinalepoch;test_best.csvrepresents the test set results at the epoch with the highest average validation error;test_best_epoch.csvrepresents the test set results at the epoch with the highest validation error, averaged over different random seeds.3 Analyze the resultsWe provides a handy tool to automatically provide an overview of a batch of experiments inanalysis/example.ipynb.cdanalysis
jupyternotebook
example.ipynb# automatically provide an overview of a batch of experiments4 User customizationA highlight of GraphGym is that it allows users to easily register their customized modules.
The supported customized modules are provided in directorygraphgym/contrib/, including:Activationgraphgym/contrib/act/,Customized configurationsgraphgym/contrib/config/,Feature augmentationgraphgym/contrib/feature_augment/,Feature encodergraphgym/contrib/feature_encoder/,GNN headgraphgym/contrib/head/,GNN layergraphgym/contrib/layer/,Data loadergraphgym/contrib/loader/,Loss functiongraphgym/contrib/loss/,GNN network architecturegraphgym/contrib/network/,Optimizergraphgym/contrib/optimizer/,GNN global pooling (graph classification only)graphgym/contrib/pooling/,GNN stagegraphgym/contrib/stage/,GNN training pipelinegraphgym/contrib/train/,Data transformationsgraphgym/contrib/transform/.Within each directory, (at least) an example is provided, showing how to register user customized modules.
Note that new user customized modules may result in new configurations; in these cases, new configuration fields
can be registered atgraphgym/contrib/config/.Note: Applying to your own datasets.A common use case will be applying GraphGym to your favorite datasets.
To do so, you may follow our example ingraphgym/contrib/loader/example.py.
GraphGym currently accepts a list ofNetworkXgraphs
orPyGdatasets.Use case: Design Space for Graph Neural Networks (NeurIPS 2020 Spotlight)Reproducing experiments inDesign Space for Graph Neural Networks, Jiaxuan You, Rex Ying, Jure Leskovec,NeurIPS 2020 Spotlight.
You may refer to thepaperorproject webpagefor more details.# NOTE: We include the raw results with GraphGym# If you run the following code, the results will be overridden.cdrun/scripts/design/
bashrun_design_round1.sh# first round experiments, on a design space of 315K GNN designsbashrun_design_round2.sh# second round experiments, on a design space of 96 GNN designscd../analysis
jupyternotebook
design_space.ipynb# reproducing all the analyses in the paperFigure 4: Overview of the proposed GNN design space and task space.Use case: Identity-aware Graph Neural Networks (AAAI 2021)Reproducing experiments inIdentity-aware Graph Neural Networks, Jiaxuan You, Jonathan Gomes-Selman, Rex Ying, Jure Leskovec,AAAI 2021.
You may refer to thepaperorproject webpagefor more details.# NOTE: We include the raw results for ID-GNN in analysis/idgnn.csvcdrun/scripts/IDGNN/
bashrun_idgnn_node.sh# Reproduce ID-GNN node-level resultsbashrun_idgnn_edge.sh# Reproduce ID-GNN edge-level resultsbashrun_idgnn_graph.sh# Reproduce ID-GNN graph-level resultsFigure 5: Overview of Identity-aware Graph Neural Networks (ID-GNN).ContributorsJiaxuan Youinitiates the project and majorly contributes to the entire GraphGym platform.Rex Yingcontributes to the feature augmentation modules.
Jonathan Gomes Selman enables GraphGym to have OGB support.GraphGym is inspired by the framework ofpycls.
GraphGym adoptsDeepSNAPas the default data representation.
Part of GraphGym relies onPytorch Geometricfunctionalities.ContributingWe warmly welcome the community to contribute to GraphGym.
GraphGym is particularly designed to enable contribution / customization in a simple way.
For example, you may contribute your modules tographgym/contrib/by creating pull requests.Citing GraphGymIf you find GraphGym or our paper useful, please cite our paper:@InProceedings{you2020design,
  title = {Design Space for Graph Neural Networks},
  author = {You, Jiaxuan and Ying, Rex and Leskovec, Jure},
  booktitle = {NeurIPS},
  year = {2020}
}