Package: graph-ood
GOODGOOD: A Graph Out-of-Distribution BenchmarkDocumentation|PaperWe are actively building the document.OverviewWhy GOOD?InstallationQuick tutorialModule usage (recommended: use only GOOD datasets)Project usage (recommended: OOD algorithm researches & developments)ReproducibilityContactOverviewGOOD(Graph OOD) is a graph out-of-distribution (OOD) algorithm benchmarking library depending on PyTorch and PyG
to make develop and benchmark OOD algorithms easily.Currently, GOOD contains 8 datasets with 14 domain selections. When combined with covariate, concept, and no shifts, we obtain 42 different splits.
We provide performance results on 7 commonly used baseline methods (ERM, IRM, VREx, GroupDRO, Coral, DANN, Mixup) with 10 random runs.The GOOD dataset summaries are shown in the following figure.Why GOOD?Whether you are an experienced researcher of graph out-of-distribution problems or a first-time learner of graph deep learning,
here are several reasons to use GOOD as your Graph OOD research, study, and development toolkit.Easy-to-use APIs:GOOD provides simple APIs for loading OOD algorithms, graph neural networks, and datasets so that you can take only several lines of code to start.Flexibility:Full OOD split generalization code is provided for extensions and any new graph OOD dataset contributions.
OOD algorithm base class can be easily overwritten to create new OOD methods.Easy-to-extend architecture:In addition to playing as a package, GOOD is also an integrated and well-organized project ready to be further developed.
All algorithms, models, and datasets can be easily registered byregisterand automatically embedded into the designed pipeline like a breeze!
The only thing the user needs to do is write your own OOD algorithm class, your own model class, or your new dataset class.
Then you can compare your results with the leaderboard.Easy comparisons with the leaderboard:We provide insightful comparisons from multiple perspectives. Any research and studies can use
our leaderboard results for comparison. Note that this is a growing project, so we will include new OOD algorithms gradually.
Besides, if you hope to include your algorithms in the leaderboard, please contact us or contribute to this project. A big welcome!Reproducibility:OOD Datasets: GOOD provides full OOD split generalization code to reproduce or generate new datasets.Leaderboard results: One random seed round results are provided, and loaded models pass the test result reproduction.InstallationConda dependenciesGOOD depends onPyTorch (>=1.6.0),PyG (>=2.0), andRDKit (>=2020.09.5). For more details:conda environmentNote that we currently test on PyTorch (==1.10.1), PyG (==2.0.3), RDKit (==2020.09.5); thus we strongly encourage to install these versions.Attention! Due to a known issue, please install PyG through Pip to avoid incompatibility.Pip (Beta)Only use modules independently (pending)pipinstallgraph-oodTake the advantages of whole project (recommended)gitclonehttps://github.com/divelab/GOOD.git&&cdGOOD
pipinstall-e.Quick TutorialModule usageGOOD datasetsThere are two ways to import 8 GOOD datasets with 14 domain selections and a total 42 splits, but for simplicity, we only show one of them.
Please refer toTutorialfor more details.# Directly importfromGOOD.data.good_datasets.good_hivimportGOODHIVhiv_datasets,hiv_meta_info=GOODHIV.load(dataset_root,domain='scaffold',shift='covariate',generate=False)GOOD GNNsThe best and fair way to compare algorithms with the leaderboard is to use the same and similar graph encoder structure;
therefore, we provide GOOD GNN APIs to support. Here, we use an objectified dictionaryconfigto pass parameters. More
details about the config:Documents of configTo use exact GNNfromGOOD.networks.models.GCNsimportGCNmodel=GCN(config)To only use parts of GNNfromGOOD.networks.models.GINvirtualnodeimportGINEncoderencoder=GINEncoder(config)GOOD algorithmsTry to apply OOD algorithms to your own models?fromGOOD.ood_algorithms.algorithms.VREximportVRExood_algorithm=VREx(config)# Then you can provide it to your model for necessary ood parameters,# and use its hook-like function to process your input, output, and loss.Project usageIt is a good beginning to make it work directly. Here, we provide the command line scriptgoodtg(GOOD to go) to access the main function located atGOOD.kernel.pipeline:main.
Choosing a config file inconfigs/GOOD_configs, we can start a task:goodtg--config_pathGOOD_configs/GOODCMNIST/color/concept/DANN.yamlSpecifically, the task is clearly divided into three steps:ConfigfromGOODimportconfig_summonerfromGOOD.utils.argsimportargs_parserfromGOOD.utils.loggerimportload_loggerargs=args_parser()config=config_summoner(args)load_logger(config)LoaderfromGOOD.kernel.pipelineimportinitialize_model_datasetfromGOOD.ood_algorithms.ood_managerimportload_ood_algmodel,loader=initialize_model_dataset(config)ood_algorithm=load_ood_alg(config.ood.ood_alg,config)Train/test pipelinefromGOOD.kernel.pipelineimportload_taskload_task(config.task,model,loader,ood_algorithm,config)Please refer toTutorialfor more details.ReproducibilityFor reproducibility, we provide full configurations used to obtain leaderboard results inconfigs/GOOD_configs.We further provide two tests: dataset regeneration test and test result check.Dataset regeneration testThis test regenerates all datasets again and compares them with the datasets used in the original training process locates.
Test details can be found attest_regenerate_datasets.py.
For a quick review, we provide afull regeneration test report.Leaderboard results testThis test loadsall checkpoints in round 1and
compares their results with saved ones. Test details can be found attest_reproduce_round1.py.
For a quick review, we also post ourfull round1 reproduce report.These reports are inhtmlformat. Please download them and open them in your browser.: )Training plots:The training plots for all algorithms in round 1 can be foundHERE.Sampled testsIn order to keep the validity of our code all the time, we link our project with circleci service and provide several
sampled tests to go through (because of the limitation of computational resources in CI platforms).DiscussionPlease submitnew issuesor starta new discussionfor any technical or other questions.ContactPlease feel free to contactShurui Gui,Xiner Li, orShuiwang Ji!