Package: grb
Homepage|Datasets|Leaderboard|DocumentationGraph Robustness Benchmark (GRB)providesscalable,general,unified, andreproducibleevaluation on the adversarial robustness of graph machine learning, especially Graph Neural Networks (GNNs). GRB haselaborated datasets,unified evaluation pipeline,reproducible leaderboards, andmodular coding framework, which facilitates a fair comparison among various attacks & defenses on GNNs and promotes future research in this field.Get StartedInstallationInstall grb viapip:pipinstallgrbInstall grb viagit:gitclonegit@github.com:THUDM/grb.gitcdgrb
pipinstall-e.PreparationGRB provides all necessary components to ensure the reproducibility of evaluation results.
Get datasets fromlinkor download them by running the following script:cd./scripts
shdownload_dataset.shGet attack results (adversarial adjacency matrix and features) fromlinkor download them by running the following script:shdownload_attack_results.shGet saved models (model weights) fromlinkor download them by running the following script:shdownload_saved_models.shUsage of GRB ModulesTraining a GNN modelAn example of training Graph Convolutional Network (GCN) ongrb-coradataset.importtorch# pytorch backendfromgrb.datasetimportDatasetfromgrb.model.torchimportGCNfromgrb.utils.trainerimportTrainer# Load datadataset=Dataset(name='grb-cora',mode='easy',feat_norm='arctan')# Build modelmodel=GCN(in_features=dataset.num_features,out_features=dataset.num_classes,hidden_features=[64,64])# Trainingadam=torch.optim.Adam(model.parameters(),lr=0.01)trainer=Trainer(dataset=dataset,optimizer=adam,loss=torch.nn.functional.nll_loss)trainer.train(model=model,n_epoch=200,dropout=0.5,train_mode='inductive')Adversarial attackAn example of applying Topological Defective Graph Injection Attack (TDGIA) on trained GCN model.fromgrb.attack.tdgiaimportTDGIA# Attack configurationtdgia=TDGIA(lr=0.01,n_epoch=10,n_inject_max=20,n_edge_max=20,feat_lim_min=-0.9,feat_lim_max=0.9,sequential_step=0.2)# Apply attackrst=tdgia.attack(model=model,adj=dataset.adj,features=dataset.features,target_mask=dataset.test_mask)# Get modified adj and featuresadj_attack,features_attack=rstGRB EvaluationEvaluation scenarioGRB provides a unified evaluation scenario for fair comparisons between attacks and defenses. The scenario isBlack-box,Evasion,Inductive,Injection. Take the case of a citation-graph classification system for example. The platform collects labeled data from previous papers and trains a GNN model. When a batch of new papers are submitted, it updates the graph and uses the trained model to predict labels for them.Black-box: Both the attacker and the defender have no knowledge about the applied methods each other uses.Evasion: GNNs are already trained in trusted data (e.g. authenticated users), which are untouched by the attackers but might have natural noises. Thus, attacks will only happen during the inference phase.Inductive: GNNs are used to classify unseen data (e.g. new users), i.e. validation or test data are unseen during training, which requires GNNs to generalize to out of distribution data.Injection: The attackers can only inject new nodes but not modify the target nodes directly. Since it is usually hard to hack into users' accounts and modify their profiles. However, it is easier to create fake accounts and connect them to existing users.GRB LeaderboardGRB maintainsleaderboardsthat permits a fair comparision across various attacks and defenses. To ensure the reproducibility, we provide all necessary information including datasets, attack results, saved models, etc. Besides, all results on the leaderboards can be easily reproduced by running the following scripts (e.g.leaderboard forgrb-coradataset):shrun_leaderboard_pipeline.sh-dgrb-cora-g0-s./leaderboard-n0Usage:run_leaderboard_pipeline.sh[-d<string>][-g<int>][-s<string>][-n<int>]Pipelineforreproducingleaderboardonthechosendataset.-hDisplayhelpmessage.-dChooseadataset.-sSetadirectorytosaveleaderboardfiles.-nChoosethenumberofanattackfrom0to9.-gChooseaGPUdevice.-1forCPU.SubmissionWe welcome researchers to submit new methods including attacks, defenses, or new GNNS to enrich the GRB leaderboard. For future submissions, one should follow theGRB Evaluation Rulesand respect the reproducibility.Please submit your methods via the google formGRB submission. Our team will verify the result within a week.Requirementsscipy==1.5.2numpy==1.19.1torch==1.8.0networkx==2.5pandas~=1.2.3cogdl~=0.3.0.post1scikit-learn~=0.24.1ContactIn case of any problem, please contact us via email:cogdl.grbteam@gmail.comorqinkai.zheng1028@gmail.com. We also welcome researchers to join ourGoogle Groupfor further discussion on the adversarial robustness of graph machine learning.