Package: beyonddefer
Quick StartIn this package, we provide the code to reproduce the experiments in the paper "Is Learn to Defer Enough? Optimal Predictors that Incorporate Human Decisions".Quick InstallationUsing pip (Recommended)This package could be installed easily using pip with the following commands. It is recommended that you create a virtual environment and install everything there:# Creating a virtual environment (optional)python3-mvenvbeyonddefer-venvsourcebeyonddefer-venv/bin/activate# Adding the PackagepipinstallbeyonddeferCloning RepositoryAnother way to use the package is to clone this repository and then add the package's path to the python path (usingPYTHONPATHenvironmental variable):# cloning the repositiry and installing requirementsgitclone<repo-url>cdBeyondDefer
pipinstall-rrequirements.txt# adding the package to python pathexportPYTHONPATH=$PWD# run your python script which includes beyonddeferUsage ExampleIn this section, we go through an example of usingbeyonddeferpackage and writing a simple python code. In this example, we simply train theAdditional Beyond Defermethod withWideResNetmodel with the syntheticCIFAR10Kdataset fork = 5. Then we test for the results and print them out.Note:Before running any experiments, you should first create thedata,models, andResultsdirectories in the directory of your python script:mkdirdatamodelsResultsHere is code for the introduced example:# import beyonddefer itself for some initializationsimportbeyonddefer# import the required modulesfrombeyonddefer.human_ai_defer.datasetsdefer.cifar_synthimport\CifarSynthDatasetfrombeyonddefer.MyMethod.additional_deferimportAdditionalBeyondfrombeyonddefer.MyNet.call_netimportnetworks,optimizer_schedulerfrombeyonddefer.metrics.metricsimportcompute_additional_defer_metricsimporttorchimportloggingdevice=torch.device("cuda:0"iftorch.cuda.is_available()else"cpu")# Setting log level to INFO to see training logslogging.getLogger().setLevel(logging.INFO)# Adding the datasetk=5# expert kdataset=CifarSynthDataset(k,False,batch_size=512)dataset_name="cifar_synth"epochs=150num_classes=10# Adding the networks and modelclassifier,human,meta=networks(dataset_name,"Additional",device)AB=AdditionalBeyond(10,classifier,human,meta,device)# Optimizer and scheduleroptimizer,scheduler=optimizer_scheduler()# Training the modelAB.fit(dataset.data_train_loader,dataset.data_val_loader,dataset.data_test_loader,num_classes,epochs,optimizer,lr=1e-3,scheduler=scheduler,verbose=False)# Generating test resultstest_data=AB.test(dataset.data_test_loader,num_classes)# Extracting useful information from the raw test datares_AB=compute_additional_defer_metrics(test_data)print(res_AB)ExperimentsThe main set of experiments shown in the paper are inExperiments/(Section 7). In fact,inExperiments/acc_vs_c.pythe code corresponding to the accuracy of methods based on additional defer cost is provided,inExperiments/CIFAR10K.pythe code corresponding to the CIFAR10K experiment for different $K$
is provided,inExperiments/cost_sensitive_cov_acc.pythe code of accuracy vs. coverage for cost-sensitive methods is provided,inExperiments/SampleComp.pythe role of sample complexity is studied, andinExperiments/no_loss_cov_acc.pythe code of accuracy vs. coverage for methods for 0-1 losses is provided.