Package: torchbench
Easily benchmark PyTorch models on selected tasks and datasets.InstallationRequires Python 3.6+.pipinstalltorchbenchUsageThis library can be used together with thesotabenchwebsite, or standalone. Below we'll describe the usage with the sotabench website.Steps to benchmark your model on the sotabench website:Create asotabench.pyin the root of your repository. Below you can see an examplesotabench.pyfile added to thetorchvisionrepository to test one of its constituent models:fromtorchbench.image_classificationimportImageNetfromtorchvision.models.resnetimportresnext101_32x8dimporttorchvision.transformsastransformsimportPIL# Define the transforms need to convert ImageNet data to expected model inputnormalize=transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225])input_transform=transforms.Compose([transforms.Resize(256,PIL.Image.BICUBIC),transforms.CenterCrop(224),transforms.ToTensor(),normalize,])# Run the benchmarkImageNet.benchmark(model=resnext101_32x8d(pretrained=True),paper_model_name='ResNeXt-101-32x8d',paper_arxiv_id='1611.05431',input_transform=input_transform,batch_size=256,num_gpu=1)Run it locally on your machine to verify it works:$pythonsotabench.pyIn addition, you can check your parameters (such as model names and arxiv IDs) by running:$sbcheckAlternatively you can run the same logic within a Notebook if that is your preferred workflow.Login and connect your repository tosotabench. After you connect your repository the website will re-evaluate your model on every commit of themasterbranch, to ensure the model is working and results are up-to-date - including if you add additional models to the benchmark file.You can also use the library without the sotabench website, by simply omitting step 3. In that case you also don't need to put in the paper details into thebenchmark()method.BenchmarksImage Classification on ImageNetImage Classification on ImageNet benchmark is implemented in theimage_classification.ImageNetclass.Benchmarking PipelineThe model is put into evaluation mode and sent to the deviceThe ImageNet validation dataset is loaded and transformed usinginput_transformThe dataset is put into a DataLoader with optionsbatch_sizeandnum_workersThe model and dataset are passed into an evaluation function for the task, along with an optionalmodel_output_transformfunction that can transform the outputs after inferenceThe transformed output is compared to expected output and Top 1 and Top 5 accuracy are calculatedExpected Inputs/OutputsModeloutput(followingmodel.forward()and optionallymodel_output_transform) should be a 2Dtorch.Tensorcontaining the model output; first dimension should be output for each example (lengthbatch_size) and second dimension should be output for each class in ImageNet (length 1000).Object Detection on COCOObject Detection on the COCO benchmark is implemented in theobject_detection.COCOclass.Benchmarking PipelineThe model is put into evaluation mode and sent to the deviceThe COCO dataset is loaded.Withouttransforms, COCO dataset returns a tuple for an index where:The first entry is aPIL.ImageThe second entry is a labels dictionary, with keys'boxes', 'labels', 'masks', 'image_id', 'area', 'iscrowd', containing the labels.Withdefault transforms, COCO dataset returns a tuple for an index where:The first entry is atorch.tensor(representing the image)The second entry is a labels dictionary, with keys'boxes', 'labels', 'masks', 'image_id', 'area', 'iscrowd', where the data is of typetorch.tensorrather than lists.You can specify your owntransformsto transform the data so your model can process it correctly.The dataset is put into a DataLoader with optionsbatch_sizeandnum_workers, and collated usingcoco_collate_fn. Alternatively you can pass in your own collate function.The model and dataset are passed into an evaluation function for the task, along withmodel_output_transformfunction that can transform the outputs and targets after inference. The defaultmodel_output_transformfollows that used in the torchvision examples, but you can provide your ownmodel_output_transformto get your model output in the right format.The expected output is a list of dictionaries (length = batch_size), where each dictionary contains keys for'boxes', 'labels', 'scores', 'masks', and each value is of thetorch.tensortype.The (transformed) model output is then converted to a dictionary with keys as the image ids, and values as a dictionary with the predictions (boxes, labels, scores, ...)result = {tar["image_id"].item(): out for tar, out in zip(target, output)}The result is passed into a COCO Evaluation pipeline and the results (Mean Average Precision) are calculatedMore benchmarks coming soon...ContributingAll contributions welcome!