Package: torcheval
TorchEvalThis library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.Installing TorchEvalRequires Python >= 3.8 and PyTorch >= 1.11From pip:pipinstalltorchevalFor nighly build versionpipinstall--pretorcheval-nightlyFrom source:gitclonehttps://github.com/pytorch/torchevalcdtorcheval
pipinstall-rrequirements.txt
pythonsetup.pyinstallQuick StartTake a look at thequickstart notebook, or fork it onColab.There are more examples in theexamplesdirectory:cdtorcheval
pythonexamples/simple_example.pyDocumentationDocumentation can be found at atpytorch.org/torchevalUsing TorchEvalTorchEval can be run on CPU, GPU, and in a multi-process or multi-GPU setting. Metrics are provided in two interfaces, functional and class based. The functional interfaces can be found intorcheval.metrics.functionaland are useful when your program runs in a single process setting. To use multi-process or multi-gpu configurations, the class-based interfaces, found intorcheval.metricsprovide a much simpler experience. The class based interfaces also allow you to defer some of the computation of the metric by callingupdate()multiple times beforecompute(). This can be advantageous even in a single process setting due to saved computation overhead.Single ProcessFor use in a single process program, the simplest use case utilizes a functional metric. We simply import the metric function and feed in our outputs and targets. The example below shows a minimal PyTorch training loop that evaluates the multiclass accuracy of every fourth batch of data.Functional Version (immediate computation of metric)importtorchfromtorcheval.metrics.functionalimportmulticlass_accuracyNUM_BATCHES=16BATCH_SIZE=8INPUT_SIZE=10NUM_CLASSES=6eval_frequency=4model=torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE,NUM_CLASSES),torch.nn.ReLU())optim=torch.optim.Adagrad(model.parameters(),lr=0.001)loss_fn=torch.nn.CrossEntropyLoss()metric_history=[]forbatchinrange(NUM_BATCHES):input=torch.rand(size=(BATCH_SIZE,INPUT_SIZE))target=torch.randint(size=(BATCH_SIZE,),high=NUM_CLASSES)outputs=model(input)loss=loss_fn(outputs,target)optim.zero_grad()loss.backward()optim.step()# metric only computed every 4 batches,# data from previous three batches is lostif(batch+1)%eval_frequency==0:metric_history.append(multiclass_accuracy(outputs,target))Single Process with Deferred ComputationClass Version (enables deferred computation of metric)importtorchfromtorcheval.metricsimportMulticlassAccuracyNUM_BATCHES=16BATCH_SIZE=8INPUT_SIZE=10NUM_CLASSES=6eval_frequency=4model=torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE,NUM_CLASSES),torch.nn.ReLU())optim=torch.optim.Adagrad(model.parameters(),lr=0.001)loss_fn=torch.nn.CrossEntropyLoss()metric=MulticlassAccuracy()metric_history=[]forbatchinrange(NUM_BATCHES):input=torch.rand(size=(BATCH_SIZE,INPUT_SIZE))target=torch.randint(size=(BATCH_SIZE,),high=NUM_CLASSES)outputs=model(input)loss=loss_fn(outputs,target)optim.zero_grad()loss.backward()optim.step()# metric only computed every 4 batches,# data from previous three batches is includedmetric.update(input,target)if(batch+1)%eval_frequency==0:metric_history.append(metric.compute())# remove old data so that the next call# to compute is only based off next 4 batchesmetric.reset()Multi-Process or Multi-GPUFor usage on multiple devices a minimal example is given below. In the normaltorch.distributedparadigm, each device is allocated its own process gets a unique numerical ID called a "global rank", counting up from 0.Class Version (enables deferred computation and multi-processing)importtorchfromtorcheval.metrics.toolkitimportsync_and_computefromtorcheval.metricsimportMulticlassAccuracy# Using torch.distributedlocal_rank=int(os.environ["LOCAL_RANK"])#rank on local machine, i.e. unique ID within a machineglobal_rank=int(os.environ["RANK"])#rank in global pool, i.e. unique ID within the entire process groupworld_size=int(os.environ["WORLD_SIZE"])#total number of processes or "ranks" in the entire process groupdevice=torch.device(f"cuda:{local_rank}"iftorch.cuda.is_available()andtorch.cuda.device_count()>=world_sizeelse"cpu")metric=MulticlassAccuracy(device=device)num_epochs,num_batches=4,8forepochinrange(num_epochs):foriinrange(num_batches):input=torch.randint(high=5,size=(10,),device=device)target=torch.randint(high=5,size=(10,),device=device)# Add data to metric locallymetric.update(input,target)# metric.compute() will returns metric value from# all seen data on the local process since last reset()local_compute_result=metric.compute()# sync_and_compute(metric) syncs metric data across all ranks and computes the metric valueglobal_compute_result=sync_and_compute(metric)ifglobal_rank==0:print(global_compute_result)# metric.reset() clears the data on each process so that subsequent# calls to compute() only act on new datametric.reset()See theexample directoryfor more examples.ContributingWe welcome PRs! See theCONTRIBUTINGfile.LicenseTorchEval is BSD licensed, as found in theLICENSEfile.