Package: nerfbaselines
NerfBaselinesNerfBaselines is a framework forevaluating and comparing existing NeRF methods. Currently, most official implementations use different dataset loaders, evaluation protocols, and metrics, which renders the comparison of methods difficult. Therefore, this project aims to provide aunified interfacefor running and evaluating methods on different datasets in a consistent way using the same metrics. But instead of reimplementing the methods,we use the official implementationsand wrap them so that they can be run easily using the same interface.Please visit theproject page to see the resultsof implemented methods on dataset benchmarks.Project Page + ResultsGetting startedStart by installing thenerfbaselinespip package on your host system.pipinstallnerfbaselinesNow you can use thenerfbaselinescli to interact with NerfBaselines.WARNING: the default installation only installs the core nerfbaselines package which does not depend on either PyTorch or JAX.
However, the LPIPS metric requires PyTorch to be installed and will be disabled otherwise. Similarly, if you install JAX and
have a GPU available, the dataloading and evaluation will be faster as some parts of the pipeline will be moved to GPU.
Therefore, we recommend installing theextraspackage by following theAdvanced installationsection.The next step is to choose the backend which will be used to install different methods. At the moment there are the following backends implemented:docker: Offers good isolation, requiresdockerto be installed and the user to have access to it (being in the docker user group).apptainer: Similar level of isolation asdocker, but does not require the user to have privileged access.conda(not recommended): Does not require docker/apptainer to be installed, but does not offer the same level of isolation and some methods require additional
dependencies to be installed. Also, some methods are not implemented for this backend because they rely on dependencies not found onconda.python(not recommended): Will run everything directly in the current environment. Everything needs to be installed in the environment for this backend to work.The backend can be set as the--backend <backend>argument or using theNERFBASELINES_BACKENDenvironment variable.Advanced installationThe LPIPS metric requires PyTorch to be installed and will be disabled otherwise. Similarly, if you install JAX and
have a GPU available, the dataloading and evaluation will be faster as some parts of the pipeline will be moved to GPU.
In this section we describe how to install the packages required for LPIPS and accelerated dataloading.
We recommend this as the default installation (unless there is a reason for not installing PyTorch or JAX).
Select one of the following configurations:CPU-only installpipinstalltorchtorchvisiontorchaudio--index-urlhttps://download.pytorch.org/whl/cpu
pipinstalljax[cpu]pipinstall'nerfbaselines[extras]'CUDA 11.8 installpipinstalltorchtorchvisiontorchaudio--index-urlhttps://download.pytorch.org/whl/cu118
pipinstalljax[cuda11_pip]pipinstall'nerfbaselines[extras]'CUDA 12.1 installpipinstalltorchtorchvisiontorchaudio
pipinstalljax[cuda12_pip]pipinstall'nerfbaselines[extras]'Downloading dataFor some datasets, e.g. Mip-NeRF 360 or NerfStudio, the datasets can be downloaded automatically. You can specify the argument--data external://dataset/sceneduring training
or download the dataset beforehand by runningnerfbaselines download-dataset dataset/scene.
Examples:# Downloads the garden scene to the cache folder.nerfbaselinesdownload-datasetmipnerf360/garden# Downloads all nerfstudio scenes to the cache folder.nerfbaselinesdownload-datasetnerfstudio# Downloads kithen scene to folder kitchennerfbaselinesdownload-datasetmipnerf360/kitchen-okitchenTrainingTo start the training, use thenerfbaselines train --method <method> --data <data>command. Use--helpargument to learn about all implemented methods and supported features.RenderingThenerfbaselines render --checkpoint <checkpoint>command can be used to render images from a trained checkpoint. Again, use--helpto learn about the arguments.Interactive viewerGiven a trained checkpoint, the interactive viewer can be launched as follows:nerfbaselinesviewer--checkpoint<checkpoin>--data<dataset>Even though the argument--data <dataset>is optional, it is recommended, as the camera poses
are used to perform gravity alignment and rescaling for a better viewing experience.
Again, you can use the--backend <backend>flag orNS_BACKEND=<backend>environment variable to change the backend.ResultsIn this section, we present results of implemented methods on standard benchmark datasets. For detailed results, visit the project page:https://jkulhanek.com/nerfbaselinesMip-NeRF 360Mip-NeRF 360 is a collection of four indoor and five outdoor object-centric scenes. The camera trajectory is an orbit around the object with fixed elevation and radius. The test set takes each n-th frame of the trajectory as test views.
Detailed results are available on the project page:https://jkulhanek.com/nerfbaselines/mipnerf360MethodPSNRSSIMLPIPSTimeGPU mem.Zip-NeRF28.5160.8280.1385h 30m 49s26.2 GBMip-NeRF 36027.6700.7920.1967h 29m 42s127.0 GBGaussian Splatting27.4390.8140.18022m 45s11.1 GBNerfStudio26.3480.7300.25719m 50s3.8 GBTetra-NeRF25.4680.6700.35217h 32m 35s13.4 GBInstant NGP24.8990.6730.3554m 16s5.6 GBBlenderBlender (nerf-synthetic) is a synthetic dataset used to benchmark NeRF methods. It consists of 8 scenes of an object placed on a white background. Cameras are placed on a semi-sphere around the object.
Detailed results are available on the project page:https://jkulhanek.com/nerfbaselines/blenderMethodPSNRSSIMLPIPSTimeGPU mem.Zip-NeRF33.6700.9730.0205h 21m 57s26.2 GBGaussian Splatting33.3080.9690.0236m 6s3.1 GBInstant NGP32.1910.9590.0312m 23s2.6 GBTetra-NeRF31.9510.9570.0316h 53m 20s29.6 GBMip-NeRF 36030.3450.9510.0383h 29m 39s114.8 GBNerfStudio29.1910.9410.0499m 38s3.6 GBNerfstudioNerfstudio Dataset includes 10 in-the-wild captures obtained using either a mobile phone or a mirror-less camera with a fisheye lens. We processed the data using either COLMAP or the Polycam app to obtain camera poses and intrinsic parameters.
Detailed results are available on the project page:https://jkulhanek.com/nerfbaselines/nerfstudioMethodPSNRSSIMLPIPSTimeGPU mem.Zip-NeRF24.8150.7980.1785h 21m 41s26.2 GBInstant NGP20.6530.6010.4524m 33s4.2 GBNerfStudio20.0640.6170.35313m 30s4.8 GBGaussian Splatting-----Implementation statusMethods:NerfactoInstant-NGPGaussian SplattingMip-SplattingTetra-NeRFMip-NeRF 360Zip-NeRFCamPTensoRF (Blender, LLFF datasets)Mip-NeRFNeRFDatasets/features:Mip-NeRF 360 datasetBlender datasetany COLMAP datasetany NerfStudio datasetautomatic dataset downloadinteractive viewerundistorting images for methods that do not support complex camera models (Gaussian Splatting)logging to tensorboard, wandbLLFF dataset (only supported in TensoRF now)Tanks and TemplesHDR images supportRAW images supporthandling large datasetsloading/creating camera trajectories in the interactive viewerContributingContributions are very much welcome. Please open a PR with a dataset/method/feature that you want to contribute. The goal of this project is to slowly expand by implementing more and more methods.LicenseThis project is licensed under the MIT license.ThanksA big thanks to the authors of all implemented methods for the great work they have done.