Package: nerfstudio
A collaboration friendly studio for NeRFsQuickstartLearn moreSupported FeaturesAboutIt’s as simple as plug and play with nerfstudio!Nerfstudio provides a simple API that allows for a simplified end-to-end process of creating, training, and testing NeRFs.
The library supports amore interpretable implementation of NeRFs by modularizing each component.With more modular NeRFs, we hope to create a more user-friendly experience in exploring the technology.This is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions.
Nerfstudio initially launched as an opensource project by Berkeley students inKAIR labatBerkeley AI Research (BAIR)in October 2022 as a part of a research project (paper). It is currently developed by Berkeley students and community contributors.We are committed to providing learning resources to help you understand the basics of (if you're just getting started), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more!Have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset?We welcomecontributions!Please do not hesitate to reach out to the nerfstudio team with any questions viaDiscord.Have feedback? We'd love for you to fill out ourNerfstudio Feedback Formif you want to let us know who you are, why you are interested in Nerfstudio, or provide any feedback!We hope nerfstudio enables you to build faster :hammer: learn together :books: and contribute to our NeRF community :sparkling_heart:.SponsorsSponsors of this work includesLuma AIand theBAIR commons.QuickstartThe quickstart will help you get started with the default vanilla NeRF trained on the classic Blender Lego scene.
For more complex changes (e.g., running with your own data/setting up a new NeRF graph), please refer to ourreferences.1. Installation: Setup the environmentPrerequisitesYou must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.8 of CUDA. You can find more information about installing CUDAhereCreate environmentNerfstudio requirespython >= 3.8. We recommend using conda to manage dependencies. Make sure to installCondabefore proceeding.condacreate--namenerfstudio-ypython=3.8
condaactivatenerfstudio
pipinstall--upgradepipDependenciesInstall PyTorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) andtiny-cuda-nn.cuda-toolkitis required for buildingtiny-cuda-nn.For CUDA 11.8:pipinstalltorch==2.1.2+cu118torchvision==0.16.2+cu118--extra-index-urlhttps://download.pytorch.org/whl/cu118

condainstall-c"nvidia/label/cuda-11.8.0"cuda-toolkit
pipinstallninjagit+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torchSeeDependenciesin the Installation documentation for more.Installing nerfstudioEasy option:pipinstallnerfstudioORif you want the latest and greatest:gitclonehttps://github.com/nerfstudio-project/nerfstudio.gitcdnerfstudio
pipinstall--upgradepipsetuptools
pipinstall-e.ORif you want to skip all installation steps and directly start using nerfstudio, use the docker image:SeeInstallation-Use docker image.2. Training your first model!The following will train anerfactomodel, our recommended model for real world scenes.# Download some test data:ns-download-datanerfstudio--capture-name=poster# Train modelns-trainnerfacto--datadata/nerfstudio/posterIf everything works, you should see training progress like the following:Navigating to the link at the end of the terminal will load the webviewer. If you are running on a remote machine, you will need to port forward the websocket port (defaults to 7007).Resume from checkpoint / visualize existing runIt is possible to load a pretrained model by runningns-trainnerfacto--datadata/nerfstudio/poster--load-dir{outputs/.../nerfstudio_models}Visualize existing runGiven a pretrained model checkpoint, you can start the viewer by runningns-viewer--load-config{outputs/.../config.yml}3. Exporting ResultsOnce you have a NeRF model you can either render out a video or export a point cloud.Render VideoFirst we must create a path for the camera to follow. This can be done in the viewer under the "RENDER" tab. Orient your 3D view to the location where you wish the video to start, then press "ADD CAMERA". This will set the first camera key frame. Continue to new viewpoints adding additional cameras to create the camera path. We provide other parameters to further refine your camera path. Once satisfied, press "RENDER" which will display a modal that contains the command needed to render the video. Kill the training job (or create a new terminal if you have lots of compute) and run the command to generate the video.Other video export options are available, learn more by runningns-render--helpGenerate Point CloudWhile NeRF models are not designed to generate point clouds, it is still possible. Navigate to the "EXPORT" tab in the 3D viewer and select "POINT CLOUD". If the crop option is selected, everything in the yellow square will be exported into a point cloud. Modify the settings as desired then run the command at the bottom of the panel in your command line.Alternatively you can use the CLI without the viewer. Learn about the export options by runningns-exportpointcloud--help4. Using Custom DataUsing an existing dataset is great, but likely you want to use your own data! We support various methods for using your own data. Before it can be used in nerfstudio, the camera location and orientations must be determined and then converted into our format usingns-process-data. We rely on external tools for this, instructions and information can be found in the documentation.DataCapture DeviceRequirementsns-process-dataSpeed📷ImagesAnyCOLMAP🐢📹VideoAnyCOLMAP🐢🌎360 DataAnyCOLMAP🐢📱PolycamIOS with LiDARPolycam App🐇📱KIRI EngineIOS or AndroidKIRI Engine App🐇📱Record3DIOS with LiDARRecord3D app🐇📱Spectacular AIIOS, OAK,othersApp/sai-cli🐇🖥MetashapeAnyMetashape🐇🖥RealityCaptureAnyRealityCapture🐇🖥ODMAnyODM🐇👓AriaAria glassesProject Aria🐇🛠CustomAnyCamera Poses🐇5. Advanced OptionsTraining models other than nerfactoWe provide other models than nerfacto, for example if you want to train the original nerf model, use the following commandns-trainvanilla-nerf--dataDATA_PATHFor a full list of included models runns-train --help.Modify ConfigurationEach model contains many parameters that can be changed, too many to list here. Use the--helpcommand to see the full list of configuration options.ns-trainnerfacto--helpTensorboard / WandB / ViewerWe support four different methods to track training progress, using the viewertensorboard,Weights and Biases, and ,Comet. You can specify which visualizer to use by appending--vis {viewer, tensorboard, wandb, comet viewer+wandb, viewer+tensorboard, viewer+comet}to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps. The viewer only works for methods that are fast (ie. nerfacto, instant-ngp), for slower methods like NeRF, use the other loggers.Learn MoreAnd that's it for getting started with the basics of nerfstudio.If you're interested in learning more on how to create your own pipelines, develop with the viewer, run benchmarks, and more, please check out some of the quicklinks below or visit ourdocumentationdirectly.SectionDescriptionDocumentationFull API documentation and tutorialsViewerHome page for our web viewer🎒EducationalModel DescriptionsDescription of all the models supported by nerfstudio and explanations of component parts.Component DescriptionsInteractive notebooks that explain notable/commonly used modules in various models.🏃TutorialsGetting StartedA more in-depth guide on how to get started with nerfstudio from installation to contributing.Using the ViewerA quick demo video on how to navigate the viewer.Using Record3DDemo video on how to run nerfstudio without using COLMAP.💻For DevelopersCreating pipelinesLearn how to easily build new neural rendering pipelines by using and/or implementing new modules.Creating datasetsHave a new dataset? Learn how to run it with nerfstudio.ContributingWalk-through for how you can start contributing now.💖CommunityDiscordJoin our community to discuss more. We would love to hear from you!TwitterFollow us on Twitter @nerfstudioteam to see cool updates and announcementsFeedback FormWe welcome any feedback! This is our chance to learn what you all are using Nerfstudio for.Supported FeaturesWe provide the following support structures to make life easier for getting started with NeRFs.If you are looking for a feature that is not currently supported, please do not hesitate to contact the Nerfstudio Team onDiscord!:mag_right: Web-based visualizer that allows you to:Visualize training in real-time + interact with the sceneCreate and render out scenes with custom camera trajectoriesView different output typesAnd more!:pencil2: Support for multiple logging interfaces (Tensorboard, Wandb), code profiling, and other built-in debugging tools:chart_with_upwards_trend: Easy-to-use benchmarking scripts on the Blender dataset:iphone: Full pipeline support (w/ Colmap, Polycam, or Record3D) for going from a video on your phone to a full 3D render.Built OnEasy-to-use config systemDeveloped byBrent YiLibrary for accelerating NeRF rendersDeveloped byRuilong LiCitationYou can find a paper writeup of the framework onarXiv.If you use this library or find the documentation useful for your research, please consider citing:@inproceedings{nerfstudio,
	title        = {Nerfstudio: A Modular Framework for Neural Radiance Field Development},
	author       = {
		Tancik, Matthew and Weber, Ethan and Ng, Evonne and Li, Ruilong and Yi, Brent
		and Kerr, Justin and Wang, Terrance and Kristoffersen, Alexander and Austin,
		Jake and Salahi, Kamyar and Ahuja, Abhik and McAllister, David and Kanazawa,
		Angjoo
	},
	year         = 2023,
	booktitle    = {ACM SIGGRAPH 2023 Conference Proceedings},
	series       = {SIGGRAPH '23}
}Contributors