Package: f3rm
Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation🌐 Project Website|📝 Paper|🎥 VideoF3RM:Feature Fields for Robotic ManipulationWe distill features from 2D foundation models into 3D feature fields, and enable few-shot language-guided manipulation
that generalizes across object poses, shapes, appearances and categories.William Shen*1,Ge Yang*1,2,Alan Yu1,Jansen Wong1,Leslie Kaelbling1,Phillip Isola11MIT CSAIL,2Institute of AI and Fundamental Interactions (IAIFI)*Indicates equal contributionCoRL 2023 (Oral)Table of ContentsInstallationUsageDownloading Example DatasetsTraining a Feature FieldUsing our Custom ViewerUsing the Nerfstudio ViewerExtracting CLIP and DINO FeaturesLanguage-Guided Pose OptimizationTroubleshootingCitationCodeWe provide the official implementation of F3RM for:Training Feature Fields6-DOF Pose Optimization for Open-Text Language-Guided ManipulationF3RM is built on top ofNerfstudiofollowing theirguide for adding new methods. For a summary
of the codebase structure, seeassets/code_structure.md.InstallationNote:this repo requires an NVIDIA GPU with CUDA 11.7+ for NeRF and feature field distillation.1. Setup conda environment# We recommend that you use conda to manage your environmentcondacreate-nf3rmpython=3.8
condaactivatef3rm2. Install Nerfstudio dependencies# Install torch per instructions here: https://pytorch.org/get-started/locally/# Choose the CUDA version that your GPU supports. We will use CUDA 11.8pipinstalltorch==2.0.1+cu118torchvision==0.15.2+cu118--extra-index-urlhttps://download.pytorch.org/whl/cu118# Install CUDA toolkit, you can skip this if you already have CUDA 11.8 installed# You can check your existing CUDA installation with `nvcc --version`condainstall-c"nvidia/label/cuda-11.8.0"cuda-toolkitexportCUDA_HOME=$CONDA_PREFIX# Install tiny-cuda-nn, this will take a few minutespipinstallninjagit+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch3. Clone and install F3RMgitclonehttps://github.com/f3rm/f3rm.gitcdf3rm# Install F3RM and its dependenciespipinstall-e.# Install command-line completions for nerfstudions-install-cli# Test your installation, check that 'f3rm' is a valid methodns-train--helpNote:if you have a previous installation of Nerfstudio, make sure it does not conflict with the new installation
in thef3rmconda environment. Runwhich -a ns-trainand check that the first entry points to$CONDA_PREFIX/bin/ns-train. If it doesn't, then you may need to deactivate all conda environments and only activate
thef3rmenvironment.4. (Optional) Install dependencies for robot manipulation codeMake sure your conda environment is activated before running the following commands.# Install robot dependenciespipinstall-e".[robot]"# Install PyTorch3D, we recommend you build from source which may take a few minutes# Alternatively, check: https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.mdpipinstall"git+https://github.com/facebookresearch/pytorch3d.git@stable"# Install the latest version of viser, temporary workaround as current nerfstudio release uses older versionpipinstallviser==0.17.0# Test your installation. If you see a help message, everything should be workingf3rm-optimize--helpUsageDownloading Example DatasetsWe provide example datasets of tabletop and room-scale environments which you can download using
thef3rm-download-datacommand. By default, the script will download all the datasets (requires ~350MB disk space)
into thedatasets/f3rmdirectory relative to your current directory.Runf3rm-download-data -hto see how to download specific datasets or set your own save directory. We provide a short
description and preview of each dataset inassets/datasets.md.Training a Feature FieldWe provide the functionality to train a NeRF and distill features in parallel. The default features we distill are CLIP
features. You can distill DINO features instead by adding--pipeline.datamanager.feature-type DINO.ns-trainf3rm--data<data_folder>You can try F3RM with the example datasets which you can download following theinstructions here(try outf3rm/panda/scene_001). Alternatively, you can prepare your
own datasets following the instructions in theNerfstudio documentation.Note that while we focused on tabletop environments in the paper, F3RM can be scaled up to much larger environments. Try
training feature fields on the examplerooms datasets.You do not need to run the training to completion. We save a checkpoint every 2000 steps by default. To see all the
options available for training, runns-train f3rm -h.Resuming Training from a CheckpointUse the--load-dirflag to resume training from a checkpoint. Nerfstudio writes the checkpoint files to theoutputs/directory relative to your current directory.ns-trainf3rm--data<data_folder>--load-dir{outputs/.../nerfstudio_models}Checkout theNerfstudio documentationfor more details
on functionality.Using our Custom ViewerOur custom web viewer is coming soon! Keep an eye on out for updates.Using the Nerfstudio ViewerOnce you have started training the feature field withns-train, Nerfstudio will print a URL to the viewer in the
terminal (the URL will start withhttps://viewer.nerf.studio). You can open this URL to open the Nerfstudio viewer in
your browser to visualize training progress and the feature field. Alternatively, to visualize a trained model you can
runns-viewer --load-config {outputs/.../config.yml}.Note that if you are using a remote server, you will need to forward the port to your local machine
(instructions).
The default port used by Nerfstudio is 7007, but check the viewer URL to make sure. For a general guide on how to use
the Nerfstudio viewer, check out
theirdocumentation.Visualizing the Feature Field PCATo visualize the PCA of the features, selectfeature_pcain theRender Options -> Output Renderdropdown box.
Note that the initial PCA projection matrix is computed based on the features rendered at your current viewpoint.To recompute the PCA projection based on your current viewpoint, click the "Refresh PCA Projection" button underTrainer/pipeline/modelnear the bottom of the controls.Language Interaction with CLIP Feature FieldsIf you are distilling CLIP features (the default feature type), then you will see the following additional controls
underTrainer/pipeline/modelnear the bottom of the controls panel. You can enter positive and negative text queries
(separated by,commas), which will compute similarity heatmaps. You will need to click out of the text box or press
the enter key to submit the query.To visualize these heatmaps, selectsimilarityin theRender Options -> Output Renderdropdown box. It may take a
few seconds for this option to show up on the first query, as we load CLIP lazily.We show the similarity heatmap over thef3rm/panda/scene_001dataset for the "Baymax" query, with the negatives in the
controls above (you can download this dataset using thef3rm-download-data pandacommand). Try playing around with
different language queries and see what results you get!Note:if multiple positive queries are specified, we average their CLIP embeddings before computing the pair-wise
softmax described in Section 3.3 of thepaper. The default temperature of 0.1 works
well. If no negative queries are specified, then we show the cosine similarity between the positive query and the
feature field.Extracting CLIP and DINO FeaturesWe provide scripts to demonstrate how to extract CLIP and DINO features from their respective vision models. You can
use these features for your own NeRF pipeline or for other downstream applications.Runpython f3rm/scripts/demo_extract_features.pyfor a general demo on how to extract CLIP and DINO features.
This will create a plot showing the PCA of the CLIP and DINO features. The plot is saved
todemo_extract_features.png.For details on how to extract CLIP features and compare the extracted features with CLIP text embeddings, runpython f3rm/scripts/demo_clip_features.py. This script will create a plot showing the similarity heatmaps for a
given
text query, and will save a plot todemo_clip_features-{text_query}.png.Language-Guided Pose OptimizationFor details on how to run the 6-DOF pose optimization code for language-guided manipulation, please
checkf3rm_robot/README.md. A detailed tutorial is provided.TroubleshootingLanguage queries are not working in Nerfstudio viewerThe Nerfstudio viewer can sometimes fail to register the input you type into the text boxes if you are use the same
browser tab for different training runs. This meansfeature_pcaandsimilaritymay not appear in the Render Options.
To fix this issue, try closing the tab with the viewer in your browser and opening it again. If this doesn't work,
please open an issue in this repository.Running out of GPU memoryThis codebase was tested on a RTX3090 with 24GB of GPU memory. We observe a peak memory usage of ~6GB when training a
CLIP feature fieldwithoutusing the viewer. When the viewer is used in conjunction with training, the peak memory
usage is ~12GB.If you are running out of memory when using the Nerfstudio viewer, try:Decreasing the number of rays per batch when rendering by using the--pipeline.model.eval-num-rays-per-chunk 8192flag when runningns-train.The default rays per chunk is 16384, which uses ~12GB of memory (atMax Res = 512).8192 rays per chunk uses ~10GB memory. Decreasing this number will further reduce memory usage, at the cost of
slower rendering.Decrease the rendering resolution inMax Resunder Render Options in the Nerfstudio viewer.If you are running out of memory during any other stages, please open a GitHub issue and we will try to help.AcknowledgementsWe thank the authors of the following projects for making their code open source:NerfstudioLERFCLIPDINOanddino-vit-featuresViserCitationIf you find our work useful, please consider citing:@inproceedings{shen2023F3RM,
    title={Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation},
    author={Shen, William and Yang, Ge and Yu, Alan and Wong, Jansen and Kaelbling, Leslie Pack and Isola, Phillip},
    booktitle={7th Annual Conference on Robot Learning},
    year={2023},
    url={https://openreview.net/forum?id=Rb0nGIt_kh5}
}