NDF

Code for Depth Estimators Are Implicit Neural Fields for 3D Scene Geometry Inpainting and Reconstruction.

Neural Depth Field (NDF) unifies depth inpainting and geometry reconstruction in a common training objective, supporting accurate and consistent scene depth completion.

Structure

├── ndf/
│   ├── models/
│   ├── adaptation/
│   ├── geometry/
│   ├── reconstruction/
│   └── height/
├── train.py
├── infer.py
├── evaluate.py
├── configs/
├── data/
├── checkpoints/
├── scripts/
│   ├── data/
│   ├── train/
│   ├── infer/
│   ├── inpaint/
│   └── evaluate/
└── requirements.txt

Preparation

python -m venv .venv
source .venv/bin/activate
git clone https://github.com/zju3dv/InfiniDepth.git dependencies/InfiniDepth
git clone https://github.com/facebookresearch/vggt.git dependencies/VGGT
git clone https://github.com/nv-dvl/capa.git dependencies/CAPA
pip install -r requirements.txt
export PYTHONPATH="$PWD:$PWD/dependencies/InfiniDepth:$PWD/dependencies/VGGT:$PWD/dependencies/CAPA"

Data

Dataset	Download	Preparation
ScanNet	link	link
DrivingStereo	link	link
DDAD	link	link
Waymo	link	link
Birmingham	link	link

Birmingham raw point clouds: link.

ScanNet preparation:

python scripts/data/prepare_scannet.py --source-root data/raw/scannet --output-root data/scannet
python scripts/data/prepare_scannet_density.py --dataset-root data/scannet --output-root data/scannet_100 --points 100

Pretrained weights

Model	Code	Download
InfiniDepth	link	link
VGGT	link	link
CAPA	link	—

Save infinidepth.ckpt as checkpoints/pretrained/InfiniDepth/rgb.ckpt, infinidepth_depthsensor.ckpt as checkpoints/pretrained/InfiniDepth/depthsensor.ckpt, and VGGT's model.pt as checkpoints/pretrained/VGGT/model.pt.

Usage

Fitting

python train.py --config configs/scannet/ndf.json --scene scene0011_01

For CAPA+NDF, use --config configs/scannet/capa_ndf.json. Use --epochs 100 for 100 epochs and --resume to continue fitting.

Street fitting:

python scripts/train/driving/street_run.py \
  --field configs/driving/fields/drivingstereo/drivingstereo_2018-07-11-14-48-52_clip00.json \
  --epochs 10 \
  --output outputs/street

For input variations, use --variation constant_image or --variation no_spatial_depth_prompt with infer.py. For sparse-depth density, use the prepared sparse map and --points 100, 250, or 1000.

VGGT fitting:

python scripts/train/multiview/models/run_ndf.py --config configs/multiview/ndf.json

Gaussian reconstruction fitting:

python scripts/train/gaussian_splatting/run_gs.py \
  --manifest data/scannet/gs/scene0146_01/H/adaptation.json \
  --field scene0146_01 --condition H --stage warmup \
  --initialization data/scannet/gs/scene0146_01/H/initialization.npz \
  --output outputs/gs/scene0146_01/H/warmup
python scripts/train/gaussian_splatting/teacher/teacher_runner.py \
  --manifest data/scannet/gs/scene0146_01/H/adaptation.json \
  --field scene0146_01 --condition H \
  --output outputs/gs/scene0146_01/H/teacher
python scripts/train/gaussian_splatting/run_gs.py \
  --manifest data/scannet/gs/scene0146_01/H/adaptation.json \
  --field scene0146_01 --condition H --stage reconstruction \
  --warmup outputs/gs/scene0146_01/H/warmup/gs.pt \
  --teachers outputs/gs/scene0146_01/H/teacher \
  --output outputs/gs/scene0146_01/H/reconstruction

Height fitting:

python scripts/train/height/run.py --config configs/height/train.yaml

Evaluation

python evaluate.py \
  --prediction-dir outputs/predictions \
  --ground-truth-dir data/scannet/gt_heldout \
  --max-depth 10 \
  --output-dir outputs/metrics

Street evaluation:

python scripts/evaluate/driving/evaluate.py \
  --field configs/driving/fields/drivingstereo/drivingstereo_2018-07-11-14-48-52_clip00.json \
  --prediction-dir outputs/street/predictions \
  --output outputs/street/metrics.json

Inference

Use the scene parameters saved by fitting for inference.

RGB images and sparse depth maps share the same spatial resolution. Sparse depth maps use NumPy .npy format, with depth in meters and zero for unknown pixels.

python infer.py \
  --rgb data/scannet/rgb/scene0011_01_000000.png \
  --sparse-depth data/scannet/sparse/scene0011_01_000000.npy \
  --checkpoint checkpoints/pretrained/InfiniDepth/depthsensor.ckpt \
  --adapter outputs/ndf/NDF/scene0011_01/e10/adapter.pt \
  --output outputs/depth.npy

For CAPA+NDF, use --adapter outputs/ndf/CAPA+NDF/scene0011_01/e10/adapter.pt and --prompt outputs/ndf/CAPA/scene0011_01/prompt/adapter.pt.

Python API:

from pathlib import Path
from ndf.models import load_field, predict_depth

model = load_field(
    Path("checkpoints/pretrained/InfiniDepth/depthsensor.ckpt"),
    adapter=Path("outputs/ndf/NDF/scene0011_01/e10/adapter.pt"),
)
depth = predict_depth(
    model,
    rgb=Path("data/scannet/rgb/scene0011_01_000000.png"),
    sparse=Path("data/scannet/sparse/scene0011_01_000000.npy"),
)

Street depth:

python scripts/infer/street/run.py \
  --field configs/driving/fields/drivingstereo/drivingstereo_2018-07-11-14-48-52_clip00.json \
  --adapter outputs/street/adapter.pt \
  --output outputs/street

VGGT depth:

python scripts/infer/multiview/run.py \
  --scene scene0011_01 \
  --adapter outputs/multiview/scenes/scene0011_01/depth_head.pt \
  --output outputs/multiview

Gaussian reconstruction:

python scripts/infer/gaussian_splatting/run.py \
  --checkpoint outputs/gs/scene0146_01/H/reconstruction/gs.pt \
  --manifest data/scannet/gs/scene0146_01/evaluation.json \
  --output outputs/gs

Height completion:

python scripts/infer/height/run.py \
  --checkpoint checkpoints/Satellite-Birmingham-NDF/ndf.ckpt \
  --heldout-mask data/birmingham/heldout_mask.png \
  --output outputs/birmingham/height.npy

Variants

Task	Dataset	Method	Path
Indoor	ScanNet	NDF	Link
Indoor	ScanNet	CAPA+NDF	Link
Indoor	ScanNet	VGGT+NDF	Link
Indoor	ScanNet	GS+NDF	Link
Street	DrivingStereo	NDF	Link
Street	DDAD	NDF	Link
Street	Waymo	NDF	Link
Satellite	Birmingham	NDF	Link