Package: manydepth
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular DepthJamie Watson,Oisin Mac Aodha,Victor Prisacariu,Gabriel J. BrostowandMichael Firman–CVPR 2021[Link to paper]We introduceManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available.✅Self-supervised: We train from monocular video only. No depths or poses are needed at training or test time.✅ Good depths from single frames; even better depths fromshort sequences.✅Efficient: Only one forward pass at test time. No test-time optimization needed.✅State-of-the-artself-supervised monocular-trained depth estimation on KITTI and CityScapes.pip installManydepth can be installed through pippipinstallgit+https://github.com/AdityaNG/manydepth@pip-moduleRun the webcam demo with :python-mmanydepthTo use the class to create a manydepth2 object as follows :frommanydepthimportmanydepthmd=manydepth()# Load in a frame along with previous framedepth=md.eval(frame,prev_frame)OverviewCost volumes are commonly used for estimating depths from multiple input views:However, cost volumes do not easily work with self-supervised training.In our paper, we:Introduce an adaptive cost volume to deal with unknown scene scalesFix problems with moving objectsIntroduce augmentations to deal with static cameras and start-of-sequence framesThese contributions enable cost volumes to work with self-supervised training:With our contributions, short test-time sequences give better predictions than methods which predict depth from just a single frame.✏️ 📄 CitationIf you find our work useful or interesting, please cite our paper:@inproceedings{watson2021temporal,
    author ={Jamie Watson and
              Oisin Mac Aodha and
              Victor Prisacariu and
              Gabriel Brostow and
              Michael Firman},
    title ={{The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth}},
    booktitle ={Computer Vision and Pattern Recognition (CVPR)},
    year ={2021}}📈 ResultsOurManyDepthmethod outperforms all previous methods in all subsections across most metrics, whether or not the baselines use multiple frames at test time.
See our paper for full details.👀 Reproducing Paper ResultsTo recreate the results from our paper, run:CUDA_VISIBLE_DEVICES=<your_desired_GPU>\python-mmanydepth.train\--data_path<your_KITTI_path>\--log_dir<your_save_path>\--model_name<your_model_name>Depending on the size of your GPU, you may need to set--batch_sizeto be lower than 12. Additionally you can train
a high resolution model by adding--height 320 --width 1024.For instructions on downloading the KITTI dataset, seeMonodepth2To train a CityScapes model, run:CUDA_VISIBLE_DEVICES=<your_desired_GPU>\python-mmanydepth.train\--data_path<your_preprocessed_cityscapes_path>\--log_dir<your_save_path>\--model_name<your_model_name>\--datasetcityscapes_preprocessed\--splitcityscapes_preprocessed\--freeze_teacher_epoch5\--height192--width512Note here the--freeze_teacher_epoch 5command - we found this to be important for Cityscapes models, due to the large number of images in the training set.This assumes you have already preprocessed the CityScapes dataset using SfMLearner'sprepare_train_data.pyscript.
We used the following command:pythonprepare_train_data.py\--img_height512\--img_width1024\--dataset_dir<path_to_downloaded_cityscapes_data>\--dataset_namecityscapes\--dump_root<your_preprocessed_cityscapes_path>\--seq_length3\--num_threads8Note that while we use the--img_height 512flag, theprepare_train_data.pyscript will save images which are1024x384as it also crops off the bottom portion of the image.
You could probably save disk space without a loss of accuracy by preprocessing with--img_height 256 --img_width 512(to create512x192images), but this isn't what we did for our experiments.💾 Pretrained weights and evaluationYou can download weights for some pretrained models here:KITTI MR (640x192)KITTI HR (1024x320)CityScapes (512x192)To evaluate a model on KITTI, run:CUDA_VISIBLE_DEVICES=<your_desired_GPU>\python-mmanydepth.evaluate_depth\--data_path<your_KITTI_path>\--load_weights_folder<your_model_path>--eval_monoMake sure you have first runexport_gt_depth.pyto extract ground truth files.And to evaluate a model on Cityscapes, run:CUDA_VISIBLE_DEVICES=<your_desired_GPU>\python-mmanydepth.evaluate_depth\--data_path<your_cityscapes_path>\--load_weights_folder<your_model_path>--eval_mono\--eval_splitcityscapesDuring evaluation, we crop and evaluate on the middle 50% of the images.We provide ground truth depth filesHERE,
which were converted from pixel disparities using intrinsics and the known baseline. Download this and unzip intosplits/cityscapes.If you want to evaluate a teacher network (i.e. the monocular network used for consistency loss), then add the flag--eval_teacher. This will
load the weights ofmono_encoder.pthandmono_depth.pth, which are provided for our KITTI models.🖼 Running on your own imagesWe provide some sample code intest_simple.pywhich demonstrates multi-frame inference.
This predicts depth for a sequence of two images cropped from adashcam video.
Prediction also requires an estimate of the intrinsics matrix, in json format.
For the provided test images, we have estimated the intrinsics to be equivalent to those of the KITTI dataset.
Note that the intrinsics provided in the json file are expected to be innormalised coordinates.Download and unzip model weights from one of the links above, and then run the following command:python-mmanydepth.test_simple\--target_image_pathassets/test_sequence_target.jpg\--source_image_pathassets/test_sequence_source.jpg\--intrinsics_json_pathassets/test_sequence_intrinsics.json\--model_pathpath/to/weightsA predicted depth map rendering will be saved toassets/test_sequence_target_disp.jpeg.👩‍⚖️ LicenseCopyright © Niantic, Inc. 2021. Patent Pending.
All rights reserved.
Please see thelicense filefor terms.