Package: mars-nerfstudio
MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous DrivingCICAI 2023 Best Paper Runner-up AwardFor business inquiries, please contact us atzhaohao@air.tsinghua.edu.cn.We have just finished a refactorization of our codebase. Now you can usepip installto start using mars instantly! Please contact us without hesitation if you encounter any issues using the latest version. Thanks!1. Installation: Setup the environmentPrerequisitesYou must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.7 of CUDA. You can find more information about installing CUDAhere.Create environmentNerfstudio requirespython >= 3.7. We recommend using conda to manage dependencies. Make sure to installCondabefore proceeding.condacreate--namemars-ypython=3.9
condaactivatemarsInstallationThis section will walk you through the installation process. Our system is dependent on thetiny-cuda-nnproject.pipinstallmars-nerfstudiocd/path/to/tiny-cuda-nn/bindings/torch
pythonsetup.pyinstall2. Training from ScratchThe following will train a MARS model.Our repository provides dataparser for KITTI and vKITTI2 datasets, for your own data, you can write your own dataparser or convert your own dataset to the format of the provided datasets.From DatasetsData PreparationThe data used in our experiments should contain both the pose parameters of cameras and object tracklets. The camera parameters include the intrinsics and the extrinsics. The object tracklets include the bounding box poses, types, ids, etc. For more information, you can refer to KITTI-MOT or vKITTI2 datasets below.KITTITheKITTI-MOTdataset should look like this:.(KITTI_MOT_ROOT)
├── panoptic_maps                    # (Optional) panoptic segmentation from KITTI-STEP dataset.
│   ├── colors
│   │   └── sequence_id.txt
│   ├── train
│   │   └── sequence_id
│   │       └── frame_id.png
└── training
    ├── calib
    │   └── sequence_id.txt
    ├── completion_02                # (Optional) depth completion
    │   └── sequence_id
    │       └── frame_id.png
    ├── completion_03
    │   └── sequence_id
    │       └── frame_id.png
    ├── image_02
    │   └── sequence_id
    │       └── frame_id.png
    ├── image_03
    │   └── sequence_id
    │       └── frame_id.png
    ├── label_02
    │   └── sequence_id.txt
    └── oxts
        └── sequence_id.txtWe use amonocular depth estimation modelto generate the depth maps for KITTI-MOT dataset.Hereis the estimation result of 0006 sequence of KITTI-MOT datasets. You can download and put them in theKITTI-MOT/trainingdirectory.We download the KITTI-STEP annotations and generate the panoptic segmentation maps for KITTI-MOT dataset. You can download the demo panoptic mapshereand put them in theKITTI-MOTdirectory, or you can visit the official website ofKITTI-STEPfor more information.To train a reconstruction model, you can use the following command:ns-trainmars-kitti-car-depth-recon--data/data/kitti-MOT/training/image_02/0006or if you want to use the Python script (please refer to thelaunch.jsonfile in the.vscodedirectory):pythonnerfstudio/nerfstudio/scripts/train.pymars-kitti-car-depth-recon--data/data/kitti-MOT/training/image_02/0006vKITTI2ThevKITTI2dataset should look like this:.(vKITTI2_ROOT)
└── sequence_id
    └── scene_name
        ├── bbox.txt
        ├── colors.txt
        ├── extrinsic.txt
        ├── info.txt
        ├── instrinsic.txt
        ├── pose.txt
        └── frames
            ├── depth
            │   ├── Camera_0
            │   │   └── frame_id.png
            │   └── Camera_1
            │   │   └── frame_id.png
            ├── instanceSegmentation
            │   ├── Camera_0
            │   │   └── frame_id.png
            │   └── Camera_1
            │   │   └── frame_id.png
            ├── classSegmentation
            │   ├── Camera_0
            │   │   └── frame_id.png
            │   └── Camera_1
            │   │   └── frame_id.png
            └── rgb
                ├── Camera_0
                │   └── frame_id.png
                └── Camera_1
                    └── frame_id.pngTo train a reconstruction model, you can use the following command:ns-trainmars-vkitti-car-depth-recon--data/data/vkitti/Scene06/cloneor if you want to use the python script:pythonnerfstudio/nerfstudio/scripts/train.pymars-vkitti-car-depth-recon--data/data/vkitti/Scene06/cloneYour Own DataFor your own data, you can refer to the above data structure and write your own dataparser, or you can convert your own dataset to the format of the dataset above.From Pre-Trained ModelOur model uses nerfstudio as the training framework, we provide the reconstruction and novel view synthesis tasks checkpoints.Our pre-trained model is uploaded to Google Drive, you can refer to the below table to download the model.DatasetSceneSettingStart-EndStepsPSNRSSIMDownloadWandbKITTI-MOT0006Reconstruction65-120400k27.960.900modelreport0006Novel View Synthesis 75%65-120200k27.320.890modelreport0006Novel View Synthesis 50%65-120200k26.800.883modelreport0006Novel View Synthesis 25%65-120200k25.870.866modelreportVitural KITTI-2Scene06Novel View Synthesis 75%0-237600k32.320.940modelreportScene06Novel View Synthesis 50%0-237600k32.160.938modelreportScene06Novel View Synthesis 25%0-237600k30.870.935modelreportYou can use the following command to train a model from a pre-trained model:ns-trainmars-kitti-car-depth-recon--data/data/kitti-MOT/training/image_02/0006--load-diroutputs/experiment_name/method_name/timestamp/nerfstudioModel ConfigsOur modular framework supports combining different architectures for each node by modifying model configurations. Here's an example of using Nerfacto for background and our category-level object model:model=SceneGraphModelConfig(background_model=NerfactoModelConfig(),object_model_template=CarNeRFModelConfig(_target=CarNeRF),object_representation="class-wise",object_ray_sample_strategy="remove-bg",)If you choose to use the category-level object model, please make sure that theuse_car_latents=Trueand the latent codes exists. We provide latent codes of some sequences on KITTI-MOT and vKITTI2 datasetshere.For more information, please refer to our provided configurations atmars/cicai_configs.py. We use wandb for logging by default, you can also specify other viewers (tensorboard/nerfstudio-viewer supported) with the--visconfig. Please refer to the nerfstudio documentation for details.RenderIf you want to render with our pre-trained model, you should visithereto download our checkpoints andconfig. To run the render script, you need to ensure that your config is the same as theconfig.ymlthat you load in.You can use the following command to render. You can modify output format and directory by specificing--output-formatand--output-path:pythonscripts/cicai_render.py--load-configoutputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/config.yml--output-formatvideo

or

pythonscripts/cicai_render.py--load-configoutputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/config.yml--output-formatimages--output-path/path/to/your/output/directoryCitationYou can find our paperhere. If you use this library or find the repo useful for your research, please consider citing:@article{wu2023mars,
  author    = {Wu, Zirui and Liu, Tianyu and Luo, Liyi and Zhong, Zhide and Chen, Jianteng and Xiao, Hongmin and Hou, Chao and Lou, Haozhe and Chen, Yuantao and Yang, Runyi and Huang, Yuxin and Ye, Xiaoyu and Yan, Zike and Shi, Yongliang and Liao, Yiyi and Zhao, Hao},
  title     = {MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving},
  journal   = {CICAI},
  year      = {2023},
}AcknoledgementPart of our code is borrowed fromNerfstudio. This project is sponsored by Tsinghua-Toyota Joint Research Fund (20223930097) and Baidu Inc. through Apollo-AIR Joint Research Center.NoticeThis open-sourced version will be actively maintained and regularly updated. For more features, please contact us for a commercial version.