Package: bev-toolbox
BEVPerception-Survey-RecipeAwesome BEV perception papers and toolbox for achieving SOTA results.🤝Fundamental VisionTable of contentsBEVPerception-Survey-RecipeIntroductionMajor FeaturesWhat's NewLiterature SurveyBEV ToolboxGet StartedInstallationA simple exampleUse BEV toolbox withmmdet3dUse BEV-toolbox withdetectron2Playground on WaymoSetupConfig with PerformanceOngoing FeaturesLicense & Citation & AcknowledgementIntroductionThis repo is associated with the survey paper "Delving into the Devils of Bird’s-eye-view Perception: A Review, Evaluation and Recipe", which provides an up-to-date literature survey for BEVPercption and an open source BEV toolbox based on PyTorch. In the literature survey, it includes different modalities (camera, lidar and fusion) and tasks (detection and segmentation). As for the toolbox, it provides useful recipe for BEV camera-based 3D object detection, including solid data augmentation strategies, efficient BEV encoder design, loss function family, useful test-time augmentation, ensemble policy, and so on. We hope this repo can not only be a good starting point for new beginners but also help current researchers in the BEV perception community.Major FeaturesUp-to-date Literature Survey for BEV PerceptionWe summarized important methods in recent years about BEV perception including different modalities and tasks.Convenient BEVPerception ToolboxWe integrate bag of tricks in the BEV toolbox that help us achieve 1st in the camera-based detection track of the Waymo Open Challenge 2022, which can be grouped as four types -- data augmentation, design of BEV encoder, loss family and post-process policy. This toolbox can be used indedependly or as a plug-in formmdet3danddetectron2.Bag of TricksMultiple View Data AugmentationBEV encoderLoss & Heads familyPost-ProcessRandom FlipRandom Multi-scale ResizeGrid MaskTBATBATest-time AugmentationWeighted Box FusionTwo-stage EnsembleSupport Waymo Open Dataset (WOD) for camera-only detectionWe provide a suitable playground for new-beginners in this area, including hands-on tutorial and small-scale dataset (1/5 WOD in kitti format) to validate idea.What's Newv0 was released in 10/13/2022.Integrate some practical data augmentation methods for BEV camera-based 3D detection in the toolbox.Offer a pipeline to process the Waymo dataset (camera-based 3D detection).Release a baseline (with config) for Waymo dataset and also 1/5 Waymo dataset in Kitti format.Please refer tochangelog.mdfor details and release history.Literature SurveyThe general picture of BEV perception at a glance, where consists of three sub-parts based on the input modality. BEV perception is a general task built on top of a series of fundamental tasks. For better completeness of the whole perception algorithms in autonomous driving, we list other topics as well. More detail can be found in thesurvey paper.We have summarized important datasets and methods in recent years about BEV perception in academia and also different roadmaps used in industry.Academic Summary of BEV PerceptionDatasets for BEV PerceptionBEV CameraBEV LidarBEV FusionIndustrial Roadmap of BEV PerceptionWe have also summarized some conventional methods for different tasks.Conventional Methods Camera 3D Object DetectionConventional Methods LiDAR DetectionConventional Methods LiDAR SegmentationConventional Methods Sensor FusionBEV ToolboxGet StartedInstallationpipinstallnumpyopencv-python
pipinstallbev-toolboxA simple exampleWe provide an example with a sample from Waymo dataset to introduce the usage of this toolbox.importcv2importnumpyasnpfrombev_toolbox.data_augimportRandomScaleImageMultiViewImage# Declare an augmentation pipelinetransform=RandomScaleImageMultiViewImage(scales=[0.9,1.0,1.1])# multiple-view imagesimgs=[cv2.imread(f'example/cam{i}_img.jpg')foriinrange(5)]# intrinsic parameters of camerascam_intr=[np.load(f'example/cam{i}_intrinsic.npy')foriinrange(5)]# extrinsic parameters of camerascam_extr=[np.load(f'example/cam{i}_extrinsic.npy')foriinrange(5)]# transformations from lidar to imagelidar2img=[np.load(f'example/cam{i}_lidar2img.npy')foriinrange(5)]# Augment an imageimgs_new,cam_intr_new,lidar2img_new=transform(imgs,cam_intr,cam_extr,lidar2img)For more details like the coordinate systems or visualization, please refer toexample.mdUse BEV toolbox withmmdet3dWe provide wrappers of this BEV toolbox for mmdet3d and detectron2.Add the following code totrain_video.pyortest_video.py.from bev_toolbox.init_toolbox import init_toolbox_mmdet3d
init_toolbox_mmdet3d()Use functions in the toolbox just like mmdet3d. For example, you can just addRandomScaleImageMultiViewImageto the configure file.train_pipeline=[...dict(type='RandomScaleImageMultiViewImage',scales=[0.9,1.0,1.1]),...]Use BEV-toolbox withdetectron2We plan to make this toolbox compatible with detectron2 in the future.Playground on WaymoWe provide a suitable playground on the Waymo dataset, including hands-on tutorial and small-scale dataset (1/5 WOD in kitti format) to validate idea.SetupPlease refer towaymo_setup.mdabout how to run experiments on Waymo.Config with PerformanceWe provide the improvement of each trick compared with the baseline on the Waymo validation set. All the models are trained with 1/5 training data of Waymo v1.3 which is represented as Waymo mini here. It's worthy noting that the results were run on data withpngformat. We are revalidating these results on the data withjpgformat. So, the actual performance may be different.✓: DONE, ☐: TODO.BackboneHeadTrain dataTrick and corresponding configLET-mAPLLET-mAPHL1/mAPH (Car)StatusResNet101DETRWaymo miniBaseline34.946.325.5✓ResNet101DETRWaymo miniMulti-scale resize, Flip35.646.926.8✓ResNet101DETRWaymo miniConv offset in TSA35.948.125.6☐ResNet101DETRWaymo miniDeformable view encoder36.148.125.9☐ResNet101DETRWaymo miniCorner pooling35.646.926.0☐ResNet101DETRWaymo mini2x BEV scale]--25.5☐ResNet101DETRWaymo miniSync BN--25.5☐ResNet101DETRWaymo miniEMA--25.6☐ResNet101DETRWaymo mini2d auxiliary loss35.347.424.6☐ResNet101DETRWaymo mini2d auxiliary loss, Learnable loss weight36.248.125.4☐ResNet101DETRWaymo miniSmooth L1 loss--26.2☐ResNet101DETRWaymo miniLabel smoothing36.046.7-☐Ongoing FeaturesLiterature SurveyAdd new papers.BEV toolboxData augmentation methods for BEV perceptionRandom horizontal flipRandom scaleGrid maskNew data augmentationIntegrate more tricksPost-processTest-time AugmentationWeighted Box FusionTwo-stage EnsembleBEV EncoderLoss FamilyAdd Visualization in BEVImprove the current implementations.Add documentation to introduce the APIs of the toolboxLicenseThis project is released under theApache 2.0 license.CitationIf you find this project useful in your research, please consider cite:@article{li2022bevsurvey,title={Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe},author={Li, Hongyang and Sima, Chonghao and Dai, Jifeng and Wang, Wenhai and Lu, Lewei and Wang, Huijie and Xie, Enze and Li, Zhiqi and Deng, Hanming and Tian, Hao and Zhu, Xizhou and Chen, Li and Gao, Yulu and Geng, Xiangwei and Zeng, Jia and Li, Yang and Yang, Jiazhi and Jia, Xiaosong and Yu, Bohan and Qiao, Yu and Lin, Dahua and Liu, Si and Yan, Junchi and Shi, Jianping and Luo, Ping},journal={arXiv preprint arXiv:2209.05324},year={2022}}@misc{bevtoolbox2022,title={{BEVPerceptionx-Survey-Recipe} toolbox for general BEV perception},author={BEV-Toolbox Contributors},howpublished={\url{https://github.com/OpenPerceptionX/BEVPerception-Survey-Recipe}},year={2022}}AcknowledgementMany thanks to these excellent open source projects and also the stargazers and forkers:detr3dBEVFormermmdet3d↳ Stargazers↳ Forkers