Image-to-Voxel Model Translation for 3D Scene Reconstruction and Segmentation
Channel: 3DOM-FBK
Date: 2020-08-23

- ECCV 2020 paper - 
Objects class, depth, and shape are instantly reconstructedby a human looking at 2D image. While modern deep models solve each of this challenging tasks separately, they struggle to perform simultaneous scene 3D reconstruction and segmentation. We propose a single shot image-to-semantic voxel model translation framework. We train a generator adversarially against a discriminator that veries object's poses. Furthermore, trapezium shaped voxels, volumetric residual blocks, and 2D-to-3D skip connections facilitate our model learning explicit reasoning about 3D scene structure. We collected a SemanticVoxels dataset with116k images, ground-truth semantic voxel models, depth maps, and 6Dobject poses. Experiments on ShapeNet, UP-3D, and our SemanticVoxels datasets demonstrate that our framework achieves and surpasses state-of-the-art in reconstruction of scenes with multiple non-rigid objects of different classes. We made our model and dataset publicly available at http://www.zefirus.org/SSZ