Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel method to perform knowledge distillation between a parent model that is only trained on 2d data and a child model trained on 3d data. To my knowledge this is the first paper to show that it is possible to distill knowledge between models that accept different inputs. This in itself is a novel and interesting contribution.  The model that they construct with this approach outperforms a baseline model while using only 3.8% as many parameters.
strength_and_weaknesses: Strengths:
* Proves that knowledge distillation is possible between two models that accept different inputs
* Uses this method to produce a high-quality model that performed as-good or better than one baseline model with a fraction of the parameters and runtime.
* Introduced a new dataset benchmark to encourage further development of this task.
Weaknesses:
* Only compares to one older (2016) baseline method when there are several more recent works that include code that could be used.
clarity,_quality,_novelty_and_reproducibility: The quality of this work is very good and the work is very novel. Just the insight that knowledge distillation can be applied to models with different inputs is significant. The authors have agreed to release code and data upon acceptance so hopefully reproducing the results will be trivial, but the writing is clear enough that for a researcher with a background in 3d geometrical reasoning should be able to reproduce independently. 

The main drawback is that there is not enough comparison to state of the art 3d semantic segmentation methods.  I would recommend looking at the Scannet benchmark challenge and at minimum finding some of the top entries with code and adding them to the papers comparisons. 
summary_of_the_review: This is a novel approach for knowledge distillation between a 2d model and a 3d model for 3d semantic segmentation. The paper presents a new dataset to support this problem and presents nominal results showing that the approach is effective. Unfortunately there is not enough comparison to state of the art methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a pipeline for 3D scene-level segmentation based on an efficient 3D Transformer structure. On top of that, the authors also propose to distill information from pre-trained/finetuned 2D segmentation networks to improve the performance for 3D segmentation. The distillation is made possible by differentiable panoramic 2D rendering of the 3D scene. To train and evaluate the proposed model, a photorealistic synthetic 3D dataset is proposed. The performance of the proposed model is shown to be better than the 3DUnet structure but with much less parameters.

strength_and_weaknesses: Strength

+The 2D-to-3D distillation process is novel: the submission proposes to use differentiable panoramic rendering of 3D scene to bridge the 3D and 2D segmentation tasks.

+The proposed 3D segmentation backbone is based on the Video SwinTransformer, thus leading to high efficiency in terms of number of parameters. In the experiment, it has been shown that the 3D Transformer-based segmentation backbone archives better performance than 3DUNet with only ~4% number of parameters.

+The proposed synthetic dataset is photorealistic and complete. This could be useful for training and evaluating 3D segmentation backbones.

Weakness

-As mentioned in the limitation, the panorama rendering does not handle occlusion well. As a result, the occluded objects/parts when the camera is positioned in the center of the scene would be missing in the segmentation result.

-Experiment benchmark is limited: The proposed method is only tested on the proposed 3D segmentation synthetic dataset. The performance of the proposed pipeline on real-world datasets such as ScanNet and MatterPort3D is unknown. 

-Compared method is limited: The proposed method is only compared against 3DUNet, which was proposed in 2016. There are lots of backbones and scene representations (such colored point cloud) on which the backbone has been performed since then (eg. PointTransformer).  As a result, it is hard to conclude that the proposed method archives state-of-the-art (SOTA) performance, even though it is efficient in terms of number of parameters.

clarity,_quality,_novelty_and_reproducibility: Quality and clarity:

The paper is clearly written and easy to follow. The figures are illustrative enough for manifesting the main idea. However, the experiment is lacking and therefore not convincing enough (see the weakness part above for more details).

Novelty:
The 2D-to-3D distillation method proposed in the paper is novel and interesting and could be inspiring for future works for 3D segmentation tasks.
summary_of_the_review: Although the idea of using the differentiable panoramic rendering for 2D-to-3D distillation is novel, the experiments are lacking in terms of both benchmarks and compared method. As a result, I’m not sure if the proposed backbone and the distillation method are good enough to achieve the SOTA performance. As a result, I’m on the fence (towards rejection) unless convincing feedbacks are provided.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to solve the 3D semantic segmentation problem by distilling the knowledge embedded in the latent space of powerful 2D models. Unlike traditional knowledge distillation approaches, where student and teacher models take the same input，the inputs of the 2D teacher model in this paper are the panorama images stitched from 6 different views of 3D scenes. Thus, it is possible for gradient backpropagates from 2D to 3D. Video Swin Transformer is adapted as the 3D segmentor to segment the input 3D scenes, and a Kullback–Leibler divergence loss is used to measure the difference of distribution between student 3D segmentation maps and teacher 2D segmentation maps. To facilitate the experiments, this paper also proposed a large-scale 3D volumetric dataset with a clean voxel-wise semantic label and well-aligned corresponding 2D panoramic renderings with pixel-wise semantic labels. Experimental results on this dataset demonstrate the superiority of the proposed transformer based 3D segmentor over the 3D convolutional Unet segmentor, and the effectiveness of the proposed knowledge distillation strategy as well.


strength_and_weaknesses: Strength:
This paper is the first one to utilize the data-abundant and pretrain-ready 2D semantic segmentation to improve 3D semantic segmentation, which is reasonable and novel.
Experiments demonstrate the effectiveness of the proposed model over the 3D-Unet segmentor with convolutional architecture, and the proposed model consumes less parameters. The proposed distillation strategy is also proved to be useful.
The proposed PANOROOMS3D dataset is beneficial to the community.

Weakness:
How to get the detail correspondences between 2D panorama image pixels and 3D patches? There is lack of descriptions.
It seems that the proposed model only compares with 3D-Unet segmentor which is proposed in 2016. How about other baseline models like more powerful 3D segmentation methods proposed in recent years?
Can the proposed distilling strategy also improve 3DSeg-M and 3DSeg-L performance? There is also lack of experiments and demonstration.
clarity,_quality,_novelty_and_reproducibility: The organization and demonstration of this paper is clear and easy to follow. The 2D-to-3D distillation idea is novel. The authors have stated that the code and dataset will be released after the acceptance of the paper.


summary_of_the_review: This paper proposed a knowledge distillation method to utilize the pretrain-ready 2D semantic segmentation to improve 3D semantic segmentation, which is novel and reasonable. Taking 2D panorama images collected from 3D input scenes as the input to the 2D segmentor is also beneficial to correlate the 2D-3D data. I think the overall idea of this paper is good. But other 3D segmentors except 3D Unet is also should be compared, and some details should be demonstrated more clearly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Multi-modality between 2D and 3D is important. The paper mainly introduces a method that achieves 2D-to-3D knowledge distillation. They also propose PanoRooms3D, a large-scale 3D volumetric dataset that has aligned 2D (pano images) and 3D data (3D voxels). 
strength_and_weaknesses: The task of aligning 2D and 3D is important, and the current benchmarks are not good enough (including both diversity and quantity) to achieve this. The paper is good to read and easy to follow. But there miss some important experiments and comparisons: 
(1) About the dataset. The ScanNet also contains about 1500 scans. Instead of providing Pano images, they have a sequence of images and corresponding labels that are used to rebuild the scan. The author should claim the strength of the proposed dataset. For example, why using pano images can achieve higher accuracy than a sequence of single images. 
(2) The distillation idea to find 2D-3D corresponding samples is not new. BPNet, which is designed to combine 2D and 3D info, can also modified to the knowledge distillion. 
(3) More comparisons should be provided, as current method applies sparse convolution, instead of fully 3D CNN to address the task.
(4) Some technique is not clearly stated. How to transfer a p X p X p voxel to a token embedding. Is this achieved by fully 3D CNN? If so, will this waste a lot of memory and computation on invalid space?
(5) The backbone is designed to use 3D SwinTransformer. Is the parameter initialized with pretrained video swin-transformer model? Or just training from scratch. I am curious about the effect if the params are initialized with a model from a video pretrain. 
clarity,_quality,_novelty_and_reproducibility: The novelty is not clear, as the author fails to make a comparison with other benchmarks. So the superiority of the method is not clear. The writing of the paper is good and easy to follow. If the author provides the dataset, I think the experiment is easy to reproduce. 
summary_of_the_review: See the weakness part.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper proposes a 2D-to-3D knowledge distillation method to improve 3D semantic segmentation.    The work proposes a 3D segmenter based on Video Swin Transformer.  For 2D-to-3D knowledge distillation, a 2D segmenter (the teacher model) is pretrained on ImageNet-21K and fined-fined on 2D panoramas, and used to train a 3D detector by using differentiable rendering of semantic volume predictions to project to panoramas.  To investigate the setting, the authors also contribute PanoRooms3D, a dataset of 5917 rooms (based on 3D-FRONT) with semantic voxels and 2D panoramic renderings.  Comparisons with prior work (3D UNet), experiments (on  PanoRooms3D and ScanNet) show that the proposed 3D Segmenter can achieve similar performance with less parameters, and that the 2D-to-3D distillation approach gives similar results as direct supervised training of the 3D detector.

Strengths:
- Proposes a 3D segmenter based on Video Swin Transformer that predicts voxel-based semantic segmentation
- The proposed 3D segmenter can achieve similar performance as prior work (3D UNet) with less parameters
- The proposed 2D-to-3D distillation approach gives similar performance as directly supervised training of the 3D detector
- Introduces a dataset with semantic voxels and aligned panoramic images 

Weaknesses:
- There is limited discussion of prior work on knowledge distillation from 2D to 3D (such as [1] and [2])
- There is limited comparisons against prior work on 3D segmentation
- In the experiments, panoramic images for the pretraining and distillation are taken from the same set of rooms.  This means that during distillation, the teacher model should have relatively good predictions as the rendered panoramas would be from the same rooms it would be trained on.  A better experimental setup would be to apply the distillation on a separate set of room for which there is no annotated 3D labels.
- The number of rooms in the val/test set is small, and there are no training variances reported
- There isn't much detail about how the evaluation metrics are computed and averaged (are they macro-averaged over classes?)

[1] Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data, Sautier et al. CVPR 2022
[2] Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining, Liu et al, 2021

Minor improvements to writing including to having space before '(' and improved typesetting of equations is recommended.

justification_for_why_not_higher_score: There is a lack of comparison with more recent 3D segmentation methods and some missing discussion of recent work in 2D-to-3D distillation.

justification_for_why_not_lower_score: The paper make several contributions that works well, including the proposed 3D segmenter, the 2D-3D distillation method, and a dataset.  The direction of 2D-to-3D distillation can be an interesting direction for the community.
Decision: Accept: poster