Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper targets at weakly supervised semantic segmentation for both 2D and 3D domain. The intuition is that the weakly supervisedly features in these two domains are complementary and can benefit each other. Based on this intuition, the paper proposes two constraints, transferring the knowledge from 2D domain to 3D domain and vice versa. Specifically, the model uses 2D sudo segmentation label as supervision for 3D segmentation learning, while encouraging the 2D feature correlation map to match its 3D counterpart. During inference, the model does not need pair 2D-3D data, but instead carry out semantic segmentation on each domain separately.
strengths_and_weaknesses: **Strengths.**
- This paper is well written and easy to follow.
- The proposed method is effective and improves existing methods by a large margin.

**Weaknesses.**
- The proposed method needs pair 2D-3D data for training, which is tricky to get. The paper doesn't demonstrate if the learned model is practical and can generalize to images or scenes outside the chosen dataset.
questions: **Method.**

My foremost concern is that the proposed method needs paired 2D and 3D data, which is hard to obtain and such datasets usually already include ground truth 2D/3D semantic segmentation maps, e.g., ScanNet. Thus it is important and practical for the model to generalize to in-the-wild images or scenes. The model in the paper is learned on ScanNet, then the question is would there be a large domain gap if we apply the model to in-the-wild 2D images or 3D scenes?

**Experiment details.**
- In section 4.1, it seems the 2D &rarr; 3D and 3D &rarr; 2D consistency constraints are included from the beginning. I am curious how the model acts in the beginning, when 3D features are not good while the 2D part is pre-trained on ImageNet, will the 3D correlation maps be misleading in the beginning?
- How to decide the occluded object in line 137?

**Paper writing.**
- I would suggest briefly introduce the Class Activation Maps in the Introduction, which would be friendly for readers without a background. For now the paper mentioned the concept throughout the Introduction but doesn't properly explain it until line 111.
- Minor spelling erros:
  - line 3: has &rarr; have
  - line 42: can &rarr; to
limitations: The authors didn't discuss limitations or societal impact.

As stated above, I would suggest the authors to discuss the generalization capacity of the proposed method on images or scenes outside the chosen dataset.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a method to do weakly supervised semantic segmentation with 2D-3D information. Existing methods have studied the use of class activation maps (CAMs) on 2D images and 3D point clouds for semantic segmentation independently. This paper proposes to leverage 2D and 3D CAMs jointly to learn a network for better 3D semantic segmentation under weak supervision. A joint use of 2D and 3D CAMs can alleviate the sharpness issue of 2D CAMs at object boundaries with the geometric information from 3D point clouds, and similarity, can mitigate the class imbalance and data sparseness problems in 3D CAMs with 2D CAMs. To this end, a 3D-to-2D loss and a 2D-to-3D loss is proposed to enforce the consistency between the similarity matrix of the 2D and projected 3D features, and 2D pseudo label  and projected 3D CAM, respectively. 
strengths_and_weaknesses: Strengths:
+ This paper is well-written, the core ideas are easy to follow.
+ The proposed framework of leveraging the information from 2D images to improve 3D semantic segmentation under weak supervision is interesting. It is really good to see that the paper is written in a way to highlight how the 3D geometrical information from point clouds and the rich and dense 2D image information can compliment each other in the training process.
+ The proposed 3D-to-2D loss on the class correlation matrix of the 2D image and the projected 3D class correlation matrix is technically sound. Similarly, the 2D-to-3D loss is a good idea to enforce the consistency between the 2D and 3D activation maps. 
+ The image classifier pretrained on imageNet can also be seen as the transfer of 2D knowledge to 3D point clouds. Specifically, there is a lack of large-scale dataset for pretraining for 3D point clouds, and thus the use of 2D information from the large scale imageNet can effectively help 3D semantic segmentation. 

Weaknesses:
- While it's understandable that the final goal is to improve 3D point cloud semantic segmentation, the proposed framework of joint 2D-3D learning should actually also help improve the results of 2D semantic segmentation. However, there is no experiments done to show that the 2D semantic segmentation actually benefited from the joint learning. 
- The experiment results are only done on one dataset, i.e. the ScanNetV2 dataset. There should be other dataset with paired 2D image and 3D point cloud available for training and testing the proposed model. It also remains unclear how generalizable is the proposed method, e.g. how well it works on other dataset when it's trained on ScanNetV2? The incompleteness of the experiments somehow weakened the paper. 
- The proposed method requires 2D image-to-3D point cloud pairs. It would be better if the method can also leverage unpaired images and 3D point clouds for training. However, this limitation seems to be difficult to alleviate as the method dependents heavily on the projection. of 3D to 2D. 
questions: 1. Why is it named 2D-to-3D loss in Equation (7)? Isn't the 3D activation map projected into 2D for comparison with the 2D pseudo-labels obtained from the 2D activation map, and thus should be also a 3D-to-2D loss?
2. How much does the joint 2D-3D learning help weakly supervised 2D image segmentation? Intuitively from the explanations in the paper, 2D segmentation results should also improve, especially in the aspect of clear boundaries of the segmentations should be seen. This would be a clear proof that 3D geometry is useful for 2D segmentation, and joint learning is effective in both directions. 
3. How is the experimental results conducted only on ScanNet-V2 convincing that the proposed method can work well on other datasets? Otherwise, it is important to show results on at least another dataset. Although the proposed method is convincing, the lack of experimental results on other dataset(s) is a major weakness of the paper.
limitations: Yes, the limitation is discussed, but the negative societal impact is not discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel 2D-3D Weakly Supervised Semantic Segmentation framework that jointly targets both domains of image and point cloud. To bridge the gap between 2D and 3D domains, the 3D outputs are projected onto the image plane. And they build the consistency of the correlation matrix of
2D features and the projected 3D features. Then they build the consistency of the 2D CAM and the projected 3D CAM.
strengths_and_weaknesses: (+) The idea is interesting and easy to follow.
(+) The ablation studies are comprehensive.
(+) SOTA performance on the ScanNetV2 and S3DIS datasets.
(-) Compared with the baseline, the proposed method requires the perspective projection matrix and images as input during training. This might cause an unfair comparison in the experiment.
(-) The Performance comparison with other state-of-the-art 2D WSSS methods e.g.[R1-R5] is missing.

[R1] Xu, Lian, et al. "Leveraging auxiliary tasks with affinity learning for weakly supervised semantic segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[R2] Yao, Yazhou, et al. "Non-salient region object mining for weakly supervised semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[R3] Wu, Tong, et al. "Embedded discriminative attention mechanism for weakly supervised semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[R4] Lee, Jungbeom, et al. "Reducing information bottleneck for weakly

[R5] Zhou, Tianfei, et al. "Regional semantic contrast and aggregation for weakly supervised semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: In consideration of the reproducibility of this work, the authors are suggested to provide their source code as supplementary.

In Figure 2, there is a projection function from the point cloud to the image. Is the image built from a point cloud? If not, please remove this projection function.

Why 3D-to-2D loss only affects the image branch, while the 2D-to-3D loss affects the point cloud branch? Can they conduct in two-ways? More comprehensive and in-depth studies are needed to justify this design.
limitations: The proposed method requires the perspective projection matrix as input during training, which might be difficult to obtain in a real-world applications.

Only ResNet38 and Point Net++ are employed in this paper. I suggest authors add more backbone to justify the generalization ability of the proposed framework.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new task formulation and a method to jointly train semantic segmentation on images and corresponding point clouds, which can be deployed to segment any of those modalities separately. It only requires image- and scan-level class labels during training but can produce pixel- or point-level segmentation, depending on the input modality. The method makes good use of a strong semantic signal of RGB images and strong geometric signal from point clouds to disambiguate weak supervision. To achieve that, the authors introduce new loss functions to match the pre-pooling activations and the statistics on intermediate features of pixels and projected features of 3D points. That leads to significant improvement in SOTA in both 2D and 3D segmentation tasks.
strengths_and_weaknesses: The paper proposes an important new task formulation and a simple but efficient method to solve it, hence I vote to accept the paper, although not very confidently as WSSS is not my area.

Significance / originality:
(+) the paper proposes a practical task formulation: datasets often have both photometric and LiDAR modalities, and it is reasonable to leverage the synergy between them in absence of strong supervision;
(+) the paper appears to be the first to formulate this specific task; the proposed method is simple but not trivial;
(+) the method relaxes test-time assumptions: it can be deployed when either images or scans are available;
(−) the paper often refer to class imbalance as a hindering factor in point cloud segmentation (e.g. l. 159): can it be solved in a simpler way with re-weighting errors depending on class frequency?

Clarity:
* (±) it is possible to understand the method, although wording can be improved in multiple cases;
* (−) occlusion handling seems to be crucial when projecting 3D features / CAM, yet the paper mentions it only briefly in l. 137; are points assumed to be disks for the purpose of culling? what radius is used? Also, the paper sometimes refers to that process as rendering – that would probably require splatting the points to cover all pixels; the described process is more like projecting and rounding;
* (−) in projecting 3D features, the paper does not specify how collisions are handled, i.e. when multiple points project to the same pixel;
* (−) unclear what the training procedure is: is (12) minimised with SGD directly, or are classifiers first pre-trained and then fine-tuned with transfer losses?

Experiments:
* (+) improvement over SOTA is significant;
* (+) ablation study is sufficient: the paper ablates almost all design choices;
* (−) the paper motivates using correlation matching by CAM loss not working well; it would be good to show it experimentally, i.e. what if 3D-to-2D loss i) just uses L1 distance on features directly, not on their pairwise correlations; ii) uses it on CAMs, not Z features.

===========

Typos / wording:
* l. 19: “studies on semantic segmentation have achieved remarkable performances”;
* l. 41: “It enables our framework can individually perform...”
* in (3), is it \propto instead of the equality?
* l.153: “using maximum logit as a confidence score”: this contradicts (6) where softmax is applied to logits before taking max.
questions: Please address the following, more important, concerns:
* please explain how occlusions and collisions are handled when projecting 3D features,
* please clarify the training schedule w.r.t. classification and transfer losses;
* comment on ablating pairwise correlation (instead of the direct difference of features).
limitations: Limitations are only briefly mentioned. Something to discuss:
* how sensitive is the method to the quality of camera poses used for re-projection?
* how sensitive is the method to class imbalance; how well does it work on rare classes? Maybe report a class-weighted metric (not weighted).
review_title: null

Meta-review: metareview: The paper proposes two constraints, transferring the knowledge from 2D domain to 3D domain and vice versa, to do weakly supervised semantic segmentation in 2D and 3D.  During inference the model doesn't require paired data.

The paper is clearly written, and explains how the 3D geometrical information complements the redundant dense 2D information.  The reviewers mention limited experiments.  However, the ablations are thorough, and show significant improvement over SotA performance.  

The authors addressed the reviewers' comments during the rebuttal.  The reviewers agree that the benefits of the proposal outweigh its flaws.

I recommend the paper for publication.
confidence: Certain
Decision: Accept