Joint Learning of 2D-3D Weakly Supervised Semantic Segmentation
Authors: Hyeokjun Kweon
Kuk-Jin Yoon
Venue: NeurIPS 2022 Accept (2022)
Keywords: Weakly supervised semantic segmentation
Point cloud

Abstract: The aim of weakly supervised semantic segmentation (WSSS) is to learn semantic segmentation without using dense annotations. WSSS has been intensively studied for 2D images and 3D point clouds. However, the existing WSSS studies have focused on a single domain, i.e. 2D or 3D, even when multi-domain data is available. In this paper, we propose a novel joint 2D-3D WSSS framework taking advantage of WSSS in different domains, using classification labels only. Via projection, we leverage the 2D class activation map as self-supervision to enhance the 3D semantic perception. Conversely, we exploit the similarity matrix of point cloud features for training the image classifier to achieve more precise 2D segmentation. In both directions, we devise a confidence-based scoring method to reduce the effect of inaccurate self-supervision. With extensive quantitative and qualitative experiments, we verify that the proposed joint WSSS framework effectively transfers the benefit of each domain to the other domain, and the resulting semantic segmentation performance is remarkably improved in both 2D and 3D domains. On the ScanNetV2 benchmark, our framework significantly outperforms the prior WSSS approaches, suggesting a new research direction for WSSS.

Joint Learning of 2D-3D Weakly Supervised Semantic Segmentation
Hyeokjun Kweon
Kuk-Jin Yoon KAIST
KAIST
0327juneCkaist.ac.
kjyoonOkaist.ack ac.kr
Abstract
The aim of weakly supervised semantic segmentation (WSSS) is learn semantic segmentation without using dense annotations. WSSS has been intensively studied for 2D images and 3D point clouds. However, the existing WSSS studies have fo- cused on a single domain, i.e. 2D or 3D, even when multi-domain data is available. In this paper, we propose novel joint 2D-3D WSSS framework taking advantage of WSSS in different domains, using classification labels only. Via projection, we leverage the 2D class activation map as sulf--upeniiiion enhance the 3D semantic perception. Conversely, we exploit the similarity matrix of point cloud features for training the image classifier to achieve more precise 2D segmentation. In both directions, we devise confidence-baseee effect of inaccurate self self-supervision. With extensive quantitative and qualitative experiments, we verify that the proposed joint WSSS framework effectively trans- fers the benefit of each domain to the other domain, and the resulting semantic segmentation performance is remarkably improved in both 2D and 3D domains. On the ScanNetV2 benchmark, our framework significantly outperforms the prior WSSS approaches, suggesting research direction for WSSS.
scoring method to reduce the
Introduction
Semantic segmentation is fundamental problem in the field of computer vision. Recently, learning- based semantic segmentation methods have achieved remarkable performance. However, they usually rely strong supervision such as pixel-wise (2D) point-wise (3D) dense semantic labels. Since dense labels are time-consuming and inborrintensive to annotate, the labeling process has remained the main obstacle to applying semantic segmentation real problems. Weakly supervised semantic segmentation (WSSS) has flourished to learn semantic segmentation by using inexpensive weak labels (e.g. bounding boxes or class lahels).
WSSS using class labels has been widely studied in the 2D domain (image) and has recently also been scaled into the 3D domain (point cloud). For acquiring pixel- point-wise class predictions without any spatial supervision, the existing 2D and 3D WSSS studies have employed Class Activation Maps (CAM) [35] extracted from classifier. Since CAM usually localizes the discriminative regions of each class (which be used as cues for classification by classifier), one can acquire pseudo-labels for semantic segmentation from the CAM. While the studies have shown promising results, they focus on using the data of a single domain and place less emphasis on the information other domains that might have complementary properties.
Fig. [ visualizes the complementary advantages 2D and 3D WSSS. With rich and dense features from a CNN-based architecture, 2D WSSS succeeds in the semantic perception of images. However, the 2D CAM is usually not sharp at the object boundary and the segmentation result tends to be imprecise as shown in (b3). On the other hand, 3D WSSS is advantageous in terms of segmentation, since the input point cloud contains the 3D geometry of the scene. However, due to the severe class
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Baselines
Ours (2D-3D joint WSSS)
Input
Ground truths
(2D WSSS and 3D WSSS)
(al) cloud
(b1) 3D w WSS result
(c1) Ours result (3D)
(d1) GT
(a2) Projected point cloud
(b2) Proj. W WSSS result
(c2) Proj. ours result (3D)
(d2) Proj. 3D GT
(a3) RGB image
(b3) 2D WSSS Result
(c3) Ours result (2D)
(d3) 2D GT
Figure 1: Comparison results from single 2D or 3D domain WSSS and the proposed multi-domain WSSS. The second column shows the results of 3D WSSS (b1, b2) and 2D WSSS (b3). 3D WSSS misclassifies the semantics of the objects (red boxes), while the 2D WSSS generates imprecise prediction object boundaries (yellow boxes). Compared to the uni-domain results, the proposed 2D-3D joint WSSS framework achieves much more precise and complete semantic segmentation results in both domains (cl-c3), by unifying the benefits of multi-domain data.
imbalance in 3D data (e.g. wall or floor exist in almost every scene) and the sparse nature of the point cloud, 3D WSSS difficulty predicting the semantics as in (b1) and (b2). The results of 2D and 3D WSSS show different tendencies originating from the distinct domains, and they are complementary for achieving semantic segmentation of the scene.
Upon this background, we propose a novel 2D-3D WSSS framework that jointly targets both domains of image and point cloud. For unifying the complementary strengths of both domains, we propagate the benefit of one domain to the other domain via self-supervision, and vice versa. It enables our framework individually perform 2D and 3D WSSS, without requiring the paired input data the inference phase. We obtain 2D pseudo-label from the 2D CAM of the image classifier and make 3D CAM follow it, which enhances the semantic perception of the point cloud classifier. On the other hand, to transfer the segmentation capability of 3D CAM to 2D, we construct correlation matrix between the 3D features. The image classifier is then trained to generate 2D features having similar correlation matrix to that of the 3D, which relieves the impreciseness problem of 2D CAM. In both directions, to prevent the networks from being spoiled by inaccurate self-supervision, the framework obtains confidence score for each region and rejects erroneous predictions.
With extensive qualitative and quantitative experiments, we verify that the proposed joint framework effectively transfers the benefit of each domain to the other domain as visualized in (c (cl-c3) of Fig. As result, in both image and point cloud domains, the semantic segmentation performance remarkably enhanced simultaneously. For example, on the ScanNetV2 [5] benchmark, our method significantly outperforms the existing WSSS studies, demonstrating an experiment on generalization capability to the S3DIS dataset, we verify that the gain of our joint framework could extended to the unobserved data. The proposed framework is the first multi-domain approach for the field of WSSS, which is not only effective for learning both semantics and segmentation but also practical for being applied in real cases. We strongly believe that 2D-3DD joint learning promising direction for conquering semantic segmentation with class labels only.
superiority. Further, with

---

Table 1: Summary of related works in the field of WSSS. Compared to the prior single 2D or 3D domain works, our method targets both 2D and 3D domains by using image-level and scene-level class labels only. CLS and SS denote class labels and semantic segmentation labels, respectively.
Domains
2D+3D
Methods
11B13 32 24 TG 25
20131 B3134
MPRM 29
WyPR122 CLSS (scene)
Proposed
CLS
SS (few points)
CL.S (subcloud)
CT.S (image and scene)
Weak Labels
(image)
2
Related works
In this section, we introduce the existing studies on 2D/3D WSSS and clarify the uniqueness of the proposed WSSS method that targets both domains in a simultancous manner. Table [ shows the setting of the weak labels used and those of existing studies. Note that the proposed framework is the first method that exploits the complementary strengths of WSSS in the 2D and 3D domains, using image- and scene-level class labels only.
WSSS in 2D domain 2D WSSS with image-level class labels only has been widely studied for the last decade. Since the class labels do not provide any spatial supervision, most WSSS studies have exploited a CAM for localizing the regions of each class. Although the CAM can serve as an acceptable estimate for localization, they are usually activated in the most discriminative regions only, rather than capturing whole object regions. Moreover, the CAM sensitive to the scale variance of the input image and less precise at the boundary of the objects. To relieve these issues, existing works have proposed various techniques such as refining [11. 13 1
32
adversarial erasing [32 24 [6], sub-categorical classification (3), and cross-image attention 7
However, their problem definition and benchmark dataset 66 [8] mainly focus on the 2D domain and less consider 3D structures of the scene, which are valuable for solving the aforementioned issues.
WSSS in 3D domain Point-wise dense semantic segmentation labeling for 3D point cloud is notorious for being difficult to annotate (22.3 min) [29]. To resolve this, several studies 200 BT
8 33 34]
have been conducted to learn semantic segmentation with only partially labeled points. However, since acquiring partial point-wise annotation is still expensive even under the "One Thing
One Click" setting (2 min) 20). a few studies have been conducted for exploiting less expensive class labels (15 sec) [29]. MPRM [29] samples subclouds from the full scene and annotates them, since using subcloud-level class labels can relieve the class imbalance issue (e.g. wall floor) commonly appearing in almost every scene). Recently, WyPR (22) jointly targets semantic segmentation and object detection of a point cloud with scene-level class labels only. The studies have shown the feasibility of WSSS in the 3D domain; however, due to the sparse and irregular nature of the point cloud, the performance of 3D WSSS limited compared to the 2D WSSS.
Semantic segmentation in combined 2D-3D domains To exploit both 2D and 3D data for semantic segmentation, several semantic segmentation studies have been conducted for uni-directional [2] or bi-directional [0] feature projection between the 2D and 3D domains, in a fullly-supervise setting.
leveraged the 2D semantic segmentation for learning the 3D semantic segmentation with multi-vie -iew or virtual-view settings; however, they still require a dense 2D GT. [26] proposed to use 3D information as guidance for enhancing 2D WSSS using bounding boxes. Recently, ( [7 incorporated 3D priors for self-supervised learning of 2D networks to achieve 3D-aware scene recognition. Unlike the existing studies, our framework is the first WSSS approach that jointly unifies the complementary benefits of 2D and 3D domains, by using the image-level and scene-level class labels only.
28 15
Method
In this section, we introduce the proposed 2D-3D joint framework for learning WSSS in both domains. Figure 2 shows our overall framework.
3

---

Image I e (T. ... I
2D CAM A
2D pscudo-label M
Image classifier F
2D feature Z
SEgISAX
2D Corr.
V
Projection (real)
2d
2D-to-3D loss
3D-to-2D loss
Scene
3D Corr.
Sdd.p
Projected
zd.p
Projected Add.p 3D CAM
Point cloud classifier g
3D feature
Projection (situal))
Projection
(virtua))
Point cloud P
3D Feature 3d
3D CAM A
3D pseudo-label Md
Figure 2: Our overall framework. As inputs, we use point cloud P and an image Is sampled from a of images corresponding the point cloud, where the projection for the image is given V Note that the image is captured from the real scene. An image classifier F and point cloud classifier are trained to extract 2D outputs (Zad and A 2d) and 3D outputs (Z and A Add), respectively. To bridge the 2D and 3D domains, the 3D outputs are projected onto the image plane by using V For 3D-to-2D), we construct a correlation matrix of 2D features (S2d) and make it to follow that of the projected 3D features (d...)) For 2D-to-3D, we generate 2D pseudo-label 2 from 2D CAM and make projected 3D CAM follow Note that blue dashed lines denote propaaaapaationn of the losses.
3.1 Acquiring CAM of each dimension
Following the prior works with the class labels only, we exploit CAM to localize the class regions from the image or point cloud. We assume that a set of K images {T² I and a point cloud are obtained from certain scene. Is is an RGB image with size of H x W. and is set of N points where each point is represented by its 3D coordinate (XYZ) and color (RGB), as in [2] view matrix v (perspective projection matrix) of the camera for the image Is is also given, as in many point cloud datasets 2). As weak supervision, we use the image-level class label 22d and scene-level class label 3d where both labels are represented as C-dimensional binary vectors. C' denotes the total number of classes, and the classes are non-exclusive since multiple classes can co-exist single image (or point cloud).
As visualized in Fig. we train an image classifier F. to extract 2D feature map 2dd DDX single linear layer having input channels and C output channels then serves as a classification head for making pixel-wise class prediction A 2d e IR x Similarly, a point cloud classifier G is trained provide point-wise features 3d E DxN and point-wiss class prediction A Add R C'X Here, we directly employ A 2d and A as the CAM of each domain, and refer to them as 2D CAM and 3D CAM throughout this paper. For both networks, we use the GAP (Global Average Pooling) layer to average the pixel- or point-wise predictions along the spatial dimension as follows:
H1
W
N
y
2d
2d ..
and
ydd
Σ T=
3d
(1)
HW
N
where 2d C and 3d E RC denote image-level and scene-level class prediction, respectively. To promote the 2D and 3D CAMs to be activated on the corresponding regions of classes, the classifiers trained with classification loss following, where lhce denotes the binary cross entropy loss.
Cad. ly (d) and
Cale loce ay (dd).
(2)

---

3.2 From 2D to 3D: Transferring semantic capability
We are inspired by an observation that the information contained in 2D and 3D data is not only distinct but also complementary. The proposed framework aims jointly learn WSSS in both domains and unify their strengths. For joint learning of both domains, we could directly provide information from one domain to the other domain via feature paths using projection, in [10]. However, in the inference phase, a model including the feature path requires the paired set data of both domains, which greatly limits the practical use case.
Therefore, instead directly swapping the features, we devise our framework to provide information from one domain to the other domain as form of self-supervision during the training process only. It enables the proposed 2D-3D framework to separately process the data each domain our framework perform 2D WSSS and 3D WSSS individually in the inference phase, without using the input data from the other domain.
We use the view matrices of images for bridging the 2D and 3D domains, visualized in Fig. 2 The projection from 3D homogeneous coordinates to the pixel coordinates in I is formulated as follows:
[ uh 1]" [Z Un. 2n. 1)²
where (In, Man 2n) the coordinate of the rith point in the point cloud P and (u wh) is the destination of the projected point in Is_ We denote the projection function with the view matrix V as proj(-, Vkk. For simplicity, we omit & throughout this paper.
With given V, the 3D point features ad and 3D CAM 3d are projected onto the image plane. Here, we discard the points located outside of the image grid. Also, we filter the points of the occluded object which should not exist on the image. The projected 3D features Zdd E B Dx No. and 3D CAM A 3d,p E RCxN can be obtained as follows:
zdp proj V) and A 3d.p pAoj 3d V),
where Ny denotes the number of preserved points. Note that the projection process is differentiable, and therefore the gradients can be pack-propa (see blue dashed lines in Fig. 2
Once the projected 3D CAM acquired, we bridge the 2D CAM and 3D CAM at the pixels where the projected points exist. To leverage the semantic advantages of the 2D domain for improving 3D CAM, we obtain pseudo-label map M from the CAM. Since there usually no explicit background class in indoor scenes, we regard the class with maximum prediction score as a predicted class at each pixel as follows:
M
argmax (t22 A c. b ct CE(1....CC)
where the non-exisie non-existing classes are ignored by using the given image-level class label 2d
We apply point-wise classification loss the 3D CAM to follow the acquired 2D pseudo-label. However, unlike GT supervision, the predicted 2D pseudo-labels may include inaccurate signals and cannot be fully trusted. To address this, regard the maximum score of class prediction of each pixel as a metric for the confidence of that pixel. The confidence map W is defined as follows:
W f.
max A 2d
We define the 2D-to-3D loss that transfers 2D semantic capability to 3D network as
No Σ
fdd-33
W
un. laee ( (A (A...p 3d.p 2d
N
where the (un, vn)) is pixel coordinate of the nth projected point as we denoted. Using maximum logit as confidence score can reduce the contributions such less confident pixels to the loss, and prevent 3D CAM from learning with erroneous self-supervision. We verify that this confidence-based scoring method brings actually increases the WSSS performance by a large gap, which will be demonstrated in Section
5

---

3.3 From 3D to 2D: Transferring segmentation capability
Thanks to the well preservee geometrical structures of the point cloud, the 3D WSSS can be advanta- geous for segmenting the scenes compared with 2D WSSS. However, unlike 2D-to-3D loss utilizing 2D pseudo-labels, this benefit cannot be directly leveraged to mitigate the impreciseness of 2D CAM due to the class imbalance issue and low semantic perception capability of the 3D CAM. In this paper, to unlock the segmentation potentials of the point cloud classifier, we train the image classifier to generate 2D features having similar correlation matrix with that of the 3D features.
Motivated by [19], we construct a correlation matrix Söd, from the projected 3D features Zd,, by using the following equation:
S Sade P
Z.d... Z.S."
(8)
where (a, b) denotes a pair of the projected points. Similarly, for the pixels corresponding to the projected points, Stdd computed from 2D features Zdd as follows:
2d 2,
2d
0.6
06,
where the (Ma: va) and (ub: 1%) denote pixel coordinates of the ath and bth projected point, respec- tively. Note that we map the features on a unit hypersphere by normalization, before computing the correlation matrices. Similar the 2D-to-3D loss, construct W the confidence of the correlation matrix of projected 3D features, by multiplying the confidence at each point as follows:
W max A A.P) max(A 3d..).
(10)
The confidence-based 3D-to-2D loss then defined as follows:
N. N
3 Sad
Sdd. 11.
(1I)
In summary, we apply the sum of the classification and the joint loss functions for training our framework as following equation, where is a weighting parameter.
Listal = L2d + Cale + (2d-3d + AL
(12)
joint
4 Experiments
4.1 Implementation details
The proposed framework is implemented with PyTorch. ResNet38 (30 and Point- Net++ 22 are employed as backbones for the image classifier and point cloud classifier, respectively. The image classifier is initialized with the weights pre-trained on ImageNet [23]. We augment the input point cloud with sub-sampling, random flipping, and random rotation. For the image, random resizing/cropping. horizontal flipping. and color jittering [14] are applied. We set À = in Eq. [2] The model is trained on two Tesla V100 GPUs with batch size 16 for 200 epochs. The initial learning rate is set to 0.003, and be decayed by 0.1 at epoch 120, 160, 180 as in 22 More details can be found in Supplementary Material.
Framework
conduct experiments on ScanNetV2 (5) dataset (MIT license, we agreed to the terms of use), which contains indoor scenes such as offices or bathrooms. ScanNetV2 composed of 1513 scans with annotations (semantic segmentation GT, camera poses, etc.) and 100 test scans serving as a benchmark. We follow the official split, where there exist 1201 train scans and 312 val scans. For the 2D data, we use the provided RGB frames and the corresponding camera view matrices. We sample only 1% of all provided frames, which is equivalent the proposed label setting is less expensive than that of MPRM (29 (using around 17 subclouds per scene), since annotating class-level labels even easier on images than on point clouds. For evaluating the semantic segmentation performance, we use the mean intersection over union (mloU) as metric on both the and 3D domains. Since the GT for the test set not publicly available, test performance is evaluated on the ScanNet evaluation server.
Dataset and metric Following the existing 3D WSSS studies 29 22
using around 17 frames per scene on average. Note that

---

Table 2: Ablation study of the proposed framework. The semantic segmentation performance (mloU, %) is evaluated on the official train and val splits of the ScanNetV2 [3]. From left to the right, each loss term denotes image-level 2D classification loss ( L scene-level 3D classification loss ( L ). 2D-to-3D loss (C3 ), and 3D-to-2D loss (c ). Confidence-basee scoring denotes using W
(in Eq. and Eq. 10). Bold numbers represent the best results.
Confidence-basse scoring
Domain
cie
33d
C3d-22d
Train
Val
38.0
33.3
v
47.8
39.2
2D
d
50.0
41.6
7
/
54.4
44.5
17.9
16.2
51.3
43.8
3D
53.5
45.6
4
59.1
49.6
Input
GT
2D and 3D CAMs of chair class
2D and 3D CAMs floor class
(a
(22)
(bl)
(b2)
(h3)
(b1)
(h2)
(h3)
Figure 3: Qualitative comparison of 2D/3D CAMs. 3D CAMs (al) uses scene-level classification loss (L3d) only, (a2) uses 2D-to-3D loss (C (C2d-33d) and (a3) uses bidirectional (2D-to-3D and 3D-to- 2D, ie., Ldd-3d joint + L joint
c3d-2d) joint loss. 2D CAMs - (b1) uses image-level classification loss (C2d)
only, (b2) uses 3D-to-2D loss (C3dr42)) and (b3) uses bidirectional joint loss. More qualitative
comparison and samples of 2D and 3D CAMs can found in Supplementary Material.
4.2 Ablation studies
To demonstrate the advantage of the proposed joint learning framework, we conduct an ablation study, as shown in Table 2 For each domain, the first row shows the result of our baseline setting, where only the image- or scene-level classification loss of each domain (C3d or Lad is applied. As reported in prior works (29 222, we also observe that the 3D WSSS suffers from the class imbalance of scene-level class labels and thereby completely fails to understand the semantics of the objects, as visualized in (al) of Fig. 3 On the other hand, as shown (b1), 2D WSSS shows the capability to roughly localize the region of each class on the image. However, due to the lack of geometrical 3D information, the 2D CAM fails be precise the object boundary. This illustrates the motivation of the proposed 2D-3D joint framework.
benefit of the 2D-to-3D loss (o is shown in the second row of the 3D domain in Table The train and val performance of 31) CAM are significantly increased from 17.9% and 16.2% to 51.3% and 43.8%, respectively. The 3D CAMs Fig. 3 (a2) show much more accurate results with the help of the 2D-to-3D joint loss. Here, to clarify the benefit of the 2D-to-3D loss, we also conduct an additional experiment. In this experiment, instead of the joint loss, apply additional classification loss to the point cloud classifier by using the image-level class labels. This setting similar to that of PCAM (99] however, the classification is done on the image level, not on the subcloud level. It achieves 43.9% and 39.8% 3 on train and val split respectively, which are the almost equivalent performance to PCAM (43. and 38.1% (%) but clearly lower than using our 2D-to-3D loss (51.3% and 43.8% The result indicates that the performance gain of the proposed 2D-to-3D loss not only comes from relieving the class imbalance problem of the 3D domain but also effectively transferring the better semantics of the 2D domain to the 3D domain.
7

---

Table 3: Performance (mloU, %) comparison with other state-of-the-art 3D WSSS methods 29
22
on ScannetV2 [3] train, val, and test split. Class- IoUs can be found in Supplementary Material. Bold numbers represent the best results, while underlined numbers are the second best results.
Methods
Backbones
Weak Labels
Train
Val
Test
Scene
22.1
PCAM 29
Subcloud
43.1
38.1
KPConv 27
Scene
24,4
MPRM 29
Subcloud
47.4
43.2
41.1
MiL--seg 22
20.7
PointNet++ 21
Scene
WyPR 22
30.7
29.6
24.0
proposed
PointNet++
Scene+Image
59.1
49.6
47.4
The results also show that the 3D-to-2D loss (C)ore?") enhances the performance of 2D WSSS, from 38.0% and 33.3% to 47.8% and 39.2%, respectively. Thanks to the geometrical information provided by the 3D domain in the form of a correlation matrix, as visualized in Fig. 3 (b2), the generated 2D CAMs are much more precise at the object boundary when being trained with the 3D-to-2D loss. The results strongly support our design intentions: the 2D-to-3D loss provides rich semantics from 2D to 3D, while the 3D-to-2D loss transfers the segmentation capability of 3D to 2D.
Further, we quantitatively verify that the confidence- based scoring strategy is effective in both the 2D>to-3D and 3D-to-2D directions, by suppressing the self-supervision at inaccurate regions while preserving reliable guidance. Finally, by using the bidirectional (2D-to-31D and 3ID-to-2D) joint losses, we even further improve the semantic segmentation performance in both domains (the fourth row of each domain in Table 2 and (a3)/(b3) in Fig. 3.. It shows the superiority of the proposed joint framework, which unlocks the complementary potential of 2D and 3D domains and unifies them.
Here, we can observe that the performance of 2D WSSS is quite lower than that of 3D WSSS after apply joint losses. One of the main reasons is the limited FoV (Field of View) of the perspective camera that captures the images. When only the corner of an object is captured in the image, it is almost impossible correctly perceive the object. Therefore, classification results and corresponding 2D CAMS could be also inaccurate. To relieve the issue, during the training process, we filter 2D CAM with ground truth image-level class labels and suppress confusing regions. However, misclassification that occured in the inference phase is difficult to be addressed in our framework. Extending the proposed framework using the cameras with large FoV (e.g. omnidirectional camera) can be an interesting future direction.
4.3 Segmentation results
In Table. we compare the point cloud semantic segmentation performance of the proposed frame- work with that of the other 3D WSSS methods (29 22). This shows the superiority of the proposed 2D-3D joint framework, surpassing the other methods by more than 6% in every split. Although we use additional 2D image data and image-level class labels compared to WyPR [22), annotating additional image-level class labels is not very expensive, particularly when we consider the significant performance gain. Compared with MPRM [29] using subcloud- labels, the annotating cost is rather inexpensive at the image-level compared to the subcloud-level. while the proposed framework outperforms MPRM by significant margin. It is also noteworthy that the proposed framework does not employ the two-stage retraining technique or dense conditional random field (CRF), unlike MPRM. In summary, the proposed 2D-3D joint learning method outperforms the existing 3D WSSS methods in both terms of label efficiency and performance.
Figure 4 visualizes qualitative comparisons of the proposed framework and basclines, in both 2D and 3D domains. Compared with the result learned with a single 2D or 3D domain, our framework shows significantly better results in terms of accuracy of semantics and preciseness of segmentation. With our 2D-3D joint framework for WSSS, the complementary advantages of both domains are effectively combined. We can observe that the unified benefits of multi-domains achieve substantial semantic segmentation results, compared to uni-domain baselines. There are a few limitations such as thin objects (e.g. legs of table) in the 3D domain, fine details in the 2D domain, and sensitivity to the pose quality. Nevertheless, our framework achieves strong baseline for the 2D-3D joint learning of WSSS. Additional semantic segmentation results are provided in Suppiementary Material.
8

---

Wall
Floor
Cabinet
Bed
Chair
Sofa
Table
Door
Window
Booksheff
Picture
Counter
Desk
Curtain
Fridge
S. C.
Toilet
SInk
Bathub
Other
Baselines
Input
Ground bruths
(2D WSSS and 3D WSSS)
2DD3D joint WSSS)
3D
Proj-
3D
2D
Baselines
Ours (2D-3D joint WSSS)
Imput
Ground truths
(2D WSSS and 3D WSSS)
3D
Proj. 3D
2D
Figure 4: Qualitative comparison of 2D/3D semantic segmentation results of uni-domain baselines and the proposed 2D-3D joint framework. From left to right: input (projected) point clouds and images, results of uni-domain 2D WSSS and 3D WSSS, our joint WSSS results, and ground truths. We visualize the region of the point cloud corresponding to the images by red or blue circles. A color map of the 20 classes of ScanNetV2 [5] is provided the top of the figure, where the black color denotes the ignore index.
4.4 Generalization Capability
Most of the existing (indoor) point cloud datasets 5 have obtained the point cloud from sequence of RGBD frames, using 3D reconstruction techniques such as Structure from Motion (SfM). Therefore, in addition to the 3D point clouds, such datasets usually provide 2D images and the poses of the 2D images. The proposed joint learning framework targets such paired multi-domain datasets. Throughout Sec 4 we demonstrated that unifying the complementary advantages of multi-domain data could bring remarkable improvements in both 2D and 3D WSSS.
Nevertheless, we agree that the paired multi-domain data is not trivial to acquire in usual, especially when we want to apply our framework to the novel data of the unobserved scene domains. Compared to the existing 2D or 3D WSSS methods which require single-domain data (e.g. images only or point cloud only), the difficulty acquire the multi-domain dataset one of the limitations of the proposed joint WSSS framework.

---

Table 4: Quantitative comparisons of the generalization capability between the proposed joint learning framework and our baselines. The semantic segmentation performance (mloU, %) is evaluated on the official val splits of the S3DIS [2] dataset. The loss terms follow the denotation of Table We use confidence-baxee scoring by default. Bold numbers represent the best results.
Domain
C2d cs
Ltd cla
C2d josnt
C3d joint
Val
36.9
2D
/
43.3
v
49.5
17.5
3D
48.0
4
54.4
To relieve this issue, we experimentally verify the generalization capability of the proposed method using the S3DIS [2] dataset, as in WyPR [22] In this experiment, we first train our model on the ScanNetV2 dataset only and then test the trained model on validation set of the S3DIS dataset. We evaluate the performance only on the overlapped class between the ScanNetV2 and S3DIS datasets. Here, it is noteworthy that we perform 2D WSSS and 3D WSSS individually, without using the input data from the other domain. Thanks to practical design philosophy, we could feed the 3D point clouds only for the 3D branch and 2D images only for the 2D.
shows the generalization capability of our method by comparing it with our baselines. Similar to the tendency in Table 2 the proposed 2D-3D joint loss provides significant performance gain in both 2D and 3D domain, even on the dataset unobserved during the training. Especially, the proposed framework still outperforms the 3D WSSS performance of WyPR [22 on the S3DIS dataset (22.3%) by a large gap. The results strongly support that the complementary advantages of 2D) and 3D domains are preserved on the other dataset, even without any further fine-tuning or domain-specine specific adaptation.
Although the proposed joint framework requires paired multi-domain data during training, thanks to the demonstrated generalization capability, our framework could be scaled into processing the unobserved general data. From a practical perspective, this generalizability of the proposed method is a strong contribution to the field of WSSS.
5 Conclusion
WSSS has been widely studied for relieving the annotation cost of semantic segmentation in both 2D and 3D domains. However, the existing studies have been focused on domain-specific methods. In this paper, we propose a novel 2D-3D joint framework for WSSS in both 2D and 3D domains. Motivated by the complementary nature of the 2D and 3D domains, we devise our framework effectively transfer the benefit of one domain to the other domain in a simultaneous manner. We bridge the 2D pixels and 3D points via projection and leverage the 2D CAM as self-supervision for improving the semantic perception of the 3D CAM. To transfer the geometrical structure of the 3D scene to the 2D domain, we exploit correlation matrix of 3D features for training the 2D features. For both directions, we propose a confidence-basee and make the framework learn from the reliable regions only. With extensive experiments, we verify that the proposed 2D)-to331) and 3D-to-2DD joint loss effectively unifies the distinct benefits of both domains, while relieving the vinadvan On the ScanNetV2 dataset, the proposed framework significantly outperforms the single-domain baselines and the other state-of-the-a methods. Further, with an experiment on the S3DIS dataset, we demonstrate the generalization capability our method, which supports its practicality. We believe that 2D-3D joint learning is and promising research direction for WSSS, and this paper provides a strong baseline for it.
scoring strategy to suppress less accurate self-supervision
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea (MVI (NRF2022R
10

---

References
J. Ahn and S. Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1981-4990, 2018.
[2] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534-1543, 2016.
[3] Y.-T. Chang, Q. Wang. W.-C. Hung. R. Piramuthu, Y.-H. Tsai, and M.-H. Yang. Seakly-supervised semantic segmentation via sub-category exploration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8991-9000, 2020.
[4] A. Dai and M. Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference Computer Vision (ECCV), pages 452-468, 2018
[5] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Seannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[6] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-333 -388, 2010.
J. Fan, Z. Zhang, T. Tan, C. Song, and J. Xiao. Cian: Cross-image affinity for weakly supervised semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10762-10769 2020.
[8] J. Hou, B. Graham, M. Nicßner, and S. Xie. Exploring data-efficient 3d seene understanding with contrastive seene contexts. In Proceedings the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 15587-15597, 2021.
[9] Hou, S. Xie, B. Graham, A. Dai, M. NieBner. Pri3d: Can 3d priors help representation learning? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5693-5702, 2021.
[10] W. Hu, H. Zhao, L. Jiang, J. Jia, and T.-T. Wong. Bidirectional projection network for cross dimension scene understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14373-14382. 2021.
[11] Z. Huang, X. Wang, J. Wang. W. Liu, and Wang. Weakly-supervised semantic segmentation network with deep seeded region growing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7014-7023, 2018.
[12] M. Jaritz, J. Gu, and H. Su. Multi-viww pointnet for 3d scene understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pages 0-0, 2019.
[13] A. Kolesnikov and C. H. Lampert. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In European conference on computer vision, pages 695-711. Springer, 2016.
[14] A. Krizhevsky, I. Sutskever, and F., Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84990, 2017.
[15] A. Kundu, X. Yin, A. Fathi, D. Ross, B. Brewington, T. Funkhouser, and C. Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. In European Conference Computer Vision, pages 518-535. Springer, 2020.
[16] H. Kweon, S.-H. Yoon, Kim, D. Park, and K.-J. Yoon. Unlocking the potential of ordinary classifier: Class-specific adversarial erasing framework for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6994-799009 1194-7003, 2021.
[17] Z. Li, Z. Chen, A. Li, L. Fang, Q. Jiang, X. Liu, J. Jiang, B. Zhou, and H. Zhao, Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for apatial-aware visual representations. arXiv preprint arXiv:2112.0 12,04880 2021.
[18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740 455. Springer. 2014.

---

[19] Y. Liu, K. Chen, C. Liu. Z. Qin, Z. Luo, and J. Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 2604-2613, 2019.
[20] Z. Liu, X. Qi, and C.-W. Fu. One thing one click: self-training approach for weakly supervised 3d semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patter Recognition, pages 1726-1736, 2021.
[21] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[22] Z. Ren, I. Misra, A. G. Schwing, and R. Girdhar. 3d spatial recognition without spatially labeled 3d. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204- 13213, 2021.
[23] O. Russakovsky. J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, Bernstein. al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211- 22-222, 2015.
[24] K. K. Singh and J. Lee. Hide-and-seck: Forcing network to be meticulous for weakly-supervised object and action localization. In 2017 IEEE international conference on computer vision (ICCV), pages 3544-3555 IEEE, 2017.
[25] G. Sun, W. Wang, J. Dai, and L. Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation, arXiv preprint arXiv:2007.01947, 2020.
[26] W. Sun, J. Zhang, and N. Barnes. 3d guided weakly supervised semantic segmentation. In Proceedings of the Asian Conference on Computer Vision, 2020.
[27] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[28] Wang, X. Rong, L.. Yang, Wang, and Tian, Towards weakly supervised semantic segmentation in 3d graph-structured point clouds of wild scenes. In BMVC, page 284, 2019.
[29] J. Wei. G. Lin, K.-H. Yap. T.-Y. Hung, and L. Xie. Multi-path region mining for weakly supervised 3d semantic segmentation point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4384-4393. 2020.
[30] 7. Wu, C. Shen, and A. Van Den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition, 90:1 19-13,,33 2019.
[31] X. Xu and G. H. Lee. Weakly supervised semantic point cloud segmentation: Towards 10x fewer labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13705-13715. 2020.
[32] X. Zhang, Y. Wei, J. Feng, Yang, and Huang. Adversarial complementary learning weakly supervised object localization, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1325-1334, 2018
[33] Y. Zhang, Z. Li, Y. Xie. Y. Qu. C. Li, and T. Mei. Weakly supervised semantic segmentation for large-scale point cloud. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 3421-3429, 2021.
[34] Y. Zhang, Y. Qu, Y. Xie, Z. Li, S. Zheng, and Li. Perturbed self-distillation: Weakly supervised large-scale point cloud semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15520-15528, 2021.
[35] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929 2016.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We clarify our contributions in Section and they are experimentally verified in Section 4
(b) Did you describe the limitations of your work? [Yes] See Section 4.3
(c) Did you discuss any potential negative societal impacts of your work? [No] As far as we know, WSSS would not make meaningful negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We carefully read the guidelines and ensure it.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] But all the code and instructions will be publicly available, soon.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] But we empirically check that the proposed method robust to random seed, and we are willing to provide the results regarding it if required.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We ScanNetV2 dataset, and cite the paper.
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See Section 4.1
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 4.1
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "rTfrHwQW7FM", "reviewer": "Reviewer_JTDc", "summary": "This paper targets at weakly supervised semantic segmentation for both 2D and 3D domain. The intuition is that the weakly supervisedly features in these two domains are complementary and can benefit each other. Based on this intuition, the paper proposes two constraints, transferring the knowledge from 2D domain to 3D domain and vice versa. Specifically, the model uses 2D sudo segmentation label as supervision for 3D segmentation learning, while encouraging the 2D feature correlation map to match its 3D counterpart. During inference, the model does not need pair 2D-3D data, but instead carry out semantic segmentation on each domain separately.", "questions": "**Method.**\n\nMy foremost concern is that the proposed method needs paired 2D and 3D data, which is hard to obtain and such datasets usually already include ground truth 2D/3D semantic segmentation maps, e.g., ScanNet. Thus it is important and practical for the model to generalize to in-the-wild images or scenes. The model in the paper is learned on ScanNet, then the question is would there be a large domain gap if we apply the model to in-the-wild 2D images or 3D scenes?\n\n**Experiment details.**\n- In section 4.1, it seems the 2D &rarr; 3D and 3D &rarr; 2D consistency constraints are included from the beginning. I am curious how the model acts in the beginning, when 3D features are not good while the 2D part is pre-trained on ImageNet, will the 3D correlation maps be misleading in the beginning?\n- How to decide the occluded object in line 137?\n\n**Paper writing.**\n- I would suggest briefly introduce the Class Activation Maps in the Introduction, which would be friendly for readers without a background. For now the paper mentioned the concept throughout the Introduction but doesn't properly explain it until line 111.\n- Minor spelling erros:\n  - line 3: has &rarr; have\n  - line 42: can &rarr; to", "limitations": "The authors didn't discuss limitations or societal impact.\n\nAs stated above, I would suggest the authors to discuss the generalization capacity of the proposed method on images or scenes outside the chosen dataset.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n- This paper is well written and easy to follow.\n- The proposed method is effective and improves existing methods by a large margin.\n\n**Weaknesses.**\n- The proposed method needs pair 2D-3D data for training, which is tricky to get. The paper doesn't demonstrate if the learned model is practical and can generalize to images or scenes outside the chosen dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eUSL2GuuAI9", "reviewer": "Reviewer_WWGT", "summary": "This paper proposes a novel 2D-3D Weakly Supervised Semantic Segmentation framework that jointly targets both domains of image and point cloud. To bridge the gap between 2D and 3D domains, the 3D outputs are projected onto the image plane. And they build the consistency of the correlation matrix of\n2D features and the projected 3D features. Then they build the consistency of the 2D CAM and the projected 3D CAM.", "questions": "In consideration of the reproducibility of this work, the authors are suggested to provide their source code as supplementary.\n\nIn Figure 2, there is a projection function from the point cloud to the image. Is the image built from a point cloud? If not, please remove this projection function.\n\nWhy 3D-to-2D loss only affects the image branch, while the 2D-to-3D loss affects the point cloud branch? Can they conduct in two-ways? More comprehensive and in-depth studies are needed to justify this design.", "limitations": "The proposed method requires the perspective projection matrix as input during training, which might be difficult to obtain in a real-world applications.\n\nOnly ResNet38 and Point Net++ are employed in this paper. I suggest authors add more backbone to justify the generalization ability of the proposed framework.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) The idea is interesting and easy to follow.\n(+) The ablation studies are comprehensive.\n(+) SOTA performance on the ScanNetV2 and S3DIS datasets.\n(-) Compared with the baseline, the proposed method requires the perspective projection matrix and images as input during training. This might cause an unfair comparison in the experiment.\n(-) The Performance comparison with other state-of-the-art 2D WSSS methods e.g.[R1-R5] is missing.\n\n[R1] Xu, Lian, et al. \"Leveraging auxiliary tasks with affinity learning for weakly supervised semantic segmentation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[R2] Yao, Yazhou, et al. \"Non-salient region object mining for weakly supervised semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[R3] Wu, Tong, et al. \"Embedded discriminative attention mechanism for weakly supervised semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[R4] Lee, Jungbeom, et al. \"Reducing information bottleneck for weakly\n\n[R5] Zhou, Tianfei, et al. \"Regional semantic contrast and aggregation for weakly supervised semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "DLEfaq2eV1_", "reviewer": "Reviewer_BnBD", "summary": "The paper proposes a new task formulation and a method to jointly train semantic segmentation on images and corresponding point clouds, which can be deployed to segment any of those modalities separately. It only requires image- and scan-level class labels during training but can produce pixel- or point-level segmentation, depending on the input modality. The method makes good use of a strong semantic signal of RGB images and strong geometric signal from point clouds to disambiguate weak supervision. To achieve that, the authors introduce new loss functions to match the pre-pooling activations and the statistics on intermediate features of pixels and projected features of 3D points. That leads to significant improvement in SOTA in both 2D and 3D segmentation tasks.", "questions": "Please address the following, more important, concerns:\n* please explain how occlusions and collisions are handled when projecting 3D features,\n* please clarify the training schedule w.r.t. classification and transfer losses;\n* comment on ablating pairwise correlation (instead of the direct difference of features).", "limitations": "Limitations are only briefly mentioned. Something to discuss:\n* how sensitive is the method to the quality of camera poses used for re-projection?\n* how sensitive is the method to class imbalance; how well does it work on rare classes? Maybe report a class-weighted metric (not weighted).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes an important new task formulation and a simple but efficient method to solve it, hence I vote to accept the paper, although not very confidently as WSSS is not my area.\n\nSignificance / originality:\n(+) the paper proposes a practical task formulation: datasets often have both photometric and LiDAR modalities, and it is reasonable to leverage the synergy between them in absence of strong supervision;\n(+) the paper appears to be the first to formulate this specific task; the proposed method is simple but not trivial;\n(+) the method relaxes test-time assumptions: it can be deployed when either images or scans are available;\n(−) the paper often refer to class imbalance as a hindering factor in point cloud segmentation (e.g. l. 159): can it be solved in a simpler way with re-weighting errors depending on class frequency?\n\nClarity:\n* (±) it is possible to understand the method, although wording can be improved in multiple cases;\n* (−) occlusion handling seems to be crucial when projecting 3D features / CAM, yet the paper mentions it only briefly in l. 137; are points assumed to be disks for the purpose of culling? what radius is used? Also, the paper sometimes refers to that process as rendering – that would probably require splatting the points to cover all pixels; the described process is more like projecting and rounding;\n* (−) in projecting 3D features, the paper does not specify how collisions are handled, i.e. when multiple points project to the same pixel;\n* (−) unclear what the training procedure is: is (12) minimised with SGD directly, or are classifiers first pre-trained and then fine-tuned with transfer losses?\n\nExperiments:\n* (+) improvement over SOTA is significant;\n* (+) ablation study is sufficient: the paper ablates almost all design choices;\n* (−) the paper motivates using correlation matching by CAM loss not working well; it would be good to show it experimentally, i.e. what if 3D-to-2D loss i) just uses L1 distance on features directly, not on their pairwise correlations; ii) uses it on CAMs, not Z features.\n\n===========\n\nTypos / wording:\n* l. 19: “studies on semantic segmentation have achieved remarkable performances”;\n* l. 41: “It enables our framework can individually perform...”\n* in (3), is it \\propto instead of the equality?\n* l.153: “using maximum logit as a confidence score”: this contradicts (6) where softmax is applied to logits before taking max.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3cD6bUZNc2k", "reviewer": "Reviewer_u5nu", "summary": "This paper proposes a method to do weakly supervised semantic segmentation with 2D-3D information. Existing methods have studied the use of class activation maps (CAMs) on 2D images and 3D point clouds for semantic segmentation independently. This paper proposes to leverage 2D and 3D CAMs jointly to learn a network for better 3D semantic segmentation under weak supervision. A joint use of 2D and 3D CAMs can alleviate the sharpness issue of 2D CAMs at object boundaries with the geometric information from 3D point clouds, and similarity, can mitigate the class imbalance and data sparseness problems in 3D CAMs with 2D CAMs. To this end, a 3D-to-2D loss and a 2D-to-3D loss is proposed to enforce the consistency between the similarity matrix of the 2D and projected 3D features, and 2D pseudo label  and projected 3D CAM, respectively. ", "questions": "1. Why is it named 2D-to-3D loss in Equation (7)? Isn't the 3D activation map projected into 2D for comparison with the 2D pseudo-labels obtained from the 2D activation map, and thus should be also a 3D-to-2D loss?\n2. How much does the joint 2D-3D learning help weakly supervised 2D image segmentation? Intuitively from the explanations in the paper, 2D segmentation results should also improve, especially in the aspect of clear boundaries of the segmentations should be seen. This would be a clear proof that 3D geometry is useful for 2D segmentation, and joint learning is effective in both directions. \n3. How is the experimental results conducted only on ScanNet-V2 convincing that the proposed method can work well on other datasets? Otherwise, it is important to show results on at least another dataset. Although the proposed method is convincing, the lack of experimental results on other dataset(s) is a major weakness of the paper.", "limitations": "Yes, the limitation is discussed, but the negative societal impact is not discussed in the paper. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This paper is well-written, the core ideas are easy to follow.\n+ The proposed framework of leveraging the information from 2D images to improve 3D semantic segmentation under weak supervision is interesting. It is really good to see that the paper is written in a way to highlight how the 3D geometrical information from point clouds and the rich and dense 2D image information can compliment each other in the training process.\n+ The proposed 3D-to-2D loss on the class correlation matrix of the 2D image and the projected 3D class correlation matrix is technically sound. Similarly, the 2D-to-3D loss is a good idea to enforce the consistency between the 2D and 3D activation maps. \n+ The image classifier pretrained on imageNet can also be seen as the transfer of 2D knowledge to 3D point clouds. Specifically, there is a lack of large-scale dataset for pretraining for 3D point clouds, and thus the use of 2D information from the large scale imageNet can effectively help 3D semantic segmentation. \n\nWeaknesses:\n- While it's understandable that the final goal is to improve 3D point cloud semantic segmentation, the proposed framework of joint 2D-3D learning should actually also help improve the results of 2D semantic segmentation. However, there is no experiments done to show that the 2D semantic segmentation actually benefited from the joint learning. \n- The experiment results are only done on one dataset, i.e. the ScanNetV2 dataset. There should be other dataset with paired 2D image and 3D point cloud available for training and testing the proposed model. It also remains unclear how generalizable is the proposed method, e.g. how well it works on other dataset when it's trained on ScanNetV2? The incompleteness of the experiments somehow weakened the paper. \n- The proposed method requires 2D image-to-3D point cloud pairs. It would be better if the method can also leverage unpaired images and 3D point clouds for training. However, this limitation seems to be difficult to alleviate as the method dependents heavily on the projection. of 3D to 2D. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept