Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a knowledge distillation strategy to improve point cloud classification. It uses images of point clouds to train an image classification network. The representation from this network can be distilled into any existing point cloud network. The distillation requires multi-stage training including -- training the image classifier, training point generator from images and training the point-cloud network assisted by the image classifier. Experiments show the effectiveness of this scheme. 
strengths_and_weaknesses: Strengths:

- The paper achieves impressive performance on both the ModelNet40 and ScanObjectNN datasets.

- Gains due to knowledge distillation seem to consistent across various networks (Table 1, Table 2)

Weakness:

- The method archives performance improvements; however, the training pipeline seems to become more complicated with many training stages. This could be a potential limitation. It would be useful if the paper discusses this and potential ways to mitigate it.

- It would be nice if the paper could provide some mean +- std measures. This could be done by running the same experiments multiple times (with random initialization) and reporting the mean and variance. This is particularly important as point-based benchmark methods can have significant variations across runs.

- What is performance of the teacher networks on the two datasets? Also, is the teacher network potentially better because of the additional image to point cloud task? I asking this because as shown in the paper 3D projections could be used and these 3D projections can be created from the point cloud, which is available at test time. 
questions: Refer to the weakness section for questions. Overall, I am (weakly) positive about the paper. I will update the score based on the rebuttal.
limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper shows a 2D-to-3D distillation framework PointCMT that improves 3D classification through cross-modal training. Despite the performance improvement in two classification datasets,  the reviewer has concerns about the effectiveness of PointCMT: (1) the theoretical argument of this approach; (2) some important ablation studies are missed; (3) more serious applications like segmentation should be considered.
strengths_and_weaknesses: PointCMT shows cross-modal training that improves performance. Check questions for the weakness. 

questions: #### Majors:

1. PointCMT might not be theoretically sound. Feature loss (Eqn. 5) will be zero if global features of image and point cloud are equal, since the same CMPT is used for both inputs. Classifier loss (Eqn. 7) will be zero if global features of image and point cloud are equal, since the same classifier (Cls^{pts}) is used. Therefore, the whole regularization term in PointCMT is just to make two global features identical, which means the PointCMT can be replaced by a simple regularization term that makes two features closer to each other.
2. Table 5 needs more clarification. It is not clear which component is ablated. Take row 1 (baseline) and row 2 (Hinton) for example, is it only the logits distillation part be ablated? In other words, are you using eqn. 6 as a replacement of eqn. 7. Or, is the feature distillation ablated such that you are studying the different possibilities of Eqn. 5? Or, both?
3. An ablation study on with/without CMPG should be added if Table 5 is not for it.
4. An ablation study on using Eqn. 6 instead of Eqn. 7 should be added if Table 5 is not for it.
5. An ablation study on values of tradeoff parameters (alpha and beta) in L217 Eqn. should be added. 30 and 0.3 are not common values. How much variance can the parameters cause to the performance? Is that possible that you outperform other distillation techniques simply because the parameters are not good for them in Table 5.
6. Overclaim. There is already a work in 2d-to-3d knowledge distillation. [1]
7. Application is limited to classification only. Classification is the most trivial task in point cloud understanding and the most unimportant task. The classification benchmark datasets in this work are rather small-scale and networks are easily saturated in performance. More serious applications such as segmentation should be considered.

[1]  Yueh-Cheng Liu, etc. Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining

#### minors:

1. Revise CMPG description. For L181, you should revise to something like: the cross-modal point generator (CMPG) is used to map the global feature representation acquired from images/points into the Euclidean space. Note that CMPG will be used to transform both images and point clouds after pretraining. Since it is the first time you define CMPG, the function of CMPG should be made clear.
2. Better illustration for Fig. 2. Fig.2 (a) dash lines from global feature to feature Enhancement can be illustrated in other colors, \eg blue, since it requires the CMPG module and thus should be highlighted differently for easier understanding. In (b), image features should be changed to global features from images or points.
3. Highlight training cost instead of inference speed. It is weird to highlight the inference speed in L257, since PointCMT only influences training speed and has no effect on inference speed. The inference speed is totally dependent on the point cloud network used in PointCMT. The author should highlight training costs instead.
4. The authors should explicitly highlight that for all experiments, they trained baseline with and without PointCMT using the same optimization techniques, data augmentation, and evaluation techniques. For example, SimpleView found these tricks matter a lot in performance.
5. Any clue why PointCMT improves PointNet++ more than other networks?
limitations: The authors did not include limitations and potential negative societal impacts. Limitations, e.g. only for classification, can be added.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a cross-modal training scheme called PointCMT which utilizes both 3D point cloud and synthetically rendered or view projected 2D image to boost point cloud classification performance. The cross-modal training is formulated as a knowledge distillation problem and can be easily combined with existing 3D point cloud based algorithms such as PointNet++ and PointMLP. Experiment shows that PointNet++ with PointCMT can achieve 1.0% and 4.4% accuracy improvements on ModelNet40 and ScanObjectNN benchmark dataset, respectively.
strengths_and_weaknesses: Strength
1. The idea of using view projected 2D image for 3D point cloud classification has been presented in SimpleView[14], but the scheme of cross-modal training using both 3D point cloud and view projected 2D image is novel and interesting.
2. A cross-modal point generator is proposed to solve the cross-modal knowledge distillation problem when the feature distribution of 3D point cloud and 2D image is different and complementary
3. The analysis and proof on the discrepancy between the discriminative image and point cloud features is good.

Weakness:
1. There are some concerns on the experiment: 
    a) In table 1, it seems that even equipped with PointCMT, The PointNet++/RS-CNN/DGCNN seems no better than SimpleView [14] which achieves 93.9% overall accuracy with speed of 2208 samples/second, does this mean that cross-modal information fusion (3D point cloud + 2D view image) may not help on the ModelNet40 dataset and only 2D view image is enough? 
    b) In table 6, the performance of "with projection" achieves 91.8% accuracy on OBJ_ONLY, which is about +4.3% improvement according to Table 2, but the performance of "with Project with color" only achieves 90.7% accuracy, dropped by 1.1%, does this mean the use of point color information hurts performance of cross-modal training? 
2. Since the cross-model training may increase the training time, it would be good to include both training cost and performance gain in the comparison experiment

questions: 1. How about the generalization capability of cross-modal training? for example, the performance of training on ModelNet40 and test on ScanObjectNN or vice versa, as similar experiment has been done in [14]
limitations: There is no potential negative societal impact found
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a method to enhance point cloud classification models by exploiting training on images. Features extracted from images may be able to capture complementary details to what is typically learned by training on point clouds alone. A distillation procedure is presented so that such complementary features can be integrated in a model that only processes a point cloud at test time.
strengths_and_weaknesses: Overall, the method is sound and generally well explained. A few mistakes in the use of English are present and could be fixed (for example page 2 line 55 "Effectively" --> "Effectiveness").

Strenghts:
- sensible approach to improve feature learning by leveraging complementary features that are more easily extracted from images
- the distillation approach allows to still work in a single-modal setting at test time
- good results showing a benefit from the proposed technique on state of the art architectures
- good ablations to validate each proposed component (CPMG/Feature enhancement)

Weaknesses:
- distillation allows to train the point cloud encoder to extract those features that are easily extracted from images but generally escape classifiers trained on point clouds, despite those features being present in the input point cloud. However, because at test time only the point cloud is used, it is not possible to exploit truly complementary features that an image might carry and that are not present in the point cloud data. This would only be possible in a truly multi-modal setting, also at test time
- the training of CPMG requires paired data with the image and point cloud of the same object. This is somewhat constrasting with the motivation of the work in page 1 line 34 highlighting the potential difficulty in having paired multimodal data  
questions: - why have chosen the EMD instead of the Chamfer distance as a metric for your loss function? 
- it seems you reference the published results for PointMLP. Have you been able to reproduce them? Based on experience and online discussions (https://github.com/ma-xu/pointMLP-pytorch/issues/1), it seems they are quite tricky to reproduce
limitations: I would like to see an extended discussion about the single-modal setting, in which only point cloud data are used at testing time (although multiple modalities may be available for training) considered in this paper against a fully multi-modal setting.
review_title: null

Meta-review: metareview: The paper focuses on the problem of distilling semantic/representation knowledge from 2D images to help further enrich 3D point cloud representation. It received four detailed reviews and a healthy interaction between authors and reviewers ensued. In that back-and-forth, the reviewers clearly stated the weaknesses/issues they saw in the paper, which the authors resolved for the most part through their additional analysis, explanation/clarification, and experiments (e.g. on ShapeNetPart). As such, some reviewers raised their initial review score. Overall, the paper targets an interesting topic in 3D representation learning and it exceeds the bar of contribution and impact expected in NeurIPS papers. The authors are expected to include their additional experiments and discussions in the final version of the paper.
confidence: Certain
Decision: Accept