Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced "inflation" (2D CNN kernel to 3D one by repeating on the 3rd axis) to transform a 2D image pre-train backbone into a 3D version so that 3D point cloud task can be benefit from 2D image pretraining. Detailed experiments are performed to address this idea. Results show minimal finetuning efforts can achieve competitive performance on 3D tasks. The authors are kind of "surprised" by these results.
main_review: 
Strengths: 
1. Idea is good and helpful for the 3D point cloud field.
2. The experiments are designed to test different advantage of this idea. The results support the arguments. 

Weaknesses / questions: 
1. Since such idea has already used in other 2D-3D domain, the novelty is limited. Such paper needs extra emphasis on discussing why it works and ablation studies. 
  - In discussion section, the author provides some experiments, but the result seems quite random (figure 4). The "shape representation" are better transferred seems make sense, but why? The author tried to explain the finding but seems not quite convincing.
  - When designing inflation, is there a difference on which axis to inflate. x, y or z axis. This is different to video since 3 axis is symmetric. 
2. The biggest concern is the parameter sizes and network structure.
  - In 4.1, the author did not compare the parameter size. I doubt the fairness of the comparison with the baseline since it is possible that the performance gain is fully from the increasing parameters. 
  - In 4.2, there is a possibility that the ResNet structure is not good structure to train on point cloud making it quite bad on scratch training. For pointnet++, the training method is quite strange. 
3. The writing for this paper is interesting... not formal enough. The author used 10 "surprising(ly)" in this paper...
summary_of_the_review: This is an interesting idea. This paper provided detailed experiments to test different advantage of this idea. However, the discussion on why it work seems not quite convincing. Also, part of the experiments missed some details. Some conclusions may not correct.
We expect the author provided more convincing discussion and more comprehensive comparison detail.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a pipeline for transferring convolutional network weights that are pre-trained on 2D images to 3D convolution networks. The proposed approach is to inflate the 2D kernels to 3D kernels similar to video models. The experiments are conducted on 2D image datasets such as ImageNet, and 3D datasets such as ModelNet.
main_review: Strength:

- This paper investigates a new problem of transferring 2D pre-training to 3D.
- The presentation is clear and easy to follow.
- The experiments are extensive. The performance gain using the 2D pre-training is notable and non-trivial, showing the proposed weight transferring technique has the ability to convert useful learned information on 2D to 3D.

Weakness:

- It's good that the authors showed some visualizations to help understand the proposed framework. However, as the authors pointed out, the visualization cannot show why the transfer works or what information can be transferred.
- The last sentence on page 8 is questionable: the conclusion of "shape representations are better transferred from image to point-cloud" cannot be inferred from overall dataset performances. Also, what does "shape representations" mean? 

summary_of_the_review: This paper proposed a solution to a new problem of transferring 2D weights to 3D. The solution works well on many dataset pairs. Though some claims made in the paper are questionable, this paper deserves to be published at the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to transfer the 2D image features for 3D point cloud understanding tasks. Specifically, the proposed method first inflates the 2D convolution filters from a pretrained model to 3D and then optimizes only input, output and optionally batch normalization layer.  Detailed experiments show that the proposed finetuned-image-pretrained models can improve the performance and data efficiency.

main_review: Strengths
- The paper proposes a model to connect 2D and 3D representations in transfer learning.
- Detailed experiments show that the proposed model improves the performance and data efficiency compared to training from scratch.

Weaknesses
- Will larger dataset leads to better performance? A lot of comparisons between results from datasets of different size are missing in Table 1. 
- The paper shows that using 2D pretrained features improve performance. However, results of state-of-the-art task specific methods should also be listed. In addition, it would be better to compare with different self-supervised learning method.
For example, for shape classification on ModelNet40:
Foldingnet: Point cloud auto-encoder via deep grid deformation. CVPR 2018
Pointcontrast: Unsupervised pretraining for 3d point cloud understanding. ECCV 2020
-  Explanation why this kind of inflating 2D filter to 3D is reasonable. There is a huge domain gap between 2D dataset and 3D dataset. It's still unclear why this kind of transfer is reasonable. It would be better to add theoretical analysis besides experiments.
summary_of_the_review: Although the experiments in the paper show that the proposed model can improve the performance and data efficiency compared to training from scratch, the overall experiments results are not strong enough. In addition, it's still unclear why this kind of 2D to 3D transfer is reasonable. I am inclined to reject.

Update: After reading the feedbacks from the author, I raise my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper described an experiment of transfer learning between image and point cloud data. Despite of apparent dissimilarity between the two domain, the author was able to use structure and weights from pertained models in image domain to get reasonable performance in point cloud tasks like classification and segmentation. The author also made comparison of using pertained weights versus training from scratch, the result shows training pertained weights improves data efficiency. The proposed method is also proven to be useful when the training data is limited by training under few-shot learning setting. 

main_review: Pros: The idea of transfer learning between image and point cloud data seems to be novel and surprising.

Cons: 
1. The idea of inflating model weight from 2D to 3D is not new[1].  
2. The performance on point cloud segmentation on SemanticKitti dataset, while is reasonably well, still has gap to the state-of-the-art(This one is minor)
3. The effectiveness of pretrained weight would be more convincing if the author could do further ablation study to initialize new model with expanding subsets of layers and exploring the effect on model performance. It will indicate which part of the pertained model is really useful in this transferred learning setting.

=====
Update after rebuttal: After reading the author's response and the new experimental results amended by the author, I raise my score to 6.

[1] Carreira, Joao, and Andrew Zisserman. "Quo vadis, action recognition? a new model and the kinetics dataset." proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
summary_of_the_review: Overall the paper has interesting experimental result on transfer learning between image and point cloud data. The authors did a good job explaining the idea, concepts, procedures and experiments, showing the effectiveness of the transferred model on performance. data efficiency and usefulness on few-shot setting. The experiment itself could be further extended though.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed to perform the 3D point cloud understanding based on the pretrained 2D image models. Simple inflation technical, same as the work in I3D, are used to convert the 2D convnet to 3D convnet.

Extensive experiments performed on the point cloud classification and segmentation tasks demonstrate the effectiveness of the work.

main_review: Strength:

The paper is simple yet very interesting, which simple inflate the 2D convolution kernels to 3D kernels to learn the representation of the 3D point clouds. 2D images and 3D point cloud are of great differences. Thus, it seems to impractical to simply performed the inflation and make it work. However, the authors proposed one simple idea and demonstrate that the idea does work.  I like the simple idea and would like to recommend the acceptance.


Weakness:
1. Some detailed information is missing. Therefore, I am not able to follow the detailed process. In Table 1, the authors list the performances of pointnet++ and the performances. I am curious how to performance the inflation in point net++
2. The authors perform the experiments on the classification and segmeantion. What about the performances on detection task.




=========
Post-rebuttal

I have read the authors rebuttal as well as the other fellow reviewers' comments and the corresponding discussions. 

I agree with the other fellow reviewers' comment, 2D inflating to 3D has been studied in video action recognition. However, to me, inflating 2D CNN of image for 3D pointcloud is new. Therefore, I do think the paper is of novelty. 
Also I agree with the other reviewers' comments that the discussion of why the proposed method works does not been clearly addressed. 


Considering this, I am lowering my rating from 8 to 6.

summary_of_the_review: It seems to the reviewer that this is the first paper to use 2D image pertained models for 3D point cloud understanding. Although the idea is simple, the work is interesting and inspiring. Therefore, I am recommending the acceptance.
review_title: null

Meta-review: This paper proposes to transfer the image-pretrained model to a point cloud model by inflating 2D convolutional filters to 3D convolutional filters and finetuning the inflated image-pretrained model, so that 3D point cloud tasks can benefit from 2D image pretraining. Extensive experiments are conducted to validate the effectiveness of the proposed method. Even though the performance gain using the 2D pre-training is notable, the novelty of the paper is limited since inflating 2D model to 3D video action recognition has been studied, and theoretical understanding of the proposed model is lacking. During the rebuttal period, the authors addressed most of the reviewers’ concerns by conducting additional experiments. Even though the performance is compelling, all reviewers agree that the novelty for the paper is limited and the discussion on why this method work is not convincing. Meanwhile, one reviewer points out that some claims made by authors are not well supported. Besides, one reviewer points out that the paper might have a broader impact in a computer vision conference but only provide a limited contribution to the ICLR community. After an internal discussion with reviewers. the AC agrees with the reviewers on their judgments and recommends rejecting the paper because of the limited novelty of the paper.
Decision: Reject