Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The submission presents a self-supervised method for fusing LiDAR and camera data in self-driving applications. The approach builds upon MAEs to reconstruct masked LiDAR data by fusing features from dense spherical LiDAR projections and features from fish-eye camera crops with a similar field of view.
strengths_and_weaknesses: ### Strengths:
1. The submission is well-motivated and clear.
2. The proposed method is simple and requires less overhead compared to related works.

### Weaknesses:
- **Reproducibility** : Although the authors have used the open-source KITTI dataset, they have not provided the necessary code to reproduce the results. Furthermore, the training-validation split seems small (92%-8%), raising concerns about the generalizability of the approach to different random samplings of the dataset.
- **Correctness** :  While the authors provide qualitative results demonstrating the effectiveness of their approach, they have not conducted a quantitative evaluation of the method against at least one of the existing related works to determine its correctness.

suggested_changes: 1. While I find the proposed idea simple yet promising, I suggest that the authors compare their approach with at least one baseline to provide a justification for their claims.
2. Additionally, in the "Introduction" section, the authors state that their "contributions are twofold...we perform evaluation on a self-driving dataset". However, performing evaluation on a dataset is not a contribution but rather a means of justifying the correctness of the approach. I recommend that the authors rephrase this sentence.
3. **Clarification**: In Section 2 "Method," the authors state that "our LiDAR encoder achieves the same computational efficiency as a vanilla MAE encoder." However, in the appendix, they mention that they used a smaller patch size (8x8) compared to the vanilla MAE. Therefore, the computational efficiency should be different. I recommend the authors provide further clarification in this regard.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The paper proposes a MAE-inspired method for self-supervised representation learning with paired LiDAR and camera data. The LiDAR data is masked and masked areas are reconstructed via cross-attention with camera data.
strengths_and_weaknesses: Strengths
* The paper is clearly written and Fig 1 is very useful for giving an overview
* Good references to existing literature
* The cross-modal application of MAEs to LiDAR+camera data is an interesting direction to investigate as self-supervised learning could help to make perception models much more data efficient.

Weaknesses
* It would be best practice to keep a held-out test set - to help avoid overfitting on a validation set. 
* It wasn't clear to me that range and intensity of the lidar were being reconstructed until Fig 3. It may be worth adding some detail about that?
* If camera features are held out (Fig 2), how can reconstruction be achieved at all? Fig 1 suggests there is no self-attention mechanism for LiDAR features - should this be added? In that case would that be like training a regular MAE on LiDAR-only data?


suggested_changes: I rated GS for the moment due to some of the items in the weaknesses section above which require clarification. If these can be reasonably addressed, I think the paper will be between HP<>CCR. To expand the work into a full paper in the future, it would be interesting to run a comparative study of vision-only vs lidar-only vs the cross-modal approach here, and to better understand the usefulness of the learned representations - e.g. using them to measure improvements in few-shot learning of 3D detection models. A closely related work from ECCV last year which might provide further inspiration is https://multimae.epfl.ch/

Minor suggestions / typos
* "lacks the semantic richness" - this term is a bit vague, maybe more specifically e.g. lacks the photometric qualities and sampling density of camera data?
* "Therefore, we reduce the learned transformations to moderate perspective transformation and do not require additional modules to generate dense LiDAR representations." - this 
* up-on -> upon
* 16x16 -> $16\times16$, same for 8x8
* the main text says a patch size of 16 was used, the appendix says 8? Is one for LiDAR, the other for camera?
* querys -> queries
* two sentences are a bit repetitive and could be combined - "Accordingly, our model learns to reconstruct LiDAR data by querying camera features. Finally, a MAE decoder reconstructs the spherical LiDAR projections from the fused tokens."
* Fig 3: difference/error images may be interesting to plot 
* Figs 2+3: add text for what each image row is to make comparison easier for reviewers
* why train for 57 epochs? What was the stopping criteria?
review_title: null

Decision: Invite to revise