Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper addresses the challenging task 4D panoptic segmentation by utilizing both LiDAR and RGB inputs. To this end, the authors propose a novel method called 4D-Former that leverages both modalities to predict semantic masks along with temporally consistent object masks for the given input sequence. This paper employs concise queries to encode semantic classes and objects from both modalities. Further, this work proposes a learnable association mechanism to associate objects over time that considers both appearance and spatial location. 
Lastly, 4D former achieves SOTA results for nuScenes and SemanticKITTI dataset

review: - The paper is easy to understand and logically organized into sections.
- The figures complement the text and help in understanding the tasks easily.
-This work is one of the first to address the task of 4D panoptic segmentation in a multi-modal setting.
-4D-Former achieves SOTA results on both Semantic KITTI and Panoptic-nuScenes.
-The ablation experiments exhibit the usefulness of each architecture design choice quantitatively. 

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: To the best of my knowledge, this study represents the first effort in tackling the multi-modal 4D panoptic segmentation task. The results achieved by the 4D Former model are convincing. It is my anticipation that this work will lay the groundwork for subsequent advancements in the realm of multi-modal research, serving as a good starting point.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a system that incorporates 3D panoptic segmentation, 4D tacking, and multi-modal fusion to fulfill multimodal 4D Panoptic Segmentation. It fuses image information in the 3D feature extraction stage and decode stage. It also designs a Tracklet Association Module to associate object tracks.
review: Strengths:
1. Comprehensive Approach: The paper integrates different modules to address the challenges of multimodal 4D panoptic segmentation, making the system complete and sensible.
2. Clear Methodology: The methodology is presented in a straightforward manner, allowing readers to easily follow the proposed approach. Figure 1 provides a clear overview of the system's structure, while subsequent subfigures and sections delve into the details.
3. Analysis of Limitations: The paper acknowledges the limitations of their methods, providing an insightful analysis of the deficiencies and inspiring future work in the field.

Weaknesses:
1. Fragmented Components: The proposed components sometimes lack sufficient insight into the problems they aim to solve. This issue is also evident in the related works section. Instead of highlighting the novelty of specific fields, the paper introduces components from other fields to showcase superiority without thoroughly addressing the underlying problem.
2. Incomplete Experiments: This paper misses some important comparisons and metrics in the experiment part. While the focus is on 4D panoptic segmentation, it is crucial to include comparisons with 3D panoptic segmentation methods, as it serves as a fundamental task for 4D panoptic segmentation. And the metric Panoptic Quality (PQ) is both important in 3D and 4D panoptic segmentation. Besides, 3D panoptic segmentation is more mature. This method, incorporating multi-modal and temporal information, is expected to surpass previous 3D panoptic segmentation to validate its effectiveness. The comparison on the Semantickitti dataset is also missing in terms of PQ.
3. Lack of Related Works: The paper should discuss more related works, particularly in the field of 3D unified panoptic segmentation, which forms an important foundation for this work. Including works such as PUPS (Zhao et al., 2023) and P3Former (Xiao et al., 2023) would enhance the comparison and provide a broader context for the proposed system.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Considering the limitations as stated in the weaknesses, my rating is Weak Reject.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The task of this paper is to conduct semantic segmentation, instance segmentation and tracking of Lidar point clouds across time, and the paper proposed to incorporate multi-view camera images as input to enhance the features and improve the performance.
review: Originality:

This paper is the first to use monocular images for point cloud 4D panoptic segmentation. From the technical perspective, this paper combined recent successful solutions for 3D point cloud segmentation and multi-view monocular 3D object detection together for this new task, and the authors also proposed a novel method for temporal association of objects. The discussion of related work is proper. A missing reference is the current SOTA of 4D panoptic segmentation on semantic-kitti: EQ-4DSTOP (https://arxiv.org/pdf/2303.15651.pdf). Another work Panoster (https://arxiv.org/abs/2010.15157) presented a post-processing-free instance prediction module, which is also relevant to this paper's contribution. Overall, this paper has decent novelty. 

Quality:

The paper presents a well-assembled framework to process multi-modal inputs to conduct the segmentation task. Besides, I found three designs particularly interesting. First, the point-level fusion has a pseudo fusion module for points without RGB correspondence so that the features are better aligned with those with RGB information, with a corresponding loss function. Second, it employed a query-based approach for cross-modal fusion and segmentation. In terms of segmentation, it naturally solved the problem of consistency between instance and semantic labels. Third, the tracklet association module goes beyond IoU-based association, making use of both the semantic and geometric features. The experiments validate the effectiveness of the methodology. Overall, this work is complete in the standard of a conference paper. 

Clarity:

The paper is overall well written, but I found the section of soft-masked cross-attention confusing. The K and V do not match the dimension of M_v. Since K and V are of shape TxD, representing the query features (Q also query features), while M_v is the attention matrix between queries and voxels, it is hence unclear whether this module is self-attention or cross-attention, and where this module is placed in the overall network. 

Significance:

The results are important. It pushed forward the state-of-the-art 4D panoptic segmentation in nuScenes, and more importantly showed a viable way of handling this task with additional image input. 

Relevance:

The panoptic segmentation task with temporal knowledge is a very relevant problem to the robotic community. This paper brings new modality to this task and obtained good results, thus I believe can attract interest of CoRL audience. 

Limitations:

A major weakness of this paper is the performance on the SemanticKITTI dataset. The performance is only marginally better than the baselines while having extra monocular image input, which could make people question the effectiveness of the multimodal fusion. However, the authors did provide a paragraph of discussion on the possible reasons, which I found acceptable. 

Strengths:

1. First to propose to solve point cloud 4D panoptic segmentation with camera modality. 

2. Reasonably designed framework, fusion, prediction, and association modules. Incorporated recent progress in different subfields. 

3. Good results on the nuScenes dataset. 

Weakness:

1. Confusing introduction of the soft-masked cross attention module. 

2. Mediocre SemanticKITTI performance.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposed to solve the 4D panoptic segmentation task in a new way by incorporating extra camera input. The method has good incorporation of existing solutions in building the overall framework, and also proposed novel submodules to improve the performance. The experimental results are good overall, although not as strong on the SemanticKITTI dataset. I would like to see the soft-masked cross attention clarified, and it is a good paper overall. 

After rebuttal:

I'd like to keep my score at Weak Accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a multi-modality transformer-based framework for 4D LiDAR-based point clouds panoptic segmentation. Specifically, it takes multi-view images and LiDAR-based point clouds as input, uses a multimodal encoder to fuse image and point features, and subsequently, designs a query-based unified decoder for panoptic segmentation prediction. Furthermore, a tracking module named "Tracklet Association Module (TAM)" is devised to associate tracklets across longer frame gaps in the multi-frame panoptic segmentation problem. The proposed framework achieves new state-of-the-art on two popular benchmarks, nuScenes and SemanticKITTI, and detailed ablation studies show the efficacy of each component.
review: Strengths:

- This paper proposes the first multi-modality transformer-based framework for 4D LiDAR panoptic segmentation, and it is also effective and practical, as shown in the experiments.
- The basic idea is easy to follow, and the overall methodology is simple yet effective.
- Detailed ablation studies show the importance of each component.
- It achieves new state-of-the-art and surpasses previous methods by a large margin, validating its effectiveness.

Weaknesses:

- There are some missing related works regarding transformer-based methods for LiDAR-based panoptic segmentation, such as PUPS[1] and P3Former[2]. From this perspective, the contribution of a unified query-based decoder for LiDAR-based panoptic segmentation should be weakened. Technically, there are few new things in the multimodal encoder and transformer-based panoptic decoder except for incorporating multimodal data fusion during the procedure.
- (Minor) It would be better to have more ablation studies for alternative model designs. For example, how about sharing several MLP layers in Eq. (1), i.e., using the same MLP layers to encoder point clouds and using additional layers to fuse part of them with image features?
- (Minor) The overall training is two-stage and there are no experiments comparing two-stage and one-stage training, although the author gives the reason and claims end-to-end training may yield better results.

[1] PUPS: Point Cloud Unified Panoptic Segmentation

[2] Position-Guided Point Cloud Panoptic Segmentation Transformer
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposes an effective transformer-based framework for multi-modality 4D LiDAR-based panoptic segmentation. Although there are minor issues regarding missing related works and potentially weakened contributions regarding the transformer-based framework, the overall paper has strong performance and detailed ablation studies, making it potentially beneficial to the community. Therefore, I vote for weak accept.
review_title: null

Meta-review: This paper proposes an approach for 4D panoptic segmentation from camera images and LiDAR scans. The reviewers find the paper to be clearly written and addressing an important problem. Some concerns are convincingly addressed in the rebuttal whereas some regarding the incremental nature of the contributions still remain. Nevertheless, all the reviewers agree that the work is technically correct and well executed.
Decision: Accept (Poster)