Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new technique called PathFusion that aims to fuse the LiDAR and camera features, especially the deep layer features. In order to fuse the features at multiple stages, the method design a path-consistency loss to lift and align the 2D features in the camera branch to the 3D features in the LiDAR branch. With the path-consistent regularization, the PathFusion further improves the detection accuracy on both KITTI and NuScenes datasets compared to the baseline Focals Conv method. 
strength_and_weaknesses: Strength: 
+ The motivation is good. Compared to the previous methods that only fuse the features in shallow layers, this method considers the fusion of deep-layer features which is valuable to multi-modal detection.
+ The method is easy to follow and reproducable. The definition of path consistency loss is simple. 
+ Experiments on KITTI and NuScenes datasets demonstrate the robustness of the proposed method.

Weaknesses:
- The contribution is slightly incremental. The main part of this work is the design of path consistency regularization.
- The forced alignment of the camera and LiDAR features is doubtful. As mentioned in the introduction, the LiDAR branch mainly extracts the geometry features, while the image branch produces the semantic features. Though the 2D semantic features are transformed into the 3D space, the distribution of lifted features remains intrinsic differences. Instead of aligning the features, learning an accurate transformation is more critical. 
- The description of the lift operator is unclear. Are the 2D features in different stages upsampled to the origin resolution before the lift transformation? Is that possible to adjust calibration parameters to directly lift 2D features to 3D features?
- In practice, the point cloud and image cannot be aligned perfectly due to the synchronization error and motion distortion. For these cases, the shallow fusion is not reliable. Is deep fusion helpful in such situations?
clarity,_quality,_novelty_and_reproducibility: Please see the strength and weaknesses.
summary_of_the_review: This paper present PathFusion to fuse the multi-stage features between LiDAR and camera branches to improve 3D detection. A path consistency regularization is designed to align the 2D image and 3D lidar features. Experiments validate the performance of the proposed method. However, the contribution is limited, and the practice of forced alignment is doubtful. Considering the above factors, I vote for the weak reject in this round of review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Empirical results show deep 3d fusion of lidar and camera has inferior performance compared to early stage fusion. A possible reason is the misalignment in deep fusion when deep features have large receptive fields and include overlapping regions of large areas. This paper aims to enable path-consistency between early features and deep features to enforce alignment in deep features. Path consistency is negative cosine similarity or l1 distance between features of lifted 2D features and 3D features whose input is lifted 2D features from previous stage . The experiments show 1.2%+ mAP on nuscenes test set and 0.6%+ on KITTI AP3d moderate compared to its baseline Focals Conv.
strength_and_weaknesses: Strength:
The approach is reasonable. Paper is generally well-written. Ablation study is extensive.
Weaknesses:
- The biggest concern is the results only show marginal gain compared to baseline. Especially on KITTI the results are known to be volatile.
- This paper claims other fusion methods are not optimized in model complexity but no latency comparison is provided.
clarity,_quality,_novelty_and_reproducibility: This paper is generally well-written and I believe it's easy to implement. 

A few questions to better understand this paper:
1) In figure 4(b),  should the input of F_{t+1} be concatenation or summation of y_t and P_t\circle x_t because the 3D branch is handling fused features? If so, how to separate lifted 2D features from this 3D branch? And are feature channel size the same across different stages?
2) Is data augmentation applied in KITTI experiments? How does the proposed method handle misalignment caused by data augmentation especially gt sampling? 

A minor question: in Section 6 paragraph PathFusion settings 'back-progates to parameters in stage i and the corresponding lifting operator P_i'. The lifting operator is upsampling and mapping the center of 2D regions to 3D voxels. There should be no parameters in lifting operators?
summary_of_the_review: The biggest concern is the results only show marginal gain compared to baseline. Without latency comparison to other fusion methods and other sota lidar-only methods, the proposed method has not justify the potential of wide application.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submitted manuscript considers the task of fusing camera and lidar data to produce 3D object detections. The paper focuses on a specific type of fusion mechanism where 2D camera features are mapped to 3D for inference and discusses the pros and cons of late vs. early fusion. The main contribution of the paper is a loss function that penalizes inconsistencies across different paths through the network. The experiments demonstrate that the proposed loss function can lead to improved performance on both the KTTI and NuScenes benchmarks. 
strength_and_weaknesses: The proposed idea (the loss function) is interesting, and I believe it has some merit. The fact that the loss function improves performance on both KITTI and nuScenes data is, of course, a significant strength. At the same time, it is also a simple idea, and I don’t think the authors have done enough to motivate that it is important. 

A few aspects that would be good to discuss further: 

* How general is this idea? The authors only demonstrate it on a very specific architecture called FocalsConv, and I personally doubt that FocalsConv will be commonly used in the future. Can the authors mention other networks that the loss function could be combined with to motivate that the generality of the idea goes beyond FocalsConv. Ideally, the authors should include such examples in the evaluation, but I am not that is necessary. 

* What is the relation between the proposed loss and self-supervised learning? My understanding is that the loss function can be used in a self-supervised manner. In the current version, this has not been discussed or explored in a satisfying manner. 
clarity,_quality,_novelty_and_reproducibility: As for the clarity and quality of the text itself, the authors initially state that they build on FocalsConv and introduce an additional loss function for path consistency. However, from section two and onwards, the separation between proposed novelties and existing methods is far from obvious. I personally find that this is an important weakness in the current version of the manuscript and something that I hope the authors can improve substantially; it should be easy to fix. 

Another important aspect is that I don't think the description of the state-of-the-art is somewhat misleading in the sense they argue that most methods are shallow, whereas many modern methods are arguably deep according to the terminology used in the paper. 
summary_of_the_review: To conclude, 

*  (pros) the proposed loss function is intuitively appealing,
*  (pros) the results demonstrate that the loss function improves performance,
*  (con) the presented solution is not state-of-the-art,
*  (con) some aspects are not sufficiently well explored,
* (con) the paper could be more well written.

My current recommendation is marginally below acceptance, but I would be willing to change that if the paper is adjusted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of fusing 2D image frame features and 3D LiDAR features in a way that maximizes their alignment. The paper points out that in the input resolution there is an exact alignment between 3D scene locations and 2D pixel locations provided by camera calibration and the camera projection equations. However, during 2D and 3D featurization, this precise alignment is lost, and later fusion of 2D image features lifted in the 3D grid to be fused with LiDAR features is suboptimal due to the coarsening of 2D and 3D grids and their non precise alignments. The paper proposes multiple paths of fusing 2D image and LiDAR features in early or later layers, and imposes consistency losses between these paths, to encourage alignment of 2D and 3D features even at deeper layers. Though this design choice makes sense, the empirical impact in performance is minor, as reviewers point out, and concerns a specific model architecture. The proposed design choice is motivated from a performance viewpoint in the introduction of the paper, and since its empirical validation is not convincing the reviewers and area chair find that it is not noteworthy enough to be published in ICLR. 
justification_for_why_not_higher_score: Though this design choice makes sense, the empirical impact in performance is minor, as reviewers point out, and concerns a specific model architecture. The proposed design choice is motivated from a performance viewpoint in the introduction of the paper, and since its empirical validation is not convincing the reviewers and area chair find that it is not noteworthy enough to be published in ICLR.
Decision: Reject