Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose CoBEVT for bird's eye view (BEV) semantic segmentation from multiple cameras in a multi-agent environment in this paper. CoBERT is a Transformer-based method, in which the BEV representation is extracted by the SinBEVT Trasnformer in each vehicle, and then aggregated by FuseBEVT. These Transformers include a fused axial (FAX) module with both global and local attention, which reduces the computational cost and shows high performance in this problem setting. The proposed method has shown high performance in a variety of different settings at various metrics.
strengths_and_weaknesses: - Strengths:
  - The paper is well organized and readable.
  - The proposed method, FAX, is less expensive than full attention, yet very effective in the current problem setting, and could be useful in this research area in the future.
  - The experiments in this paper have been evaluated with various metrics (e.g., segmentation and processing speed) and settings (e.g., number of agents, camera dropout) on different methods and datasets, which are highly reliable.

- Weakness:
  - The authors assume that "poses of all the agents are accurate and transmitted messages are synchronized" but as the authors state in the limitation section, this is not necessarily the case in a real vehicle. Since this conference is also focused on the feasibility of hardware robots, the authors need more discussion on the extent to which these assumptions may not hold in reality, the extent to which these difficulties affect the current problem setting, and whether the proposed method has the potential to address these difficulties.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although I am not an expert in BEV map understanding, I believe that the proposed method is novel and will be useful in this community in the future. I would like to accept this paper, although I have some concerns that it has not been sufficiently discussed for real-world applications.
issues: As pointed out in "weakness", I would like the authors to discuss more how well the hypotheses in this setting are valid in the real world and how well the proposed method can be used in the real world.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. Each view is first passed through an image encoding to extract image images, which are then in turn used to form BEV features. The features are concatenated and fused together to make BEV predictions. The method is tested on OPV2V and nuScenes.
strengths_and_weaknesses: Strengths

- The papers utilize the latest advance in transformer architectures to tackle the V2V communication problem.

Weaknesses

- It is not clear whether those proposed transformer structures are necessary for the problem setup or not. Although it makes sense, but there is no comparison with the proposed structure with the existing methods. For example, we can compare the BEV features produced by single camera with the existing models through map segmentation and 3D object detection on nuScenes to justify the model designs to extract features from single cameras.

- The experiment detail on nuScenses is really sparse, which leads to confusion. On Line 242, it claims that "Compared to the stat-of-the-art method CVT, we are 1.1% higher with similar parameters and latency." This statement is too vague, especially compared to the results on OPV2V. The supplementary material only provides quantitative results.

- It is strange to use ResNet34 as the image encoder while the authors try to put latest transformer models on top of it, instead of designing an end-to-end transformer for image encoding to BEV features.

- It is better to know where the multi-agent fusion can improve the prediction accuracies, and whether the improvements depend on the distances of the vehicles.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I am on the borderline for the paper, but leaning towards rejection now. While fusing multiple cameras for 3D predictions is an interesting topic, the present paper leaves too many details out to understand and justify the model designs. The experiment detail on nuScenes is too little for a comprehensive understanding.
issues: Similar to the weaknesses above.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method for RGB-alone BEV perception across multiple agents and views. Each agent uses a perspective-to-BEV attention mechanism to construct a BEV feature map, and  sends its BEV feature maps to other agents. Then, each agent stacks all BEV feature maps and computes a fused BEV feature map.  The paper proposes as its contribution a hierarchical transformer in the BEV space, where attention is first performed within local windows, and then across far away locations on the map. They evaluate their model on both single agent and multi-agent RGB alone BEV segmentation and LiDAR 3D detection.
strengths_and_weaknesses: + The paper is overall clearly written and addresses an important problem

-The main contribution of the paper seems to be the sparse attention module (FAX). There are no ablations where everything is kept the same in the model, and instead a generic transformer is used to fuse information in the BEV map, or a SWIN transformer or a 2D CNN. the ablations consider previous fusion mechanisms of other methods (Table 1), and the reader does not know neither what fusion method the previous methods used, nor whether there are more changes between the present model and the models of the previous paper. The proposes sparse  transformer is never evaluated in apples-to-apples comparison with other choices.

- recent papers on arxiv can obtain way higher numbers than the present paper without any transformer  for fusion of information and without even any multi agent collaboration, at least in the tasks of BEV segmentation, e.g., https://arxiv.org/pdf/2206.07959.pdf table 2. This casts some doubt on the validity of the conclusions of the present paper, since hyperparameters such as resolution or batchsize are not investigated, and the reader does not know if the gains survive such hyperparameter changes.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper addresses an important problem but does not provide convincing ablations of its main contribution, which is the FAX module and its numbers are significantly lower than recent papers in the literature. 
issues: Could you use your model architecture and compare FAX with a generic 2D transformer in the BEV space, a SWIN transformer and a 2D CNN with varying number of layers in terms of accuracy and computational cost? Could you repeat these comparisons with double the resolution of the BEV map?


Why building a BEV per agent is important? What if the multiview images across all agents are broadcasted, and a transformer directly attends into them to build the complete BEV map of the scene?

I am happy to update my score if the authors address the questions above.


Post rebuttal: I greatly appreciate the effort  of the authors for providing additional experiments. 

The authors argue that https://arxiv.org/pdf/2206.07959.pdf showed up on arxiv post CoRL submission which is correct. I was pointing though  to the Table 2 of that paper, where you can see *earlier* papers, such as  BEVformer having *much* higher IoU in Nuscenes than the proposed model. Essentially this means the present paper is much below the SOTA. (BEVformer was submitted on arxiv on 31 Mar 2022)

Fro the comparison against other transformer variant and number of layers of those that the authors provide,  we see that the proposed model has marginally better performance (2%) with slightly less number of FLOPS . I think 2% difference is small given that changing the resolution had a 7% boost (as they report), and the best proposed method's variant provided at rebuttal time is already 5% below the SOTA in Nuscenes in comparison to BEVformer. 

By broadcasting images i meant broadcasting perspective features instead of BEV features, apologies if this was confusing. 

Overall, this is a reasonable paper that is simply not moving as fast as the SOTA in the field of  multi-view BEV fusion. I deeply appreciate the effort of the authors, and I think their additional experiments have elucidated what really matters for performance in BEV. For the reasons outlined above, I maintain my original rating.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a solution to attention-based multi-sensor multi-agent camera data fusion for the task of bird's eye view (BEV) semantic segmentation. The proposed framework CoBEVT depends on a novel fused axial attention (FAX) mechanism at both the feature extraction and fusion phases as part of Transformer architectures. The FAX mechanism conducts attention at both the local and global scales, while remaining tractable. CoBEVT is empirically validated as part of a simulated (in CARLA) vehicle-to-vehicle benchmark and on the real-world NuScenes dataset. CoBEVT outperforms state-of-the-art baselines across cooperative BEV semantic segmentation, singe-agent multi-camera BEV semantic segmentation, and 3D object detection with multi-agent LiDAR sensors tasks with real-time inference speed.
strengths_and_weaknesses: Strengths:
* The biggest strength of the paper is its empirical validation. The authors do their due diligence in thoughtfully constructing experiments that showcase the advantages of the CoBEVT framework as compared to a significant number of baselines across several tasks. The ablation studies clearly show the benefits of both local and global attention, robustness to sensor failure, multi-agent sensor fusion, and BEV feature map compression on inference speed.
* The camera-based multi-sensor multi-view CoBEVT framework appears to be novel given a quick search of existing literature. The authors do a good job contextualizing their approach in prior work.
* The limitations section is honest and thorough providing helpful context and opportunities for future work.
* Fig. 1 is an effective illustration of the CoBEVT framework.

Weaknesses:

My biggest concerns with the paper center around the presentation and polish.
* In Sec. 2.2 and 2.3 of the literature review, it would be helpful to include a sentence in each section relating how the described literature relates to the presented approach.   
* Fig. 2 lacks clarity and polish. It is unclear how the BEV image on the left in Fig. 2(a) corresponds to the predicted scenes on the right. Occluded vehicles should be highlighted in the figure. The captions for the two subfigures should be better spaced out. In Fig. 2(b), the different views (side, back, front) should be labeled. From the caption and the text in the body of the paper, it is not entirely clear what is the takeaway from Fig. 2(b). For example, which car is torn apart into multiple views? Is it the one in the lower row? How does the illustrated attention combat this?
* In methods lines 128-137, it was not really clear the difference between local and global attention. The notation makes it seem like they are the same thing but transposed. Further illustrating the difference between the approaches would be helpful here. Additionally, since the sparsity in the attention is critical to the proposed CoBEVT approach, a description of how sparsity is achieved should be included either in the main paper or in the appendix.
* In methods lines 143-145, it is not clear why the equations for local and global attention have notation that differs by one index value or why they are sequential. More context and explanation is necessary here. The 'LN' acronym is not defined in the paper.
* Table 1, 2, and 3 captions are too close together and have spelling errors (PointPillar, nuScene - both should be plural).
* In Fig. 4, the white boxes (vehicles) were not explained in the caption. The occluded and far-away vehicles should be indicated in the figure to showcase the performance of CoBEVT. The merging of vehicles in the predictions should be listed as a limitation.
* In Table 5, it is unclear what is the structure of the no local and global attention FAX architecture (first table row).

The paper currently contains many typos and requires polishing. The following is a non-exhaustive list of typos as well as grammatical and consistency issues I found:

1. Lines 3-5: incorrect grammar in 'they are all based on single-agent camera-based systems which are difficult to handle occlusions and detect distant objects in complex traffic scenes'.
2. Fig. 1: white blocks are not indicated to be cars.
3. Line 51-52: 'FAX contains global attention to reason high-level contextual information' should read 'about high-level contextual information'.
4. Line 59: 'show that CoBEVT exhibit' should be 'exhibits'.
5. Line 112: 'We propose a novel 3D attention mechanism ... as the core components' should be 'component'. 
6. Line 113: there should be no comma before 'that'.
7. Line 115: 'sing-agent' should be 'single-agent'.
8. Line 124: 'ahead the vehicle' should be 'ahead of the vehicle'.
9. Lines 140-143: 'Our 3D FAX ... more expressive than local approaches [27, 28].' sentence is too long and has grammatical errors.
10. Lines 176-177: 'The ego vehicle first stacks the received and projected BEV features ..., then feed into' should be 'then feeds them into'.
11. Line 182: 'sparsely sampled tokens are interacted globally' should be 'tokens interact globally'.
12. Line 192: '4 cameras towards distinction to cover' is incorrect grammar.
13. Line 201: 'the detection performance is measured with Average Precisions (AP) at IoU thresholds 0.7' should be 'Average Precision (AP) at an IoU threshold of 0.7'.
14. Line 206: 'the size of resolutions of the BEV grid is 0.5m' should be 'the resolution of the BEV grid is 0.5 m'.
15. Line 212: 'layer3' should be 'layers'.
16. Line 213: 'Adaw' should be 'Adam'.
17. Line 243: 'stat-of-the-art' should be 'state-of-the-art'.
18. Line 246: Missing space between sentences.
19. Line 247: Missing space between 'the' and '$1 \times 1$'.
20. The references have typos and consistency issues. For example, 'Bev' in [1] should be capitalized. Conference acronyms are sometimes included (e.g., IV in [6]) and sometimes not (e.g., [3]). When possible, the conference venue should be cited instead of ArXiv (e.g., [1]). The references should in general be proofread (e.g., [6] lists the year and IEEE twice).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the paper presents a novel approach to multi-agent multi-sensor BEV semantic segmentation, which is of interest to the autonomous driving community. A new scalable attention mechanism is introduced for the task. The experiments extensively validate the proposed approach. However, the current presentation lacks polish and clarity. As it stands, I am leaning towards rejecting the paper in its current state. I am open to increasing my score based on the rebuttal discussion and updates made to the paper during the rebuttal phase. That being said, I think this is a promising paper, and with some improvements to the presentation it will make a good contribution.

**Post-Rebuttal**: Upon going through the authors' rebuttal, I am inclined to raise my score to a Weak Accept. The authors put in significant effort in terms of clarifications and further exposition to address my concerns. Although I still have some concerns surrounding clarity and polish, I believe that the proposed method is novel and interesting with extensive and promising empirical results.
issues: In addition to the weaknesses listed above, I am hoping the authors can address the following clarifications and discussion points during the rebuttal period.

1. Are the stacked BEV features from $N$ agents (line 130) already transformed into the same spatial area?
2. How could object disappearance and merging of multiple vehicles in the predictions be addressed (e.g., in the appendix Fig. 3, third row example). 
3. Are there ways to enforce known road geometry to avoid deformation in the predictions?
4. In Fig. 5, is $32 \times 32$ the spatial resolution used in the earlier experiments?
5. It would be helpful to include the ground truth in the appendix Figs. 4 and 5. What is the difference between CoBEVT and DiscoNet qualitative results in these two figures?
review_title: null

Meta-review: metareview: The paper is well-written and well-organized. It presents an interesting and novel method, especially with regard to the fused axial attention model, which could potentially have a major impact. While some of the reviewers were convinced by the experimental evaluation, others pointed to important shortcomings: 

* motivation of the approach should be made clearer: which problems do the transformers actually solve in this setting?
* more details should be discussed on where multi-agent fusion can improve the prediction accuracies, and whether the improvements depend on the distances of the vehicles.
* notation (especially in section 3.1) should be revised to make the difference between local and global attention more clear
* the paper contains a significant number of typos and grammatical mistakes which should be corrected
* at least for BEV segmentation, other approaches in the literature exist which achieve better performance without transformers for fusion of information and without multi-agent collaboration, so are the conclusions of the paper really justified?
* evaluation is insufficient to really assess progress: sparse transformer should be evaluated against other natural candidates in BEV space, such as generic 2D transformers, SWIN transformers or a 2D CNN
* robustness to hyperparameter changes is not investigated
* the authors assume an unrealistic setting with accurate and synchronized transmission of agent poses, and the potential of the method to overcome this limitation should be discussed

===

Post rebuttal/discussion update:

The reviewers appreciated the extensive effort which the authors put into their rebuttal. Although weaknesses with respect to clarity, presentation, and explanation of the design choices remain (and should be improved), overall, the rebuttal addressed concerns in a rather comprehensive manner.  The method is interesting, novel and offers promising empirical results. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)