Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper tackles the task of ‘collaborative’ scene completion, where multiple robots communicate their own observations (top-down maps) so that each can build a more complete map. Subsequent tasks e.g. detection/segmentation can then be performed on such a completed map.

The key technical question this work addresses is regarding the representation of a robot’s own map that it should communicate. Inspired by MAEs, this work shows that a small number of patch embeddings allow decoding a map (and trains a MAE-like encoder and decoder to predict occupancy maps). One innovation here is that the paper extends MAEs to possibly use patch embeddings from previous timesteps (this helps reduce the communication at each timestep, as only transmitting a subset of embeddings suffices).

The paper reports scene completion results on outdoor driving datasets and shows that the proposed approach leads to accurate reconstructions despite low-bandwidth communications.
strengths_and_weaknesses: Strengths:

a) The overall task studied is a well-motivated one i.e. how should agents communicate to effectively reconstruct a scene while minimizing the information transmitted, and I have not seen prior works which explore this question in detail.

b) The proposed approach to learn a MAE-like network that. can leverage embeddings across timestamps is a nice contribution. In particular, it helps with temporal amortization of communication and is shown to only result in slight performance drops (compared to the communication efficiency gains). 

Weaknesses:

a) For a paper claiming to be about ‘collaborative’ completion, I feel the approach is anything but collaborative. Given the asynchronous training, it seems the final training procedure is to simply learn a MAE-like network (albeit with possible temporal data indicating past views from the same robot) to independently reconstruct the scene for each robot given only the observations from the perspective of that robot. While the MAE-like training can help get a lower bandwidth representation of the map (i.e. in terms of a subset of transformer features), this completely sidesteps the ‘collaborative’ nature of the task (e.g. robot one may have observed one end of a long table, robot 2 may have observed the other end, and a true collaborative approach would leverage both to possibly hallucinate the middle!).

b) Several important details are missing, and in its current form, it is rather difficult to parse what the work exactly does! In particular:
- what precisely is $\Gamma$ in L177?
- The details of the STAR decoder are not explained well. L169-L177 seem to imply that this uses tokens (output by the encoder, possibly across timesteps) to then predict a mask — but it is not clear what fraction of tokens are masked.
- what does the CNN backbone baseline in Table 1 actually correspond to?

c) If the main goal is to reduce the dimensionality of the messages (so that the one robot can decode the maps sent by others), why not leverage something akin to a VQ-VAE which, using discrete tokens, significantly reduces the communication bandwidth? For example, given a vocabulary of 512 and a spatial resolution of 32 x 32 for the tokens, the size of each message would be around 1 KB — leading to a significantly smaller bandwidth! Again, if the paper were actually doing some collaborative completion, this would bot have been as issue, but in its current form, it is only searching for ways to effectively compress the maps observed by each robot so they can be communicated and (independently) decoded.

d) I am also not convinced regarding the presentation of the work as ‘task-agnostic perception’. In fact, there is a very specific defined task that the network is trained for (i.e. MAE-like completion). Given a completed scene, one does need to use separately trained networks to perform any desired task (like detection, segmentation). This would be akin to claiming that ‘image completion’ is ’task agnostic visual understanding’ as given the completed image,  one can use off-the-shelf systems to do later tasks.

e) On a note related to the ‘multi-robot’ application setting, the work here actually abstracts away the actions the robots must take (which is a rather key part of a multi-robot. setting i.e. how should agents act). This work actually treats ‘robots’ as  as LiDAR sensors that passively move, and I am not sure this is a very good setup to investigate.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: First and foremost, I believe this work does not tackle ‘collaborative completion’ as each robot’s observations are independently processed into a corresponding map (which are then trivially combined) (W1). The primary benefit of the proposed approach is a more compressed representation of a map s.t. it can be communicated easily, but there alternate methods like VQ-VAEs would perhaps be even better (W3). Further, there are also concerns regarding the presentation (W2) and the positioning (W4).

While the temporal amortization of communication is an interesting and novel nugeet, I think the above concerns far outweigh this aspect.

issues: - Please do correct me if I am misinterpreting the ‘independent’ nature of the per-robot map reconstructions.

- I would be curious to hear the authors’ thoughts on why alternate compression schemes e.g. VQ-VAEs would be less preferred compared to the method here of using a MAE-like network.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a self-supervised learning strategy based on multi-robot scene completion, for which multiple robots collaboratively reconstruct the full view based on each robot’s partial observation. The training is built on top of a MAE type training strategy, for which the data can be asynchronized collected from different robots.  
strengths_and_weaknesses: Disclaimer: I am not the expert in this area.

Strength. The paper proposes to apply a spatial-temporal MAE for efficient task-agnostic multi-robot scene completion. The proposed method has achieved good performance on downstream detection and segmentation tasks.

Limitations. 

-- Contribution. Though I understand the general design strategy of applying spatial-temporal MAE for asynchronous training, I am not following how the proposed method is more advanced/better compared to other baseline methods. The authors did mention that the proposed method requires a much lower bandwidth compared to CNN counter-part, but it limits the reconstruction quality as shown in Table 1.

-- Memory. The proposed method is built on top of MAE, which should require a much large memory compared to CNN baseline. Would the memory be the bottleneck for communication as well?

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents an interesting framework for collaborative scene completion based on MAE. I feel like the design fits human intuition, though the experiments can be improved in terms of clarity -- 1.) adding a small introduction to each task-specific baselines.  ii) Explain why task-specific methods can achieve even higher performance than task-agnostic upper-bounds. iii) Additional insights and advantages compared to other designs.  
issues: See weaknesses.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a novel way to collaboratively perceive a scene through multiple robots that can work with asynchronous datasets. While prior work uses task specific losses to train bandwidth-limited encoders and decoders that handle message passing between robots, their method STAR learns to reconstruct the scene at each robot, and this scene can be used for downstream tasks. Additionally, training STAR is designed for asynchronously collected datasets, where robot messages are not aligned in time. STAR learns to encode and decode just the view of any single robot at a time, and then at inference time will use messages from each robot to reconstruct each robot’s current view. Then each view can be easily aggregated into one scene view. This scene view can be used for downstream task learning. They demonstrate their method achieves decent performance under much lower bandwidth with extensive experiments on an existing dataset.
strengths_and_weaknesses: Strengths

- Using temporal information to reduce bandwidth is a nice insight
- The scene reconstruction is a natural choice for a task-agnostic representation for ease of downstream task learning
- The authors run many ablations on their method, and compare against a good set of baselines.

Weaknesses

- At times, the writing style was confusing, with many critical details glossed over
- Conceptually, reconstructing the full scene from each robot’s messages seems like a much more challenging task than simply learning the downstream task. This is exemplified by the lower performance of STAR on the downstream tasks than task specific models, suggesting that the reconstructed scenes are not accurate enough for downstream task learning. While I agree task agnostic learning makes sense for quick transfer across tasks, the choice in using the full scene point cloud as that task agnostic representation is not convincing based on these results.
- Limited datasets: Only one task (one dataset) is considered, and it is a simulated task. It remains to be seen if STAR can perform well on datasets collected with real world hardware.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The contribution of the paper is quite novel and the experiments and ablations are mostly quite thorough. Although the chosen dataset is limited and simulated, other works in this field seem to compare against the same dataset.
issues: See “Weaknesses” for concerns for the authors to address.

Other types of task-agnostic feature learning

- Rather than learning to reconstruct the exact full scene, the authors should compare against methods that learn an invariant scene representation using multiple dowstream tasks, e.g. contrastive learning methods or meta-learning. This would help convince the reader that reconstructing the exact scene is a better choice for this intermediate representation.

Writing Style/Figure Changes

- It can be confusing to parse the Method Section, and was not immediately clear to me how the design desiderata in section 4.1 are incorporated in the method. I think the paper would greatly benefit from a method figure (possibly replacing the asynchronous vs. synchronous figure or some part of it) that shows the transformer design, the multi-robot-aggregation, and then downstream task prediction all in one figure. Then the authors could refer to this throughout section 4.
- The Temporal amortization section in 4.1 was quite confusing on first read, I think the issue in prior work could be more clearly stated or re-stated here and tied into STAR. Again having part of the method figure refer to the sub-sampling process would be quite helpful.
- Each desiderata in section 4.1 could benefit from more information, as a lot of details are glossed over here. For example in the Synchonization section, the authors say “Specifically, we use single-view observation as the supervision” without explaining how that supervision is employed during training.
- In the appendix, an algorithm outline would also be quite beneficial to outline the exact encoder / decoder steps during both training and inference.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a task-agnostic collaborative perception paradigm that can support different downstream tasks through learning collaborative scene completion. To amortize the communication cost, a spatial-temporal-aware autoencoder (STAR) is proposed. The approach can be summarized as follows:
- A grid map observation from a robot is divided into patches, each patch is encoded using the STAR encoder and then a subset of patches are selected to be broadcasted along with the robot's global pose to all other robots.
- The receiving robots aggregate the information from all other robots in addition to previously received information using the STAR decoder to form the full observation
- The model is trained asynchronously on each robot.

strengths_and_weaknesses: The paper shows a lot of promise regarding aggregating information from multiple sources that is beneficial for multiple downstream tasks. The work is also well motivated and structured, however a number of technical clarifications and experiments are missing:
1. How does the pose-aware aggregator work?
2. It was not very clear if the inference is happening synchronously or asynchronously
3. What is spatial sub-sampling and temporal mixing?
4. Given that not all information is transmitted and at the decoder side, historic tokens are aggregated with the current ones to predict the scene, how is the decision made over which patches should be transmitted? As mentioned in the paper, entirely masking a dynamic object means it cannot be properly reconstructed at decoding time. Similarly, given the dynamic nature of the scene one can imagine that the historic tokens can be no longer relevant or contain outdated scene information if the selection strategy is faulty.
5. How does the CNN baseline differ from STAR?
6. In the results, it's a bit misleading to say that STAR achieves roughly the same performance as V2V and DiscoNet when the difference in AP is around 10%
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The idea of the paper is quite intriguing, however the current draft is lacking quite a bit of detail that is pertinent to understanding the technical contribution of the work.
issues: See above
review_title: null

Meta-review: metareview: The paper presents a method for multiple robots to communicate among themselves so that each one can build a spatial map using information from the others.  The focus is on efficient information sharing through the spatial-temporal-aware auto-encoder (STAR) concept.

All the reviewers agree that this is an important problem in robotics, and appreciate the focus on bandwidth constrained communication and asynchronous training.  The authors have addressed most of the reviewers' original concerns through supplemental material and revisions to the paper. 

I encourage the authors to simplify their language and avoid jargon to better communicate their results to the CoRL audience.  Simply put, the robots in this work build an occupancy map using a LIDAR sensor and information communicated from neighbors.  Describing the contribution with clear, concrete language in the abstract and intro will help improve the ultimate impact of the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)