Multi-Robot Scene Completion: Towards Task-Agnostic Collaborative Perception
Authors: Yiming Li
Juexiao Zhang
Dekun Ma
Yue Wang
Chen Feng
Venue: CoRL 2022 Poster (2022)
Keywords: Multi-Robot Perception
Scene Completion
Representation Learning

Abstract: Collaborative perception learns how to share information among multiple robots to perceive the environment better than individually done. Past research on this has been task-specific, such as detection or segmentation. Yet this leads to different information sharing for different tasks, hindering the large-scale deployment of collaborative perception. We propose the first task-agnostic collaborative perception paradigm that learns a single collaboration module in a self-supervised manner for different downstream tasks. This is done by a novel task termed multi-robot scene completion, where each robot learns to effectively share information for reconstructing a complete scene viewed by all robots. Moreover, we propose a spatiotemporal autoencoder (STAR) that amortizes over time the communication cost by spatial sub-sampling and temporal mixing. Extensive experiments validate our method's effectiveness on scene completion and collaborative perception in autonomous driving scenarios. Our code is available at https://coperception.github.io/star/.

Multi-Robot Scene Completion: Towards Task-Agnostic Collaborative Perception
Yiming Li* New York University yiminglienyu. edu
Juexiao Zhang' New York University juexiao zhangenyu edu
Dekun Ma
Yue Wang Massachusetts Institute of Technology yuewangêcsail
Chen Fengi New York University cf cfeng@nyu. edu
New York University dm4524@nyu. edu
eeaaangaaaaammm..ee...... mit.edu
Abstract: Collaborative perception learns how to share information among mul- tiple robots to perceive the environment better than individually done. Past re- search on this has been task-specific, such as detection or segmentation. Yet this leads different information sharing for different tasks, hindering the large-scale deployment of collaborative perception. We propose the first task-agnostic col- laborative perception paradigm that learns single collaboration module in self- supervised manner for different downstream tasks. This is done by a novel task termed multi-robot scene completion, where each robot learns effectively share information for reconstructing a complete scene viewed by all robots. Moreover, we propose a spatiotemporal autoencoder (STAR) that amortizes over time the communication cost by spatial samplan and temporal mixing. Extensive experiments validate our method's effectiveness on scene completion and col- laborative perception in autonomous driving scenarios. Our code is available at https: https://coperceptioon github so/star/.
Keywords: Multi-Robot Perception, Scene Completion, Representation Learning
Introduction
Single robot perception has been widely studied on tasks such as object detection [1] and semantic segmentation [2]. However, suffers from various challenges, such as occlusion and sparsity in raw observations. Collaborative perception is promising alleviate those issues. It provides more environment observations from different perspectives by information sharing to improve perception performance and robustness. Amongst different collaboration strategies, feature-level collabora- tion [3, 4, 5] transmits the intermediate representations generated by deep neural networks (DNNs) of each robot. Since these intermediate features are easy compress and can preserve contextual information of the scene, feature-level collaboration demonstrates better iefffrmanne-bandwiinn trade-off compared raw-data-level and output-level collaboration [6, 7].
However, existing feature-level collaboration methods [8, 4, 3] are fully supervised by task-specinf losses to learn the entire model, including feature extractor, collaboration module, and a decoder, as shown in Fig. (a). Such a task-specific framework requires re-training the whole model for different perception tasks. Besides, existing collaborative perception requires training data record- ings to be synchronized among all robots in time, which is more demanding than data collection in design collaborative perception framework (1) independent from downstream tasks and (2) trainable from asynchronous datasets?
single-robot perception. How
To answer this question, propose novel self-supervised learning task termed multi-robot scene completion. It enables multiple robots to collaboratively use an autoencoder to reconstruct a com- plete scene based on shared latent features. The completed scene could then be fed into various
"indicates equal contribution. Coorreppondig author
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Perception
(a) Task-specific frumework
Single Extractor
Decoder Output Header
Specifii Perception Task
Scene Completion
(b)
Task-agnostic framework
Single
Multi-View Decoder
Encoder
Many Offer
Peroaption Tralo
Figure 1: Task-specific Task-agnostic collaboration. Task-specifie paradigm learns different models with different losses for each task. Whereas for the task-agnostic paradigm, reconstruction of the multi-robot scene is learned, which is independent of yet still usable by all downstream tasks.
downstream tasks without additional training, as shown in Fig. 1 (b). This allows us to decou- ple the collaboration training from downstream task learning. Moreover, it scamlessly supports synchronous and asynchronous training datasets with different learning objectives: complete scene reconstruction synchronous and individual view reconstruction if asynchronous.
Yet naive autoencoders are not designed to balance scene reconstruction performance and communi- cation volume, which is an established criterion evaluate collaborative perception. To address this challenge, we further design spatiotemporal autoencoder (STAR) inspired by the recent masked au- toencoders (MAE) [9]. reconstructs scene using a spatiotemporal mixture patch tokens: some tokens are encoded from randomly sub-sampled patches in the current frame and others are cached from the past. The sampling ensures that all patches the mixture can jointly cover the whole spa- tial region while being self-disjoint. This allows each robot to only transmit the sub-sampled tokens in the current frame instead of the entire latent feature maps, leading to much lower communication bandwidth than prior works. Our key insight behind such amortized communication cost is that features of many patches (e.g., static or nearly static) do not need to be shared every frame.
In summary, our main contributions are threefold:
We propose a brand-new task aggnosicc collaborative perception framework based on multi-robot scene completion, decoupling the collaboration learning from downstream tasks.
We propose asynchronous training and synchronous inference with a shared autoencoder to solve the proposed task, eliminating the need for synchronous data for collaboration learning.
We develop novel spatiotemporal autoencoder (STAR) that reconstructs scenes based on tempo- rally mixed information. amortizes the spatial communication volume over time to improve the iefformance-bandviie trade-off.
We conduct extensive experiments to verify our method's effectiveness for scene completion and downstream perception in autonomous driving scenarios.
2
Related Works
Collaborative perception. ception's 's flexibility, resilience, and efficiency [10, 11, 12]. With recent advances in deep learning, researchers have developed feature-level collaborative perception in which intermediate representa- tions produced by deep neural networks (DNNs) from multiple viewpoints are propagated in team of robots, e-g., swarm of drones [8, 3] or group of vehicles [4, 6]. Existing works commonly consider a specific downstream task and use the corresponding loss function to learn collabora- tion module such lutional neural network [5. 15]. Several downstream tasks have been investigated collaborative scenarios, such object detection [4], semantic segmentation [8], and depth estimation [3]. Un- like the existing task-specific collaborative perception sharing task-dependent representations, we define task task-agnostic collaborative perception the feature-level collaborative perception sharing task-independent representations amongst multiple robots.
Collaborative perception has been proposed to improve individual per-
graph neural network (GNN) [4, 3]. a Transformer [13, 14), and convo-
Herein "task" denotes the downstream perception task such object detection and semantic segmentation.

---

Scene completion. Autonomous navigation [16] requires robots to understand the geometry and semantics of 3D scenes. However, vision sensors only capture partial observations because of a limited field of view and sparse sensing, leading to an incomplete spatial representation. Therefore, scene completion (SC) has been proposed to infer the complete 3D scene geometry given sparse 2D/3D observations [17, 18, 19]. Following scene completion, semantic scene completion (SSC) has been introduced to jointly estimate both geometry and semantic information based on partial observation [2. 20, 21, 22]. On the one hand, single robot scene completion can rely on prior semantic knowledge to complete the partially-observed objects. On the other hand, it is unrealistic to through full occlusions. Unlike single robot scene completion depending on prior knowledge, the multi-robot scene completion task utilizes information shared by teammates for scene completion.
Self-supervised representation learning, Self-supervised representation learning (SSRL) aims to provide powerful features without the need for massive annotated datasets [23]. SSRL generally composed of: (1) task-agnostic pre-training via aarefully-designed self-supervised pretext tasks such as contrastive learning (24, 25] or autoencoding [26, 27, 9], and (2) task-specific adaptation to fine- tune the pre-trained model on the downstream tasks such object detection or image classification. Masked autoencoder (MAE) achieves great performance with a simple reconstruction objective [9]. It employs asymmetric architecture with a large encoder that only processes unmasked patches and a lightweight decoder that reconstructs the masked patches from the latent representation. Re- cent works extend MAE into multimodal representation learning [28, 29]. video [30, 31], and 2D image completion [32]. In this work, we employ similar autoencoding to learn the shared repre- sentations and enable quick adaptation to the downstream perception: the reconstructions could be seamlessly utilized by off-the-shelf individual perception model trained on single-view data without any fine-tuning, bridging the gap between collaborative perception and individual perception.
3 Multi-Robot Scene Completion: Motivation, Formulation, and Evaluation
Motivation. Even though single-robot data already requires arduous annotations like 3D bounding boxes and pixel-wise semantic labeling, multi-robot data even demands multiple times much work. To relax the task-dependent supervision for collaboration learning, we propose multi-robot scene completion to enable task-agnostic collaborative perception. It can utilize self-supervision to learn shared representations instead of expensive task-dependent supervision. We will introduce its overall workflow, training objective, and evaluation metrics hereafter.
Problem setup. We consider N robots in the same geographical location simultaneously perceiving the 3D environment, such a fleet of autonomous vehicles located at crossroad. These robots communicate with each other about their observations to better understand the surrounding envi- ronment. Each robot indexed by is equipped with a 3D sensor such LiDAR to generate binary occupancy grid map M; E {0, 1 WXx XC defined in its local coordinate, where H, W, and C' respectively denote the length, width, and height resolution.
Feature extraction. We employ intermediate collaboration with better performance brrdowiance-bandwidth trade-off [5]. Each robot encodes its observation into a feature map denoted by F; = @(M)), where θ is a feature extractor. Now F, E f xC has lower spatial resolution H x W, while keeping a higher feature dimension Ć compared to the original map M, Then, each robot will broadcast F, to its peers as well as its pose & E se(3) defined in the global coordinate.
Feature decoding. The robot receives the messages from the neighboring robots {F and then uses decoder & and pose-aware aggregator A for fusion, and output a completed occupancy grid map Y, (AF,,, i & {F,, &, )=+)), where Y, has the same dimension and describe the same spatial range as M, yet is a more comprehensive spatial representation for the scene. The pose-aware aggregator A transforms the feature maps of robot j(j * i) into the coordinate of the target robot i, then sums all the coordinate- sysiiiante-ycccroiieeeedd feature maps:
N-1
A(F,, E. (F = F, IT),(FF))
(1)
where Ty- E SE(3) is the transformation from robot j's coordinate to i's, obtained by the expo- nential map of poses E, and &,, Bi-linear interpolation is used to transform discrete map, and the positions out of the spatial range H x W after transformation are padded with zero.
3

---

Training loss. We treat the scene completion task as a binary classification problem and use cross- en loss train a neural network composed of θ and o. Specifically, the ground-truth Y; (0.1) ) xc defined in the coordinate of robot i represents multi view occupancy voxel grid with two classes, Le,, free and occupied. Therefore, the loss can be computed by:
N-14-1
e.i.k Ee Eebiaee
L -= 1=0 1=
(2)
where i is the robot index, k is the voxel index, L is the total number of the voxel (L = HXW x C e is the number of class (2 in our case), ki, is the predicted logits for the k-th voxel belonging
to class Vi, is the k-th element of Y; and is one-hot vector (Hi.k.c 1 if voxel & of robot d belongs to class c). Here we show the training objective using synchronous multi-robot data created by aggregating multi-robot observations based robots' poses similar Eq. (1): Y, A(M, E., [M Ei)r)) (j <. M), where M is the number of robots for the ground truth generation. Note that @ and 4 can also be trained by individual view reconstruction shown in Eq. (3), while being deployed on synchronous data for inference as discussed Section 4. Hallucinating the invisible scene is possible if the ground-truth Y; is generated with more robots than those involved in the training phase, i.e., M >N, Yet only synchronous training can achieve this because the ground truth requires aggregation of different viewpoints obtained synchronously.
asynchronous data as
Evaluation metrics. We follow the evaluation protocol in single-robot scene completion [19, 33], which uses the voxel-level intersection over union (IoU) between predicted voxel labels Y, and ground truth labels Y, for each robot. Note that only non-empty voxels are evaluated.
4 STAR: Spatiotemporal Autoencoder
In addition to the multi-robot scene completion task, we also propose novel architecture called Saptiotemporal autoencoder (STAR) to tackle this problem. We will present our key design moti- vation, detailed modules, training, and inference procedures.
4.1 Design rationale
Partially broadcasting, Inspired by the idea of "masking" in MAE [9], we employ similar asym- metric design as MAE yet with different purposes: MAE is to design nontrivial superviiry task for pre-training via randomly masking, while the goal of STAR is to reduce the communica- tion volume in multi-robot systems via partial broadcasting. More specifically, STAR deploys an encoder at the sender robol to map the entire observation to an intermediate feature representation that is selectively transmitted to lower the bandwidth. Mcanwhile, STAR deploys decoder at the receiver robot that reconstructs the original observation from the received partial representation.
Spatiotemporal amortization. Simply applying random masking to a complete observation is not a good idea. Unlike MAE mainly for object-level recognition, we aim large-scale dynamic scene modeling. Once objects are completely masked during encoding, the decoder cannot hallucinate the corresponding objects without such knowledge. Therefore, we propose to exploit historical tokens to replace mask lokens during decoding. In this way, we can amortize the communication cost over the temporal domain by spatial sub-sampling and temporal mixing. Specifically, at the sender's side, part of the spatial patches are sampled from the entire observation in each timestamp. Only the features of these subsampled spatial patches are communicated with the teammates. Upon receiving the transmitted patch features, the receiver's decoder combines these features with the historical patch features and reconstructs an entire observation. Since the patches are subsampled a spatially complementary manner, the temporally mixed patches jointly cover the whole spatial region.
Synchronization-free training. Traditional collaborative perception approaches consider a syn- chronization training strategy that requires synchronous (potentially with a slight temporal latency) multi-robot recordings to train a feature-space collaboration strategy with task-specific loss func- tions [4, 5]. In this work, we try to relax the requirement that multiple robots simultaneously capture perception data by using single-view observations as the supervision during the training phase. We will describe this in detail in section 4.3.

---

Asynchronous Training (Ours)
Synchronous Inference
Mis
Pir Pla-d
Scene Recomstrucdon
M,x
Fpp-s
Ejz= 4
Synchronous Training
Lasses
Apgregation
Completed Bcene
Single-rbot Perception Model
Task Specific Outputs
Figure 2: Asynchronous training and synchronous inference. In the top right, asynchronous train- ing does not require communication between robots. In contrast, the bottom right, synchronous training requires communication and optimization w.r.t. each specific task loss. The synchronous inference is illustrated on the left. The sender transmit encoded representations to the receiver. The receiver sub-sampling. T: temporal mixing.
mixture of spatiotemporal tokens to complete the scene observation. S: spatial
4.2 Architecture
We consider a set of robots deploying the same neural network following [4, 5J. Each robot serves both message sender and receiver during collaboration and is equipped with an encoder for ob- servation abstraction and a decoder for view reconstruction.
STAR encoder. Different from MAE [9], the STAR encoder uses vision transformer (ViT) [34] backbone, which operates on all patches yet only sends out a subset (spatial sub-sampling). Specif- ically, the entire grid map for robot i at time denoted by M,, is divided into multiple patches, and each patch is encoded with a linear projection with additional positional embedding, and then processed using series of Transformer blocks to generate the final message F.,-- Note that adopt a complementary transmission strategy in the temporal domain regarding the patch index (i.e., the observed spatial locations) avoid the loss of information for the dynamic scenes.
STAR decoder. Different from MAE which uses mask tokens to replace the missed patch embed- dings. robot i receiver aggregates the historic tokens F Fj,-1 and the current tokens F,, from robot j (temporal mixing), which approximately form a complete observation towards the entire spa- tial range. Temporal embeddings added to the tokens from the respective timestamps to enhance the temporal awareness before feeding them into Transformer blocks. Note that here we use two- timestamp and t- 1 example for simple explanation. The STAR decoder is also able to pro- cess more historical timestamps. After decoding all robots' views [M..jjjiii the ultimate prediction of the complete view is computed by coordinate synchronization: Y, = A(M,, & {M,,s & 3144, and its calculation process is similar to Eq. (1).
4.3 Training and inference
Asynchronous training. The model is trained with single view ground M,. and adopt cross- entropy loss during training:
N-IL-1 f= --
e tn
mi.k.clog K= c=0
(3)
where is the robot index, is the voxel index, L is the total number of the voxels, c 2 is number of class, mikkee denotes the k-th element of M, and is a one-hot vector same Vi,k.c in Eq. 2, rit the prediction for the k-th voxel belonging to class c. Note that the training loss is calculated voxel-wise with respect to the self-supervision signal from each robot's single-view observation. This design decouples the training phase from communication with other robots: the model on each robot does not require synchronous observations from neighbor robots in the training
5

---

phase, making the training asynchronous (asynchronous training in Fig. 2). This is greatly different from the training framework in previous collaborative perception works such as [5] (synchronous training in Fig. 2). Our training framework can relax the need for the carefully-collected and hard- to-annotate multi-robot dataset and can exploit a large amount of single-robot data to learn powerful as well as compact feature representations.
Synchronous inference. During inference, each robot is equipped with the same model. The sender robots' encoders will encode and broadcast subset of their current timestamp's observation. Then, the decoders on the receiver side will leverage the transmitted intermediate representation along with the pose information to reconstruct the corresponding view, optionally with historical features as described above. Then, the receivers use corresponding pose information aggregate the single observations into a multi-view completed scene. We illustrate the pipeline on the left side of Fig. 2.
5 Experimental Results
5.1 Experimental setup
Dataset. We conduct experiments on the V2X-Sim Dataset (15), a large-scale dataset that simulates urban multi-vehicle driving scenes with CARLA [35]. We use 80 scenes for training and ten scenes for testing. The dataset is sampled at 5 Hz. We pre-prcesss the voxels grids with range [-32m, 32m] in the and y-axis and [-3m, 2m]] in the z-axis. Finally, we can get the voxel grids with spatial resolution of 256 x 256 x 13.
Baselines. The nower-bounde refers to single-robot perception model trained and tested using only individual observations. The task-specific models all optimize the collaboration based on the specific perception head. When2com [8] uses the attention mechanism to fuse the collaborators' information. Who2com [36] employs a handshake mechanism. The V2VNet [4] trains a graph neural network to propagate the agent's information. DiscoNet [5] selectively fuse messages from the informative regions. For task-sgnostic models, we use a modified FaFNet [37] backbone as the CNN baseline and substitute the detection head with a classification head that outputs the logits for binary classification. VQ-VAE [26] learns a variational autoencoder to reconstruct the scene and employs a vector quantization technique reduce communication costs.
Implementation details. A 6-block ViT encoder with hidden dimension 384 is used for the STAR encoder. Then an MLP is used to compress the intermediate representations to 32 dimensions and feed them to the decoder, where they are projected back to 256 dimensions and sent to a 4-layer transformer decoder. An FaFNet [37] is used for single-robot object detection. A UNet [38] serves the same purpose for the semantic segmentation task. Note that all the perception models take the three-dimensional voxel grids input and output results in bird's eye view (BEV), such bounding boxes and semantic labels. Our models are all trained on single-view data.
Evaluation metrics. For the scene completion task, we measure the completion quality using the intersection-ovvr-unioo (IoU) at three different scales by down-sampling the voxels accordingly. For the perception task. report the average precision (AP) at thresholds 0.5 and 0.7 for vehicle detection, loU for the vehicle category, and the overall mIoU for semantic segmentation.
5.2 Quantitative results on scene completion
We present the quantitative results of the multi-robot scene completion task in Table I, measured by IoU at different scales and the corresponding communication bandwidth.
Spatial resolution. Among the three tested resolutions, we can see that, in general, higher spatial resolution leads to better completion quality: the spatial resolution 32 x 32 which has a patch size of 8 achieves the best performance.
Timestamps. Our method allows the multi-robot system to amortize the spatial communication see that from timestamps to 4, the
bandwidth over the temporal domain. From Table performance only varies slightly while largely reducing the bandwidth.
Bandwidth. Bandwidth is calculated to reflect the required data volume for communication per second. trade-off between performance and communication is clear. The CNN baseline requires much higher bandwidth because multiple feature maps are transmitted during communication due

---

IoU scale 1:1
IoU scale 1:2
IoU scale 1:4
Communication Bandwidth
Timestamp
32x32
32x32
32x32
16x16
STAR TS1
55.13
53.11
50.79
77.40
72.16
66.55
83.28
79.30
73.33
1.3MB/
320.0KI3/
80.0KB/
STAR TS2
54,93
52.07
50.40
75.71
69.63
64.86
82.51
76.24
70.81
640.0KR/s
160.0KB/%
40.0KBK
STAR TS3
53.35
51.56
50.39
72.52
68.
64.75
79.20
74.51
70.55
427 OKB/S
106.7KBA
26.7KB/
STAR TS4
51.64
72.98
79.73
74.36
3/0.0KB/S
20,0RRS
CNN backbone
55.57
77.17
83.51
155.0MB/s
Table I: Quantitative results on scene completion. Results across different spatial resolutions and timestamps are presented. Note TSX means fusing temporal information across X TimeStamps
Timestamp All
Partial
Timestamp
Multi
Single
65.19
65.19
64.68
61.36
64.68
52.07
64.53
63.77
64,53
51.97
(a) Patches to encode. All: encodes all. Partial: only encodes those patches being transmitted.
(h) Timestamps to decode. For timestamp 1, de- coding a single timestamp is equivalent to multi.
Timestamp Temporal Emmh
Timestamp
Strategy
w/ w/o
2(50%)
3(66%)
4(75%)
64.68
64.29
random
52.36
64.53
61.83
compl.
64.45
64,20
63.27
(c) Temporal embedding. W/ means temporal embeddings are added. W/o means not.
(d) Masking strategy. The ratio of random mask- ing is set equivalent to complementary masking.
Table 2: Ablation studies. The performance is reported IoU 1:2 for the spatial resolution 16x16.
The observations under other settings consistent.
to the skip connections in the model. STAR requires much lower bandwidth. A finer- aner-grained spatial resolution with better performance requires a higher bandwidth.
5.3 Ablation studies
completion
We conduct several ablation studies to investigate the effectiveness of the key components in our method. Results are presented in Table 2 and are discussed in detail below.
Patches to encode. As shown in Table 2a, only encoding the patches that will be transmitted can re- sult in minor drop in performance. Yet it can reduce some computations for computtation-rsstriette robotic systems.
Timestamps to decode. We investigate the effect of whether the decoder incorporates previous timestamps or just the current single timestamp combined with learnable mask tokens. Results in Table 2b indicate that historical information is essential.
Temporal embedding. In the STAR decoder, we add temporal embedding the patches of different timestamps, similar to the approach in [30, 31j. The ablation study in Table 2c shows that adding temporal embedding is beneficial.
We compared our complementary masking strategy with the random masking strategy proposed in MAE [9] in Table 2d. Results show that switching from complementary to random masking leads to a degradation in the completion performance.
Masking strategy.
5.4 Quantitative results downstream perception
We directly feed the single-view to the single-robot perception model termed lower-bound without any fine-tuning, and the results are shown in Table Our best STAR method improves the lower- bound by 25.9% and 22.8% in object detection (AP@IoU=0.7) and semantic segmentation (IoU of vehicle) respectively. Achieved by simply combining the completion model with off-the-shelf single-robot perception models, these improvements are promising because our framework: (1) has no knowledge about downstream tasks (lass-annostic); (2) does not require synchronous data the training phase Lyynchronizationnfrree)) (3) is learned without manual annotations (self-supervised), We also investigated the single-robot perception model directly taking ground truth multi-view mea- surements without additional training, termed upper- hopper-bound, and find that it can achieve nearly com-
7

---

Delection
Semantic Segmentation mIoU
Parudign
Method
AP@1oU==0.5
AP@IoU=0.7
Single-robol perceptiion
Lower bound
49.90
44.21
45.93
36.64
When2com [8]
44.02
39.89
47.87
34.49
Who2com [36]
44.02
Task specific multi roboi perception
V2VNet [4]
68.35
62.83
58.35
41.17
DiscoNet [S]
69.03
63.44
55.84
41.34
STAR TS1
62.84
57.22
56.41
39.09
Task-agnostic multi-robot perception
35.75
VQ VAE
60.27
54.08
55.40
38.48
CNN
Upper-bound
65.09
69.26
60.34
40.45
Table 3: Quantitative results on downstream tasks. The task-specific methods achieve excellent results via elaborate supervised learning with synchronous multi-robot recordings. The task-agnostic methods use single-robot perception models with reconstructed observations.
parable performance with DiscoNet [5] and V2VNet [4], both trained with full supervision using synchronous data for specific tasks. This demonstrates the potential of our proposed task: when the completions approach the ground truth scenes, it can perform similarly to the upper bound on many downstream tasks. Moreover, using a stronger single-robot perception model can further enhance the final performance.
5.5 Qualitative results on scene completion and downstream perception
We present few qualitative results the completion and the downstream tasks in Figure 3. Due to the limited space, more visualizations can be found in the appendix.
(a) Completion
True Observation
Completed Observation
Difference
Thue Observation
Completed Observation
(b) Detecttion
True Segmentation
Prodicted Soguxcntation
Thie Observation
Completed Observation
Detection
(C) Semantic Segmentation
Figure 3: Qualitative results. (a), (b) and (c) each presents a qualitative example scene comple- tion, detection, and segmentation tasks respectively. Refer to the appendix for more visualizations.
Limitation
There is still performance gap between our method and the upper bound on the downstream per- ception due to the non-perfect scene completion. We believe when trained with more single-robot recordings, our method is able to achieve comparable performance to task-specifi approaches while maintaining excellent flexibility. Currently the spatial tokens are sub-sampled randomly at individ- ual timestamp, and we believe this could be improved in the future. We also inherit the common limitation in most existing collaborative perception works: experiments are on simulated datasets due to the lack of public real-world datasets. We further ignore the influence of pose noises, although previous works [5] already revealed reasonable robustness.
Conclusion
We propose the first task agnosstc collaborative perception paradigm, where single collaboration module is learned and can transferred to different downstream tasks. Our key observation that we can move communication between robots to the temporal domain, which achieves an excellent performance-bandwee trade-off. Also, our self-supervised learning method sheds new light on collaborative perception that reduces the importance of human annotations.

---

Acknowledgments
We thank the anonymous reviewers for their valuable comments in revising this paper. This work was supported by the NSF CPS Program under Grant CMMI-1932187 and CNS-2121391.
References
[1] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning. pages 180-191. PMLR, 2021.
[2] R. Cheng, C. Agia, Y. Ren, X. Li, and L. Bingbing. S3enet: A sparse semantic scene comple- tion network for lidar point clouds. In Conference Robot Learning, 2020.
[3] Y. Zhou, J. Xiao, Y. Zhou, and G. Loianno. Multi-robot collaborative perception with graph neural networks. IEEE Robotics and Automation Letters, 2022.
[4] T.-H. Wang. S. Manivasagam, M. Liang, B. Yang, W. Zeng. and R. Urtasun. V2vnet: Vehicle- to-vehicle communication for joint perception and prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 605 605-621, 2020.
[5] Y. Li, S. Ren, P. Wu, S. Chen, C. Feng, and W. Zhang. Learning distilled collaboration graph for multi-agent perception. In Advances in Neural Information Processing Systems, volume 34, 2021.
[6] Q. Chen, S. Tang, Q. Yang, and S. Fu. Cooper: Cooperative perception for connected au- tonomous vehicles based on 3d point clouds. In IEEE International Conference on Distributed Computing Systems (ICDCS), pages 514 524, 2019.
[7] E. Arnold, M. Dianati, R. de Temple, and S. Fallah. Cooperative perception for 3d object detection driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems, 2020.
[8] Y-C. Liu, J. Tian, N. Glaser, and Z. Kira. When2com: multi-agent perception via communi- cation graph grouping. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 4106-4115, 2020.
[9] K. He, X. Chen, S. Xie, L., Dollár, and Girshick. Masked autoencoders are scalable vi- sion learners. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] Y. Hu, S. Fang, Z. Lei, Y. Zhong, and S. Chen. Where2comm: Communication-effan collab- orative perception via spatial confidence maps. In Advances in neural information processing systems, 2022.
[11] Z. Lei, S. Ren, Y. Hu, W. Zhang, and S. Chen. Latency-aware collaborative perception. In Proceedings the European Conference on Computer Vision (ECCV), 2022.
[12] S. Su, Y. Li, S. He, S. Han, C. Feng, C. Ding, and F. Miao. Uncertainty quantification of collaborative detection for self-driving, arXiv preprint axXiv:2209.88122 2022.
[13] R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, and J. Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. 6th Annual Conference on Robot Learning.
[14] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma. V2x-vit: Vehicle-to-everythinng cooperative perception with vision transformer. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[15] Y. Li, D. Ma. Z. An, Z. Wang, Y. Zhong, S. Chen, and C. Feng. V2x-sim: Multi-agent col- laborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 1(4):1091411021, 2022.

---

[16] S. Garg, N. Sünderhauf, F. Dayoub, D. Morrison, A. Cosgun, G. Carneiro, Q. Wu, T.-J. Chin, I. Reid, S. Gould, et al. Semantics for robotic mapping. perception and interaction: A survey. Foundations and Trendso in Robotics, 8(1-2):1 2020.
[17] J. Davis, S. R. Marschner, M. Garr, and M. Levoy. Filling holes in complex surfaces using volumetric diffusion. Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission, pages 428 -41. IEEE, 2002.
[18] M. Firman, O. Mac Aodha, S. Julier, and G. J. Brostow. Structured prediction of unobserved voxels from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5431-544) 2016.
[19] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene com- pletion from a single depth image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1746-1754, 2017.
[20] L. Roldao,
R. de Charette, and A. Verroust-Blondet. Lmscnet: Lightweight multiscale 3d semantic completion. In 2020 International Conference on 3D Vision (3DV), pages 111-119. IEEF, 2020.
[21] C. B. Rist, D. Emmerichs, M. Enzweiler, and M.M. Gavrila. Semantic scene completion using local deep implicit functions lidar data. IEEE transactions on pattern analysis and machine intelligence, 44(10):7205-7218, 2021.
[22] X. Yan, J. Gao, J. Li, R. Zhang, Z. Li, R. Huang, and S. Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3101-3109, 2021.
[23] L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales. Self-supervised representation learn- ing: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 39(3):42-62, 2022.
[24] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 77299-9388 2020.
[25] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and Isola. What makes for good views for contrastive learning? In Advances in Neural Information Processing Systems, volume 33, pages 6827-6839, 2020.
[26] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In Advances in neural information processing systems, volume 30, 2017.
[27] D. P. Kingma and M. Welling. Auto-encoding variational bayes. 1, 2013.
arXiv preprint
[28] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. Multimae: Multi-modal multi-task masked autoencoders. arXiv preprint arXiv:2204.00 2022.
[29] X. Geng, H. Liu, L. Lee, D. Schuurams, S. Levine, and P. Abbeel. Multimodal masked autoen- coders learn transferable representations. arXiv preprint arXiv:2205. 14204, 2022.
[30] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for sulf-supervised video pre-training. In Advances in neural information processing systems, 2022.
[31] C. Feichtenhofer, H. Fan, Li, and K. He. Masked autoencoders as spatiotemporal learners. In Advances in neural information processing systems, 2022.
[32] C. Zheng, T.-J. Cham, and Cai. Tfill: Image completion via a transformer-based architecture. arXiv preprint arXiv:2104.00845, 2021.
[33] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- mantickitti: dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297-9307, 2019.
10

---

[34] A. Dosovitskiy. L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghami, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. In International Conference on Learning Representations, 2020..
[35] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[36] Y.-.. Liu, J. Tian, C.-Y. Ma, N. Glaser, C.-W. Kuo, and Z. Kira. Who2com: Collaborative perception via learnable handshake communication. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6876-6883, 2020.
[37] W. Luo, B. Yang, and R. Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3569-3577, 2018
[38] O. Ronneherger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.


Reviews:
[{"review_id": "dnvCWc54uz", "reviewer": "Reviewer_uiAy", "summary": "This paper proposes a novel way to collaboratively perceive a scene through multiple robots that can work with asynchronous datasets. While prior work uses task specific losses to train bandwidth-limited encoders and decoders that handle message passing between robots, their method STAR learns to reconstruct the scene at each robot, and this scene can be used for downstream tasks. Additionally, training STAR is designed for asynchronously collected datasets, where robot messages are not aligned in time. STAR learns to encode and decode just the view of any single robot at a time, and then at inference time will use messages from each robot to reconstruct each robot’s current view. Then each view can be easily aggregated into one scene view. This scene view can be used for downstream task learning. They demonstrate their method achieves decent performance under much lower bandwidth with extensive experiments on an existing dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Using temporal information to reduce bandwidth is a nice insight\n- The scene reconstruction is a natural choice for a task-agnostic representation for ease of downstream task learning\n- The authors run many ablations on their method, and compare against a good set of baselines.\n\nWeaknesses\n\n- At times, the writing style was confusing, with many critical details glossed over\n- Conceptually, reconstructing the full scene from each robot’s messages seems like a much more challenging task than simply learning the downstream task. This is exemplified by the lower performance of STAR on the downstream tasks than task specific models, suggesting that the reconstructed scenes are not accurate enough for downstream task learning. While I agree task agnostic learning makes sense for quick transfer across tasks, the choice in using the full scene point cloud as that task agnostic representation is not convincing based on these results.\n- Limited datasets: Only one task (one dataset) is considered, and it is a simulated task. It remains to be seen if STAR can perform well on datasets collected with real world hardware.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The contribution of the paper is quite novel and the experiments and ablations are mostly quite thorough. Although the chosen dataset is limited and simulated, other works in this field seem to compare against the same dataset.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "See “Weaknesses” for concerns for the authors to address.\n\nOther types of task-agnostic feature learning\n\n- Rather than learning to reconstruct the exact full scene, the authors should compare against methods that learn an invariant scene representation using multiple dowstream tasks, e.g. contrastive learning methods or meta-learning. This would help convince the reader that reconstructing the exact scene is a better choice for this intermediate representation.\n\nWriting Style/Figure Changes\n\n- It can be confusing to parse the Method Section, and was not immediately clear to me how the design desiderata in section 4.1 are incorporated in the method. I think the paper would greatly benefit from a method figure (possibly replacing the asynchronous vs. synchronous figure or some part of it) that shows the transformer design, the multi-robot-aggregation, and then downstream task prediction all in one figure. Then the authors could refer to this throughout section 4.\n- The Temporal amortization section in 4.1 was quite confusing on first read, I think the issue in prior work could be more clearly stated or re-stated here and tied into STAR. Again having part of the method figure refer to the sub-sampling process would be quite helpful.\n- Each desiderata in section 4.1 could benefit from more information, as a lot of details are glossed over here. For example in the Synchonization section, the authors say “Specifically, we use single-view observation as the supervision” without explaining how that supervision is employed during training.\n- In the appendix, an algorithm outline would also be quite beneficial to outline the exact encoder / decoder steps during both training and inference."}}, {"review_id": "dFpRuqF-CX", "reviewer": "Reviewer_8X6f", "summary": "This paper proposes a self-supervised learning strategy based on multi-robot scene completion, for which multiple robots collaboratively reconstruct the full view based on each robot’s partial observation. The training is built on top of a MAE type training strategy, for which the data can be asynchronized collected from different robots.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Disclaimer: I am not the expert in this area.\n\nStrength. The paper proposes to apply a spatial-temporal MAE for efficient task-agnostic multi-robot scene completion. The proposed method has achieved good performance on downstream detection and segmentation tasks.\n\nLimitations. \n\n-- Contribution. Though I understand the general design strategy of applying spatial-temporal MAE for asynchronous training, I am not following how the proposed method is more advanced/better compared to other baseline methods. The authors did mention that the proposed method requires a much lower bandwidth compared to CNN counter-part, but it limits the reconstruction quality as shown in Table 1.\n\n-- Memory. The proposed method is built on top of MAE, which should require a much large memory compared to CNN baseline. Would the memory be the bottleneck for communication as well?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an interesting framework for collaborative scene completion based on MAE. I feel like the design fits human intuition, though the experiments can be improved in terms of clarity -- 1.) adding a small introduction to each task-specific baselines.  ii) Explain why task-specific methods can achieve even higher performance than task-agnostic upper-bounds. iii) Additional insights and advantages compared to other designs.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See weaknesses.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "__pNrYQHT_f", "reviewer": "Reviewer_vXCo", "summary": "The paper proposes a task-agnostic collaborative perception paradigm that can support different downstream tasks through learning collaborative scene completion. To amortize the communication cost, a spatial-temporal-aware autoencoder (STAR) is proposed. The approach can be summarized as follows:\n- A grid map observation from a robot is divided into patches, each patch is encoded using the STAR encoder and then a subset of patches are selected to be broadcasted along with the robot's global pose to all other robots.\n- The receiving robots aggregate the information from all other robots in addition to previously received information using the STAR decoder to form the full observation\n- The model is trained asynchronously on each robot.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper shows a lot of promise regarding aggregating information from multiple sources that is beneficial for multiple downstream tasks. The work is also well motivated and structured, however a number of technical clarifications and experiments are missing:\n1. How does the pose-aware aggregator work?\n2. It was not very clear if the inference is happening synchronously or asynchronously\n3. What is spatial sub-sampling and temporal mixing?\n4. Given that not all information is transmitted and at the decoder side, historic tokens are aggregated with the current ones to predict the scene, how is the decision made over which patches should be transmitted? As mentioned in the paper, entirely masking a dynamic object means it cannot be properly reconstructed at decoding time. Similarly, given the dynamic nature of the scene one can imagine that the historic tokens can be no longer relevant or contain outdated scene information if the selection strategy is faulty.\n5. How does the CNN baseline differ from STAR?\n6. In the results, it's a bit misleading to say that STAR achieves roughly the same performance as V2V and DiscoNet when the difference in AP is around 10%", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea of the paper is quite intriguing, however the current draft is lacking quite a bit of detail that is pertinent to understanding the technical contribution of the work.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See above", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "04Ea5EmVO0c", "reviewer": "Reviewer_MFnL", "summary": "This paper tackles the task of ‘collaborative’ scene completion, where multiple robots communicate their own observations (top-down maps) so that each can build a more complete map. Subsequent tasks e.g. detection/segmentation can then be performed on such a completed map.\n\nThe key technical question this work addresses is regarding the representation of a robot’s own map that it should communicate. Inspired by MAEs, this work shows that a small number of patch embeddings allow decoding a map (and trains a MAE-like encoder and decoder to predict occupancy maps). One innovation here is that the paper extends MAEs to possibly use patch embeddings from previous timesteps (this helps reduce the communication at each timestep, as only transmitting a subset of embeddings suffices).\n\nThe paper reports scene completion results on outdoor driving datasets and shows that the proposed approach leads to accurate reconstructions despite low-bandwidth communications.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\na) The overall task studied is a well-motivated one i.e. how should agents communicate to effectively reconstruct a scene while minimizing the information transmitted, and I have not seen prior works which explore this question in detail.\n\nb) The proposed approach to learn a MAE-like network that. can leverage embeddings across timestamps is a nice contribution. In particular, it helps with temporal amortization of communication and is shown to only result in slight performance drops (compared to the communication efficiency gains). \n\nWeaknesses:\n\na) For a paper claiming to be about ‘collaborative’ completion, I feel the approach is anything but collaborative. Given the asynchronous training, it seems the final training procedure is to simply learn a MAE-like network (albeit with possible temporal data indicating past views from the same robot) to independently reconstruct the scene for each robot given only the observations from the perspective of that robot. While the MAE-like training can help get a lower bandwidth representation of the map (i.e. in terms of a subset of transformer features), this completely sidesteps the ‘collaborative’ nature of the task (e.g. robot one may have observed one end of a long table, robot 2 may have observed the other end, and a true collaborative approach would leverage both to possibly hallucinate the middle!).\n\nb) Several important details are missing, and in its current form, it is rather difficult to parse what the work exactly does! In particular:\n- what precisely is $\\Gamma$ in L177?\n- The details of the STAR decoder are not explained well. L169-L177 seem to imply that this uses tokens (output by the encoder, possibly across timesteps) to then predict a mask — but it is not clear what fraction of tokens are masked.\n- what does the CNN backbone baseline in Table 1 actually correspond to?\n\nc) If the main goal is to reduce the dimensionality of the messages (so that the one robot can decode the maps sent by others), why not leverage something akin to a VQ-VAE which, using discrete tokens, significantly reduces the communication bandwidth? For example, given a vocabulary of 512 and a spatial resolution of 32 x 32 for the tokens, the size of each message would be around 1 KB — leading to a significantly smaller bandwidth! Again, if the paper were actually doing some collaborative completion, this would bot have been as issue, but in its current form, it is only searching for ways to effectively compress the maps observed by each robot so they can be communicated and (independently) decoded.\n\nd) I am also not convinced regarding the presentation of the work as ‘task-agnostic perception’. In fact, there is a very specific defined task that the network is trained for (i.e. MAE-like completion). Given a completed scene, one does need to use separately trained networks to perform any desired task (like detection, segmentation). This would be akin to claiming that ‘image completion’ is ’task agnostic visual understanding’ as given the completed image,  one can use off-the-shelf systems to do later tasks.\n\ne) On a note related to the ‘multi-robot’ application setting, the work here actually abstracts away the actions the robots must take (which is a rather key part of a multi-robot. setting i.e. how should agents act). This work actually treats ‘robots’ as  as LiDAR sensors that passively move, and I am not sure this is a very good setup to investigate.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "First and foremost, I believe this work does not tackle ‘collaborative completion’ as each robot’s observations are independently processed into a corresponding map (which are then trivially combined) (W1). The primary benefit of the proposed approach is a more compressed representation of a map s.t. it can be communicated easily, but there alternate methods like VQ-VAEs would perhaps be even better (W3). Further, there are also concerns regarding the presentation (W2) and the positioning (W4).\n\nWhile the temporal amortization of communication is an interesting and novel nugeet, I think the above concerns far outweigh this aspect.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Please do correct me if I am misinterpreting the ‘independent’ nature of the per-robot map reconstructions.\n\n- I would be curious to hear the authors’ thoughts on why alternate compression schemes e.g. VQ-VAEs would be less preferred compared to the method here of using a MAE-like network.\n"}}]

Author rebuttal:

Decision: Accept (poster)