CoViS-Net: A Cooperative Visual Spatial Foundation Model for Multi-Robot Applications
Authors: Jan Blumenkamp
Steven Morad
Jennifer Gielis
Amanda Prorok
Venue: CoRL 2024 (2024)
Keywords: Multi-Robot Systems
Robot Perception
Foundation Models
TL;DR: We propose a decentralized, platform-agnostic visual spatial foundation model for multi-robot systems, enhancing spatial understanding through data-derived priors for real-world, online deployment.

Abstract: Autonomous robot operation in unstructured environments is often underpinned by spatial understanding through vision. Systems composed of multiple concurrently operating robots additionally require access to frequent, accurate and reliable pose estimates. Classical vision-based methods to regress relative pose are commonly computationally expensive (precluding real-time applications), and often lack data-derived priors for resolving ambiguities. In this work, we propose CoViS-Net, a cooperative, multi-robot visual spatial foundation model that learns spatial priors from data, enabling pose estimation as well as general spatial comprehension. Our model is fully decentralized, platform-agnostic, executable in real-time using onboard compute, and does not require existing networking infrastructure. CoViS-Net provides relative pose estimates and a local bird's-eye-view (BEV) representation, even without camera overlap between robots, and can predict BEV representations of unseen regions. We demonstrate its use in a multi-robot formation control task across various real-world settings. We provide supplementary material online and will open source our trained model in due course.
https://sites.google.com/view/covis-net

CoViS-Net: A Cooperative Visual Spatial Foundation Model for Multi-Robot Applications
Jan Blumenkamp, Steven Morad, Jennifer Gielis and Amanda Prorok Department Computer Science and Technology
University of Cambridge, United Kingdom {jb2270, sm2558, jag233, asp45]8cst. cam ac.uk uk
Abstract: Autonomous robot operation in unstructured environments is often un- derpinned by spatial understanding through vision. Systems composed of multiple concurrently operating robots additionally require access frequent, accurate and reliable pose estimates. In this work, we propose CoViS-Net, decentralized vi- sual spatial foundation model that learns spatial priors from data, enabling pose estimation as well as spatial comprehension. Our model is fully decentralized, platform-agnose executable in real-time using onboard compute, and does not require existing networking infrastructure. CoViS-Net provides relative pose es- timates and a local bird's-eys purd's-eye-viet (BEV) representation, even without camera overlap between robots (in contrast classical methods). We demonstrate its use in multi-robot formation control task across various real-world settings. We provide code, models and supplementary material online!
Keywords: Multi-Robe Nubti-Robot Systems, Robot Perception, Foundation Models
Figure 1: Our model can be used to control the relative pose of multiple follower robots (depicted yellow and magenta) to leader robot (blue) following a reference trajectory using visual cues in the environment only (the field-of-view is depicted as colored area). Even if there is no direct visual overlap (intersection of colored areas), the model able to estimate relative pose.
Introduction
Spatial understanding is cornerstone of robotic operation in unstructured environments, relying effective pose estimation and environmental perception [1, 2, 3, 4, 6. 7, 8]. While GNSS, LiDAR, and UWB sensors have been instrumental in advancing robotics, they are limited by constraints such as indoor operation, unreliability when operating around reflective surfaces or in bright envi- ronments. In contrast, color cameras offer a low-cost, energy-efficient and rich data source, aligned with the vision-centric design of real-world, human-designed environments. Classical vision-based techniques [1, 6, 5] struggle with the ill- ill-posed nature of image processing, cannot leverage semantic priors to resolve ambiguous situations, and may converge a solution only after multiple time steps. These challenges are exacerbated in multi-robot scenarios that require fast and accurate relative pose
hatops://proookllbb http:///rrrrrrroaiiiuuuu...bbooe github..oo/Cois--Met
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

1/1
1/3
1.
Prediction Ground Trith y [m]]
+ [m]]
-2.5 0.0 2.5
2.5
0.0
2.5 2.5
0.0
2.5
2.5
0.0
2.5 2.5
0.0
2.5
x [m]]
x [m]
[mm]
x [m]]
[m]]
Figure 2: By leveraging spatial priors, our model provides rough relative positions even without overlapping fields of view, which classical approaches cannot achieve. This figure shows sample from the simulation validation dataset Dian for five nodes, each represented by a column. The top row displays the respective camera image 7, The middle row shows the ground truth labels in the coordinate system Fü, with each robot centered and facing upwards at (0, 0), and the ground truth poses Pl, of other robots, along with their field view. The background shows the BEV represen- tation BEV, for each robot. The bottom row presents the corresponding predictions, displaying pose predictions Pi,iy R,,i5 with low uncertainty o R.yy' predicted uncertainties of, dashed oval and of ij similar to the FOV, and the predicted BEV representation BÉV; in the background.
estimates for tasks like flocking [9], path planning [10], and collaborative perception [11]. Explicit agent detection [4, 3, 2, 7, 8] requires line-of-sight and is not
whereas deep pose predictors [12, 13, 14, 15, 16, 17, 18] show promise but have are not widely adopted multi-robot systems due to hardware constraints. Additionally, incorporating uncertainty metrics is essential for assessing the reliability of predictions in these applications.
Acknowledging these limitations, we introduce CoViS-Net decentralized visual spatial foundation model targeting real-world multi-robot applications. This model operates in real-time on onboard computers without the need for pre-existing network infrastructure, thus addressing critical scalabil- ity, flexibility, and robustness challenges. CoViS-Net features distributed architecture designed for uncertainty-awwn pose estimation and BEV representation prediction in areas without local camera coverage. We demonstrate its effectiveness in real-world scenarios, including multi-robot control tasks in real-wort world indoor and outdoor environments.
In summary, our contributions are threefold: (1) We introduce a novel architecture for decentralized, real-time multi-robot pose estimation from monocular images, with built-in uncertainty awareness. (2) We extend it to predict BEV representations, facilitating spatial understanding in occluded areas through communication and by leveraging spatial priors. (3) We validate our model the real-world for multi-robot control task, demonstrating robustness even in scenarios with no image overlap.
2 Related Work
Multi-robot systems require coordination of fast-moving robots in close proximity, necessitating spatial understanding and accurate pose estimates. Key applications include flocking [9], formation control [19], trajectory deconfliction [20], multi-agent object manipulation [21, 22], area cover-

---

age [23, 24], and exploration [25, 26, 27, 28). Recent advances use Graph Neural Networks (GNNs) for enhanced control and perception, enabling inter-robot communication through latent messages to address the challenge of partial observability [29, 30, 31, 32, 33, 34, 35]. Environment inference deals with predicting unobserved environmental information [36, 37j. However, significant gap remains in acquiring relative pose knowledge with uncertainty estimates.
Map-free relative pose regression [17] estimates the relative pose between two camera images without scene-specifie training. Traditional methods [38] and learning-based approaches [39, 40, 41, 42] require significant visual overlap between views. Integrating scale involves depth maps or Siamese networks [12, 43, 44, 45, 14, 13, 15]. These models lack uncertainty estimates, limiting their roboties applicability, and have not been tested in real-world, real-time deployments. Our approach provides reliable relative poses with aleatoric uncertainty estimates, beyond visual overlap constraints, using unmodified pre-trained foundation models, enabling transfer learning [46]-
BEV representations are an effective way of providing detailed spatial comprehension by merg- ing data from a set of sensors. Learning-baa -bsen approaches have recently gained popular- ity [47, 48, 49, 50, 51J. Multi-agent scenarios [34, 33, 52, 35, 53, 54, 25] rely on GNSS for global pose knowledge, precluding application indoor environments, or rely on an existing SLAM stack. Our work proposes BEV representation predictor that utilizes relative poses predicted from monoc- ular images, instead of known static poses of camera rig or poses derived from external sensors.
Foundation models are large neural networks trained on broad data using self-supervised learning, exhibiting emergence and homogenization [55]. These models, often based on the transformer ar- chitecture [56], span several types: Large Language Models (LLLMs) [57, 58), vision models [59], Vision-Language Models (VLMs) [60, 61], and Vision-Language-Aciin Models (VLAs) [62, 63]. Recent trends include open-source development [64, 63] and significant scaling of model parame- ters. Deploying these models in robotics faces challenges due to memory and latency constraints, necessitating techniques like distillation [65] and quantization [66] for real-time processing.
3 Problem Formulation
Given a multi-robot system, our goals are: (i) for each robot to predict its pose and uncertainty relative other robots as well as corresponding embeddings using non-line-of-sight visual cor- respondences, and (ii) to use these embeddings for downstream tasks like local occupancy grid prediction. Consider a multi-robot system represented by a set of nodes V. Each node has position P and an orientation Rw, in the world coordinate frame Fu. The set of edges C V x V defines the communication topology and thus the graph g = (V,E). For each edge (u;, 0y) E E, the relative position of U in the coordinate frame of v, (denoted as Fi) is Pi,ú = R (Pw.j Pw,s), and the relative rotation is R.,,j R R.w.j- Fach node vs is equipped with a camera C., providing an image 4, From the images, estimate relative poses and uncertainties (pr..j. op.3. Ri,ig, R..y)) Vu (N(u), with respect to v; in its coordinate frame Fi This approach does not assume overlap between images or line-of-sight visibility between robots, relying exclusively on the relationship between images, thus making it platform-agnostic Using these estimated poses, the robots cooperatively generate a BEV representation BÉV, around each node v, which can be used for various downstream multi-robot applications.
V
4 Approach
In this section, we detail our architecture and training approach. We explain dataset generation, de- fine our model architecture, and introduce the uncertainty estimation strategy and training protocol.
4.1 Model
We employ four primary models; The encoder fewe generates common embedding of images, fpose estimates pairwise poses, fagg aggregates node features, and feev predicts two-dimensional BEV
3

---

f
E
Eij
Pl. o R, R
f poss
|
Fíj
Jeac
fage
>
E
F
Ei
F
Paii Daske R, R
fosse
|
Fi
/BEV
fene
Ik
Ex
BÊV
Figure 3: Overview of the model architecture: We illustrate the decentralized evaluation with spect to ego-robot 24 which receives embeddings E,, Es generated by the encoder fene from the neighbors 8j UA (N;).. For each received embedding, concatenate the pose embedding E, with the image embedding E, (and Eg with Eg) and subse- quently aggregate this into node feature F, Model outputs are framed with dashed line.
employ fpose to compute the pose. We
representation. An overview of the model architecture is provided in Sec. 4.1. Our model uses transformers [56, 67] in fencs fpose and fagg- We provide more details in Appendix A.1.
Image Encoder: We use the smallest distilled version of Dino V2 [59] (ViT-S/14) as image encoder due to its robust feature representation and spatial understanding. Each robot & processes the image I, through fane generate the communicated node embedding E, E 5xP where S represents the sequence length, and F the feature vector size. We freeze the weights of DinoV2, enabling the use of various open-source downstream fine-tuned image detectors without additional forward pass.
Pairwise pose encoder: Each robot v;) broadcasts the image encoding E; to neighboring robots. Given encodings E, and Ej. fpose estimates edge embeddings E, and relative poses with un- certainties (Pi,aj op.ey. Ru,ig, k.yy)) For each incoming E, V ey E NN(u;), we concatenate the node features along the sequence dimension, add the positional embedding, run the trans- former and then extract the first element the sequence dimension to obtain the edge embed- ding. We then use this edge embedding to estimate position, rotation and uncertainties, resulting in Spoce E, E,) = (Pr.i, p.újs Ri,i k..)).
Multi-Node Aggregation: We perform position-ar posttion-aware E, from fpoee The function fags integrates information from all nodes vj E eN(1)) Uw;, producing feature vector F; for each node U; encapsulating the aggregated information from its local vicinity.
aware feature aggregation using node embeddings
We concatenate node features along the sequence dimension and add the positional embedding. We then concatenate a nonlinear transformation the edge features along the feature dimension. This operation is performed for all U; E (w;),, including self-loops to handle cases where (N;;) empty by zeroing the edge embedding. We aggregate over the neighbors and the ego-robot to produce the output embedding F, from all edge embeddings Fij, which is used for downstream tasks, such estimating BEV representation BÊV, fRE)
4.2 Training
For training our models, we use supervised learning. This section describes the dataset, loss func- tions, and uncertainty estimation.
Dataset: We use the Habitat simulator [68, 69] and the HM3D dataset [70], consisting of 800 scenes of 3D-scanned multi-floor buildings. The data divided into training Dšim (80%), testing Sim (19%), and validation Sia (1%) sets. We provide more details in Appendix A.2.
Uncertainty Estimation: Two types of uncertainties be modeled: epistemic uncertainty, which captures uncertainty in the model and can be reduced with more data, and aleatoric uncertainty,

---

which captures noise inherent to specific observations [71]. For pose estimation and pose control in robotic systems, aleatoric uncertainty is more important since downstream applications can act on to reduce Traditional methods like Monte Carlo Dropout [72] and Model Ensembles [73] model epistemic uncertainty. Instead, we propose using the Gaussian Negative Log Likelihood (GNLL) Loss [74, 71], a generalization of the mean squared error (MSE) loss, allowing the model predict both mean ji and a variance o², learned from data points J.. The loss is defined as
1 log (62) + (A 14) 0²
CGNLL (, R. (ON
=
The GNLL loss penalizes high variance predictions and scales the MSE loss by the predicted vari- ance, penalizing errors more when predicted uncertainty is high.
Estimating Rotations: Estimating rotation is difficult due to discontinuities in many standard ori- entation representations. Quaternions are often used for their numerical efficiency and lack of sin- gularities. Related work [75] represents rotations with symmetric matrix defining a Bingham distribution over unit quaternions to predict epistemic uncertainty. We combine the chordal loss and GNLL to predict aleatoric uncertainty for rotations, as shown in the following equation:
dquaa (q, q) min (1lq 41l2 |/q áll2)
(2)
Lehord (q, ④ 2d (q, ④) (4 Lauat (4 q)
(3)
log (6²) Sod (á q)
ehord (q. q, 02)
(4)
Training:
We use a combination of the Dice loss [76] LDice and Binary Cross Entropy (BCE) Loss Lee weighted by the parameter 0 S a < 1. This is commonly referred to as combo loss [77] and typically utilized for slight class imbalance. We define the total loss as
L,B = a Diic BEV, ,BV.) +(1 (1-aa) LBCE BEV,, BÉV,
(5)
Lisj, = -1- 3) CONLL (pic) p.is)) (ois))+ + cchood (Ri,ij. Rasy, R..))
(6)
L= Σ V.V
L.BEV +
Σ MJEN(M))
Liss
The parameter 0 S B < balances between the position and orientation loss. Uncertainty terms are necessary for training, as many dataset samples lack overlap and may be adjacent rooms, leading to significant errors from difficult samples that could hinder learning. We train on simulated data Dim and provide further details in Appendix A.4.
5 Experiments and Results
We demonstrate the performance of our trained models on the test set Dijn and custom real-world test sets DReal First, we outline the metrics used. Then, we introduce the real-world dataset. Next, we analyze model performance on both simulation and real-world data. Finally, we compile and deploy our model in real-world, real-time trajectory tracking task.
We compute the Euclidean distance between the predicted and ground truth poses as D Dpos (Pi.ij Pr,i)) and the geodesic distance between rotations as Droc (R,,j, R.,.)) for all edges and report median errors. We categorize the median pose enror into four categories. All includes all edges. Visible and Invisible edges are determined by FOV overlap (an edge invisible Drot (Ri,ij, 0) > FOV). Invisible Filtered samples use the predicted position uncertainty of.3 to reject high- high-uncertainty pose estimations based on the Youden's index [78]. For simulation results, also report the Dice and loU metric. We provide more details in Appendix B.I.
Metrics:
5

---

Table I: Ablation study over the number of patches S and size of features F per patch. We report the BEV representation performance and the median error for poses on the dataset Dim and D
Model
Direct
Deal Invisible
s
F
Dice
IoU
All
Invis. Filt.
Visible
256
48
69.1
57.1
36 cm
8.3
61 cm
6.8
97 cm
7.9
33 cm
5.8
128
96
68.8
56.8
40 cm
8.4"
55 cm
7.7
97 cm
7,4
32 cm
5.6
128
48
67.9
56.1
38 cm
7.7
67
9.9
113
9.6"
29
5.7
128
66.7
54.6
50 cm
9.5"
83 cm
11.2
112 cm
9.7"
cm
5.7
61.0
43.5
51 cm
9.6
cm
9.49
119 cm
10.8
36 cm
6.3
3072
47.0
144 em 84 cm
89.9
123 em 150 cm
25.7"
122 cm 135 cm
25.8"
em
138,2
348
47.1
1.4
11.7
164.1
37.9"
80 cm
11.1
5.1 Dataset Evaluation
We conducted experiments across various model variants trained with different parameters, reporting on our model's 's efficiency. We present quantitative results in Tab. and qualitative findings in Fig. 2.
Real-World Dataset:
Our setup includes four Cambridge RoboMasters [79] and one Unitree Gol Quadruped, each equipped with local compute, monocular camera and WiFi dongles for adhoc networking. Our custom real-world test dataset DReal covers diverse and challenging range of indoor and outdoor scenes. We provide more details in Appendix A.3.
Results: Tab. details the performance across seven different experiments. We evaluated the impact of image embedding sizes E, on the performance. The IoU and Dice metrics for the BEV represen- tation, alongside median pose estimates for all samples in the dataset, provided. Experiments show that the sequence length S has more significant influence BEV representation and pose estimate performance than the feature size F. We set S to 1 for two experiments: (I) increasing F to match the size of the embeddings for other experiments, and (2) decreasing F, effectively reducing the transformer to a CNN-only architecture by removing the attention mechanism. Both experiments result reduced performance, highlighting the importance of attention in our model. The best- rest-performing model with S 256 achieves an IoU of 0.571, Dice score of 0.691, and me- dian pose accuracy of 36 em and 8.3%, which is roughly equal to one robot body length. In contrast, models with S show significant degradation, with nearly unusable BEV representation predic- tions and pose errors on the scale of meters. Models with F = 24 and S = 64 performed slightly worse than those with larger F' or S. We provide additional results in Appendix B.2. Evaluation on the real-world dataset Real focuses on position and rotation accuracy, excluding quantitative BEV performance due to unavailable ground truth data. The results align with Diin showing a correlation between the number of patches S and features F and pose estimation performance. The rotation estimates do not significantly differ between visible and invisible samples, demonstrating robust pose estimation without requiring visual overlap. The most accurate model achieves median localization error of 97 em and 7.9% for invisible samples, and 33 em and 5.8" for visible samples. Test samples are at most 2 m apart, resulting in a worst-case error of up to 4 m. These findings demonstrate our model's capability to reliably estimate poses, and accurately predict higher uncertainties under adverse conditions. We attribute this performance to our pose prediction model, as well as the robust spatial representation capabilities of the DinoV2 encoder. The model goes be- yond mere feature matching, achieving sophisticated scene understanding that allows to "imagine" unseen portions of the scene based on learned data priors, thus facilitating pose estimation through environment inference [36, 37]. We provide additional results in Appendix B.3.
Note that Real
Finally, we conduct a baseline comparison with two feature- mawiurr-aatccing--e maase approaches, as summa- rized in Table 2. Both baseline methods extract and match features across two images to estimate the essential matrix, deriving rotation and scale-less translation vectors. The first baseline uses OpenCV for ORB feature extraction followed by brute-force matching. The second baseline. LightGlue, uses neural network- based approach for feature extraction and matching. Given both baselines estimate scale-less transformations, we adopt the same metric used in LightGlue [42], computing the AUC

---

Table 2: We report the FPS and AUC metric at 20, 45 and on two baselines.
All
Invisible
Visible
AUC@
FPS
20
45
90
20
45
90
20
45
90
ORB/OpenCV [80]
2.79
6.45
15.43
28.26
0.16
1.60
7.43
9.41
21.92
38,03
LightGlue [42]
1.17
16.30
27.14
37.63
0.02
0.09
0.31
23.94
39.82
55.14
Ours
42.73
25.23
47.63
66.34
9.22
24.40
45.74
32.74
58.53
76.01
at thresholds of 20, 45, and 90 degrees based the maximum error in rotation and translation. Our findings indicate that our model surpasses both baselines in performance and speed across image pairs with and without visual overlap. Notably, while both baselines require visual overlap to func- tion effectively, our approach does not, highlighting its robustness and applicability a broader range of scenarios.
5.2 Real-Worke Pose Control
In this section, we describe our experiment setup (including model deployment, wireless communi- cation, and controllers), demonstrating real-world multi-robot control tasks.
Model deployment: We deploy model with F 24 and S 128. After training in PyTorch, compile fenc, Spose. and fage for decentralized deployment. Using TensorRT, we achieve sub-30 processing times. We provide more details and runtime analysis in Appendix B.4.
Ad-hoc WiFi: Image embeddings are sent between robots at 15 Hz, each over 6 KiB. The system's distributed nature suits broadcast communication, allowing scalable operation. However, IEEE 802. I/WiFi's fallback to low bit rates for broadcast increases frame loss probability, worsening with more robots. Higher-level systems like ROS2's CycloneDDS manage retransmissions but can't detect network overloads, leading increased packet loss. We design custom protocol that dynam- ically adjusts messaging load based on network conditions and uses TDMA scheduling to maximize frame reception without compromising data rates. We provide more details in Appendix B.4.
Experiment: We conduct qualitative experiments various environments, including outdoors, and quantitative experiments. Two follower robots maintain a fixed orientation and distance from leader robot predefined trajectory. A PD controller manages relative pose control, adjusting based predicted uncertainty and selectively deactivating if uncertainty thresholds exceeded. This ensures that even with unreliable estimates, robots maintain orientation toward the leader un- til reliable estimates are available. We assess tracking accuracy by comparing ground truth and estimated poses, focusing on position and rotation errors and predicted uncertainties.
Results: Fig. shows snapshots from four different real-world deployments, and Fig. 4 shows the trajectories of all robots for all quantitative experiments, well as visualization of the position and rotation error over time. We provide additional results in Appendix B.4.
Pos Std. dew. [m]]
Rot Std. dev. ddgg
Pos Ear [n]]
Rot Eer (dec)
Phoor
1.0
0.4
80
Leader
200
0.0
0.0
[m]]
Time
Time
Time
Time [+]
Figure 4: We evaluate the tracking performance of our model and uncertainty- aware controller reference trajectories with two follower robots (blue and orange) positioned to the left and right of
the leader robot (green). We report tracking performance over time for position and rotation, well as the predicted uncertainties. The leader always faces the direction of movement. We show the trajectory for 120 and the tracking error for the first 60

---

Os
18
2s
3s
4s
5s
Figure 5: We show snapshots of real-world deployments in different scenes with up to four robots. Each row contains six frames from a video recording, each spaced 1s in time. We indicate the posi- tions of the robots in the first frame, where the leader is cireled yellow and the followers blue. The first three samples show indoor scenes, and the bottom sample is an outdoor deployment. The sec- ond sample shows heterogeneous deployment, combining robots of different sizes and dynamics.
The qualitative experiments show a robust system that generalizes to a wide range of scenarios, including outdoors, albeit with scale prediction that is less reliable outdoors. Our model was trained exclusively on indoor data and generalizes to outdoor data due to the wide range of training data used the Dino encoder. We furthermore show platform agnosticism by deploying our model to heterogeneous team of robots, lead by quadruped (second row in Fig. 5).
In quantitative experiments, the tracking error oscillates throughout the trajectories as the followers reactively adjust their positions to the leader's movements. Follower A starts facing the opposite di- rection of the leader, resulting in high position variance and low rotation variance, which decreases as the robot aligns with the leader. An erroneous rotation prediction is accompanied by high uncer- tainty. As the leader moves, one follower is inside and the other outside the trajectory, switching roles depending on the section of the figure. We report median tracking error of 48 em and 5,3 for Follower A, and 26 cm and 5,4" for Follower B (corresponding roughly to one robot body length).
6 Limitations
Foundation models are commonly trained unsupervised [55], whereas our model is trained on the HM3D indoor dataset in a supervised manner. Our model requires peer-to-peet communication to be available to merge information from other robots and GPU-accelerated onboard computing for real-time deployment. Previous work showed the scalability of GNNs due the principle of locality (number of neighboring nodes are constrained by the communication range) [29]. The number of real robots is currently constrained by our custom networking stack.
7 Conclusion
We introduced first-of-its-kind cooperative visual spatial foundation model that leverages exist- ing large vision models and demonstrated its use in multi-robot control. From monocular camera images only, our model is able to output accurate relative pose estimates and BEV representations by aggregating information across neighboring robots. We validated our model on custom real- world test set and demonstrated real-time multi-robot control. Our model accurately predicts poses, even without pixel-level correspondences, and correctly predicts BEV representations for obscured regions. In further real-world tests, we demonstrated accurate trajectory tracking, paving the way for more complex vision-only robot tasks. Our model estimates relative poses with a median error of 33 em and 5.8 for visible edges, and 97 em and 5.9 for invisible edges, roughly equivalent to one body length of our robot. In future work, we plan to include temporal data, apply this model to other downstream tasks and combine it with learning- based multi-agent control policies.
8

---

Acknowledgments
This work was supported in part by European Research Council (ERC) Project 949949 (gAla) and in part by ARL DCIST CRA 1911NF-17-2-0188. J. Blumenkamp acknowledges the support of the 'ttudienttifuung des deutschen Volkes' and an EPSRC tuition fee grant. We also acknowledge gift from Arm. We gratefully acknowledge Toshiba Europe Ltd, Cambridge Research Laboratory for providing the Unitree Gol used in our real-world experiments.
References
[1] D. Scaramuzza and F. Fraundorfer. Visual Odometry [Tutorial]. IEEE Robotics & Au- tomation Magazine, 18(4):80-92, 2011. doi: Ntt100.100/MRA22011...0 //ieeexplore ieee.
URL https:
[2] M. Saska, J. Vakula, and I.. Pfeućil. Swarms of miero aerial vehicles stabilized under visual relative localization. In 2014 IEEE International Conference on Robotics and Au- lomation (ICRA), pages 3570-3575, 2014, doi:10. 109.1//////AA221.060000
URL https:
// //ieeexplore ieee. org/ httgg//sstactt///oee (oocument/6907335
[3] O. De Silva, G. K. I. Mann, and R. G. Gosine. An Ultrasonic and Vision-Based Rela- tive Positioning Sensor for Multirobot Localization. IEEE Sensors Journal, 15(3):1716- 1726, 2015. doi: 111...11///EENN210.223266
URL https://leeexpporrt seeeeeeiiee.... ieee.
[4] V. Walter, M. Saska, and A. Franchi. Fast Mutual Relative Localization of UAVs using Ultraviolet LED Markers. In 2018 International Conference on Unmanned Aircraft Sys- tems (ICUAS), pages 1217-1226, 2018. 100110..10//CCAAS2211..... //1esexplore.ieed htt.://.......ooooeaaaarrrrrrren /ttp///ttaactcccccut////555....
URL https:
[5] J. L. Schönberger and J.-M. Frahm. Structure- froii--ttooooooonn ference on Computer Vision and Pattern Recognition (CVPR), pages 4104-4113, 2016. doi: 10.1109/CVP.22016.444 7780814.
Motion Revisited. In 2016 IEEE Con-
URL https://ieeexplore ieee. wtgg/abstraat//ocmmentt
[6] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and D. Tardós. ORB-SLAM3: An Accurate Open-Source Library for Visual, Transactions Robotics, 37(6):177-1890,, 2021. doi: 10. 100.1110000RRR222.....00 https://keeexpporee ieee
and Multimap SLAM. IEEE
URL
[7] A. Moldagalieva and W. Hönig. Virtual Omnidirectional Perception for Downwash Prediction within a Team of Nano Multirotors Flying in Close Proximity, 2023. URL https://arxiv. org/abs/2303 htte/abb/23033038888 03898.
[8] P. T. Morón, S. Salimpour, L. Fu, X. Yu, J. P. Queralta, and T. Westerlund. Benchmarking UWB-Baser Infrastructure-Free Positioning and Multi-Robot Relative Localization: Dataset and Characterization, 2023. URL attps://aarae //axxiv. . org/ abs/2305 08532.
[9] C. W. Reynolds. Flocks, Herds and Schools: A Distributed Behavioral Model. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 877, pages 25-34, New York, NY, USA, 1987. Association for Computing Machinery. ISBN (- URL btttps///oot http://ii..orro///// 01114//77711374000
89791-227-6. doi: 145.1..155//7411333444
[10] G. Sharon, Stern, A. Felner, and N. R. Sturtevant. Conflict-based search for optimal multi- agent pathfinding. Artificial intelligence, 219:40-66, 2015.
[11] Y. Zhou, J. Xiao, Y. Zhou, and Loianno. Multi-Robot Collaborative Perception With Graph Neural Networks. IEEE Robotics and Automation Letters, 7()):289-2296. 2022. doi: 1109/LRA.2022.3141666 URL ftttps://ieeexplore ieee org/ bbstract//ocumentt 9676458.

---

[12] I. Melekhov, J. Ylioinas, J. Kannala, and E. Rahtu. Relative Camera Pose Estimation Us- ing Convolutional Neural Networks. In Blanc-Talon, R. Penne, W. Philips, D. Popescu, and P. Scheunders, editors, Advanced Concepts for Intelligent Vision Systems, pages 675-687, Cham, 2017. Springer International Publishing. ISBN 978-3-319-70353-4 URL https: //link.springse.e.eeeeee...ee.ep springer. com/ 2taapter/1..0007978---3333333
[13] K. Rajendran, S. Mishra, L. F. Vecchietti, and D. Har. RelMobNet: End-to-Enn Relative Camera Pose Estimation Using Robust Two-Stage Training. In Karlinsky, T. Michaeli, and Nishino, editors, Computer Vision ECCV 2022 Workshops, pages 238-252, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-25075- URL https: www.ii.ssiiinge.... com/ Com/chapter//i 10.
K.
25075-0_18.
[14] C. Yang, Y. Liu, and A. Zell. RCPNet: Deep-Leaming based Relative Camera Pose Estimation for UAVs. In 2020 International Conference Unmanned Aircraft Systems (ICUAS), pages 1085-1092, 2020. 00:10001100/CUAA/887-22202222222
URL https://ieeexppore.
[15] C. Yang. Y. Liu, and A. Zell. Learning- Camera Relocalization with Domain Adaptation via Image-to-Image Translation. In 202/ International Conference on Unmanned Aircraft Systems (ICUAS), pages 1047-1054, 2021. doi: 06:110.1099/CUU511844.4000 20.767777..... URL ieee. http:e/pppprrre.....oorrraaaaaaa
[16] R. Cai, B. Hariharan, N. Snavely, and H. Averbuch-Elor. Extreme Rotation Estimation using Dense Correlation Volumes. IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), 2021. URL https://openacee
http:/f.coocooott/eete Rotation__Ettinatioon Estimation Suiit_snnnnnn_e
CVPR2021/html/Ca. Extreme Rotation Dolumes_CVPR__0 2UPPP__2221____111 pwwppp__o2_pppper_.___ www.pppppprttt. html.
[17] E. Amold, J. Wynn, S. Vicente, G. Careia-Hermando, A. Monszpart, V. A. Prisacariu, D. Tur- mukhambetov, and E. Brachmann. Map-free Visual Relocalization: Metric Pose Rela- tive to Single Image. In ECCV, 2022. URL https: acm wtt.cooor////o1.1100... 178-3-031-197633
[18) C. Rockwell, J. Johnson, and D. F. Fouhey. The 8-Point Algorithm an Inductive Bias for Relative Pose Prediction by ViTs. In 2022 International Conference on 3D Vision (3DV), pages 1-11, 2022. doi: 200i000..100/DD55568.2222222
abstraatt//ocumee//1044433 202..0000002220000.222 URL attps://1eeexploo. ieee. org/
[19] K.-K. Oh, M.-C. Park, and H.-S. Ahn. A survey of multi-agent formation control. Automatica, 53:424-440, 2015.
[20] Van den Berg, M. Lin, and D. Manocha. Reciprocal velocity obstacles for real-time multi- agent navigation. IEEE International Conference on Robotics and Automation, pages 1928- 1935, 2008
[21] A. Martinoli, K. Easton, and W. Agassounon. Modeling swarm robotic systems: A case study in collaborative distributed manipulation. The International Journal of Roboties Research, 23 (4-5):415-436, 2004.
[22] Alonso-Moot Knepper, Siegwart, and Rus. Local motion planning for collaborative multi-robot manipulation of deformable objects. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5495-5502, 2015. 000:0..109//CRA.211...11
[23] A. Zelinsky, A. Jarvis, J. Byrne, S. Yuta, and others. Planning paths of complete coverage of unstructured environment by mobile robot. In Proceedings of international conference on advanced robotics, volume 13, pages 533-538. Citeseer, 1993.
10

---

[24] S. L. Smith. M. Schwager, and D. Rus. Persistent Robotic Tasks: Monitoring and Sweeping in Changing Environments. IEEE Transactions on Robotics, 28(2):410-426, 2012. doi:10. 109/ TRO.2011.217444
[25] A. H. Tan, S. Narasimhan, and G. Nejat. 4CNet: A Confidence-Awarr. Contrastive, Condi- tional, Consistency Model for Robot Map Prediction in Multi-Robo Environments, Feb. 2024. URL htw.:ttpp///// /ttp///axvvvv.. httpaaawww....aaaa///aaaaaaaaara 200p././04..... 17904.
[26] H. Wang, A. H. Tan, and G. Nejat. NavFormer: Transformer Architecture for Robot Target- Driven Navigation in Unknown and Dynamic Environments. IEEE Robotics and Automation Letters, 9()):888-668 6815, Aug. 2024. ISSN 2377-3766, 2377-3774. doi:10. 109/LRA. 3412638. URL
arg/bbs/24022 06838.
[27] A. H. Tan, Bejarano, Y. Zhu, R. Ren, and G. Nejat. Deep Reinforcement Learning for De- centralized Multi-Robot Exploration With Macro Actions. IEEE Roboties and Automation Let- ters, 8(1 8(1):272-279, Jan. 2023. ISSN 2377-3766, 2377-3774, doi: 109/1 URL https: petttps://ieexpplo.0
leee. org/ /
[28] S. C. Mohamed, A. Fung, and G. Nejat. Multirobot Person Search System for Finding Mul- tiple Dynamic Users in Human-Centeree Environments. IEEE Transactions Cybernetics, 53(1))628-640) Jan. 2023. ISSN 2168-2267, 2168-2275. httttttss///eeee.t
leee. org/ document/97657763
[29] E. Tolstaya, Gama, J. Paulos, G. Pappas, Kumar, and A. Ribeiro. Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 671-682. PMLR, Nov. 2020. URL https: //proceedings. mlr www.ppressseee tolstoyalasaaaaa.a html.
[30] F. Gama, E. Tolstaya, and A. Ribeiro. Graph Neural Networks for Decentralized Controllers. In ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 5260-5264, 2021. doi: 10. 10011////ASSS9999882000000
9411539..... URL
https://ieeexplorr/ ieee. hedd/deetttt1445550
[31] Gama, Q. E. Tolstaya, A. Prorok, and Ribeiro. Synthesizing Decentralized Controllers With Graph Neural Networks and Imitation Learning. IEEE Transactions Signal Process- ing, 70:1932-1946. 2022. 1oi:10.1109/11 10911900009997)222223336661) URL
ieee. wrgg/d/cuent////e0 9755001.
[32] Blumenkamp. S. Morad, J. Gielis, Q. Li, and A. Prorok. A Framework for Robot Systems Running Decentralized GNN-Based Policies. In 2022 International Conference on Robotics and Automation (ICRA), pages 8772-8778, 2022. doi: 1oi:10.11 11/11100/////44333...00 9811744. URL https: /ttp///eeeeeeereeeee// ieee org/ document/998111
Multi-
[33] X. Runsheng, X. Hao, T. Zhengzhong, X. Xin, Y. Ming-Hsuar and M. Jiaqi. V2X-VIT. -ehieli-to-Exeryytiing Cooperative Perception with Vision Transformer. In Proceedings the European Conference Computer Vision (ECCV), 2022. URL https: //d1 acm. org/ doi/ abs/10. 1007 / 0/////97733001 -1188-3--011----- -984-1--22---.7
[34] R. Xu, Z. Tu, H. Xiang, W. Shao, Zhou, and J. Ma. CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers. In K. Liu, D. Kulic, and J. Ich- nowski, editors, Proceedings of The 6th Conference Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 989-1000. PMLR, Dec. 2023. URL https://proceedingg mlr ppwss/v205/xuiii xu2aa htmll.
[35] Y. Li, J. Zhang, D. Ma, Y. Wang, and C. Feng. Multi-Robot Scene Completion: Towards Task- Agnostic Collaborative Perception. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning

---

Research, pages 2062-2072 PMLR, Dec. 2023. URL https://pprcceedngs mlr, press/ v205/1i23e html.
[36] A. H. Tan and G. Nejat. Enhancing Robot Task Completion Through Environment and Task Inference: A Survey from the Mobile Robot Perspective. Journal of Intelligent & Robotic Sys- tems, 106(4):73, Dec. 2022. ISSN 0921-0296, 1573-0409. 110:::0..0077111184002--0-111110
//iink.//nn////// http://iinn...iiner.r.r. wwwpenger..ommm//o/ com/10. 1007 001.00//1000400022-0777700100000
[37] A. Smith and G. A. Hollinger. Distributed inference-based multi-robot exploration. Au-
tonomous Robots, 42(6):1651 Dec. 2018. ISSN 0929-5593, 1573-7527. doi:10.1007/
URL htt.::ttt::/iiiti httpa/a......oooo.oo.o////000000
[38] D. G. Lowe. Distinctive image features from scale-invariant keypoints. Interna-
lional journal of computer vision, 60:91-110, 2004. URL httpss//iinsssprinngg
com/article/ certiccee100.. 1023//: VIw.:...........00004....00 99615. 94. Publisher: Springer .eprint:
[39] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. SuperGlue: Learning Feature Matching With Graph Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL httpstt:::/ Utt.https://ppeaaccee wtw.//cccoccoooooo ctt. cttpvv..cc//oooeeenee contenttttttttccccee CVPR_ htt00///tmm/araannSaaaeeeeeeee/e Ltwrnprrrrleeeeeaennnnnnngn Feature thecvf.
Matching_Ma Siigg_WWttte Wiith_o Graph_Neuraa Neural Networks_ Saarrrkss_vPP______ B_VPR_202______21_______ _appppppp.p...- pappp..pml...................m.p html.
[40] Sun, Shen, Y. Wang, H. Bao, and X. Zhou. LoFTR: Detector-Free Local Feature Matching With Transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattenn Recognition (CVPR), pages 8922-8931, June 2021. URL https: //openaccess. thecvf wwt.coocccccccccec http://ceeeet////2////////////// Local_ Leral__FeatureeMMatchte With Tra.tthhrrrannrmerrrrssaeaa 2021 1app21pppppp____ html.
[41] R. Pautrat, Suárez, Y. Yu, M. Pollefeys, and V. Larsson. GlueStick: Robust Image Matching by Sticking Points and Lines Together. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 9706-9716, Oct. 2023. URL https://openaccess. thecvf. htteerr.....caoeeee//////////aaa Robust_Immage Baaes___maaa_Maccinnnnn.. paper.html. .ttl..
[42] Lindenberger, P.-E. Sarlin, and M. Pollefeys. LightGlue: Local Feature Match- ing Light Speed. In Proceedings of the IEEE/CVF International Confer-
on Computer Vision (ICCV), pages 17627-17638, Oct. 2023. URL https: popeenaccess.hecrree
html.
[43] Z. Laskar, I. Melekhov. S. Kalia, and J. Kannala. Camera Relocalization by Computing Pair- wise Relative Poses Using Convolutional Neural Network. In The IEEE International Confer- ence on Computer Vision (ICCV), Oct. 2017. URL https://githun com www.aab.ccc//aatoiiiiioooo aamerarrelocalssatioon
[44] S. En, A. Lechervy, and Jurie. RPNet: an End-to-End Network for Relative Camera Pose Es- timation. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018. URL https: httph://tiiiii h/t::://nnnnnne. springer. wwwp:ngerr.co/ccceaaaeeeeeeee http:p/gggg.....a/aa/////////./a
[45] Y. Abouelnaga, M. Bui, and S. Ilie. DistillPose: Lightweight Camera Localization Us- ing Auxiliary Learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7919-7924, 2021. 11011011199900051111822222933939 URL
https://seeexploreeeeer ieee wwwp//ocmet/t//aaaa....
12

---

[46] K. Weiss, T. M. Khoshgoftaar, and D. Wang. A survey of transfer learning, Journal of Big data, 3:1-40, 2016.
[47] Roddick and R. Cipolla. Predicting Semantic Map Representations From Im- ages Using Pyramid Occupancy Networks. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), June 2020. URL.
https://openaccesat thecvf com/ Seatteett_VPP__220000 htRodddikm/ http:/cii..Seommccoc..eeeeeeeeee hrw.r//ppppperraaaoooooooooooooa
Sltages_Usinn_Pyae
paper... html.
[48] Philion and S. Fidler. Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting 3D. In Proceedings of the European Conference Computer Vision, 2020. URL https:t////// //dl. acm. org/ /ts////aa///////
aat/aaaaaa/00.//. 1007000000010000 / 99/0//799---11
12.
[49] C. Yang. Y. Chen, H. Tian, C. Tao, X. Zhu, Z. Zhang, G. Huang, H. Li, Y. Qiao, L. Lu, Zhou, and J. Dai. BEVFormer v2: Adapting Modern Image Backbones Bird's-Eyer View Recognition via Perspective Supervision. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 17830-17839, June 2023. URL ctttps://penaccess.. thecvf com http///eenn////2222//ttlll////// Bttp///een_BBBoooeee... ddapting_Modese 2023 2023_paper html.
[50] Y. Liu, Yan, F. Jia, S. Li. A. Gao, T. Wang, and X. Zhang. PETRv2: A Unified Frame- work for 3D Perception from Multi-Camera Images. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 3262-3272, Oct. 2023. URL https://openaccess: /tterrtccc///oottt//////////////
Unified_Framework for 3D f htg.://utttitttaaaeaamaaa..oooL. paper html.
[51] L. Peng. Chen, Fu, P. Liang, and E. Cheng. BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs. In Proceedings the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 5935-5943, Jan. 2023. URI. thecvf. com/ Leads_Eye_Viece http://eeeeaattee.ppaaeaeaaeeeea 2023 2023_paper: html.
_
[52] Y. Li, D. Ma, Z. An, Z. Wang, Y. Zhong, S. Chen, and C. Feng. V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving. IEEE Robotics and Automation Letters, 7(4):10914-1092,, 2022. 100:10..10//LRA.222..12222 ieee. www.eeeeeeeoooeeeeeeeeeeeeeeeeee www.:/o///ooe//e/eeeeeooo
URL
[53] T.-H. Wang. S. Manivasagam, M. Liang, B. Yang, W. Zeng, and R. Urtasun. V2VNet: Vehicle- to- Vehicle Communication for Joint Perception and Prediction. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision ECCV 2020, pages 605-621, Cham, 2020. Springer International Publishing. ISBN 978- 978-3-030-58536-55 URL https://link springer com/ http/chapter/eeee / 0007/998-3-000 58536-5_36.
[54] Y. Han. H. Zhang, H. Li. Y. Jin, C. Lang, and Y. Li. Collaborative Perception in Au- tonomous Driving: Methods, Datasets, and Challenges. IEEE Intelligent Transportation Systems Magazine, 15(6):131-151 2023.
URL https:
// //ieeexplore. ieee org/ org/document/1024800 110244449
[55] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, Bohg, Bosselut, E. Brunskill, E. Brynjolfsson, Buch, D. Card, R. Castellon, N. Chat- terji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong,
13

---

K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchan- dani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, C. Nicbles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, Papadimitriou. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang. On the Opportunities and Risks of Foundation Models, 2021. URL argg/bs///ii 07258.
[56] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URL https: .ips. cc/paper_ fi files/paper/2a
/ httpa///222aaeeeeee2ee22ee2aa112 Abstrrac. tatsstraat.ttt........ html.
[57] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for I Language Understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pages 4171-4186. Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. 10i:10.18653// 18653/v1/N19-14235
URL hatp:p:ss//aaaatthooiooo aaclaaccnaooooyooooo org/N19- MergNN1--423..
[58] T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Vosss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Rad- ford, Sutskever, and D. Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, Curran As- sociates, Inc., 2020. URL https::/pproceeednngg nwwriigg...........ee...r.... wttppppppcpppp....eeeeeeee files/ files/paper/ (o00://iie//Ssoeeeecrro/////ooo0
pdf.
[59] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, 2023. URL https::::///ttttt cww./aaccbooorreeerc//////////// mremmmainn Publication Title: arXiv:2304.07199 eeprin:: https:/aarrivorr//p////000.....0
[60] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Mod- els From Natural Language Supervision. In M. Meila and T. Zhang, editors, Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pages 8748-8763. PMLR, July 2021. URL https: html.
//proceedings.nit mlr www.ppesssvee catoss//v3//raafoooro
[61] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-Shot Texa-to-Image Generation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR, July 2021. URL https: htap:s:/pproceeeee http:///ooceeens.... mlr. wwwps/vvvivv html..
14

---

[62] A. Brohan, N. Brown, J. Carbajal. Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, and others. Rt-1: Rohotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.068// 2022.
[63] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, Nair, R. Rafailov, Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An Open-Source Vision-Languuge-Action Model, 2024.
URL https: /tapps://eei hates://aarxe /ttp///aavvvv///e///a//eeeeawwva orgg/bs////// 099./4440.00066....
[64] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar, and others. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[65] G. Hinton, O. Vinyals, and J. Dean. Distilling the Knowledge in Neural Network, 2015, URL https://arxiv org/
[66] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arihmetti-nnly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704- 2713, 2018.
[67] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Con- ference on Learning Representations, 2021. URL https: //tpp//////eeeeiee.// net/forummidd YicbFaNTTy.
[68] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied Al Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. htt.o///11///ttt////aaaaaaaaaaa. A_Platform_ for Baltooorm_ofoo__aooieeeeeee Ewbodied_ooobtooo _tw.eAleeeeeeeehee__ ICCV_2019 2ape001111.p..__111111_1111 html.
URL https: /ttp:///oppccccccceec
[69] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https: ptttppe:/ppooceeedggg neurips. cttccc///paao wctppppr/202//////// 2ttp///222///h/////// 022107/2207111666000006666-----1 Abstraact .ttml.
[70] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, M. Turner, E. Un- dersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitat- Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https:t///// /ittap:///ahaabtatt org/ datdsetsttttaaaa
[71] A. Kendall and Y. Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Com- puter Vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL htattts:://ppeeee nips. ww/paper.pppp..ppppppp f ptt.es//ppppee/11//// hash 2aa/9966669990660066622228826 Abstrratttttttt. html.
[72] Y. Gal and Ghahramani. Dropout as a Bayesian Approximation: Representing Model Un- certainty in Deep Learning. In M. Balcan and K. Q. Weinberger, editors, Proceedings The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, June 2016. PMLR. URL html. html..
https: /ttpes////ccceeeeeeeeeee mlr, http:/ingg.mll...ess//////// /aal///ga16..///
15

---

[73] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Gameett, editors, Advances in Neural Information Processing Sys- lems, volume 30. Curran Associates, Inc., 2017. URL https: mips.. cc/paper. http:///ppp.p..oeoao/////aarttt.
Abstract html.
[74] D. Nix and A. Weigend. Estimating the mean and variance of the target probability dis- tribution. Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume I, pages 55-60 vol.1, 1994. 00111..1199/CN.11944443335 URL https://ieeexplorrs ieee org/document/S74138.
[75] V. Peretroukhin, M. Giamou, W. N. Greene, D. Rosen, J. Kelly, and N. Roy. A Smooth Repre- sentation of Belief over SO(3) for Deep Rotation Learning with Uncertainty. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. doi: 1..11.../RSSS....... 20000. XVI.007. URL https:: http:://wwwww www.woocppppcccccogggg.oooooooe
htmll.
[76] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso. Generalised dice over- lap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3, pages 240-248. Springer, 2017. URL pettps///ing springer wtt http///aaptt...... 10. 1007/
[77] Jadon. survey of loss functions for semantic segmentation. In 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pages 1-7, 2020. doi:10. (0WCCCBBCBB81111220009977711
[78] W. J. Youden. Index for rating diagnostic tests. Cancer, 3(1):32-35, 1950.
[79] J. Blumenkamp, A. Shankar, M. Bettini, J. Bird, and A. Prorok. The Cambridge RoboMaster:
An Agile Multi-Robot Research Platform, 2024.
[80] E. Rublee, Rabaud, K. Konolige, and G. Bradski. ORB: An efficient alternative SIFT SURF. In 201/ International Conference Computer Vision, pages 2564-2571, 2011. doi: 0001001109//CC22011....11
[81] Deng, W. Dong, Socher, L..J. Li, Li, and L. Fei-Fei. ImageNet: large-scale hier- archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recogni- tion, pages 248-255, 2009. 1.0. ieee. org httg//documenn/520888555 /
URL httht:////eee..p.orre.
16

---

Appendices
A
Model Training
This section details our model architecture, datasets, and training process. We describe the key components of our model, explain the creation of training and validation datasets, and outline training methodology.
A.1
Model
This section introduces all hyperparameters and implementation details the model architecture.
Our methodology utilizes the Transformer architecture [56]. Self-attention computes the output as weighted sum of the values (V), with the weights determined by the compatibility of queries (Q) and keys (K) as per the formula: Attention Q, K, V) softmax (QQK)) / /dvd,) VV, where dN is the dimensionality of the keys. Typically, Q. K. and V are set to the same input features.
D2 is a vision foundation model based on the vision transformer (ViT) and pre-trained self-supervised manner the LVD-142M dataset, which is cumulation of various filtered datasets, including various versions of ImageNet [81] and Google Landmarks.
The encoder fene consists of the pre-trained D2 vision transformer and two more transformer layers previously described as senc We use the smallest distilled version of Dino based on ViT- S/14 with 22.1 parameters due the fastest inference on our hardware. The input feature size 224 224, of which DinoV2 generates an embedding size $ 256 and F = 384. We add linear layer that transforms F' down to the parameter described in this paper (e.g. 24) while maintaining S. The output of this is then cropped to the S described in this paper (e.g. 128) by cropping the sequence length at that position.
fenc = fdino Jtruns Senc (I,) E,
(8)
(9)
The pairwise pose encoder Spoce consists of transformer frans which is assembled from five trans- former blocks, and four MLPs fp" fp. JR.
We define the concatenation operation of two tensors A and B along axis C' as (A lle B) and the indexing operation on tensor A by the first element along axis as A We introduce an instance of another transformer /pose and four MLPs fb" fp fKk f&. Furthermore,
introduce learnable transformer positional embedding for the sequence dimension S Spose. For each incoming E, V vy (w:),, we perform pose estimation
Eij = frons ((E; 11s E2) + Spose) $1
(10)
Pieej Ri.i f3' (Erj), ) SK (E.j)
(11)
2 R..j = f (E.y). fR (E.))
(12)
Spose (E E,) = (Pr.g) : R,,,, of R.
We concatenate the node features along the sequence dimension, add the positional embedding, run the transformer and then extract the first element of the sequence dimension (Eq. 10), and eventually use this edge embedding to estimate poses and orientations (Eq. 11) and uncertainties (Eq. 12). We first project the embedding E, to larger feature size F 192 using linear layer. Each transformer layer has F 192 in and out features. The learnable transformer positional embedding Spose is initialized to match F and S, with normal distribution initialization of a = 0.02. The MLPs f/ fo¹ Jk _ are represented by one linear layer mapping the output of the transformer from = 192 to F = 17 (position: 3, position uncertainty: 3, orientation: 10, orientation uncertainty: 1).
Lastly, we perform the position-aware feature aggregation, utilizing the node embeddings E, generated in fpese for each corresponding edge. The feature aggregation fune- tion fage integrates information from the set of nodes ey e ((x))) U {v.). The resulting output is
Multi-node aggregation:
17

---

feature vector F; for each node U; encapsulating the aggregated information from its local vicinity, which can be used for any downstream task in a transfer-learning context [46], requiring geometric information robotic swarm, for example, to predict a BEV of the environment around the robot that captures areas that might not necessarily visible by the robot itself.
We introduce instance of two more transformers trans and firnss MLP faggpos. learnable transformer positional embedding Sagg: as well as BEV decoder fBEV. We perform the node aggregation as
The feature aggregation function Sazg is built from Jage which consists of five transformer layers, and ppest which consists of one transformer layer, all of which have an embedding size of F = 192. The MLP Sagepes consists of three layers with 17, and 48 neurons each, the positional embedding Sags is initialized similar to the one fpuse. and the BEV decoder fBEV takes the first sequence element of Juans as input and scales it up through seven alternations of convolutions and upsampling.
X;j = (EE Ils E,) + Sagx)
(13)
Fij (Xi) F Ingepos (Eij))
(14)
F5 (Xis ||5 Sagapos (0))
(15)
F; Strass
Σ
(F (5)
(16)
Irans
BÉV; = SBEV (F:).
(17)
We concatenate the node features along the sequence dimension and add the positional embedding (Eq. 13). We then concatenate nonlinear transformation of the edge features along the feature dimension to the output of the previous operation (Eq. 14). The previous operations are performed for all v (N;),, but it is essential to include self-loops to the graph topology, as a full forward pass of the neural network would otherwise not be possible if (((x) were empty. We therefore set the edge embedding to the zero tensor O in that case (Eq. 14). We then aggregate over the set of neighbors and itself to produce the output embedding (Eq. 16) and finally use this to estimate any other downstream task, for example, estimating a BEV (Eq. 17).
All transformer blocks have 12 attention heads. The MLP used in the transformer has a hidden size of 4F. All blocks utilize a dropout of 0.2. The model has a total of 43.7 M parameters (22.1 M of which are frozen D2 parameters and 21.6 M trainable parameters).
A.2 Training Dataset
We utilize the Habitat simulator [68, 69] along with the training set of the HM3D dataset [70], which comprises 800 scenes of 3D-scanned real-world multi-floor buildings. We select Habitat and HM3D for their photorealism and the availability of ground-truth pose information, well detailed nav- igation mesh information that be converted to BEV representation of the scene. We instantiate single agent equipped with calibrated, rectified camera with a field of view of 120", which cor- responds to the camera configuration on our robots. For each scene, we randomly sample several uniform agent positions based on the total area of the scene. For each sample, the camera orien- tation is randomized across all three axes. Additionally, for each scene, we extract the processed navigation mesh, which indicates areas that are navigable by mobile robot, or not navigable due to obstacles or overhangs. We extract 3, 816, 288 images from calibrated camera with 120 field of view from 800 scenes and 1, 411 floors, along with corresponding BEV representations.
The samples of the datasets Diim DSim and DSin are randomly sampled to be within orientation range [- π I for roll and pitch, and within a height of 0.0 to 2.0 meters, or less if the ceiling is at 32: 32
a lower height.
The samples of the datasets DSim Di D Dii TesteD are randomly sampled be within orientation range [-m, π] for roll and pitch. We furthermore randomize the FOV ranging from 60 to 120" per camera.
18

---

Figure 6: To collect the real-world testset, we construct a static base station equipped with two cameras facing in opposite directions and lidar. We add a marker on the robot that its pose be tracked with the lidar and then manually move the robot around the base station.
We divide our data into training Diin (80%), testing DSim (19%), and validation D (1%) sets. From these sets, we sample pairs of Nuax = 5 images and poses by (1) uniformly sampling 25% of individual images from each dataset, and then (2) uniformly sampling Nmax images within a distance of dmax = 2 m around each sample from (1). This process yields 763, 257 training and 276, 680 test tuples, each of which are at most 2dmax apart and pointing in random directions, thus providing realistically close and distant data samples for estimating both poses and uncertainties in case of no overlaps.
The ground truth BEV BEV, is a cropped 6 m x 6 m view of the floor's BEV around generated through Habitat's navigation mesh. Each agent e is centered and facing north.
A.3 Validation Dataset
Data collection involves a base station with dual cameras and a 2D Lidar, plus distinctively shaped marker on each robot. We manually navigated robots around the base station to capture variety of relative poses, then used custom tool to label Lidar data for pose extraction.
All robots are equipped with an NVidia Jetson Orin NX 16GB and forward-facing camera, rectified to 120 FOV with an image resolution of 224 x 224 pixels. We collected the real-world test set DReaal in eight indoor scenes. It consists of 5692 images. We sample image pairs to ensure uniform distribution relative positions from m to 2 m. This results 14008 sets of samples, each with three images, totaling 84048 pose edges, 32% which have no visual overlap. Our dataset covers a diverse and challenging range of indoor scenes, as well as one outdoor scene.
We explain the setup of our data collection unit in Fig. 6. We visualize the five different scenes collected the dataset from as well as the data distribution from each respective scene in Fig. 7.
A.4 Training
We use Pytorch Lightning to train our model for 15 epochs on two NVidia A100 GPUs. This training takes approximately 24 hours. We optimize the weights using the AdamW optimizer and use the Icycle learning rate annealing schedule throughout the training and configured weight decay. We choose the best model based on the performance on the validation set.
We set the initial learning rate 1 and configure learning rate schedule to increase this over two epochs up to 1 13, followed by sinusoidal decline to 0 over 20 epochs.
19

---

We collect the real-world dataset Real from eight unique scenes, showing different chal- Test lenges, from cluttered environments over scenes with high ceilings sunny floors. The first row shows picture summary of the scene, taken from fisttperson-view perspective. The second to fourth row shows three samples from the real-world dataset, from the perspective of the robot. The fifth row shows the distribution of positions for each scene and the sixth row shows the distribution of relative angles between nodes.
Figure 7:
-1
Epoch
Figure 8: We show the training and validation loss for model trained with F 48 and $ 128 over N 5 different training seeds. We report low average standard deviation of 0.01 over all epochs on the validation loss.
We show the train and validation loss for N = 5 training runs with different seeds in Fig. 8. Training with F = 48 and S 128 across = 5 different seeds reveals a low average standard deviation of 0.011374 leading us to report results for only N = seeds in Tab.
The parameters a balances between Dice and BCE loss, and set balances between position and orientation prediction accuracy, and we set it to 1, given equal weight to both.
0.5, and the parameter
20

---

Angle diff. (deg)
180
135
45
0.0 0.5 1.0 1.5 2.0
0.0
0.5
1.0
1.5
2.0
22.5 45
67.5
22.5 45
67.5
90
Pos. Err
Pos. Var
Rot. Err
Rot. Var (deg)
Figure 9: We show qualitative distributions for position and rotation errors and variances over the relative angle between two robots on all edges of the simulation testset D Two agents facing the same direction would have angle difference of 0%, whereas two agents facing the opposite direction would have an angle difference of 180". We show the FOV as dashed line, where any results below the line indicate that the edge has some image overlap and above image overlap. The horizontal marginals show the distribution of each individual plot while the vertical marginal shows the distribution of angle differences increase over the threshold, the majority of the predictions are still useful. The rotation error is consistent across angle differences, while the uncertainty increases.
the dataset. While the position error and variance
Table 3: Ablation over different modes for the BEV prediction on the simulation testset Diest-
Experiment
Dice
loU
Median Pose Err.
None
0.628
0.495
N/A 31 cm, 5.0" 0 cm, 0.0
Predicted
0.683
0.561
Ground truth
0.743
0.632
B Experiments and Results
B.1 Metrics
We compute absolute Euclidean distance between the predicted and ground truth poses and geodesic distance between rotations as
Doss (Pr.j. Pr.es) = || Prasll
Diot (Ry,y, (R,,,,,,,,,)) = 4 arcsin dqua (Ri,es .R..),R...)))
edges and report median errors.
We perform further evaluation the distributions of pose errors and variations the simulation test set Diim in Fig. 9. The results are in line with the results performed on Test in the main manuscript.
B.2 Simulation
We provide additional quantitative experiments Fig. and Tab. visualize four additional qualitative samples from the simulation dataset DSin Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14 and Fig, 15.
BEV Experiments: We assess the effectiveness of BEV predictions, focusing on the contribution of pose predictions to enhancing BEV accuracy, in an ablation study detailed in Tab. 3.
We conduct three experiments: (/) model trained without incorporating any pose prediction into the BEV aggregator. This model can estimate local information and, to a degree, information ag- gregated through neighbors. However, it must implicitly lear to predict the relative poses between observations to successfully merge the BEV representations from multiple agents. (2) A model
21

---

2/1
1/3
1.
Prediction Ground Trith
y [m]]
y [m]
2.5
0.0
2.5
2.5
0.0
2.5
2.5 0.0
2.5
2.5
0.0
2.5
2.5
0.0
2.5
[m]
x [m]
[mm]
x [m]]
[m]
Figure 10: Sample A from Dim
Test
U0
21
2/2
V/
14
Prediction Ground Truth
[m
[m]
2.5
2.5
0.0
2.5
2.5
2.5
0.0
2.5
2.5
0.0
2.5
[m]
x
[m]
[m]
x [m]
[m]
Figure 11: Sample B from Dii
Test
trained described in this work, integrating pose estimates with BEV predictions and aggregating this information from neighbor nodes. For this experiment, use the model configuration with F = 48 and S 128. (3) A BEV model trained with ground truth pose information from the simulation training set DSim This model's performance is considered an upper limit, assuming perfect pose estimation. Evaluation on the simulation test set DSim Test indicates our model performs between the two baselines. significantly outperforms the pose-free free-fre model with an 8. 8.75% improve- ment in accuracy, while the model using ground truth poses shows an 18.31% performance increase compared to the baseline without pose information. These results highlight the substantial benefit of incorporating pose predictions into BEV estimation, positioning approach notably effective middle ground between the two extremes.
22

---

2/
1/3
1.
Prediction Ground Trith
& [m]]
y [m]
2.5 0.0
2.5
2.5
0.0
2.5
2.55 0.0
2.5
225
0.0
2.5
2.5
0.0
2.5
x [m]]
x [m]
[mm]
x [m]
[m]
Figure 12:
Sample C from Sim
Test
2/
1/3
14
Prediction Ground Truth y [m]]
K [m]
-2.5 0.0
2.5
2.5
0.0
2.5
225
0.0
2.5
2.5
0.0
2.5
-2.
0.0
2.5
x [m]
x
[m]
x [m]
x [m]
x [m]
Figure 13: Sample D from DSim
6D Dataset: We train an additional set of seven models with differing S and F' on the dataset DSSim Train6D- Notably, this dataset contains fully randomized 6D camera poses (opposed to roll and pitch constrained to т T/32 in DSi.... We furthermore randomize the camera's FOV. We show the results in Tab. While the simulated results are not comparable to the one reported in Tab. as the dataset is different, the real-world dataset identical, and the reported performance is approximately identical with slight degradation. The rotation error Dim TestoD is higher there is a larger number of cameras without any direct visual overlap, and the maximum possible pose error increases.
23

---

1/1
1/3
1.
Prediction Ground Trith
y [m]]
y [m]
2.5
0.0
2.5
2.5
0.0
2.5
2.5 0.0
2.5
2.5
0.0
2.5
2.5
0.0
2.5
[m]
x [m]
[mm]
x [m]]
[m]
Figure 14:
Sample E from D
Test
V0
V1
V2
V/
04
1
Prediction Ground Truth
[m])
[m]
-2.5
0.0
2.5
225
0.0
2.5
225 0.0
2.5
-2.
0.0
2.5
2.5
0.0
2.5
[m]
[m]
[m]]
[m]
[m]
Figure 15: Sample F from Dim
Test
B.3 Real-World
We provide additional quantitative experiments in Fig. 16, Tab. 5 and Tab. 6. We visualize four additional qualitative samples from the real-world dataset Dieaal in Fig. 17 and Fig. 18.
We investigate the efficiency the uncertainty estimation as well as the distributions of pose errors more closely. Fig. 16 illustrates the distributions position and rotation errors, along with the predicted variances over the angle difference between robot pairs in the dataset D angle difference of 0 signifies robots facing the same direction, potentially resulting in significant image overlap, while angle difference of 180" indicates opposite directions, with no overlap. The Field Of View (FOV) overlap threshold is highlighted in red, delineating errors with overlap (below the
24

---

Table 4: Ablation study over the number of patches S and size of features F per patch for models trained on the dataset D error for poses on the dataset Drim
Trainen- We report the BEV representation performance and the median
TesttD and D
Model
D Sim
Dest
S
Dice
loU
All
Invis. Filt.
Invisible
Visible
256
48
65.1
53.8
39 cm
46.5
75 cm
9.3
112 cm
16.0
44 cm
7.1
128
96
64.8
53.3
42 cm
46.6
100 cm
13.3
119 cm
16,8
41 cm
6.8
128
48
64.9
53.4
43 cm
46.6
88 cm
13.0
113
14.9
41
6.5
128
24
66.8
54.1
48 cm
47.2
107 em
23.7
120 em
23.3º
cm 7.0%
64
48
64.3
52.7
52 cm
47.9º
138 cm
16.0
126 cm
16.2
cm
7.1"
32
24
66.0
53.5
59 cm
48.8"
166 cm
15.4
127 cm
19.8
55 cm
7.50
48
58.3
46.4
106 cm
58.5
153 cm
101.6
132 cm
66.4
102 cm
24.3º
Angle diff. (dog)
180
90
0.0 0.5 1.0 1.5
2.0
0.0
0.5
1.0
1.5
2.0
22.3 67.5
90
Pos. Err
Pos. Var
Rot. Err
Rot. Var (deg)
Figure 16: We show qualitative distributions for position and rotation errors and variances over the relative angle between robots on all edges of the real-world testset DReal containing 84048 unique edges. Two agents facing the same direction would have an angle difference of 0%, whereas two agents facing the opposite direction would have an angle difference of 180%. We show the FOV as dashed line, where any results below the line indicate that the edge has some image overlap and above no image overlap. The horizontal marginals show the distribution of each individual plot while the vertical marginal shows the distribution of angle differences over the testset. Note that the angle distribution is not uniform due the real-world dataset collection where least two cameras always face the opposite direction.
line) from those without (above the line). The analysis reveals that position errors remain consistent up to the FOV threshold, beyond which they escalate, as reflected in the predicted position variance. Conversely, rotation eтrоrs stay uniform across angle differences, with only marginal increase for higher angles, though the predicted rotation variance notably rises.
In Tab. 5, investigate the performance of our model across the different scenes in the testing dataset DReal We provide detailed breakdown of the median pose error for Invisible, Invisible Filtered and Visible cases for each of the eight scenes. While the pose error for visible samples is consistently between 22 cm/5.2 and 48 cm/7.5". there are more outliers for invisible samples. This due to some scenes having more or less symmetry, thus making it more challenging for the model to accurately predict poses. The pose prediction for the outdoor scene are least accurate, which is to be expected since the model was trained on indoor scenes. The Sunny scene has severe shadows and direct sunlight, while still providing low pose errors. The pose predictions are useful in all scenarios, demonstrating impressive generalization ability across wide range of scenes.
Next, we evaluate the magnitude of pose prediction errors over different thresholds of distances between image pairs in Tab. 6. We consider five bands of distances separated in 0.4 m chunks, ranging up to 2m. Note that the worst-case pose error predictions. While the error increases with an increasing distance between samples, the rotational error in particular stays low, and the position errors stay useful, ranging from 12 em 65 em for visible samples, 34 em to 134 em for invisible filtered samples (samples that not visible but for which he model predicts high confidence) and 76 em to 153 em for all invisible samples.
d due to the direedimmessioal pose
25

---

Table 5: The median pose estimation error of our model on each of the eight Real
Test
scenes.
Scenario
Invisible
Invisible Filtered
Visible
Corridor A
138
129.6'
90
11.4
22
5.2
Corridor B
112 cm
5,9
261 em
8,2
cm
5.2
Office A
86 cm
14.4
38 cm
9.6
33 cm
7.5e
Office B
146 cm
10.4
248 cm
4.6"
20 cm
4.7
Outdoor
134 cm
9.5
193 cm
28.7
49 cm
6.9
Study A
132
6.7
40
5.1
28
5.3
Study
115 cm
8.9
56 cm
9.6
cm
5.5
Sunny
77 cm
9.3"
49 cm
8,7"
22 cm
6.0"
Table 6: We show median pose prediction errors for different distance thresholds on the dataset Diteal
d
Invisible
Invisible Filtered
Visible
<0.4
76 cm
7.6
34 cm
6.7
12 cm
4,9
<0.8
80
8.8
48 cm
8.6
21
5.2
<1.2
cm
9.5
56 cm
8,5
cm
5.8
<1.6 m
123 cm
10.6
97 cm
12.9
45 cm
6,0"
<2.0 m
153 cm
10.5
134 cm
16.2
65 cm
6,8
B.4 Geal-World Pose Control
Ad-hoc WiFi: Image embeddings are sent between nodes at rate of 15hz, with each being just over 6KiB. The fully distributed nature of our system matches well with a broadcast communications topology, which theoretically can permit operation at scale. Broadcast messaging has significant drawback however, in that modern wireless data networking standards (IEEE 802.11/WiFD fallback very low bit rates to ensure maximum reception probability.
802.11 uses low bit rates for broadcast due to a lack of acknowledgment messages, which are the typical feedback mechanism for data rate control and retransmissions, The probability of a frame being lost due to the underlying CSMA/CA frame scheduler non-trivial (3%+); a condition which worsens with increasing participating network nodes.
A higher level system, such ROS2 communications middleware package CycloneDDS man- age retransmissions, however currently available systems are not well suited to the requirements of a decentralized robotic network. In particular they lack the ability to detect network overload con- ditions, which can result in retransmissions that further load the network; finally resulting in a rapid increase in packet loss rates which renders the network as a whole unusable. While it is possible tune such communications packages manually, the best parameter values are sensitive to highly variable deployment configuration parameters including ambient wireless network traffic. Manual tuning of network hardware, such forcing increased broadeast bit rates, alleviate this problem but are generally not resilient as scale.
Our custom messaging protocol is targeted to this specific use case, and includes the ability to dynamically backoff messaging load based upon detected network conditions. Our system is also uses shared slot TDMA message scheduling system which works with the 802.1 frame scheduler maximize frame reception probability without compromising overall data rates.
Results: We benchmark the runtime performance of the model on the Jetson Orin NX in Tab. by averaging 100 sequential forward passes. We compare 16-bit and 32-bit float evaluated on the GPU or CPU and compilation with TensorRT, a proprietary NVidia tool for ahead-of-time compilation of neural networks. The model fenc. generating the communicated embedding E,, has total of 30 M parameters, and takes ca. 20 for one forward pass FP16 optimized with Torch TensorRT on the GPU, This is 50x faster than running the same model on the CPU. We also note that changing from 32-bit floats to 16-bit floats results in 2.75x speedup. The model frose processing pair of
26

---

Up
14
4
Ground Tmth
Ground Tmih
2
Prediction
Prediction
2.5-2.5 2.5-2.5
Figure 17: We visualize a sample from our custom real-world testset DReal for three nodes, similar to the simulation sample in Fig. 2. We do not show the ground-truth BEV segmentation since we do not have labels for this. Additionally to the pose predictions Ps.ij. RR,... we also visualize the predicted uncertainties op.00 and o..
Grond Truth
Ground Truth
2
2
Prediction
Prediction
2
2.5
0.0
2.5-2.5
0.0
2.5
2.5
0.0
2.5
2.5
0.0
2.5
2.5
0.0
2.5-2.5
0.0
2.5
Figure 18: Sample and C from Deal Test
embeddings into relative pose, has total of 6 M parameters and takes ca. ms for one forward pass. The model is in custom C++ environment integrated with ROS2 Humble and ad-hoc WiFi network for inter-node communication to communicate image embeddings E,, achieving 15 Hz processing rate for the whole pipeline, including image pre-processing, model evaluation, communication and pose prediction.
Fig. 20 and Tab. contain additional quantitative and qualitative evaluation for the multi-robot pose- tracking experiment, consisting of additional reference trajectories. Figure 8 static is similar to the Figure 8 dynamic trajectory elaborated in the main paper, but all robots face the same direction, resulting in two shifted Figure 8s for the two follower robots. Rectangle dynamic is also similar to
27

---

0s
1s
2x
3s
4s
5s
Figure 19: We show additional snapshots of real-world deployments in different scenes with up four robots. Each row contains six frames from a video recording, each spaced s in time.
Table Model runtime evaluation for 128 and F 24.
Type
Mode
Dev.
few
foeee
FP16 TRT
GPU
19.22
+0.92
4.15
+0.49
FP16
GPU
36.79 ms
+1.33 ms
8.04 ms
+0.73 ms
FP32
JIT
GPU
99.32 ms
+1.56 ms
13.57 ms
+1.10 ms
FP32
JIT
CPU
952.04 ms
111..72 ms
248.41 ms
1:4598 ms
the Figure 8 dynamic trajectory elaborated in the main paper, but instead of a Figure 8, the reference rounded rectangle, with robots moving with up to 0.8 m/s.
Tab. reports quantitative evaluations for the same results, which are in line with the results reported in Tab. Note that the velocities reported are average velocities of the follower robots. The veloc- ities are consistent for both Figure trajectories, but differ for the rectangle trajectory, as the inner robot moves slowly than the outer robot.
B.5 Single-Robot Trajectory Recording
We perform a single-robot homing/trajectory replay experiment further show the versatility of Nee. In this experiment, treat list of recorded keyframes as reference embedding timate the relative pose to given the currently observed image. If either the distance the predicted
28

---

Figure (1)
0.6
Rot Std. dev. (dgg]
Pos Std. dew. [m]]
Flax.
Pos Eir [aa]
Rot Eer (deg)
Plaar.
1.0
0.4
80
Leadee
20
0.0
0.0
Time
Time
Time
Time [c]
Roctangle dynaemic (2)
Rot Stå. dev. [dog]
Poe Std. dev. [n]
Pos Ear [aa]
Rot Erz [dea]
Flwa.
1.00
0.4
60
Lender
20
Time
Time
Time
Figure 20: We evaluate the tracking performance of model and uncertainty-awarr controller on two additional reference trajectories, with two follower robots (in blue and orange), positioned left and right of the leader robot (in green). For each trajectory, we report the tracking performance over time for position and rotation, as well as the predicted uncertainties. The top trajectory (1) is Figure Eight with the leader always facing in the same direction. The bottom trajectory (2) is rounded rectangle. We show the trajectory for 120 and the tracking error for the first 60 s. The tracking performance is consistent across multiple runs.
Table 8: We report the and absolute tracking error for both leaders for all three trajectories, well as average velocities. The errors are consistent with the results reported in Tab.
Trajectory
Robot
Mean Abs.
Median
Figure dynamic
A
38 cm, 5.6"
38 cm, 4.8"
0.59 m/s
28 cm, 5.1
25 em, 4.7"
0.58 m/s
Figure static
A
51 em, 5.2"
48 cm, 5.3"
0.60 m/s
28 cm, 5.6
26 cm, 5.4%
0.61 m/s
Rectangle dynanic
em. 4.4
42 cm, 4.4"
0.32 m/s
29 cm, 5.1%
27 em, 5.1
0.81 m/s
uncertainty increases by some predetermined threshold, append new keyframe to the list. After teaching the robot the reference trajectory, we command it into keyframe- mode, where the current reference keyframe is set based on the recorded list. We provide qualitative experiment on website.
29


Reviews:
[{"review_id": "2NqjSCDA7A", "reviewer": "Reviewer_3vko", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a method for relative pose estimation with uncertainty quantification using monocular visual images. The method is built upon DinoV2 for image encoding, where DinoV2 can address relative pose estimation without overlapping views. A pairwise pose encoder is then used to estimate the relative poses, and multi-node aggregation is utilized to construct a BEV representation. In addition, the uncertainty in the relative pose estimation is quantified using the Gaussian NLL loss. The lightweight model design and GPU acceleration allow the method to run onboard in real-time.", "summary_of_recommendation": "The theoretical novelty of the paper is limited, and several key aspects of the approach, such as addressing non-overlapping views and scale issues, are not well explained or justified. The efforts to achieve real-time performance are commendable.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please see the review section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XE52zwvNKm", "reviewer": "Reviewer_vrDg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for estimating relative poses between robots and \"birds eye view\" map representations in the setting of multi-robot navigation. The model is based on a neural network that uses pre-trained foundation model features (DINO) to predict relative poses and birds-eye view representations. The model is tested in real multi-robot experiments.", "summary_of_recommendation": "I am recommending \"weak accept\" on the basis that the method itself seems interesting and the technical evaluation seems reasonable to me, although some aspects of the presentation could be clarified.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "See comments above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJqq31CbDG", "reviewer": "Reviewer_bCfq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes CoViS-Net, a decentralized, platform-agnostic visual spatial foundation model designed for multi-robot applications in unstructured environments. CoViS-Net leverages spatial priors learned from data to provide real-time pose estimation and bird’s-eye-view (BEV) representations, even without camera overlap between robots. The model operates on onboard compute without requiring existing networking infrastructure.", "summary_of_recommendation": "This work demonstrates several strengths, including an effective ablation study that underscores the importance of key components, strong real-world validation with comprehensive experimental analysis, and the incorporation of uncertainty estimation. The approach is robot/platform-agnostic, does not rely on external networking infrastructure, and supports real-time processing capabilities.     However, there are some areas for improvement. The literature review should be enhanced by including more papers on multi-robot search and exploration, with a particular focus on map prediction methods and scalability analysis. Additionally, comparison metrics should be made more consistent or clearly justified, and an ablation study should be conducted comparing scenarios with and without BEV generation to highlight its benefits.  I recommend weak accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- The literature review needs improvement. Can the authors discuss related work on map prediction methods and how BEV representations relate to these approaches? Additionally, include relevant studies on multi-robot exploration and search. \n- The use of different metrics for ablation and comparison methods is confusing. Can the authors streamline the metrics and use consistent measures such as Euclidean distance and geodesic distance for evaluating pose estimation and BEV representation?\\ \n- Can the authors include an ablation study comparing results with and without BEV generation to highlight its benefits? \n- Please include limitations of your approach.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Ms3ZZBsUH", "reviewer": "Reviewer_mxQi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a transformer-based architecture to enable accurate real-time pose estimation and spatial understanding in multi-robot systems, validated through comprehensive simulations and real-world experiments.", "summary_of_recommendation": "The specialized nature, limited training scope, and lack of demonstrated versatility across tasks suggest that CoViS-Net is better classified as a highly specialized model rather than a general-purpose foundation model. This distinction is important for setting appropriate expectations and understanding the potential and limitations of the proposed approach.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Could the authors provide a clear definition of what they consider a \"foundation model\" and how CoViS-Net fits into this category?\n\nHow does CoViS-Net compare to well-known foundation models in terms of architecture, training scope, and adaptability?\n\nWhat is the extent of diversity in the training data used for CoViS-Net? How do the authors ensure that the model generalizes well beyond the specific scenarios presented in the paper?\n\nWhat is the parameter count of CoViS-Net, and how does it compare to traditional foundation models? Can the authors justify the scale of the model in the context of foundation models?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strength: 

- Reviewers praised the paper's focus on real-time performance and the demonstration on real robots. (Reviewer 3vko, Reviewer vrDg, Reviewer bCfq)
- Comprehensive evaluation: The paper includes both simulation and real-world experiments, uses multiple metrics, and compares against baseline methods, strengthening the results' validity. (Reviewer mxQi)
- Novel use of BEV representations: Employing BEV representations for improving relative pose estimation was deemed relatively original. (Reviewer bCfq)

Weaknesses:

- Limited theoretical novelty: Several reviewers found the paper's reliance on existing techniques
- Unclear justification for “foundation model" claim: The paper's characterization of its model as a "foundation model" drew criticism. Reviewers questioned the training data scope, specialized architecture, and lack of comparison with established foundation models. (Reviewer vrDg, Reviewer mxQi)

After rebuttal, the majority of the reviews lean towards accepting this paper. This work has strong real world evaluation and demonstration, while needs more polishing in presentation and positioning. I would recommend accepting this paper and encourage the authors to further improve the paper according to the suggestions.
Decision: Accept