CoViS-Net: A Cooperative Visual Spatial Foundation Model for Multi-Robot Applications

Authors: jan  blumenkamp
steven  morad
jennifer  gielis
amanda  prorok
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-04 10:12:18

Abstract: Autonomous robot operation in unstructured environments is often underpinned by spatial understanding through vision. Systems composed of multiple concurrently operating robots additionally require access to frequent, accurate and reliable pose estimates. Classical vision-based methods to regress relative pose are commonly computationally expensive (precluding real-time applications), and often lack data-derived priors for resolving ambiguities. In this work, we propose CoViS-Net, a cooperative, multi-robot visual spatial foundation model that learns spatial priors from data, enabling pose estimation as well as general spatial comprehension. Our model is fully decentralized, platform-agnostic, executable in real-time using onboard compute, and does not require existing networking infrastructure. CoViS-Net provides relative pose estimates and a local bird's-eye-view (BEV) representation, even without camera overlap between robots, and can predict BEV representations of unseen regions. We demonstrate its use in a multi-robot formation control task across various real-world settings. We provide supplementary material online and will open source our trained model in due course.
https://sites.google.com/view/covis-net

Main paper: 
 introduction: 
 Spatial understanding is a cornerstone of robotic operation in unstructured environments, relying on effective pose estimation and environmental perception [1, 2, 3, 4, 5, 6, 7, 8] . While GNSS, LiDAR, and UWB sensors have been instrumental in advancing robotics, they are limited by constraints such as indoor operation, unreliability when operating around reflective surfaces or in bright environments. In contrast, color cameras offer a low-cost, energy-efficient, and rich data source, aligned with the vision-centric design of real-world, human-designed environments. Classical vision-based techniques [1, 6, 5] struggle with the ill-posed nature of image processing, cannot leverage semantic priors to resolve ambiguous situations, and may converge to a solution only after multiple time steps. These challenges are exacerbated in multi-robot scenarios that require fast and accurate relative pose EQUATIONWe conducted experiments across various model variants trained with different parameters, reporting on our model's efficiency. We present quantitative results in Tab. 1 and qualitative findings in Fig. 2 .Real-World Dataset: Our setup includes four Cambridge RoboMasters [79] and one Unitree Go1 Quadruped, each equipped with local compute, a monocular camera and WiFi dongles for adhoc networking. Our custom real-world test dataset D Real Test covers a diverse and challenging range of indoor and outdoor scenes. We provide more details in Appendix A.3.Tab. 1 details the performance across seven different experiments. We evaluated the impact of image embedding sizes E i on the performance. The IoU and Dice metrics for the BEV representation, alongside median pose estimates for all samples in the dataset, are provided. Experiments show that the sequence length S has a more significant influence on BEV representation and pose estimate performance than the feature size F . We set S to 1 for two experiments: (1) increasing F to match the size of the embeddings for other experiments, and (2) decreasing F , effectively reducing the transformer to a CNN-only architecture by removing the attention mechanism. Both experiments result in reduced performance, highlighting the importance of attention in our model. The best-performing model with S = 256 achieves an IoU of 0.571, a Dice score of 0.691, and median pose accuracy of 36 cm and 8.3 • , which is roughly equal to one robot body length. In contrast, models with S = 1 show significant degradation, with nearly unusable BEV representation predictions and pose errors on the scale of meters. Models with F = 24 and S = 64 performed slightly worse than those with larger F or S. We provide additional results in Appendix B.2. Evaluation on the real-world dataset D Real Test focuses on position and rotation accuracy, excluding quantitative BEV performance due to unavailable ground truth data. The results align with D Sim Test , showing a correlation between the number of patches S and features F and pose estimation performance. The rotation estimates do not significantly differ between visible and invisible samples, demonstrating robust pose estimation without requiring visual overlap. The most accurate model achieves a median localization error of 97 cm and 7.9 • for invisible samples, and 33 cm and 5.8 • for visible samples.Test samples are at most 2 m apart, resulting in a worst-case error of up to 4 m. These findings demonstrate our model's capability to reliably estimate poses, and accurately predict higher uncertainties under adverse conditions. We attribute this performance to our pose prediction model, as well as the robust spatial representation capabilities of the DinoV2 encoder. The model goes beyond mere feature matching, achieving sophisticated scene understanding that allows it to "imagine" unseen portions of the scene based on learned data priors, thus facilitating pose estimation through environment inference [36, 37] . We provide additional results in Appendix B.3.Finally, we conduct a baseline comparison with two feature-matching-based approaches, as summarized in Table 2 . Both baseline methods extract and match features across two images to estimate the essential matrix, deriving rotation and scale-less translation vectors. The first baseline uses OpenCV for ORB feature extraction followed by brute-force matching. The second baseline, LightGlue, uses a neural network-based approach for feature extraction and matching. Given both baselines estimate scale-less transformations, we adopt the same metric used in LightGlue [42] , computing the AUC at thresholds of 20, 45, and 90 degrees based on the maximum error in rotation and translation. Our findings indicate that our model surpasses both baselines in performance and speed across image pairs with and without visual overlap. Notably, while both baselines require visual overlap to function effectively, our approach does not, highlighting its robustness and applicability in a broader range of scenarios. 
problem formulation: 
 Given a multi-robot system, our goals are: (i) for each robot to predict its pose and uncertainty relative to other robots as well as corresponding embeddings using non-line-of-sight visual correspondences, and (ii) to use these embeddings for downstream tasks like local occupancy grid prediction. Consider a multi-robot system represented by a set of nodes V. Each node v i ∈ V has a position p w,i and an orientation R w,i in the world coordinate frame F w . The set of edges E ⊆ V × V defines the communication topology and thus the graph G = (V, E). For each edge (v i , v j ) ∈ E, the relative position of v j in the coordinate frame of v i (denoted as F i ) is p i,ij = R -1 w,i • (p w,j -p w,i ), and the relative rotation is R i,ij = R -1 w,i • R w,j . Each node v i is equipped with a camera C i , providing an image I i . From the images, we estimate relative poses and uncertainties (p i,ij , σ 2 p,ij , Ri,ij , σ 2 R,ij ) ∀ v j ∈ N (v i ) with respect to v i in its coordinate frame F i . This approach does not assume overlap between images or line-of-sight visibility between robots, relying exclusively on the relationship between images, thus making it platform-agnostic. Using these estimated poses, the robots cooperatively generate a BEV representation BEV i around each node v i , which can be used for various downstream multi-robot applications. 
approach: 
 In this section, we detail our architecture and training approach. We explain dataset generation, define our model architecture, and introduce the uncertainty estimation strategy and training protocol.We employ four primary models; The encoder f enc generates a common embedding of images, f pose estimates pairwise poses, f agg aggregates node features, and f BEV predicts a two-dimensional BEVf enc f enc f enc f pose pi,ij , σ 2 p,ij , Ri,ij , σ 2 R,ij f pose pi,ik , σ 2 p,ik , Ri,ik , σ 2 R,ik || F || F || F f agg f BEV BEV i 0 I i I k I j E k E j E i E ik E ij F ik F ij F ii F iFigure 3 : Overview of the model architecture: We illustrate the decentralized evaluation with respect to ego-robot v i , which receives embeddings E j , E k generated by the encoder f enc from the neighbors v j , v k ∈ N (v i ). For each received embedding, we employ f pose to compute the pose. We concatenate the pose embedding E ij with the image embedding E j (and E ik with E k ) and subsequently aggregate this into a node feature F i . Model outputs are framed with a dashed line.representation. An overview of the model architecture is provided in Sec. 4.1. Our model uses transformers [56, 67] in f enc , f pose and f agg . We provide more details in Appendix A.1.Image Encoder: We use the smallest distilled version of DinoV2 [59] (ViT-S/14) as image encoder due to its robust feature representation and spatial understanding. Each robot v i processes the image I i through f enc to generate the communicated node embedding E i ∈ R S×F , where S represents the sequence length, and F the feature vector size. We freeze the weights of DinoV2, enabling the use of various open-source downstream fine-tuned image detectors without an additional forward pass.Pairwise pose encoder: Each robot v i ) broadcasts the image encoding E i to neighboring robots.Given encodings E i and E j , f pose estimates edge embeddings E ij and relative poses with uncertainties (p i,ij , σ 2 p,ij , Ri,ij , σ 2 R,ij ). For each incoming E j ∀ v j ∈ N (v i ), we concatenate the node features along the sequence dimension, add the positional embedding, run the transformer and then extract the first element of the sequence dimension to obtain the edge embedding. We then use this edge embedding to estimate position, rotation and uncertainties, resulting inf pose (E i , E j ) = (p i,ij , σ 2 p,ij , Ri,ij , σ 2 R,ij). Multi-Node Aggregation: We perform position-aware feature aggregation using node embeddings E j from f pose . The function f agg integrates information from all nodes v j ∈ N (v i ) ∪ v i , producing a feature vector F i for each node v i , encapsulating the aggregated information from its local vicinity. We concatenate node features along the sequence dimension and add the positional embedding. We then concatenate a nonlinear transformation of the edge features along the feature dimension. This operation is performed for all v j ∈ N (v i ), including self-loops to handle cases where N (v i ) is empty by zeroing the edge embedding. We aggregate over the neighbors and the ego-robot to produce the output embedding F i from all edge embeddings F ij , which is used for downstream tasks, such as estimating a BEV representation BEV i = f BEV (F i ).For training our models, we use supervised learning. This section describes the dataset, loss functions, and uncertainty estimation.We use the Habitat simulator [68, 69] and the HM3D dataset [70] , consisting of 800 scenes of 3D-scanned multi-floor buildings. The data is divided into training D Sim Train (80%), testing D Sim Test (19%), and validation D Sim Val (1%) sets. We provide more details in Appendix A.2. Uncertainty Estimation: Two types of uncertainties can be modeled: epistemic uncertainty, which captures uncertainty in the model and can be reduced with more data, and aleatoric uncertainty, which captures noise inherent to specific observations [71] . For pose estimation and pose control in robotic systems, aleatoric uncertainty is more important since downstream applications can act on it to reduce it. Traditional methods like Monte Carlo Dropout [72] and Model Ensembles [73] model epistemic uncertainty. Instead, we propose using the Gaussian Negative Log Likelihood (GNLL) Loss [74, 71] , a generalization of the mean squared error (MSE) loss, allowing the model to predict both a mean μ and a variance σ2 , learned from data points µ. The loss is defined asEQUATIONThe GNLL loss penalizes high variance predictions and scales the MSE loss by the predicted variance, penalizing errors more when predicted uncertainty is high.Estimating Rotations: Estimating rotation is difficult due to discontinuities in many standard orientation representations. Quaternions are often used for their numerical efficiency and lack of singularities. Related work [75] represents rotations with a symmetric matrix defining a Bingham distribution over unit quaternions to predict epistemic uncertainty. We combine the chordal loss and GNLL to predict aleatoric uncertainty for rotations, as shown in the following equation:EQUATIONEQUATIONTraining: We use a combination of the Dice loss [76] L Dice and Binary Cross Entropy (BCE) Loss L BEC weighted by the parameter 0 ≤ α ≤ 1. This is commonly referred to as combo loss [77] and is typically utilized for slight class imbalance. We define the total loss asEQUATIONEQUATIONEQUATIONThe parameter 0 ≤ β ≤ 1 balances between the position and orientation loss. Uncertainty terms are necessary for training, as many dataset samples lack overlap and may be in adjacent rooms, leading to significant errors from difficult samples that could hinder learning. We train on simulated data D Sim Train and provide further details in Appendix A.4. 
experiments and results: 
 We demonstrate the performance of our trained models on the test set D Sim Test and custom real-world test sets D Real Test . First, we outline the metrics used. Then, we introduce the real-world dataset. Next, we analyze model performance on both simulation and real-world data. Finally, we compile and deploy our model in a real-world, real-time trajectory tracking task.We compute the Euclidean distance between the predicted and ground truth poses as D pos (p i,ij , pi,ij ) and the geodesic distance between rotations as D rot (R i,ij , Ri,ij ) for all edges and report median errors. We categorize the median pose error into four categories. All includes all edges. Visible and Invisible edges are determined by FOV overlap (an edge is invisible if D rot (R i,ij , 0) > FOV). Invisible Filtered samples use the predicted position uncertainty σ 2 p,ij to reject high-uncertainty pose estimations based on the Youden's index [78] . For simulation results, we also report the Dice and IoU metric. We provide more details in Appendix B.1. .In this section, we describe our experiment setup (including model deployment, wireless communication, and controllers), demonstrating real-world multi-robot control tasks.We deploy a model with F = 24 and S = 128. After training in PyTorch, we compile f enc , f pose , and f agg for decentralized deployment. Using TensorRT, we achieve sub-30 ms processing times. We provide more details and a runtime analysis in Appendix B.4.Ad-hoc WiFi: Image embeddings are sent between robots at 15 Hz, each over 6 KiB. The system's distributed nature suits broadcast communication, allowing scalable operation. However, IEEE 802.11/WiFi's fallback to low bit rates for broadcast increases frame loss probability, worsening with more robots. Higher-level systems like ROS2's CycloneDDS manage retransmissions but can't detect network overloads, leading to increased packet loss. We design a custom protocol that dynamically adjusts messaging load based on network conditions and uses TDMA scheduling to maximize frame reception without compromising data rates. We provide more details in Appendix B.4.We conduct qualitative experiments in various environments, including outdoors, and quantitative experiments. Two follower robots maintain a fixed orientation and distance from a leader robot on a predefined trajectory. A PD controller manages relative pose control, adjusting based on predicted uncertainty and selectively deactivating if uncertainty thresholds are exceeded. This ensures that even with unreliable estimates, robots maintain orientation toward the leader until reliable estimates are available. We assess tracking accuracy by comparing ground truth and estimated poses, focusing on position and rotation errors and predicted uncertainties.Results: Fig. 5 shows snapshots from four different real-world deployments, and Fig. 4 shows the trajectories of all robots for all quantitative experiments, as well as a visualization of the position and rotation error over time. We provide additional results in Appendix B.4. The qualitative experiments show a robust system that generalizes to a wide range of scenarios, including outdoors, albeit with a scale prediction that is less reliable outdoors. Our model was trained exclusively on indoor data and generalizes to outdoor data due to the wide range of training data used on the DinoV2 encoder. We furthermore show platform agnosticism by deploying our model to a heterogeneous team of robots, lead by a quadruped (second row in Fig. 5 ).In quantitative experiments, the tracking error oscillates throughout the trajectories as the followers reactively adjust their positions to the leader's movements. Follower A starts facing the opposite direction of the leader, resulting in high position variance and low rotation variance, which decreases as the robot aligns with the leader. An erroneous rotation prediction is accompanied by high uncertainty. As the leader moves, one follower is inside and the other outside the trajectory, switching roles depending on the section of the figure. We report a median tracking error of 48 cm and 5.3°for Follower A, and 26 cm and 5.4°for Follower B (corresponding roughly to one robot body length). 
limitations: 
 Foundation models are commonly trained unsupervised [55] , whereas our model is trained on the HM3D indoor dataset in a supervised manner. Our model requires peer-to-peer communication to be available to merge information from other robots and GPU-accelerated onboard computing for real-time deployment. Previous work showed the scalability of GNNs due to the principle of locality (number of neighboring nodes are constrained by the communication range) [29] . The number of real robots is currently constrained by our custom networking stack. 
conclusion: 
 We introduced a first-of-its-kind cooperative visual spatial foundation model that leverages existing large vision models and demonstrated its use in multi-robot control. From monocular camera images only, our model is able to output accurate relative pose estimates and BEV representations by aggregating information across neighboring robots. We validated our model on a custom realworld test set and demonstrated real-time multi-robot control. Our model accurately predicts poses, even without pixel-level correspondences, and correctly predicts BEV representations for obscured regions. In further real-world tests, we demonstrated accurate trajectory tracking, paving the way for more complex vision-only robot tasks. Our model estimates relative poses with a median error of 33 cm and 5.8°for visible edges, and 97 cm and 5.9°for invisible edges, roughly equivalent to one body length of our robot. In future work, we plan to include temporal data, apply this model to other downstream tasks and combine it with learning-based multi-agent control policies. 
 
 Appendix: This section details our model architecture, datasets, and training process. We describe the key components of our model, explain the creation of our training and validation datasets, and outline our training methodology.This section introduces all hyperparameters and implementation details of the model architecture.Our methodology utilizes the Transformer architecture [56] . Self-attention computes the output as a weighted sum of the values (V), with the weights determined by the compatibility of queries (Q) and keys (K) as per the formula:Attention(Q, K, V) = softmax((QK T )/ √ d k )V, where d k is the dimensionality of the keys. Typically, Q, K, and V are set to the same input features.DinoV2 is a vision foundation model based on the vision transformer (ViT) and pre-trained in a self-supervised manner on the LVD-142M dataset, which is a cumulation of various filtered datasets, including various versions of ImageNet [81] and Google Landmarks.The encoder f enc consists of the pre-trained DinoV2 vision transformer and two more transformer layers previously described as f enc trans . We use the smallest distilled version of DinoV2 based on ViT-S/14 with 22.1 M parameters due to the fastest inference on our hardware. The input feature size is 224×224, of which DinoV2 generates an embedding of size S = 256 and F = 384. We add a linear layer that transforms F down to the parameter described in this paper (e.g. 24) while maintaining S. The output of this is then cropped to the S described in this paper (e.g. 128) by cropping the sequence length at that position.f enc = f dino • f enc trans (8) f enc (I i ) = E i (9)The pairwise pose encoder f pose consists of transformer f pose trans , which is assembled from five transformer blocks, and four MLPs f µ p , f σ p , f µ R , f σ R . We define the concatenation operation of two tensors A and B along axis C as (A || C B) and the indexing operation on tensor A by the first element along axis B as A B1 . We introduce an instance of another transformer f pose trans , and four MLPs f µ p , f σ p , f µ R , f σ R . Furthermore, we introduce a learnable transformer positional embedding for the sequence dimension S pose . For each incoming E j ∀ v j ∈ N (v i ), we perform a pose estimation asEQUATIONEQUATIONf pose (E i , E j ) = (p i,ij , Ri,ij , σ 2 p,ij , σ 2 R,ij). We concatenate the node features along the sequence dimension, add the positional embedding, run the transformer and then extract the first element of the sequence dimension (Eq. 10), and eventually use this edge embedding to estimate poses and orientations (Eq. 11) and uncertainties (Eq. 12). We first project the embedding E i to a larger feature size F = 192 using a linear layer. Each transformer layer has F = 192 in and out features. The learnable transformer positional embedding S pose is initialized to match F and S, with normal distribution initialization of σ = 0.02. The MLPs Multi-node aggregation: Lastly, we perform the position-aware feature aggregation, utilizing the node embeddings E j generated in f pose for each corresponding edge. The feature aggregation function f agg integrates information from the set of nodes v j ∈ N (v i ) ∪ {v i }. The resulting output is a feature vector F i for each node v i , encapsulating the aggregated information from its local vicinity, which can be used for any downstream task in a transfer-learning context [46] , requiring geometric information in a robotic swarm, for example, to predict a BEV of the environment around the robot that captures areas that might not necessarily be visible by the robot itself.f µ p , f σ p , f µ R , f σ R areWe introduce an instance of two more transformers f agg trans and f post trans , an MLP f aggpos , a learnable transformer positional embedding S agg , as well as a BEV decoder f BEV . We perform the node aggregation asThe feature aggregation function f agg is built from f agg trans , which consists of five transformer layers, and f post trans , which consists of one transformer layer, all of which have an embedding size of F = 192. The MLP f aggpos consists of three layers with 17, 48 and 48 neurons each, the positional embedding S agg is initialized similar to the one in f pose , and the BEV decoder f BEV takes the first sequence element of f post trans as input and scales it up through seven alternations of convolutions and upsampling.EQUATIONEQUATION)EQUATIONWe concatenate the node features along the sequence dimension and add the positional embedding (Eq. 13). We then concatenate a nonlinear transformation of the edge features along the feature dimension to the output of the previous operation (Eq. 14). The previous operations are performed for all v j ∈ N (v i ), but it is essential to include self-loops to the graph topology, as a full forward pass of the neural network would otherwise not be possible if N (v i ) were empty. We therefore set the edge embedding to the zero tensor 0 in that case (Eq. 14). We then aggregate over the set of neighbors and itself to produce the output embedding (Eq. 16) and finally use this to estimate any other downstream task, for example, estimating a BEV (Eq. 17).All transformer blocks have 12 attention heads. The MLP used in the transformer has a hidden size of 4F . All blocks utilize a dropout of 0.2. The model has a total of 43.7 M parameters (22.1 M of which are frozen DinoV2 parameters and 21.6 M trainable parameters).We utilize the Habitat simulator [68, 69] along with the training set of the HM3D dataset [70] , which comprises 800 scenes of 3D-scanned real-world multi-floor buildings. We select Habitat and HM3D for their photorealism and the availability of ground-truth pose information, as well as detailed navigation mesh information that can be converted to a BEV representation of the scene. We instantiate a single agent equipped with a calibrated, rectified camera with a field of view of 120 • , which corresponds to the camera configuration on our robots. For each scene, we randomly sample several uniform agent positions based on the total area of the scene. For each sample, the camera orientation is randomized across all three axes. Additionally, for each scene, we extract the processed navigation mesh, which indicates areas that are navigable by a mobile robot, or not navigable due to obstacles or overhangs. We extract 3, 816, 288 images from a calibrated camera with a 120 • field of view from 800 scenes and 1, 411 floors, along with corresponding BEV representations.The samples of the datasets D Sim Train D Sim Test and D Sim Val are randomly sampled to be within orientation range [-π 32 , π 32 ] for roll and pitch, and within a height of 0.0 to 2.0 meters, or less if the ceiling is at a lower height.Train6D D Sim Test6D are randomly sampled to be within orientation range [-π, π] for roll and pitch. We furthermore randomize the FOV ranging from 60 • to 120 • per camera. The ground truth BEV BEV i is a cropped 6 m × 6 m view of the floor's BEV around v i , generated through Habitat's navigation mesh. Each agent v i is centered and facing north.Data collection involves a base station with dual cameras and a 2D Lidar, plus a distinctively shaped marker on each robot. We manually navigated robots around the base station to capture a variety of relative poses, then used a custom tool to label Lidar data for pose extraction.All robots are equipped with an NVidia Jetson Orin NX 16GB and a forward-facing camera, rectified to a 120 • FOV with an image resolution of 224 × 224 pixels. We collected the real-world test set D Real Test in eight indoor scenes. It consists of 5692 images. We sample image pairs to ensure a uniform distribution of relative positions from 0 m to 2 m. This results in 14008 sets of samples, each with three images, totaling 84048 pose edges, 32% which have no visual overlap. Our dataset covers a diverse and challenging range of indoor scenes, as well as one outdoor scene.We explain the setup of our data collection unit in Fig. 6 . We visualize the five different scenes we collected the dataset from as well as the data distribution from each respective scene in Fig. 7 .We use Pytorch Lightning to train our model for 15 epochs on two NVidia A100 GPUs. This training takes approximately 24 hours. We optimize the weights using the AdamW optimizer and use the 1cycle learning rate annealing schedule throughout the training and configured weight decay. We choose the best model based on the performance on the validation set.We set the initial learning rate to 1 -5 and configure a learning rate schedule to increase this over two epochs up to 1 -3 , followed by a sinusoidal decline to 0 over 20 epochs. The parameters α balances between Dice and BCE loss, and we set it to α = 0.5, and the parameter β balances between position and orientation prediction accuracy, and we set it to 1, given equal weight to both. Rot. Var (deg)Figure 9 : We show qualitative distributions for position and rotation errors and variances over the relative angle between two robots on all edges of the simulation testset D Sim Test . Two agents facing the same direction would have an angle difference of 0 • , whereas two agents facing the opposite direction would have an angle difference of 180 • . We show the FOV as dashed line, where any results below the line indicate that the edge has some image overlap and above no image overlap. The horizontal marginals show the distribution of each individual plot while the vertical marginal shows the distribution of angle differences over the dataset. While the position error and variance increase over the threshold, the majority of the predictions are still useful. The rotation error is consistent across angle differences, while the uncertainty increases. We compute the absolute Euclidean distance between the predicted and ground truth poses and the geodesic distance between rotations asD pos (p i,ij , pi,ij ) = ∥p i,ij -pi,ij ∥ D rot (R i,ij , Ri,ij ) = 4 • arcsin 1 2 d quat ( Ri,ij , R i,ij )for all edges and report median errors.We perform a further evaluation on the distributions of pose errors and variations on the simulation test set D Sim Test in Fig. 9 . The results are in line with the results performed on D Real Test in the main manuscript.We provide additional quantitative experiments in Fig. 9 BEV Experiments: We assess the effectiveness of BEV predictions, focusing on the contribution of pose predictions to enhancing BEV accuracy, in an ablation study detailed in Tab. 3.We conduct three experiments: (1) A model trained without incorporating any pose prediction into the BEV aggregator. This model can estimate local information and, to a degree, information aggregated through neighbors. However, it must implicitly learn to predict the relative poses between observations to successfully merge the BEV representations from multiple agents. (2) A model Test indicates our model performs between the two baselines. It significantly outperforms the pose-free model with an 8.75% improvement in accuracy, while the model using ground truth poses shows an 18.31% performance increase compared to the baseline without pose information. These results highlight the substantial benefit of incorporating pose predictions into BEV estimation, positioning our approach as a notably effective middle ground between the two extremes. Train ). We furthermore randomize the camera's FOV. We show the results in Tab. 4. While the simulated results are not comparable to the one reported in Tab. 1 as the dataset is different, the real-world dataset is identical, and the reported performance is approximately identical with a slight degradation. The rotation error on D Sim Test6D is higher as there is a larger number of cameras without any direct visual overlap, and the maximum possible pose error increases. EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONWe provide additional quantitative experiments in Fig. 16 , Tab. 5 and Tab. 6. We visualize four additional qualitative samples from the real-world dataset D Real Test in Fig. 17 and Fig. 18 . We investigate the efficiency of the uncertainty estimation as well as the distributions of pose errors more closely. Fig. 16 illustrates the distributions of position and rotation errors, along with the predicted variances over the angle difference between robot pairs in the dataset D SimTest . An angle difference of 0 • signifies robots facing the same direction, potentially resulting in significant image overlap, while an angle difference of 180 • indicates opposite directions, with no overlap. The Field Of View (FOV) overlap threshold is highlighted in red, delineating errors with overlap (below the Test , containing 84048 unique edges. Two agents facing the same direction would have an angle difference of 0 • , whereas two agents facing the opposite direction would have an angle difference of 180 • . We show the FOV as dashed line, where any results below the line indicate that the edge has some image overlap and above no image overlap. The horizontal marginals show the distribution of each individual plot while the vertical marginal shows the distribution of angle differences over the testset. Note that the angle distribution is not uniform due to the real-world dataset collection where at least two cameras always face the opposite direction. line) from those without (above the line). The analysis reveals that position errors remain consistent up to the FOV threshold, beyond which they escalate, as reflected in the predicted position variance. Conversely, rotation errors stay uniform across angle differences, with only a marginal increase for higher angles, though the predicted rotation variance notably rises.In Tab. 5, we investigate the performance of our model across the different scenes in the testing dataset D Real Test . We provide a detailed breakdown of the median pose error for Invisible, Invisible Filtered and Visible cases for each of the eight scenes. While the pose error for visible samples is consistently between 22 cm/5.2°and 48 cm/7.5°, there are more outliers for invisible samples. This is due to some scenes having more or less symmetry, thus making it more challenging for the model to accurately predict poses. The pose prediction for the outdoor scene are least accurate, which is to be expected since the model was trained on indoor scenes. The Sunny scene has severe shadows and direct sunlight, while still providing low pose errors. The pose predictions are useful in all scenarios, demonstrating impressive generalization ability across a wide range of scenes.Next, we evaluate the magnitude of pose prediction errors over different thresholds of distances between image pairs in Tab. 6. We consider five bands of distances separated in 0.4 m chunks, ranging up to 2m. Note that the worst-case pose error is 2 • d due to the three-dimensional pose predictions. While the error increases with an increasing distance between samples, the rotational error in particular stays low, and the position errors stay useful, ranging from 12 cm to 65 cm for visible samples, 34 cm to 134 cm for invisible filtered samples (samples that are not visible but for which he model predicts a high confidence) and 76 cm to 153 cm for all invisible samples. 802.11 uses low bit rates for broadcast due to a lack of acknowledgment messages, which are the typical feedback mechanism for data rate control and retransmissions. The probability of a frame being lost due to the underlying CSMA/CA frame scheduler is non-trivial (3%+); a condition which worsens with increasing participating network nodes.A higher level system, such as ROS2 communications middleware package CycloneDDS can manage retransmissions, however currently available systems are not well suited to the requirements of a decentralized robotic network. In particular they lack the ability to detect network overload conditions, which can result in retransmissions that further load the network; finally resulting in a rapid increase in packet loss rates which renders the network as a whole unusable. While it is possible to tune such communications packages manually, the best parameter values are sensitive to highly variable deployment configuration parameters including ambient wireless network traffic. Manual tuning of network hardware, such as forcing increased broadcast bit rates, can alleviate this problem but are generally not resilient as scale.Our custom messaging protocol is targeted to this specific use case, and includes the ability to dynamically backoff messaging load based upon detected network conditions. Our system is also uses a shared slot TDMA message scheduling system which works with the 802.11 frame scheduler to maximize frame reception probability without compromising overall data rates.Results: We benchmark the runtime performance of the model on the Jetson Orin NX in Tab. 7 by averaging 100 sequential forward passes. We compare 16-bit and 32-bit float evaluated on the GPU or CPU and compilation with TensorRT, a proprietary NVidia tool for ahead-of-time compilation of neural networks. The model f enc , generating the communicated embedding E i , has a total of 30 M parameters, and takes ca. 20 ms for one forward pass as FP16 optimized with Torch TensorRT on the GPU. This is 50× faster than running the same model on the CPU. We also note that changing from 32-bit floats to 16-bit floats results in a 2.75× speedup. The model f pose , processing a pair of the Figure 8 dynamic trajectory elaborated in the main paper, but instead of a Figure 8 , the reference is a rounded rectangle, with robots moving with up to 0.8 m/s.Tab. 8 reports quantitative evaluations for the same results, which are in line with the results reported in Tab. 1. Note that the velocities reported are average velocities of the follower robots. The velocities are consistent for both Figure 8 trajectories, but differ for the rectangle trajectory, as the inner robot moves more slowly than the outer robot.We a single-robot homing/trajectory replay experiment to further show the versatility of CoViS-Net. In this experiment, we treat a list of recorded keyframes as reference embedding to estimate the relative pose to given the currently observed image. If either the distance or the predictedhttps://proroklab.github.io/CoViS-Net/ 


Decision: Strength: 

- Reviewers praised the paper's focus on real-time performance and the demonstration on real robots. (Reviewer 3vko, Reviewer vrDg, Reviewer bCfq)
- Comprehensive evaluation: The paper includes both simulation and real-world experiments, uses multiple metrics, and compares against baseline methods, strengthening the results' validity. (Reviewer mxQi)
- Novel use of BEV representations: Employing BEV representations for improving relative pose estimation was deemed relatively original. (Reviewer bCfq)

Weaknesses:

- Limited theoretical novelty: Several reviewers found the paper's reliance on existing techniques
- Unclear justification for “foundation model" claim: The paper's characterization of its model as a "foundation model" drew criticism. Reviewers questioned the training data scope, specialized architecture, and lack of comparison with established foundation models. (Reviewer vrDg, Reviewer mxQi)

After rebuttal, the majority of the reviews lean towards accepting this paper. This work has strong real world evaluation and demonstration, while needs more polishing in presentation and positioning. I would recommend accepting this paper and encourage the authors to further improve the paper according to the suggestions.

Reviews: review: title: title: A paper with low theoretical novelty but good efforts to make the approach run in real-time
paper_summary: paper_summary: The paper introduces a method for relative pose estimation with uncertainty quantification using monocular visual images. The method is built upon DinoV2 for image encoding, where DinoV2 can address relative pose estimation without overlapping views. A pairwise pose encoder is then used to estimate the relative poses, and multi-node aggregation is utilized to construct a BEV representation. In addition, the uncertainty in the relative pose estimation is quantified using the Gaussian NLL loss. The lightweight model design and GPU acceleration allow the method to run onboard in real-time.
main_review: main_review: Strengths:

+ Efforts to make the approach run in real-time on a robot's onboard GPU are appreciated.

+ The video demonstrations on real robots are illustrative.

+ Overall, the paper is easy to follow.

Weaknesses:
- The theoretical novelty of this paper is low. The main contribution to relative pose estimation in non-overlapping cases relies heavily on the pre-trained DinoV2 model. The loss function for uncertainty quantification is also widely used. 

- It is not clear how the proposed method addresses non-overlapping multi-robot views for relative pose estimation. If two images have no overlap, why does the method work and can maintain small rotation uncertainty. Line 190 states, "...allows it to 'imagine' unseen portions of the scene..." What theoretical and experimental supports justify this statement?

- Visual cameras alone do not provide direct measurements of the absolute scale of objects or the distance between them. How does the proposed method address scale issues?

- How can the quantified uncertainties be used to improve pose estimation, especially given that DinoV2 is a component?

- How is the proposed method decentralized? Is it achieved by running the same full model on each robot?

- It appears that the experiments evaluate pairwise situations where each follower follows the leader. How can the proposed approach generalize to estimating poses of more than one robot? How does it handle potential conflicting pose estimation results (e.g., when two robots try to estimate each other's poses)?

- Minor: Why is the proposed method referred to as a foundation model? Using a pre-trained DinoV2 as an image encoder does not make the proposed method a foundation model. This terminology seems overstated and misleading.
questions: questions: Please see the review section.
limitations: limitations: 1
review_summary: review_summary: The theoretical novelty of the paper is limited, and several key aspects of the approach, such as addressing non-overlapping views and scale issues, are not well explained or justified. The efforts to achieve real-time performance are commendable.
title: title: Review of Submission 176
paper_summary: paper_summary: This paper presents a method for estimating relative poses between robots and "birds eye view" map representations in the setting of multi-robot navigation. The model is based on a neural network that uses pre-trained foundation model features (DINO) to predict relative poses and birds-eye view representations. The model is tested in real multi-robot experiments.
main_review: main_review: Overall, I think the proposed method is interesting, and there could be some technical insights in the methodology (as far as I can tell, it seems sound). The paper includes a nice demonstration on real hardware and lots of technical details (probably more than necessary for a CoRL paper).

Strengths:
- The idea to use DINO features from multiple robots as an intermediate encoding representation for multi-robot relative pose estimation seems reasonable.
- The model design and loss seem reasonable to me,
- The method is tested on real hardware, which is great to see.

Weaknesses:
- There are many seemingly misleading claims in the paper, including the presentation of this work as a "foundation model." As I understand it, this work uses pre-trained DINO features as an encoding. DINO itself is a "foundation model" in the common usage, but I'm not sure that this makes every downstream application which uses DINO immediately a "foundation model" as well. Calling the proposed model a foundation model would seem to imply that the proposed model itself is trained on a broad, large-scale dataset. As far as I can tell, this is not the case, but maybe I am missing something.
- Similarly, in the abstract the paper states "classical vision-based methods to regress relative pose are commonly computationally expensive (precluding real-time applications)." I suspect I must have misunderstood this claim, because I believe multi-robot SLAM broadly (and the related technologies it encompasses) is a relatively well understood area of research at this point. This includes, for example, visual feature matching / place recognition and relative pose estimation from feature correspondences. These problems are by no means _solved_, but computational expense does not seem to me to be the primary issue precluding the widespread use of these systems.
- The presentation seems unnecessarily verbose. It is clear that the authors have a lot of confidence in the proposed approach, and they want to communicate their enthusiasm, but occasionally I felt that too much information was being thrown out at once, and it would serve the paper well to more concisely highlight just those key ideas which are most relevant to the proposed work. To give some specific examples to guide potential revisions, phrases like: "a decentralized, platform-agnostic visual spatial foundation model designed for real-world multi-robot applications," "a deep distributed stateless architecture to tackle multi-robot uncertainty-aware relative pose estimation," "platform-agnostic, decentralized, real-time multi-robot pose estimation from monocular images," really make it hard as a reader to identify what the most salient aspect(s) of the proposed contribution is.
- One of the major proposed benefits of the presented work is that it will attempt to estimate a relative pose even between non-overlapping images based on an imagined context. While this is demonstrated to work within the scope of the presented experimental results, it's not actually obvious to me that this is a desirable property of such a system. I still think the method is interesting and there could be some benefits to predicting local context outside the field of view (for example for local planning), but it seems for practical purposes that it might be better in those instances to simply recognize that there is no overlap and not attempt to register the relative pose between the robots.
- The "Limitations and Conclusions" section does not appear to contain any description of the limitations of the proposed system.
questions: questions: See comments above
limitations: limitations: 1
review_summary: review_summary: I am recommending "weak accept" on the basis that the method itself seems interesting and the technical evaluation seems reasonable to me, although some aspects of the presentation could be clarified.
title: title: Official Review for Paper176 From Reviewer bCfq
paper_summary: paper_summary: The paper proposes CoViS-Net, a decentralized, platform-agnostic visual spatial foundation model designed for multi-robot applications in unstructured environments. CoViS-Net leverages spatial priors learned from data to provide real-time pose estimation and bird’s-eye-view (BEV) representations, even without camera overlap between robots. The model operates on onboard compute without requiring existing networking infrastructure.
main_review: main_review: ## Quality  
The paper presents several strong technical contributions. The ablation study effectively motivates key components of the model, providing insights into the importance of each aspect. The inclusion of real-world experiments to validate the model’s performance significantly enhances the credibility of the proposed approach, demonstrating its practical applicability. However, the paper could benefit from a scalability analysis, particularly in larger multi-robot systems, and the potential challenges and limitations associated with it. The benchmarking section, which compares CoViS-Net with only two feature matching methods, is somewhat limited. Adding a third benchmark will help to better highlight the advantages of the proposed model. 
   
## Clarity  
The paper is generally well-structured and easy to follow, with a clear flow of information from the introduction to the experimental results. However, the literature review could be improved by discussing related work on map prediction methods and how BEV representations relate to these similar approaches. Specifically, citing necessary map prediction papers such as:  
- 4CNet: A Confidence-Aware, Contrastive, Conditional, Consistency Model for Robot Map Prediction in Multi-Robot Environments, 2024 
- Enhancing robot task completion through environment andtask inference: a survey from the mobile robot perspective, 2022 
- Distributed Inference based multi-robot exploration, 2018 
 
Additionally, other applications in the literature review should be included, such as multi-robot exploration/navigation/search: 
- NavFormer: A Transformer Architecture for Robot Target-Driven Navigation in Unknown and Dynamic Environments, 2024
- Deep Reinforcement Learning for Decentralized Multi-Robot Exploration with Macro Actions,2023 
- A multirobot person search system for finding multiple dynamic users in human-centered environments, 2023 
 
A more detailed explanation of why feature matching benchmarks were chosen for comparison would improve clarity. While the paper proposes a method for improving relative pose estimations and BEV representations, the comparison with feature matching methods like ORB and LightGlue is not entirely clear. This table should also include FPS to see how the proposed method compares in terms of performance. It is also not clear why the ablation and comparison methods use different metrics, which could be. As Euclidean distance between the predicted and ground truth poses and the geodesic distance between rotations seem like more direct metrics when evaluating pose estimation and BEV representation. 
 
## Originality  
Using BEV representations to improve relative pose estimation is relatively original. However, the paper does not sufficiently compare this approach to other map prediction methods or tasks such as robot exploration, which could potentially address the same issues. A more detailed comparison with these areas would better highlight the novelty and advantages of CoViS-Net. 
   
## Significance  
The paper is significant because it provides a robot/platform-agnostic solution to multi-robot pose estimation and spatial understanding without requiring external networking infrastructure. Its real-time processing capabilities make it highly practical for real-world applications.   
 
## Strengths: 
- Effective ablation study to motivate key components. 
- Strong real-world validation with detailed experimental analysis. 
- Incorporation of uncertainty estimation. 
- Robot/platform-agnostic approach that does not require external networking infrastructure. 
- Real-time processing capabilities. 

## Weaknesses: 
- Improve the literature review by including additional multi-robot search and exploration papers, focusing on map prediction methods and adding a scalability analysis. 
- Ensure comparison metrics are consistent or clearly justified, and include an ablation study comparing with and without BEV generation to highlight its benefits.
questions: questions: - The literature review needs improvement. Can the authors discuss related work on map prediction methods and how BEV representations relate to these approaches? Additionally, include relevant studies on multi-robot exploration and search. 
- The use of different metrics for ablation and comparison methods is confusing. Can the authors streamline the metrics and use consistent measures such as Euclidean distance and geodesic distance for evaluating pose estimation and BEV representation?\ 
- Can the authors include an ablation study comparing results with and without BEV generation to highlight its benefits? 
- Please include limitations of your approach.
limitations: limitations: 1
review_summary: review_summary: This work demonstrates several strengths, including an effective ablation study that underscores the importance of key components, strong real-world validation with comprehensive experimental analysis, and the incorporation of uncertainty estimation. The approach is robot/platform-agnostic, does not rely on external networking infrastructure, and supports real-time processing capabilities.     However, there are some areas for improvement. The literature review should be enhanced by including more papers on multi-robot search and exploration, with a particular focus on map prediction methods and scalability analysis. Additionally, comparison metrics should be made more consistent or clearly justified, and an ablation study should be conducted comparing scenarios with and without BEV generation to highlight its benefits.  I recommend weak accept.
title: title: A decentralized, platform-agnostic visual spatial model for multi-robot system.
paper_summary: paper_summary: This paper introduces a transformer-based architecture to enable accurate real-time pose estimation and spatial understanding in multi-robot systems, validated through comprehensive simulations and real-world experiments.
main_review: main_review: [Strengths]
1. Innovative Approach: The use of a transformer-based architecture for pose prediction is novel and interesting. The integration of visual and inertial data through a cooperative network enhances pose estimation accuracy, as evidenced by the experimental results.

2. Comprehensive Evaluation: The authors provide a thorough evaluation of their model, including both simulated and real-world datasets. The use of different metrics (Euclidean distance, geodesic distance) to report errors adds credibility to their findings.

3. Baseline Comparisons: The inclusion of baseline comparisons with ORB and LightGlue methods effectively highlights the superior performance of CovisNet. This comparative analysis is crucial for establishing the novelty and effectiveness of the proposed method.

[Weaknesses]

The characterization of CoViS-Net as a "foundation model" warrants a closer examination. Here are several points that challenge the classification of CoViS-Net as a true foundation model.

1. Scope of Training Data: Foundation models are typically trained on extensive and diverse datasets that cover a wide array of scenarios and contexts. While CoViS-Net uses a substantial dataset, including the HM3D dataset and simulations, its focus remains narrow within the domain of multi-robot systems. The training data is tailored to specific robotic tasks, limiting its generalizability.

2. Specialized Architecture: The architecture of CoViS-Net, while innovative, is specialized for the task of relative pose estimation and BEV representation in multi-robot systems. Foundation models usually feature a more generic architecture that can be adapted for multiple, diverse tasks. The specialized nature of CoViS-Net’s architecture suggests a more application-specific model rather than a general-purpose foundation model.

3. Scale: Foundation models are often exceptionally large, with billions of parameters, enabling them to capture vast amounts of knowledge and nuances from their training data. CoViS-Net, while complex, does not reach the scale typically associated with foundation models. Its parameter count and computational requirements are more in line with specialized deep learning models rather than the large-scale models seen in NLP or vision foundation models.
questions: questions: Could the authors provide a clear definition of what they consider a "foundation model" and how CoViS-Net fits into this category?

How does CoViS-Net compare to well-known foundation models in terms of architecture, training scope, and adaptability?

What is the extent of diversity in the training data used for CoViS-Net? How do the authors ensure that the model generalizes well beyond the specific scenarios presented in the paper?

What is the parameter count of CoViS-Net, and how does it compare to traditional foundation models? Can the authors justify the scale of the model in the context of foundation models?
limitations: limitations: 3
review_summary: review_summary: The specialized nature, limited training scope, and lack of demonstrated versatility across tasks suggest that CoViS-Net is better classified as a highly specialized model rather than a general-purpose foundation model. This distinction is important for setting appropriate expectations and understanding the potential and limitations of the proposed approach.