CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems
Authors: Rui Liu
Yu Shen
Peng Gao
Pratap Tokekar
Ming Lin
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Modal Learning
Multi-Agent Systems
Primary area: general_machine_learning
TL;DR: A novel multi-modal multi-agent framework that enables agents to collaborate and share multi-modal data during training while allowing inference with reduced modalities at test time.

Abstract: Multi-modal learning has emerged as a key technique for improving performance across domains such as autonomous driving, robotics, and reasoning. However, in certain scenarios, particularly in resource-constrained environments, some modalities available during training may be absent during inference. While existing frameworks effectively utilize multiple data sources during training and enable inference with reduced modalities, they are primarily designed for single-agent settings. This poses a critical limitation in dynamic environments such as connected autonomous vehicles (CAV), where incomplete data coverage can lead to decision-making blind spots. Conversely, some works explore multi-agent collaboration but without addressing missing modality at test time. To overcome these limitations, we propose Collaborative Auxiliary Modality Learning (CAML), a novel multi-modal multi-agent framework that enables agents to collaborate and share multi-modal data during training, while allowing inference with reduced modalities during testing. Experimental results in collaborative decision-making for CAV in accident-prone scenarios demonstrate that CAML achieves up to a 58.1% improvement in accident detection. Additionally, we validate CAML on real-world aerial-ground robot data for collaborative semantic segmentation, achieving up to a 10.6% improvement in mIoU.

CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems
Rui Liu', Yu Shen?, Peng Gao Pratap Tokekar', Ming Lin University of Maryland, College Park
AAdobe Research
North Carolina State University
Abstract
Multi-modal learning has emerged as key technique for improving performance across domains such as autonomous driving, robotics, and reasoning. However, certain scenarios, particularly in essource-consrrinee environments, some modal- ities available during training may be absent during inference. While existing frameworks effectively utilize multiple data sources during training and enable inference with reduced modalities, they are primarily designed for single-agent settings. This poses critical limitation in dynamic environments such as connected autonomous vehicles (CAV), where incomplete data coverage can lead to decision- making blind spots. Conversely, some works explore multi-agent collaboration but without addressing missing modality at test time. To overcome these limita- tions, we propose Collaborative Auxiliary Modality Learning (CAML), a novel multi-modal multi-agent aggett framework that enables agents to collaborate and share multi-modal data during training, while allowing inference with reduced modalities during testing. Experimental results in collaborative decision Decision-making for CAV in accident-pre prone scenarios demonstrate that CAMI. MI. achieves up to 58.1% 1% im- provement in accident detection. Additionally, we validate CAMI, on real-world aerial-ground robot data for collaborative semantic segmentation, achieving up to a 10.6% 6% improvement in mloU.
Introduction
Multi-modal learning has become an essential approach in wide range of machine learning systems, particularly in areas such as autonomous driving [6. 27, 46], robotics [20, 26, 36], and reasoning [21, 28, 291, where the availability of multiple data sources (c.g., RGB, LiDAR, radar, text) improves model performance by providing complementary information. However, these multi-modal systems often incur increased computational overhead and latency at inference time. More critically, in real-world conditions, some modalities may be unavailable, or too costly acquire test time, which challenges the reliability and efficiency of such systems.
Recent work in machine learning [9, 10, 16. 37, 45] has sought address these problems by enabling models to leverage additional modalities during training while supporting inference with fewer modalities. Such approaches reduce computational costs and improve robustness conditions where some sensors may be unavailable at inference time. Shen et al. [41] formalized these tasks under the framework of Auxiliary Modality Learning (AML), which effectively reduces the dependency on expensive or unreliable modalities.
Despite the benefits of AML, notable limitations remain. AML is primarily designed for single-agent settings, where an individual model is trained to handle reduced modalities during inference. A key challenge in current AML frameworks insufficient data coverage, particularly in dynamic environments such as connected autonomous vehicles (CAV). In these scenarios, single agent's data
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

is often incomplete due to occlusions or limited sensor range, resulting in blind spots and increased uncertainty decision-making making. In contrast, aggett systems, enabled by technologies such as vehicle-to-vehicle (V2V) communication or collaborative robotics, can access complementary sensory information from other agents. Sharing this information allows for more informed and safer decisions, especially in accident-pros prone scenarios. However, existing single-agent AML approaches fail to leverage this collaborative potential.
To address these limitations, we propose Collab- orative Auxiliary Modality Learning (CAML), a novel framework for multi-modal, multi-agent systems. CAML enables agents to collabo- rate and share multi-modal data during train- ing while supporting inference under reduced- modality conditions at test time, as illustrated Fig. 1. This collaborative learning paradigm enhances scene understanding and data cover- age by allowing agents to compensate for each other's blind spots, leading to more informed and coordinated decision- decision-making. To ensure robust deployment in scenarios with missing modalities (c.g., due to sensor failures or re- source constraints), CAML employs knowl- edge distillation (KD) [15] to transfer knowl- edge from teacher model trained with full- modality inputs to a student model that performs effectively with limited modalities. By enabling cross-modal and cross-agent knowledge transfer, CAML extends traditional KD to handle partial observability and support team-level coordina- tion. The distillation process embeds rich, full- uall-modlity knowledge into the student model, preserving the benefits of collaborative training and ensuring reliable inference even when some modalities are unavailable. For example, autonomous driving, multiple vehicles can share LiDAR and RGB data during training to build robust shared representations, while at deployment each vehicle performs inference using only RGB inputs.
Training
Testing with missing modalities and agents
Figure 1: Illustration of CAML. CAML enables (I) multiple agents collaborate and share multi- modal data during training while allowing for run- time inference with reduced modalities during test- ing: (2) the number of agents can vary between training and testing, ensuring flexibility and ro- bustness in deployment.
Unlike prior work that either focuses on multi-agent collaboration but without addressing missing modality at time, explores AML ideas solely in single-agent settings, CAML unifies these concepts. To the best of our knowledge, this work is the first to propose a flexible and principled framework for Overall, this work offers the following key contributions:
multi multi-agent systems where limited modalities are available for inference.
We introduce CAML, novel framework for multi-agent systems that allows agents to share multi-modal data during training, while supporting reduced-modality inference during testing. This collaborative approach enhances data coverage, improves robustness to missing modalities, which is crucial for deployment in in dynamic, resource-cone conouruce- environments.
We validate CAML through extensive experiments in collaborative decision-makinn making for connected autonomous driving in accident-prone scenarios, and collaborative semantic segmentation for real-world data of aerial-ground robots. CAML achieves up to 58.1% improvement in accident detection for autonomous driving, and up to 10.6% improvement for more accurate semantic segmentation.
2 Related Work
Multi-Agent Collaboration. Collaboration in multi-agent systems has been widely studied across fields such as autonomous driving and robotics. In autonomous driving, prior research has explored various strategies, including spatio-temporal graph neural networks [8], LiDAR-based end-to-end systems [5], decentralized cooperative lane-changing [35] and game-theoretic models [13]. In robotics, Mandi et al. [34] presented hierarchical multi-robot collaboration approach using large language models, while Zhou et [49] proposed a perception framework for multi-robot systems built on graph neural networks. A review of multi-robot systems in search and rescue operations

---

was provided by [39], and Bae et al. [1] developed a reinforcement learning (RL) method for multi- robot path planning. Additionally. various communication mechanisms, such as Who2com [31], When2com [30], and Where2comm [17), have been created to optimize agent interactions.
Despite these advancements, existing multi-agent collahoration frameworks remain limited by their focus on specific tasks and the assumption that agents will have consistent access to the same data modalities during both training and testing, an assumption that may not hold in weal-world applications. To address these gaps, CAML enables agents to collaborate during training by sharing multi-modal data, but test time, each agent performs inference using reduced modality. This reduces the dependency on certain modalities for deployment, while still allowing agents to leverage additional information during training to enhance overall performance and robustness.
Auxiliary Modality Learning. Auxiliary Modality Learning (AML) [41] has emerged as an effective solution to reduce computational costs and the amount of input data required for inference. By utilizing auxiliary modalities during training, AML minimizes reliance on those modalities at inference time. For example, Hoffmman et al. [16] introduced a method that incorporates depth images during training to enhance test-time RGB-only detection models. Similarly, Wang et al. [45] proposed PM-GANs to learn a full-modal representation using data from partial modalities, while Garcia et al. [9, 10] developed approaches that use depth and RGB videos during training but rely solely on RGB data for testing. Piasco et al. [37] created a localization system that predicts depth maps from RGB query images test time. Building on these works, Shen et al. [41] formalized the AML framework, systematically classifying auxiliary modality types and AML architectures.
However, existing AML frameworks typically designed for single-agent settings, failing exploit the potential benefits of multi-agent collaboration for improving multi-modal learning. Transition- ing from single-agent to multi-agent learning fundamentally non-trivial, as multi-agent systems introduce unique challenges such as dynamic team compositions and team-level decision-making CAML effectively addresses these challenges and allows agents to collaboratively learn richer multi-modal representations. By sharing complementary information, CAML enhances overall data coverage and mitigates information loss during reduced-modee modality inference, which is particularly important when an individual agent's 's observations are incomplete due to occlusions or limited sensor range.
Knowledge Distillation. Knowledge distillation (KD) [15] is widely used technique in many domains to reduce computation by transferring knowledge from a large, complex model (teacher) to simpler model (student). In computer vision, Gou et al. [11] provided a comprehensive survey of KD applications, while Beyer et al. [2] conducted an empirical investigation to develop a robust and effective recipe for making large-scale models more practical. Additionally, Tung and Mori [44] introduced KD loss function that aligns the training of student network with input pairs producing similar activation the teacher network. In natural language processing, Xu et al. [47] reviewed the applications of KD in LLMs, while Sun et al. [42] proposed a Patient KD method to compress larger models into lightweight counterparts that maintain effectiveness. Hahn and Choi [12] suggested a self-distillation approach that leverages the soft target probabilities of model to train other neural networks, while Liu et al. [25] explored KD under uncertainty. In autonomous driving, Lan and Tian [19] presented an approach for visual detection, Cho et al. [4], Sautier et al. [40] used KD for 3D object detection.
Notice that existing KD mostly distills knowledge from a larger model to smaller one to reduce computation, Shen et al. [41] aimed to design a cross-modal learning approach using KD to utilize the hidden information from auxiliary modalities within the AML framework. But AML is limited by the scope of single-agent paradigm. In contrast, we leverage KD within multi- multi-agent settings, where the teacher models are trained with access to shared multi-moda data (e.g., RGB and LiDAR) from multiple agents. By distilling this collaborative knowledge into each agent's reduced modality (e.g.. RGB), CAML enables robust inference during deployment, even with fewer modalities. This collab- orative distillation process enhances each agent's performance by providing richer, complementary knowledge from the collaborative training phase.
3 Collaborative Auxiliary Modality Learning
In single-agent frameworks such as AML [41], the missing modalities during testing are referred to as auxiliary modalities, while those that remain available are called the main modality. In contrast, in our framework CAML, each agent can process different number of modalities during training and
3

---

Observations Ay
Collaborative Feature Embeddings
leacher Model
Encoder k
Featurey
Teacher Predictior
Encoder
Feature
Aggregation
Encoder k
Feature
Multimodal Aggregation
Prediction
Encoder
Feacuree
X2
Encoder 1M
Feature !1
Aggregation
X
Encocier m
Feattrre
Knowledge Distillation
Observations Xust
Collaborative Festure Embeddings
Student Model Prodiction
Avr:
Encoder /
Fenture 2t
Studen1 Prediction
Aggregation)
Encoder
Fenturte
Muttimodal Aggregation
Predictionn
Mr
Enceader
Featurei 14
Aggregation 12
As
X
Encoder
Feature
Figure 2: Overview of the Pipeline of CAML. The teacher model (top) aggregates and shares multi- modal embeddings across agents make predictions using the full set of modalities. In contrast, the student model (bottom) processes subset of modalities per agent and shares them to form multi- modal embedding. Through knowledge distillation from the teacher, the student learns to produce robust predictions despite missing modalities, enabling effective inference during deployment. In the teacher model, the set of agents denoted as Aurin = [Ai: A2. .... Ar). The set of modalities is denoted as Turain The observations of all agents are denoted as X (2, ,.......''' NNN. ). where rF is the observation acquired by the i-th agent A, E Avain for the k-th modality. In the student model, the set of agents is denoted A as Traa, which is a subset of Train- The set of agents that have access to the j-th modality I, Ieest is denoted as AT and the number of agents in this set is given by | = Mj, with T'M, represents the observation acquired by the Mt agent AM, E Alest for the j-th modality.
A2.......... Av.. The set of modalities is denoted
different agents can have different main modalities and auxiliary modalities. There is no correlation between the number of agents and the number of modalities.
Problem Formulation. We define our problem in both training and testing phases. In the training phase, we consider multi-agent system with N agents collaboratively completing task. The set of agents is denoted as Aurain {A1 A2,..., AN J. The observations of all agents are denoted as X [ EN }. where z; is the observation acquired by the i-th agent A; E Arran- The ground truth denoted as Y. which can be object label, semantic class, or a control command (e.g., brake for an autonomous vehicle). The set of modalities is denoted as Tarain I I.....,.. .IKK), J. such as RGB, LiDAR, Depth, etc, where is the number of modalities available during training. During training, each agent has access to all these K modalities. In the testing phase, we assume there are M agents. The set of test agents is denoted as Agst = [A1, .. AM In addition, the set of modalities is denoted Iess which is a subset of Imain- The number of modalities available during testing is denoted as L, where < K. The set of agents that have access to the j-th modality
I, E Irstt is denoted as ALe where A E Aues, and the number of agents in this set is given by A My. This means that during testing, each agent may have access different number of modalities.
Approach. Given the problem definition, we aim to estimate the posterior distribution P of the ground truth label y given all agents' observations X. During training, we first train teacher model where each agent has access to all modalities in Iurin and then student model where each agent has access to partial modalities in Zeeu- We employ Knowledge Distillation (KD) to transfer the richer multi-modal knowledge from the teacher to the student, enabling the student to benefit from information beyond own input, as illustrated in Fig. the student model, which operates solely on the available test modality observations X
test time, we perform inference using

---

Specifically, in the teacher model, each agent has access to all multi-modal observations and indepen- dently processes its local observations to produce embeddings. These embeddings are then shared among agents based on whether the system operates in centralized or decentralized manner. If the system is centralized, all collaborative agents share their embeddings with one designated ego agent for centralized processing. If the system is decentralized, each agent shares the embeddings with nearby agents. Subsequently, the shared embeddings corresponding to the same modality are aggre- gated together. Then we fuse (e.g., via concatenation cross-attention) the aggregated embeddings of different modalities to create a comprehensive multi-modal embedding. This multi-modal embedding is then passed through a prediction module to produce the teacher model's final prediction. The student model follows a similar network architecture as the teacher. However, instead of processing all modalities, each agent processes only single or a subset of modalities, which can vary across agents. By sharing these embeddings among agents, the student model also constructs a multi-modal embedding, leveraging the different modalities observed by various agents. This multi-modal embed- ding is then used to generate the student model's prediction. Our approach enables the student model to maintain effective performance despite missing modalities during testing, significantly enhancing its robustness.
We further provide an intuitive analysis offering insights and explanations to justify the effectiveness of CAML by examining how our approach enhances data coverage and leads to greater information gain compared to single-agent approaches. Specifically, we show that aggregating observations from multiple agents enables broader coverage of the data space, capturing complementary information that would be missed by an individual agent. From information theory perspective, demonstrate that joint observations from multiple agents generally yield higher mutual information with respect to the ground truth, providing more informative signal. Please refer to Appendix A. for more details.
System Complexity. After designing the CAML framework, we present a detailed complexity analysis. If the system is centralized, all collaborative agents share their data with one designated ego agent for centralized processing. Each of the N - collaborative agents performs its local computation independently, with time complexity of (T²)) and a space complexity of OSa), where T, represents the time required for local computation, and S. is the associated space. Thus, the total computation time and space complexities for all collaborative agents ( - 1)) and O(S((N respectively. For simplicity, assuming each communication from one collaborative agent to the ego agent consumes (D) time complexity and ((M) space complexity, where D is the time required for communication and M is the corresponding space. Therefore, the total communication time and space complexities for gathering information the ego agent are O(D(N 1)) and OMM(N - 1))- respectively. Then the ego agent aggregates the received data, rumning a model, having time and space complexity OTT)) and (Se), where Te and S. represent the time and space required for the ego agent's 's computation. So the total time and space complexities are ((Tr(N-1)-)) DIN and (N - 1) + M (NN 1) + S.). respectively.
1)).
If the system is decentralized, each agent performs its local computation and shares information with nearby agents. For simplicity, let the local computation for single agent has time complexity of OTT), where T is the time required for local computation. Assume that communication from one agent to another agent requires (DD) time complexity and ((M) space complexity, where D represents the time of communication between two agents, and M denotes the space required for such communication. For N agents, the total computation time complexity is O(NT). In the worst case, each agent share data with all other agents, this can result in O(N D) for pairwise sharing. So the total time complexity is O + +N For space complexity, the storage requirement for all agents is (), where S is the space needed per agent. Communication between agents adds an additional complexity of 2(N2 So the total space complexity is O(NS + N2 M). In the typical case, if each agent communicates with only other i agents (k < N) rather than all N - 1 agents. The total time and space complexities become O(NT + NkD) and O(NS NKM), respectively.
4 Experiments
4.1 Collaborative Decision-Making
To evaluate our approach, we first focus on collaborative decision-making in connected autonomous driving (CAV). This involves making critical decisions for the ego vehicle in accident-prone scenarios, such as determining whether not to take a braking action.
5

---

Data Collection. The dataset is generated using the AUTOCASTSIM benchmark [38], which features three complex and accident- prone traffic scenarios designed for CAV. These scenarios are characterized by limited sensor coverage and obstructed views, as illustrated in Fig. 6. Following prior works COOPERNAUT [5] and STGN [8], we employ an expert agent with full access to the traffic scene information to collect data. This expert agent is deployed in the environment to generate the data trails, where each trail captures RGB images and LiDAR point clouds from both the ego vehicle and collaborative vehicles, along with the control actions of the ego vehicle. In line with the setup used in STGN [8], the ego vehicle can have up to three collaborative vehicles at each timestep, provided they are within a 150-meter radius. Each trail represents unique instantiation of the traffic scenario, with differences arising from the randomized scenario configurations. For further dataset details, please refer to Appendix A.2.1.
Experimental Setup. After data collection, we train the models using Behavior Cloning (BC) [3, 23], following the setup established by prior works COOPERNAUT [5] and STGN [8]. For each vehicle, both RGB and LiDAR data are used during training, while only RGB data is used during testing in CAML, Please refer to Appendix A.2.3 for more details on the BC procedure. All experiments conducted across four repeated runs, and we report the mean and standard deviation of the results.
For processing RGB data, we first resize the image to 224 x 224 and use ResNet-18 18 [14] as the encoder to extract feature map for each vehicle. We then apply self-attention on the feature map to dynamically compute the importance of features at different locations. After the self-attention, we apply three convolution layers with each followed by a ReLU activation. Finally, we obtain a 256-d feature representation after passing through fully connected layer. To aggregate RGB feature embeddings from connected vehicles to the ego vehicle, we use the cross-attention mechanism. For processing the LiDAR data, we use the Point Transformer as the encoder for each vehicle and utilize the COOPERNAUT [5] model to aggregate LiDAR feature embeddings between connected vehicles. Then we concatenate the final RGB and LiDAR embeddings for the ego vehicle's 's decision-making with a three-layer MLP as the prediction module to output the action.
For the training of Knowledge Distillation (KD), we first train a teacher model offline using a binary cross-entropy loss, where each vehicle has both RGB and LiDAR data. Then we train a student model to mimic the behavior of the teacher model with only RGB data for each vehicle. For each data sample, the student model receives the same RGB image that the teacher model was given. For further details on the KD training process, please refer to Appendix A.5. And for the detailed training settings, please see Appendix A.4. We employ the following two metrics for evaluation: (1) Accident Detection Rate (ADR): This is the ratio of accident-prom. cases correctly detected by the model compared to the total ground truth accident- when the ego vehicle performs a braking action. This metric measures the model's 's effectiveness in identifying potential accidents. (2) Expert Imitation Rate (EIR): This denotes the percentage of actions accurately replicated by the model out of the total expert actions. It serves evaluate how well the model mimics expert driving behavior.
cases. An accident-prom case is identified
Baselines. We implement the following baselines for comparison: (1) AML [41]: In the AML setting, the ego vehicle operates independently without collaboration with other vehicles (non- collaborative). Both RGB and LiDAR data are available during training for the vehicle, while only RGB data is available during testing. (2) COOPERNAUT [5]: A collaborative method that processes LiDAR data during both training and testing. It employs the Point Transformer [48] as the backbone, encoding raw 3D point clouds into keypoints. (3) STGN [8]: A collaborative approach that utilizes spatial-temporal graph networks for decision-making, with RGBD data used for both training and testing.
Baselines Comparison. How well does CAML perform against other methods for decision in CAV? We evaluate CAML against the baselines and present the results in Fig. 3, which demonstrate clear performance advantage of CAML across all three accident-prone scenarios.
Compared to single-agent agent systems like AML, CAML achieves notable improvements in mean ADR: 13.00 3.0% in the overtaking scenario, 34.6% in the left turn scenario, and significant 58.1% in the red light violation scenario. These improvements highlight the advantages of CAML'' 's collaborative framework, which enables the ego vehicle to aggregate complementary sensory data from connected vehicles. This richer, multi-agent perspective significantly enhances situational awareness, allowing the ego vehicle to detect potential accidents and respond proactively, such as

---

100 009001
350
109
A16-14
193
150
504
20
20
CANL
AML
OXCPERAUTT
STOR
CAML
AML
COOMEMNUT SIOR
CAM
AML
COO
STOM
(a) Overtaking
(b) Left Turn
(c) Red Light Violation
Accident Detection Rate (ADR)
Expert Imitation Rate (EIR)
Figure 3: Performance Comparison of CAML Against Baselines. We evaluate performance using two metrics: Accident Detection Rate (ADR) and Expert Imitation Rate (EIR) across three accident- prone scenarios: (a) Overtaking, (b) Left Turn, and (c) Red Light Violation. CAML demonstrates superior performance across all scenarios compared to these baselines by up to 58.1 1 ,,, henefiting considerably from the multi-modal multi-agent collaboration.
braking in time to avoid collisions, especially in scenarios involving occlusions or restricted views. When compared COOPERNAUT, CAML achieves improvements in mean ADR by up to 21.5%, further demonstrating the strength of its collaborative and multi-modal approach.
Additionally, how does CAML compare with other approaches that have access more modalities during testing? We evaluate CAML against STGN [8] under multi-agent settings. CAML uses only RGB data during testing but STGN uses both RGB and depth data. Despite with fewer modality, CAML achieves comparable, and in some cases superior performance while relying solely on RGB data, as illustrated in Fig. 4. Notably, CAML exceeds the mean ADR of STGN by 10.9% in the left-turn scenario, demonstrating that our model can enhance driving safety even when constrained to fewer modalities. This further underscores the strength of CAML, which effectively leverages LiDAR data as an auxiliary modality during training to boost performance. The fact that CAML matches or exceeds the performance of model that uses more data at test time highlights the modality efficacy of our approach.
System Generalizability. How effectively does the system generalize when we have fewer agents during testing compared to training? (e.g., we have multi-agent collaboration during training but only single agent during testing). We test the case where multi-agent collaboration used during training, but only single agent is present during testing. This test is also motivated by practical constraints, where many real-world situations, multi-vehicle connected systems are not available, we only have a single vehicle. But it is reasonable to have multi-vehicle connected systems with multiple modalities during training to develop robust models. After training, we can then apply the model on a single vehicle for testing, which is very valuable in practice and provides cost-effective solution.
We compare the performance of our approach with COOPERNAUT [5] and STGN [8] under single- agent settings, with results shown in Fig. 4. CAML outperforms the other baselines across all scenarios, for both ADR and EIR metrics. This demonstrates that even with single agent during testing, CAML remains highly effective, by utilizing the multi-agent collaboration and auxiliary modalities provided by the teacher model during training.
Efficiency Analysis. We evaluate the communication and inference efficiency of CAML compared other cooperative ap- proaches, COOPERNAUT [5] and STGN [8]. To assess communication bandwidth, we follow STGN [8] and use the shared package size (PS) metric, which measures the size of data exchanged between connected vehicles, serving as an indicator of communication efficiency in collaborative decision-making
Table 1: Communication and inference effi- ciency comparison. CAML can operate more efficiently with lower communication overhead and faster inference than both [5] and [8].
Approach
PS (KB)
Latency (ms)
COOPERNAUT [5]
65.5
90.0
STGN [8]
4.9
18.5
CAML
1.0
3.7
For inference efficiency, we measure latency per batch, which reflects the average time required to process a batch during inference. The evaluation results are summarized in Table 1. As shown, CAML achieves lower PS and latency compared to other approaches, highlighting its superior communication and inference efficiency.
Ablation Studies. the ablation studies, we first compare the performance of CAML with two other basclines: the cully-e teacher model, and a multi-agent BC model trained and
7

---

100
100
160
123-11
733
513
Rote th1
600
Hote
40
40-
40-
CAML
COOPERNAUT
STGN
CAML COOPERNALT STGN
CAML COOPERNAUT STGN
(a) Overtaking
(b) Left Turn
(c) Red Light Violation
Accident Detection Rate (ADR)
Expert Imitation Rate (EIR)
Figure 4: Single-Agent System Generalizability of CAML. We evaluate the generalizability of
CAML by testing the case where we have multi-agent collaboration during training, but only single agent during testing. We compare the performance of our approach with COOPERNAUT and STGN under single-agent settings. CAML with single agent during testing consistently oulperforms the other baselines across all scenarios, offering a valuable and cost-effective solution for practical applications.
tested using only RGB images, without knowledge distillation (KD). As shown in Table 2, while CAML shows slight performance drop compared to the teacher, due to missing modalities at inference, it still achieves strong and robust performance. This highlights the effectiveness of our method in handling modality reduction scenarios, making it especially suitable for resource- constrained environments. Furthermore, CAML consistently outperforms the RGB-only BC baseline. The results demonstrate the advantage of incorporating multi-modal learning into multi- systems. Specifically, leveraging additional modalities during training through knowledge distillation allows CAML to encode richer information into the student model, resulting in improved performance at test time.
Next, we evaluate the case involving more modalities beyond RGB and LiDAR. Each vehi- ele now receives RGB, LiDAR, and state infor- mation (e.g.. position and velocity) during train- ing, while only RGB inputs are available dur- ing inference. As shown in Table 3, incorporat- ing state information as an additional modality further improves system performance, demon- strating CAML's is ability to effectively leverage richer multimodal data during training while maintaining strong performance under reduced- modality conditions.
Table 2:
Performance comparison (%) of CAML with the teacher model and the RGB-only BC baseline. While CAMI shows slight per- formance drop compared to the teacher model due to missing modalities at inference, it still achieves strong and robust performance. More- over, CAML outperforms the RGB-only BC base- line across all scenarios.
Real ADRI
ADR1 Wa
RGB only BC
852=10
$3.2.00
564215
786612
81021.1 661114 83051.
53.213
CAME
928208
$5.581
673310
797611
We also investigate the complementary role of different modalities by retaining LiDAR and re- moving RGB during testing. As presented in Table 4, the performance of retaining LiDAR is slightly better than retaining RGB, as Li- DAR provides more precise spatial localization, which is beneficial for accident detection.
Table 3: Performance evaluation with three modali- ties: RGB, LiDAR, and state information. Incorpo- rating state information as an additional modality further enhances overall system performance.
Modaltio
ADR
Ovenaking
IR
ADRY 077310
Lof Tam
Sal Liin Vitatoo
ADRI
IR1
ILIDAR
Furthermore, we compare CAML, which adopts an intermediate cooperation strategy, with a late cooperation baseline. the late cooperation set- ting, each vehicle independently processes RGB and LiDAR inputs using the same encoders as in CAML (ResNet-18 for RGB and PoiníTrans- former for LiDAR). followed by MLP heads to generate action logits. For each modality. we first fuse the logits from connected vehicles via averaging and then fuse the averaged logits across modalities to determine the ego vehicle's control actions. We apply the same KD procedure as CAML, distilling from the RGB-LiDAR teacher model to student model that uses only RGB at test time. As shown in Table 5, CAML outperforms
Table 4: Performance comparison (%) of CAML when retaining either RGB or LiDAR dur- ing testing. Both settings achieve comparable per- formance, but retaining LiDAR yields slightly bet- ter results due to more precise spatial localiza- tion, which benefits accident detection.
Modalit
AD&
Ovenaking
Lighighh Villation IRT
ROR
92.8+08
95.5
1
79
83001. 66
98.3+09
56.1-06
0-000

---

the late cooperation approach by enabling the model to learn richer and more complementary repre- sentations across agents and modalities, whereas late cooperation relies solely on output-leved fusion, leading to potential information loss.
Overall, the experimental results clearly illus- trate the superiority of our CAML framework. The ability of CAML to learn a more effec- tive driving policy stems from the collaborative multi-modal behavior of multiple agents, which together capture a wider and more nuanced rep- resentation of data. This broader data coverage enables the ego vehicle to make better- inttrrinformee decisions, improving safety and performance, particularly in complex, dynamic, and accident-prone environments where isolated agents with limited sensing.
Table 5:
Performance comparison (%) be-
tween CAML and the late cooperation approach. CAML achieves superior results by enabling the model to learn richer and more complementary representations across agents and modalities.
Approach
Uvemking
m
111
XDR
LAMI
127-1
Lase Cooperaton
$35210
102-13
785-09
582-1.5
81.7.15
4.2 Collaborative Semantic Segmentation
To further evaluate our approach, we focus on collaborative semantic segmentation by conducting experiments with real-world data from groiudd robots. We use the dataset CoPeD [50]. with one aerial robot and one ground robot, in two different real-world scenarios of the indoor NYUARPL. and the outdoor HOUSEA. For more details about the dataset, please refer to [50]. Additionally, we introduce noise to the RGBD data collected by the ground robot. For both aerial and ground robots, RGB and depth data are used during training, while only RGB data is used during testing in CAML.
Experimental Setup. We adopt the FCN [32] architecture the backbone for semantic segmenta- tion. To process RGB and depth data locally for each robot, we use ResNet-18 [14] as the encoder to extract feature maps of size x 7. Please refer to Appendix A.3.1 for more details of the exper- imental setup. We first train a teacher model offline with aerial-ground robots collaboration using cross-entropy loss, where each robot has both RGB and depth data. Then train student model to mimic the behavior of the teacher model with only RGB data for both acrial and ground robots through KD. The KD process is similar that of the collaborative prcision-makine in CAV, but here we use a cross-entropy loss as the student task loss. Please see Appendix A.4 for the detailed training settings. We evaluate performance using the Mean Intersection over Union (mloU) metric, which quantifies the average overlap between predicted segmentation outputs and ground truth across all classes. We compare the performance of CAML with other baselines including AML [41] and FCN [32]. In the AML approach, only the ground robot operates, with RGB and depth data available during training but only RGB data used for testing. The FCN approach involves only the ground robot operating with RGB data for both training and testing.
Experimental Results. We first present the experimental results of baselines comparison in Table 6, where CAML demonstrates superior performance in terms of mIoU across both in- door and outdoor environments. Specifically, CAML achieves an improvement of mIoU for 8.9% in indoor scenario and 10. 1..% in outdoor scenario compared AML [41]. We also show the qualitative results in Fig. 5. As we can see, despite the noisy input image from the ground robot, CAML produces predictions that are clos- est to the ground truth. This improvement is attributed to CAML's multi-agent collaboration, which provides complementary information to enhance data coverage and offers a more comprehensive understanding of the scenes. Additionally, the utilization of auxiliary depth data during training results in more precise segmentation outputs.
Table 6: Baseline Comparison of Semantic Seg- mentation on real-world dataset CoPeD [50] using grrial-ground robots in indoor and outdoor environ- ments. CAML achieves the highest mloU in both environments, with upto 10.6 % higher accuracy.
mloU (%)
Approach
Indoor
Outdoor
FCN [32]
51.2
56.2
AML [41]
55.9
60.3
CAML
60.1
66.8
Improvement over SOTA
4.2-8.9
6.5-10.6
Ablation Studies. We also investigate another variant of CAML, called Pre-fusion CAML, as ablation studies. In this variant, each robot first locally extracts feature maps of size 7 x for both RGB and depth modalities. Instead of separately fusing the RGB and depth features between the robots, we first fuse the feature maps of RGB and depth within each single robot using cross-attention. Then we share and merge the fused RGBD features between robots via concatenation. We also apply 1 x convolution to reduce the feature maps to the original channel dimensions. The multi-modal multi-agent feature aggregations then pass through the decoder. Finally, we obtain the output map by
9

---

Input
Ground truth
FCN
AML
CAML
Figure 5: Qualitative results of different approaches semantic segmentation real-world data from acrial-ground robots in scenarios of both indoor and outdoor environments. From left to right, input image for the ground robot, ground truth segmentation map. FCN prediction, AML prediction, and CAML prediction. CAML prediction is the closest to the ground truth.
upsampling to match the input image size. The mloU of the Pre-fusion CAML is similar to that of CAML, achieving 59.2% and 65.8% for indoor and outdoor environments, respectively. Although the fusion order is different, both versions benefit from robust feature aggregation and multi-agen collaboration, which ultimately results in better segmentation performance. And CAML can easily shift Pre-fusion CAML because of the flexibility of our framework. Please refer to Appendix A.3.2 for more details.
Conclusions
In conclusion, we propose Collaborative Auxiliary Modality Learning (CAML), unified frame- work for multi-modal multi-agent systems. Unlike prior methods that either focus on multi-agent collaboration without modality reduction address multi-modal learning in single-agent settings, CAML integrates both aspects. enables agents to collaborate using shared modalities during training while allowing efficient, mndalltty-rdduce inference. This not only lowers computational costs and data requirements at test time but also enhances predictive accuracy through multi-agent collaboration. We provide intuitive analysis of CAML in terms of data coverage and information gain, justifying its effectiveness. CAML demonstrates up to a 58.1 88.1% improvement in accident detection for connected autonomous driving in complex scenarios and up 10.6% mIoU gain in real-world aerial-groune collaborative semantic segmentation. These improvements underscore the practical implications of framework, especially for resource-constrained environments. For discussion on limitations and future work, please see Appendix A.6.
Acknowledgment
This research is supported in part by National Science Foundation, Dr. Barry Mersky & Capital One E-Nnovate Endowed Professorships.
References
[1] Hyansu Bae, Gidong Kim, Jonguk Kim, Dianwei Qian, and Sukgyu Lee. Multi-robot path planning method using reinforcement learning. Applied sciences, 911))3305,, 2019.
[2] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEFF/CV conference on computer vision and pattern recognition, pages 10925-10934,
2022.
10

---

[3] Amisha Bhaskar, Rui Liu, Vishnu D Sharma, Guangyao Shi, and Pratap Tokekar. Lava: Long- horizon visual action based food acquisition. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1929-8933 IEEE, 2024.
[4] Hyeon Cho, Junyong Choi, Geonwoo Back, and Wonjun Hwang. itkd: Interchange transfer- based knowledge distillation for 3d object detection. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 13540-13544 2023.
[5] Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17252-17262, 2022.
[6] Khaled EI Madawi, Hazem Rashed, Ahmad El Sallab, Omar Nasr, Hanan Kamel, and Senthil Yogamani. Rgb and lidar fusion based 3d semantic segmentation for autonomous driving. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 7-12. IEEE, 2019.
[7] Hongbo Gao, Bo Cheng, Jianqiang Wang, Keqiang Li, Jianhui Zhao, and Deyi Li. Object classification using enn-based fusion of vision and lidar in autonomous vehicle environment. IEEE Transactions on Industrial Informatics, 14(9):4224-4235 2018.
[8] Peng Gao, Yu Shen, and Ming C Lin. Collaborative decision-making using spatiotemporal graphs in connected autonomy. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4983-4989. IEEE, 2024.
[9] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pages 103-118, 2018.
[10] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Learning with privileged information via adversarial discriminative modality distillation. IEEE transactions on pattern analysis and machine intelligence, 42(10))2581-2555 2019.
[11] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey, International Journal of Computer Vision, 1789-11788-1819, 2021.
[12] Sangchul Hahn and Heeyoul Choi. Self-knowledge distillation in natural language processing. arXiv preprint arXiv: 1908, 01851, 2019.
[13] Peng Hang, Chao Huang. Zhongxu Hu, Yang Xing, and Chen Lv. Decision making of connected automated vehicles at an unsignalized roundabout considering personalized driving behaviours. IEEE Transactions Vehicular Technology, 70(5):4051 4064, 2021.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[15] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint 2015.
[16] Judy Hoffman, Saurabh Gupta, and Trevor Darrell. Learning with side information through modality hallucination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 826-834, 2016.
[17] Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication-efinn efffcceent collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35:4874 2022.
[18] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014.
[19] Qizhen Lan and Qing Tian. Instance, scale, and teacher adaptive knowledge distillation for visual detection autonomous driving. IEEE Transactions on Intelligent Vehicles, 8(3):235--23777 2022.
11

---

[20] Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3): 582-596, 2020.
[21] Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, et al. Self-rewarding vision-language model via reasoning decomposition. arXiv preprint arXiv:2508. 10008.99635 19652, 2025.
[22] Rui Liu, Guangyao Shi, and Pratap Tokekar. Data-driven distributionally robust optimal control with state-dependent noise. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9986-9991, IEEE, 2023.
[23] Rui Liu, Amisha Bhaskar, and Pratap Tokekar. Adaptive visual imitation learning for robotic assisted feeding across varied bowl configurations and food types. arXiv preprint arXiv:2403. 12891, 2024.
[24] Rui Liu, Anish Gupta, Erfaun Noorani, and Pratap Tokekar. Towards efficient risk-sensitive policy gradient: An iteration complexity analysis. arXiv preprint arXiv:2403.08955 2024.
[25] Rui Liu, Peng Gao, Yu Shen, Ming Lin, and Pratap Tokekar. Aukt: Adaptive uncertainty-guii knowledge transfer with conformal prediction. arXiv preprint 1xXiv:2502.16736 2025.
[26] Rui Liu, Zahiruddin Mahammad, Amisha Bhaskar, and Pratap Tokekar. Imrl: Integrating visual, physical, temporal, and geometric representations for enhanced food acquisition. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 741-747. IEEE, 2025.
[27] Rui Liu, Zikang Wang. Peng Gao, Yu Shen, Pratap Tokekar, and Ming Lin. Mmed: Multi-modal collaborative decision-making for connected autonomy with knowledge distillation. arXiv preprint arXiv:2509. 2025.
[28] Rui Liu, Dian Yu, Lei Ke, Haolin Liu, Yujun Zhou, Zhenwen Liang, Haitao Mi, Pratap Tokekar, and Dong Yu, Stable and efficient single-rollout rl for multimodal reasoning. arXiv preprint arXiv:2512. 18215, 2025.
[29] Rui Liu, Dian Yu. Tong Zheng, Runpeng Dai, Zongxia Li. Wenhao Yu. Zhenwen Liang, Linfeng Song, Haitao Mi, Pratap Tokekar, et al. Vogue: Guiding exploration with visual uncertainty improves multimodal reasoning. arXiv preprint arXiv:2510001444, 2025.
[30] Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 4106-4115, 2020.
[31] Yen-Cheng Liu, Junjiao Tian, Chih- Yao Ma. Nathan Glaser, Chia-Wer Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In 2020 JEEE International Conference on Robotics and Automation (ICRA), pages 6876 6883. IEEE, 2020.
[32] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[33] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016
[34] Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 286-299. IEEE, 2024.
[35] Jianqiang Nie, Jian Zhang, Wanting Ding, Xia Wan, Xiaoxuan Chen, and Bin Ran. Decentralized cooperative lane-changing decision maaiiinn-maiiggggg 4:9413-949 2016.
making for connected autonomous vehicles. IEEE access,
[36] Kuniaki Noda, Hiroaki Arie, Yuki Suga, and Tetsuya Ogata. Multimodal integration learning of robot behavior using deep neural networks. Robotics and Autonomous Systems, (266):221 2014.
12

---

[37] Nathan Piasco. Désiré Sidibé, Valérie image description with auxiliary modality for visual localization in challenging conditions. International Journal of Computer Vision, 129(1))85-2002 2021.
and Cédric Demonceaux. Improving
[38] Hang Qiu, Pohan Huang, Namo Asavisanu, Xiaochen Liu, Konstantinos Psounis, and Ramesh Govindan. Autocast: Scalable infrastructure--les cooperative perception for distributed collabo- rative driving. arXiv preprint arXiv:21
2021.
[39] Jorge Pena Queralta, Jussi Taipalmaa, Bilge Can Pullinen, Victor Kathan Sarker, Tuan Nguyen Gia, Hannu Tenhunen, Moncef Gabbouj, Jenni Raitoharju, and Tomi Westerlund. Collaborative multi-robot search and rescue: Planning, coordination, perception, and active vision. leee
Access, 8:191617-19164. 2020.
[40] Corentin Sautier, Gilles Puy. Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self supervisdd distillation for autonomous driving data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9891-9901, 2022.
[41] Yu Shen, Xijun Wang, Peng Gao, and Ming Lin. Auxiliary modality learning with generalized curriculum distillation. In International Conference Machine Learning, pages 31057-31076. PMLR, 2023.
[42] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: /908.09090.0000 00355,, 2019.
[43] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 10699, 2019.
[44] Frederick Tung and Greg Mori. similarity-preservinn knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1365-1374, 2019.
[45] Lan Wang, Chenqiang Gao, Luyu Yang, Yue Zhao, Wangmeng Zuo, and Deyu Meng. Pm- gans: Discriminative representation learning for action recognition using partial-modalitee In Proceedings of the European Conference on Computer Vision (ECCV), pages 384-401, 2018.
[46] Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, and Antonio M López. Multimodal end-to-end autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23 (1):537-547, 2020.
[47] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. survey on knowledge distillation of large language models. arXiv preprint 2311:22402.13116, 2024.
[48] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[49] Yang Zhou, Jiuhong Xiao, Yue Zhou and Giuseppe Loianno. Multi-robot collaborative per- ception with graph neural networks. IEEE Robotics and Automation Letters, 7(2):2289-2296, 2022.
[50] Yang Zhou, Long Quang, Carlos Nieto-Granda, and Giuseppe Loianno. Coped-advancing multi-robot collaborative perception: A comprehensive dataset in real-world environments. IEEE Robotics and Automation Letters, 2024.
13

---

A Appendix
A.1 Analysis
We provide an intuitive analysis to justify the effectiveness CAML and illustrate why outperforms single-agent approaches such AML. Please note that this analysis is intended to offer intuitive insights and explanations into the benefits of CAML, rather than serving as a formal mathematical proof, this may be intractable in this context.
We analyze from two key perspectives: Data Coverage and Information Gain. We aim to address the following key questions: (a) Does the collaboration of multiple agents provide complementary information that increases data coverage? Specifically, does combining observations from each agent lead to more accurate and comprehensive prediction compared using single agent? (b) Does the collaboration increase the mutual information between the observations and the true label?
Data Coverage. To address Question (a), we study data coverage and information provided by each agent multi-agent system. Let the entire data space denoted as DD which consists of various subsets. Each agent A: in the system covers subset of this data space: C₁ C 'D. The overall coverage by the system is given by the union of all subsets covered by individual agents: Creutti = , This ensures that [Cmaits] 2 max [C,] If only single agent is available, it can only observe portion of the data space, leaving parts of the space unobserved, which leads to incomplete information for
estimating the true label y. We show a qualitative example of multi-agent collaboration providing complementary information to enhance data coverage in Fig. Appendix A.2.2.
From probabilistic perspective, when multi-agent collaboration is in place, the combined likelihood P(X is modeled as multivariate distribution. This approach provides a broader and more accurate representation of the data space by integrating information from all agents and modeling the dependencies and correlations between them. Compared to univariate distribution P(asly) for single agent A,, the multivariate distribution covers a larger portion of the data space D, thus enhancing data coverage. This allows the exploration of more complex patterns, relationships, and complementary information from different agents. By capturing richer set of interactions and correlations among the agents' observations, the multivariate distribution supports more informed decision-male making. The model's predictions are based on comprehensive view of the environment, thus leading to more accurate outcomes.
Information Gain. To address Question (b) about information gain, we analyze using information theory. Let [(y; x,) represent the mutual information between the true label y and agent A,'s observation z4, which quantifies how much information v, provides about the estimation of y. The joint mutual information between y and the set of all observations X is ( X). In the context of multi-agent collaboration, the joint observations X from multiple agents typically provide more comprehensive information about the true label y compared to the observation of any single agent. Therefore, the mutual information (y; X) is always greater than equal to the mutual information from a single agent: [y;; X) 2 (3y; 2.). We can formally justify this using the chain rule for mutual information:
I(y; X)
[1}; + (1y; x2 x1) .... 133 En I1,..., (I,...,,n-1)) In-1).
Each term (1 F1,..., Tk 2 0) is non-negative because mutual information is always non-negative. It follows that:
T(y; 2 T(,,,,)) for any individual 24.-
2
Adding more observations (e.g., expanding from 2; to X) does not reduce the information about y. Even if some observations are redundant (e.g.. zyj duplicates v.;), the joint mutual information f(y; X) remains at least as large as T(y; a.)). Thus, the combined observations from multi-agent collaboration generally provide more information about y than single observation, improving the overall estimate.
14

---

A.2
Connected Autonomous Driving
A.2.1 Dataset Details
The dataset is generated using the AUTOCASTSIM [38] benchmark, which features three complex and accident-prone traffic scenarios for connected autonomous driving, characterized by limited sensor coverage or obstructed views. These scenarios are realistic and include background traffic of 30 vehicles. They involve challenging interactions such as overtaking, lane changing, and red-light violations, which inherently increase the risk of accidents: (1) Overtaking: A sedan is blocked by a truck on narrow, two-way road with dashed centerline. The truck also obscures the sedan's view of oncoming traffic. The ego vehicle must decide when and how to safely pass the truck. (2) Left Turn: The ego vehicle attempts a left turn at a yield sign. Its view partially blocked by truck waiting in the opposite left-turn lane, reducing visibility of vehicles coming from the opposite direction. (3) Red Light Violation: As the ego vehicle crosses an intersection, another vehicle runs a red light. Due to nearby vehicles waiting to turn left, the ego vehicle's 's sensors unable detect the violator.
Following the setup established by prior works COOPERNAUT [5] and STGN [8], we collect 24 data trails for each scenario, using 12 trails for training and the remaining for testing. Each trail represents unique instantiation of the traffic scenario, with differences arising from the randomized scenario configurations. These variations include different types of vehicles present the scene, varying initial positions and trajectories of collaborative vehicles, difference in traffic flow and interactions, resulting in different visual, LiDAR inputs and control actions for each trail. Each trail contains RGB images and LiDAR point clouds from multiple vehicles, resulting in substantially large dataset with a total size of approximately 400 GB.
Overtaking
Left Turn
Red Light Violation
eludee
Occlud
iC2
C
C2
Acclernd
Ego
Ego
Figure 6: Three accident-prone scenarios in connected autonomous driving benchmark AUTOCAST- SIM: overtaking, left turn, and red light violation.
A.2.2 Data Coverage
We present qualitative example highlighting how multi-agent collaboration provides complementary information to enhance data coverage in Fig. 7. In a red-light violation scenario for connected autonomous driving, as shown in the following figure, the ego vehicle's view obstructed, rendering the occluded vehicle invisible. However, collaborative vehicles are able to detect the occluded vehicle, providing critical complementary information. This additional data helps the ego vehicle overcome its occluded view, enabling it to make more informed decisions and avoid potential collisions with the occluded vehicle.
A.2.3 Behavior Cloning Details
In the CAV task, we apply Behavior Cloning (BC) following prior works COOPERNAUT [5] and STGN [8]. The process involves first running an expert agent to collect data from both the ego vehicle and collaborative vehicles. This data includes RGB images, LiDAR point clouds, and the control actions of the ego vehicle, specifically, braking decisions. Please see more details of the data collection in Appendix A.
15

---

Red Light Violation
Ego view
C1 view
C2 view
ci
Octtided
Ego
Figure 7: Qualitative example of multi-agent collaboration provides complementary information to enhance data coverage.
To integrate BC into our multi-agent setup, we train a model that takes as input the RGB and LiDAR data from both the ego and collaborative vehicles, and it outputs the control actions for the ego vehicle. BC is trained using a cross-entropy loss detect accident-prone cases. The purpose of BC is to let the ego vehicle effectively mimic the expert's ' behavior, allowing it to better assess whether a situation dangerous or not and make informed decisions, such as braking or continuing to drive. This enhances safety in accident- prone environments by ensuring more accurate and context-aware decision-making
A.3 Real-Worle Aerial-Ground Scenarios
A.3.1 Experimental Setup
We resize the input RGB and depth images to 224 x 224. To process RGB and depth data locally for each robot, we use ResNet-18 [14] as the encoder to extract feature maps of size 7 x 7. The RGB features from both robots are shared and fused through channel-wise concatenation, and the depth features are processed similarly. Then we apply x convolution to reduce the fused feature maps to the original channel dimensions for RGB and depth, respectively. We subsequently apply cross-altention fuse the RGB and depth feature maps generate multi-agent multi-modal feature aggregations. These aggregated features are passed through the decoder and upsampled to produce an output map matching the input image size.
A.3.2 Advantages of CAML and Pre-fusion CAML
Both CAML and its variant Pre-fusion CAML have their advantages, CAML fuses the same modalities across different agents, which provides better alignment because it ensures consistency in feature representation. And this approach is particularly beneficial when individual agent views are limited, as CAML effectively leverages diverse viewpoints to provide complementary information, enhancing overall data coverage. On the other hand, Pre-fusion CAML allows agent-specific contextual understanding by fusing different modalities locally within each agent. Furthermore, the system avoids redundant communication between agents by transmitting multi-modal aggregated features rather than modality features separately. CAML can easily shift to Pre-fusion CAML because of the flexibility of our framework, depending on application scenarios.
A.4 Training Complexity
We report the training complexity of AML [41] and CAML for the experiments of collaborative decision-making in CAV, and collaborative semantic segmentation for groiald robots in Table 7 and Table 8, respectively. For experiments, we employ batch size of and the Adam optimizer [18] with an initial learning rate of 1e-3, and Cosine Annealing Scheduler [33] to adjust the learning rate over time. The model trained on an Nvidia RTX 3090 GPU with AMD Ryzen 5900 CPU and 32 GB RAM for 200 epochs.
16

---

Table 7: Training complexity of AML and CAML in collaborative prccision-makingg for connected autonomous driving.
Approach
Parameters
Time/epoch
AMI [41]
19.5M
34s
CAML
39.3M
73s
Table 8: Training complexity of AML and CAML in collaborative semantic segmentation for acrial- ground robots.
Approach
Parameters
Time/epoch
AML [41]
13.5M
3s
CAML
25.5M
7s
A.5 Knowledge Distillation
We begin by training a teacher decision-making model T offline using both RGB and LiDAR data, with a binary cross-entropy loss: LBCE(Y, T) = - [y: log(p:) + (1-9) ys) log(1 p:)]. where 'D is the dataset, ys is the ground truth indicating whether the vehicle should brake, P: is the predicted probability by the teacher model T. The student model S is trained to mimic the behavior of the teacher model while having less modalities. For each data point, the student model receives the same RGB image that the teacher model was given. The loss for the student model is a combination of two terms: the distillation loss using KL divergence between the student output and teacher output (soft targets), and the student task loss, which is the binary cross entropy loss between the student output and the true labels (hard targets). The soft targets from the teacher enrich learning with class similarities, while hard targets ensure alignment with true labels. The soft targets are generated by applying a temperature scaling to the logits. Following prior works [15, 41, 43), the exp (s.ft)
scaled logits are defined as: 2 = (ex/))++exp (R. where 2; is the logit for class and t = 4.0
the temperature. The distillation loss defined as: LKD(S, 7) = - x7 log(2;)) ), where 27 and z/ are the soft target probability from the teacher and student model, respectively. The overall loss for the student model is a weighted sum of the distillation loss and the binary cross-entropy loss: Ls- (1 - (1-a) CCCE (y, S) + at? ( T) with a 0.9 as the weight. After the training of knowledge distillation process, we obtain a student model that uses only RGB data while learning from a teacher model that has access both RGB and LiDAR data. This enables the student model to be effective during testing with only RGB data. Additionally, by leveraging knowledge distillation, the student model benefits from the additional insights provided by the LiDAR data during training, learning more effectively compared to training solely with RGB data.
Training a teacher model and then distilling it into student model allows the student model benefit from the richer information that the teacher model has. Since student models are typically smaller or operate with fewer modalities, they are designed to be more efficient, with fewer parameters or reduced complexity. Deploying such a student model in a resource-constraing especially when the teacher model is too costly to deploy due its computation requirements.
environment is crucial,
A.6 Limitations and Future Work
CAML is capable of handling different and mixed modalities across agents at test time. Feature embeddings are first extracted using modality-specific encoders and then fused across agents (e.g.. via concatenation or cross-attention) for downstream decision-making. In the student model of CAML, the modality configuration can be specified individually for each agent, allowing flexibility without requiring all agents to share the same modalities, Although the current student model does not dynamically vary modalities over time, such capability can be achieved without modifying the architecture. A promising direction is to incorporate a modality dropout strategy during knowledge distillation, where certain modalities are randomly masked or removed for each agent during training. This would enable the student model to better adapt to dynamically changing modality availability. Moreover, CAML can be extended with uncertainty estimation [22, 24] to assess the reliability of each modality and enhance robustness when sensor inputs are noisy or degraded (e.g., under adverse weather conditions in connected autonomous driving).
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: can be found in Section 3 Approach and Section 4 Experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: can be found in Appendix A.6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

We do not have formal theoretical results with solid math proof, though we have an intuitive analysis help explain and understand the advantages of our approach in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: It can be found in Section 4 Experiments and corresponding appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Datasets used in this paper are publicly available. Implementations are detailed in Section Experiments and corresponding appendices.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
ol olycc) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ httplllc/ggidee///e
wwwihbmmmssioooo www.siiiisssoooooioooooco onPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: It can be found in Section 4 and corresponding appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars suitably for the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: can be found in Section 4 and corresponding appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //teuenuuss..// httpp///ceerpp...pppppct///////
Question:
Answer: [Yes]
Justification: The research conducted in this paper is in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper is positive for multi-modal multi-agent systems, and especially ben- eficial for resource-constrained environments where some data modalities may be missing during testing.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators original owners of assets (e.g., code, data, models) used in the paper have been properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: httpaa/:///upps...eee http://erppp..c/ooeeeeeeeeeeeeee / for what should or should not be described.
24


Reviews:
[{"review_id": "I7wJwmjfzh", "reviewer": "Reviewer_yoA6", "summary": "The paper introduces CAML, a framework that integrates auxiliary modality learning with multi-agent collaboration. The core idea is to enable agents to share multi-modal data during training and still function effectively at inference time with reduced modality input. This is particularly useful for real-world applications like connected autonomous vehicles and collaborative robotics, where some modalities may be expensive or unavailable during deployment. CAML leverages knowledge distillation  to transfer information from a full-modality teacher model to a reduced-modality student model.", "questions": "In light of the previously mentioned weaknesses:\n\n- Can the authors include a brief experiment showing how CAML handles more than 2 modalities? and whether it is possible to handle modalities dynamically at test time?\n\n- Can the authors clarify whether CAML learns modality preferences implicitly, and if so, how it would behave if modalities become available intermittently?", "limitations": "- Can the authors add a short discussion in the limitations section about how CAML might behave in cases where all available modalities are uninformative or corrupted (e.g., in bad weather)? Could the system detect such cases or be extended to estimate uncertainty?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper is clearly written and well-structured. The authors effectively motivate the problem and identify key limitations of existing approaches.\n- The proposed architecture is conceptually sound, and its components are well-justified. The support for both centralized and decentralized communication setups is clearly discussed.\n- The authors provide a complexity analysis of the approach in both centralized and decentralized settings, which adds clarity to the framework’s practical implications.\n\nWeaknesses:\n\n- While the framework is well described and motivated, the evaluation could benefit from a broader experimental setup to more thoroughly support the authors’ claims. Specifically, the evaluation relies only on simulated data and scenarios using the AUTOCASTSIM benchmark, which may not fully capture the complexity of real-world environments. Experiments in real-world settings, using datasets such as RCooper [1], would help validate the approach more robustly.\n\n- The setup assumes that modality availability remains static at inference time, and only two modalities (RGB and LiDAR) are considered. This raises several questions: How would the model perform under dynamically varying modalities at test time? For instance, if RGB is intermittently unavailable while LiDAR becomes available, how does the system adapt? Would it prioritize one modality over another, and does the model learn such preferences?\n\n- The evaluation would also benefit from an extension to other tasks, such as collaborative tracking, as this would enable more direct comparisons with additional baselines like CoBEVT [2], F-Cooper[3], and Where2Comm [4].\n\n- Related to the previous points: The paper does not discuss scenarios where all available modalities might be uninformative or corrupted (e.g., in adverse weather conditions). A short discussion on whether the system could support uncertainty estimation in modality effectivenes, or at least detect such failure cases, would be a valuable addition to the limitation section of the paper.\n\nReferences:\n\n[1]  R. Hao et al., \"RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception,\" in CVPR, 2024.\n\n[2] R. Xu et al., \"CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers,\" in CoRL, 2022.\n\n[3] Q. Chen et al., \"F-Cooper: Feature-Based Cooperative Perception for Autonomous Vehicles Using 3D Point Clouds,\" in Proc. ACM/IEEE SEC, 2019.\n\n[4] Y. Hu et al., \"Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps,\" NeurIPS 2022.", "was_revised": true, "final_justification": "The authors provided thorough explanations and additional results that addressed my concerns to a significant extent. While theoretical scalability to many modalities and their informativeness remains unclear, the practical constraints based on benchmarks are acknowledged. I appreciate the clarifications and encourage the authors to include these points and limitations in the manuscript.", "rebuttal": "Thanks for your comments and feedback. We’re glad you found our paper **well-written**, **well-motivated**, and the **architecture sound** with **clear discussion of both communication setups and complexity analysis**. Below we address your concerns and will incorporate all feedback in the final version. We hope you'll consider raising the score. \n\n>Experimental evaluation\n\nWe conduct additional experiments as requested, using three modalities for the CAV task, where each agent receives RGB, LiDAR, and state information (position and velocity) during training. At test time,  only the RGB modality is available for each vehicle. We show the results in the following table, with mean and std over four repeated runs. As we can see, having state information as another modality improves the system performance. This setup further demonstrates CAML’s ability to leverage richer multi-modal data during training while maintaining effective performance under reduced-modality conditions at inference.\n\n|Modalities during training| Overtaking|| Left Turn||Red Light Violation||\n|-|-|-|-|-|-|-|\n|| ADR↑|EIR↑| ADR↑|EIR↑| ADR↑| EIR↑|\n|RGB+LiDAR|$92.8\\pm0.8$|$85.5\\pm0.5$|$67.3\\pm1.0$|$79.7\\pm1.1$|$66.1\\pm1.4$|$83.0\\pm1.3$|\n|RGB+LiDAR+State|$94.0\\pm0.7$|$86.4\\pm0.6$|$69.7\\pm1.2$|$81.7\\pm1.3$|$68.3\\pm1.2$|$84.7\\pm1.0$|\n\nPlease note that our evaluation does not only rely on simulation data and scenarios using the AUTOCASTSIM benchmark, which is used for the experiments of the connected autonomous vehicles (CAV). In addition, we evaluated on aerial-ground robot collaboration with real-world dataset CoPeD for collaborative semantic segmentation. This verifies the effectiveness of our approach in real-world environments.  \n\nThank you for the suggestions to evaluate on tasks such as collaborative tracking and to explore datasets like RCooper. We agree that these directions are valuable and complementary to our work, but incorporating such additional experiments within the very limited rebuttal period is not feasible. Moreover, our collaborative decision-making experiments for CAV actually already implicitly involve aspects of collaborative tracking, as vehicles must first be localized and tracked before making decisions. We appreciate the suggestions and will be sure to include discussions of these potential extensions in the future work section of the final version.\n\n>Modalities \n\nWe conduct additional experiments including state information (position and velocity) as another modality for the CAV tasks, as shown in the above table. Moreover, our work does not only consider RGB and LiDAR modalities. In the aerial-ground robot experiments, we incorporate Depth as a different modality. These evaluations further demonstrate CAML’s flexibility in handling diverse sensory inputs across different tasks and environments.\n\nYes, it is possible to handle modalities dynamically at test time or when modalities become available intermittently. CAML does not learn modality preferences implicitly, it can utilize all modalities available at time time. In the student model of CAML, the modalities for each agent can be configured individually. The agents do not all need to have the same modalities. Although the current student model training does not dynamically change the modalities over time. As noted in the future work section, we can handle dynamic changes without changing the architecture. Our planned solution is to incorporate a modality dropout strategy. During training of knowledge distillation, we can randomly drop out or mask some modality for each agent in the student model. This allows the student model to learn robustness to dynamic modality availability and adapt effectively over time.\n\n>Discussion in limitation section\n\nIn extreme scenarios where all available modalities are uninformative (e.g., due to adverse weather), system performance may degrade. CAML improves robustness by leveraging auxiliary modalities and multi-agent collaboration during training. Additionally, our system can support uncertainty estimation to access modality effectiveness. CAML can be extended to incorporate uncertainty estimation mechanisms to further improve system robustness when available modalities are noisy. This would allow the system to detect and better handle such cases. We will include these discussions into the limitations section of the final version."}, {"review_id": "scWHVo32o5", "reviewer": "Reviewer_eJXK", "summary": "The paper introduces ​CAML, a novel framework for multi-agent systems that leverages multi-modal data during training (e.g., LiDAR + RGB) but operates with reduced modalities (e.g., RGB-only) during inference.", "questions": "1. The knowledge distrillation  has been proposed in AML, so what is the key innovation of this work?\n\n2. How the knowledge is distilled from the teacher model to the student model is not clearly introduced. \n\n3. There have been many collaborative perception frameworks. Can this method work in existing collaborative perception method?", "limitations": "As disscussed in the questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. ​Multi-agent knowledge distillation: A teacher model (full modalities) distills knowledge into a student model (reduced modalities), preserving collaborative insights.\n​\n2. Flexible modality/agent configurations: Supports varying numbers of agents/modalities between training and testing.\n\n​3. Strong empirical results: 58.1% higher accident detection in autonomous driving and 10.6% mIoU gain in aerial-ground robot segmentation versus baselines.\n\nWeakness:\n\n1. Technical innovation: It seems that the paper is just an extension from the single-agent AML method to the multi-agent CAML scenario. \n\n2. How the knowledge is distrillated from teacher model to the student model is not clearly introduced. \n\n3.  The performance difference from AML to CAML is very limited.", "was_revised": true, "final_justification": "4: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.", "rebuttal": "Thanks for your comments and feedback. We are glad that you found our approach **preserving collaborative insights, flexible for modality/agent, with strong empirical results**. Below we answer your questions and will incorporate all feedback to the final version. We hope you'll consider raising the score. \n\n>Key innovation\n\nPlease note that our proposed CAML is not merely an extension of AML. Multi-agent settings introduces substantial complexity, requiring careful consideration of new challenges. CAML specifically addresses the unique challenges of multi-agent systems, including heterogeneity in sensing modalities, dynamic team compositions, and team-level decision-making. CAML has broader applications of connected autonomous driving, collaborative search and rescue, as well as intelligent transportation. To the best of our knowledge, this work is the first to propose a flexible and principled framework capable of tackling a broad range of collaborative tasks in multi-agent systems where limited modalities are available for inference. Unlike prior work that either focuses on multi-agent collaboration but without addressing missing modality at test time, or explores AML ideas solely in single-agent settings, CAML unifies these directions for the first time. \n\n>Knowledge distillation\n\nThe knowledge is distilled from the teacher to the student by having the student mimic the teacher’s behavior. For example, in the CAV experiments, we first train a teacher model offline using a cross-entropy loss, where each vehicle has access to both RGB and LiDAR data. Then, we train a student model that receives only RGB input, learning to replicate the behavior of the teacher model. For each data sample, the student receives the same RGB image as the teacher. The training loss for the student model combines two terms: (1) a distillation loss using KL divergence between the student and teacher outputs, which encourages the student to closely match the teacher’s output, and (2) a task loss using cross-entropy between the student’s prediction and the ground-truth label. We provided these details in Appendix A.5 in paper.\n\n>Performance difference from AML to CAML\n\nPlease note that the performance improvement of CAML over AML is not marginal, but substantial. In collaborative decision-making for CAV, CAML achieves up to a 58.1% improvement in accident detection compared to AML. Additionally, in real-world aerial-ground robot experiments for collaborative semantic segmentation, CAML achieves up to a 10.6% improvement in mIoU over AML. These results highlight the significant advantages of CAML compared to AML.\n\n>Working in existing collaborative perception method\n\nYes, CAML can work with existing collaborative perception methods. It is a general framework that is not limited to specific modalities and can flexibly incorporate different types of sensory inputs. CAML is particularly well-suited for resource-constrained environments where certain modalities may be missing during inference. By enabling effective reduced-modality inference, CAML helps reduce computational cost and improve system robustness during inference, otherwise the system may not work due to missing modality."}, {"review_id": "xA8VYazNlU", "reviewer": "Reviewer_G7GG", "summary": "This paper proposes a multi-agent system named CAML to address the problem of missing modalities at test time. It demonstrates a 58.1% improvement in accident detection performance and further validates the effectiveness of CAML in real-world scenarios.", "questions": "1. The paper evaluates only two modalities. Can the proposed framework generalize to settings with more modalities, and is it capable of handling arbitrary missing modalities at inference time?\n\n2. In autonomous driving scenarios, when the ego vehicle has limited field of view, the V2V system aims to compensate by leveraging visual information from surrounding vehicles. This constitutes information-level missingness rather than modality-level. Can the authors clarify whether their method can handle such information-missing situations?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a collaborative multi-agent system, enabling complementary information exchange between agents, which enhances the robustness of the system.\n\n2. The effectiveness of CAML is validated on real-world data, further demonstrating its practical value for deployment.\n\nWeaknesses:\n\n1. The paper only evaluates the performance under missing LiDAR signals. However, the more informative setting would be to retain LiDAR and remove RGB input, which would better reflect the model’s robustness to missing modalities rather than reliance on a dominant one.\n\n2. The paper does not mention existing work on missing modality problems, such as [1] [2], nor does it discuss the differences between CAML and these methods in terms of design or empirical performance.\n\n[1] Ma M, Ren J, Zhao L, et al. Are multimodal transformers robust to missing modality?[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18177-18186.\n[2] Lee Y L, Tsai Y H, Chiu W C, et al. Multimodal prompting with missing modalities for visual recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14943-14952.", "was_revised": true, "final_justification": "The experimental results in rebuttal address my concerns to a large extent.", "rebuttal": "Thanks for your comments and feedback. We are pleased that you found our collaborative multi-agent design **effective in enabling complementary information exchange** and **enhancing system robustness**. We are also encouraged that you recognized the **practical value of CAML for deployment**, as **validated on real-world data**. Below, We address your concerns and will incorporate all feedback into the final version. We hope you'll consider raising the score.\n\n>Additional experiments of retaining LiDAR and removing RGB\n\nWe conduct additional experiments by retaining LiDAR and removing RGB during testing for the experiments of collaborative decision-making for CAV. We present the results in the following table, with mean and std over four repeated runs. The performances of retaining RGB or LiDAR are similar but retaining LiDAR shows slightly better performance as LiDAR provides more precise spatial localization which is better for accident detection. \n\n|Modality during testing| Overtaking|| Left Turn||Red Light Violation||\n|-|-|-|-|-|-|-|\n|| ADR↑|EIR↑| ADR↑|EIR↑| ADR↑| EIR↑|\n|RGB|$92.8\\pm0.8$|$85.5\\pm0.5$|$67.3\\pm1.0$|$79.7\\pm1.1$|$66.1\\pm1.4$|$83.0\\pm1.3$|\n|LiDAR|$93.3\\pm0.9$|$86.1\\pm0.6$|$68.0\\pm0.9$|$80.4\\pm1.3$|$67.2\\pm1.2$|$83.8\\pm1.4$|\n\n>Modalities\n\nYes, the proposed framework can generalize to settings with more modalities, and it is capable of handling arbitrary missing modalities at inference time. Please note that we did not evaluate only two modalities, in the experiments of CAV, we evaluated RGB and LiDAR, in the experiments of aerial-ground robot collaboration, we also evaluated RGB and Depth. \n\nIn addition, we conduct additional experiments, using three modalities for the CAV task, where each agent receives RGB, LiDAR, and state information (position and velocity) during training. At test time, only the RGB modality is available for each vehicle. As shown below, incorporating state information during training improves the system performance.\n\n|Modalities during training| Overtaking|| Left Turn||Red Light Violation||\n|-|-|-|-|-|-|-|\n|| ADR↑|EIR↑| ADR↑|EIR↑| ADR↑| EIR↑|\n|RGB+LiDAR|$92.8\\pm0.8$|$85.5\\pm0.5$|$67.3\\pm1.0$|$79.7\\pm1.1$|$66.1\\pm1.4$|$83.0\\pm1.3$|\n|RGB+LiDAR+State|$94.0\\pm0.7$|$86.4\\pm0.6$|$69.7\\pm1.2$|$81.7\\pm1.3$|$68.3\\pm1.2$|$84.7\\pm1.0$|\n\n>Handling information-missing situations\n\nYes, CAML can handle such information-missing situations. It is not only a multi-modal system but also a multi-agent framework. From the multi-modal perspective, CAML addresses missing modalities at the modality level. From the multi-agent collaboration perspective, CAML explicitly handles information-level missingness, such as when the ego vehicle has a limited field of view, by leveraging complementary information from surrounding vehicles through V2V communication. This enables the system to compensate for partial observations by integrating diverse views across agents, thereby enhancing perception and decision-making capabilities.\n\n>Discussion on related work\n\nThanks for the suggestion. CAML differs from the two papers in following: \n\n1. CAML leverages multi-agent collaboration and knowledge distillation to address both missing modality and partial agent coverage, which is not explored by the two referenced single-agent works. \n\n2. The other two works focus on improving single-agent robustness to missing sensor inputs, either through transformer fusion strategies or prompting, without multi-agent collaboration or cross-agent information transfer. \n\nWe will be sure to discuss these related works in the camera-ready version."}, {"review_id": "enxuZ7NCCq", "reviewer": "Reviewer_fUqC", "summary": "In this paper, the authors introduce Collaborative Auxiliary Modality Learning (CAML), a framework that allows agents to collaborate and share reduced modalities during inference. The proposed framework is trained using a knowledge distillation approach. The experimental results on collaborative decision-making and semantic segmentation tasks demonstrate the effectiveness of CAML.", "questions": "Is CAML an intermediate cooperation approach? If so, what are the key differences between it and other RGB-based intermediate cooperation methods? It appears that the primary distinction lies in the enhancement of RGB representation through knowledge distillation.\n\nRegarding the training complexity of pre-fusion CAML, how does it compare to other methods? Is it a more favorable option that achieves a balance between training complexity and inference performance?\n\nFurthermore, how does CAML's performance compare to that of late cooperation (result-level cooperation), which is also robust to modality missing?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The proposed CAML extends Auxiliary Modality Learning (AML) to multi-agent cooperative scenarios, enhancing the robustness of multi-agent systems to missing modalities. The proposed approach is technically sound. The performance improvements highlight the effectiveness of CAML.\n\nHowever, CAML is essentially a simple combination of AML and a cooperative system, and its additional contributions are somewhat limited. The agents in the framework are restricted to the RGB modality, and dynamic modality dropout is not yet supported. This limitation prevents CAML from effectively addressing the challenges posed by dynamic environments (claimed by the authors), such as those encountered in Connected and Autonomous Vehicles (CAV). Technically, CAML can be seen as a cooperative framework with an enhanced RGB representation. The distinctions between CAML and general cooperative paradigms should be clearly articulated. Moreover, a more comprehensive comparison with existing cooperative approaches is needed to better demonstrate its performance advantages.", "was_revised": true, "final_justification": "The experimental results address my concerns to a large extent. I have carefully read the authors' response as well as the discussion with other reviewers. I have decided to change the score to relative positive.", "rebuttal": "Thanks for your comments and feedback. We are glad that you found our paper **technically sound** with **effective performance improvements**. We address your concerns below and will incorporate all feedback into the final version. We hope you could consider raising the score.\n\n>Contributions of CAML\n\nPlease note that CAML is not simple combination of AML and cooperative system. Multi-agent settings introduces substantial complexity, requiring careful consideration of new challenges. CAML specifically addresses the unique challenges of multi-agent systems, including heterogeneity in sensing modalities, dynamic team compositions, and team-level decision-making. CAML has broader applications of connected autonomous driving, collaborative search and rescue, as well as intelligent transportation. To the best of our knowledge, this work is the first to propose a flexible and principled framework capable of tackling a broad range of collaborative tasks in multi-agent systems where limited modalities are available for inference. Unlike prior work that either focuses on multi-agent collaboration but without addressing missing modality at test time, or explores AML ideas solely in single-agent settings, CAML unifies these directions for the first time. \n\nThe agents in the framework are not restricted to the RGB modality. CAML is a general framework and can work with any modality. In the experiments of CAV and aerial-ground robot collaboration, we use RGB as the available modality during inference because it is the most commonly used and widely available in robotics and autonomous driving. \n\nAdditionally, we conduct new experiments where LiDAR is retained while RGB is removed during testing, please see the results in the following table, with mean and std over four repeated runs. The performances of retaining RGB or LiDAR are similar but retaining LiDAR shows slightly better performance as LiDAR provides more precise spatial localization which is better for accident detection.\n\n|Modality during testing| Overtaking|| Left Turn||Red Light Violation||\n|-|-|-|-|-|-|-|\n|| ADR↑|EIR↑| ADR↑|EIR↑| ADR↑| EIR↑|\n|RGB|$92.8\\pm0.8$|$85.5\\pm0.5$|$67.3\\pm1.0$|$79.7\\pm1.1$|$66.1\\pm1.4$|$83.0\\pm1.3$|\n|LiDAR|$93.3\\pm0.9$|$86.1\\pm0.6$|$68.0\\pm0.9$|$80.4\\pm1.3$|$67.2\\pm1.2$|$83.8\\pm1.4$|\n\n>Performance comparison to late cooperation\n\nWe conduct additional experiments using a late cooperation strategy for the CAV tasks. For each vehicle and each modality (RGB, LiDAR), we compute its decision output (action logits). Specifically, we use the same encoders as in CAML, ResNet-18 for RGB and PointTransformer for LiDAR, to extract features independently from each modality. These features are then passed through MLP prediction heads to generate action logits. To perform late cooperation, we first fuse the logits of the same modality from connected vehicles to the ego vehicle by averaging. Next, we fuse the averaged logits across modalities by taking their mean. The resulting fused multi-modal, multi-agent logits are used to determine the ego vehicle’s control actions for accident detection. We then apply the same knowledge distillation procedure as CAML, distilling from the RGB-LiDAR teacher model to a student model that only uses RGB at test time. The results are presented in the following table. As shown, CAML outperforms the late cooperation approach, as it enables the model to learn richer and more complementary representations across agents and modalities. In contrast, late cooperation only fuses output decisions, which may lead to information loss.\n\n|Approach| Overtaking|| Left Turn||Red Light Violation||\n|-|-|-|-|-|-|-|\n|| ADR↑|EIR↑| ADR↑|EIR↑| ADR↑| EIR↑|\n|CAML|$92.8\\pm0.8$|$85.5\\pm0.5$|$67.3\\pm1.0$|$79.7\\pm1.1$|$66.1\\pm1.4$|$83.0\\pm1.3$|\n|Late Cooperation|$88.0\\pm1.0$|$83.5\\pm1.0$|$60.2\\pm1.3$|$78.5\\pm0.9$|$58.2\\pm1.5$|$81.7\\pm1.5$|\n\n>Difference between other RGB-based intermediate cooperation methods\n\nCAML is an intermediate cooperation approach. It has following advantages compared to existing RGB-based intermediate cooperation methods.\n\n1. **Robustness under modality reduction**: Unlike typical RGB-based cooperation methods that assume consistent modality availability during both training and inference, CAML is explicitly designed to handle modality reduction at test time. It enables robust inference even when high-cost or unreliable modalities (e.g., LiDAR) are unavailable during deployment.\n\n2. **Flexibility to agent and modality heterogeneity**: CAML is a general framework that supports a varying number of agents and heterogeneous modality availability per agent. This makes it particularly suited for resource-constrained environments where sensors may fail, communication may be intermittent, or hardware capabilities differ across agents.\n\n3. **Cross-agent, cross-modal knowledge distillation**: CAML goes beyond traditional distillation by performing cross-agent, cross-modal knowledge transfer. It enhances the RGB representation by leveraging auxiliary modalities and collaborative knowledge shared among agents, leading to improved performance even in RGB-only inference settings.\n\n>Training complexity of pre-fusion CAML\n\nIn terms of training complexity, Pre-fusion CAML is very similar to that of CAML, since both share a similar overall network architecture and use same modality-specific encoders. We illustrate in the table below with the collaborative semantic segmentation task.\n\n|Approach|Parameters|Time/epoch|\n|-|-|-|\n|AML|13.5M|3s|\n|CAML|25.5M|7s|\n|Pre-fusion CAML|25.7M|7s|\n\nBoth CAML and its variant Pre-fusion CAML have their advantages, the preferable choice depends on the specific task requirements. If cross-agent modality alignment and consistency in feature representation are critical, CAML is more suitable. If preserving agent-specific contextual understanding and reducing inter-agent communication are priorities, Pre-fusion CAML may be preferred."}]

Author rebuttal:

Meta-review: This paper considers the missing-modality problem for multi-modal collaborative multi-agent systems. The approach centers on a knowledge distillation method, where a full-modality teacher model is distilled into a reduced-modality student model, extending auxiliary modality learning methods. Strong results are shown on challenging accident-prone scenarios where agents can have limited visibility. The reviewers found the paper well-motivated and written, and appreciated the strong performance on real-world data and ablation/complexity analysis. Several concerns regarding clarification of the contribution (especially compared to AML), having more thorough sets of modality experiments (e.g. no RGB), and clarification of the training complexity. The authors provided a strong rebuttal, including with new results answering some of these questions (results of additional modalities, training complexity results, etc.), and all reviewers were satisfied with the paper. I recommend acceptance, and for the authors to incorporate the clarifications and results in the final paper.
Decision: Accept (poster)