Title: CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Modal Learning

Multi-Agent Systems
Abstract: Multi-modal learning has emerged as a key technique for improving performance across domains such as autonomous driving, robotics, and reasoning. However, in certain scenarios, particularly in resource-constrained environments, some modalities available during training may be absent during inference. While existing frameworks effectively utilize multiple data sources during training and enable inference with reduced modalities, they are primarily designed for single-agent settings. This poses a critical limitation in dynamic environments such as connected autonomous vehicles (CAV), where incomplete data coverage can lead to decision-making blind spots. Conversely, some works explore multi-agent collaboration but without addressing missing modality at test time. To overcome these limitations, we propose Collaborative Auxiliary Modality Learning (CAML), a novel multi-modal multi-agent framework that enables agents to collaborate and share multi-modal data during training, while allowing inference with reduced modalities during testing. Experimental results in collaborative decision-making for CAV in accident-prone scenarios demonstrate that CAML achieves up to a 58.1% improvement in accident detection. Additionally, we validate CAML on real-world aerial-ground robot data for collaborative semantic segmentation, achieving up to a 10.6% improvement in mIoU.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce Collaborative Auxiliary Modality Learning (CAML), a framework that allows agents to collaborate and share reduced modalities during inference. The proposed framework is trained using a knowledge distillation approach. The experimental results on collaborative decision-making and semantic segmentation tasks demonstrate the effectiveness of CAML.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed CAML extends Auxiliary Modality Learning (AML) to multi-agent cooperative scenarios, enhancing the robustness of multi-agent systems to missing modalities. The proposed approach is technically sound. The performance improvements highlight the effectiveness of CAML.\n\nHowever, CAML is essentially a simple combination of AML and a cooperative system, and its additional contributions are somewhat limited. The agents in the framework are restricted to the RGB modality, and dynamic modality dropout is not yet supported. This limitation prevents CAML from effectively addressing the challenges posed by dynamic environments (claimed by the authors), such as those encountered in Connected and Autonomous Vehicles (CAV). Technically, CAML can be seen as a cooperative framework with an enhanced RGB representation. The distinctions between CAML and general cooperative paradigms should be clearly articulated. Moreover, a more comprehensive comparison with existing cooperative approaches is needed to better demonstrate its performance advantages.\n\nQUESTIONS:\nIs CAML an intermediate cooperation approach? If so, what are the key differences between it and other RGB-based intermediate cooperation methods? It appears that the primary distinction lies in the enhancement of RGB representation through knowledge distillation.\n\nRegarding the training complexity of pre-fusion CAML, how does it compare to other methods? Is it a more favorable option that achieves a balance between training complexity and inference performance?\n\nFurthermore, how does CAML's performance compare to that of late cooperation (result-level cooperation), which is also robust to modality missing?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multi-agent system named CAML to address the problem of missing modalities at test time. It demonstrates a 58.1% improvement in accident detection performance and further validates the effectiveness of CAML in real-world scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper presents a collaborative multi-agent system, enabling complementary information exchange between agents, which enhances the robustness of the system.\n\n2. The effectiveness of CAML is validated on real-world data, further demonstrating its practical value for deployment.\n\nWeaknesses:\n\n1. The paper only evaluates the performance under missing LiDAR signals. However, the more informative setting would be to retain LiDAR and remove RGB input, which would better reflect the model’s robustness to missing modalities rather than reliance on a dominant one.\n\n2. The paper does not mention existing work on missing modality problems, such as [1] [2], nor does it discuss the differences between CAML and these methods in terms of design or empirical performance.\n\n[1] Ma M, Ren J, Zhao L, et al. Are multimodal transformers robust to missing modality?[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18177-18186.\n[2] Lee Y L, Tsai Y H, Chiu W C, et al. Multimodal prompting with missing modalities for visual recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14943-14952.\n\nQUESTIONS:\n1. The paper evaluates only two modalities. Can the proposed framework generalize to settings with more modalities, and is it capable of handling arbitrary missing modalities at inference time?\n\n2. In autonomous driving scenarios, when the ego vehicle has limited field of view, the V2V system aims to compensate by leveraging visual information from surrounding vehicles. This constitutes information-level missingness rather than modality-level. Can the authors clarify whether their method can handle such information-missing situations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

comments: SUMMARY:
The paper introduces ​CAML, a novel framework for multi-agent systems that leverages multi-modal data during training (e.g., LiDAR + RGB) but operates with reduced modalities (e.g., RGB-only) during inference.

STRENGTHS AND WEAKNESSES:
Strength:

1. ​Multi-agent knowledge distillation: A teacher model (full modalities) distills knowledge into a student model (reduced modalities), preserving collaborative insights.
​
2. Flexible modality/agent configurations: Supports varying numbers of agents/modalities between training and testing.

​3. Strong empirical results: 58.1% higher accident detection in autonomous driving and 10.6% mIoU gain in aerial-ground robot segmentation versus baselines.

Weakness:

1. Technical innovation: It seems that the paper is just an extension from the single-agent AML method to the multi-agent CAML scenario. 

2. How the knowledge is distrillated from teacher model to the student model is not clearly introduced. 

3.  The performance difference from AML to CAML is very limited.

QUESTIONS:
1. The knowledge distrillation  has been proposed in AML, so what is the key innovation of this work?

2. How the knowledge is distilled from the teacher model to the student model is not clearly introduced. 

3. There have been many collaborative perception frameworks. Can this method work in existing collaborative perception method?

LIMITATIONS:
As disscussed in the questions.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CAML, a framework that integrates auxiliary modality learning with multi-agent collaboration. The core idea is to enable agents to share multi-modal data during training and still function effectively at inference time with reduced modality input. This is particularly useful for real-world applications like connected autonomous vehicles and collaborative robotics, where some modalities may be expensive or unavailable during deployment. CAML leverages knowledge distillation  to transfer information from a full-modality teacher model to a reduced-modality student model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is clearly written and well-structured. The authors effectively motivate the problem and identify key limitations of existing approaches.\n- The proposed architecture is conceptually sound, and its components are well-justified. The support for both centralized and decentralized communication setups is clearly discussed.\n- The authors provide a complexity analysis of the approach in both centralized and decentralized settings, which adds clarity to the framework’s practical implications.\n\nWeaknesses:\n\n- While the framework is well described and motivated, the evaluation could benefit from a broader experimental setup to more thoroughly support the authors’ claims. Specifically, the evaluation relies only on simulated data and scenarios using the AUTOCASTSIM benchmark, which may not fully capture the complexity of real-world environments. Experiments in real-world settings, using datasets such as RCooper [1], would help validate the approach more robustly.\n\n- The setup assumes that modality availability remains static at inference time, and only two modalities (RGB and LiDAR) are considered. This raises several questions: How would the model perform under dynamically varying modalities at test time? For instance, if RGB is intermittently unavailable while LiDAR becomes available, how does the system adapt? Would it prioritize one modality over another, and does the model learn such preferences?\n\n- The evaluation would also benefit from an extension to other tasks, such as collaborative tracking, as this would enable more direct comparisons with additional baselines like CoBEVT [2], F-Cooper[3], and Where2Comm [4].\n\n- Related to the previous points: The paper does not discuss scenarios where all available modalities might be uninformative or corrupted (e.g., in adverse weather conditions). A short discussion on whether the system could support uncertainty estimation in modality effectivenes, or at least detect such failure cases, would be a valuable addition to the limitation section of the paper.\n\nReferences:\n\n[1]  R. Hao et al., \"RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception,\" in CVPR, 2024.\n\n[2] R. Xu et al., \"CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers,\" in CoRL, 2022.\n\n[3] Q. Chen et al., \"F-Cooper: Feature-Based Cooperative Perception for Autonomous Vehicles Using 3D Point Clouds,\" in Proc. ACM/IEEE SEC, 2019.\n\n[4] Y. Hu et al., \"Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps,\" NeurIPS 2022.\n\nQUESTIONS:\nIn light of the previously mentioned weaknesses:\n\n- Can the authors include a brief experiment showing how CAML handles more than 2 modalities? and whether it is possible to handle modalities dynamically at test time?\n\n- Can the authors clarify whether CAML learns modality preferences implicitly, and if so, how it would behave if modalities become available intermittently?\n\nLIMITATIONS:\n- Can the authors add a short discussion in the limitations section about how CAML might behave in cases where all available modalities are uninformative or corrupted (e.g., in bad weather)? Could the system detect such cases or be extended to estimate uncertainty?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}