CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

Authors: Rui Liu, Yu Shen, Peng Gao, Ming Lin
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Collaborative Auxiliary Modality Learning, Multi-Agent Collaboration, Cross-Modality Knowledge Distillation

Abstract:
Multi-modality learning has become a crucial technique in enhancing the performance of machine learning applications across various domains, including autonomous driving, robotics, and perception systems. Existing frameworks, such as Auxiliary Modality Learning (AML), effectively utilize multiple data sources during training and enable inference with reduced modalities, but they primarily operate in a single-agent context. This limitation is particularly critical in dynamic environments, such as connected autonomous vehicles (CAV), where incomplete data coverage can result in decision-making blind spots. To address these challenges, we introduce Collaborative Auxiliary Modality Learning ($\textbf{CAML}$), a novel extension of the AML framework for multi-agent systems. $\textbf{CAML}$ facilitates collaboration among agents by allowing them to share multimodal data during training. During inference, each agent operates effectively with fewer modalities, ensuring robustness in performance even with missing data. We analyze the effectiveness of $\textbf{CAML}$ from the perspective of uncertainty reduction and data coverage, providing a theoretical support to understand and explain why $\textbf{CAML}$ works better than AML. We then validate $\textbf{CAML}$ through experiments in collaborative decision-making for CAV in accident-prone scenarios. Experimental results show that $\textbf{CAML}$ outperforms AML across all tested scenarios, achieving up to a ${\bf 58.3}$% improvement in accident detection. Additionally, we validate our approach on real-world data from aerial-ground vehicles for collaborative semantic segmentation, achieving up to ${\bf 10.8}$% improvement in mIoU compared to AML.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729488430102

thread_type:
official_review

<thread_object>
creator: reviewer_fwch

<content>
summary:
This paper introduces Collaborative Auxiliary Modality Learning (CAML), a framework that extends Auxiliary Modality Learning (AML) to multi-agent systems. CAML allows agents to collaborate and share multimodal data during training, while enabling efficient, reduced-modality inference during testing. The authors provide theoretical analysis of CAML's effectiveness from the perspectives of uncertainty reduction and data coverage enhancement, explaining why CAML outperforms AML. The framework is validated through experiments on collaborative decision-making tasks for connected autonomous vehicles in accident-prone scenarios. The paper shows CAML's modality-efficient superiority and generalizability. The paper positions CAML as a unified approach that leverages multi-agent collaboration to improve robustness and accuracy in predictions, while reducing computational costs and data requirements at test time. This has potential applications in various domains where multi-agent collaboration and multimodal learning are crucial.

strengths:
The paper demonstrates several notable strengths across the dimensions of originality, quality, clarity, and significance: 

  

**Originality:** CAML creatively combines ideas from knowledge distillation, multi-agent systems, and multi-modal learning, resulting in a novel approach to handling complex, real-world scenarios. 

  

**Quality:**The paper provides a solid theoretical analysis of CAML's effectiveness, examining it from the perspectives of uncertainty reduction and data coverage enhancement. This adds depth and credibility to the proposed approach. 

  

**Clarity:** Well-structured: The paper follows a logical flow, clearly presenting the problem, methodology, theoretical analysis, and experimental results. 

   

**Significance:** CAML's ability to perform well with reduced modalities during testing offers potential computational and resource efficiency benefits in deployed systems. While focused on autonomous driving, the framework's principles could be applied to other multi-agent, multi-modal learning scenarios, such as distributed sensing systems. 

   

In summary, the paper presents an interesting approach to a problem in multi-agent systems, supported by thorough theoretical analysis and experimental validation.

weaknesses:
Here are the key areas where the paper could be strengthened: 

1. Modalities in Experimentation: It’s unclear which modalities are used for the experiments. While some sections mention depth, RGB, and LIDAR, the modalities should be clearly stated in Section 5.1 (Data Collection). Clarifying the exact modalities would provide better insight into the data used. 

2. Overly Simplistic Driving Scenarios: The driving scenarios seem too basic. Including more complex environments with domain randomization (e.g., varying numbers of pedestrians and vehicles) would enrich the evaluation. Results from real-world scenarios would also be valuable to demonstrate generalization. 

3. Limited Analysis of Failure Cases: The paper focuses on scenarios where CAML performs well, but lacks an exploration of its weaknesses or failure cases. To strengthen the paper, include an analysis of these cases and discuss possible improvements or limitations of the approach. 

4. Insufficient Training Details: More details about the training setup would improve reproducibility. It would be useful to specify the GPU type, training time, and hardware resources used for training. Providing these details ensures transparency and supports future replication. 

Typos: 

1. L377: “CAML ." -> “CAML.”  There is a space between “L” and “.” 

By addressing these points, the paper can offer a more thorough evaluation of CAML and broaden its contributions.

questions:
Questions: 

1. Clarification on Modalities: Can you clarify the exact modalities used in the experiments? Section 5.1 (Data Collection) does not specify this clearly, and it is inferred that depth, RGB, and LIDAR were utilized from other sections. Are these the only modalities involved, or are there others? 

2. Complexity of Driving Scenarios: How realistic were the driving scenarios used in the experiments? Did you incorporate domain randomization, and how many pedestrians or vehicles were simulated? Additionally, are there plans to test the approach in real-world settings? 

3. Failure Case Analysis: What are the specific failure cases where CAML underperforms? Including more analysis or examples of these cases would provide a clearer understanding of the framework’s limitations and potential areas for improvement. 

4. Training Setup: Can you provide more details about the training setup, such as the GPU types, number of hours required for training, and hardware resources used? This would help improve reproducibility and enable more accurate comparisons in future work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690468447

thread_type:
official_review

<thread_object>
creator: reviewer_97b9

<content>
summary:
This paper attempts to address the Auxiliary Modality Learning problem by means of a multi-agent system, where models are permitted to employ additional modalities during training while enabling inference with fewer or even a single modality.

strengths:
1. The topic of  Auxiliary Modality Learning is interesting and practical. It provides solutions to deal with scenarios where limited modalities are available for inference, making it applicable in real-world situations with constrained data. 

2. The author's attempt to address AML with a multi-agent system is interesting. The use of a multi-agent system in this context is an innovative concept that can lead to more efficient and effective learning processes.

3. The analysis part puts forward three interesting questions. These questions cover important aspects of agent collaboration and can lead to a deeper understanding and evaluation of the related mechanisms and effects.

weaknesses:
1. Many experimental settings are not described clearly. For instance, is there a correlation between the number of agents and the number of modalities? The number of agents appears not to be explicitly stated in the experiment. Additionally, I am curious about how the experiment is devised when the number of modalities exceeds 3, and what the outcome would be.

2. Accurate numbers should preferably be marked on the histograms in the experiment for convenient viewing. For example, in Figure 4(a), it is difficult to tell the highs and lows. Meanwhile, I'm also curious about why while STGN uses both RGB and depth data during testing, CAML relies solely on RGB, yet achieves comparable, or even better performance.

3. There should be a comparative analysis of training complexity. Multiple agents inevitably increase the parameters of the network. Is the source of the performance improvement due to the increase in network parameters?

4. Discussions on limitations are necessary as there is no description of the limitations of the proposed method.

questions:
Please see weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691723067

thread_type:
official_review

<thread_object>
creator: reviewer_8jqu

<content>
summary:
The paper aims to generalize Auxiliary Modality Learning (AML) to the context of cooperative perception. AML benefits machine learning by enabling inferencing on reduced modality space. The paper validates its proposed CAML method from the perspective of effective uncertainty reduction and data coverage to justify its superiority over AML. Empirical experiments on accident detection further support the conclusion.

strengths:
The paper is well-organized, with a clear objective and statement of its design logic. The experiment design comparing the proposed CAML with its baseline AML effectively showcased the benefits of extending to a cooperative context well.

weaknesses:
1. This paper's main contribution is an extension of an existing maturely developed machine learning technique, AML, to the context of cooperative perception, which limits its scope of novelty. Despite experiments conducted by the authors to investigate the benefits of CAML compared to single-agent AML, they only investigate one of the plausible paradigms for applying the AML - ego-only paradigm. Since we are looking into a cooperative perception task, will it be more sensical to use AML for training the encoder for single vehicles in parallel before fusion (**pre-fusion**) and compare this application with the proposed CAML, which is essentially applying a joint-version of AML to train the fusion module (**on-fusion**)?
2. The authors have cited several related works to demonstrate the benefits of applying AML to a perception system. However, it would be much better if the authors could present a comparative study of CAML, AML, and other cooperative perception methods on traditional cooperative perception tasks, such as objective detection, to showcase the need for applying AML in cooperative perception.

questions:
Problems have been raised in the discussion about the weakness of this paper. Here are a few of them in short:
1. Can you justify the need to use AML for cooperative perception in general? If it was beneficial, why had no one before this paper explored using it for more traditional tasks such as cooperative object detection?
2. What is CAML's time and space complexity concerning the number of connected agents? From your description, I presume it is quadratic. Would this trade-off in extra time and space complexity be worth it?
3. Can you compare applying AML on training the entire fusion module to using it only for the encoder and decoder on single vehicles (pre- or post-fusion)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733435507

thread_type:
official_review

<thread_object>
creator: reviewer_bryx

<content>
summary:
This paper extends Auxiliary Modality Learning (AML) for multi-agent systems, resulting in Collaborative Auxiliary Modality Learning (CAML). Then each single-agent can maintain robustness robustness even with missing modalities during inference through multi-agent information sharing. Analysis about uncertainty reduction, and data coverage provides a theoretical support to understand and explain why CAML works better than AML. And the CAML system is validated in the application of decision-making for CAV in accident-prone scenarios, demonstrating the effectiveness of the proposed CAML.

strengths:
1. The writing is easy to follow.  
2. The motivation and proposed method are clearly stated.  
3. This work represents a novel effort to integrate multi-agent systems into Auxiliary Modality Learning.  
4. The experiments demonstrate the effectiveness of CAML compared to previous baselines by incorporating multi-agent collaboration.  
5. It is promising that CAML can achieve comparable performance with RGBD using only RGB modality, and the benefits of multi-agent training can extend to single-agent testing scenarios.

weaknesses:
1. Lack of ablation studies on a wider range of modalities. For instance, what happens when training involves both LiDAR and RGB modalities, while testing uses only RGB or LiDAR? Would the performance still improve when the modality gap is significant?
2. Lack of validation on real-world benchmarks. Validating real-world data is crucial, especially for applications in connected autonomous vehicles. Numerous real-world benchmarks exist, such as DAIR-V2X, V2V4Real, and TUMTraf, making real-world validation essential.
3. Lack of quantitative analysis regarding uncertainty reduction and data coverage. Additionally, providing qualitative visualization examples would enhance the presentation.

questions:
1. If different modality embeddings are combined through concatenation, can the system only accommodate a fixed modality?  
2. What about heterogeneous scenarios where agents possess different modalities, such as in an RGBD context, including single agents with either RGB or depth data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732411456382

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Area Chairs and Reviewers,

We thank all the reviewers for their detailed and helpful comments on our work. We appreciate the reviewers for acknowledging our strengths and contributions, such as
- a **novel and interesting approach** to integrate multi-agent systems into Auxiliary Modality Learning (bryx, 97B9, fWcH);
- **practical solutions** (97B9) to deal with real-world scenarios where limited modalities are available for inference;
- **significance** (fWcH) of the proposed approach being applied to other multi-agent, multi-modal learning scenarios; 
- **interesting and solid theoretical analysis** which leads to a deeper understanding and adds depth and credibility to the proposed approach (97B9, fWcH);
- **clear motivation, method and logic** (bryx, 8JqU, fWcH); 
- experiments demonstrating the **effectiveness** of CAML (bryx, 8JqU); and 
- **well written and organized** (bryx, 8JqU, fWcH).

Below we address the concerns raised by the Reviewers.  We include more requested experiments with real-world data and more ablation studies.  We also provide more discussions on limitations and failure case analysis, more experimental and training details, and add clarity and depth to address any ambiguities. Our responses to specific concerns are detailed below. We thank you all for the opportunity to improve our work with your constructive feedback.

Best regards,  
The Authors
___
Here we refer to some general questions:

**Question 1 (bryx, 8JqU, fWcH): Experiments on real-world data and other collaborative perception tasks.**

**Experimental Setup.** We conducted additional experiments with real-world data from aerial-ground vehicles for collaborative semantic segmentation. We use the dataset CoPeD (Zhou et al., 2024), with one aerial vehicle and one ground vehicle, in two different real-world scenarios of the indoor NYUARPL and the outdoor HOUSEA. For more details about the dataset, please refer to Zhou et al. (2024). Additionally, we introduce noise to the RGBD data collected by the ground vehicle. For both aerial and ground vehicles, RGB and depth data are used during training, while only RGB data is used during testing in CAML.

We adopt the FCN (Long et al., 2015) architecture as the backbone for semantic segmentation. We resize the input RGB and depth images to $224\times224$. To process RGB and depth data locally for each vehicle, we use ResNet-18 as the encoder to extract feature maps of size $7\times7$. The RGB features from both vehicles are shared and fused through channel-wise concatenation, and the depth features are processed similarly. Then we apply $1\times1$ convolution to reduce the fused feature maps to the original channel dimensions for RGB and depth, respectively. We subsequently apply cross-attention to fuse the RGB and depth feature maps to generate multi-agent multi-modal feature aggregations. These aggregated features are passed through the decoder and upsampled to produce an output map matching the input image size. 

**Experimental Results.**
We evaluate performance using the Mean Intersection over Union (mIoU) metric, which quantifies the average overlap between predicted segmentation outputs and ground truth across all classes. We compare the performance of CAML with AML and FCN. In the AML approach, only the ground vehicle operates, with RGB and depth data available during training but only RGB data used for testing. The FCN approach involves only the ground vehicle operating with RGB data for both training and testing.

We present the experimental results in the following Table. CAML demonstrates superior performance in terms of mIoU across both indoor and outdoor environments. Specifically, CAML achieves an improvement of mIoU for 7.4% in indoor scenario and 10.8% in outdoor scenario compared to AML. We also present the qualitative results in **Fig. 7 in the Appendix**. Despite the noisy input image from the ground vehicle, CAML produces predictions that are closest to the ground truth. This performance improvement can be attributed to CAML's multi-agent collaboration, which provides complementary information to enhance data coverage and offers a more comprehensive understanding of the scenes. Additionally, the utilization of auxiliary depth data during training results in more precise segmentation outputs.

**Table 1**: Experimental results of semantic segmentation on real-world dataset CoPeD using aerial-ground vehicles in indoor and outdoor environments. CAML achieves the highest mIoU in both environments.

| **Approach** | **Indoor mIoU (%)** | **Outdoor mIoU (%)** |
|--------------|---------------------|----------------------|
| FCN          | 51.20               | 56.22                |
| AML          | 55.89               | 60.32                |
| CAML         | 60.05               | 66.83                |
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary: This paper generalizes Auxiliary Modality Learning (AML) to the task of cooperative perception in the context of connected and automated vehicles.
(b) Strengths: The paper is generally well-written and easy to follow. The proposed approach seems reasonable. The experimental results seem to support the authors' claims to some extent.
(c) Weaknesses: The reviewers pointed out a few concerns and issues of the paper, such as lack of baseline comparisons, lack of technical details and explanations, lack of discussions on limitations, and some claims are not fully convincing.
(d) The authors' rebuttal addressed some of the major concerns in clarity. However, there still remain major issues unresolved after the rebuttal, such as the lack of baseline comparison on the real-world dataset. The paper still needs significant further refinement. The majority of reviewers gave a final rating of below borderline.