Title: Multi-Modal Interactive Agent Layer for Few-Shot Universal Cross-Domain Retrieval and Beyond
Venue: NeurIPS 2025
Accepted: yes
Keywords: Few-Shot Universal Cross Domain Retrieval

Vision-Language Models

Cross-Domain Interaction

Agent Layer
Abstract: This paper firstly addresses the challenge of few-shot universal cross-domain retrieval (FS-UCDR), enabling machines trained with limited data to generalize to novel retrieval scenarios, with queries from entirely unknown domains and categories. To achieve this, we first formally define the FS-UCDR task and propose the Multi-Modal Interactive Agent Layer (MAIL), which enhances the cross-modal interaction in vision-language models (VLMs) by aligning the parameter updates of target layer pairs across modalities.
Specifically, MAIL freezes the selected target layer pair and introduces a trainable agent layer pair to approximate localized parameter updates. A bridge function is then introduced to couple the agent layer pair, enabling gradient communication across modalities to facilitate update alignment. The proposed MAIL offers four key advantages: 1) its cross-modal interaction mechanism improves knowledge acquisition from limited data, making it highly effective in low-data scenarios; 2) during inference, MAIL integrates seamlessly into the VLM via reparameterization, preserving inference complexity; 3) extensive experiments validate the superiority of MAIL, which achieves substantial performance gains over data-efficient UCDR methods while requiring significantly fewer training samples; 4) beyond UCDR, MAIL also performs competitively on few-shot classification tasks, underscoring its strong generalization ability. Code.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on few-shot universal cross-domain retrieval (FS-UCDR), a newly introduced scenario that considers limited data in traditional universal cross-domain retrieval. Authors experimentally find that modality-coupled methods work well under the proposed FS-UCDR to parameter-efficiently adapt VLMs for this task, and correspondingly design the Multi-Modal Interactive Agent Layer (MAIL) to align parameter updates of different layers. Extensive experiments on retrieval and few-shot classification validate the effectiveness and efficiency of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The overall structure is clear and easy to follow.\n\n2. The newly introduced FS-UCDR scenario is practical, exploring whether a few samples from each source domain can lead to good generalization performance.\n\n3. Experiments are solid. MAIL is evaluated both on FS-UCDR and few-shot classification, with results on multiple datasets supporting the effectiveness.\n\nWeakness:\n\n1. The motivation for introducing modality-coupling into the partially fine-tuned methods is unclear. Authors claim that the lack of such endeavors in the literature derives from the non-trivial nature. However, a hard-to-implement design does not necessarily guarantee better performance. Therefore, more clarifications are required.\n\n2. It seems that the authors introduce a general PEFT method that works for all tasks that require the adaptation of VLMs. Motivation mainly focuses on inference efficiency and the fact that modality-coupling has not been explored in the fine-tuned-based approaches. Little motivation is given on why this is a good design for Universal Cross-Domain Retrieval and the newly introduced Few-Shot-UCDR. \n\n3. It's quite confusing for readers to understand the original layer, target layer, and agent layer without context and specifications. Please define earlier in the Introduction Section.\n\nQUESTIONS:\n1. Given the non-trivial nature of the partially fine-tuned methods mentioned in the paper, why still try to introduce modality coupling to such methods? Are there any reasons other than the under-explored attempts for including modality coupling in partially fine-tuned methods?\n\n2. What's the connection between the MAIL and FS-UCDR? Why would it address the difficulties of generalizing with limited training data?\n\n3. What's the rationale behind aligning parameter updates? Why would it help the adaptation of VLMs? More analyses are encouraged to be provided.\n\nOverall, my concerns focus on the motivation of the designed approach, the connection with FS-UCDR, and the rationale behind aligning parameter updates. I would appreciate the authors’ clarifications on these questions. Provided that the requested clarifications are addressed, I will improve the recommendation score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper proposes a new method for cross-model interaction in visual language models (in particular CLIP like models) through synchronous parameter updates for the two modalities. The main difference with existing methods is that the alignment is operating at a higher level i.e. more at the semantic level than the "signal" level. The method is particularly targeting few-shot universal cross-domain retrieval. Results on a number of datasets are showing that the method outperforms existing methods most of the time.

STRENGTHS AND WEAKNESSES:
Strengths: 
- New approach to the problem, by linking the two modalities at a different level. 
- The method at inference remains to have the same complexity, no significant additional overhead.
- Results on existing benchmark compared to other methods is good.

Weaknesses: 
- Performance not that different from MMA with same number of tunable parameters.
- Figure 7 very misleading. Y-axis is not starting at 0 (which for a barchart should always be the case) and y-range are not even given.
- Figure 7 indicates there is no benefit in using this method apart from the improvements in accuracy. So also figure 1 is misleading. 
- It is interesting to move the alignment to higher levels with less parameters. But why would that be interesting, if training time is still higher and testing time only slightly better.

QUESTIONS:
- See the final weakness. Why is it intrinsically interesting to have this method? 
- Why doesn't the reduced number of parameters not lead to more efficiency?

LIMITATIONS:
One limitation is indicated, so quite minimal.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

comments: SUMMARY:
This paper proposes MAIL, a method for Few-Shot Unsupervised Cross-Domain Retrieval using coupled agent layers with scaling-and-shifting reparameterization to enhance CLIP's cross-modal alignment. MAIL maintains CLIP’s inference efficiency while improving adaptability, outperforming baselines on three benchmarks. However, its sequential processing incurs minor training and memory overhead.

STRENGTHS AND WEAKNESSES:
Strengths

1. The coupled agent layers concept leveraging scaling-and-shifting reparameterization is an interesting approach. Its key strength is enhancing adaptation without compromising CLIP's original inference efficiency.  
2. The propoesd method demonstrates consistent superiority over competitors across several benchmarks, provides evidence for MAIL's effectiveness in the Few-Shot Unsupervised Cross-Domain Retrieval setting.  

Weaknesses

1. Relying on CLIP’s pre-trained features, MAIL may underperform in domains with unique visual-linguistic distributions (e.g., medical imaging or remote sensing), where pre-trained representations lack task-specific nuances. This limits adaptability in specialized fields.
2. I would like to see the performance comparision of MAIL with a similar PEFT methods named SVFT [1]. SVFT updates the weight as a sparse combination a diagonala matrix outer products of its singular vectors, training only the coefficients. The proposed MAIL learns a diagonal matrix with a scaling vector as its diagonal elements. 

[1] Lingam, Vijay Chandra, et al. Svft: Parameter-efficient fine-tuning with singular vectors. Advances in Neural Information Processing Systems, 2024.

QUESTIONS:
No

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03