MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples
Authors: Chen Tao, Enwei Zhang, Yuting Gao, Ke Li, Xing Sun, Yan Zhang, Hui Li


Reviewer paper summaries: The paper proposes a novel fine-tuning strategy, Multi-Modal Fine-Tuning with In-Context Tuning (MMICT), which utilizes a frozen visual encoder and an LLM to solve various vision-language tasks. Specifically, a new module called M-Hub is introduced. M-Hub architecture is similar to Q-former but capable of processing learnable query or text inputs and generating image/video-guided features via cross-attention. MMICT can be used to input multiple demonstrations (task-specific image-text pairs) to enhance the performance of the target task sample. Experimental evaluations across various multi-modal tasks and datasets show that MMICT and in-context demonstrations can be helpful.
Summary of strengths: * M-Hub has several advantages because it can take both query tokens and/or text tokens. For example, frozen LLM does not need to be modified (as the authors discussed in Appendix A) to integrate visual information. Also, M-Hub is more flexible to adapt to different vision-language tasks.
* The paper provides a comprehensive overview of the background and related works, sufficiently acknowledging the contributions of prior works.
Summary of weaknesses: * Details regarding the training and inference process are somewhat confusing. For example, these points are not clear to me. Please consider rewriting or clarifying the details.
  * Is the training process task-dependent? Is training only done once and then the model can be used for multiple vision-language tasks? For different tasks, do you use different configurations for training/inference (as depicted in Figure 2)?
  * During training, are the video frames and their captions only used data type? Or (image, caption) pairs are also used as in previous works?
  * M-Hub needs to understand the correlation between (video/image, caption) and (video/image, instruction). For caption tokens paired with videos, they do not get gradients from next-token prediction. Is this correct?
  * Which dataset did you use for training?
* The effect of in-context examples is not shown clearly. In Table 3, more examples do not lead to better performance. Can you provide an explanation for this?
* Although the proposed method does not change LLM itself, it needs to run a heavy EVA model during inference for every example. Can you compare the total computation cost / total model size between MMICT and Otter?
Comments, suggestions and typos: * Figure 1 is too small. Please consider enlarging the figure for improved readability.
* Typo in Conclusion, line 641 (“M-Hubyh”)
* M-Hub is initialized using Q-former, requiring additional dependencies, and potentially limiting MMICT’s practical use. Is this choice necessary?
* Please consider moving the comparison with Otter (Appendix A) to the main sections. Also, can we see the Otter’s performance on the same tasks (Table 1)?
Overall assessment scores: 2.5