MINGLE: Multi-level Integration via Language Guided Learning\\for Comprehensive Visual Understanding
Authors: Sungjune Park, Beomchan Park, Hyunjun Kim, Dahye Lee, Yong Man Ro


Reviewer paper summaries: With the success of large language models (LLMs), recent works have been working on exploiting language representations to  address vision problems. However, given that visual semantics vary at different levels (e.g., scene-level, coarse-level, fine-grained, etc.), the language integration to visual semantics needs further modifications. This paper investigates how to fuse language representations into visual semantic learning at multiple levels, and proposes Multi-level Integration via Language Guided Learning (MINGLE) for comprehensive visual understanding. At coarse level, a semantic reasoner helps a visual encoder to understand global contents. At fine-grained level, a language information transport loss is proposed to transfer knowledge of language representations in order to  improve robustness towards instance-level variances of a category The experimental results and qualitative results both demonstrate the effectiveness of MINGLE on drone-view object detection tasks.
Summary of strengths: 1. MINGLE is simple and effective, where it outperforms prior state-of-the-art on both UAVDT and VisDrone benchmarks.
2. The qualitative results and ablation studies justify the design choies of MINGLE.
3. MINGLE can be further applied to other vision problems, not only object detection.
Summary of weaknesses: 1. How to obtain description embeddings (L269-L270)? The variant of the exemplar description prompt has varying lengths. Are these any aggregations over the token representations or the prompt is passed into an LLM to obtain the representation?
2. A further ablation on learnable categorical prompts on the model’s performance on downstream tasks can be done to further understand the effectiveness of this component.
3. Majority of prior works use ViT as its visual backbone on UAVDT. A further ablation with ViT can make the comparisons more fair and direct.
Comments, suggestions and typos: 1. It will be interesting to carry out cross-task generalization experiments to examine the generalizability of MINGLE.
Overall assessment scores: 3.5