u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Authors: name: Jinjin Xu; name: Liwu Xu; name: Yuzhe Yang; name: Xiang Li; name: Yanchun Xie; name: Yi-Jie Huang; name: Yaqian Li
Published: 2023-11-09 13:18:27

Abstract: Recent advances such as LLaVA and Mini-GPT4 have successfully integrated
visual information into LLMs, yielding inspiring outcomes and giving rise to a
new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods
struggle with hallucinations and the mutual interference between tasks. To
tackle these problems, we propose an efficient and accurate approach to adapt
to downstream tasks by utilizing LLM as a bridge to connect multiple expert
models, namely u-LLaVA. Firstly, we incorporate the modality alignment module
and multi-task modules into LLM. Then, we reorganize or rebuild multi-type
public datasets to enable efficient modality alignment and instruction
following. Finally, task-specific information is extracted from the trained LLM
and provided to different modules for solving downstream tasks. The overall
framework is simple, effective, and achieves state-of-the-art performance
across multiple benchmarks. We also release our model, the generated data, and
the code base publicly available.