ImageBind-LLM: Multi-modality Instruction Tuning
Authors: name: Jiaming Han; name: Renrui Zhang; name: Wenqi Shao; name: Peng Gao; name: Peng Xu; name: Han Xiao; name: Kaipeng Zhang; name: Chris Liu; name: Song Wen; name: Ziyu Guo; name: Xudong Lu; name: Shuai Ren; name: Yafei Wen; name: Xiaoxin Chen; name: Xiangyu Yue; name: Hongsheng Li; name: Yu Qiao
Published: 2023-09-07 17:59:45

Abstract: We present ImageBind-LLM, a multi-modality instruction tuning method of large
language models (LLMs) via ImageBind. Existing works mainly focus on language
and image instruction tuning, different from which, our ImageBind-LLM can
respond to multi-modality conditions, including audio, 3D point clouds, video,
and their embedding-space arithmetic by only image-text alignment training.
During training, we adopt a learnable bind network to align the embedding space
between LLaMA and ImageBind's image encoder. Then, the image features
transformed by the bind network are added to word tokens of all layers in
LLaMA, which progressively injects visual instructions via an attention-free
and zero-initialized gating mechanism. Aided by the joint embedding of
ImageBind, the simple image-text training enables our model to exhibit superior
multi-modality instruction-following capabilities. During inference, the
multi-modality inputs are fed into the corresponding ImageBind encoders, and
processed by a proposed visual cache model for further cross-modal embedding
enhancement. The training-free cache model retrieves from three million image
features extracted by ImageBind, which effectively mitigates the
training-inference modality discrepancy. Notably, with our approach,
ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
significant language generation quality. Code is released at
https://github.com/OpenGVLab/LLaMA-Adapter.