Edit model card

πŸ”₯ Mantis

Paper | Website | Github | Models | Demo

Mantis

Summary

  • Mantis-Fuyu is a Fuyu based LMM with interleaved text and image as inputs, train on Mantis-Instruct under academic-level resources (i.e. 36 hours on 16xA100-40G).
  • Mantis is trained to have multi-image skills including co-reference, reasoning, comparing, temporal understanding.
  • Mantis reaches the state-of-the-art performance on five multi-image benchmarks (NLVR2, Q-Bench, BLINK, MVBench, Mantis-Eval), and also maintain a strong single-image performance on par with CogVLM and Emu2.

Multi-Image Performance

SoTA performance on 5 multi-image benchmarks

Single-Image Performance

Strong performance on single image results

How to use

Installation

# This only installs minimum packages (torch, transformers, accelerate) for inference, no redundant packages are installed.
pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

Run example inference:

from mantis.models.mllava import chat_mllava
from PIL import Image
import torch


image1 = "image1.jpg"
image2 = "image2.jpg"
images = [Image.open(image1), Image.open(image2)]

# load processor and model
from mantis.models.mfuyu import MFuyuForCausalLM, MFuyuProcessor
processor = MFuyuProcessor.from_pretrained("TIGER-Lab/Mantis-8B-Fuyu")
attn_implementation = None # or "flash_attention_2"
model = MFuyuForCausalLM.from_pretrained("TIGER-Lab/Mantis-8B-Fuyu", device_map="cuda", torch_dtype=torch.bfloat16, attn_implementation=attn_implementation)

generation_kwargs = {
    "max_new_tokens": 1024,
    "num_beams": 1,
    "do_sample": False,
    "pad_token_id": processor.tokenizer.eos_token_id,
}

# chat
text = "Describe the difference of <image> and <image> as much as you can."
response, history = chat_mllava(text, images, model, processor, **generation_kwargs)

print("USER: ", text)
print("ASSISTANT: ", response)

text = "How many wallets are there in image 1 and image 2 respectively?"
response, history = chat_mllava(text, images, model, processor, history=history, **generation_kwargs)

print("USER: ", text)
print("ASSISTANT: ", response)

"""
USER:  Describe the difference of <image> and <image> as much as you can.
ASSISTANT:  The second image has more variety in terms of colors and designs. While the first image only shows two brown leather pouches, the second image features four different pouches in various colors and designs, including a purple one with a gold coin, a red one with a gold coin, a black one with a gold coin, and a brown one with a gold coin. This variety makes the second image more visually interesting and dynamic.
USER:  How many wallets are there in image 1 and image 2 respectively?
ASSISTANT:  There are two wallets in image 1, and four wallets in image 2.
"""

Training

See mantis/train for details

Evaluation

See mantis/benchmark for details

Citation

@inproceedings{Jiang2024MANTISIM,
  title={MANTIS: Interleaved Multi-Image Instruction Tuning},
  author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
  publisher={arXiv2405.01483}
  year={2024},
}
Downloads last month
365
Safetensors
Model size
9.41B params
Tensor type
BF16
Β·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for TIGER-Lab/Mantis-8B-Fuyu

Base model

adept/fuyu-8b
Finetuned
(3)
this model

Dataset used to train TIGER-Lab/Mantis-8B-Fuyu

Collection including TIGER-Lab/Mantis-8B-Fuyu