LLaVA-NeXT
Source: transformers

# LLaVA-NeXT## OverviewThe LLaVA-NeXT model was proposed in [LLaVA-NeXT: Improved reasoning, OCR, and world knowledge](https://llava-vl.github.io/blog/2024-01-30-llava-next/) by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon [LLaVa](llava) by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning.The introduction from the blog is the following:*In October 2023, we released LLaVA-1.5 with a simple and efficient design along with great performance on a benchmark suite of 12 datasets. It has since served as the foundation of many comprehensive studies of data, model, and capabilities of large multimodal models (LMM), and has enabled various new applications.Today, we are thrilled to present LLaVA-NeXT, with improved reasoning, OCR, and world knowledge. LLaVA-NeXT even exceeds Gemini Pro on several benchmarks.Compared with LLaVA-1.5, LLaVA-NeXT has several improvements:Increasing the input image resolution to 4x more pixels. This allows it to grasp more visual details. It supports three aspect ratios, up to 672x672, 336x1344, 1344x336 resolution.Better visual reasoning and OCR capability with an improved visual instruction tuning data mixture.Better visual conversation for more scenarios, covering different applications. Better world knowledge and logical reasoning.Efficient deployment and inference with SGLang.Along with performance improvements, LLaVA-NeXT maintains the minimalist design and data efficiency of LLaVA-1.5. It re-uses the pretrained connector of LLaVA-1.5, and still uses less than 1M visual instruction tuning samples. The largest 34B variant finishes training in ~1 day with 32 A100s.*<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/llava_next_overview.png"alt="drawing" width="600"/><small> LLaVa-NeXT incorporates a higher input resolution by encoding various patches of the input image. Taken from the <a href="https://arxiv.org/abs/2310.03744">original paper.</a> </small>This model was contributed by [nielsr](https://huggingface.co/nielsr).The original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main).## Usage tips- We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating.<Tip warning={true}>- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is "left-padding" if model is in `eval()` mode, otherwise "right-padding".</Tip>- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys "role" and "content". The "content" should be a list of dictionaries, for "text" and "image" modalities. Below is an example of how to do that and the list of formats accepted by each checkpoint.We will use [llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) and a conversation history of text and image. Each content field has to be a list of dicts, as follows:```pythonfrom transformers import LlavaNextProcessorprocessor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")conversation = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "What’s shown in this image?"},        ],    },    {        "role": "assistant",        "content": [{"type": "text", "text": "This image shows a red stop sign."},]    },    {        "role": "user",        "content": [            {"type": "text", "text": "Describe the image in more details."},        ],    },]text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your imagesprint(text_prompt)>>> "
 <image>\nWhat's shown in this image? 
 This image shows a red stop sign. 
 Describe the image in more details. 
"```- If you want to construct a chat prompt yourself, below is a list of possible formats.[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:```bash"
 <image>\nWhat is shown in this image? 
"```[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:```bash"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"```[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:```bash"system\nAnswer the questions.user\n<image>\nWhat is shown in this image?assistant\n"```[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:```bash"system\n\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.user\n\n<image>\nWhat is shown in this image?assistant\n\n"```[llava-next-72b-hf](https://huggingface.co/llava-hf/llava-next-72b-hf) and [llava-next-110b-hf](https://huggingface.co/llava-hf/llava-next-110b-hf) require the following format:```bash"system\nYou are a helpful assistant.\nuser\n<image>\nWhat is shown in this image?\nassistant\n"```## Usage example### Single image inferenceHere's how to load the model and perform inference in half-precision (`torch.float16`):```pythonfrom transformers import LlavaNextProcessor, LlavaNextForConditionalGenerationimport torchfrom PIL import Imageimport requestsprocessor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) model.to("cuda:0")# prepare image and text prompt, using the appropriate prompt templateurl = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"image = Image.open(requests.get(url, stream=True).raw)conversation = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "What is shown in this image?"},        ],    },]prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")# autoregressively complete promptoutput = model.generate(**inputs, max_new_tokens=100)print(processor.decode(output[0], skip_special_tokens=True))```### Multi image inferenceLLaVa-Next can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference). Here is how you can do it:```pythonimport requestsfrom PIL import Imageimport torchfrom transformers import AutoProcessor, LlavaNextForConditionalGeneration# Load the model in half-precisionmodel = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, device_map="auto")processor = AutoProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")# Get three different imagesurl = "https://www.ilankelman.org/stopsigns/australia.jpg"image_stop = Image.open(requests.get(url, stream=True).raw)url = "http://images.cocodataset.org/val2017/000000039769.jpg"image_cats = Image.open(requests.get(url, stream=True).raw)url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"image_snowman = Image.open(requests.get(url, stream=True).raw)# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is notconversation_1 = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "What is shown in this image?"},            ],    },    {        "role": "assistant",        "content": [            {"type": "text", "text": "There is a red stop sign in the image."},            ],    },    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "What about this image? How many cats do you see?"},            ],    },]conversation_2 = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "What is shown in this image?"},            ],    },]prompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)prompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)prompts = [prompt_1, prompt_2]# We can simply feed images in the order they have to be used in the text prompt# Each "<image>" token uses one image leaving the next for the subsequent "<image>" tokensinputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors="pt").to(model.device)# Generategenerate_ids = model.generate(**inputs, max_new_tokens=30)processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)```## Model optimization### Quantization using BitsandbytesThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model. First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:```pythonfrom transformers import LlavaNextForConditionalGeneration, BitsAndBytesConfig# specify how to quantize the modelquantization_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.float16,)model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", quantization_config=quantization_config, device_map="auto")```### Use Flash-Attention 2 to further speed-up generationFirst make sure to install flash-attn. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:```pythonfrom transformers import LlavaNextForConditionalGenerationmodel = LlavaNextForConditionalGeneration.from_pretrained(    model_id,     torch_dtype=torch.float16,     low_cpu_mem_usage=True,    use_flash_attention_2=True).to(0)```## LlavaNextConfig[[autodoc]] LlavaNextConfig## LlavaNextImageProcessor[[autodoc]] LlavaNextImageProcessor    - preprocess## LlavaNextProcessor[[autodoc]] LlavaNextProcessor## LlavaNextForConditionalGeneration[[autodoc]] LlavaNextForConditionalGeneration    - forward