brightxiaolei/light-vlm-0.6b

Light VLM 0.6B

Light VLM is a lightweight multimodal vision-language model built upon Qwen2.5-0.5B-Instruct and CLIP ViT-base-patch16.

Demonstration

Below are some examples demonstrating the model's capabilities on various vision-language tasks.

Image Captioning

Image
Prompt	what can you see in this picture?
Response	In this picture, you can see a man with an axe in his hand. He is standing on the edge of a wooden fence and appears to be working at something that involves cutting wood or possibly chopping down trees. The scene suggests he might be engaged in some kind of woodworking activity such as sawing logs into pieces for construction purposes or preparing materials for other projects like furniture making or building structures.

Image
Prompt	describe this picture
Response	The picture depicts a group of children, specifically boys and girls in their 10s or younger. The scene is set on the beach with an outdoor setting.\n\nIn the foreground, there are several adults sitting around a table where they appear to be discussing something related to sports activities such as soccer or baseball. There's also a man standing near one of them who appears to have his arms outstretched for support during a game.\n\nOn either side of these individuals, there are two young men playing

Model Architecture

• Language Model: Qwen2.5-0.5B-Instruct
• Vision Encoder: openai/clip-vit-base-patch16
• Projector: MLP (768 → 2048 → 896)
• Fine-tuning: LoRA (rank=64, alpha=128)

Total Parameters

~590M (0.6B)

Features

• Image captioning
• Visual question answering
• Multi-device support (CUDA/MPS/CPU)
• OpenAI-compatible API

Quick Start

from PIL import Image
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import CLIPVisionModel, CLIPImageProcessor

# Load model components
model_name = "brightxiaolei/light-vlm-0.6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
vision_encoder = CLIPVisionModel.from_pretrained(f"{model_name}/vision_model")
image_processor = CLIPImageProcessor.from_pretrained(f"{model_name}/vision_model")

# Example: Image captioning
image = Image.open("example.jpg")
inputs = image_processor(images=image, return_tensors="pt")
pixel_values = inputs['pixel_values']

API Usage

curl -X POST "http://localhost:8000/v1/chat/completions" \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "light-vlm-0.6b",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "描述这张图片"},
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
        ]
      }
    ],
    "max_tokens": 256
  }'

Limitations

• Designed for basic visual understanding tasks
• Not suitable for complex reasoning tasks
• Performance may vary on diverse image types

License

MIT License