YifanXu/libra-vision-tokenizer

Libra Vision Tokenizer

Libra: Building Decoupled Vision System on Large Language Models

This repo provides the pretrained weight of Libra vision tokenizer trained with lookup-free quantization.

!!! NOTE !!!

1. Please merge the weights into llama-2-7b-chat-hf-libra (huggingface version of LLaMA2-7B-Chat).

2. Please download the pretrained CLIP model in huggingface and merge it into the path. The CLIP model can be downloaded here.

The files should be organized as:

llama-2-7b-chat-hf-libra/
|
│   # original llama files
|
├── ...
│   
│   # newly added vision tokenizer
│   
├── vision_tokenizer_config.yaml
├── vqgan.ckpt
│
│   # CLIP model
│
└── openai-clip-vit-large-patch14-336/
    └── ...