Add QLIP into transformers
Author: lavinal712
Pull request: no
Labels: name: New model

### Model description

Introduction:

We introduce Quantized Language-Image Pretraining (QLIP), a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image understanding. QLIP trains a binary-spherical-quantization-based autoencoder with reconstruction and language-image alignment objectives. We are the first to show that the two objectives do not need to be at odds. We balance the two loss terms dynamically during training and show that a two-stage training pipeline effectively mixes the large-batch requirements of image-language pre-training with the memory bottleneck imposed by the reconstruction objective. We validate the effectiveness of QLIP for multimodal understanding and text-conditioned image generation with a single model. Specifically, QLIP serves as a drop-in replacement for the visual encoder for LLaVA and the image tokenizer for LlamaGen with comparable or even better performance. Finally, we demonstrate that QLIP enables a unified mixed-modality auto-regressive model for understanding and generation.

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2502.05178

Code: https://github.com/NVlabs/QLIP

Comments:
Hey @lavinal712!  

Awesome paper! We've been adding more quantizer-based VLMs to `transformers` recently, so it’s exciting to see another one in the mix. Since there aren't well-established standards for these models yet, the review process might take a bit longer than usual.  

To start, I’d recommend adding the model as custom code on the Hub. You can find the documentation [[here](https://huggingface.co/docs/transformers/en/custom_models#model)](https://huggingface.co/docs/transformers/en/custom_models#model). Keep in mind that being a custom code model doesn’t make it any less significant—many highly popular, high-performance models, like [[Phi-3.5-vision](https://huggingface.co/microsoft/Phi-3.5-vision-instruct)](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) (>250k downloads/month), are custom code models without a Transformers PR. This is often the best first step for most authors!  

That said, feel free to submit a PR if you’d like - just note that starting with custom code will get your model out there faster for users to experiment with. Looking forward to seeing the model in action! 🚀

The model code has been added to the hub.

model: https://huggingface.co/nvidia/QLIP-B-8-256 https://huggingface.co/lavinal712/QLIP-B-8-256


```python
from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
from PIL import Image
import torch
from torchvision.transforms import Normalize
from diffusers.utils import load_image

model_name = "lavinal712/QLIP-B-8-256"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(model_name, trust_remote_code=True)

image_path = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png"
image = load_image(image_path)
captions = ["a dog", "a logo", "a cat"]

input_ids = tokenizer(captions, return_tensors="pt", padding=True).input_ids
input_pixels = processor(images=image, return_tensors="pt", padding=True).pixel_values
outputs = model(input_ids, input_pixels)


inverse_normalize = Normalize(
    mean=[-m / s for m, s in zip(processor.image_mean, processor.image_std)],
    std=[1 / s for s in processor.image_std]
)
input2 = inverse_normalize(input_pixels[0]).clamp_(0, 1)
recon = inverse_normalize(outputs["reconstructions"][0]).clamp_(0, 1)


with torch.no_grad():
    image_features = outputs["image_embeds"] # equivalent to: `image_features = model.get_image_features(input_pixels)`
    text_features = outputs["text_embeds"]   # equivalent to: `text_features = model.get_text_features(input_ids)`
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

label_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print(f"model label_probs: {label_probs}")

collage = torch.cat([input2, recon], dim=2)
collage = (collage * 255).to(torch.uint8).numpy().transpose(1, 2, 0)
Image.fromarray(collage).save("collage.png")
```

![Image](https://github.com/user-attachments/assets/f86188d8-d114-4fae-9148-14eea0398cf4)

```
label_probs: tensor([[3.3339e-04, 1.0870e-05, 9.9966e-01]], grad_fn=<SoftmaxBackward0>)
```