|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# OFA-tiny |
|
This is the **tiny** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework. |
|
|
|
To use it in Transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers and download the directory of transformers. After installation, you can use it as shown below: |
|
|
|
``` |
|
>>> from PIL import Image |
|
>>> from torchvision import transforms |
|
>>> from transformers import OFATokenizer, OFAForConditionalGeneration |
|
|
|
>>> mean, std = [0.5, 0.5, 0.5], [0.5, 0.5, 0.5] |
|
>>> resolution = 256 |
|
>>> patch_resize_transform = transforms.Compose([ |
|
lambda image: image.convert("RGB"), |
|
transforms.Resize((resolution, resolution), interpolation=Image.BICUBIC), |
|
transforms.ToTensor(), |
|
transforms.Normalize(mean=mean, std=std) |
|
]) |
|
|
|
>>> model = OFAForConditionalGeneration.from_pretrained(ckpt_dir) |
|
>>> tokenizer = OFATokenizer.from_pretrained(ckpt_dir) |
|
|
|
>>> txt = " what is the description of the image?" |
|
>>> inputs = tokenizer([txt], max_length=1024, return_tensors="pt")["input_ids"] |
|
>>> img = Image.open(path_to_image) |
|
>>> patch_img = patch_resize_transform(img).unsqueeze(0) |
|
|
|
>>> gen = model.generate(inputs, patch_img=patch_img, num_beams=4) |
|
>>> print(tokenizer.decode(gen, skip_special_tokens=True, clean_up_tokenization_spaces=False)) |
|
``` |
|
|
