JaxNN conversion of the timm vit_large_patch16_224.orig_in21k Vision Transformer checkpoint.

Model Details

• Architecture: vit_large_patch16_224
• Source: timm/vit_large_patch16_224.orig_in21k

Model card for vit_large_patch16_224.orig_in21k

A Vision Transformer (ViT) image classification model. Pretrained on ImageNet-21k in JAX by paper authors, ported to PyTorch by Ross Wightman. This model does not have a classification head, useful for features and fine-tune only.

Model Details

• Model Type: Image classification / feature backbone
• Model Stats:
  • Params (M): 303.3
  • GMACs: 59.7
  • Activations (M): 43.8
  • Image size: 224 x 224
• Papers:
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: https://arxiv.org/abs/2010.11929v2
• Dataset: ImageNet-21k
• Original: https://github.com/google-research/vision_transformer

Model Usage

Image Classification

from urllib.request import urlopen

import jax
from PIL import Image

import jaxnn

img = Image.open(urlopen(
    "https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg"
))

model = jaxnn.create_model("vit_large_patch16_224.orig_in21k", pretrained=True)
model.eval()

data_config = jaxnn.data.resolve_model_data_config(model)
transforms = jaxnn.data.create_transform(**data_config, is_training=False)

x = jax.numpy.expand_dims(transforms(img), 0)
output = model(x, deterministic=True)

top5_probabilities, top5_class_indices = jax.lax.top_k(
    jax.nn.softmax(output, axis=-1) * 100,
    k=5,
)

Image Embeddings

from urllib.request import urlopen

import jax
from PIL import Image

import jaxnn

img = Image.open(urlopen(
    "https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg"
))

model = jaxnn.create_model(
    "vit_large_patch16_224.orig_in21k",
    pretrained=True,
    num_classes=0,
)
model.eval()

data_config = jaxnn.data.resolve_model_data_config(model)
transforms = jaxnn.data.create_transform(**data_config, is_training=False)

x = jax.numpy.expand_dims(transforms(img), 0)
output = model(x, deterministic=True)

Citation

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}