Convolutional Vision Transformer (CvT)

## [](#overview)Overview

The CvT model was proposed in [CvT: Introducing Convolutions to Vision Transformers](https://arxiv.org/abs/2103.15808) by Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan and Lei Zhang. The Convolutional vision Transformer (CvT) improves the [Vision Transformer (ViT)](vit) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs.

The abstract from the paper is the following:

_We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks._

Tips:

-   CvT models are regular Vision Transformers, but trained with convolutions. They outperform the [original model (ViT)](vit) when fine-tuned on ImageNet-1K and CIFAR-100.
-   You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace [ViTFeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/vit#transformers.ViTFeatureExtractor) by [AutoImageProcessor](/docs/transformers/v4.30.0/en/model_doc/auto#transformers.AutoImageProcessor) and [ViTForImageClassification](/docs/transformers/v4.30.0/en/model_doc/vit#transformers.ViTForImageClassification) by [CvtForImageClassification](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtForImageClassification)).
-   The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of 14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).

This model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/microsoft/CvT).

## [](#resources)Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CvT.

-   [CvtForImageClassification](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtForImageClassification) is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
-   See also: [Image classification task guide](../tasks/image_classification)

If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## [](#transformers.CvtConfig)CvtConfig

### class transformers.CvtConfig

[](#transformers.CvtConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/configuration_cvt.py#L29)

( num\_channels = 3patch\_sizes = \[7, 3, 3\]patch\_stride = \[4, 2, 2\]patch\_padding = \[2, 1, 1\]embed\_dim = \[64, 192, 384\]num\_heads = \[1, 3, 6\]depth = \[1, 2, 10\]mlp\_ratio = \[4.0, 4.0, 4.0\]attention\_drop\_rate = \[0.0, 0.0, 0.0\]drop\_rate = \[0.0, 0.0, 0.0\]drop\_path\_rate = \[0.0, 0.0, 0.1\]qkv\_bias = \[True, True, True\]cls\_token = \[False, False, True\]qkv\_projection\_method = \['dw\_bn', 'dw\_bn', 'dw\_bn'\]kernel\_qkv = \[3, 3, 3\]padding\_kv = \[1, 1, 1\]stride\_kv = \[2, 2, 2\]padding\_q = \[1, 1, 1\]stride\_q = \[1, 1, 1\]initializer\_range = 0.02layer\_norm\_eps = 1e-12\*\*kwargs )

This is the configuration class to store the configuration of a [CvtModel](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtModel). It is used to instantiate a CvT model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the CvT [microsoft/cvt-13](https://huggingface.co/microsoft/cvt-13) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import CvtConfig, CvtModel

>>> 
>>> configuration = CvtConfig()

>>> 
>>> model = CvtModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.CvtModel)CvtModel

### class transformers.CvtModel

[](#transformers.CvtModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_cvt.py#L590)

( configadd\_pooling\_layer = True )

Parameters

-   [](#transformers.CvtModel.config)**config** ([CvtConfig](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare Cvt Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.CvtModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_cvt.py#L605)

( pixel\_values: typing.Optional\[torch.Tensor\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → `transformers.models.cvt.modeling_cvt.BaseModelOutputWithCLSToken` or `tuple(torch.FloatTensor)`

The [CvtModel](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoImageProcessor, CvtModel
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/cvt-13")
>>> model = CvtModel.from_pretrained("microsoft/cvt-13")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 384, 14, 14]```

## [](#transformers.CvtForImageClassification)CvtForImageClassification

### class transformers.CvtForImageClassification

[](#transformers.CvtForImageClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_cvt.py#L651)

( config )

Parameters

-   [](#transformers.CvtForImageClassification.config)**config** ([CvtConfig](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Cvt Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the \[CLS\] token) e.g. for ImageNet.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

The [CvtForImageClassification](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtForImageClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoImageProcessor, CvtForImageClassification
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/cvt-13")
>>> model = CvtForImageClassification.from_pretrained("microsoft/cvt-13")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> 
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tabby, tabby cat```

## [](#transformers.TFCvtModel)TFCvtModel

### class transformers.TFCvtModel

[](#transformers.TFCvtModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_tf_cvt.py#L762)

( \*args\*\*kwargs )

Parameters

-   [](#transformers.TFCvtModel.config)**config** ([CvtConfig](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.TFPreTrainedModel.from_pretrained) method to load the model weights.

The bare Cvt Model transformer outputting raw hidden-states without any specific head on top.

This model inherits from [TFPreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.TFPreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a [tf.keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TF 2.0 models accepts two formats as inputs:

-   having all inputs as keyword arguments (like PyTorch models), or
-   having all inputs as a list, tuple or dict in the first positional arguments.

This second option is useful when using `tf.keras.Model.fit` method which currently requires having all the tensors in the first argument of the model call function: `model(inputs)`.

#### call

[](#transformers.TFCvtModel.call)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_tf_cvt.py#L768)

( pixel\_values: tf.Tensor | None = Noneoutput\_hidden\_states: Optional\[bool\] = Nonereturn\_dict: Optional\[bool\] = Nonetraining: Optional\[bool\] = False ) → `transformers.models.cvt.modeling_tf_cvt.TFBaseModelOutputWithCLSToken` or `tuple(tf.Tensor)`

The [TFCvtModel](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.TFCvtModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoImageProcessor, TFCvtModel
>>> from PIL import Image
>>> import requests

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/cvt-13")
>>> model = TFCvtModel.from_pretrained("microsoft/cvt-13")

>>> inputs = image_processor(images=image, return_tensors="tf")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state```

## [](#transformers.TFCvtForImageClassification)TFCvtForImageClassification

### class transformers.TFCvtForImageClassification

[](#transformers.TFCvtForImageClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_tf_cvt.py#L826)

( \*args\*\*kwargs )

Parameters

-   [](#transformers.TFCvtForImageClassification.config)**config** ([CvtConfig](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.CvtConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.TFPreTrainedModel.from_pretrained) method to load the model weights.

Cvt Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the \[CLS\] token) e.g. for ImageNet.

This model inherits from [TFPreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.TFPreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a [tf.keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TF 2.0 models accepts two formats as inputs:

-   having all inputs as keyword arguments (like PyTorch models), or
-   having all inputs as a list, tuple or dict in the first positional arguments.

This second option is useful when using `tf.keras.Model.fit` method which currently requires having all the tensors in the first argument of the model call function: `model(inputs)`.

#### call

[](#transformers.TFCvtForImageClassification.call)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cvt/modeling_tf_cvt.py#L844)

( pixel\_values: tf.Tensor | None = Nonelabels: tf.Tensor | None = Noneoutput\_hidden\_states: Optional\[bool\] = Nonereturn\_dict: Optional\[bool\] = Nonetraining: Optional\[bool\] = False ) → `transformers.modeling_tf_outputs.TFImageClassifierOutputWithNoAttention` or `tuple(tf.Tensor)`

The [TFCvtForImageClassification](/docs/transformers/v4.30.0/en/model_doc/cvt#transformers.TFCvtForImageClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoImageProcessor, TFCvtForImageClassification
>>> import tensorflow as tf
>>> from PIL import Image
>>> import requests

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/cvt-13")
>>> model = TFCvtForImageClassification.from_pretrained("microsoft/cvt-13")

>>> inputs = image_processor(images=image, return_tensors="tf")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> 
>>> predicted_class_idx = tf.math.argmax(logits, axis=-1)[0]
>>> print("Predicted class:", model.config.id2label[int(predicted_class_idx)])```