VAN

## [](#overview)Overview

The VAN model was proposed in [Visual Attention Network](https://arxiv.org/abs/2202.09741) by Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu.

This paper introduces a new attention layer based on convolution operations able to capture both local and distant relationships. This is done by combining normal and large kernel convolution layers. The latter uses a dilated convolution to capture distant correlations.

The abstract from the paper is the following:

_While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel large kernel attention (LKA) module to enable self-adaptive and long-range correlations in self-attention while avoiding the above issues. We further introduce a novel neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN outperforms the state-of-the-art vision transformers and convolutional neural networks with a large margin in extensive experiments, including image classification, object detection, semantic segmentation, instance segmentation, etc. Code is available at [this https URL](https://github.com/Visual-Attention-Network/VAN-Classification)._

Tips:

-   VAN does not have an embedding layer, thus the `hidden_states` will have a length equal to the number of stages.

The figure below illustrates the architecture of a Visual Aattention Layer. Taken from the [original paper](https://arxiv.org/abs/2202.09741).

![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/van_architecture.png)

This model was contributed by [Francesco](https://huggingface.co/Francesco). The original code can be found [here](https://github.com/Visual-Attention-Network/VAN-Classification).

## [](#resources)Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with VAN.

-   [VanForImageClassification](/docs/transformers/v4.30.0/en/model_doc/van#transformers.VanForImageClassification) is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
-   See also: [Image classification task guide](../tasks/image_classification)

If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## [](#transformers.VanConfig)VanConfig

### class transformers.VanConfig

[](#transformers.VanConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/van/configuration_van.py#L30)

( image\_size = 224num\_channels = 3patch\_sizes = \[7, 3, 3, 3\]strides = \[4, 2, 2, 2\]hidden\_sizes = \[64, 128, 320, 512\]depths = \[3, 3, 12, 3\]mlp\_ratios = \[8, 8, 4, 4\]hidden\_act = 'gelu'initializer\_range = 0.02layer\_norm\_eps = 1e-06layer\_scale\_init\_value = 0.01drop\_path\_rate = 0.0dropout\_rate = 0.0\*\*kwargs )

This is the configuration class to store the configuration of a [VanModel](/docs/transformers/v4.30.0/en/model_doc/van#transformers.VanModel). It is used to instantiate a VAN model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the VAN [Visual-Attention-Network/van-base](https://huggingface.co/Visual-Attention-Network/van-base) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import VanModel, VanConfig

>>> 
>>> configuration = VanConfig()
>>> 
>>> model = VanModel(configuration)
>>> 
>>> configuration = model.config```

## [](#transformers.VanModel)VanModel

### class transformers.VanModel

[](#transformers.VanModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/van/modeling_van.py#L425)

( config )

Parameters

-   [](#transformers.VanModel.config)**config** ([VanConfig](/docs/transformers/v4.30.0/en/model_doc/van#transformers.VanConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare VAN model outputting raw features without any specific head on top. Note, VAN does not have an embedding layer. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.VanModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/van/modeling_van.py#L435)

( pixel\_values: typing.Optional\[torch.FloatTensor\]output\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → `transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention` or `tuple(torch.FloatTensor)`

The [VanModel](/docs/transformers/v4.30.0/en/model_doc/van#transformers.VanModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoImageProcessor, VanModel
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("Visual-Attention-Network/van-base")
>>> model = VanModel.from_pretrained("Visual-Attention-Network/van-base")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 512, 7, 7]```

## [](#transformers.VanForImageClassification)VanForImageClassification

### class transformers.VanForImageClassification

[](#transformers.VanForImageClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/van/modeling_van.py#L480)

( config )

Parameters

-   [](#transformers.VanForImageClassification.config)**config** ([VanConfig](/docs/transformers/v4.30.0/en/model_doc/van#transformers.VanConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

VAN Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

The [VanForImageClassification](/docs/transformers/v4.30.0/en/model_doc/van#transformers.VanForImageClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoImageProcessor, VanForImageClassification
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("Visual-Attention-Network/van-base")
>>> model = VanForImageClassification.from_pretrained("Visual-Attention-Network/van-base")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> 
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tabby, tabby cat```