AltCLIP

## [](#overview)Overview

The AltCLIP model was proposed in [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://arxiv.org/abs/2211.06679v2) by Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu. AltCLIP (Altering the Language Encoder in CLIP) is a neural network trained on a variety of image-text and text-text pairs. By switching CLIP’s text encoder with a pretrained multilingual text encoder XLM-R, we could obtain very close performances with CLIP on almost all tasks, and extended original CLIP’s capabilities such as multilingual understanding.

The abstract from the paper is the following:

_In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding._

## [](#usage)Usage

The usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention and we take the \[CLS\] token in XLM-R to represent text embedding.

AltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image classification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similar score.

To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches, which are then linearly embedded. A \[CLS\] token is added to serve as representation of an entire image. The authors also add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder. The [CLIPImageProcessor](/docs/transformers/v4.30.0/en/model_doc/clip#transformers.CLIPImageProcessor) can be used to resize (or rescale) and normalize images for the model.

The [AltCLIPProcessor](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPProcessor) wraps a [CLIPImageProcessor](/docs/transformers/v4.30.0/en/model_doc/clip#transformers.CLIPImageProcessor) and a [XLMRobertaTokenizer](/docs/transformers/v4.30.0/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) into a single instance to both encode the text and prepare the images. The following example shows how to get the image-text similarity scores using [AltCLIPProcessor](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPProcessor) and [AltCLIPModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPModel).

```
>>> from PIL import Image
>>> import requests

>>> from transformers import AltCLIPModel, AltCLIPProcessor

>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AltCLIPProcessor.from_pretrained("BAAI/AltCLIP")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  
>>> probs = logits_per_image.softmax(dim=1)  ```

Tips:

This model is build on `CLIPModel`, so use it like a original CLIP.

This model was contributed by [jongjyh](https://huggingface.co/jongjyh).

## [](#transformers.AltCLIPConfig)AltCLIPConfig

### class transformers.AltCLIPConfig

[](#transformers.AltCLIPConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/configuration_altclip.py#L246)

( text\_config = Nonevision\_config = Noneprojection\_dim = 768logit\_scale\_init\_value = 2.6592\*\*kwargs )

Parameters

-   [](#transformers.AltCLIPConfig.text_config)**text\_config** (`dict`, _optional_) — Dictionary of configuration options used to initialize [AltCLIPTextConfig](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPTextConfig).
-   [](#transformers.AltCLIPConfig.vision_config)**vision\_config** (`dict`, _optional_) — Dictionary of configuration options used to initialize [AltCLIPVisionConfig](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPVisionConfig).
-   [](#transformers.AltCLIPConfig.projection_dim)**projection\_dim** (`int`, _optional_, defaults to 512) — Dimentionality of text and vision projection layers.
-   [](#transformers.AltCLIPConfig.logit_scale_init_value)**logit\_scale\_init\_value** (`float`, _optional_, defaults to 2.6592) — The inital value of the _logit\_scale_ paramter. Default is used as per the original CLIP implementation.
-   [](#transformers.AltCLIPConfig.kwargs)**kwargs** (_optional_) — Dictionary of keyword arguments.

This is the configuration class to store the configuration of a [AltCLIPModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPModel). It is used to instantiate an AltCLIP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AltCLIP [BAAI/AltCLIP](https://huggingface.co/BAAI/AltCLIP) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import AltCLIPConfig, AltCLIPModel

>>> 
>>> configuration = AltCLIPConfig()

>>> 
>>> model = AltCLIPModel(configuration)

>>> 
>>> configuration = model.config

>>> 

>>> 
>>> config_text = AltCLIPTextConfig()
>>> config_vision = AltCLIPVisionConfig()

>>> config = AltCLIPConfig.from_text_vision_configs(config_text, config_vision)```

#### from\_text\_vision\_configs

[](#transformers.AltCLIPConfig.from_text_vision_configs)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/configuration_altclip.py#L382)

( text\_config: AltCLIPTextConfigvision\_config: AltCLIPVisionConfig\*\*kwargs ) → [AltCLIPConfig](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPConfig)

An instance of a configuration object

Instantiate a [AltCLIPConfig](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPConfig) (or a derived class) from altclip text model configuration and altclip vision model configuration.

## [](#transformers.AltCLIPTextConfig)AltCLIPTextConfig

### class transformers.AltCLIPTextConfig

[](#transformers.AltCLIPTextConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/configuration_altclip.py#L32)

( vocab\_size = 250002hidden\_size = 1024num\_hidden\_layers = 24num\_attention\_heads = 16intermediate\_size = 4096hidden\_act = 'gelu'hidden\_dropout\_prob = 0.1attention\_probs\_dropout\_prob = 0.1max\_position\_embeddings = 514type\_vocab\_size = 1initializer\_range = 0.02initializer\_factor = 0.02layer\_norm\_eps = 1e-05pad\_token\_id = 1bos\_token\_id = 0eos\_token\_id = 2position\_embedding\_type = 'absolute'use\_cache = Trueproject\_dim = 768\*\*kwargs )

This is the configuration class to store the configuration of a [AltCLIPTextModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPTextModel). It is used to instantiate a AltCLIP text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AltCLIP [BAAI/AltCLIP](https://huggingface.co/BAAI/AltCLIP) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Examples:

```
>>> from transformers import AltCLIPTextModel, AltCLIPTextConfig

>>> 
>>> configuration = AltCLIPTextConfig()

>>> 
>>> model = AltCLIPTextModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.AltCLIPVisionConfig)AltCLIPVisionConfig

### class transformers.AltCLIPVisionConfig

[](#transformers.AltCLIPVisionConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/configuration_altclip.py#L142)

( hidden\_size = 768intermediate\_size = 3072projection\_dim = 512num\_hidden\_layers = 12num\_attention\_heads = 12num\_channels = 3image\_size = 224patch\_size = 32hidden\_act = 'quick\_gelu'layer\_norm\_eps = 1e-05attention\_dropout = 0.0initializer\_range = 0.02initializer\_factor = 1.0\*\*kwargs )

This is the configuration class to store the configuration of a [AltCLIPModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPModel). It is used to instantiate an AltCLIP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AltCLIP [BAAI/AltCLIP](https://huggingface.co/BAAI/AltCLIP) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import AltCLIPVisionConfig, AltCLIPVisionModel

>>> 
>>> configuration = AltCLIPVisionConfig()

>>> 
>>> model = AltCLIPVisionModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.AltCLIPProcessor)AltCLIPProcessor

### class transformers.AltCLIPProcessor

[](#transformers.AltCLIPProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/processing_altclip.py#L24)

( image\_processor = Nonetokenizer = None\*\*kwargs )

Parameters

-   [](#transformers.AltCLIPProcessor.image_processor)**image\_processor** ([CLIPImageProcessor](/docs/transformers/v4.30.0/en/model_doc/clip#transformers.CLIPImageProcessor)) — The image processor is a required input.
-   [](#transformers.AltCLIPProcessor.tokenizer)**tokenizer** ([XLMRobertaTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast)) — The tokenizer is a required input.

Constructs a AltCLIP processor which wraps a CLIP image processor and a XLM-Roberta tokenizer into a single processor.

[AltCLIPProcessor](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPProcessor) offers all the functionalities of [CLIPImageProcessor](/docs/transformers/v4.30.0/en/model_doc/clip#transformers.CLIPImageProcessor) and [XLMRobertaTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast). See the `__call__()` and [decode()](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPProcessor.decode) for more information.

This method forwards all its arguments to XLMRobertaTokenizerFast’s [batch\_decode()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.batch_decode). Please refer to the docstring of this method for more information.

This method forwards all its arguments to XLMRobertaTokenizerFast’s [decode()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.decode). Please refer to the docstring of this method for more information.

## [](#transformers.AltCLIPModel)AltCLIPModel

### class transformers.AltCLIPModel

[](#transformers.AltCLIPModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1478)

( config: AltCLIPConfig )

#### forward

[](#transformers.AltCLIPModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1605)

( input\_ids: typing.Optional\[torch.LongTensor\] = Nonepixel\_values: typing.Optional\[torch.FloatTensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonetoken\_type\_ids = Nonereturn\_loss: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → `transformers.models.altclip.modeling_altclip.AltCLIPOutput` or `tuple(torch.FloatTensor)`

The [AltCLIPModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AltCLIPModel

>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(
...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  
>>> probs = logits_per_image.softmax(dim=1)  ```

#### get\_text\_features

[](#transformers.AltCLIPModel.get_text_features)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1512)

( input\_ids: typing.Optional\[torch.Tensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.Tensor\] = Nonetoken\_type\_ids = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → text\_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The [AltCLIPModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoProcessor, AltCLIPModel

>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)```

#### get\_image\_features

[](#transformers.AltCLIPModel.get_image_features)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1559)

( pixel\_values: typing.Optional\[torch.FloatTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → image\_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

Parameters

-   [](#transformers.AltCLIPModel.get_image_features.pixel_values)**pixel\_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [AutoImageProcessor](/docs/transformers/v4.30.0/en/model_doc/auto#transformers.AutoImageProcessor). See [CLIPImageProcessor.**call**()](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor.__call__) for details.
-   [](#transformers.AltCLIPModel.get_image_features.output_attentions)**output\_attentions** (`bool`, _optional_) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
-   [](#transformers.AltCLIPModel.get_image_features.output_hidden_states)**output\_hidden\_states** (`bool`, _optional_) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
-   [](#transformers.AltCLIPModel.get_image_features.return_dict)**return\_dict** (`bool`, _optional_) — Whether or not to return a [ModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.

Returns

image\_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The image embeddings obtained by applying the projection layer to the pooled output of [AltCLIPVisionModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPVisionModel).

The [AltCLIPModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AltCLIPModel

>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features(**inputs)```

## [](#transformers.AltCLIPTextModel)AltCLIPTextModel

### class transformers.AltCLIPTextModel

[](#transformers.AltCLIPTextModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1386)

( config )

#### forward

[](#transformers.AltCLIPTextModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1405)

( input\_ids: typing.Optional\[torch.Tensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.Tensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Noneinputs\_embeds: typing.Optional\[torch.Tensor\] = Noneencoder\_hidden\_states: typing.Optional\[torch.Tensor\] = Noneencoder\_attention\_mask: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = None ) → `transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection` or `tuple(torch.FloatTensor)`

The [AltCLIPTextModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPTextModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoProcessor, AltCLIPTextModel

>>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")

>>> texts = ["it's a cat", "it's a dog"]

>>> inputs = processor(text=texts, padding=True, return_tensors="pt")

>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  ```

## [](#transformers.AltCLIPVisionModel)AltCLIPVisionModel

### class transformers.AltCLIPVisionModel

[](#transformers.AltCLIPVisionModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/altclip/modeling_altclip.py#L1156)

( config: AltCLIPVisionConfig )

The [AltCLIPVisionModel](/docs/transformers/v4.30.0/en/model_doc/altclip#transformers.AltCLIPVisionModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AltCLIPVisionModel

>>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> inputs = processor(images=image, return_tensors="pt")

>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  ```