BridgeTower

## [](#overview)Overview

The BridgeTower model was proposed in [BridgeTower: Building Bridges Between Encoders in Vision-Language Representative Learning](https://arxiv.org/abs/2206.08657) by Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan. The goal of this model is to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder thus achieving remarkable performance on various downstream tasks with almost negligible additional performance and computational costs.

This paper has been accepted to the [AAAI’23](https://aaai.org/Conferences/AAAI-23/) conference.

The abstract from the paper is the following:

_Vision-Language (VL) models with the TWO-TOWER architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BRIDGETOWER, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the crossmodal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BRIDGETOWER achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BRIDGETOWER achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BRIDGETOWER achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets._

![drawing](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/bridgetower_architecture%20.jpg) BridgeTower architecture. Taken from the [original paper.](https://arxiv.org/abs/2206.08657)

## [](#usage)Usage

BridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers. The goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder. In principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.

The [BridgeTowerProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) wraps [RobertaTokenizer](/docs/transformers/v4.30.0/en/model_doc/roberta#transformers.RobertaTokenizer) and [BridgeTowerImageProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) into a single instance to both encode the text and prepare the images respectively.

The following example shows how to run contrastive learning using [BridgeTowerProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) and [BridgeTowerForContrastiveLearning](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerForContrastiveLearning).

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning
>>> import requests
>>> from PIL import Image

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")
>>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")

>>> 
>>> scores = dict()
>>> for text in texts:
...     
...     encoding = processor(image, text, return_tensors="pt")
...     outputs = model(**encoding)
...     scores[text] = outputs```

The following example shows how to run image-text retrieval using [BridgeTowerProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) and [BridgeTowerForImageAndTextRetrieval](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerForImageAndTextRetrieval).

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
>>> import requests
>>> from PIL import Image

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
>>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

>>> 
>>> scores = dict()
>>> for text in texts:
...     
...     encoding = processor(image, text, return_tensors="pt")
...     outputs = model(**encoding)
...     scores[text] = outputs.logits[0, 1].item()```

The following example shows how to run masked language modeling using [BridgeTowerProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) and [BridgeTowerForMaskedLM](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerForMaskedLM).

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM
>>> from PIL import Image
>>> import requests

>>> url = "http://images.cocodataset.org/val2017/000000360943.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
>>> text = "a <mask> looking out of the window"

>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
>>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

>>> 
>>> encoding = processor(image, text, return_tensors="pt")

>>> 
>>> outputs = model(**encoding)

>>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())

>>> print(results)
.a cat looking out of the window.```

This model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).

Tips:

-   This implementation of BridgeTower uses [RobertaTokenizer](/docs/transformers/v4.30.0/en/model_doc/roberta#transformers.RobertaTokenizer) to generate text embeddings and OpenAI’s CLIP/ViT model to compute visual embeddings.
-   Checkpoints for pre-trained [bridgeTower-base](https://huggingface.co/BridgeTower/bridgetower-base) and [bridgetower masked language modeling and image text matching](https://huggingface.co/BridgeTower/bridgetower-base-itm-mlm) are released.
-   Please refer to [Table 5](https://arxiv.org/pdf/2206.08657.pdf) for BridgeTower’s performance on Image Retrieval and other down stream tasks.
-   The PyTorch version of this model is only available in torch 1.10 and higher.

## [](#transformers.BridgeTowerConfig)BridgeTowerConfig

### class transformers.BridgeTowerConfig

[](#transformers.BridgeTowerConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/configuration_bridgetower.py#L250)

( share\_cross\_modal\_transformer\_layers = Truehidden\_act = 'gelu'hidden\_size = 768initializer\_factor = 1layer\_norm\_eps = 1e-05share\_link\_tower\_layers = Falselink\_tower\_type = 'add'num\_attention\_heads = 12num\_hidden\_layers = 6tie\_word\_embeddings = Falseinit\_layernorm\_from\_vision\_encoder = Falsetext\_config = Nonevision\_config = None\*\*kwargs )

This is the configuration class to store the configuration of a [BridgeTowerModel](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerModel). It is used to instantiate a BridgeTower model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the bridgetower-base [BridgeTower/bridgetower-base](https://huggingface.co/BridgeTower/bridgetower-base/) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import BridgeTowerModel, BridgeTowerConfig

>>> 
>>> configuration = BridgeTowerConfig()

>>> 
>>> model = BridgeTowerModel(configuration)

>>> 
>>> configuration = model.config```

#### from\_text\_vision\_configs

[](#transformers.BridgeTowerConfig.from_text_vision_configs)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/configuration_bridgetower.py#L350)

( text\_config: BridgeTowerTextConfigvision\_config: BridgeTowerVisionConfig\*\*kwargs )

Instantiate a [BridgeTowerConfig](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerConfig) (or a derived class) from BridgeTower text model configuration. Returns: [BridgeTowerConfig](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerConfig): An instance of a configuration object

#### to\_dict

[](#transformers.BridgeTowerConfig.to_dict)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/configuration_bridgetower.py#L361)

( ) → `Dict[str, any]`

Dictionary of all the attributes that make up this configuration instance,

Serializes this instance to a Python dictionary. Override the default [to\_dict()](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig.to_dict).

## [](#transformers.BridgeTowerTextConfig)BridgeTowerTextConfig

### class transformers.BridgeTowerTextConfig

[](#transformers.BridgeTowerTextConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/configuration_bridgetower.py#L121)

( vocab\_size = 50265hidden\_size = 768num\_hidden\_layers = 12num\_attention\_heads = 12initializer\_factor = 1intermediate\_size = 3072hidden\_act = 'gelu'hidden\_dropout\_prob = 0.1attention\_probs\_dropout\_prob = 0.1max\_position\_embeddings = 514type\_vocab\_size = 1initializer\_range = 0.02layer\_norm\_eps = 1e-05pad\_token\_id = 1bos\_token\_id = 0eos\_token\_id = 2position\_embedding\_type = 'absolute'use\_cache = Trueclassifier\_dropout = None\*\*kwargs )

This is the configuration class to store the text configuration of a [BridgeTowerModel](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerModel). The default values here are copied from RoBERTa. Instantiating a configuration with the defaults will yield a similar configuration to that of the bridgetower-base [BridegTower/bridgetower-base](https://huggingface.co/BridgeTower/bridgetower-base/) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import BridgeTowerTextConfig

>>> 
>>> configuration = BridgeTowerTextConfig()

>>> 
>>> configuration```

## [](#transformers.BridgeTowerVisionConfig)BridgeTowerVisionConfig

### class transformers.BridgeTowerVisionConfig

[](#transformers.BridgeTowerVisionConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/configuration_bridgetower.py#L35)

( hidden\_size = 768num\_hidden\_layers = 12num\_channels = 3patch\_size = 16image\_size = 288initializer\_factor = 1layer\_norm\_eps = 1e-05stop\_gradient = Falseshare\_layernorm = Trueremove\_last\_layer = False\*\*kwargs )

This is the configuration class to store the vision configuration of a [BridgeTowerModel](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerModel). Instantiating a configuration with the defaults will yield a similar configuration to that of the bridgetower-base [BridgeTower/bridgetower-base](https://huggingface.co/BridgeTower/bridgetower-base/) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import BridgeTowerVisionConfig

>>> 
>>> configuration = BridgeTowerVisionConfig()

>>> 
>>> configuration```

## [](#transformers.BridgeTowerImageProcessor)BridgeTowerImageProcessor

### class transformers.BridgeTowerImageProcessor

[](#transformers.BridgeTowerImageProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/image_processing_bridgetower.py#L114)

( do\_resize: bool = Truesize: typing.Dict\[str, int\] = 288size\_divisor: int = 32resample: Resampling = <Resampling.BICUBIC: 3>do\_rescale: bool = Truerescale\_factor: typing.Union\[int, float\] = 0.00392156862745098do\_normalize: bool = Trueimage\_mean: typing.Union\[float, typing.List\[float\], NoneType\] = Noneimage\_std: typing.Union\[float, typing.List\[float\], NoneType\] = Nonedo\_center\_crop: bool = Truedo\_pad: bool = True\*\*kwargs )

Constructs a BridgeTower image processor.

#### preprocess

[](#transformers.BridgeTowerImageProcessor.preprocess)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/image_processing_bridgetower.py#L391)

( images: typing.Union\[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List\[ForwardRef('PIL.Image.Image')\], typing.List\[numpy.ndarray\], typing.List\[ForwardRef('torch.Tensor')\]\]do\_resize: typing.Optional\[bool\] = Nonesize: typing.Union\[typing.Dict\[str, int\], NoneType\] = Nonesize\_divisor: typing.Optional\[int\] = Noneresample: Resampling = Nonedo\_rescale: typing.Optional\[bool\] = Nonerescale\_factor: typing.Optional\[float\] = Nonedo\_normalize: typing.Optional\[bool\] = Noneimage\_mean: typing.Union\[float, typing.List\[float\], NoneType\] = Noneimage\_std: typing.Union\[float, typing.List\[float\], NoneType\] = Nonedo\_pad: typing.Optional\[bool\] = Nonedo\_center\_crop: typing.Optional\[bool\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonedata\_format: ChannelDimension = <ChannelDimension.FIRST: 'channels\_first'>\*\*kwargs )

Preprocess an image or batch of images.

## [](#transformers.BridgeTowerProcessor)BridgeTowerProcessor

### class transformers.BridgeTowerProcessor

[](#transformers.BridgeTowerProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/processing_bridgetower.py#L26)

( image\_processortokenizer )

Parameters

-   [](#transformers.BridgeTowerProcessor.image_processor)**image\_processor** (`BridgeTowerImageProcessor`) — An instance of [BridgeTowerImageProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor). The image processor is a required input.
-   [](#transformers.BridgeTowerProcessor.tokenizer)**tokenizer** (`RobertaTokenizerFast`) — An instance of \[‘RobertaTokenizerFast\`\]. The tokenizer is a required input.

Constructs a BridgeTower processor which wraps a Roberta tokenizer and BridgeTower image processor into a single processor.

[BridgeTowerProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) offers all the functionalities of [BridgeTowerImageProcessor](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) and [RobertaTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/roberta#transformers.RobertaTokenizerFast). See the docstring of [**call**()](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerProcessor.__call__) and `decode()` for more information.

#### \_\_call\_\_

[](#transformers.BridgeTowerProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/processing_bridgetower.py#L48)

( imagestext: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\]\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0pad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = Truereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = None\*\*kwargs )

This method uses [BridgeTowerImageProcessor.**call**()](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor.__call__) method to prepare image(s) for the model, and [RobertaTokenizerFast.**call**()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.__call__) to prepare text for the model.

Please refer to the docstring of the above two methods for more information.

## [](#transformers.BridgeTowerModel)BridgeTowerModel

### class transformers.BridgeTowerModel

[](#transformers.BridgeTowerModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1211)

( config )

Parameters

-   [](#transformers.BridgeTowerModel.config)**config** ([BridgeTowerConfig](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on top. This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`\_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.BridgeTowerModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1268)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Nonepixel\_values: typing.Optional\[torch.FloatTensor\] = Nonepixel\_mask: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Noneimage\_embeds: typing.Optional\[torch.FloatTensor\] = Noneimage\_token\_type\_idx: typing.Optional\[int\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = Nonelabels: typing.Optional\[torch.LongTensor\] = None ) → `transformers.models.bridgetower.modeling_bridgetower.BridgeTowerModelOutput` or `tuple(torch.FloatTensor)`

The [BridgeTowerModel](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerModel
>>> from PIL import Image
>>> import requests

>>> 
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> text = "hello world"
>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base")
>>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base")

>>> inputs = processor(image, text, return_tensors="pt")
>>> outputs = model(**inputs)
>>> outputs.keys()
odict_keys(['text_features', 'image_features', 'pooler_output'])```

## [](#transformers.BridgeTowerForContrastiveLearning)BridgeTowerForContrastiveLearning

### class transformers.BridgeTowerForContrastiveLearning

[](#transformers.BridgeTowerForContrastiveLearning)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1768)

( config )

Parameters

-   [](#transformers.BridgeTowerForContrastiveLearning.config)**config** ([BridgeTowerConfig](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss.

This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`\_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.BridgeTowerForContrastiveLearning.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1782)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Nonepixel\_values: typing.Optional\[torch.FloatTensor\] = Nonepixel\_mask: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Noneimage\_embeds: typing.Optional\[torch.FloatTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Truereturn\_dict: typing.Optional\[bool\] = Nonereturn\_loss: typing.Optional\[bool\] = None ) → `transformers.models.bridgetower.modeling_bridgetower.BridgeTowerContrastiveOutput` or `tuple(torch.FloatTensor)`

The [BridgeTowerForContrastiveLearning](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerForContrastiveLearning) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning
>>> import requests
>>> from PIL import Image
>>> import torch

>>> image_urls = [
...     "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg",
...     "http://images.cocodataset.org/val2017/000000039769.jpg",
... ]
>>> texts = ["two dogs in a car", "two cats sleeping on a couch"]
>>> images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]

>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")
>>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")

>>> inputs = processor(images, texts, padding=True, return_tensors="pt")
>>> loss = model(**inputs, return_loss=True).loss

>>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt")
>>> loss_swapped = model(**inputs, return_loss=True).loss

>>> print("Loss", round(loss.item(), 4))
Loss 0.0019

>>> print("Loss with swapped images", round(loss_swapped.item(), 4))
Loss with swapped images 2.126```

## [](#transformers.BridgeTowerForMaskedLM)BridgeTowerForMaskedLM

### class transformers.BridgeTowerForMaskedLM

[](#transformers.BridgeTowerForMaskedLM)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1550)

( config )

Parameters

-   [](#transformers.BridgeTowerForMaskedLM.config)**config** ([BridgeTowerConfig](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

BridgeTower Model with a language modeling head on top as done during pretraining.

This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`\_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.BridgeTowerForMaskedLM.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1566)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Nonepixel\_values: typing.Optional\[torch.FloatTensor\] = Nonepixel\_mask: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Noneimage\_embeds: typing.Optional\[torch.FloatTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = Nonelabels: typing.Optional\[torch.LongTensor\] = None ) → [transformers.modeling\_outputs.MaskedLMOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.MaskedLMOutput) or `tuple(torch.FloatTensor)`

The [BridgeTowerForMaskedLM](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerForMaskedLM) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM
>>> from PIL import Image
>>> import requests

>>> url = "http://images.cocodataset.org/val2017/000000360943.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
>>> text = "a <mask> looking out of the window"

>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
>>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

>>> 
>>> encoding = processor(image, text, return_tensors="pt")

>>> 
>>> outputs = model(**encoding)

>>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())

>>> print(results)
.a cat looking out of the window.```

## [](#transformers.BridgeTowerForImageAndTextRetrieval)BridgeTowerForImageAndTextRetrieval

### class transformers.BridgeTowerForImageAndTextRetrieval

[](#transformers.BridgeTowerForImageAndTextRetrieval)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1657)

( config )

Parameters

-   [](#transformers.BridgeTowerForImageAndTextRetrieval.config)**config** ([BridgeTowerConfig](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the \[CLS\] token) for image-to-text matching.

This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`\_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.BridgeTowerForImageAndTextRetrieval.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bridgetower/modeling_bridgetower.py#L1668)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneattention\_mask: typing.Optional\[torch.FloatTensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.LongTensor\] = Nonepixel\_values: typing.Optional\[torch.FloatTensor\] = Nonepixel\_mask: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.FloatTensor\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Noneimage\_embeds: typing.Optional\[torch.FloatTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = Nonelabels: typing.Optional\[torch.LongTensor\] = None ) → [transformers.modeling\_outputs.SequenceClassifierOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or `tuple(torch.FloatTensor)`

The [BridgeTowerForImageAndTextRetrieval](/docs/transformers/v4.30.0/en/model_doc/bridgetower#transformers.BridgeTowerForImageAndTextRetrieval) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
>>> import requests
>>> from PIL import Image

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

>>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
>>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

>>> 
>>> scores = dict()
>>> for text in texts:
...     
...     encoding = processor(image, text, return_tensors="pt")
...     outputs = model(**encoding)
...     scores[text] = outputs.logits[0, 1].item()```