BridgeTower
/

bridgetower-large-itm-mlm-gaudi

@@ -25,7 +25,7 @@ BridgeTower got accepted to [AAAI'23](https://aaai.org/Conferences/AAAI-23/).
 The abstract from the paper is the following:
 Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.
-## Intended uses & limitations(TODO)
 ### How to use
@@ -79,10 +79,6 @@ print(results)
 #.a cat looking out of the window.
 ```
-### Limitations and bias
-TODO
 ## Training data
 The BridgeTower model was pretrained on four public image-caption datasets:
@@ -95,10 +91,6 @@ The total number of unique images in the combined data is 4M.
 ## Training procedure
-### Preprocessing
-TODO
 ### Pretraining
 The model was pre-trained for ___ steps on an "Intel AI supercomputing cluster" using 512 Gaudis and 128 Xeons with a batch size of 4096.

 The abstract from the paper is the following:
 Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.
+## Intended uses & limitations
 ### How to use
 #.a cat looking out of the window.
 ```
 ## Training data
 The BridgeTower model was pretrained on four public image-caption datasets:
 ## Training procedure
 ### Pretraining
 The model was pre-trained for ___ steps on an "Intel AI supercomputing cluster" using 512 Gaudis and 128 Xeons with a batch size of 4096.