GIT: A Generative Image-to-text Transformer for Vision and Language
Authors: Jianfeng Wang
Zhengyuan Yang
Xiaowei Hu
Linjie Li
Kevin Lin
Zhe Gan
Zicheng Liu
Ce Liu
Lijuan Wang
Venue: Accepted by TMLR ()

Abstract: In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder
under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on numerous challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.

Published in Transactions on Machine Learning Research (11/2022)
GIT: A Generative Image-to-text Transformer for Vision and Language
Jianfeng Wang
jianfu@microsoff.. com zhengyong fmicrosofff com rioowei. hu muccrosoff. com lindsey. l&@ sémicrosoft. com keli deli@miccosoft.. com zhe. gan @ @microsoft. com zliu miccrooot... com ce. li Siirrmooooo.... com kijuan@@micosoff... com
Zhengyuan Yang
Xiaowei Hu
Linjie Li
Kevin Lin
Zhe Gan
Zicheng Liu
Ce Liu
Lijuan Wang
Microsoft Cloud and AI
Reviewed on OpenReview:
https: openreview. net/ forum? id= by b4tMNNOOJG
Abstract
In this paper, we design and train Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While genera- tive models provide consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multis (uni/multi-modal encoder (eolderr and depends on external modules such as object detectors/ ttagggsss/ and optical character recogni- lion (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on numerous challenging benchmarks with large margin. For instance, our model surpasses the human performance for the first time Text (138.2 vs. 125.5 in CIDEr). Furthermore, present scheme of generation-baser image classification and scene text recognition, achieving decent performance on standard benchmarks.
1
Introduction
Table 1: Comparison with prior SOTA on image/video captioning and question answering (QA) tasks. *: evaluated
the public server. CIDEr scores are reported for Captioning tasks. Prior SOTA: COCO Zhang al.] 2021a)
nocaps (Yu et al.] 2022), VizWiz-Caption (Gong et al.] 2021), Text (Yang et al.]
(Lin ST-VQA [Biten et al.
2021c).
2022).
VizWiz- VQA (Alayrac al. 2022
O.ROOCRROO VQA (Biten al. 2022),MSVD MSVVD
2021 MSRVTT (Seo
et al.
2022
VATEX (Tang et al.) 2021),1 T VC TTang et al.
2021), MSVD-QA (Wang et al. 2022a) TGIF-
Zellers
el
2021
Text Recog.
[Lyu al.)
2022
Details of GIT2 are presented supplementary materials.
Image caplioning
Image QA
Video captioning
Video QA
Text Rec.
COCO
nocaps
VizWiz
TextCCaps
ST- VT-VQA
VizWiz
ICR-VQA
MSVD
MSRVTT
VATEX
TVC*
MSVD-QA
TGIF- TGIF-Framm Frame
Avg on
Prior SOTAI
138.7
120.6
94.1
109.7
69.6
65.4
67.9
120.6
60
86.5
64.5
48.3
69.5
93.8
GIT (ours)
148.8
123.4
114.4
138.2
69.6
67.5
68.1
180.2
73.9
93.8
61.2
56.8
72.8
92.9
+10.1
+2.8
20.33
+28.5
+0.0
+2.1
+0.2
-5.6
+13.9
+7.3
-3.3
+8.5
+3.3
-0.9
G1T2 (ours)
149.8
124.8
120.8
145.0
75.8
70.1
70.3
185.4
75.9
96.6
65.0
58.2
74.9
94.5
+11.1
+ 4.2
-26.7
+35.3
+6.2
+4.7
+2.4
64.8
+15.9
+10.1
+0.5
+9.9
+5.4
+0.7
Prior SOTA: among all the numbers reported publications before 8/2022, as far know.

---

Published in Transactions on Machine Learning Research (11/2022)
scene text
long text
curved text
blurry text
occluded text
table
chart
549.500 131
Shakesspeare, successful glover originally from Snitterflac Manvitiinnne, Mary Arden.
025-240
11 259-199
book by henry
paper that says shakespeare was the son of john shakespeare, alderman and successful glaver originally from soltterfleld in warvickshire, and mary arden.
poster for the nationsal administrative professionals day shown.
person holding bottle bacon
baseball player with the blue jeys on profilee matrix shown his jersey is on table. hit a ball.
chevron comgetitive
pie chart with 28% over
titled regalo de los reves magos.
food
money bill
logo
landmark
character
celebrity
product tag/photo
Friskiest
of chinese food called mapo tafu.
person holding five dollar bill with a picture abraham lincaln the front.
dellaa plane parked on the airport.
white mahal is reflected in
A slaar movie poster with a Darth Vader helmet.
Marlivn Monros photo with black background.
apple with green label that says 94133.
putina triskies franso adultos cat food.
Figure 1: Example captions generated by GIT. The model demonstrates strong capability of recognizing scene text, tables/charts. food, banknote, logos, landmarks, characters, products, etc.
Tremendous advances have been made in recent years on vision-language (VL) pre-training, especially based on the largo-scale data of image-text pairs, e.g., CLIP (Radford et al.] 2021). Florence (Yuan et al.) 2021). and SimVLM (Wang et al. 2021bb. The learned representation greatly boosts the performance on various downstream tasks, such image captioning (Lin et al. 2014). visual question answering (VQA) (Goyya et al.
2017 and image-text retrieval.
During pre-training, Masked Language Modeling (MLM) and Image-Text Matching (FTM) tasks have been widely used
[Wang et al.
2020 Fang et al. 2021c Li et al. 2020 Zhang et al. 2021a; Chen et al., 2020b Dou
et al.
2021 Wang et al.
2021a Kim et al. 2021 However, these losses are different from the downstteea
tasks,
and task-specific adaptation has to be made. For example, ITM removed for image captioning (Wang
et al.
2021a
et al.
2020), and an extra randomly initialized multi-layer perceptron is added for VQA (Wang
ct al.
2021b
Li al. 2020). To reduce this discrepancy, recent approaches (Cho et al. 2021 Wang et al.
Yang et al.J 20221 Wang el al., 2022b) have allempted to design unified generalive models for pre-training, as most VL tasks can be cast as generation problems. These approaches typically leverage a multi-modal encoder and a text decoder with carcful design on the text input and the text target. To further push the frontier of this direction, we present a simple Generative Image-to-text Transformer, named GIT, which consists only of one image encoder and one text decoder. The pre-training task just to map the input image to the entire associated text description with the language modeling objective. Despite its simplicity, GIT achieves new state of the arts across numerous challenging benchmarks with large margin, as summarized in Table 1
2021b
The image encoder is Swin-like vision transformer (Dosovitskiy et al. 2021 Yuan et al. 2021 pre-trained on
massive image-text pairs based on the contrastive task (Jia et al. 2021 Radford et al. 2021 Yuan et al. 2021). ).
This eliminates the dependency on the object detector, which is used in many existing approaches (Andersor
et al. 2018 Li et al., 20200 Wang et al. 2020 Zhang et al. 2021a Chen et al. 2020b Fang al. 2021c). To extend to the video domain, we simply extract the features of multiple sampled frames and concalenate them the video representation. The text decoder is transformer network to predict the associated text. The entire network is trained with the language modeling task. For VQA, the input question is treated as a text prefix, and the answer is generated in an auto-regressive way. Furthermore, we present a new generation-based scheme for ImageNet classification, where the predicted labels come directly from our generative model without pre-defining the vocabulary.
The approach is simple, but the performance is surprisingly impressive after we scale up the pre-training data and the model size. Fig. showw captions generated by the GIT fine-tuned with TextCaps. The samples
2

---

Published in Transactions on Machine Learning Research (11/2022)
demonstrate the model's strong capability of recognizing and describing scene text, tables, charts, food, banknote, logos, landmarks, characters, celebrities, products, elc., indicating that our GIT model has encoded rich multi-modal knowledge about the visual world.
Our main contributions are as follows.
We present GIT, which consists of only one image encoder and one text decoder, pro-trained on 0.8 billion image-text pains with the language modeling lask.
We demonstrate new state-of-the-art performance over numerous tasks on image/vides captioning and QA (Table D without the dependency on object detectors, object tags, and OCR. On TextCaps, we surpass the human performance for the first time. This implics that a simple network architecture can also achieve strong performance with scaling.
We demonstrate that GIT pre-trained on the image-text pairs capable of achieving new state-of-the-art. performance even on video tasks without video-dedicated encoders.
We present new scheme of generation-based image classification. On ImageNet-IK, we show a decent performance (88.79% top-1 accuracy) with our GIT.
2 Related Work
In VL pre-training, multi-task pre-training has been widely used to empower the network with multiple or enhanced capabilities. For example, MLM and ITM are widely adopted pre-training tasks (Li et al. 2020
Kim et al.) 2021 Zhang et al.] 2021a Wang et al. 2020 Xue et al. 2021: Lu et al. 2019 Tan & Bansal 2019). Recently, the image-text contrastive loss has also been added in Yu et al. (2022): Li et al. (021a));
Wang et al.] (2021a). Since most VL. tasks can be formulated as the text generation task @Cho et al. 2021 ), a
single generation model can be pre-trained to support various downstream tasks. The input and output texts are usually carefully designed to pre-train such a generation model. For example in Cho et al. (202)), the text is properly masked as the network input and the goal is to recover the masked text span. SimVLM (Wang 2021b) randomly splits text sentence into the input and the target output. In these methods,
ct al. multi-modal transformer encoder is utilized to incorporate the text inputs before decoding the output.
For image representation, Faster RCNN has been used in most existing approaches (Anderson et al. 2018
ct al. 2020 Wang et al. 2020 Zhang et al.] 2021a Chen ct all. 2020b Fang et al. 2021c) to extract the
region uealures. Recently, a growing interest is in dense representation (Huang el al.] 2020; Wang et al., 2021b Kim et al. 2021 Fang et al. 2021b: Dou et al. 2021 Li et al.)
2021a) from the feature map, which requires no bounding box annotations. Meanwhile, it is easy train the entire network in an end-to-end way.
In addition to the representation from the feature map, object tags (Li et al. 2020 Wang et al. 2020 Zhang
et al. 2021a Cornia et. al. 2021 Fang et al. (021b) are leveraged to facilitate the transformer to understand tasks, OCR is invoked to generate the scene text as additional network input, e.g., in Hu et al. (2020); Yang et al. (2021c). For the text prediction, A transformer network is typically used, which can incorporate the cross-attentia module to fuse the image tokens, e.g., Cho et al. (2021): |Alayrac et al. (2022); Yang et al. (2021b): Yu et al. (2022), or only the self-attention modules where the image tokens are concatenated with the text tokens, e.g., Li al. 2020)
the context, especially the novel objects. For scene-text-relat
Chen et al. (2020b Zhang et al. (2021a): Wang et al.
(200 Fang et al. (021bb)
Along the direction of scaling on VL tasks, LEMON Hu et al. 2021a) studies the behavior of the detector- based captioning model with MLM. CoCa Yu et al.
2022 studies different model sizes, but on the same
pre-training data. In this paper, we present a comprehensive study on 9 various benchmarks (3 in main paper and 6 in supplementary materials, image/video captioning & QA tasks) with different model sizes and 3 different pro-training data scales (9 data points for cach benchmark).
3 Generative Image-to-text Transformer
With largo-scale image-text pairs, our goal is to pre-train a VL model which is simple yet effective to benefit image/video captioning and QA tasks. As the input is the image and the output is the text, the minimal set.
3

---

Published in Transactions on Machine Learning Research (11/2022)
tscsun radio
12
54
FOs
Text decoder
N x
Text decoder
Feed forward
Multi-head self-attention
BOS what time
Q: what time it?
(b) VQA
BOS
tscsun
radio
Image encoder
Tokenize & Embed
temporal embedding
temporal embedding
tecsun radio with the time of 12: 54.
Image encoder
Image encoder
Frame 1
Frame
(a) Pre-training/captiinnng
(c) Video
Figure 2: Network architecture of our GIT, composed of one image encoder and one text decoder. (a): The training task in both pre-training and captioning is the language modeling task to predict the associated description. (b): In VQA, the question is placed as the text prefix. (c): For video, multiple frames are sampled and encoded independently. The features are added with an extra learnable temporal embedding (initialized as 0) before concatenation.
of components could be one image encoder and one text decoder, which are the only components of our GIT as illustrated in Fig.
3.1 Network Architecture
The image encoder is based the contrastive pre-trained model (Yuan et al., 2021 ). The input is the image and the output is compact 2D fcature map, which is flattened into list of features. With an extra linear layer and layernorm layer, the image fealures projected into D dimensions, which are the input to the text decoder. We use the image encoder pre-trained with contrastive tasks because recent studies show superior performance with such image encoder, c.g. Yuan et al. (2021): Dou et al. (2021 Alayrac et al. (2022). Sec 4.6 and supplementarr materials, we also observe the VL performance boosts signilicantly with stronger image encoder. This consistent with the observation object detection- approaches,
e.g. in Wang al. (2020): Zhang al. (2021a). The concurrent work of CoCa (Yu et al. 2022) unifies the contrastive task and the generation task. as pre-training phase. Our approach is equivalent to separating the two lasks sequentially: (i) using the contrastive task to pre-train the image encoder followed by (fi) using the generation task to pre-train both the image encoder and text decoder.
The text decoder a transformer module to predict the text description. The transformer module consists of multiple transformer blocks, each which is composed of one self-attention layer and one leed-forward layer. The text is tokenized and embedded into dimensions, followed by an addition of the positional encoding and a layernorm layer. The image features are concatenated with the text embeddings the input to the transformer module. The text begins with the [BDS] token, and is decoded in auto-regressive way until the [EOS] token or reaching the maximum steps. The seq2seq attention mask as in Fig. 3 is applied such that the text token only depends on the preceding tokens and all image tokens, and image tokens can attend to each other. This is different from unidirectiona attention mask, where not every image token can rely on all other image tokens.
Instead of well initializing the image encoder, we randomly initialize the text docoder. This design choice is highly motivated from the experiment studies of Wang al. 2020), in which the random initialization shows
4

---

Published in Transactions on Machine Learning Research (11/2022)
similar performance, compared with the BERT initialization. This could be because the BERT initialization cannot understand the image signal, which is critical for VT. tasks. Without dependency of the intialization, we can easily explore different design choices. The concurrent work of Flamingo (Alayrac et al. 2022) employs a similar architecture of image encoder + text decoder, but their decoder is pro-trained and frozen to preserve the generalization capability of the large language model. In our GIT, all parameters are updated to better fit the VL tasks.
An alternative architecture is the pros--attention-assee decoder to in- corporate the image signals instead of concatenation with self-attention. Empirically as shown in supplementary material (Appendix G.2), with large-scale pre-training, we find the self-attention-baseed decoder achieves better performance overall, while in small-scale setting, the cross-attention- based approach wins. plausible explanation is that with sufficient training, the decoder parameters can well process both the image and the text, and the image tokens can be better updated with the self-attention for text generation. With cross- attention, the image tokens cannot attend to each other.
image
text
1
1
1
0
0
0
image
1 1
1
O
0
0
1
1
0
0
0
| 1
1
0
0
text
1
1
1
1
0
1
1
1
1
1
3.2 Pre-training
Figure 3: seq2seq attention mask
applied to the transformer. (i, j) is 1, the i-th output can depend on the j-th input; otherwise, not.
For each image-text pair, let / be the image, yo, E {1,.... N be the text tokens, yo be the [BOS] token and UN+1 be the [EDS] token. We apply the language modeling (LM) loss to train the model. That is,
N+1 Σ
1
CE(,,,(Mu Aug. = 0, 0.. - 1))),
(1)
N+1
where CE is the cross-entropy loss with label smoothing of 0.1.
An alternative choice MLM, which predicts typically 15% of imput tokens in each iteration. To predict all tokens, we have to run at least 1/0.15 6.7 epochs. For LM, each itcration can predict all tokens, which is more efficient for large-scale pre-training data. In Hu al. (2021a), the ablation studies also show that LM can achieve better performance with limited epochs. In our large-scale training, the number of epoch is only 2 due to computational resource limitation, and thus we choose LM. Meanwhile, most of the recent large-scale language models are also based on LM, e.g. Brown et al. (2020): Chowdhery et al. (2022).
Without the image input, the model is reduced to a decoder-only language model, similar to GPT3 (Bronn et al. 2020) in the architecture wise. Thus, this design also enables the possibility to leverage the text-only data enrich the decoding capability with a scaled-up decoder. We leave this as future work.
3.3 Fine-tuning
For the image captioning task, as the training data format is the same as that in pre-training, we apply the same LM task to fine-tune our GIT.
For visual question answering, the question and the ground-truth answer are concatenated as a new special caption during the fine-tuning, but the LM loss is only applied on the answer and the [EDS] tokens. During inference, the question interpreted as the caption prefix and the completed part the prediction. Compared with the existing approaches (Wang et al. 2021a b. Zhang et al. 2021a Li et al. 2022b) for VQAv2 (Goyal et al. 2017), our model is generative without pre-defining the candidate answers, even in inference. This imposes more challenges as the model has to predict at least two correct tokens: one for the answer and another for [EOS]. In contrast, the existing work pre-collects the answer candidate, recasts the problem as a classification problem, and only needs to predict once. However, considering the benefit of the free-form answer, we choose the generative approach. Due to difficulty of the generative model, we observe slightly worse performance on VQAv2 than the discriminative existing work. For the sceno-text related VQA tasks, Hu et al. 2020 typically leverages the OCR engine to generate the
existing approaches (Yang et al. 2021c
5

---

Published in Transactions on Machine Learning Research (11/2022)
scene text and use dynamic pointer network to decide the current output token should be OCR the general text. Here, our approach depends on no OCR engine, and thus no dynamic pointer network. Empirically, we find the model gradually learns how to read the scene text with large-scale pre-trat pre-training, and our model achieves new SoTA performance on these tasks.
Our model not specifically designed for the video domain, but we find our model can also achieve competitive or even new SOTA performance with simple architecture change. That is, we sample multiple frames from each video clip, and encode each frame via the image encoder independently. Afterwards, we add learnable temporal embedding (initialized zeros), and concatenate the features from sampled frames. The final reprosentation is used in a similar way as the image representation for captioning and question answering.
We also apply our generation model to the image classification lask, where the class names are interpreted as image captions, and our GIT is fine-tuned to predict the result in an auto-regressive way. This is different from existing work which normally pro-defines the vocabulary and uses a lincar layer (with softmax) predict the likelihood of each category. This new generation-based scheme is beneficial when new data and new categories are added to the existing dataset. In this case, the network can continuously train the new data without introducing new parameters.
4 Experiments
4.1
Setting
We collect 0.8B image-text pairs for pre-training, which include COCO
Lin et al.
2014). Conceptual Captions
(CC3M) (Sharma et al. 2018). SBU (Ordonez et al. 2011). Visual Genome (VG) (Krishna et al.] 2016),
Conceptual Captions (CC12M) (Changpinyo et al.] 2021 ALT200M (Hu et al.
2021a), and an extra 0.6B
data following a similar collection procedure in Hu ct al. (2021a). The image encoder is initialized from the pre-trained contrastive model (Yuan et al.) 2021). The hidden dimension (D) is 768. The text decoder consists of mandomly-inttialinn
transformer blocks. The total number of model parameters is 0.7 billion. The learning rates of the image encoder and the decoder are 1e and 5e-5; respectively, and follow the cosine decay to O. The total number of epochs is 2. During inference, the beam size is 4 and the length penalty (Wu el al. 2016) is 0.6 by default.
Supplementary materials show results on two smaller model variants (GIT B and GIT,) and one even larger model (GIT2) with full details. When comparing with existing approaches, the reference numbers are the best one reported in the corresponding paper unless explicitly specified.
4.2 Results on Image Captioning and Question Answering
We comprchensively evaluate the captioning performance on the widcly-usod Karpathy split (Karpathy & Li, 2015) of COCO [Ln et al., 2014) and Flickr30K (Young al. 2014)), the COCO test set,, nocaps (Agrawal et al. 2019 which focuses novel objects, TextCaps (Sidorov et al. 2020) which focuses on scene-text understanding, and CiapitCasssions (Guari et al.] 2020) which focuses on the real use case by the vision- impaired people. The results in CIDEr (Vedantam et al. (0115) are shown in Table 2 and 3 From the results, we can see our model achieves the new SOTA performance on all these metrics except on COCO Karpathy test. On nocaps, compared with CoCa YYu et al.) 2022). our model is much smaller in the model size (0.7B vs 2.1B), but achieves higher performance (123.0 vs 120.6 in CIDEr). On Textcaps, our solution outperforms the previous SOTA (TAP Yang et al.) (2021c)) by a breakthrough margin (28.5 points in CIDEr), and also surpassos the human performance for the first time. For zero/fewe ferr/ffwwshot evaluation as shown in Table our model can significantly benefit from more shots. With 32-shots, our approach is also better than Flamingo.
On VQA, the evaluation benchmarks include VQAv2 (Goyal et al.; 2017). Vex Singh et al. 2019 VizWiz-VQA (Gurari et al. 2018). ST-VQA (Biten et al., 2019), and OCR-VQA Mishhaa et al. 2019). Before fine-tuning the model, we run an intermediate fine-tuning on the combination of the training data of VQAv2,
TextVQA, ST-VQA, OCR-VQA, VizWiz- VQA, Visual Genome QA (Krshha et al. 2016). GQA (Hudson &
We compare all approaches including using external image-text datasets.

---

Published in Transactions on Machine Learning Research (11/2022)
Table 2: Results on image captioning. *: the nubmers are from Sidorov al. 2020 ; CE: cross-entropy optimization.
All numbers are CIDEr scores, and other metries are shown in supplementary materials. winner entry of the CVPR
2021 workshop challenge Anc.-Cap.: Xu al. (2021) AoANet: [uuang al. [2019) BUTD: Anderson et al. (2018)
CoCa:
Yu et al. (2022), DistilIVLM: |Fang et al. (2021c). Flamingo: Allayrac et al. (2022), Human: [Agrawal et al.
(2019
LEMON: Hu et al.
(2021a),
M4C-Cap.: Hu et al. (2020)
MiniVLM: Wang al. (2020). MTMA: Gong et al.
(2021
OFA: Wang et al.
(2022b).
OSCAR: Li et al. [2020], UFO: Wang al. [2021a] Universal (Cornia
et al.
2021)
ViTCCa: Fang et al.
2021b), VinVL:
Zhang et al.
2021a
VIVO: Hu et al. (0221 SimVLM: Wang
et al.
2021b1.
TAP: Yang et al.
2021cp.
Method
CE
Method
C
Method
Test
Method
Test
MiniVLM
119.8
BUTD
120.5
OSCAR
80.9
BUTD*
33.8
DistilIVLM
120.8
VinVL
138.7
Human
85.3
AoANet*
34.6
ViTCap
125.2
VIVO
86.6
MAC-Cap.
81.0
GIT
148.8
OSCAR
127.8
VinVL
92.5
Anc.-Cap.
87.4
VinVL
130.8
(b) COCO test (c40)
UFO
92.3
ТАР
103.2
UFO
131.2
SimVLM
115.2
TAP#
109.7
Flamingo
138.1
Method
test-std
LEMON
114.3
Human
125.5
LEMON
139.1
UniversalCap
119.3
MTMA
94.1
GIT
138.2
SimVLM
143.3
CoCa
120.6
CoCa
143.6
GIT
114.4
TextCaps
GIT
123.4
OFA
145.3
(c) VizWiz-Captions
(d) nocaps
GIT
144.8
(a) COCO Karp.
Table 3: Jero/Few/Puull-iii evaluation Flickr30K with Karpathy split.
Shot
16
32
290 (1%)
full
Zhou el al. 2020
68.5
Flamingo GIT
67.2
78.9
75.4
49.6
78.0
80.5
86.6
98.5
Manning 2019 and OK-VQA (Marino et al. 2019 To avoid data contamination, we remove the duplicate images of the test and validation set of the target benchmarks. As illustrated in Table we achieve new SOTA on VizWiz- VQA and OCR-VQA, and same performance with prior SOTA of LaTr (Biten et al. 2022) ST-VQA. Compared with the concurrent work of Flamingo (Alayrac et al.) 2022).
achieve higher accuracy (+5.4) on VQA and lower (-3.29) VQAv2. Note that Flamingo's model size is 80B, which is 114 times of ours (0.7B). On VQAv2, we observe that our model performs worse in 1.5 points than the discriminative model of Florence (Yuan et al. 2021), which shares the same image encoder. The reason might be the increased difficulty of the generative model. That is, each correct answer requires at least two correct predictions (answer and [EOS]; 2.2 on average), while the discriminative model requires only one correct prediction. In (Wang et al., 2021b). the ablation study also shows the better performance by around point than the discriminative counterpart. Another reason could be that the model of Florence for VQA leverages RoBerta Liu et al. 2019 as the text encoder, which implicitly uses the text-only data to improve the performance.
4.3 Results on Video Captioning and Question Answering
On the video captioning task, the performance is evaluated MSVD (Chen & Dolan 2011 with the widely-used splits from Venugopalan et al.] (2014), MSRVTT (Xu et al. 2016). YouCook2 (Zhou et al. 2018 (results in supplementary materials.) VATEX (Wang el al. 2019b), and TVC (Lei et al. 2020) (results in supplementary materials)). On VATEX, the performance is evaluated on both the public test and private test (evaluated on the server). Video QA evaluated on MSVD-QA (Xu al. 20177] Chen Dolan 2011);
MSRVTT-QQ (Xu al. 2017 2016), , and TGIF-Frae Frame (Jang et al. 2017), which are all open-ended tasks.
The results shown in Table 5 and Table for captioning and QA, respectively. Although model not.
7

---

Published in Transactions on Machine Learning Research (11/2022)
Table Results on visual question answering. (a): for VQÁv2, approaches are divided according to whether the answer vocabulary is pre-defined (Closed) not (Open) during inference. The model with closed vocabulary classification model or generation model with constrained outputs, e.g., [Wang et al. (2022b); [i et al. (2022b).
The two numbers in parenthesis are the number of parameters and the number of images (the images for pre-trained
modules are not counted) VL pretraining. (b): for Text Mia (Qiao et al. 2021 # is the winner entry of
Tex Challenge 2021 with fine-tuned T5-3B (Raffel et al. 2020 model. (c): **; winner entry of 2021 VizWiz
Grand Challenge Workshop. ALBEF: Li et al.
(021a), BLIP: Li et al.
2022b),
Mishra et al.
2019
CLIP-ViLL [Shen et al. (2021). CoCa: Yu et al. (2022). CRN:
Liu et al. (2020a). Flamingo: Alayrac et al.
2022
Florence: Yuan et al. (2021), ), LaAP-Net: Han et al. (2020). LaTr: Biten et al. (2022)
MAC: [Hu et al. (2020)
M4C:
Hu et al. (2020), METER: [Dou et al. (2021), Mia: QQao al.
(2021). Pllgg: Li al.
2022a
OSCAR:
et al.
2020p.
OFA: Wang et al.) (2022b), UFO: Wang et al. (2021a).
UNITER: (Chen et al.
2020b),
UNIMO: Li
et
c021cc.
SA-M4C: Kant et (2020) SimVLM: Wang al. (2021b)
SMA Gao al.
2020
SMA:
Gao al.
(2020
TAP:
Yang et al. (021c)).
VinVL: Zhang et al.
(2021a).
VILLA: Gan et al. 2020
MettholarMetthod
lest-sid
Method
Lest
Method
Test ANLS
M4C
46.2
OSCAR
73.82
M4C
40.46
SMA
46.6
UNITER
74.02
LaAP-Net
41.41
CRN
48.3
VILLA
74.87
SA-M4C
44.6
LaAP-Net
48.5
UNIMO
75.27
SMA
45.51
SA-MAC
50.4
ALBEF
76.04
TAP
53.97
TAP
59.7
VinVL
76.60
Flamingo
54.1
LaTr
69.6
UFO
76.76
Mia
73.67
Closed
CLIP-ViL
76.70
GIT
69.6
CIT
59.75
METER
77.64
(d) ST-VQA
(b) Tex
BLIP
78.32
SimVLM (-. 1.8B)
80.34
Method
test
Florence (0.9B, 14M)
80.36
Method
test
BLOCK-CNN+WDD
48.3
mPlug (0.6B, 14M)
81.26
Liu al.
2021 ##
60.6
MCC
63.9
OFA (0.9B, 54M)
82.0
LaAP-Net
61.1
Flamingo
65.4
CoCa (2.1B, 4.8B)
82.3
LaTr
67.9
CIT
67.5
Flamingo (80B, 2.3B)
82.1
GIT
68.1
Open
GIT (0.7B, 0.8B)
78.81
(c) VizWiz-QA
(e) OCR-VQA
(a) VQAv2
dedicated for vidco tasks, our model achieve new SOTA on MSRVD, MSRVTT, and VATEX for captioning and MSVD-QA and TGIF-Frame for QA. For example VATEX private test, our results even better (93.8 vs 86.5) than CLiP4Caption++ (Tang et al. 2021). which relies model ensemble and additional subtitle input. This is also better than Flamingo [Alayrac et al. 2022 (84.2) with 80B parameters.
4.4 Results on Image Classification
We fine-tune GIT on ImageNet-Ik. Each category is mapped to a unique class name, and the prediction is correct only if it is exactly matched with the ground-truth label subject to more or fewer whitespaces As shown in Table 7 our approach
achieve descent accuracy without pre-defining the vocabulary. Compared with Florence (Yuan et al. 2021 (same image encoder), our approach is worse in about 1.2 points. The reason might be similar to the case VQAv2. That is, the generative approach needs to predict more tokens correctly to make one correct prediction, which increases the difficulty.
Zero-shot Zero-shot/Few-shle Fewwshott The result shown in Table With no knowledge of the vocabulary, the pretrained GIT cannot infer the expected vocabulary, and thus the exactly-match accuracy only 1.93% (in the column of equal). However, we relax the requirement and take correct if the prediction contains the ground-truth, the accuracy is 40.88% (in the column of in), which shows the predicted caption well identify the image content. If we have the vocabulary a prior and limit the output lokens to be within the vocabulary, the accuracy drops to 33.48% (in the column of voc-prior).. This may suggest the network is loss natural to
3 rpredceep 1, = geplacce )
8

---

Published in Transactions on Machine Learning Research (11/2022)
Table 5: Results on video captioning. 9: model ensemble: f: with the subtitle as additional input. Cap.
Tang
et al. 2 GRU-EVE: [Aalaq et al. 2019 MGSA: [Chen & Jiang (2019) MGSA: [Chen & Jiang 2019
MV-GPT:
Seo et al. (2022) PickNet: Chen et al. (2018) PMI-CAP: Chen et al. (2020a) SibNet: Liu et al.
(2020b)
OA-BTG:
Zhang & Peng 2019) ORG-TRL: [hhan et al. (2020) OpenBook: Zhang et al. (2021b) POS+
POS+VCT:
VCT:
Hou et al.
(2019
POS+CG: Wang et al. 2019ah SAAT: Zheng et al. (2020). STG-KD:
Pan el al. (2020) SwinBERT:
Lin el al.
2021
Support-set: Patrick
al. (2021
VaTeX: [Wang al. (2019b)
VALUE:
Li et al. (2021b)
Method
B@4
C
Method
B@4
C
Method
C
PickNet
52.3
76.5
SAAT
39.9
51.0
VaTeX
45.1
GRU-EVE
47.9
78.1
MGSA
42.4
47.5
OpenBook
57.5
SAAT
46.5
81.0
POS+VCT
42.3
49.1
VALUE"
58.1
MGSA
53.4
86.7
SibNet
40.9
17.5
SwinBERT
73.0
POS+VCT
52.8
87.8
POS+CG
42.0
48.7
C.4Cap. ET
85.7
SibNet
54.2
88.2
OA-BTG
41.4
46.9
GIT
91.5
POS+CG
52.5
88.7
STG-KD
40.5
47.1
OА-BTG
56.9
90.6
Support-sel
38.9
48.6
(a) VATEX public test
STG-KD
52.2
93.0
PMI-CAP
42.1
49.4
PMI-CAP
54.6
95.1
ORG-TRL
43.6
50.9
Method
C
ORG-TRL
54.3
95.2
OpenBook
33.9
52.9
X-L.+T. 81.4
SwinBERT
58.2
120.6
SwinBERT
41.9
53.8
Flamingo
84.2
MV-GPT
48.9
60
GIT
79.5
180.2
C.4Cap. ET
86.5
(a) MSVD
GIT
53.8
73.9
GIT
93.8
(b) MSRVTT
VATEX private test
Table 6: Results on video question answering. All are open-ended question answering tasks. All-in-one: Wang et al
(202aa
ClipBERT: Lei et al. 22021), CoMVT: Seo et al. (2021), Flamingo: Alayrac et al. (2022), JustAsk: Yang
el al.
2021a
MERLOT: Zellers et al. (2021). MV-GPT: Seo et al. 2022). QueST:
Jiang et al. 2020
HCRN: Le
et al.
2021
VIOLET:
Fu al. [2021).
Method
Accuracy
Method
Accuracy
Method
Accuracy
QueST
34.6
JustAsk
HCRN
55.9
HCRN
36.1
MV-GPT
41.7
QueST
59.7
CoMVT
42.6
MERLOT
ClipBERT
60.3
JustAsk
46.3
VIOLET
13.9
All-in-one
66.3
VIOLET
17.9
All-in-one
VIOLET
68.9
All-in-one
48.3
Flamingo
47.4
MERLOT
69.5
GIT
56.8
GIT
43.2
GIT
72.8
(a) MSVD-QA
(b) MSRVTT-QA
(c) TGIF-Frame
directly predict the category name. By fine-tuning the model with only 1 shot or shots per category, we observe that the accuracy significantly improved. This demonstrates model can be easily adapted to downstream tasks even with few training samples. With the shot increased from to 5, the gap between voc-prior and the other two columns (equal and in) becomes smaller. This is expected as more shots can be better to guide the network to predict in-vocabulary output.
Compared with Flamingo, our GIT achieves higher accuracy. Flamingo conducts the few-shot learning without parameter update, but each test image combined with the support training examples as extra network inputs. Meanwhile, different test image requires different support shots based on Yang et al. (2022). These may increase the inference cost. In contrast, model updates the parameters by lightweight, fine-tuning once, and then all these training shots are not required during inference.
4.5 Results on Scene Text Recognition
The task (Graves et al. 2006 aims to read scene text directly from the image. We evaluate our model in two settings. One the GIT fine-tuned on TextCaps. The prediction considered correct if the caption

---

Published in Transactions on Machine Learning Research (11/2022)
Table 7: Results ImageNet-1k classification task. Our approach takes the class name as the caption and predict the label in auto-regressive way without pre-defining the vocabulary.
Table 8: Results scene text recognition. MJ and ST indicate the MJSyuth (MJ) (Jaderberg et al. 2014 2016) and SynthText (ST) (Gupta et al 2016)
datasets used for training scene text recognition mod- els.
VocabularyMethood
Top-1
Method
FT data
Average
ALIGN (Jia et al. 2021
88.64
Closed
Florence Yuan et al.
2021
90.05
SAM
Liao et al.
2019
MJ+ST
87.8
CoCa Yu et al. 2022
91.0
Ro.Scanner Yuue et al.
2020
MJ+ST
87.5
SRN Yuu et al. 2020
MJ+ST
89.6
Open
GIT
88.79
ABINet (Fang et al.
2021a
MJ+ST
91.9
S-GTR (He al. 2022
MJ+ST
91.9
MaskOCR (Lyu et al.
2022
MJ+ST 93.8
TextCaps
89.9
GIT
MJ+ST 92.9
Table 9: Zero/Fewt Pero/Few-shot evaluation on ImageNet with 3 metrics. equal: the unrestricted prediction should exactly matched to the ground-truth. in: the unrestricted prediction should contain the ground-truth label name. voc-prior: the vocabulary is pre-defined as a prior. For our GIT, a trie structure is constructed motivated from |aang et al. (2022b) to limit the candidate tokens during each token prediction, such that the predicted result is guaranteed to be within the vocabulary.
Zero-shot
1-shot per class
5-shot per class
Accuracy type equal
in
voc-prior
equal
in
voc-prior equal
in
voc-prior
Flamingo
71.7
77.3
GIT
1.93
40.88
33.48
64.54
66.76
72.45
79.79
80.15
80.95
contains the ground-truth scene text. word. The other is to fine-tune the model on two large scene text datasets: MJSynth (MJ) (Jaderberg et al. 2014 2016) and SynthText (ST) (Gupta et al. 2016), where the
ground-truth scene text is used as the coption. The prediction is correct if the output is the exact match to
the ground-truth. Following the established setup, we evaluate on six standard benchmarks, including ICDAR
2013 (IC13) (Karatzas et al. 2013), ICDAR 2015 (IC15) Karatzas et al., 2015), IIIT 5K-Words (IIIT) (Mishra
lt al. 2012 Street View Text (SVT) (Wang et al. 2011 Street View Text-Perspective (SVTP) (Phan el al. 2013), and CUTESO (CUTE) (Rismumawan et al. 2014p. The average accuracy is reported in Table g
The accuracy on individual test sets in supplementary materials. Our TextCaps-fine fine-ttnnetedddd captioning model achieves 89.9 accuracy, which demonstrates the strong scene text comprehension capability of our captioning model. After fine-tuning the model on the standard MJ+ST datasets, GIT achieves 92.9 that
surpasses the prior arts (Fang et al. 2021a He et al. 2022 of 91.9.
4.6 Analysis
Model and data scaling. To study the trending with data scales, we construct two smaller pre-training datasets: one the combination of COCO, SBU, CC3M and VG, leading to 4M images or 10M image-text pairs; the other to further combine CC12M, leading to about 14M images or 20M image-text pairs. When pre-training on small-scale datasets, we use 30 epochs rather than 2 epochs as on the 0.8B data. For the network structure,
name our model as Huge and replace the image encoder with ViT-B/16 and ViT-L/14 14 from CLIP Radford et al. (2021) Base and Larye, respectively. Fig, 4 shows the results COCO, Tept and VizW Wi..Q.Q.A. On COCO, the base model benefits from 4M to 14M, but the performance drops with 0.8B data. The 14M data are more similar to COCO than the majority of the noisy 0.8B data. Meanwhile, the Base model with limited capacity may not be able to benefit effectively from large-scale data. Similar observations are also reported in Kolesnikov al. (2020) for ImageNet-Ik. 1k classification. On TextCaps and VizWiz-QA, all model variants benefit significantly from more pre-training data. Also, larger backbone improves more especially with 0.8B data.
10

---

Published in Transactions on Machine Learning Research (11/2022)
145
Bode
Large
Hage
140
120
56
CD0
CDDR
Nouuaiy
135
130
Bast
80
Bess Lange Hug:
125
Huse
10'
1d
10
10
10"
10
million per traiing
miliion moining
(a) COCO
(b) Caps
(c) ViaWiz-QA
Figure 4: Performance with different pre-training data scales and different model sizes.
Table 10: Ablation study of larger text decoders. The models pre-trained a subset of 0.4B image-text pairs. No. beam search and no SCST are performed.
Layers
COCO
nocaps
B@4
M
C
S
C
S
6
38.9
30.7
136.4
24.6
119.3
15.9
12
38.9
30.6
136.0
24.2
118.1
15.5
24
39.1
30.2
134.6
23.8
115.4
15.1
Here, we scale the image encoder. Empirically, we find it difficult to effectively scale up the text decoder. Preliminary results are shown in Table [O] which shows a larger decoder shows no improvement. The reason might be that it is difficult to effectively train with limited amount of text by LM. Another plausible reason is that the image encoder is responsible for object recognition, and the decoder is responsible for organizing the object terms in natural language way. The latter task might be casy since most of the descriptions follow similar patterns, e.g. object + verb + subject, and thus a small decoder is enough during end-to-end training. Larger decoders increase the learning difficulty, which might degrade the performance.
Flamingo (Alayrac et al., 2022) shows a larger docoder improves the performance. However, their decoder is pre-trained and frozen during the VL. pre-training, which avoids the problem of how to effectively train the decoder. In LEMON (Hu et al. 2021a), the transformer can be scaled up to 32 layers. The reason could be that LEMON uses MLM, instead of LM, which might be more difficult to train.
Scene text in pre-training data. To understand the capability of scene text comprehension, we examine the pre-training dataset and study how many image-text pairs contain the seene text. We first run the Microsoft Azure OCR AP against images in CC12M and 500K images in the web crawled images. The OCR result is compared with the associated text. It is considered matched only if the text contains an OCR result that is longer than characters. It is estimated that 15% of CC12M and 31% of the downloaded images contain scene text descriptions. As the training task is to predict the texts, the network gradually learns to read the scene text.
5 Conclusion
In the paper, we design and train a simple generative model, named GIT, to map the input image to the associated text description on large-scale image-text pairs. On image/video captioning and question answering lasks, our model achieves new stale-of-the-arl performance across numerous benchmarks and surpasses the human performance on TextCaps for the first time. For the image classification, we apply the generation task to predict the label name directly. The strategy is different from the existing work with a pre-defined and fixed vocabulary, and is beneficial especially when new category data are added.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Limitations. We focus on the wawraaiiing-addfinnttiiii strategy to improve the absolute performance. Empirically, we find it is unclear on how to control the generated caption and how to perform in-context. learning without parameter update, which we leave as future work.
Societal impact. Compared with the existing work, our model clearly improves the performance and be more appropriate to help visually-impaired people. The model is pre-trained on large-scale data, and the data are not guaranteed contain no toxic language, which may poison the output. Although we observe few such instances qualitatively, special care should be taken to deploy the model in practice and more rescarch exploration is required to control the output.
References
Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In CVPR, 2019.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019.
BaptiBe Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Xiv preprint arXiv:2204.1 14 198, 2022.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019.
Ali Furkan Bilen, Ron Lilman, Yusheng Xie, Srikar Appalaraju, and R Manmatha. Latr: Layout-aware transformer for scene-text vqa. In CVPR, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Noss,, Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. Xr preprint Xrv::20005.1 1 14165, 2020.
Voss, Gretchen
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pro-training to recognize long-tail visual concepts. In CVPR, 2021.
David Chen and William Dolan Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.
Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning. In AAAI, 2019.
Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. Learning modality interaction for temporal sentence localization and event captioning in videos. In ECCV, 2020a.
Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. Less is more: Picking informative frames for video captioning. In ECCV, 2018
Ven-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV, 2020b.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and- vision-and-languagg tasks via text generation. In ICML, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Xii preprint arXi::2204.0255 2022.
12

---

Published in Transactions on Machine Learning Research (11/2022)
Marcella Cornia, Lorenzo Baraldi, Giuseppe Fiameni, and Rita Cucchiara. Universal captioner: Long-tail vision-and-languuage model training through content-style separation. arXiv preprint arXiv:2111.12771 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Golly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognilion at scale. In ICLR, 2021.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianleng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, La Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-end vision-and-languaee transformers. arXiv preprint arxiv: 2111. 10317, 2021.
Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR, 2021a.
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Injecting semantic concepts into end-to-end image captioning. arXiv preprial arxiv:21 12. 05230, 2021b.
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Compressing linual-lig model via knowledge distillation. In ICCV, 2021c.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, Willian Yang Wang, Lijuan Wang, and Zicheng Liu. VIOLET End- to-end video-language transformers with masked visual-token modeling, arXiv preprint arXiv:2111.12681, 2021.
Zhe Gan, Yen-Chur Chen, Linjic Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Largo-scale adversarial training for vision-anddlanguagee representation learning. In NewrIPS, 2020.
Chenyu Gao, Qi Zhu, Peng Wang, Hui Li, Yuliang Liu, Anton van den Hengel, and Qi Wu. Structured multimodal attentions for textvqa. arxiv preprint Yiw:2006:00753, 2020.
Xuchao Gong, Hongji Zhu, Yongliang Wang, Biaolong Chen, Aixi Zhang, Fangxun Shu, and Si Liu. Multiple transformer mining for vizwiz image caption. In 2021 Viz Wiz Grand Challenge Workshop, 2021.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In JCML, 2006.
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In CVPR, pp. 2315-2324, 2016.
Danna Gurari, Qing Li, Abigalc J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018
Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. Captioning images taken by people who are blind. arXiv preprint ar Xim:2000 2,5666,,,, 2020.
Wei Han, Hantao Huang, and Tao Han. Finding the evidence: Localization-aware answer prediction for text. visual question answering. In COLING, 2020.
Yue He, Chen Chen, Jing Zhang, Juhua Liu, Fengxiang He, Chaoyue Wang, and Bo Du. Visual semantics allow for textual rcasoning better scene text recognition. In AAAI, 2022.
Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jicbo Luo, and Yunde Jia. Joint syntax representation learning and visual cue translation for video captioning. In ICCV, 2019.
Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In CVPR, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. or preprint ar Xrxiv:21 11. 1111223,, 2021a.
Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianleng Gao, and Zicheng Liu. VTVO: surpassing human performance in novel object captioning with visual vocabulary pre-training. In AAAI, 2021b.
Lun Huang, Wenmin Wang, Jie Chen, and Xiao Yong Wei. Attention on attention for image captioning. In ICCV, 2019.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint or N.N0:20044.088/.4 2020.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint 406. 2014.
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. IJCV, 2016.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. TGIF-QA: toward spatio-temporal reasoning in visual question answering. In CVPR, 2017.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. Divide and conquer: Question- guided spatio-temporal contextual attention for video question answering. In AAAT, 2020.
Yash Kant, Dhruv Batra, Peter Anderson, Alexander G. Schwing, Devi Parikh, Jiasen Lu, and Harsh Agrawal. Spatially aware multimodal transformers for textvqa. In ECCV, 2020.
Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lhuis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In ICDAR, 2013.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Twamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu,, et al. Todar 2015 competition robust reading, In ICDAR, 2015.
Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languagg transformer without convolution region supervision. In ICML, 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXio preprint arXiw:1602.07893 2016.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hicrarchical conditional relation networks for multimodal video question answering. IJCV, 2021.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal, Tvr: A largo-scale dataset for vidco-subtitle moment retrieval. In ECCV, 2020.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, 2021.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint ar Xix: 2022a.
Jumnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NewrIPS, 2021a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven H. Hoi. BLIP: bootstrapping language-image pre- training for unified lanoun-a understanding and generation. arXiv preprint arXiv:2201.12086, 2022b.
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang. William Yang Wang, et al. Value: multi-task benchmark for aidoo-and-language understanding evaluation. In NewrIPS, 2021b.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachon Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. In ACI, 2021c.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for languanlanguage tasks. In ECCV, 2020.
Minghui Liao, Pengyuan Lyu, Minghang He, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. PAMI, 2019.
Kevin Lin, Linjie Li, Chung- Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning, ar preprint arXiv:2111.191 2021.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. arXiv preprint arxiv:1405, 0312, 2014.
Fen Liu, Guanghui Xu, Qi Wu, Qing Du, Wei Jia, and Mingkui Tan. Cascade reasoning network for text-based visual question answering. In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (eds.), ACM MM, 2020a.
Sheng Liu, Zhou Ren, and Junsong Yuan. Sibnet: Sibling convolutional encoder for video captioning. IEEE TPAMI, 2020b.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xr:i 11602, 2019.
Yu Liu, Lianghua Huang, Liuyihang Song, Bin Wang, Yingya Zhang, and Pan Pan. Enhancing textual cues in multi-modal transformers for vqa. In 2021 VizWiz Grand Challenge Workshop, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NewrIPS, 2019.
Pengyuan Lyu, Chengquan Zhang, Shanshan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Maskocr: Text recognition with masked encoder-decoder pretraining. arXiv preprinl ar arXic:2206.00:: 2022.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Anand Mishra, Karteck Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Auirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using million captioned photographs. In NewrIPS, 2011.
Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In CVPR, 2020.
Mandola Patrick, Yao Huang, Yuki Asano, Florian Motze, Alexander Hauptmann, Joao Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2021.
Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion in natural scenes. In ICCV, 2013.
Yixuan Qiao, Hao Chen, Jun Wang, Yihao Chen, Xianbin Ye, Ziliang Li, Xianbiao Qi. Peng Gao, and Guotong Xie. Winner team mia at textvqa challenge 2021: Vision-and-language representation learning with pre-trained sequence-to-seqqeeee model. Xiv preprint arXic:2106. 15332, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020.
Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text delection system for natural scene images. Espert Systems with Applications, 2014.
Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. Look before you speak: Visually contextualized utterances. In CVPR, 2021.
Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end generative pretraining for multimodal video captioning. arXie preprint arXiw:2201.082 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Auna Rohrbach, Kai- Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit vision-and- langguugee tasks? arXio preprint orXi::2107.0620 063888 2021.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Hao Tan and Mohit Bansal. LXMERT: learning cross modalit encoder representations from transformers. In EMNLP, 2019.
Mingkang Tang, Zhanyu Wang, Zhaoyang Zeng, Fengyun Rao, and Dian Li. Clip4caption ++: Multi-clip for video caption. arXie preprint arXiv:2110.00 10. 05.00004 2021.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basse image description evaluation. In CVPR, 2015.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Subhashini Vemugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, and Kate Saenko. Translating videos to natural language using deep recurrent neural networks arXiv preprinl arXiv:1412.4722, 2014.
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv preprint arXte:2203.072 07303, 2022a.
Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, and Wei Liu. Controllable video captioning with pos sequence guidance based gated fusion network. In ICCV, 2019a.
Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. Minivlm: A smaller and faster vision-language model. arXiv preprinl Xix::200 269446, 2020.
Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: A unified transformer for linion-language representation learning. arXiv preprint arXiv:2111.1002;
2021a.
Kai Wang, Boris Babenko, and Serge Belongie. End-Lo-end scene text recognition. In ICCV, 2011.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to-ssqqueec learning framework. arXiv preprint ar Xiv:2202.03055 2022b.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-andllanguagge research. ICCV, 2019b.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. ar preprint orXiv:208.. 10904, 2021b.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 1rXiv:1609.088 0444 2016.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, and Qi Wu. Towards accurate text-based image captioning with content diversity exploration. In CVPR, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention for vision-language pro-training. In NewrIPS, 2021.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In ICCV, 2021a.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Crossing the format boundary of text and boxes: Towards unified vision-language modeling. arXio preprint xrXiv:2111.10065 2021b.
Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei A. F. Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. TAP: text-aware pre-training for text-vqa and text-caption. In CVPR, 2021c
17

---

Published in Transactions on Machine Learning Research (11/2022)
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 3081 3089, 2022.
Peter Young, Alice Lai, Micah Hodosh, and Julia. Hockenmaier. From image descriptions to visual denotations:
New similarity metrics for semantic inference event descriptions. Transactions of the Association for Compulational Linguislics, 2014.
Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Towards accurate scene text. recognition with semantic reasoning networks. In CVPR, 2020.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint XXtx:2205.01911,, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianleng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: new foundation model for computer vision. arXiv preprint 0rXiv:2111.11492, 2021.
Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, and Wayne Zhang. Robustscanner: Dynamically enhancing positional clues for robust text recognition. In ECCV, 2020.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. MERLOT: multimodal neural script knowledge models. In NewrIPS, 2021.
Junchao Zhang and Yuxin Peng. Object-aware aggregation with bidirectional temporal graph for video captioning. CVPR, 2019.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. In CVPR, 2021a.
Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommendee learning for video captioning. In CVPR, 2020.
Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, and Weiming Hu. Open-book video captioning with serreve-copy-generaae network. In CVPR, 2021b.
Qi Zheng, Chaoyue Wang, and Dacheng Tao. Syntax-aware action targeting for video captioning, In CVPR, 2020.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In A AAAT, 2018.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and VQA. In AAAL, 2020.
18


Reviews:
[{"review_id": "UsNG3gxuXy", "reviewer": "Reviewer_a1U9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The experiments are very extensive. With the majority of the tasks focused on vision-and-language understanding, the paper evaluates on all the vision-and-language task I am aware of, and achieves impressive results there.\n+ I like the architecture simplicity, and overall direction that tries to scale the pre-training data and model size on simple models to make progress\n\n- Against the simplicity is that approach initializes the visual encoder from image-text contrastive learning (CLIP-like). The additional stage is adding complications and is not friendly for reproducing results or better scientific understanding. \n- Overall the paper lacks good ablations. This may due to the fact that a lot of the space is devoted to state-of-the-art, system-level comparisons. However, I believe analytical experiments are equally important and should be taking up more space.\n- Relative to the above point is the \"novelty\". I understand novelty is subjective and for simple approaches it is a bit demanding. However, since there are already so many works showing vision-and-language works existing and as a reader I am not so surprised to see size matters and yet another better number is achieved with the next-scale model. Therefore, I believe insight or rigorous analysis is the key here to at least remedy the concern to an extent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to use a simple encoder-decoder design for pre-training a vision-and-language Transformer. The model is then fine-tuned to a number of tasks, mainly for vision-and-language understanding but also for vision-only tasks, and shows good results. It shows that the simple encoder-decoder design is powerful enough to achieve state-of-the-art.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* In light of the first weakness, I am curious about the role of the pre-trained visual encoder and think more experiments can be devoted there for ablations. E.g. what if randomly initialize the visual encoder; what if initialized from supervised/self-supervised learning purely on images.\n", "broader_impact_concerns": "Not aware of.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dd9LM8B7TX", "reviewer": "Reviewer_q1NN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The proposed framework is simple yet effective.\n\n2. Very strong empirical results.\n\n3. The paper is presented overall clearly.\n\nCons:\n\n1. It is hard to understand the true source of the improvement in comparisons with existing methods. Due to the difference in pretraining data & its size, model size, number of pretrained encoders/decoders used and fine-tuning data & its size, from the main SOTA comparison Tables, it is impossible to understand the possible reasons leading to the improvements. Therefore, it is important to at least add details from these aspects for the most competitive compared methods to clarify the main source of improvements. \n\n2. Lack further insightful analysis of the behavior of the model. \n\n- a. The text decoder is randomly initialized but there is no analysis on the behavior of the model when the text decoder is initialized from some pretrained LMs. This is actually counter-intuitive and very different from existing work where leveraging pretrained text decoder can even avoid the vision-text pretraining process [r1,r2,r3,r4]. The authors should discuss this different behavior of the proposed model and show more insights on the possible reasons.\n\n- b. The different trend in terms of the size between image encoder and text decoder. In Flamingo, text decoder is much larger than the image encoder but in this work the image encoder is usually larger than the text decoder. Is this correlated to the fact that currently the text decoder is randomly initialized? What is the principle on the deciding the size of each component? What is the performance when using a relatively much larger text decoder?\n\n- c. The reviewer understands that the authors have genuinely listed the zero/few-shot ability as a limitation. But it is still very interesting to see a comprehensive performance comparison to understand the gain of fine-tuning. This could be also helpful to understand the benefit from different fine-tuning datasets.\n\n3. Minor: the details of how videos are sampled/preprocessed is missing.\n\n[r1] VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs, CVPR 2021\n\n[r2] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language\n\n[r3] Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning\n\n[r4] Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors proposed a simple generative image-to-text framework with a unified encoder-decoder design for various downstream tasks. New state-of-the-art is achieved on various downstream tasks. This framework is also shown to be effective on directly generating image class names for image classification.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please check the cons and broader impact concerns for details of required new items.\n\nPost-discussion:\nThanks for the new results and discussions! But the reviewer noticed that many of the discussions are not added to either the main paper or the supplementary in the revision.\n\nThe reviewer tends to recommend acceptance if all the discussions are added. If the reviewer missed anything, it would be very helpful if the authors could provide a change-list for the revisions.", "broader_impact_concerns": "The open-vocabulary generation property makes this proposed foundation model potentially easier to have negative impact. Therefore, it will be interesting and somewhat necessary to further evaluate the bias learned in this model. For example, a further test could be evaluating the model for image captioning and see if it has severe issue of racial bias [r5].\n\n[r5] Understanding and Evaluating Racial Biases in Image Captioning, ICCV 2021", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VvGZ8eFTdA", "reviewer": "Reviewer_hDXf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors propose an end to end model capable of performing well across a variety of image and captioning tasks.\n\n2. The architecture proposed is simple and the model is trained using a single pre-training task instead of relying on more supervision and multiple pre-training tasks. \n\n3. The experimental section of the paper is clear and the authors have specified all the details regarding the various experiment and analysis conducted. \n\n4. In general, the paper is well written and is easy to follow.\n\n**Weaknesses**\n\n1. Although the results in section 4.5 showcase that the model is able to achieve good performance on standard OCR datasets, it is still not clear how the model learns to do OCR well. Is the model able to do OCR well by just adding more images which contains scene text in it, or is it the architecture of the model that contributes to good OCR performance? Also, the paper does not contain experiment on a recent TextOCR dataset. \n\n3. To compare GIT to recent large scale vision and language models, instead of just comparing performance the authors should also compare the amount of training data (pre-training and fine-tuning) and the number of parameters used by the baselines and GIT. Including this analysis might strengthen the paper further.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Contributions**\n\nThe paper proposes a unified architecture GIT for vision and language tasks. The GIT model is able to perform well across a variety of vision and language tasks like video/image captioning, video/image question answering and scene text recognition without relying on external feature extractors like object detectors and OCR. The model performs significantly well on tasks which involve reasoning over scene text in images. The approach used is simple and scales well with more pre-training data and the model size.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Highlighted in the weaknesses section above.\n\n**Decision**\n\nI am recommending accept as the authors have provided requested changes in feedback.", "broader_impact_concerns": " The model is pre-trained on large-scale data, and the data used for training might contain toxic content and biases therefore the model might generate predictions which contain biases or toxic language.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E98lLa0nCd", "reviewer": "Reviewer_6mzR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \n- Large-scale pretrained vision-and-language mode have been very successful and has seen a lot of innovation in the last couple of years. But existing methods require careful pre-training, multiple training objectives and discriminative tasks (VQA, etc) and image-captioning tasks are handled differently. This paper presents a minimalist approach – treat all tasks as generative language modeling tasks and show impressive performance on a range of downstream tasks.\n- Overall results are great, especially the results on nocaps and TextCaps! I enjoyed seeing the qualitative results present in the supplementary.\n- I was pleasantly surprised by how well the model works on the TextCaps dataset both qualitatively and quantitatively. Without using the output of a OCR module as input, the model was able to produce captions that correctly recognized text present in the image, and use it in the output caption.\n\n## Weaknesses: \n1. As mentioned in the manuscript, one of the limitations of the approach is the requirement of fine-tuning on a range of tasks. The paper doesn't show few-shot results on any of the tasks. \n  1.1 Specifically, on tasks like COCO, nocaps, being trained on 0.8 billion image-text pairs, it would have been nice to see zero-shot/few-shot performance at least on the captioning tasks. \n  1.2 Similarly, tasks like image classification can also be evaluated in a zero-shot/few-shot manner, right? It would have been great to see these results in the paper \n  1.3 Flamingo (Table 6) (https://arxiv.org/pdf/2204.14198.pdf) show few-shot results on a range of benchmarks, and it would be good to discuss the model's ability for few-shot learning. \n\n2. Discussion of failure modes: While the authors discuss reasons for why the performance is a few points lower than other methods (e.g., VQA performance is lower because of posing the task as a generative task vs discriminative), I would have liked to see some discussion of what authors think are fundamental limitations of the paper. For instance, if this model is not capable of few-shot learning, why is that?\n\n3. I'd also like to point out that the author's evaluation on nocaps is not consistent with the guidelines in the benchmark. According to the guidelines in the original paper, large-scale training on additional image-caption pairs that don't come from COCO should not be used. The authors of the nocaps benchmark, specifically created a server for this usecase (nocaps-XD) which should be used when using extra image-caption paired data.\n\n3. A very minor comment on the supplementary – the text, tables and figures are put together in a way that is hard to read. For instance, 2 lines of the previous section show up after the big table 7 on page 19. I recommend reformatting it such that text for each section appear in one place. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a new architecture GIT which pose a diverse range of vision-and-language tasks as a generative language modeling task. The authors show posing all tasks as a single language modeling task has many desirable properties – simplification of training paradigm (pre-training and fine-tuning on downstream task is consistent), data can be scaled for pre-training and there is no need for expert modules like OCR, object detection. Using GIT, authors show state-of-the-art performance on many vision and vision-and-language tasks including tasks that require reading text in image. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I am tending to recommend the paper for acceptance, but it'd be nice to include these results\n- Few-shot results similar to Flamingo, CoCa on the various benchmarks. While these won't come very close to the sota results, it'd still be good to see what few-shot learning performance looks like. \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper overall successfully supported the claims experimentally with throughout strong and extensive experimental evaluation.
Overall, all reviewer recommend accept, but with some hesitation.

Specifically, the reviewers and action editor, like to ask the author to improve the following points in a minor revision
* integrate missing discussion of the author/reviewer discussions in paper.
* add any additional insights the authors might have to provide more understanding the authors have gained in the meantime to provide more details for how the model is working and how the improvements are achieved
* thorough proof read of the paper
Decision: Accept (poster)