Tag2Text: Guiding Vision-Language Model via Image Tagging
Authors: Xinyu Huang
Youcai Zhang
Jinyu Ma
Weiwei Tian
Rui Feng
Yuejie Zhang
Yaqian Li
Yandong Guo
Lei Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Image Recognition
Vision-Language Pretraining
Image Tagging
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This paper presents Tag2Text, a strong image recognition model which achieves a superior tagging ability and effectively enhances vision-language tasks.

Abstract: This paper presents Tag2Text, a vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with a limited detector, our approach utilizes tags parsed from its paired text to learn an image tagger and meanwhile provides guidance to vision-language models. Given that, Tag2Text can utilize large-scale annotation-free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. Strikingly, Tag2Text showcases the ability of a foundational image tagging model, with superior zero-shot performance even comparable to full supervision manner. Moreover, by leveraging tagging guidance, Tag2Text effectively enhances the performance of vision-language models on both generation-based and alignment-based tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the proposed tagging guidance.

Published as a conference paper at ICLR 2024
TAG2TEXT: GUIDING VISION-LANGUAGE MODEL VIA IMAGE TAGGING
Xinyu Huang 1.2 Youcai Zhang²
Jinyu Ma²
Weiwei Tian
Rui Feng 1.4+
Yuejie Zhang
Yaqian Li
Yandong Guo? Lei Zhang*
"Shanghai Key Lab of Intell, Info. Processing, School of Computer Science, Fudan University
OPPO Research Institute 'International Digital Economy Academy (IDEA)
4 Academy for Engineering and Technology, Fudan University
ABSTRACT
This paper presents Tag2Text, vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learn- ing of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with an off-the-shel de- tector with limited performance, our approach explicitly learns an image tagger using tags parsed from image-paired text and thus provides a strong semantic guidance to vision-language models. In this way, Tag2Text can utilize large- scale annotation-fis free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. As a result, Tag2Text demonstrates the ability of a foundational image tagging model, with superior zero-shot per- formance even comparable to fully supervised models. Moreover, by leveraging the tagging guidance, Tag2Text effectively enhances the performance of vision- language models on both generation-based and alignment- tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the pro- posed tagging guidance. Codes, demo and pre-trained models are available attps://e //itt/u//ttu com wwt./ccc..//eeeeeeeeeee http:/iiyyl2////ecggnooo http://rrrooiiiieeatiaiooaa
INTRODUCTION
Vision language pre-training (VLP) has shown an effective approach for learning a generic multi- modal representation and improving lasion-language (VL) tasks including ceneration-based (e.g. image captioning) and alignment-based (e.g., image-tex text retrieval). As large-scale datasets of image-
text pairs (Sharma et al. 2018 Changpinyo al. [222]] Schuhmann et al., [021 Radford et al.)
2021 Jia et al. 2021) become available, recent works mainly focus on using transformer-based
models to perform contrastive (Radiord al. 202T Tia al. 202T Li al.
2021 Bao et al.
2021
Li et al. 2022; Yu et al.
2022) or generative learning (Wang et al. 2021b] Li et al. 2022
Wang et al.) 2022a] Chen et al.
2022 Yu et al., 2022; Wang et al. 2022c di Li al.] (023) from
massive image-text pairs. While great progress has been made, such studies normally rely on brute force pre-training manners that involve direct interaction with different modality features, modeling
weakly-supervised learning due to the lack of explicit alignment supervision between image and
text (Li et al. 2020b Hu et al. 2021 Zeng et al. (022))
Prior approaches (e.g., OSCAR (Li et al.) 2020b), VIVO (Hu et al. 2021). X-VLM Zeng et al.) 202T]) introduce the use of object tags as anchor points to ease the learning of semantic alignments between images and texts. However, these approaches rely on obsolete detector-based VLP frame- works, which employ off-the-shelf object detectors (e.g., Faster RCNN (Ren et al.] 2015) to extract image features (as shown in Figure I
(). The primary limitation of ditector-basee models is that the used object detectors are normally not perfect but have to be kept frozen during VLP to maintain detection ability, thereby restricting the capacity of vision-language models (Li et al. 2021] Dou et al. 2022 Huang et al.) 2022). Moreover, utilizing object detectors leads to a substantial increase model parameters and running time (Li al. 202 Kim et al. 2021). Consequently, more re-
cent works (Li et al. 2021 2022 Dou et al. 2022 et al. 2023) primarily utilize detector-free VL
Corrrsponniig author.

---

Published as a conference paper at ICLR 2024
Object Tags person, surfboore
Text
"A mon in block wetsuit riding Hove in the acean surfboord
Object Detector
1
Marhod
Object
Tagainng
No Manual
Image Encoder
Modality
Text Encoder
Annotation
Interaction
Emable
End-te-Fae Eod
Image
13:
Categories Additional
Tag
Objects
Ubjects, Scenes,
Actions
Tagging Head
mon, black, wove, ocean. surfboard
Guide
Parmaters
42M
& 5M
Detector based Model
Kunning
153ms
~~40ms
2 Detector-Free free Model
Time
Image Tagging
3 Image lagging Guidance
(a)
(b)
Figure 1: (a) Q: Prior works demonstrate the effectiveness of incorporating object tags into VL models based on an off-the-shelf detector. @:: Since the detector restricts the model's capacity and is time-consuming, recent VI. models normally avoid using detector, resulting in poor utilization of valuable tags. ①: We re-introduce tag guidance into detector-free VI. models via image tagging with a simple tagging head. The tagging head is supervised by annotation-free image tags parsed from its paired text. Our model achieves a superior tagging ability and effectively enhances vision- language tasks. (b) Comparison of object detector and image tagging used in VL models.
models to address these limitations, resulting in the discarding of valuable tags (as shown in Figure )).
In this work, shown in Figure 1 C re-introduce tag guidance into detector-free VL models via the novel approach of image tagging. We demonstrate that integrating image tagging with other natural and effective approach from two crucial perspectives. 1) Data: The pre-training image tags are obtained through automatically text semantic parsing, en- abling large-scale annotation-free image tags accordance with image-text pairs can be utilized, without the requirement of expensive grounding annotations which are necessary for object detec- tors. Image tagging also provides better bridge between image and text, given that the parsed tag categories are more diverse and beyond objects, such as scenes, attributes, actions, etc. 2) Architec- ture: Image tagging merely necessitates adding a recognition head followed by the original image encoder, ensuring efficient end-to-end pre-training and resulting in fewer parameters and improved efficiency. Figure [[b) provides a comprehensive comparison between image tagging and object detection.
VLP tasks in multi-task manner
Concretely, we present Tag2Text, a VLP framework which introduces image tagging into vision- language models to guide the learning of visual-linguistic features. For image tagging. previous approaches primarily rely on limited manually annotated datasets (Lin et al., 2014; Everingham et al. 2015), resulting in poor generalization capability. In contrast, Tag2 Text utilizes large- scale image-text pairs, achieving an exceptional tag recognition capability of 3,429 commonly human-used categories. Remarkably, Tag2Text demonstrates foundational image tagging capa- bility with superior zero-shot performance, which significantly outperforms other state-of-the-an (SOTA) dision-languee models such as CLIP (Radford et al.. (2021). BLIP (Li et al. 20022), and BLIP-2 (ii et al. 2023) and is even comparable to fully supervised models (Ridnik et al.) 2023).
Moreover, Tag2Text effectively leverages tagging guidance to enhance the performance of vision- language models. For generation-based tasks, we design the training task as image-tag-text gen- eration which empowers the model to produce text descriptions based on the image features in accordance with assigned tags. As depicted in Figure 2 Tag2Text generates more comprehensive text descriptions with the guidance of comprehensively recognized tags. Additionally, Tag2Text per- mits users to input desired tags, providing the flexibility in composing corresponding texts (Zheng ct al. 2219). For alignment-baas based tasks, while previous models rely on the alignment of multi- modal features which considered as black-box approaches, Tag2Text augments these methods by incorporating tags as visible alignment indicators.
Our key contributions be summarized follows:
For the first time, Tag2Text demonstrates the potential of foundational image tagging model by utilizing large-scale annotation-free image tags parsed from image-paine paireed text, exhibiting zero-shot capabilities rivalling full supervision manners.
2

---

Published as a conference paper at ICLR 2024
BUP::
A man riding a bike next to frain
painting of turtic and jellyfish.
Tag2Taat::
Tag2Test:
A mon riding bioycle next to cad passenger frain on the tracks.
A zointing of a gea turtle sximming the ocean with fish and jellyfish.
Tag2Tiest (User Speciffed):
Tag2Test (User Specified::
commuter troin possing under
brown turtls and yellow fish under blue watorcolon,
electric cables.
BLI:
BLP:
A group of people stonding on top of a lush green field.
A dog standing next to cot on couch.
Tag2Text:
Tag2Tiest:
weman sitting on couch with brown deg stording next to her and granas laying the
group yours peosle plaving game
frisbee on gressy field.
Tag2Tisst Specified): yallow looking of frisbee player.
Tag2Tast (User Specified):
A dos that is looking af the comern.
Figure 2: Comparison of image captioning results between Tag2Text (pre-training on 14M images) and BLIP (Li et al.] 2022) (pre-training on 129M images). Tag2Text integrates recognized image tags as guiding elements into text generation, resulting in the generation with more comprehensive text descriptions (see Table 2 for quantitative results). Tag2TTex also allows users to input specified tags to generate corresponding captions, offers a way of controlling caption generation through the use of input tags.
Tag2Texx re-introduces tag guidance into detector-fre vision- langual models by seam- lessly integrating image tagging, effectively enhancing the performance of both generation- based tasks and alignment-based tasks.
A wide range of downstream benchmarks, along with series of qualitative results, demon- strate the superior tagging ability of Tag2Text and the efficacy of incorporating tagging guidance information into vision-language models.
2
RELATED WORK
Vision-Language Models consist of generation-base and dlignment-basedd models. Generation- based models involve generating text related to an input image. The initial approach of generation- based models relies on two-stage process of recognizing tags from an image and then using them to compose a caption (Fang et al., 2015). Notably, image features do not participate in the text generation stage. With remarkable progress of language models (Devlin al., 2018, Brown al. 2020 Ouyang et al. 2002)), language modeling gradually becomes dominant pre- pre-training objective vision- languag generation based models (Wang et al., 202Tb Li et al. 2022 Wang et al. 2022c Chen et al. 2022; Wang et al. [2022a d). Such approach endows vision language models with the capability generate expressive captions conditioned on visual information. Distinguished from existing works, our proposed approach is novel scheme of image-tag-tex generation, enabling our model to effectively regulate the content and quality of the generated text based on assigned tags.
Alignment-based models involve determining whether an image and text are matched. Previous models perform either image-text contrastive learning (Radford et al.] 2021 Jia et al.) 2021] Li
ct al. 2021 Bao et al. 2021 Li al. 2022, HHang et al. 2022) with the dual-encoder architecture
or image-text matching (Li et al. 2020b; 2021 Bao et al. 2021 Dou et al. 2022 Li et al., 2022)
with the fusion-encoder architecture. IDEA (Huang al. 2022) introduces identified tags addi-
tional text supervision only enhancing image classífication accuracy. These models predominantly rely on the alignment of multi-modal features which considered as black-box approaches for trieval task. Tag2Text augments these methods by incorporating tags as visible alignment indicators, leading to further performance improvements.
Image Tagging, also known as lalti-label image recognition, is a fundamental computer vision task that involves identifying multiple tags for a given image. Traditional approaches rely on fully connected classifier and Binary Cross-Entropy loss (BCE) for optimization. Recent studies propose transformer- features, as well as robust loss functions (Ridnik et al. 2021 Zhang et al. 2021b) to address the Desttive-eggatie samples. Most existing multi-label
classifiers (Liu et al. 202 la Ridnik et al. (003)) to better leverage visual
issues of missing samples and unbalanced datasets (Lin et al., 2014] Everingham et al. (013)) rely on manual annotations, which are labor- intensive and difficult scale up. Our study employs text semantic parsing to efficiently obtain
3

---

Published as a conference paper at ICLR 2024
Image
Generation
image-Tag Interaction Encoder
Image-Tag-Ted Tage-TTa--extt Generation Decoder
Image Feature
Recognice
Tagging
Text
cat, lay, suitcose, pillow
leying next To pillow.
Image Feature
Interact Encoder
Gemeratee Decoder
[Ottine))
Tag
Text
Tag
[BOS]
Image Encoder
Cross
Tagging
Image-Tag
Captioning
Recognition Decoder
Image Feature
Align Encader
Toxt
Image-Text
Rocogrtic
Match
Alignment
Alignment Encoder
Tag
(d) Image-Test Retrieval
Illustration of Tag2Texxt framework. The core of Tag2Text lies in the introduction of image tagging supervised by the annotation-free image tags parsed from its paired text. Generation: Tag2Text learns to generate text related to the image by leveraging the automatically parsed tags, resulting in comprehensive and controllable texts with the guidance of recognized tags. Alignment: Tag2Text aligns the image and text, providing tags visible alignment indicators during inference.
Figure 3:
image tags and constructs large-scale image tagging dataset comprising 3,429 commonly used categories, resulting a superior tag recognition ability.
3
APPROACH
3.1 OVERVIEW FRAMEWORK
We present Tag2Text, a VLP framework that enhances the performance vision-language models by incorporating tagging guidance. Figure shows the framework of Tag2Text. With large-scale image-text pairs, the core of Tag2Text lies in the utilization of image tags from texts. Initially, the image tags extracted through text semantic parsing, providing large-scale of tags without expensive manual amnotations. Afterward, the parsed tags can serve as ground-truth labels for image tag recognition tasks. Moreover, design novel scheme of image-tag-text generation, enabling the model to effectively regulate the content and quality of the generated text with the guidance of recognized tags. Furthermore, Tag2Text encompasses image-text alignment and leverages tags as visible alignment indicators.
3.2 MINING TAGS FROM TEXTS
Text Semantic Parser is adopted to parse text into image tags, The parser (Wu 2019) first identifies entities (= head + modi and relationships from the input sentence based on the rules of the dependency tree, which grammatical structure that maps syntactic relationships within sentence. Subsequently, we obtain the tags (including objects, scenes, altributes, and actions) of the image based on contrast maps from head > object Jbjecl/scene scene, modifie ier > attribute, and relationship - action. For instance, given the sentence "A red alarm clock is wooden desk", the parser automatically parse this as: "head": f'alarm clock', 'desk") ], "modifier"; ['red', 'wooden "relation". ['on'].
Tag Category System Construction is based on the principle that tags with higher frequency considered more significant since they reflect common elements in the image descriptions. By em- ploying the semantie parser, we process 4 million open-source image-text pairs and select the 5,000 most frequently occurring tags. Further filtering by human annotation results in the selection of the most commonly human-used 3,429 categories of tags (e.g., synonyms such as "person" and "huamn" are merged). More statistics and details are presented in Appendix B
3.3 TAG2TEXT PRE-TRAINING
With triplet image-tag-text as inputs, Tag2Text employs a multi-tase pre- pre-training approach, which consists of Tagging, Generation, and Alignment. Both generation-based and alignment-hasee task
4

---

Published as a conference paper at ICLR 2024
utilize the guidance from image tagging improve their performance. Concretely, the shared visual features obtained from the image encoder are interacted with various pre-training tasks through cross-attention.
Image Tagging aims to associate image features with the corresponding tags. We apply the image- tag recognition decoder (Liu et al., 2021a) with the robust alignment loss function for optmmization. Compared to CLIP, which relies on the alignment of global image features with text via dot product interaction. Tag2Text introduces more fine tne-grained alignment of visual spatial features with tags (parsed from texts) through an efficient recognition decoder. This approach particularly effective for multi-tag recognition, since the tags often correspond to multiple image regions and reside at the token level within the texts.
Image-Tag-Tedd Generation aims to generate texts based on the image features in accordance with assigned tags. To achieve image-tag-texx generation, Tag2Text employs the transformer encoder-decoder (Vaswani et al.) (017)) architec- ture. The [BOS] token is prepended ginning of the text to indicate the start of a se- quence. To climinate positional bias, the image tags are rearranged prior to processing. Both tags and text are transformed into embeddings through tokenization and a word embedding matrix. The tag embeddings are integrated with image features in the image-tag interaction encoder and subse- quently forwarded to the image-tage tag-text generation decoder for text generation. The text embeddings are utilized as ground truths to optimize the model via Language Modeling Loss (LM).
Cart pioua
Image
Image
Text
Generate
the be-
Figure 4: Image-text generation comparison. (a) Early works (Fang et al., 2013) primarily employ multi-stage approach with separate tag recogmition and text composition. The im- age features are not utilized during the text composition stage. (b) Recent works typi- cally generate text directly from image fea- tures, which is challenging to control the gen- erated text. (c) Our approach incorporates tags bridge to guide image features for text generation, improving content and qual- ity control.
The distinction between image-tag-text generation and other generation approaches is illustrated in Figure 4 Our image-tag-text generation incorpo- rates tags bridge to guide image features for text generation in end-to-end manner. This approach enables the model to generate more comprehensive and controllable captions, provided that many accurate tags are given as the guidance signal.
Image-Texx Text Alignment aims to determine whether a given pair of image and text aligned. Fol- lowing Li et al. (2022), Tag2Text leverages additional image-text alignment encoder. The text is converted into embeddings via tokenization and word embedding. Then the text embeddings pass through the encoder and undergo coarse-grained Image- Contrastive Loss (ITC) with image features. Subsequently, the text embeddings undergo fine- fine-graines grained Image-Tex Matching Loss (ITM) with image features through cross-attention. The negative samples with higher ITC similarity will be selected for ITM with greater probability for hard mining.
3.4 TAG-GUIDED V+L TASKS
Image Tagging, also known as multi-label image recognition, demands the model recognize all relevant tags of an image. Image tagging can serve as an effective indicator of the model's recognition abilities. As illustrated in Figure 3(a), Tag2Text achieves this task by directly utilizing the image-tag recognition decoder.
Image Captioning entails the model to generate a textual description for a given image. Figure 3 shows that the same components for image-tag-ter generation pre-training are utilized during fine- tuning, Previous image-text generation models are challenging to control the content of the gen- erated description. By incorporating comprehensive tags recognized by the image-tag recognition decoder, approach effectively improves the performance the generated text. Furthermore, users can also input alternate guidance tags to generate descriptions highlighting specific aspects of the image.
5

---

Published as a conference paper at ICLR 2024
Query Text
Rank1
Rank2
Bank3
Rankd
RavbS
Eamily pasingg wearing
slopes
meanton
alaaos mouthsh deer Fall people, ceupla.
zi cips, sill El, people free, all paik. past. soay
hill man people coople anow. podz: srivey
Alales nouttein Sean pelpl caipee
shee Iill poople
halding filled with tiny deughnutt.
deall paltry. hond, CL..
deggnnit denut, hald. miphin frestin
dlggeeu denat hele, hald CONEr
Jerses, derat. Truss tdl: 28 miki coffee
pastry, teble hand, Sex. BESEL bala, plattic
Figure 5: Example results of Tag2Text on image-text retrieval. For each query text, the top 5 in- stances from the retrieval set are ranked from left to right. Tag2Text provides tags as additional visible alignment indicators.
Image-Text Text Retrieval encompasses both image-to-text and lext-to-image retrieval. Previous meth- ods match image-text pairs based solely on the features of different modalities, resulting in lack of control and interpretability. Our approach, as depicted in Figure d),,, augments these methods
by incorporating tags as visible alignment indicators (Image Recognsse Tag, Text Parge Tog). By weighing the number of matching tags with feature similarity, Tag2Text boosts the retrieval results. Besides, real-world applications frequently involve users' searching for images using several key- words rather than a sentence, highlighting the advantages of our approach. Figure presentss some examples of visible alignment indicators that enable effective alignment between image and text.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Following Li et al. 2021 2022, we pre-train our model on two widely used dataset settings, in- cluding 4 million image dataset and 14 million image dataset, respectively. The 4M image dataset setting includes two human-an aaman-annotated datasets (COCO Lin et al. (2014) and VG KKishhal al. (2017)) and two web datasets (SBU Captions Ordonez et al. (201 and CC-3M [Sharma al. (2088). ). The 14M image dataset setting builds upon the 4M setting by adding more noisy web dataset CC-12M Changpinyo al. (202I). We adopt two most widely used backbones pre-trained on ImageNet (Deng et al. 2009) as the image encoder: ViT (Dosovitskiy et al.) (02T) and Sain (Liu et al. (02 Unless illustrated with subscript, the default model vision refers to Swin as the image encoder. More implementation details are provided in Appendix A
4.2 EVALUATION ON IMAGE TAGGING
To assess the tagging capability of Tag2Texx, we conduct evaluation on two multi-label recogni- tion tasks: COCO and OpenImages (Kuznetsova al. 2020). Given the significant number of rare categories with missing labels in OpenImages, we curated subset that encompasses common cate- gories with high quality labels. We also employed internal high-quality annotated test set, known as OPPO, to provide a comprehensive evaluation of tagging performance. Our model is fine-tuned on the COCO training dataset using texts and tags parsed from the texts provided in COCO caption annotations, since the original COCO multi-label annotations encompass only 80 categories of tags. More details can be found Appendix C Addition zero-shot evaluations on NUS- BUSD (Chual
al. 2009) provided in Appendix
These tagging benchmarks serve to gauge the recognition capabilities of image recognition models for prevalent categories. The comparison of Tag2Text with other SOTA recognition models (both classification models and vision-language models) is shown in Table 2 With regard to classifica- tion models, Tag2Text demonstrates superior zero-shot recognition capabilities, even comparable to full supervision manners of ML-Decoder. With regard to languae models, for alignment vasion-language models, we calculate the similarity between image and all tag categories with

---

Published as a conference paper at ICLR 2024
thresholding obtain image tags. For captioning vision-language models, parse the caption and classify them into synonyms to obtain image tags. Remarkably, both the tagging and captioning capabilities of Tag2Text significantly exceeds other SOTA vision vision-language models (including CLIP, BLIP, BLIP-2) in common category recognition.
Pre-train #lmages
Evaluation Paradigm
Methods
OPPO
OpenImages
COCO
ML.-Decoder (iddik ct al.
2023)
9M
Tagging
82.4
85.8
72.8
MKT (He al. 2022)
400M
Tagging
78.2
77.8
62.9
Tag2Text (Ours)
4M
Tagging
83.0
82,9
78.3
Tag2Text (Ours)
14M
Tagging
85.4
83.4
78.2
Table 1: Performance comparison of image tagging with classification models in mAP. Blue refers
zero-shot performance: Green refers to fully supervised learning; Yellow denotes that the model has seen the corresponding training images, but not the annotations. Notably, Tag2Text's zero-shot generalization on Openlmages is even comparable with ML-Decoder's full supervision.
Pre-train
Evaluation
OPPO
Openimages Precision Recall
COCO
Methods
#Images
Paradigm
Fl
Precision
Recall
Fl
F1
Precision
Recull
CLIP Radford et al.
2021
400M
Alignment
63.4
76.6
54.1
63.0
77.9
52.9
48.2
64.0
38.7
DiHT
Radenivic
al
2023
438M
Alignment
66.8
75.3
60.0
66.3
77.0
65.3
48.9
51.4
46.7
BLIP (L. el al.
2022
129M
A Aligmeet
65.7
76.7
57.5
64.8
78.6
55.1
54.3
65.2
46.5
BLIP (L. et
2022
129M
Captioning
58.6
79.1
46.6
56.6
73.7
45.9
55.7
93.0
39.8
BLIP-2 eLi
2023
129M
Captioning
72.8
48.5
58.1
74.2
47.8
95.5
42.8
Tag2Text (Ours)
14M
Captioning
65.9
82.4
54.9
62.7
76.7
53.0
62.7
93.2
47.2
Tag2/eex (Ours)
4M
Tagging
75.7
76.6
74.8
71.8
79.7
65.3
72.6
80.5
66.1
Tag2'lext (Ours)
Tagging
79.4
66.6
64.5
Table 2: Performance comparison of image tagging with vision-language models. Notably, Tag2Text showcases superior zero-shot image recognition capabilities, surpassing other vision- language models with significantly larger training dataset.
4.3 EVALUATION ON IMAGE CAPTIONING
In Section we provide novel captioning evaluation paradigm based on image tagging bench- marks, which effectively gauges the caption recognition capability for prevalent categories. In this section, we evaluate Tag2Text on two established Image Captioning benchmarks: COCO Cap- tions (Karpathy& & Fei-Fei 2015) and NoCaps (Agrawal et al., 2019), with the latter focusing more on recognizing novel objects. The comparison of Tag2Text with other SOTA generation models can be found in Table 3 To ensure fairness, we compare the results with base version of all methods without utilizing the CIDEr optimization (Rennie et al. 2017).
The experimental results demonstrate Tag2Text outperforms other methods across all metrics on both benchmarks with similar model size and data scale. Furthermore, Tag2Text surpasses most met- rics of BLIP+ Boott which employs a dataset bootstrapping approach, as well as LEMON and SIMVLM, which are pre-trained 200 million and 1.8 billion images, respectively. Notably, due to the dual capability for both generation and alignment of Tag2Texx, the performance of Tag2Texx can also be further enhanced through Bootstrap, which aim to accomplish in our future work.
4.4 EVALUATION ON IMAGE-TEXT RETRIEVAL
The Image-Text Retrieval task is evaluated on two benchmarks: COCO and Flickr30K (Plummer jet al., 2015). for both image-to-text retrieval (I2T) and text-to-image retrieval (T2I). The perfor- mance comparsion with other methods are shown in Table Under equivalent equivveent pre-training data and image encoder configurations, Tag2Text demonstrates comparable or superior performance com- pared to ALBEF, VLMO, and BLIP. Tag2Text-Swin leads to further substantial improvement in performance. More importantly, the integration of tag alignment in Tag2Text makes it well-suited for practical search scenarios, where users search through the query of several keywords.

---

Published as a conference paper at ICLR 2024
COCO Caption -Finctuning
NoCaps Validation Zem-Sher out-domain
Methods
Pre-train #lmages
in-domain
neur-domain
overall
@@4
C
Pre-trained with 4M images (COCO, VG, SBU,
CC-MM)
Di [Fang et
2021
4M
35.6
120.8
UFO (Wang atal.
10
4M
36.0
122.8
94.5
13.4
82.7
12.8
64.9
11.0
80.7
12.5
OSCAR (L) et
20200
4M
36.5
123.7
80.0
12.1
80.4
12.2
75.3
10.6
79.3
11.9
VIVO (H
90.4
13.0
12.5
83.0
10.7
BLIP [ el 2022
4M
37.0
123.6
91.3
13.5
86.4
13.0
84.2
11.8
86.6
12.8
ViTCap Fang 0l
4M
36.3
125.2
98.7
13.3
92.3
13.3
95.4
12.7
93.8
13.0
Thg2 Vil (Ours)
4M
37.3
124.6
95.1
13.5
89.0
13.0
86.9
12.1
89.5
12.9
Tag2Text-Swin (Ours)
4M
38.4
128.7
101.0
13,9
96.0
13.6
96.7
12,9
96.9
13.5
Pre-trained with more imarges:
Enc- Dec (Changpinyo et al 202
15M
110.9
92.6
12.5
88.3
12.1
94.5
11.9
90.2
12.1
MiniVLM (Wang c
35.6
119.8
ViaVL Zhang et u. 2021a
6M
38.2
129.3
103.7
13.7
95.6
13.4
83.8
11.9
94.3
13.1
LEMON et
104.5
100.7
96.7
12.4
BLIP (Li at RUZ44
14M
38.0
127.8
102.2
13.9
1ag2 Text- Vil (Ours)
14M
38.4
128.9
104.8
14.6
100.4
13.9
102.3
13.3
101.3
14.0
Tag2 Swxi-Swiin (Ours)
39.1
131.8
106.7
14.5
105.7
14.4
110.7
14.3
106.9
14.4
Addition Data Augmentetion or Higher-Scale Dutasets:
BLIP Brotatenn Li al. 2022
14M
38.6
129.7
111.3
15.1
14.4
102.4
13.7
105.1
14.4
SIMVLM (Wang et a 2021D
1.8B
39.0
134.8
94.8
13.1
LEMON
200M
107.7
14.7
106.2
14.3
107.9
106.8
14.1
Table 3: Performance comparison of image captioning on the COCO and NoCaps Caption bench-
marks. BLIP* refers to the result of our reproduction. Bootstrap indicates the two stage dataset
bootsrapping approach using generation and alignment tasks.
COCOLSK
Flickr8OK(TK
Almages
R@1
R@5
RB 10
R@]
RB5
R@10
R@1
RBS
R@ 10
R@1
R@9
R@10
Pre-trinned with 4M imoger (COCO, VG, SBU, CC.MM
UNITER Chen al. 2020
4M
65.7
88.6
93.8
52.9
79.9
88.0
87.3
98.0
99.2
75.6
94,1
96.8
VILLA (Gint
95.5
METER-SWii or 212
4M
73.0
92.0
96.3
54.9
$1.4
89.3
94.3
99.6
99.9
82.2
96.3
98.4
ALBEF Liet d= 2011
4M
73.1
91.4
96.0
56.8
$1.5
89.2
94.3
99.1
99.8
82.8
96.7
98.4
VLMO
2001
4M
74,8
93.1
96.9
57.2
82.6
89.8
92.3
99.
79.3
95.7
97.8
96.2
56.9
95.0
98.0
Omni Wang
89.5
97.0
Tag Val (GUS))
4NI
74.9
92.5
96.2
36.6
31.5
$3.3
04.3
98.5
59.6
95.5
97.66
Tag2Test Swin (Ours)
4M
77.5
94,1
97.2
60.0
83.3
$9.9
94.8
99.6
100.0
84.2
96.7
98.5
Pre-imined with more Impges::
FLAVA (Sinehet al 2122
70M
61.5
82.1
89.6
50.1
74.4
83.2
85.4
95.7
98.3
73.2
92.7
95.5
UNIMO
5.7M
89.4
98..9
99.8
78.0
94.22
97.1
METER-CTIR
404M
76.2
93.2
96.8
$7.1
82.7
94.3
99.6
82.2
96.3
98.4
93.5
96.9
59.9
ALBEF
77.6
94.1
97.2
60.7
84.3
90.5
95.9
99288
97.5
98.9
BIIPP [M
2012
14M
78.4
Tog2Teu Vil
14M
77.8
93.9
97.0
39.3
83
89.7
94.9
99.7
100.0
83.3
96.4
98.1
Tag-Test-Siin
98.5
Reststrop (L) (L) al
80.6
05.2
97.6
63.1
91.1
96.6
97.5
ONN
Table Performance comparison of image-text retrieval on the COCO and Flickr30K benchmarks. BLIP* refers to the result of our reproduction. + Bootstrop indicates the two stage dataset bootsrap- ping approach using generation and alignment tasks.
4.5 ANALYSIS OF TAGGING GUIDANCE
In this section, we present a detailed analysis to investigate the effectiveness of tagging guidance.
Evaluation of Tagging Guidance. In Table 5 we verify the superiority of incorporating tagging guidance wide range of downstream benchmarks, including four generation benchmarks, two retrieval benchmarks, and two recognition benchmarks.
Image Captioning
Image-Text Resieval
Image Tagging
Pre train
Methods
OPPO-ZS
Opealmages-22 225
COCU-FT NoClap--ZS
COCO-PT
Hickee2
CUCO-FT Upentmages-22
#lmages
Precision
Recall
Precision
Recall
CIDir
CIDEr
TR@1
1R:01
mAP
w/ Guiddane
4M
45.3
72.0
88.8
56.1
W Tag Gaidanes
4M
$1.4.19
$2.0
75.1
52.0.14
124.6..
89.5.
74.9.6
50.6.35
9014
75.2
78.3
82.9
W Tag Guidmeee
14M
82.4
$4.9
76.7
53.0
128.9
101.5
77.8
59.3
92.7
78.7
78.2
83.4
Table 5: Evaluation of tagging guidance on eight downstream benchmarks with finetuning (FT) and zero-shot (ZS) settings. refers to the method which cannot be directly transferred to the corresponding benchmark.
8

---

Published as a conference paper at ICLR 2024
Captioning Performanes (a (inee)
132
CIDE
Precision Recall
Tooging Performnne (nniess
Tagafeet
MiL Mll-Decader
Bruuk Dacage
Detli (oajec Desectinn
1.0
frotes fonce, din,
Plos, winh Aoragra on
periort Lentng suarf, frist inittes thes rubber hond
131
wene werve now
sland,
0.8
groesy young, yellow
230
eceat eattr. seggel, diore, flick, lend, radey,
phottemoph Mack of woter
bird peraon boot,
horms, camn agn cartann stroctlight
canninge, streat
vor corote, uqu dos.
227
0.2
people. Horkk
transport property.
wagon postenger, sh
comn reseeed
persors owning
0.90
Tagging Threshold
Figure 6: The strong correlation between cap- tioning performance and tagging guidance per- formance of tag2text demonstrates that tagging guidance exerts significant control over image captioning. lines with the left axis: image captioning performance. lines with the right axis: tagging guidance performance.
Figure 7: The comparison of recognized tags between Tag2Text and other SOTA models for multi-label recognition (ML-Decoder |Ridnik et al. (2023) ) and object detection (Detic (Zhou al (0222) ). Tag2Text offers more comprehen- sive and commonly used tags including objects, scenes, attributes, and actions.
Controllability Analysis. We provide the analysis of the controllability of tagging guidance for image captioning. We manipulate the threshold of the tagging head to obtain tagging guidance of varying quality. As depicted in Figure the captioning performance (evaluation on COCO) declines when the precision or recall of tagging (evaluation Openlmages) is low. These results effectively establish that tagging guidance exerts significant control over image captioning.
Better Bridge between Image and Text. In order highlight the superiority of Tag2Text in tag recognition, we compare the recognized tags with other SOTA open-source models on multi-label recognition and object detection. For multi-label recognition, we employ the ML-Decoder (Ridiik et al. 2023) model based on Openlmages (Kuznetsova et al. (2020) of 9,600 categories. For object detection, we employ the Detic (Zhou et al., 2022) model based on LVIS (Gupta et al. 2019) of 1,203 categories. The comparison results are illustrated in Figure Z ML-Decoder recognizes many tags which are not frequently used and lacks many obvious common tags. the other hand, Detie limited to only recognizing object categories. In contrast, Tag2Text provides more comprehensive and widely used set of tags, including objects, scenes, attributes, and actions.
Ablation Study. Despite the presence of noise for tags parsed from the texts, our model design enables Tag2Text to leverage tags with noise and achieve exceptional image tagging perfor- mance. As demonstrated in Table [6] the in- tegration of vision-language pre-training tasks into the model also improves the tag recogni- tion ability. Furthermore, Table 6 highlights two-stage "pre-training + finetuning" paradigm in the context of multi-label recognition. The model, trained solely on the limited COCO dataset, fails generalize well the OpenIm- ages dataset, attaining an mAP score of 57.5.
Methods
Pre: Pre-train #Images
Finetune
COCO
Openlmages mAP
on COCO
mAP
Training with full annotations image tags:
MI-
12.8
85.8
Trxining with image text pairs
Tagging
Tag2Texi
4M
X
74.7
76.4
Tag2Text
Teg2Text
4M
78.3
82.9
Tug2
14M
78.2
83.4
Table 6: Ablation study image tagging. The representation of background color is consistent with Table
However, when pre-trained on large dataset, our model exhibits remarkable performance, even in the absence any exposure to the training images from the OpenImages dataset, achieving an mAP score of 83.4, which comparable to the fully supervised performance of 85.8 mAP.
CONCLUSION
This paper has presented Tag2Text, a vision-language pre-training framework, which introduces im- age tagging into vision-language models. Tag2Text achieves superior image tag recognition ability by exploiting fine- ame-grained text information. Moreover, Tag2Text leverages tagging guidance and ef- fectively enhances the performance and controllability of rision-language models. On wide range of Dision-language tasks, Tag2Text demonstrates the value of tag as a bridge between image and text to infuse structure and knowledge information into vision-language models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 62172101), the Science and Technology Commission of Shanghai Municipality (No.22D21 100101, No. 21511100500), and the OPPO Research Foundation.
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 8948-8957 2019.
Hangho Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Sub- hojit Som, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of mixture-of-modalit of-oodaliy--- experts. arXiv preprint wrXiv:2111.023333 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recog- nizing human-object interactions in images. In Proceedings of the IEEE international conference on computer vision, pp. 1017-1025, 2015.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209. 06794, 2022.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. European conference on computer vision, pp. 104-120. Springer, 2020.
Tat-Seng Chua, Jinhui Tang, Richang Hong. Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: real-world web image database from national university of singapore. In Proceedings of the ACM international conference on image and video retrieval, pp. 1-9, 2009.
Ji Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. ICLR, 2021.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision- and-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18166-18176. 2022.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-136 2015.
10

---

Published as a conference paper at ICLR 2024
Hao Fang. Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1473-1482, 2015.
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Com- pressing visual- -isuul-inguuistc model via knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1428-1438, 2021.
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Injecting semantic concepts into end-to-end image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18009-18019, 2022.
The Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adver- sarial training for vision-and- llnguuuge representation learning. Advances in Neural Information Processing Systems, 33:6616 666688, 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Saurabh Gupta and Jitendra Malik. Visual semantic role labeling, arXiv preprint 2015.
Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Bo Ren, and Shu-Tao Xia. voca-boca multi- label classification via multi-modal knowledge transfer. CoRR, abs/2207.01887, 2022. doi: 10. 48550/arXiv.2207. 008877. URL https: //o::///oiiioe/ wtwpoooooo....ooo http:///...4550///xxx..x. 2207. 01887
Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang, Jianfeng Gao, Lijuan Wang, and Zicheng Liu. Vivo: Visual vocabulary pre-training for novel object captioning. proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1575-1583, 2021.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17980-17989, 2022.
Xinyu Huang, Youcai Zhang, Ying Cheng, Weiwei Tian, Ruiwei Zhao, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Xiaobo Zhang. Idea: Increasing text diversity via online multi- label recognition for vision-language pre-training. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4573-4583, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual--emantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision Vision-and-Aa and man-lang transformer without convo- lution or region supervision. In International Conference on Machine Learning, pp. 5583-5594 PMLR, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73 2017.
11

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-T Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-1981, 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- imagg pre- training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-
image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang- Unimo: Towards mndal understanding and generation via cross-modal contrastive learn- ing. arXiv preprint arXiv:2012. /5409, 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 127-137. Springer, 2020b.
Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point de- tection and matching for real-time human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 482-490, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label classification, arXiv preprint arXiv:2 10834, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 10012-10022, 2021b.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17X 2017.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint ww..:::203.02/1121
2022.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phrass correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard nega- tives for vasion-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6967-6977, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7008-7024, 2017.
Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zennik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 82-91, 2021.
Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch, and Asaf Noy. MI-decoder: Scal- able and versatile classification head. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 32-41, 2023.
Christoph Schubmann, Richard Vencu. Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint arXiv:2 1::::2111.00 1.02/ 14, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565, 2018
Amanprect Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
15638-15650, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. Minivlm: smaller and faster vision-language model. arXiv preprint arXiv:2012. 06946, 2020.
Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. arXiv preprint 211iv2211..10223 2021a.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint w:Xiv:2205.14000 2022a.
Junke Wang. Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXiv:2209.07522 2022b.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple eqquence-to-sequene learning framework. In International Conference on Machine Learming, pp. 23318-23344 PMLR, 2022c.
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for all vision and vision- languaee tasks. arXiv preprint arXiv: 10442, 2022d.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108, 2021b.
13

---

Published as a conference paper at ICLR 2024
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei- Ying Ma. Unified semnnl-s embeddings: Bridging vision and language with structured meaning representa- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6609-6618, 6616,66626 2019.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint urXiv:2111.08277 2021.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021a.
Youcai Zhang, Yuhao Cheng, Xinyu Huang, Fei Wen, Rui Feng, Yaqian Li, and Yandong Guo. Simple and robust loss design for multi-label learning with missing labels. arXiv preprint arXiv:2112.07366, 2021b.
Yue Zheng, Yali Li, and Shengjin Wang. Intention oriented image captions with guiding objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3395-8404, 2019.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. ECCV, 2022.
14

---

Published as a conference paper at ICLR 2024
A PRE-TRAINING DETAILS
A. IMPLEMENTATION DETAILS.
The encoder-decoder used for generation and encoder for image-text alignment 12-layer transformers Vaswani et al. 2017) initialized from BERT Base (Devlin et al. 2018). The generation decoder and alignment encoder share parameters with the cross-attention layers. The tag recognition decoder is 2-layer transformer initialized from BERT Buse and shares parameters with the lowest 2-layer of the interaction encoder. The models are pre-trained for 20 epochs with the batch size of 960 on 8 NVIDIA A A100 GPUs. The optimizer is the Adam (Loshchilov & Hutter 2017) with weight decay of 0.05. The learning rate warmed-up to le in the first 3,000 iterations, and then follows linear decay with rate of 0.9. The input images resized to 224 224 uniformly during the pre-training stage. Due to the presence missing labels in the parsed tags and an imbalanced distribution of positive and negative samples, we employ Asymmetric Loss (ASL) (Ridnik et al., 2021 for image tagging optimization.
A.2 PRE-TRAINING OBJECTIVES.
Image Tagging. Image tagging is generally decomposed into multiple binary classification with binary cross-entropy loss (BCE) to optimize:
Tragning = -Byw ((CE(y,(yy)))
c
= -By
Σ yi log (y:) + - y:) log(1 P P(xi))
where yt represents the label for the i-th category and denotes the total number of categories. We employ Asymmetric Loss (ASL) (Ridnik al. 2021 for optimization instead of BCE.
Lragging
-By-D [ASL(y, P(y))]
C Σ
Bynn
y: P(y.)) log (y::
(2)
+ (1- P (yx)))
Image-Tag-Tees Generation. The pre-training objective for image-tag-ter generation is Language Modeling Loss (LM) (Brown et al. 2020 to maximize the likelihood of the text in an autoregressive manner:
LIM = -Ex [CE(x, P(x))]
N
= -
Σ
log P (xi xci)
where I; represents the i-th token in the text and N denotes the total number of text tokens. Com- pared with the bidirectional Masked Language Modeling (MLM) (Devlin et al.] 2018). the unidi- rectional LM Loss is also gradually widely used in recent VLP studies (Wang et al.) 202 2022a
Chen et al. 2022 Li et al. 2022), as it enables seamless transfer of the model to text generation lasks.
Image- Text Alignment. The pre-training objectives for alignment utilizes Image- Text Contrastive Loss (ITC) based multi-modal feature similarity [Radford al. 2021 Li et al.) 2021 Bao
et al. 2021: et al. 2022) and Image-Tex: Matching Loss (TTM) based on multi-modal feature
fusion (Li et 2021
Bao et al. 2021 Li al. 2022
15

---

Published as a conference paper at ICLR 2024
Liee -I,TDD P0)
CITM = -I,TDD BCCCCCE,PPP P(L, T))
(5)
B TAG CATEGORY DETAILS
Pre-training Dataset. The pre-training dataset statistics, including the number of texts and tags parsed from texts, are presented Table 7
COCO
VG
SBU
CC-3M
CC-12M
#images
113K
100K
849K
2.81M
10.26M
#texts
567K
769K
849K
2.81M
10.26M
#avg texts
5.02
7.69
1.00
1.00
1.00
#tags
791K
607K
1.56M
5.93M
23.16M
#avg tags
7.00
6.07
1.84
2.11
2.26
Table 7: The statisties of the pre-training datasets.
Tag Category System. We obtain annotation image tags parsed from its paired text. We con- struct our tag calegory system based the principle that tags with higher frequency more significant, as they represent common elements in image descriptions. To this end, we process million open-source image-text pairs (COCO, VG, SBU, CC-3M), and select the 5,000 most fre- quently occurring tags. Further filtering by human annotation results the selection of the most commonly human-used 3,429 categories of tags (including objects, scenes, attributes, actions). The tag category statistics are presented in Table g An illustration of the tag categories is provided in Figure 88 where the size of each word proportional to the frequency of the category in the open-source image-text pairs.
Object/Scene
Attribution
Action
Total
Categories
3,012
177
240
3429
Table 8: The statistics of tag categories recognized Tag2Text.
hountain
S
grgenn
rocks
treet
Flower
peach
Fiwe
Will
ky
whi
dress
Show
ca
Malko
teansstage
tage
tabl
fustinn
laos
bluee
wiwn
window
arrisshhuus
e
bad
dogs Bara e
Fend
S
SOO
Poomter
perform
Youn
city
woma
playlay
black
ield
player
Tedi
people
building
two
Ball
Peeooc Footballl
gnight
Figure 8: Illustration of the most-frequent categories in our tag category system. The word size is proportional the category frequency in the pre-training image-text pair dataset.
Comparison with Public Datasets. This section provides the statistics of the overlap between our
tag categories and other widely used public datasets. Table 9 shows the statistics for object/scene
categories with other datasets including OpenImages (Kuznetsova et al. 2020). COCO (Lin et al.)
2014), ImageNet (Deng et al. 2009), CIFAR100 (Krizhevsky et al."
2009). Table 10 presents
16

---

Published as a conference paper at ICLR 2024
the statistics for action categories with other datasets including HICO (Chao al. 2013). V- COCO (Gupta & Malik 2013), HOI-W (Liao et al., 2020). To the best of our knowledge, we do not find appropriate public datasets for the recognition of attribute tag categories.
Categories Original 19982
OpenImages
COCO
ImageNet
CIFARIO0
91
1000
120
Overlapping
1988
73
358
94
Table 9: The statistics of object/scene categories overlapping with other public datasets.
Categories
HICO
V-COCO
HOI-W
Oriiiall
117
23
Overlapping
90
26
Table 10: The statistics of action categories overlapping with other public datasets.
Impact on Vocabulary Set Size. In Table I we expand the vocabulary set from 3,429 to 4,585 categories and compare the performances. Notably, the tagging performance decrease with the larger vocabulary set. We attribute to two possible reasons: /) The increased complexity in training with more categories. 2) The additional categories leading more noise, as they lack sufficient training data, thereby impacting the model's efficiency.
Vocabulary Set Size
OPPO
OpenImages
3,429
81.7 80.3
84.1
4,585
83.1
Table 11: The statistics of action categories overlapping with other public datasets.
C IMAGE TAGGING DETAILS
Tuning Dataset. We employ tags parsed from COCO Caption (Lin et 2014) for image tagging finetuning. Each image in the COCO Caption dataset is accompanied by five descriptive sentences, offering comprehensive description of the image. As result, the tags parsed from these captions are considered to be close approximation of complete set of tag labels.
Test Benchmarks. We respectively take the overlapping categories of Tag2Text with the tagging benchmarks for evaluation. The statistics the image tagging benchmarks set are shown in Ta-
ble 12
Tagging Head Comparison. Table investigates the impact of various tagging recognition heads on the performance of Tag2Text. The results show that transitioning from full connection to tag recognition decoder (Liu al. 2021a) results improved performance in image tagging recog- nition, followed by improvements in caption generation. This indicates that the enhancement of tagging recognition leads improved text generation. To mitigate the increase model parame- ters, we propose sharing the parameters of image-tag recognition decoder and image-tag interaction encoder, reducing the parameters and further boosting the performance.
Control of Tagging Guidance. During the image tagging inference process, the tagging head out- puts logits (ranging from 0 to 1) for each category. These logits are compared to a set threshold to determine the output tags. When the logits exceed this threshold, the corresponding tag category is outputted. Therefore, the tagging guidance be controlled by adjusting the threshold. For in- stance, a lower threshold yields more image tags, resulting in higher recall. On the contrary, a higher threshold increases precision.
17

---

Published as a conference paper at ICLR 2024
Benchmark
#Category
#lmages
OРPO
200
44,606
OpenImages
214
57,224
COCO
80
5,000
NUS-WWIH WIIE
50,720
Table 12: The statistics of image tagging test benchmarks.
Recognition
Caption-FT (COCO)
Caption- ZS
Tagging ZS (OpenImages)
#Parameters
Head
(NoCaps)
Full Connection
392M
120.6
86.6
79.8
Recognition Decoder
409M
121.2
87.0
81.2
Recognition Decoder
394M
121.6
87.2
81.9
(Layer Shared)
Table 13: Tagging recognition head comparison.
D IMAGE CAPTIONING DETAILS
Finetuning Strategies Comparison. This section discusses two strategies employed by Tag2Text for image captioning finetuming based on input guidance tags, as illustrated in Figure 9 ! and @ The first strategy, similar image-tag-text generation in the pre-training stage, involves input guidance tags parsed from the paired text. This enables the model to utilize all available tags to create a comprehensive
description. However, Tag2Text usually recognizes tags with similar meanings (e.g.. "man", "person"), "), which may result in redundant sentences (e.g., "a man while person ...") with low evaluation metries.
The second strategy involves using the same process as the inference stage, where the input guidance tags for fine-tuning are recognized by the model. This approach allows the model select guidance tags for generating more precise text generation. In this paper, we utilize mixed training approach that combines both strategies.
Parse
Tag
Tag2Text
Text
Recognize
Finetuning
Tag
Tag2Text
Text
Recognize
Inference
Figure 9: Illustration of two finetuning strategies for Image Captioning. By controlling the ratio of
the two strategies, Tag2Text can generate more comprehensive (0) more accurate (2) descrip- tions.
More Example Results. In Figure 12, we show more examples of Tag2Text using tag guidance to generate comprehensive descriptions.
E ADDITION ZERO-SHOT EVALUATIONS
In this section, conduct addition zero-shot evaluations NUS-WIDE Chua al. (2009), well-established tagging benchmark including 81 categories. All images in LUS-WIDE are out-of- distribution data, since Tag2Text did not utilize any NUS-WDIE training images during its training process. The results are presented the Table [4] Notably. Tag2Text also demonstrates superior zero-shot performance, exceeding both CLIP and BLIP, while utilizing much less training data.
18

---

Published as a conference paper at ICLR 2024
Pre-train
Evaluation
Methods
F1
Precision
Recall
#Images
Paradigm
CLIP
400M
Alignment
46.0
54.0
40.1
BLIP
129M
Alignment
45.0
54.0
38.6
Tag2Text
14M
Tagging
46.4
54.7
40.3
Table 14: Zero-shot Performance Comparison on NUS-WIDE.
EVALUATION ON VISUAL QUESTION ANSWERING
Visual Question Answering aims to predict an answer to question based on image. Previous approaches typically treat VQA as multi-class classification problem with a fixed set of answer choices. In contrast, Tag2Text employs encoder-decoder architecture for generation, which is suited for generating free-form answers. As shown in Figure C), the question, joint with tags, interacts with image features in the encoder, and then forwards to the decoder generate free- form answer.
We conduct experiments the VQA v2 (Goyal et al. 2017) benchmark for Visual Question An- swering. Table T1 shows that Tag2 outperforms or achieves competitive results with other ap- proaches. On the one hand, Tag2Tex Tex- also ought not to be inferior ALBEF, as the structure of Tag2Text degenerates into that of ALBEF without tagging guidance. We attribute the slightly insufficient resources available for conducting hyper-parameter search.
inferior performance
On the other hand, we observe that the VQA v2 dataset is characterized primarily by straightforward questions and answers (e.g.. "Is there big tree behind the clock? Yes."), which is challenging di- rectly augment through identified tags. We anticipate that more strategic utilization of fine- fine-graing grained positioning information derived from tagging guidance, combined with more complex benchmarks, can effectively highlight the superiority of tagging guidance for VQA tasks. We leave these explo- rations for future research.
Answer
Method
Pre-train #imagos
test-dev
test-std
VILT Kim el al 20211
4M
71.26
Image Feature
Interact Encoder
Generate Decoder
UNITER
Ningh
Chen di di ATAT
4M
72.80
72.70
72.91
OSCAR at
2020b
73.16
73.44
VILLA
2020
4M
73.59
73.67
UNIMO La el al
5.7M
75.06
75.27
ALBEF
14M
75.84
76.04
Question
Tag
[BOS]
Tag2 Text- VIl
14M
75.32
75.41
Teg2Text-Swin
Figure 10: Illustration of Tag2Text on VQA
Figure 11: Performance comparison on the VQA v2 benchmark.
finetuning.
G LIMITATIONS
Hallucinatory Captions. Tag2Text benefits from its powerful tagging capabilities. As depicted in Figure 7 there is strong correlation between captioning performance and tagging guidance performance. In practical applications, we observe that incorrect user-provided tags may lead to hallucinatory captions.
Small Objects. addition, evaluating tagging model capable on 3,429 categories is also challeng- ing. Our quantitative comparison and visual validations reveal that Tag2Text efficiently recognizes common objects and scenes, yet struggles in small objects (c.g., spoon or baseball). Our empir- ical experiments indicates that increasing the resolution during fine-tuning significantly improves performance on these small objects.
19

---

Published as a conference paper at ICLR 2024
arts
Tour
A man starding af top mountain looking at flog flying over city.
Small dog playing tey lying on green grass.
woman red dress posing legs crossed posing for picture.
collection of famous loges and names.
An old truck or trailer shelves full of small pots of plants.
alan
abm
A red deor and window surrounded by green plants and
tree full bloom with pink flowens next to a body of water.
large elephont walking civer with mauntains in the background.
Business people sitting bar having drinks and talking to bartender.
tourist poses with young girl old street:
Figure 12: More image captioning results. Tag2Text integrates recognized image tags into text generation guiding elements (highlighted in green underline), resulting in the generation with comprehensive text descriptions.
20


Reviews:
[{"review_id": "VlsCVYBflf", "reviewer": "Reviewer_N6Aw", "summary": "This paper introduces a novel framework for vision-language pre-training. The core insight here is that extracting tags from text paired with images offers robust semantic guidance for vision-language pre-training, and these tags can be easily mined from an existing pipeline. Leveraging these mined tags, the paper presents a multi-task learning framework designed for vision-language pre-training. This framework concurrently handles image captioning, image tagging (multi-label classification), and Image-Text Matching. The paper conducts an extensive array of experiments, with the results demonstrating the significant impact of image tagging in this pre-trained framework.", "questions": "1. Could the author give a short explanation of the meaning of title \"Tag2Text\"?\n2. Performing a multi-label classification with 3,429 categories is non-trivial. The reviewer wonder if the authors come across any difficulties? And what the performance of the learned classifiers on those category (for example, on a validation set of VL pretraining)?\n3. Have the authors ablate VL pretraining framework with a single task, for example, Image-Text alignment only based VL, Image-tagging only based VL, Image captioning only based VL?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Including image tagging task to vision language pretraining seems to be promising, which particularly benefit image captioning.\n2. The proposed design that combines multi-tasks for vision language pretraining is interesting.\n2. The experimental results are promising.\n3. The paper is well-written in general and easy to comprehend.", "weaknesses": "1. This paper discusses VL pretraining methods that are either based on 1) object detection or 2) image tagging (the proposed method) and argues that image tagging is faster and introduces significantly fewer parameters. However, this may not be a compelling motivation for choosing image tagging, as object detectors are fixed (with no additional learnable parameters) and only need to be executed once before training, incurring marginal computational cost compared to the training phase.\n\n2. It would be beneficial for this paper to incorporate specific mathematical formulations to provide a more comprehensive description and discussion of the fundamental problems that require resolution.\n\n3. There is room for improvement in the typesetting.\n\n4. Table 1 appears to be disorganized, making it unclear which numbers to focus on and compare. It is advisable to separate results for different tasks into distinct tables.\n\n5. The \"SOTA\" comparison for image tagging (multi-label classification) seems to omit a substantial portion of recent works. As a result, the reviewer maintains a skeptical stance concerning the associated claims and conclusions.\n\n6. Metrics such as Precision, Recall, and F1 score, which are commonly used for image tagging and multi-label classification, are notably absent from the results.\n\n7. The construction of the tag category system appears to involve human annotation in the process (Section 3.2), which contradicts the earlier claim of being an \"automatic\" approach (Section 1).\n\n8. Conducting an ablation study on the choice of vocabulary set (tag set) size to be mined could offer valuable insights into the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MtYflFGaJd", "reviewer": "Reviewer_pdv8", "summary": "This paper describes a straightforward approach to improve Vision-Language models by using tag information extracted from image captions. By utilising standard techniques and losses they show very decent results on tagging, captioning and retrieval tasks\n\n**After rebuttal**: I appreciate the authors' rebuttal. I think the original comments I made are still relevant. The paper definitely makes a decent contribution and I wouldn't mind seeing it published at ICLR. However I'm not sure I can increase my score to 8.", "questions": "No specific questions", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written and easy to follow. \n2. The method is simple and is shown to work well. The idea of using image tags to aid VL pre-training makes a lot of sense. \n3. The results are not always SOTA but are very good. Results on multiple tasks/datasets are provided. I believe the results are sufficient to show that the main idea behind the paper works as well as expected.", "weaknesses": "I think the main problem with the paper is that all of its components have been proposed before so the paper looks more like a re-implementation of known ideas with more recent architectures and pipelines which is of course expected to work better. Specifically the main idea of using tags to aid VL pre-training appears in many works including OSCAR or more recently in DiHT (Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training) while other losses used like I2T are very commonly used in most works in VL pre-training. But the method for sure can serve as strong baseline. \nSomewhat less important concern: as most experiments are on COCO/Flick which are datasets very close to the ones used for training I am wondering whether the authors could carry out an experiment on out-of-domain data .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekIi9di4rb", "reviewer": "Reviewer_gXiT", "summary": "This work introduces TAG2TEXT, incorporating object tag information into vision-language pre-training. The authors demonstrate its effectiveness across various downstream tasks. In comparison to widely-used object detectors, this method is not only faster but also enriches the model with finer-grained information.", "questions": "In Figure 1(b), the authors mention 'Actions.' Could you please clarify the source of this particular type of data or tag? Are there any documented instances of failure cases for Tag2Text?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea is straightforward and easy to grasp.\n- Given the high cost of human labeling and the limited diversity in generating captions for Image Captioning models, such as BLIP, finding new methods for generating comprehensive captions is both challenging and valuable.\n- This work offers a potential benefit in the context of person re-identification (REID).\n- The motivation behind this approach is also quite appealing.\n- Notably, the Tag method diverges from the commonly used Faster R-CNN-based object detectors and demonstrates significantly improved speed.", "weaknesses": "- Data and Pre-training Settings: The author of the paper used a 4M setting, which includes training data from COCO (Common Objects in Context) and Visual Genome. NoCaps data is sourced from OpenImages and COCO. Importantly, this work does not incorporate any out-of-distribution data. The success of this approach on COCO-related tasks is attributed to the similarity between the pre-training data and COCO style, and it's noted that many works beyond BLIP face similar challenges.\n\n- TagEval Task: TagEval is mentioned as a task, but it is not considered popular, and its persuasiveness is limited. Models trained on tags are noted to perform well in this case.\n\n- Tag Introduction in Pre-training: The introduction of tags in pre-training is not a novel idea. There are existing works, like OSCAR, that have explored the concept of using tags to improve vision-language pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7u8zGLppk", "reviewer": "Reviewer_Kr1V", "summary": "This paper proposes a new Visual Language Pretraining (VLP) framework named as Tag2Text, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. It utilizes large-scale annotation-free image tags parsed from image-text pairs through text semantic parsing, and demonstrates a foundational image tagging capability with superior zero-shot performance. Tag2Text re-introduces tag guidance into detector-free vision-language models by seamless integrating image tagging, effectively enhancing the performance of both generation-based tasks and alignment-based tasks. The proposed framework is evaluated over a wide range of downstream benchmarks, and achieves state-of-the-art results with similar model sizes and data scales.", "questions": "Please try to address the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe idea of the utilization of image tags parsing from large-scale image-text pairs is interesting, efficient and effective from improving the performance of vision-language models.\n2.\tThe framework of Tag2Text employs a multi-task pretraining approach, including Tagging, Generation, and Alignment. These tasks are reasonable and share the same visual features obtained from the image encoder, guaranteeing the efficiency of the framework.\n3.\tA large number of experimental results over image tagging, image captioning, and image-text retrieval tasks prove the effective of the proposed methods.", "weaknesses": "1.\tThe detail of text semantic parser is not clear. Although it is based on existing work of [Wu et al. 2019], it should make clear how to obtain the corresponding tags in the paper.\n2.\tIn Fig. 2, it is difficult to understand which are users’ input desired tags, since they share the same forms of the recognized image tags.\n3.\tIn the Image-Tag-Text Generation paragraph, the introduction of this task is not clear. According to Fig. 4(c), it seems the text embedding should not be used as input, which is conflicting with the introduction in the corresponding paragraph.\n4.\tIn the experiment on controllability analysis, it is not clear how the threshold of tagging head to control the tagging guidance.\n5.\tThere are some typos in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RETplhNAvl", "reviewer": "Reviewer_Nwx3", "summary": "The paper proposes the use of tags as part of an otherwise standard V&L model. Tags are incorporated by parsing the captions to produce tags within a vocabulary, and then simply predicting the tags in a standard fashion. Secondly, tags are used, in combination with image features, to predict the caption text in an autoregressive manner. Good results are shown for tagging and for captioning, while some level of control over the generated text by using input tags is also obtained.", "questions": "1) The tag system works based on pre-existing vocabulary. What happens with out-of-vocabulary terms? \n2) Have you tested retrieval based on short queries? E.g. retrieval using a term within the vocabulary vs some short retrieval query that is not part of the vocabulary (e.g. adjective + noun) vs retrieval with long captions typical of flickr/coco", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I really like the paper. It is clearly motivated, well written, technically well executed, has clear novelty, and has good experimental results. But I want to highlight one thing that is surprisingly hard to find: it seems useful. The reasons: 1) Current V&L models are trained with long captions, which means small tag-like queries are not well supported. There are indeed some object detection methods a-la CLIP (owl-vit and co) but they offer different functionalities and have different requirements. So this is a good addition to the V&L toolbox. 2) it offers a way of controlling caption generation through the use of input tags. \n\n\nMinor suggestions (up to the authors and no reply needed):\nFig. 1 might get a bit confusing as both the \"prior work strategy\" and the current strategy are included in the same flow graph.\nTable 2 shows the last 3 methods seem apart from the rest but it's unclear why they are separated.", "weaknesses": "The method is based on some relatively standard techniques that are however well executed and put together. While not much of a minus, but maybe a reason for an accept vs strong accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)