LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
Authors: Wei Wu
Kecheng Zheng
Shuailei Ma
Fan Lu
Yuxin Guo
Yifei Zhang
Wei Chen
Qingpei Guo
Yujun Shen
Zheng-Jun Zha
Venue: NeurIPS 2024 poster (2024)
Keywords: Language-Image Pre-Training
Primary area: natural_language_processing

Abstract: In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method achieves superior performance in long-text-image retrieval tasks. The project page is available at https://wuw2019.github.io/lot-lip.

LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
Wei Wu!
Kecheng Z
Shuailei Ma
Fan Lu!
Yuxin Guo
Yifei Zhang
Wei Chen
Qingpei Guo?
Yujun Shen?
Zheng-Jun Zha
University of Science and Technology of China 2 Ant Group ZZejiann University
*Northeastem University, China
Instittte of Automation, Chinese Academy of Sciences Shanghai Jiao Tong University
Abstract
Understanding long text is of great demands in practice but beyond the reach of most language-image pre-training (LIP) models. In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method achieves superior performance in long-text-image retrieval tasks. The project page is available here.
Introduction
Understanding long texts plays a key role in Natural Language Processing (NLP), e.g., Long Document Analysis [23, 28, 33], in which books, academic papers, and many other types of long texts can be the target of such analysis. Inspired by these works, in the multi-modality field, some text-to-image generation works (e.g.. DALLE-3 [1] and Pixel-art [4, 5]) employ pre-trained captioners (e.g., LLaVA [21]) to generate more accurate and detailed captions (in other words, long captions) for images. These long captions describe an image in detail which can help text-to-image models easily transfer long text to an image, improving the quality of text-image alignment. However, these text-to-image models use pure NLP encoders (e.g.. T5) to model long captions rather than CLIP. Because, in the iagguage-image pretraining task, little work has been conducted on modeling long texts in a way that effectively aligns with image representations. Despite the lack of exploration, this is a critical problem with practical demands that require nuanced understanding of textual descriptions corresponding to images.
Two primary challenges hinder the effective integration of long-text understanding in language-image pre-training. The first one is the lack of large-scale long-caption image-text paired datasets. Most
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original Image
froet cutto well hept pardes with prems white raad around the garder. lange ondic has ware with piller-
Shuffling the onder of sentenncs model snill only srea castle tnker
walloaed
"sanden taken orcnhadowed by
There well light parlen with
"aadic token
Understand the image with text caption according to attention visualization
the freec the cnll and
castle token
posel
the gindes
with hege tries avunndln in wells pillet-.
Lang cade fos
P
garden token
Teain with short caption
Underttan with long caption
Trai with shoot caption Understand with short exption
Train with long Unddrrgland with Jans caption
codlc in whllee
kept pirden with whnte and aod Feewors
laieting CLJ modele are good underttandngg short cuptions
Long cxption beinge iganden Inlenn" back into light
end bod, thowess ands perel
large caslle lass wiille wolls
belpe tnzn srrand walls pullslla..
+
castle token
castle token
Indrec the
Them well kape garden with grems while red
garden token
lept ganden with grecs
garden token
liceaad celln A lagg un white wells tire pourdii well and pille
and pased cadd bnamued the pandm lange cm white with ato with and pillnr
Figure 1: Illustration of the impacts of long v.s. short captions on image-language pre-training, as observed in the cross-attention maps of CLIP. Training images are usually paired with short captions, leaving certain tokens (e.g., garden token) easily overshadowed by salient tokens (e.g., castle token). Fortunately, the usage of long captions can help bring the overshadowed tokens back into the light, and this phenomenon is not influenced by the order of tokens within the sentence.
existing datasets focus on short captions (i.e., average text length in CC12M [3] is about 17 tokens), which limits the model's exposure to longer text forms. Consequently, models trained on these datasets tend to neglect certain tokens that are easily overshadowed by salient tokens. As shown in Fig. 1, the model trained on short captions can be good at understanding the content of short captions (i.e., garden and castle). But when increasing the length of the caption, we can that "garden token' is overshadowed by 'castle token', even we move the 'garden token' front. This bias towards the salient tokens ('castle token' in Fig. 1) of texts can severely restrict the model's ability to comprehend and generate responses based on the full context of longer inputs. The second challenge is the token length limitation of the text encoder. While directly increasing the token number limitation that a model can process appears to be a straightforward solution for accommodating longer texts, it does not have better understanding of long captions. The fundamental issue remains the model's inability to effectively interpret long texts, primarily due to the lack of appropriate training data that includes long captions.
To address these challenges, we have undertaken an extensive project to re-caption 100 million data with long captions, aiming to enrich the training environment for our models. This initiative allows us to explore the effects of increased text length in image-text pre-trained models and its impacts on model performance. Based these experiments, causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. However, directly learning with long captions may improve the long-text understanding of image-text pre-trained models, but lead to performance degradation in understanding short texts (e.g., in the image classification task) shown in Fig. 2. After integrating corner tokens to aggregate diverse textual information, we successfully enable the model regain its original proficiency in understanding short texts while significantly improving its ability to comprehend long texts. We also explore whether the model can continue to benefit from longer captions and observe a clear trade-off between performance and efficiency. Moreover, on the task of long-text image retrieval, we beat Long-CLIP [39], a competitor using long captions for fine-tuning, with 4.2% improvement (i.e., from 79.52% to 83.72%)
empirically confirm that the key reason
2 Related work
2.1 Language-Image Pre-training
Recently, using language-image pre-trained models to do zero-shot prediction has attracted a lot of attention. CLIP [24] and ALIGN [14] demonstrate contrastive pre-trained models can learn
2

---

rich visual- language correspondence knowledge from large-scale image-text pairs on the Internet and achieve good performance on zero-shot predictions, including image-text retrieval [18] and classification [9]. Following their success, various studies [16, 34, 36, 17] have been devoted to improving image-text alignment. Among them, FILIP [34] focuses on fine-grained expressiveness between text tokens and image patches by modifying the training loss. While LiT [37] finds that apply contrastive-tunins to the pre-trained models with locked image encoder and unlocked text encoder can further improve the alignment. It is recognized that larger batch size brings better performance. For this reason, SigLIP [38] proposes to replace the softmax normalization among the standard contrastive loss with the sigmoid loss to scale up training batch size. In addition, LaCLIP [11], RLEG [40] and some other works [19, 13, 22, 30] utilize multi- multi-modality generative models to improve data quality for enhancing pre-trate -re-training.
2.2 Long-text Understanding
Detailed long texts are necessary for many artificial intelligence tasks (e.g., human-computer interaction). Therefore, modeling and parsing long texts has become one of the most important research directions in natural language processing. Many studies in fields such as text generation [23, 32, 33] have shown that advanced transformer structures have the ability to interpret long texts. However, in the field of lanuuag-image pre-training. research on using long-text descriptions of images to enhance multimodal representations is still very scarce. DreamLIP [41] utilizes multi- modality large language model to re-caption image data with detailed descriptions and then use them in language-image pre-training. In fact, during training, DreamLIP randomly extracts sub-captions from the detailed description for training and does not completely use all the information of the long text. One of the reasons why previous language-image pre-training rarely directly applied long texts for training is the text encoder of traditional CLIP [24] is restricted by the token number limit (S 77). Then, Long-CLo CLIP [39] firstly introduces long captions into CLIP model to finetune, where the model is pre-trained on short-text immge datasets. The fine-tuning process equips the model with the ability to comprehend long texts, albeit at the expense of its proficiency in understanding shorter texts. In contrast, we incorporate long captions during the pre-training stage, which can not only enable the model to regain its original competency in interpreting short texts but also significantly improves its understanding of long texts. We further explored the trade-off between the benefit of long texts to the model and the training efficiency.
3 Preliminary of Language-Image Pre-training
Language-image pre-training models, i.e., CLIP [24] and LiT [37], typically consist of an image encoder and a text encoder. In the pre-training stage, the language-image image model takes image-text pairs as input and uses image encoder and text encoder to extract embeddings from images and texts, respectively. Then, two encoders are trained with contrastive objectives, ensuring that paired image and text embeddings are close in the embedding space. while unpaired pairs are far apart. Specifically, a batch images {T, /2, sampled from the pre-training dataset. Each image and its corresponding text are treated as a positive pair, while others in the same batch are negative pairs. Then, the image encoder extracts image global feature v from the i-th image 1, within the batch, while the text encoder obtains text feature t' from the j-th text Tj. These two encoders are then optimized using contrastive loss C, which consists of image-to-text loss L and text-to-image loss L can be formulated as follows:
IN} } and the corresponding short texts (Ti T2, ... TNN] J are randomly
L = 21 + 25
(1)
exp (cos(v", t²)/T)
2
log
(2)
exp (cos (v', t²)/T)²
exp (cos(t', ' vt v²)/T)
2
log
exp (cos v²) v²)/7)
where 7 learnable temperature parameter, and cos(:,)) means the cosine similarity between two normalized feature vectors.
3

---

Table 1: Dataset details of long-fext-image retrieval and short-text-image retrieval tasks. We use BERT tokenizer for tokenization. ShareGPT4V-11 and 10k are selected from the ShareGPT4V dataset. For DCI and IIW, images with human-authored long descriptions for evaluation.
Datasct
#Images
# #Texts
#Sub-captions per Text
#Tokens per Text
Long-text-imann Retrieval Dataset
DCT [31]
7,805
7,805
10.81
172.73
IIW [12]
612
612
10.16
239.73
ShareGPT4V-Ik [6]
1,000
1,000
1,
173.24
ShareGPT4V-I0k [6]
10,000
10,000
8.24
173.66
Short-text-image Retrieval Dataset
MSCOCO [18]
5,000
25,000
1.0
11.77
Flickr30k [35]
1,000
5,000
1.0
14.03
4 Long Texts in Language-Image Pre-training
Currently available image-text pair datasets, e.g. CC..C [27], typically include short texts that have an average length of approximately 17 tokens. Language-iman models pre-trained on these datasets perform well on short-text comprehension tasks. However, we find that they struggle to comprehend long texts for text-image aligmment. Concretely, they tend to overlook or neglect some tokens or sub-captions in long texts, as shown in Fig. 1. A potential reason for such a situation is the lack of long-text-image pairs in pre-training. Thus, we collected and re-captioned 100M images with long texts. In this section, we provide details of re-captioning and explore the usage long texts in kunguage-image pre-training.
4.1 Long Text-Image Pair Dataset
Training Dataset. To construct long text-image pairs for language-image pre-training, we re- captioned 100 million images with long texts. Specifically, we collected the images from CC3M [27],
CC12M [27], YFCCI5M [291, LAION [26], and COYO [2] dataset. Then, we instructed three multi- modality large language models (MLLMs), i.e.. InstructBLIP [7], LLa VA [20], and ShareGPT4V [6] to generate diverse and descriptive long texts based on the collected images. In this step. we used "Describe the image in detail." as the text prompt, following DreamL.IP [41]. Finally, each image in the collected datasets paired with four texts: raw text from the original dataset and three re-captioned long texts. The raw texts exhibit an average length of approximately 18 tokens, whereas the re-captioned long texts consist around 136 tokens.
Evaluation Dataset. Most zero-shot evaluation tasks for language-imasn pre-training primarily rely on short textual input. For example, short-exx--imag--errie fewer than 15 tokens on average, as shown in Tab. 1. Given the short texts, these tasks are not suitable for accessing the long text comprehension ability of pre-trained models. Therefore, we collected long text-image pairs from DCI [31], IIW [12], and ShareGPT4V [6] datasets to construct long-text-image retrieval evaluation tasks. Specifically, for DCI and IIW datasets, we use images with
tasks, the textual inputs contain
long descriptions for retrieval. For ShareGPT4V dataset, we sample 1,000 and 10,000 data from ShareGPT4V dataset to construct ShareGPT4V-I1 1k and ShareGPT4V- 10k retrieval dataset, respectively. The ShareGPT4v-11 1k dataset is constructed following Long- images in ShareGPT4V-L Ik and ShareGPT4V. 10k are from SA-1B [15] dataset, and all long texts generated by ShareGP44V-Capionn has one paired long text, which consists of more than sub-captions and 170 tokens on average. Here, sub-caption is complete sentence ending with period. The long-text-image retrieval task necessitates the alignment of text features from long texts with the image features from the corresponding images, which thereby evaluates the model's 's ability to comprehend long texts.
Long-CLIP CLIP [39]. All
[6]. As shown in Tab. 1, each image within these datasets
4.2 Exploring the Influence of Text length
Long-CLIPP [39] enables long text processing ability by fine-tuning CL.P model with long text-image pairs. However, the benefits and drawbacks of using long texts in pre-training are still unknown. It

---

Shore Short-tau taxt wext-txxiigeeeee iemlimmgg Rettrece RGB
Image Classfication Acci83
Leng
DC
N500000
Sub Sub-caption Num
Subicaption Num
Sub- Sub-caption Num
(a) long-text-image retrieval
(b) short-text-image retrieval
(c) image classification
The influence of text length. A significant improvement is observed across all tasks when we added one randomly sampled sub-caption from generated texts to the pre-training stage. As the number of sub-capp cub-captions increases, the performance of the pre-trained model long-text image retrieval tasks consistently improves and becomes stable (a). However, there is a performance degradation in MSCOCO retrieval task (b) and ImageNet classification task (c).
Figure 2:
Long Caption
Image Yimbeddting
In the picture, there are several objects that can be observed living room. These objects include couch, chairs xxx X00000
Image Encoder
Cumir Embedding
lest tokens
[CLS]
curner
bext toleets
CLIP Lise
Multi-Head
Transformer blocks
text
Contraslive
0000000000
tokens
Tioss
[a.s] [Coe
text tokens
Query
Fanedla
Figure 3: Overview of LoTLIP. We add multiple learnable corner tokens (Coor 1]. [Cor 2!.....) after [CLS] token. These corner tokens are initialized differently for aggregating diverse token features. Besides, an attention mask mechanism is used limit the interaction between [CLS] and corner tokens to ensure the diversity of gathered features.
is also unknown how the length of the text affects the performance of the pre-trained model. To explore the influence of using texts in different lengths for pre-training, we change the length of long texts by selecting different numbers of consecutive sub-captions, illustrated in Fig. 2. Each sub-caption in the long texts has an average of approximately 22 tokens. Sub-caption number equal to 0 indicates that the model is trained without the generated texts. When introducing one sub-caption, noticeable gain is achieved across all tasks. Moreover, the results also indicate that pre-training with longer texts, composed of more sub-captions, enhances the performance of the pre-trained model on long-text-image retrieval tasks. It confirms that using long text-image pairs for pre-training improves the model's understanding of long texts. However, the usage of longer texts negatively impacts the model's 's performance on short-text-Ne -mage retrieval and image classification tasks.
4.3
Method
In order to find solution that well balances the long and short text understanding of the pre-trained model, we design to add extra text tokens for text encoders, termed corner tokens, which can aggregate diverse text features. This strategy benefits the class (i.e., [CLS]) token by extracting more representative features for long and short text. Next, we will describe the details.
Corner Tokens. Different text encoder architectures (e.g.. BERT [10], T5 [25]) can be utilized in language-image pre- vre-training. In this section, we take BERT [10] as example of our approach. Given long text, is expressed as [CLS), [SEP].- ... [SP), where first token of every text input is its class token [CLS], and all sub-captions are separated by a special token [SEP]. The omitted part denotes the tokens obtained after tokenizing words within the sub-caption. Based on the tokenized long text, insert multiple learnable corner tokens C {[Cor 1),-.- [Cor m|} after
5

---

the class token, where m is the number of corner tokens. In this way, the form of the tokenized text input is [CLS]. [Cor 1]... [Cor m...... [SEP], ... [SEP]. Moreover, we design an attention mask mechanism A for the text encoder to promote the diversity of the aggregated features. Specifically, when calculating the attention scores, the corner tokens and [CLS] token are guided to neglect each other but attend to all other sub-caption tokens. Meanwhile, in the attention mask mechanism, each text tokens are designed to only interact with other text tokens and the [CLS] token, to keep the interactions between local and global information. The attention mask A is formulated as:
0, if (A: E C g, k E C [[CLS]]) and q A k A(g. k) k) 1, otherwise
where q and represent the query and key tokens in attention block, respectively. The features of the [CLS] and corner tokens regarded as text global feature ty and corner features te,, tezy .... t tcm respectively.
Optimization.
The short-text-image contrastive loss Lahton is calculated in the same way as Eq. (1).
Meanwhile, the long-text-image contrastive loss Liong between image global feature e and bg. teg tezy.... ... tc.. of long text is as follows:
Liong
Llong +
(4)
N Σ
exp (cos(v', , (///T)
exp (cos(v', .t/))T))
2t Long
(log
+
Σ Arm
log
) (5)
2j=1 exp cos (z)/TT)
j= exp cos(vv, tle.)/
i=1
exp (cos(t), v²)/T)
m
exp (cos(t'a io
v')/T)
Long
Σ
(log
log
(6)
Σj=1 exp (t,, v))/T)
ا1=1 j=1 exp (cos(t v2)/T)
The total training loss is LIOTLIP = Liong
Lshont
5 Experiments
5.1 Implementation Details and Datasets
Pre-training Datasets. As presented in Sec. 4.1, we collected 100M data from five publicly available image-text pair datasets and re-captioned the collected images with long texts. Based on this dataset, we construct 4 scales of pre-training data: (1) 3M, including CC3M. (2) 12M, including CC12M. (3) 30M, including CC3M, CC12M, and YFCC15M. (4) 100M, including all re-captioned data. We conduct ablation studies to validate our model on the 3M scale pre-training data. The performance of LoTLIP pre-trained with 12M and 30M scale datasets is shown in the Supplementary Material.
Downstream Datasets. To assess the ability of the pre-trained models on short-text and long-text understanding, we select downstream tasks for evaluation under the zero-shot setting, including long-text-image retrieval, short-image-ter retrieval, and image classification. For long-text-image retrieval, we present the Recall at (R @ 1) metric of the pre-trained models on DCI, IIW, and ShareGPT4V-IK Ik, and ShareGPT4v- 10k long-text-image retrieval tasks. For short-image-text retrieval, we evaluate on MSCOCO [18] and Flickr30k Caption [35] and report Recall at 1/5 (R@ 1/5) metric for comparison. For image classification, and present top-1 accuracy (Aco@ 1) on image classification. Following [24], we use class names incorporated with pre-defined text prompts as text inputs for zero-shot classification.
ImageNetlk [9] for evaluation
Implementation Details. Following LiT [37], we use a vision transformer pre-trained on ImageNet 21K as the image encoder and Bert [10] as the text encoder. The architecture of the image encoder ViT-B/16. We report other variants of vision transformers in the Supplementary Material. The images are resized to 224x 224. The maximum text token length is set to 128 unless specifically stated. Three consecutive sub-captions are randomly selected to form long texts as text input. We train 10 epochs on the 3M and 100M scale datasets. For the 3M dataset, the batch size is set to 2560, while that of 100M is set to 16384. The other pre- me-training hyperparameters are under the same setting, l.g.leannng rate, warmup steps, and weight decay.
5.2 Ablation Studies

---

Table 2: Analyze the effectiveness of LoTLIP in language-imag pre-training with long texts. The architecture the image encoder is ViT-B/16. 12T and T2I indicate R @@ on text and image retrieval, respectively. We use 3M scale dataset for pre-training. "V" indicates we add long texts in the training stage.
Long text image Retrieval
Short text image Retrieval
Classification
Method
Long Texts
DCT
IWW
ShareGPTaV.100
MSCOCO
ImageNet
12T
T21
12T
T21
12T
T21
12T
T21
Acc@!
LiT
27.14
24.13
65.20
58.50
32.73
27.01
34.20
24.07
43.76
LiT
47,96
44.92
84.97
81.70
73.66
66.73
43.52
30.06
48.87
LiT+Long CLIP"
47.21
46.61
83.82
83.01
74.60
67.49
43.78
26.74
46.82
LoTLIP
47.82
84.97
83.33
76.49
69.72
31.59
We apply the primary Component matching strategy proposed by Long-CIIIP [39] LiT and train the model with same training setting using 3M data for fair comparison.
Exploring the Influence of Token Number Limitation. Token number limitation is the maximum length of text input that the text encoder can process at a time. For language- image pre-training models, the token number limitation of text encoder can be arbitrarily set any positive integer, which is typically set to 77 [24, 38, 41]. The benefits derived from using long texts in pre-training might be constrained by the 77 token limitation. To investigate this hypothesis, we incrementally raise the token number limitation from 32 to 512. The results are illustrated in Fig. 4, which indicate a limitation of 77 tokens is insufficient for a model training with long texts. On DCI and Shar:GPT4V- 10k retrieval tasks, the best results are observed when the token number limitation is set to 192. Meanwhile, the highest performance on the IIW retrieval task is reached when the text token length limitation is 256. Since the average text length of the evaluation datasets less than 240 tokens, as shown in Tab. 1. When the text token length limitation gets larger than 256, these datasets can't 't well prove the long-tex understanding ability of the model. In Fig. 4, also illustrate the FLOPs of the text encoder, which increases with the text token number limitation. To balance the training efficiency and performance, set the token number limitation to 128 for the text encoder of LoTLIP.
Long-text- -mage
image
85
FLOPs of Text Encoder
Retrieval R@1
75
45
12T
35
BW
20
SharoGPT4V- 10k I2T
10
77 192 256
Maximum Token Length
Figure 4: Influence of token number limitation on LoTLIP. The performance of the pre-trained model on different tasks improves when the token number limitation increases up to 192, which exceeds the commonly used 77. Meanwhile, the FLOPs of the text encoder (red stars) rapidly increase with the text token number limitation.
Compare LoTLIP with Other Methods in Pre-training with Long Texts. LoTLIP aims to enhance the long-text and short-text comprehension ability of language-image pre-training. To demonstrate the effectiveness of LoTLIP, we compare it with other approaches that train with long text-image pairs. The first approach is directly using long texts in the training stage of LiT. The second approach is based on another contrastive learning method related to long texts, namely Long-CLin CLIP [39]. Concretely, we incorporate the primary component matching strategy and losses proposed by Long-CLIP to LiT LLT+Long-CLIPP For fair comparison, all models are trained with 3M scale dataset. As shown Tab. 2, directly using long texts in pre-training procedural significantly improves LiT over all tasks. Moreover, LiT+Long- CLIP exceeds LiT on the long-text- teng-eex--mmge retrieval tasks when both models incorporate long texts during pre- pre-training. But on short-text comprehension tasks, i.e., short-text-imas -mage retrieval and image classification, the performance of LiT+Long CLIP is inferior to that of LIT. Instead, LoTLIP improves LiT and LiT+Long-CiL text comprehension tasks. Concretely, LoTLIP surpasses LiT and LiT++Long-CLII by 1.97% and 1.51 % on average over three tent-aext- retrieval tasks. Besides, LoTLIP improves LiT by 2.29% and 1.47% MSCOCO retrieval and ImageNet classification, respectively. The results demonstrate
on both long and short
7

---

Table 3: Analyze the influence of the number of corner tokens and the attention mask mechanism. We use 3M scale dataset for training. The architecture of the image encoder is ViT-B/16.
Long-text-image Retrieval
Short-lext-image Retrieval
Classification
Attention Mask
#Corner Token
DCI
IIW
ShareGPT4V-10K
MSCOCO
ImageNet
12T
T21
12T
12T
12T
T2I
Acc
47.96
44.92
84,97
81,70
73.66
66.73
43.52
30.06
48.87
49.57
46.55
84.97
82.68
74.91
68.41
45.68
31.51
49.62
47.82
84.97
69.72
31.59
49.58
47.70
87.09
84.31
76.51
70.20
46.48
31.601
50.36
49.58
48.30
86.76
84.15
76,25
70.14
47.70
31.88
50,59
48.61
61.61
47.17
86.11
81.86
76.14
69.31
47.70
31.34
49.88
2
49.46
47.82
84.97
83,33
76.49
69.72
46.56
31.59
50.34
Table 4: Zero-shot evaluation of different models on long-text-image retrieval tasks. 12T and T2I indicate R@I on text and image retrieval, respectively.
Data Scale
DCI
IIW
ShareGPT4V-Ik
ShareGPT4V-10k
Model
Avg.
Short
Long
12T
T21
12T
T21
12T
T2I
12T
T2I
3M
FILIP [34]
10.85
11.36
31.54
29.08
26.50
26.80
8.64
19.21
3M
LaCLIP [11]
14.84
14.71
41.01
38.89
40.90
37.10
15.81
14.84
27.26
3M
SigLIP [38]
11.66
13.11
29.25
29.58
27.30
25.10
9.92
9.30
19.40
3M
LIT [37]
27.14
24.13
65.20
58.50
63.60
56.80
32.73
27.01
44.38
3M
LoTLIP
49.46
47.82
84.97
83.33
93.20
90.00
76.49
69.72
74.37
400M
CLIP [24]
45.45
43.01
88.24
87,58
84.50
79.80
60.22
56.16
68.12
100M
LiT [37]
41.78
40.90
88.07
82.68
86.00
80.00
61.41
50.69
66.44
700M
ALIGN [14]
56.54
57.41
92.65
90.68
86.30
85.30
65.13
62.73
74.59
12B
SigLIP [38]
57.78
56.22
91.99
85.80
83.40
83.40
63.08
76.59
400M
IM
Long- CLIP [39]
51.68
57.28
89.61
93.20
94.70
93.40
79.24
77.06
79.52
100M
LoTLIP
62.10
61.06
93.95
92.48
96.50
95.50
86.84
81.40
83.72
Long-CLI CLIP fine-tunes pre-trained CLIP model with ShareGPT4V datasct except ShareGPT4V-1k
that LoTLIP benefits from long texts and corner tokens, thereby exhibiting a better understanding of the long and short texts.
Implementation of Corner Tokens. In this part, study the influence of the attention mask mechanism and the number of corner tokens on different downstream tasks as shown Tab. LoTLIP pre-trained without the pre-defined attention mask encounters performance degradation most tasks compared to when using the pre-defined attention mask. indicates that direct interaction between the CLS token and corner tokens limits their ability to aggregate diverse textual features. On short-text-image retrieval and image classification tasks, the performance of LoTLIP improves when introducing more corner tokens. It proves that the corner tokens help with enhancing the short text understanding ability of LoTLIP. When the number of corner tokens is set to more than two, the performance improvement across all tasks is relatively small. Thus, use corner tokens in LoTLIP.
5.3 Main Results
We compare LoTLIP with the state-of-the-art methods on downstream tasks involving long texts and short texts. The experimental results are shown Tab. and Tab. 5, respectively. On 3M data scale, LoTLIP significantly improves the state-of-the-art methods on all tasks. Specifically, LoTLIP

---

Table 5: Zero-shot evaluation of different models short-text-image retrieval and classification tasks.
Data Scale
MSCOCO 12T
MSCOCO T21
Flickr30k 12T
Flickr30k T2I
Model
Short
Long
Acc.
R@1
R@S
R@5
R@1
R@s
R@1
R@5
3M
FILIP [34]
18.69
14.98
34.88
11.86
28.98
30.40
56.80
20.76
44.20
3M
LaCLiP [11]
21.50
18.94
40.40
12.42
31.04
37.00
63.90
29.32
56.04
3M
SigLIP [38]
21.28
16.30
37.36
13.22
31.65
31.00
61.10
23.76
48.64
3M
LiT [37]
43.76
34.20
61.52
24.07
48.37
61.30
89.50
48.06
75.36
3M
LoTLIP
50.34
46.56
72.02
31,59
57.65
75.20
94.20
58.20
83.58
400M
CLIP [24]
68.34
51.68
76.72
32.70
57.76
82.20
96.60
62,14
85.72
100M
LiT [37]
73.70
51.92
75.72
32.74
$7.84
80.00
95.90
60.86
84.62
700M
ALIGN [14]
65.89
60.42
82.50
42,36
67.38
88.90
98.00
74.12
92.40
12B
Iig [38]
76.04
65.46
86,22
47.14
72.10
89.10
98.00
74.66
92.30
400M
IM
Long-CLiP [39]
67.10
57.28
80.78
40.34
65.92
85.90
98.50
70.66
90.60
100M
LoTLIP
72.16
59.66
81.50
38.06
63.81
86.90
97.80
65.22
87.98
improves the second-best method LiT by 29.99% on average over four long-texte tong-text-image retrieval tasks. Moreover, LoTLIP improves the second competitor LiT by 6.58% and 10.98% on image classification task and short-text-image retrieval tasks, respectively. It proves that LoTLIP significantly enhances the language-image model for understanding short and long captions by involving long captions in pre-training and incorporating corner tokens in text inputs. It is worth noting that LoTLIP trained with 100M data exceeds all state-of-the these methods are pre-trained on a larger scale of data. Concretely, LoTLIP (trained on 100M data) exceeds SigLIP (trained with 12B) by an average of 7.13% over four long-text-image retrieval tasks. Moreover, compared to Long- CLIP, which uses long texts in CLIP for fine-tuning, LoTLIP improves averaged performance by 4.2% on seng-text-image retrieval tasks.
methods long-text-image retrieval tasks, even though
Conclusion
In this work, we empirically confirm that key issue arises because training images are typically paired with short captions, which can cause certain tokens to overshadowed by more salient ones. To address this issue, our initial strategy involved relabeling the data with long captions. However, directly learning from these long captions might lead to degraded performance in tasks requiring an understanding of short text, such as image classification. Subsequently, by incorporating corner tokens to aggregate diverse textual information, we are able to help the model regain its original proficiency in understanding short texts while significantly enhancing its capability comprehend long texts. We also investigated whether the model could continue to benefit from longer captions and observed clear trade-off between performance and efficiency. Finally, validated the effectiveness of our approach using self-constructed large-scale dataset, which consists of 100 million long- caption-oriented text-image pairs. In the task of long-text-image retrieval, our method outperforms the second-best competitor, Long-CLIPP with an improvement of 4.2%
7 Acknowledgments
This work was supported by National Natural Science Foundation of China (NSFC) under Grants 62225207, and Zhejiang Provincial Natural Science Foundation of China under Grants L.D24F02001..

---

References
[1] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, Ouyang, J. Zhuang, Lee. Y. Guo, et al. Improving image generation with better captions. Computer Science, http::/cch..peenaiicco/ppuussuud 2023.
2:8.
[2] M. Byeon, B. Park, H. Kim, Lee, W. Back, and S. Kim. Coyo-700m: Image-text pair dataset. https: figtthub. www./ooo///// /ww.abaaiaaaaaaaaaaaaaaaaaaaaaaa coyo-oateo dataset, 2022.
[3] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12M: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In IEEE Conf. Comput. Vis. Pattern Recog,, 2021.
[4] J. Chen, J. Yu, C. Ge, L. Yao. E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al. Pixart- alpha: Fast training diffusion transformer for photorealistic text-to-image synthesis, arXiv preprint arXiv:2310.04420 2023.
[5] Chen, Y. Wu, Luo, E. Xic, Paul, P. Luo, H. Zhao, and Li. Pixart- Pixart-((delta): Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.0222 05522,,, 2024.
[6] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. ShareGPT4V: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279., 2023.
[7] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. InstructBLIP. Towards purpose vision-language models with instruction tuning. Adv. Neural Inform. Process. Syst. 36, 2023.
[8] T. Darcet, M. Oquab, Mairal, and P. Bojanowski. Vision transformers need registers. In Inf. Conf. Learn. Represent., 2024.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
[10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
[11] L. Fan, D. Krishnan, P. Isola, D. Katahi, and Y. Tian. Improving CL.IP training with language rewrites. Adv. Neural Inform. Process. Syst., 36:35544-3553 35575, 2023.
[12] R. Garg, A. Burns, K. Ayan, Y. Bitton, C. Montgomery, Y. Onoe, A. Bunner, R. Krishna, J. Baldridge, and
R. Soricut. Imageinwords: Unlocking hyper-detailed image descriptions, arXiv preprint 240502240,.02793, 2024.
[13] H. A. K. Hammoud, H. Itani, F. Pizzati, P. Torr. A. Bibi, and B. Ghanem. SynthCLIP: Are we ready for fully synthetic clip training? arXiv preprint arXiv:2402.0m 01832, 2024.
[14] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.H. Sung. Z. Li, and T. Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In Int. Conf. Mach. Learn., pages 4904 4916, 2021.
[15] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg. W.-Y. Lo. et al. Segment anything. In Int. Conf. Comput. Vis., pages 4015-40260 2023.
[16] J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. UniCLIP: Unified framework for contrastive language- image pre-training. Adv. Neural Inform. Process. Syst., 35:1008 1019, 2022.
[17] Y. Li. H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language- pre-training via masking. In IEEE Conf. Comput. Vis. Pattern Recog., pages 23390-23442 -3400, 2023.
[18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, Ramanan, P. Dollár, and L. Zitmick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., pages 740 750-755. Springer, 2014.
[19] D. Liu, H. Zhang, F. Wu, and Z.-J. Zha. Learning to assemble neural module tree networks for visual grounding. In Int. Conf. Comput. Vis., pages 4673-4682, 2019.
[20] H. Liu, C. Li, Y. Li, and Y. J. Lec. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023,
[21] H. Liu, C. Li. Q Wu, and Y. J. Lee. Visual instruction tuning. Adv. Neural Inform. Process. Syst. 36, 2023.
10

---

[22] Y. Liu, K. Wang, W. Shao, P. Luo. Y. Qiao, M. Z. Shou, K. Zhang. and Y. You. MLLMs--nggmettdd visual-language representation learning. arXiv preprini arXiv:231118765, 2023.
[23] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Adv. Neural Inform. Process. Syst. 25:27730-27744. 2022.
[24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021.
[25] C. Raffel, N. Shazeer, A. Roberts. K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):166, 2020.
[26] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarezyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2 14, 2021.
[27] P. Sharma, N. Ding. S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Association for Computational Linguistics, 2018.
[28] G. Tan, D. Liu. M. Wang. and Z.-J. Zha. Learning to discretely compose reasoning module networks for vidco captioning. arXiv preprint arXiv: 2007.09049 2020.
[29] B. Thomee, D. A. Shamma, G. Friedland. B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59(2);64 73, 2016.
[30] Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. StableRep: Synthetic images from lexl-to-image models make strong visual representation learners. Ady. Neural Inform. Process. Syst. 36:48322-48002, 2023,
[31] J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V. Sharma, and A. Comero-Soriang picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. arXiv preprint arXiv:2312.08575 2023.
[32] H. Wang, Z.-. Zha, L. Li, Liu, and Luo. Structured multi-level interaction network for video moment localization via language query. In IEEE Conf. Comput. Vis. Pattern Recog., pages 7026-7035 2021.
[33] Xiong, J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungla, K. A. Sankararaman, B. Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309,1603 2023.
[34] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu. X. Liang. Z. Li. Jiang, and C. Xu. FILIP: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.077888 2021.
[35] P. Young. A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, pages 67-78, 2014.
[36] Z.-.. Zha, D. Liu, H. Zhang, Y. Zhang, and F. Wu. Context-awarc visual policy network for fine-grained image captioning. IEEE Trans. Pattern Anal, Mach. Intell., 44:710-722, 2019.
[37] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers. A. Kolesnikov, and L. Beyer. LIT: Zero-shot transfer with locked-image text tuning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 18123-18133 2022.
[38] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Int. Conf. Comput. Vis., pages 11975- 11986, 2023.
[39] B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403. 15378, 2024.
[40] L. Zhao, K. Zheng, Y. Zheng, D. Zhao, and J. Zhou. RLEG: lision-language representation learning with diffusion-bart based embedding generation. In Int. Conf. Mach. Learn., pages 2023.
[41] K. Zheng, Y. Zhang, W. Wu, F. Lu, S. Ma, X. Jin, W. Chen, and Y. Shen. DreamLIP: Language-image pre-training with long captions. arXiv preprint arXiv: 17007 2024.

---

8 Appendix
8.1 Data Statistics
As shown in Tab. 6, report the data statistics of dataset and other text-image paired datasets. Compared to the other text-image paired datasets, the re-captioned texts of LoTLIP are significantly longer, averaging 136 tokens 18 tokens. To the best of our knowledge, LoTLIP dataset is the largest dataset consisting of long texts for multi-modal learning. We are continuing to expand the size of LoTLIP by integrating additional MLLMs, well gathering more publicly available datasets for long text generation.
Table 6: Data statistic of LoTLIP dataset and other text-image paired dataset. Our dataset is the largest dataset consisting of long texts for multi-modal learning.
Dataset
#lmages
#Texts
=Sub-captions per Text
#Tokens per Text
Short-text-image Pairs Dataset
CC3M [27]
3,018,175
3,018,175
1.01
11.29
CC12M [27]
10,445,969
10,445,969
1.00
17.48
YFCCISM [29]
14,772.456
14,772,456
1.23
13.61
LAION47M [26]
49,677,119
49,677,119
1.28
18.99
COY024M [2]
24,658,004
24,658,004
1.21
17.07
Long-text-image Pairs Dataset
LoTLIP
102,571,723
307,715,169
6.16
136.14
8.2
More Experimental Analysis
8.2.1 Influence of Sub-caption Number LoTLIP and LiT
During the pre-training stage, we randomly select multiple consecutive sub-captions from long texts in LoTLIP as long text inputs. In Fig. 5, we show the influence the number of sub-captions when the length of text input is limited to 128. With the number of sub-captions increasing. LiT and LoTLIP reaches better performance in both text retrieval and image retrieval on the ShareGPT4v-100 and DCI datasets. There are small improvements when the number of sub-captions gets larger than 3. Moreover, both methods exhibit performance degradation on text-image retrieval and image classification tasks as the number sub-captions increases. But the performance drop of LoTLIP is even slower. It further indicates that LoTLIP can enhance the understanding of long texts while retaining the short-text understanding ability.
8.2.2 Compare Corner Tokens with Register Tokens
In LoTLIP, employ learnable tokens, namely corner tokens, help the model regain its original performance in understanding short texts, while significantly enhancing its capability to comprehend long texts. It mentioned that registered token [8] also adds learnable tokens in the encoder, which are used to get rid of artifacts in image feature maps. In order to fairly compare corner tokens with register tokens, we implement the same number of register tokens on the text encoder and train the model with the same scale of dataset as LoTLIP. As shown in Tab. 7, the results demonstrate that corner tokens improve the performance the model for long-text and short-text understanding tasks, which is better than register tokens.
8.2.3 Contrastive Learning without Pre-trained Weights
In Tab. 8, we present the performance of LoTLIP without loading pre-trained weights for text and image encoder on long and short-text related tasks. The results show that directly involving long texts in the pre-training stage can also enhance the performance of the language-image model with randomly initialized weights over tasks. Besides, LoTLIP further improves CL.IP on long and short text-related tasks, when they are both pre-trained with long texts. In our methodology, opt
12

---

Loni 201 inoge
022101 01 1I
Lamros SoTemeee SanTer T2 RIB--A
DOLLE
15
SubCaption Num
SubCaption Num
(a)
(h)
150000 22 100011
770 R80 11941
20302N0 Acctiii
SubCaption Num
SubCaption
(c)
(d)
(c)
Figure 5: Influence of the number of sub-captions used in the pre-training stages. Both LiT and LoTLIP are trained with long texts. The performance on ShareGPT4V and DCI retrieval are shown in (a)(b). (c)(d) represent the performance on MSCOCO retrieval. (e) shows the performance image classification on ImageNet.
Table Compare corner tokens with register tokens.
The models are trained with 3M scale dataset.
Long-text-image Retrieval
Short-text-image Retrieval
Classification
Learnable Token Type
DCI
IIW
ShareGPT4V-I0k
MSCOCO
ImageNet
12T
12T
12T
T2I
47.96
44.92
84.97
81.70
73.66
66.73
43.52
30.06
48.87
Register
45.62
44.65
83.99
81.37
74.42
68.78
44.18
30.34
48.78
Corner
49.46
47.82
84.97
83.33
76.49
69.72
46.56
31.59
50.34
to load pre-trained weights and fix the image encoder during training, following LiT. This because LiT's is strong baseline and such training setting significantly conserves computational resources.
Table Analyze the effectiveness of LoTLIP without loading pre-trained weights for image and text encoder. The architecture of image encoder is ViT-B/I6 We use 3M scale dataset for pre-training. "/" indicates use long texts in the training stage.
Long Long-text-inge text-image Retrieval
Shoot-text-image Retrieval
Classification
Method
Long Texts
DCI
IIW
ShareGPTaV-
MSCOCO
ImageNet
12T
T2I
12T
T2I
12T
T2I
12T
T2I
Acc.
CLIIP
11.67
11.01
33.17
31.37
10.69
8.77
14.74
10.93
16.54
CLIP
42.92
42.23
77.29
75.98
66.78
64.70
32.14
22.33
23,28
LeTLIP
43,91
43.83
78.76
70.20
34.40
24.11
24.65
8.2.4
Utilizing Long Captions from Different MLLMS
We use three MILMs, i.e., InstructBLIP [7], LLaVA [20], and ShareGPT4V [6], to generate diverse long captions for 100M images. In this part, we conduct experiments to analyze the influence of using long texts generated by different MILMMs in pre-training. shown in Tab. 9, LoTLIP trained with all generated long texts reach the best performance average all tasks. It means that diverse
13

---

long texts help Sanguage-image pre pre-trained models to better comprehend texts to align with images. Moreover, LoTLIP trained with long captions generated by ShareGPT4V reaches higher scores on long-text-image retrieval tasks than other MI.LMs. This indicates that the long captions generated by ShareGPT4V are of higher quality, which can enhance the long-text comprehension ability of LoTLIP.
Table 9: Utilizing long captions generated by different MILMs in the training statge.
Long-text-ingg temt-ggg-- Retrieval
Short-text-imug Retrieval
Classification
Method
MILM
DC1
IIW
ShareGPT4Y. 10k
MSCOCO
ImageNet
12T
12T
T21
12T
T21
InstructBLIP [7]
39.94
38.54
78.43
75.33
57.90
55.17
44.60
30.94
50.32
LLaVA [20]
78.26
54.50
LoTLIP
ShareGPT4V [6]
47.21
46.71
85.78
83.17
77.93
72.30
44.98
30.96
49.34
84.97
83.33
76.49
69.72
46.56
31.59
50.34
8.2.5 Pre-training with different scale of dataset
We compare LoTLIP and LiT on different scales of the pre-training dataset, i.e., 3M, 12M, 30M, and 100M on long-textimmaee retrieval, text-image retrieval, and image classification. The results on long-text text and short-text centered tasks are shown in Tab. 10 and Tab. 11, respectively. LoTLIP significantly improves LiT all tasks both ViT-B/32 and ViT-B/16 under different pre-training settings. It proves that involving long captions in the pre-training stage can help the language-image model better deal with both long and short-comprehension tasks.
8.3 Visualization
We visualize the attention map of LiT, LiT trained with long texts, Long- CLIP (our implementation), and LoTLIP in Fig. 6. Given the long caption in the first column, the attention visualization map of LiT can only activate regions corresponding to partial objects mentioned in the long caption, egg. flat rock. This situation is alleviated after incorporating long captions in LiT pre-training Benefiting from corner tokens, the highlighted image regions of LoTLIP are well aligned with the given long caption.
LiT
LiT+Long Texts
Long-CLIP
LOTLIP
An person washing dothes by scrubbing both hands.
flat rock is long enough for both of the chilly feet side by side. must rising off
and the person wearing long sleeves. the person probably bekings to group that believes the head should awered.
There is reg and gray bird in trv.. perched branch and looking its lell green they mostly There
branches have Around branches vary sky background. is sunny and dappled sunlight is on the plants and the bird.
Figure 6: Visualize the attention map of LiT, LiT trained with long texts (LiT+Long Texts), Long- CLIP, and LoTLIP. Here, both Long-CLIPP (our implementation) and LoTLIP trained with long texts. Benefiting from long texts and corner tokens, the highlighted image regions of LoTLIP are better aligned with the given long caption compared other methods.
14

---

8.4 Limitation
We re-wright the captions of 100M images using three popular open-sourced multi-modal large language models (i.c.nntrrctBLIPP [7], LLaVA [20] and ShareGPT4V [6]), but we observed hallucination elements in the synthesized long captions. The hallucinations in captions, which do not correspond with the image information, may restrict the full potential of long captions in enhancing the understanding of lengthy texts by language-image pre-trained models.
8.5 Experiment Hardware
To obtain our LoTLIP trained with 100M scale dataset, we apply A100 GPU with 80G memory for training, which costs about 133 GPU days. For models trained on datasets of other scales (i.e.M,, 12M, 30M), the training duration decreases linearly with the amount of training data.
8.6 Social Impact
The 100M images used for pre-training publicly accessible, and the re-annotated long captions are synthesized using public multi-modal large language models on these public datasets, posing no ethical risk in the data source. Our LoTLIP is incapable of generating images and text, thus there is no need for concern regarding the negative social impact resulting from fake, violent, pornographic, or discriminatory content. Moreover, LoTLIP has the potential for positive social impact, considering its excellent image-text retrieval performance, it may serve as a valuable asset in image retrieval libraries in the future.
8.7 Ethic Statement
We have conducted a thorough review to ensure that there has been no violation of the NeurIPS Code of Ethics in this paper.
15

---

Table 10: Zero-shot evaluation of different models on long-text-image retrieval.
DCI
IIW
ShareGPT4V-1k
ShareGPT4V-10k
Data Scale
Model
Avg.
12T
T21
12T
T2I
12T
T21
12T
T2I
Model Architecture:
ViT-B/32
LiT
24.11
20.56
59.48
53.92
57.80
51.00
28.19
22.73
39.72
3M
LoTLIP
44.46
41.78
83.99
79.90
91.40
87.10
71.17
63.85
70.45
LiT
38.73
33.91
78.59
77.12
73.00
69.20
43.69
38.27
56.56
12M
LoTLIP
47.35
45.86
89.87
87.58
93.80
91.90
79.01
73.00
76.04
LiT
39.41
33.71
85.13
76.63
79.40
67.70
52.09
36.47
58.82
30M
LoTLIP
51.12
49.25
89.87
89.05
94.40
92.80
82.66
76.93
78.26
LiT
42.54
39.12
86.27
83.17
82.10
76.90
$7.69
47.99
64.47
100M
LoTLIP
58.64
55.20
92.81
91.67
95.10
93.10
82.81
77.46
80.84
Model Architecture:
ViT-B/16
400M+1M
Long-CLIP
51.68
57.28
89.61
93.20
94.70
93.40
79.24
77.06
79.52
LiT
27.14
24.13
65.20
58,50
63.60
56,80
32.73
27.01
44.38
3M
LoTLIP
49.46
47.82
84.97
83.33
93.20
90.00
76.49
69.72
74.37
LiT
44.04
39,86
83.01
80.88
79.60
74.80
52.93
45.79
62.61
12M
LoTLIP
52.24
50.72
92.16
89.87
95.80
93.80
83.78
77.51
79.48
LiT
38.68
34.81
85,29
79.41
81.40
68.50
54.63
39.27
60.24
30M
LoTLIP
56.04
55,90
93.79
91.50
96.90
94.10
86.72
81.57
82.06
LIT
41.78
40.90
88.07
82.68
86.00
80.00
61.41
50.69
66.44
100M
LoTLIP
62.10
61.06
93.95
92.48
96.50
95.50
86.84
81.40
83.72
16

---

Table 11: Zero-shot evaluation of different models on image-text retrieval and classification tasks.
IN
MSCOCO 12T
MSCOCO T21
Flickr30k
12T
Flickr30k T21
Data Scale
Model
Acc.
R@1
R@5
R@1
R@5
R@1
R@5
R @
R@5
Model Architecture:
ViT-B/32
LiT
38.83
30.94
57.02
21.25
44.60
44.60
83.80
41.66
71.70
3M
LoTLIP
45,50
41.98
68.44
28.00
53.36
72.00
91.10
52.76
79.16
LiT
59.84
40.70
66.52
26.48
51,47
66.90
89.00
49.52
76.58
12M
LoTLIP
61.07
50.84
76.20
33.19
58.89
77.30
94.60
57.46
83.08
LiT
64.95
42,48
68.36
25.70
50.50
67.80
89.80
47.64
76.14
30M
LoTLIP
65.41
52.52
76.72
32.51
57.90
78.40
95.00
56.16
81.14
LiT
68.26
47.98
72.80
29.69
55.24
75.60
93.90
54.52
81.34
100M
LoTLIP
67.20
55.62
79.30
34.98
60.80
82.60
95.30
59.58
84.56
Model Architecture:
ViT-B/16
400M+1M Long-CLIP
67.10
57.28
80.78
40.34
65.92
85.90
98.50
70.66
90.60
LiT
43.76
34.20
61.52
24.07
48.37
61.30
89.50
48.06
75.36
3M
LOTLIP
50.34
46.56
72.02
31.59
57.65
75.20
94.20
58.20
83.58
LiT
65.97
44.78
70.46
29.57
54.58
72.90
92.90
54.18
81.18
12M
LoTLIP
66.70
55.18
78.54
36.22
62.05
82.60
96.30
62.74
86.78
LiT
69.91
45,46
70.48
27.66
51.96
73.40
93.10
53.16
78.94
30M
LoTLIP
70.60
55.58
79.68
34.34
59.85
81.80
95.70
60.52
84.22
LiT
73.70
51.92
75.72
32.74
57.84
80.00
95.90
60.86
84.62
100M
LoTLIP
72.16
59.66
81.50
38.06
63.81
86.90
97.80
65.22
87.98
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we explicitly state that our initial attempt is to relabel the data with long captions to facilitate the understanding of long text by the language-image pre-training (LIP) model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification: In the appendix, we discuss the limitations regarding the the potential harmfulness of the hallucination elements in long captions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification:
We primarily demonstrate through experimentation that short captions fail to unlock the potential of LIP understanding long texts, and that our approach effectively models long captions in way that aligns with image representation, as described in Sec. and Sec. 5.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
19

---

Justification: In both the description of our method in Sec. 4 and the details of experimental implementation in Sec. 5, we disclose the information needed reproduce the main experimental results. We will also release our code, model, and the new dataset to facilitate the reprodueibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code for this paper requires approval before it can be made open source, hence it is not provided in this submission. However, the code, models, and datasets of this paper will be made publicly accessible after this submission ensure the reproducibility of the experiments and to foster research progress within the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/gue ggwidss/Coeeeuuteeee
aassonooiicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / httpo/////giiiee//oeeeiiiiiiiiia
for more details.
20

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Sec. 5, we explicitly present the implementation details, including the training data, pretraining hyperparameters, the settings of comparison methods, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper proposes foundation model that incurs significant training expenses, so we fixed the random seed and trained it only once. During the evaluation, we also fixed the random seed to ensure the constant results. Consequently, the fluctuation and етror in our experimental results are slight.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
21

---

For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the appendix, have specified the GPU type, the memory, the numbers of GPUs required for one training session, and the training duration for our model.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : htt///nerripppp hwtppu/c//ttiscciiiieeeeeeeeeeee
Answer: [Yes]
Justification: We have conducted a thorough review to ensure that there has been no violation of the NeurIPS Code of Ethics in this paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consideration due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our model has the potential for positive social impact, given its strong image- text retrieval performance, it may serve valuable asset in image retrieval libraries in the future.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
22

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The image data used for training is publicly accessible, and the long captions were synthesized using public models on public datasets, posing no security risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper that produced the code and dataset in the paper. The image data used for training is publicly accessible, and the long captions were synthesized using public models.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We have rewritten captions for 100M images, but the new data requires approval before release. The new data will be made available the public with our code after submission.
23

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "PpqHOI9v74", "reviewer": "Reviewer_f34j", "summary": "The paper describes a framework to adapt language-image pre-training models to longer captions. For that, first a new dataset is created with longer captions and second, training is modified to adapt to longer captions. New corner tokens are introduced that are supposed to capture longer dependencies in the text, and the training loss is modified to take into account both short and long text. Experimental results show that the method performs better than other methods for tasks including both long-text retrieval and short-text retrieval.", "questions": "See above in weaknesses", "limitations": "There is no specific discussion on the limitations of the method", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A new dataset is created that can be used in the future for improving the ability of the models for long-text pairing with images\n- Several modifications are included in the training procedure that allow to take into account long text while preserving the performance with short text. These modification include modifying the text representation with the corner tokens, introducing a specific attention mask and defining a new loss balancing loss for long and short text.\n- Experiments show that the proposed method performs better than other existing approaches both in long text and short text retrieval. Experiments includes a detailed ablation study analyzing the impact of the different components of the model.", "weaknesses": "It is not clear the difference of the new Dora dataset with the dataset proposed in DreamLIP (reference [34]) where 30M images are also re-captioned with MLLMs. More details on how the Dora dataset has been built will be useful and also some statistics with respect of the length of the captions in the dataset, comparing with the other datasets analyzed in table 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Bellow we address your concern separately.\n\n``Q1: Difference between Dora dataset and DreamLIP.``\n\n\nThe main difference between Dora dataset and the dataset proposed in DreamLIP lies on two aspects:\n- **Larger Dataset Scale**: Our dataset has 8x image volume and 5x text volume of DreamLIP as shown in following table.  \n- **Longer Token Length**: Our dataset has average of 136 tokens per text, which is twice as many as DreamLIP (136 *v.s.* 75). \n\nIt is mentioned that the long texts of DreamLIP are splitted into short texts and sperately utilized. In this way, the usage text of DreamLIP are tokenized to around **21 tokens** on average, which not exploring how to make text encoders better understand long texts.\n\n\n| Dataset   | Num. of Image  |  Num. of Long Caption  |  Tokens per Long Text  |\n| :----: | :----: | :----: | :----: | \n| DreamLIP-15M [1] |  13,464,144| 80,784,864 |  74.86 |\n| **Dora**  | **102,571,723** | **307,715,169** | **136.14** |\n\n[1] We report the statistical results derived from 15 million data released by DreamLIP.\n\n`` Q2: More details on how the Dora dataset has been built will be useful.``\n\nThanks for your valuable suggestion. We use multi-modal large language models (MLLMs) to generate long texts by prompting 'Please describe the image in details'. To prevent the bias introduced by MLLMs' hallucinations, three kinds of MLLMs are introduced to generate the diverse long texts from different model knowledges. We also provide the hyper-parameter settings of the used MLLMs:\n|Hyper-parameters|ShareGPT4V-13B | LLaVA-v1.5-13B |InstructBLIP-Vicuna7b|\n|  :----: | :----: | :----: | :----: |\n|max_new_tokens | 1024 | 512 |  256 |\n|num_beams | 5 |1|5|\n|do_sample| True |True|False|\n|top_p | None|None|0.9|\n|temperature|0.2|0.2|1|\n\n\n`` Q3: Statistics of Dora dataset, comparing with the other datasets analyzed in table 1 of the main manuscript.``\n\nAs shown in the table below, we report some statistics of our dataset. To the best of our knowledge, **Dora is the largest dataset consisting long texts for multi-modal learning**. And we are continuing expanding the size of Dora by integrating additional MLLMs for long text generation, as well as gathering more publicly available datasets.\n\n\n| Dataset   | # Num. of Image  |  # Num. of Long Caption  |  # Tokens per Long Text  |\n| :----: | :----: | :----: | :----: |\n| MSCOCO  | 5,000|25,000|11.77|\n| Flickr30k  | 1,000 | 5,000|14.03|\n| DCI  | 7,805 | 7,805 | 172.73 |  172.73 |\n| IIW  | 612| 612 |239.73 | \n| ShareGPT4v-10k  |  10,000|10,000|  173.66 |\n| DreamLIP-15M  |  13,464,144| 80,784,864 | 74.86 |\n| **Dora**  | **102,571,723** | **307,715,169** | **136.14** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiTESFpK3U", "reviewer": "Reviewer_Zcuk", "summary": "The paper addresses a significant gap in current language-image pre-training models, which are typically trained on datasets with short captions. This limitation hinders the models' ability to effectively understand and process long texts. The proposed solution, LotCLIP, introduces methods to enhance long-text understanding without compromising the performance on short-text tasks.", "questions": "Could you please provide a detail diagram on training process encompassing the \"corner tokens ?", "limitations": "No limitations were mentioned by the authors.\n\n***** Raising the final score to 5  from 4********************", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(a) The authors re-captioned 100 million images with long texts using multi-modality large language models.\n\n(b) Introducing the concept of  corner token", "weaknesses": "(a) Not very clear why the proposed method is based on CLIP architecture only- could it have been built upon other similar algorithm as well, for example – ALIGN ?\n\n(b) Need to illustrate in detail how this “corner token” learning is actually taking place. \n\n(c) Should have also reported results on  ALIGN \n\n\n(d) The references are not correct/incomplete in some occasions -please cross check all.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Bellow we address your concern separately.\n\n``Q1: LotCLIP on other similar algorithms, *e.g.* ALIGN, CoCa.``\n\n**Our LotCLIP can be applied on other similar algorithms as well, *e.g.* ALIGN, CoCa.** The results on CC3M are presented in the table below, which demonstrate that LotCLIP also enhances the long text comprhension ability of other similar algorithm besides CLIP.\n\n[1] CoCa: Contrastive Captioners are Image-Text Foundation Models\n\n|  Algorithm   | DCI  T2I | DCI I2T| IIW T2I |IIW I2T| SV-10k T2I | SV-10k I2T | Avg\n|  :----: |  :----: | :----: | :----: | :----: | :----: | :----: |:----: |\n| ALIGN | 19.67 | 18.01 | 52.45 | 50.16 | 24.21 | 18.33 | 30.47|\n| +Long Caption | 39.68 | 38.48 | 79.58 | 76.31| 63.53 | 39.68| 56.21 **(+25.74)** |\n| +Coner Token |  40.49 | 40.87 | 81.70 | 78.76 | 69.94 | 67.50 | 63.21 **(+32.74)**|\n| CoCa | 9.04 | 8.66 | 27.45 | 27.94 | 9.46 | 9.12 | 15.28|\n| +Long Caption |32.47 | 31.21 |66.17|65.68|54.10|51.98|50.27 **(+34.99)**|\n| +Coner Token |  34.49 | 33.14 | 68.95 | 67.32 | 58.64 | 56.01 | 53.09 **(+37.81)**|\n\n``Q2: Illustrate learning process of corner tokens``\n\nThanks for the valuable suggestion. The corner tokens are learnable tokens, which are placed before text tokens and after [CLS] token. We demonstrate the training process of corner tokens in below.\n\n**Inputs**: \n- image\\_encoder\n- text\\_encoder\n- tokenizer\n- image\\_processor\n- learnable corner tokens: $[\\texttt{Cor 1}], [\\texttt{Cor 2}],\\cdots, [\\texttt{Cor m}]$\n- minibatch of long texts: $T_{\\text{long}}$ [$n,l$]\n- minibatch of short texts: $T_{\\text{short}}$ [$n,l$]\n- minibatch of images: $I$ [$n,h,w,c$]\n- learned temperature parameter: $t$\n\n**Training Process**:\n\n*\\#* The corner tokens are placed in front of text before the tokenization process.\n\n*\\#* After tokenization, the first token of text input is [CLS] token, \n\n*\\#* while the i-th token (1<i<=$m$+1) is $[\\texttt{Cor i-1}]$ token.\n\n$T_{\\text{long}}$ $=$ tokenizer($[\\texttt{Cor 1}], [\\texttt{Cor 2}],\\cdots, [\\texttt{Cor m}]$+ $T_{\\text{long}}$)\n\n$T_{\\text{short}}$ $=$ tokenizer($[\\texttt{Cor 1}], [\\texttt{Cor 2}],\\cdots, [\\texttt{Cor m}]$ + $T_{\\text{short}}$)\n\n*\\#* Image pre-processing\n$I$ $=$ image\\_processor($I$)\n\n*\\#* Build attention mask mechanism\n$\\mathcal{A}$ $=$ np.ones([$l$,$l$])\n\n*\\#* Text tokens do not attend to corner tokens\n$\\mathcal{A}[m+1:, 1:m+1]$ $*=$ 0\n\n*\\#* [CLS] token and corner tokens do not interact with each other\n$\\mathcal{A}[:m+1, :m+1]$ $=$ np.eye(m+1)\n\n*\\#* The attention mask $\\mathcal{A}$ is multiplied by original attention mask used in the text encoder \n\n*\\#* to eliminate the influence from [PAD] tokens to [CLS] and text tokens.\n\n$\\mathcal{A_{\\text{long}}}$ $=$ $\\mathcal{A}$ $\\cdot$ text\\_encoder.build_attn_mask($T_{\\text{long}}$)\n\n$\\mathcal{A_{\\text{short}}}$ $=$ $\\mathcal{A}$ $\\cdot$ text\\_encoder.build_attn_mask($T_{\\text{short}}$)\n\n*\\#* Extract text features. \n\n*\\#* The attention mask controls the interaction among tokens within in each attention block.\n\n*\\#* The text\\_encoder outputs the features of the first $m+1$ tokens, where the first \n\n*\\#* one is [CLS] token and the remains are corner tokens.\n\n$f_{lt}$ $=$ text\\_encoder($T_{\\text{long}}$, attention\\_mask $=$  $\\mathcal{A_{\\text{long}}}$) *\\#* [$n,m+1,d$]\n\n$f_{st}$ $=$ text\\_encoder($T_{\\text{short}}$, attention\\_mask $=\\mathcal{A_{\\text{short}}}$) *\\#* [$n,m+1,d$]\n\n*\\#* Extract image features.\n\n$f_\\text{i}$ $=$ image\\_encoder($I$) *\\#* [$n,d$]\n\n*\\#* Normalization\n\n$f_{lt}$ $=$ l2\\_normalize($f_{lt}$, axis=-1)\n\n$f_{st}$ $=$ l2\\_normalize($f_{st}$, axis=-1)\n\n$f_{i}$ $=$ l2\\_normalize($f_{i}$, axis=-1)\n\nlabels $=$ np.arange(n)\n\n*\\#* Loss computation\n\n$logits_{\\text{short}}$ $=$np.dot($f_{i}$, $f_{st}$[:,0,:]) $\\cdot$ np.exp($t$)\n\n$loss_{\\text{short}}^{i2t}$ $=$ cross\\_entropy\\_loss($logits_{\\text{short}}$, labels, axis=0)\n\n$loss_{\\text{short}}^{t2i}$  $=$ cross\\_entropy\\_loss($logits_{\\text{short}}$, labels, axis=1)\n\n$loss_{\\text{short}}$ $=$ ($loss_{\\text{short}}^{i2t}$+$loss_{\\text{short}}^{t2i}$)/2\n\n$loss_{\\text{long}}$ $=$ 0\n\nFor $k \\in [0, m]$:\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $logits_{\\text{long}\\_k}$ $=$ np.dot($f_\\text{image}$, $f_\\text{long}$[:,$k$,:]) * np.exp($t$)\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $loss_{\\text{long}\\_k}^{i2t}$ $=$ cross\\_entropy\\_loss($logits_{\\text{long}\\_k}$, labels, axis=0)\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $loss_{\\text{long}\\_k}^{t2i}$ $=$ cross\\_entropy\\_loss($logits_{\\text{long}\\_k}$, labels, axis=1)\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $loss_{\\text{long}}$ $+=$ ($loss_{\\text{long}\\_k}^{i2t}$+$loss_{\\text{long}\\_k}^{t2i}$)/2\n\nloss $=$ $loss_\\text{short}$ + $loss_{\\text{long}}$\n\n``Q3: Should have also reported results on ALIGN``\n\nThanks for your valuable suggestion. **We report the performance of ALIGN** pre-trained on COYO-700M in the table below. Even trained with smaller scale dataset, LotCLIP outperforms ALIGN on long-text-image retrieval task. We will include the results of ALIGN for comparision in the revised version of our paper.\n\n|  Method   | Data Scale   | DCI  T2I | DCI I2T| IIW T2I |IIW I2T| SV-10k T2I | SV-10k I2T | Avg. |\n|  :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | \n|  ALIGN | COYO-700M |56.54 | 57.41 | 92.65 | 90.68 | 65.13 | 62.73| 70.86 | \n|  LotCLIP | Dora-100M | 62.10 | 61.06 | 93.95 | 92.48 | 86.84 | 81.40 | 79.64 **(+8.78)** | \n\n|  Method   | Data Scale   | ImageNet CLS | COCO I2T| COCO T2I | Avg. |\n|  :----: | :----: | :----: | :----: | :----: | :----: | \n|  ALIGN | COYO-700M |65.89 |  60.42 | 42.36 |56.22|\n|  LotCLIP | Dora-100M | 72.16 | 59.66 | 38.06 |56.62 **(+0.40)** |\n\n``Q4: Not correct/incomplete references``\n\nWe apologize for any previous errors in the references, which we will carefully verify and rectify in the revised paper. \n\n``Q5: Diagram on corner token learning process``\n\nPlease refer to our reply to Q2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DtpkkrviLz", "reviewer": "Reviewer_sjvn", "summary": "To improve the ability of vision-language models (VLMs) for long-text understanding, the paper proposes to relabel the data with long captions, however, direct learning may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, corner tokens are introduced to aggregate diverse textual information, enabling the model to catch up to its original level of short-text understanding yet greatly enhance its capability of long-text understanding. Experiments are performed on a large-scale long caption dataset to demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weakness part.", "limitations": "Yes. Both the limitation and potential impact are well described in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The author points out the phenomenon that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens.\n2) To improve the ability of vision-language models (VLMs) for long-text understanding, the paper proposes to relabel the data with long captions; corner tokens are introduced to aggregate diverse textual information, enabling the model to catch up to its original level of short-text understanding yet greatly enhance its capability of long-text understanding.\n3) Experiments are performed on a large-scale long caption dataset to demonstrate the effectiveness of the proposed method.", "weaknesses": "1) Lack of details for addressing the limitation for the token length limitation of the text encoder in Sec. 3.4. \n2) For the attention mask part, if the corner tokens can be seen by other text tokens, how does it perform? In Table 3, how the model will the model perform if the corner taken is removed?\n3) The year of references for NeurIPS papers e.g., [10] and [14], is not correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Bellow we address your concern separately.\n\n``Q1: Lack of details for addressing the limitation for the token length limitation of the text encoder in Sec. 3.4.``\n\nSorry for confusion. We set a token length limitation of 128 for the text encoder based on the following two considerations:\n- **Token length of training text.** For pre-trained models, the token length limitation can be arbitrarily set to any positive integer value. As shown in Figure 4 of Sec. 3.4, the results indicate that a text input limitation of 77 tokens is insufficient for model that requires long text comprehension. However, the training dataset has average of 136 tokens per text, and larger token length limitation may not bring more information.\n- **Balance of training efficiency and performance.**: In Figure 4 of Sec. 3.4, a smaller token number may lead to performance degradation due to insufficient encoding of text information, while a larger token number may increase computational complexity. Thus, to balance the training efficiency and performance, we generally choose 128 as the maximum token length. \n\nWe will add more details in updated version.\n\n``Q2: For the attention mask part, if the corner tokens can be seen by other text tokens, how does it perform? In Table 3, how the model will the model perform if the corner taken is removed?``\n\n\nThanks for the valuable suggestion. We have added the following experiments:\n\na) **Performance is not improved (68.19 *v.s.* 68.20) when the corner tokens can be seen by other text tokens.** This is because allowing corner tokens to be seen by text tokens can influence the diversity of aggregated text features, leading to worse performance. \n\nb) **Removing the corner token degrades LotCLIP's average performance by 1.98%**, which further demonstrates the effectiveness of corner tokens. \n\nWe will include the results in Table 3 of the revision.\n\n\n|  Corner Token |  Attention Mask Mechanism | DCI  T2I | DCI I2T| IIW T2I |IIW I2T| SV-10k T2I | SV-10k I2T | Avg. |\n|  :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |:----: | \n|-| - | 47.96 | 44.92 | 84.97 | 81.70 | 73.66 | 66.73  | 66.65 \n|✓| - | 48.61 | 47.17 | 86.11 | 81.86 | 76.14 |  69.31|   68.20 **(+1.55%)**\n|✓| text tokens can see corner token | 48.29 | 47.23 | 85.29 | 82.26 | 76.70| 69.38| 68.19 **(+1.54%)**\n|✓| text tokens can't see corner token | 49.46 | 47.82 | 84.97 | 83.33 | 76.49 | 69.72 | 68.63 **(+1.98%)**\n\n\n``Q3: The year of references for NeurIPS papers e.g., [10] and [14], is not correct.``\n\n\nWe apologize for any previous errors in the references, which we will carefully verified and rectified in the revised paper.\n\n**Incorrect Reference in paper**:\n\n[10] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024\n\n[14] J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. Uniclip: Unified framework for contrastive language-image pre-training. Advances in Neural Information Processing Systems, 35:1008–1019, 2022\n\n[25] Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.\n\n**Rectification**:\n\n[10] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving clip training with language rewrites. Adv. Neural Inform. Process. Syst., **36:35544–35575, 2023**\n\n[14] J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. **UniCLIP**: Unified framework for contrastive language-image pre-training. Adv. Neural Inform. Process. Syst., 35:1008–1019, 2022\n\n[25] Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. StableRep: Synthetic images from text-to-image models make strong visual representation learners. Adv. Neural Inform. Process. Syst., **36:48382–48402, 2023**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7oBLorcMd9", "reviewer": "Reviewer_NJUx", "summary": "This paper introduces LotCLIP, which enhances CLIP’s capability to understand long texts. It highlights that merely increasing the length of texts (i.e., context length) is not beneficial as it adversely impacts the understanding of short texts (i.e., image classification). To mitigate this trade-off, additional learnable corner tokens are integrated into the text encoder transformer. Moreover, the attention mask is modified so that interactions between corner tokens are restricted, ensuring diversity in the output feature embeddings. The image encoder and text encoder are initialized from pre-trained models and further trained in a LiT manner. LotCLIP is trained on a self-constructed image-text dataset of 100M scale long texts using MLLMs. LotCLIP is evaluated on both long-text retrieval tasks and short-text benchmarks such as image-text retrieval and classification.", "questions": "- The reasoning behind the naming of the constructed data and model is unclear. Why are they named Dora and LotCLIP?\n\n- In the introduction section, the concept of the corner token first appears, but without supporting explanations, which creates confusion.\n\n- No training details for the baseline methods are provided.", "limitations": "- Some limitations on the long-caption data (e.g., hallucinations) are mentioned.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- This paper addresses the underexplored yet important problem of enhancing CLIP's ability to understand long texts. It introduces a simple modification to the CLIP training framework.\n\n- The 100M scale long caption data will be valuable for training VLMs with an understanding of long contexts.\n\n- Strong empirical results compared to other baselines such as LiT , CLIP, and SigLIP.", "weaknesses": "- [W1] Contributions are unclear. From an architectural perspective, there seems to be no major difference from [1, 2], which introduce additional learnable tokens in the encoders. Is there any special reason or evidence that such a technique is especially beneficial for training with long captions? It is likely to be a general technique that is also helpful for training with short captions. Such learnable tokens, named corner tokens in the paper, are only prepended to the text tokens after the [CLS] token. Corner tokens in different positions, such as after the text tokens, can be further ablated. Would corner tokens also be helpful in vision transformers? More comprehensive analysis around the corner tokens are necessary. \n\n- [W2] Connected to the [W1], the contributions of the data are unevaluable. It only mentions that some image-text pairs from various sources (e.g., CC3M/12M, YFCC15M, LAION, and COYO) are re-captioned using MLLMs such as InstructBLIP, LLaMA, and ShareGPT-4V. No other clear details are provided. Any statistics about the training data are missing. Any details on the MLLMs such as specific architecture and instruction information are completely omitted. For the LAION and COYO datasets, how is a subset selected from the entire scale? Most importantly, no verification step for the extracted long captions is provided. Due to the typical hallucinations of MLLMs, it is unclear how well the obtained long captions align with the original images or original short captions, further complicating the training of VLMs. \n\n- [W3] It is unclear whether the comparison is fair. With the LiT training mechanism, LotCLIP benefits from an ImageNet-pretrained ViT backbone, which shows strong evaluation results on ImageNet compared to other pretrained backbones. It is suggested that other visual backbones pretrained via CLIP or through unsupervised methods such as DINO be tested with LiT training. \n\n- [W4] Training and some evaluation benchmarks seem to overlap, leading to high performance results. For example, ShareGPT has LAION and Conceptual Captions images, which shares the training data. This creates a significant gap in evaluation results compared to DCI, and even between models from the default CLIP and the proposed LotCLIP in ShartGPT evaluation.\n\n- [W5] It is not directly comparable to LongCLIP. Starting from the same pretrained CLIP model, how does LotCLIP perform when fine-tuned on a 1M-scale dataset similar to that used for LongCLIP?\n\n\nIn summary, in the current version, there is no clear evidence of technical contributions and the experimental settings are unclear. \n\n\n---\n\nReferences\n\n[1] Darcet et al., Vision Transformers Need Registers, in ICLR 2024.\n\n[2] Lavoie et al., Modeling Caption Diversity in Contrastive Vision-Language Pretraining, in arXiv preprint 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "``Q1: Contributions are unclear.``\n\nSorry for confusion. We reaffirm our core contribution:\n- **We are the first to explore how to improve understanding long texts in contrastive language-image pre-training, and also firstly design learnable text [CLS] tokens (corner tokens) for this purpose.**\n\nMeanwhile, we heartily believe that it is necessary to study how to design a text tencoder that is visually aligned and has the ability to understand long texts.\n\n``Q2: From an architectural perspective, there seems to be no major difference from [1, 2].``\n\nWe are beg to differ with you on this matter. The core difference is not learnable tokens, but **text [CLS] tokens and our designed attention mask mechanism.** [1] and [2] both design learnable tokens in image encoder, which give few help to long-text understanding (refer to Reviewer NJUx, Q4). Moreover, directly **introducing register tokens [1] to text encoder** also does not improve the performance **(-0.02 performance degradation)**, because register tokens is used to get rid of artifacts in images (This problem not exists in text). Our coner token can enhance its capability of long text understanding **(+2.05 performance improvment)**\n\nAs a concurrent work with our LotCLIP, we look forward to whether using [2] into text encoder can enhance CLIP's ability to understand long texts, but unfortunately its official code is currently inaccessible.\n\n``Q3: Is there any special reason or evidence that such a technique is especially beneficial for training with long captions?``\n\nThis is really an interesting aspect. **Coner token is a general technique but significantly enhances the understanding of long text.** Moreover, our method shows a more significant performance improvement when applied to long captions (+2.05 performance improvment) compared to shorter ones (+0.18 performance improvment).\n\n`` Q4: Corner tokens in different position``\n\nThanks for you valuable suggestion. **Corner tokens around [CLS] token may bring better long-text understanding capacity.**  After Text = 67.82%,  Before [CLS] =68.49% **(+0.67)**,  After [CLS] = 68.63% **(+0.81)**. In BERT architecture, the [CLS] token is always the first token of text input. Thus, around [CLS] token may have better performance.\n\n`` Q5: Corner tokens in vision transformers``\n\nThanks for your valuable suggestion. We implement the corner tokens in the image encoder and find that it provides less improvement (0.99% performance improvement) to long-text understanding compared to utilize corner tokens in the text encoder (2.01% performance improvement).\n\n`` Q6: Statistics about the training data``\n\nAs shown in the table 1 of global pdf, we report some statistics of our dataset and compare it with other long-text datasets. \n\n`` Q7: Details on the MLLMs such as specific architecture and instruction information``\n\nWe  provide the hyper-parameter settings of the used MLLMs as shown in the table 2 of global pdf.\n\n`` Q8: For the LAION and COYO datasets, how is a subset selected?``\n\nFollowing Stable Diffusion, we filter Laion and COYO to images with aesthetic_v2>5.0 and resolution ≥512 to about 70M preserved pairs.\n\n`` Q9: Verification step for the extracted long captions.``\n\nThanks for your suggestion. We verify the quality of long texts of three MLLM for  IIW dataset, and compare with human annotated long texts. We utilize GPT4V for assessment the alignment between image and text, following Q-Bench [3] (IIW =4.51, InstructBLIP-Vicuna7B=2.80, LLaVA-v1.5-13B=3.45, ShareGPT4V-13B=3.48). They have good score and low hallucination. Training with long texts from multiple MLLMs may mitigate the inherent biases of one MLLM.\n\n[3] Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision\n\n`` Q10: It is unclear whether the comparison is fair. More visual backbones with LiT training``\n\nWe compared various methods as fairly as possible. **Our long-caption dataset and designed corner token can consistently improve various artitectures.** We use unsupervised ViT (DINO) and self-supervised ViT (MoCo-v3) as image encoder and counduct training with 3M scale dataset following LiT. \n\n|Method| Pretrained Visual Backbone  |ImageNet1k Cls | Long text-image Retrival Avg.|\n|  :----: |  :----: | :----: | :----: | \nCLIP| - | 16.54 |17.78|\nLotCLIP|-|24.65 **(+8.11)**|63.43 **(+45.65)**|\nLiT | MoCo-v3 |  35.52|32.48 |\nLotCLIP | MoCo-v3 | 41.27 **(+5.75)**|59.99 **(+27.51)**|\nLiT | DINO |37.16| 34.28 | \nLotCLIP | DINO | 42.43 **(+5.27)**| 59.24 **(+24.96)**|\n\n`` Q11: Overlap on training and evaluation dataset``\n\n**There is no overlap in the images** used for training and the ShartGPT4V evaluation, as all the images in **our used ShartGPT4V evaluation are sourced from the SAM dataset**.\n\n`` Q12: Directly comparable to LongCLIP``\n\nThanks for your valuable suggestion. **Directly comparable to LongCLIP, our LotCLIP also has better performance**. We implement LotCLIP to fine-tune pre-trained CLIP model using the 1M-scale dataset from ShareGPT4v, similar to LongCLIP. \n\n| Method |Data|Pre-trained CLIP| Long text-image Retrival Avg.|\n|  :----: | :----: | :----: | :----: |\n|  LongCLIP |ShareGPT4V-1M|ViT-B/16|67.16\n|  LotCLIP |ShareGPT4V-1M|ViT-B/16| 82.06 **(+14.9)**\n\n`` Q13: Reason behind the names (Dora and LotCLIP)``\n\nWe name the method as **LotCLIP** (**Lo**ng **T**exts in **C**ontrastive **L**anguage-**I**mage **P**re-training) and we name the dataset as **Dora** (**D**etailed Texts **o**f **R**eal Im**a**ges). We will add the explainations in revision.\n\n`` Q14: Explanation on the concept of Corner Tokens``\n\nSorry for the confusion. We will add the supporting explanations of corner tokens in the updated version. Specifically, we add several learnable tokens from different initialization after [CLS] token, termed as corner tokens.\n\n``Q15: Training Details``\n\nWe apologize for missing the training details, and we add these in table 3 of global pdf. We will fix this issue in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We thank all reviewers for their time and efforts in reviewing our paper. These constructive reviews can bring the improvements for our manuscript. We are encouraged that the reviewers appreciate our method, including

* Problem definition and analysis (Reviewer sjvn, NJUx)
* Effective method (Reviewer f34j, sjvn)
* Strong and detailed experiments (Reviewer f34j, NJUx)
* Valuable dataset (Reviewer f34j, Zcuk, NJUx)

We also have made diligent efforts to address all the concerns raised point by point. In this rebuttal, we have incorporated some new figures to more effectively address the concerns. Kindly review the newly uploaded one-page PDF.

* Table 1 gives statistics about the training data (Reviewer NJUx, Q6)
* Table 2 gives hyper-parameter settings of the used MLLMs.(Reviewer NJUx, Q7)
* Table 3 gives training hyper-parameters of our model (Reviewer NJUx, Q15)

We are open to discussions and addressing any issues from reviewers. Your constructive comments can further help us to improve our method.

Sincerely yours,

Author

Meta-review: The manuscript received positive reviews from all reviewers. The authors response further clarified aspects raised by the reviewers, while a lot of detailed information was presented by the authors.

I consider the paper worth presenting in the conference, and encourage the authors to properly integrate new information from the reviewing process in the final revision.
Decision: Accept (poster)