Title: LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
Venue: NeurIPS 2024
Accepted: yes
Keywords: Language-Image Pre-Training
Abstract: In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method achieves superior performance in long-text-image retrieval tasks. The project page is available at https://wuw2019.github.io/lot-lip.

LoTLIP: Improving Language-Image Pre-training for Long Text Understanding

Abstract

Understanding long text is of great demands in practice but beyond the reach of most language-image pre-training (LIP) models. In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions , however, directly learning with which may lead to performance degradation in understanding short text ( e.g. , in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100 M long caption oriented text-image pairs. Our method achieves superior performance in long-text-image retrieval tasks. The project page is available here.

1 Introduction

Understanding long texts plays a key role in Natural Language Processing (NLP), e.g. , Long Document Analysis [23, 28, 33], in which books, academic papers, and many other types of long texts can be the target of such analysis. Inspired by these works, in the multi-modality field, some text-to-image generation works ( e.g. , DALLE-3 [1] and Pixel-art [4, 5]) employ pre-trained captioners ( e.g. , LLaVA [21]) to generate more accurate and detailed captions (in other words, long captions) for images. These long captions describe an image in detail which can help text-to-image models easily transfer long text to an image, improving the quality of text-image alignment. However, these text-to-image models use pure NLP encoders ( e.g. , T5) to model long captions rather than CLIP. Because, in the language-image pretraining task, little work has been conducted on modeling long texts in a way that effectively aligns with image representations. Despite the lack of exploration, this is a critical problem with practical demands that require a nuanced understanding of textual descriptions corresponding to images.

Two primary challenges hinder the effective integration of long-text understanding in language-image pre-training. The first one is the lack of large-scale long-caption image-text paired datasets. Most

Train with long caption

Understand with long caption

In front of the castle is a well

kept garden with green, white

and red, flowers and a paved

road is around the garden. A

large castle has white walls

with beige trim around its walls

and pillars …

There is a well kept garden

with green, white and red,

flowers in the front of the castle

and a paved road is around the

garden. A large castle has white

walls with beige trim around its

walls and pillars …

Train with short caption

Understand with long caption

There is a well kept garden with

green, white and red, flowers in

the front of the castle and a

paved road is around the garden.

A large castle has white walls

with beige trim around its walls

and pillars …

In front of the castle is a well

kept garden with green, white

and red, flowers and a paved

road is around the garden. A

large castle has white walls with

beige trim around its walls and

pillars …

Train with short caption

Understand with short caption

In front of the castle is a well

kept garden with green, white

and red, flowers and a paved

road is around the garden.

A

large castle has white walls

with beige trim around its

walls and pillars

…

In front of the castle is a well

kept garden with green, white

and red, flowers

and a paved

road is around the garden. A

large castle has white walls

with beige trim around its walls

and pillars …

Original Image

Understand the image

with text caption

according to attention

visualization

Existing CLIP models are good at

understanding short captions

'

garden token

'

is overshadowed by

'

castle token

'

!!

Shuffling the order of sentences, model

still only sees

'

castle token

'

Long caption brings

'

garden token

'

back into light

castle token

garden token

castle token

garden token

castle token

garden token

7.

?

existing datasets focus on short captions ( i.e. , average text length in CC12M [3] is about 17 tokens), which limits the model's exposure to longer text forms. Consequently, models trained on these datasets tend to neglect certain tokens that are easily overshadowed by salient tokens. As shown in Fig. 1, the model trained on short captions can be good at understanding the content of short captions ( i.e. , garden and castle). But when increasing the length of the caption, we can see that 'garden token' is overshadowed by 'castle token', even if we move the 'garden token' front. This bias towards the salient tokens ('castle token' in Fig. 1) of texts can severely restrict the model's ability to comprehend and generate responses based on the full context of longer inputs. The second challenge is the token length limitation of the text encoder. While directly increasing the token number limitation that a model can process appears to be a straightforward solution for accommodating longer texts, it does not have a better understanding of long captions. The fundamental issue remains the model's inability to effectively interpret long texts, primarily due to the lack of appropriate training data that includes long captions.

To address these challenges, we have undertaken an extensive project to re-caption 100 million data with long captions, aiming to enrich the training environment for our models. This initiative allows us to explore the effects of increased text length in image-text pre-trained models and its impacts on model performance. Based on these experiments, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. However, directly learning with long captions may improve the long-text understanding of image-text pre-trained models, but lead to performance degradation in understanding short texts ( e.g. , in the image classification task) as shown in Fig. 2. After integrating corner tokens to aggregate diverse textual information, we successfully enable the model to regain its original proficiency in understanding short texts while significantly improving its ability to comprehend long texts. We also explore whether the model can continue to benefit from longer captions and observe a clear trade-off between performance and efficiency. Moreover, on the task of long-text image retrieval, we beat Long-CLIP [39], a competitor using long captions for fine-tuning, with 4.2% improvement ( i.e. , from 79.52% to 83.72%).

2 Related work

2.1 Language-Image Pre-training

Recently, using language-image pre-trained models to do zero-shot prediction has attracted a lot of attention. CLIP [24] and ALIGN [14] demonstrate contrastive pre-trained models can learn

rich visual-language correspondence knowledge from large-scale image-text pairs on the Internet and achieve good performance on zero-shot predictions, including image-text retrieval [18] and classification [9]. Following their success, various studies [16, 34, 36, 17] have been devoted to improving image-text alignment. Among them, FILIP [34] focuses on fine-grained expressiveness between text tokens and image patches by modifying the training loss. While LiT [37] finds that apply contrastive-tuning to the pre-trained models with locked image encoder and unlocked text encoder can further improve the alignment. It is recognized that larger batch size brings better performance. For this reason, SigLIP [38] proposes to replace the softmax normalization among the standard contrastive loss with the sigmoid loss to scale up training batch size. In addition, LaCLIP [11], RLEG [40] and some other works [19, 13, 22, 30] utilize multi-modality generative models to improve data quality for enhancing pre-training.

2.2 Long-text Understanding

Detailed long texts are necessary for many artificial intelligence tasks ( e.g. , human-computer interaction). Therefore, modeling and parsing long texts has become one of the most important research directions in natural language processing. Many studies in fields such as text generation [23, 32, 33] have shown that advanced transformer structures have the ability to interpret long texts. However, in the field of language-image pre-training, research on using long-text descriptions of images to enhance multimodal representations is still very scarce. DreamLIP [41] utilizes multimodality large language model to re-caption image data with detailed descriptions and then use them in language-image pre-training. In fact, during training, DreamLIP randomly extracts sub-captions from the detailed description for training and does not completely use all the information of the long text. One of the reasons why previous language-image pre-training rarely directly applied long texts for training is the text encoder of traditional CLIP [24] is restricted by the token number limit ( ≤ 77 ). Then, Long-CLIP [39] firstly introduces long captions into CLIP model to finetune, where the model is pre-trained on short-text-image datasets. The fine-tuning process equips the model with the ability to comprehend long texts, albeit at the expense of its proficiency in understanding shorter texts. In contrast, we incorporate long captions during the pre-training stage, which can not only enable the model to regain its original competency in interpreting short texts but also significantly improves its understanding of long texts. We further explored the trade-off between the benefit of long texts to the model and the training efficiency.

3 Preliminary of Language-Image Pre-training

Language-image pre-training models, i.e. , CLIP [24] and LiT [37], typically consist of an image encoder and a text encoder. In the pre-training stage, the language-image model takes image-text pairs as input and uses image encoder and text encoder to extract embeddings from images and texts, respectively. Then, two encoders are trained with contrastive objectives, ensuring that paired image and text embeddings are close in the embedding space, while unpaired pairs are far apart. Specifically, a batch of images { I 1 , I 2 , · · · , I N } and the corresponding short texts { T 1 , T 2 , · · · , T N } are randomly sampled from the pre-training dataset. Each image and its corresponding text are treated as a positive pair, while others in the same batch are negative pairs. Then, the image encoder extracts image global feature v i from the i -th image I i within the batch, while the text encoder obtains text feature t j from the j -th text T j . These two encoders are then optimized using contrastive loss L , which consists of image-to-text loss L i 2 t and text-to-image loss L t 2 i . It can be formulated as follows:

where τ is a learnable temperature parameter, and cos ⟨· , ·⟩ means the cosine similarity between two normalized feature vectors.

4 Long Texts in Language-Image Pre-training

Currently available image-text pair datasets, e.g. CC12M [27], typically include short texts that have an average length of approximately 17 tokens. Language-image models pre-trained on these datasets perform well on short-text comprehension tasks. However, we find that they struggle to comprehend long texts for text-image alignment. Concretely, they tend to overlook or neglect some tokens or sub-captions in long texts, as shown in Fig. 1. A potential reason for such a situation is the lack of long-text-image pairs in pre-training. Thus, we collected and re-captioned 100M images with long texts. In this section, we provide details of re-captioning and explore the usage of long texts in language-image pre-training.

4.1 Long Text-Image Pair Dataset

Training Dataset. To construct long text-image pairs for language-image pre-training, we recaptioned 100 million images with long texts. Specifically, we collected the images from CC3M [27], CC12M [27], YFCC15M [29], LAION [26], and COYO [2] dataset. Then, we instructed three multimodality large language models (MLLMs), i.e. , InstructBLIP [7], LLaVA [20], and ShareGPT4V [6] to generate diverse and descriptive long texts based on the collected images. In this step, we used 'Describe the image in detail.' as the text prompt, following DreamLIP [41]. Finally, each image in the collected datasets is paired with four texts: a raw text from the original dataset and three re-captioned long texts. The raw texts exhibit an average length of approximately 18 tokens, whereas the re-captioned long texts consist of around 136 tokens.

Evaluation Dataset. Most zero-shot evaluation tasks for language-image pre-training primarily rely on short textual input. For example, in short-text-image-retrieval tasks, the textual inputs contain fewer than 15 tokens on average, as shown in Tab. 1. Given the short texts, these tasks are not suitable for accessing the long text comprehension ability of pre-trained models. Therefore, we collected long text-image pairs from DCI [31], IIW [12], and ShareGPT4V [6] datasets to construct long-text-image retrieval evaluation tasks. Specifically, for DCI and IIW datasets, we use images with human-annotated long descriptions for retrieval. For ShareGPT4V dataset, we sample 1,000 and 10,000 data from ShareGPT4V dataset to construct ShareGPT4V-1k and ShareGPT4V-10k retrieval dataset, respectively. The ShareGPT4V-1k dataset is constructed following Long-CLIP [39]. All images in ShareGPT4V-1k and ShareGPT4V-10k are from SA-1B [15] dataset, and all long texts are generated by ShareGPT4V-Captioner [6]. As shown in Tab. 1, each image within these datasets has one paired long text, which consists of more than 8 sub-captions and 170 tokens on average. Here, a sub-caption is a complete sentence ending with a period. The long-text-image retrieval task necessitates the alignment of text features from long texts with the image features from the corresponding images, which thereby evaluates the model's ability to comprehend long texts.

4.2 Exploring the Influence of Text length

Long-CLIP [39] enables long text processing ability by fine-tuning CLIP model with long text-image pairs. However, the benefits and drawbacks of using long texts in pre-training are still unknown. It

0

1

2

3

4

5

6

Sub-caption Num

30

40

50

60

Long-text-image

Retrieval R@1

DCI T2I

ShareGPT4V-10k T2I

DCI I2T

ShareGPT4V-10k I2T

(a) long-text-image retrieval

0

1

2

3

4

5

6

Sub-caption Num

25

30

35

40

45

Short-text-image

Retrieval R@1

MSCOCO I2T

MSCOCO T2I

(b) short-text-image retrieval

0

1

2

3

4

5

6

Sub-caption Num

44

46

48

50

Image Classfication

Acc@1

ImageNet

(c) image classification

Text

Encoder

!

Long Caption

(Omit Abbreviation)

Text

Embedding

Corner

Embedding

CLS

corner

text tokens

corner

text

tokens

Key

Query

heads

In the picture, there are several

objects that can be observed in

the living room. These objects

include a couch, chairs

××××××

.

text tokens

Mask Multi-Head

Transformer blocks

Mask Multi-Head

Transformer blocks

Mask Multi-Head

Transformer blocks

Mask Multi-Head

Transformer blocks

text tokens

Image

Embedding

Image

Encoder

❄

…

…

…

CLIP

Loss

…

'()

corner

Contrastive

Loss

[CLS]

[Cor 1]

[CLS]

[Cor 1]

img

img*img3img² img²img°

cor

cls0

cls1

cls

cls

cls

is also unknown how the length of the text affects the performance of the pre-trained model. To explore the influence of using texts in different lengths for pre-training, we change the length of long texts by selecting different numbers of consecutive sub-captions, as illustrated in Fig. 2. Each sub-caption in the long texts has an average of approximately 22 tokens. Sub-caption number equal to 0 indicates that the model is trained without the generated texts. When introducing one sub-caption, a noticeable gain is achieved across all tasks. Moreover, the results also indicate that pre-training with longer texts, composed of more sub-captions, enhances the performance of the pre-trained model on long-text-image retrieval tasks. It confirms that using long text-image pairs for pre-training improves the model's understanding of long texts. However, the usage of longer texts negatively impacts the model's performance on short-text-image retrieval and image classification tasks.

4.3 Method

In order to find a solution that well balances the long and short text understanding of the pre-trained model, we design to add extra text tokens for text encoders, termed corner tokens , which can aggregate diverse text features. This strategy benefits the class ( i.e. , [ CLS ]) token by extracting more representative features for long and short text. Next, we will describe the details.

Corner Tokens. Different text encoder architectures ( e.g. , BERT [10], T5 [25]) can be utilized in language-image pre-training. In this section, we take BERT [10] as an example of our approach. Given a long text, it is expressed as [ CLS ] , · · · [ SEP ] , · · · [ SEP ] , where the first token of every text input is its class token [ CLS ] , and all sub-captions are separated by a special token [ SEP ] . The omitted part in · · · denotes the tokens obtained after tokenizing words within the sub-caption. Based on the tokenized long text, we insert multiple learnable corner tokens C = { [ Cor 1 ] , · · · , [ Cor m ] } after

the class token, where m is the number of corner tokens. In this way, the form of the tokenized text input is [ CLS ] , [ Cor 1 ] . . . [ Cor m ] , · · · [ SEP ] , · · · [ SEP ] . Moreover, we design an attention mask mechanism A for the text encoder to promote the diversity of the aggregated features. Specifically, when calculating the attention scores, the corner tokens and [ CLS ] token are guided to neglect each other but attend to all other sub-caption tokens. Meanwhile, in the attention mask mechanism, each text tokens are designed to only interact with other text tokens and the [ CLS ] token, to keep the interactions between local and global information. The attention mask A is formulated as:

̸

where q and k represent the query and key tokens in attention block, respectively. The features of the [ CLS ] and corner tokens are regarded as text global feature t g and corner features t c 1 , t c 2 , ..., t c m , respectively.

Optimization. The short-text-image contrastive loss L short is calculated in the same way as Eq. (1). Meanwhile, the long-text-image contrastive loss L long between image global feature v and t g , t c 1 , t c 2 , ..., t c m of long text is as follows:

The total training loss is L LoTLIP = L long + L short .

5 Experiments

5.1 Implementation Details and Datasets

Pre-training Datasets. As presented in Sec. 4.1, we collected 100M data from five publicly available image-text pair datasets and re-captioned the collected images with long texts. Based on this dataset, we construct 4 scales of pre-training data: (1) 3M, including CC3M. (2) 12M, including CC12M. (3) 30M, including CC3M, CC12M, and YFCC15M. (4) 100M, including all re-captioned data. We conduct ablation studies to validate our model on the 3M scale pre-training data. The performance of LoTLIP pre-trained with 12M and 30M scale datasets is shown in the Supplementary Material .

Downstream Datasets. To assess the ability of the pre-trained models on short-text and long-text understanding, we select 3 downstream tasks for evaluation under the zero-shot setting, including long-text-image retrieval, short-image-text retrieval, and image classification. For long-text-image retrieval , we present the Recall at 1 (R@1) metric of the pre-trained models on DCI, IIW, and ShareGPT4V-1k, and ShareGPT4V-10k long-text-image retrieval tasks. For short-image-text retrieval , we evaluate on MSCOCO [18] and Flickr30k Caption [35] and report Recall at 1/5 (R@1/5) metric for comparison. For image classification , we use ImageNet1k [9] for evaluation and present top-1 accuracy (Acc@1) on image classification. Following [24], we use class names incorporated with pre-defined text prompts as text inputs for zero-shot classification.

Implementation Details. Following LiT [37], we use a vision transformer pre-trained on ImageNet 21K as the image encoder and Bert [10] as the text encoder. The architecture of the image encoder is ViT-B/16. We report other variants of vision transformers in the Supplementary Material . The images are resized to 224 × 224. The maximum text token length is set to 128 unless specifically stated. Three consecutive sub-captions are randomly selected to form long texts as text input. We train 10 epochs on the 3M and 100M scale datasets. For the 3M dataset, the batch size is set to 2560, while that of 100M is set to 16384. The other pre-training hyperparameters are under the same setting, e.g. learning rate, warmup steps, and weight decay.

5.2 Ablation Studies

Exploring the Influence of Token Number Limitation. Token number limitation is the maximum length of text input that the text encoder can process at a time. For languageimage pre-training models, the token number limitation of text encoder can be arbitrarily set to any positive integer, which is typically set to 77 [24, 38, 41]. The benefits derived from using long texts in pre-training might be constrained by the 77 token limitation. To investigate this hypothesis, we incrementally raise the token number limitation from 32 to 512. The results are illustrated in Fig. 4, which indicate a limitation of 77 tokens is insufficient for a model training with long texts. On DCI and ShareGPT4V-10k retrieval tasks, the best results are observed when the token number limitation is set to 192. Meanwhile, the highest performance on the IIW retrieval task is reached

32

77

128

192

256

512

Maximum Token Length

35

45

55

65

75

85

Long-text-image

Retrieval R@1

DCI I2T

IIW I2T

ShareGPT4V-10k I2T

10

20

30

40

FLOPs of Text Encoder (G)

when the text token length limitation is 256. Since the average text length of the evaluation datasets is less than 240 tokens, as shown in Tab. 1. When the text token length limitation gets larger than 256, these datasets can't well prove the long-text understanding ability of the model. In Fig. 4, we also illustrate the FLOPs of the text encoder, which increases with the text token number limitation. To balance the training efficiency and performance, we set the token number limitation to 128 for the text encoder of LoTLIP .

Compare LoTLIP with Other Methods in Pre-training with Long Texts. LoTLIP aims to enhance the long-text and short-text comprehension ability of language-image pre-training. To demonstrate the effectiveness of LoTLIP , we compare it with other approaches that train with long text-image pairs. The first approach is directly using long texts in the training stage of LiT. The second approach is based on another contrastive learning method related to long texts, namely Long-CLIP [39]. Concretely, we incorporate the primary component matching strategy and losses proposed by Long-CLIP to LiT (LiT+Long-CLIP). For a fair comparison, all models are trained with 3M scale dataset. As shown in Tab. 2, directly using long texts in pre-training procedural significantly improves LiT over all tasks. Moreover, LiT+Long-CLIP exceeds LiT on the long-text-image retrieval tasks when both models incorporate long texts during pre-training. But on short-text comprehension tasks, i.e. , short-text-image retrieval and image classification, the performance of LiT+Long-CLIP is inferior to that of LiT. Instead, LoTLIP improves LiT and LiT+Long-CLIP on both long and short text comprehension tasks. Concretely, LoTLIP surpasses LiT and LiT+Long-CLIP by 1.97% and 1.51% on average over three long-text-image retrieval tasks. Besides, LoTLIP improves LiT by 2.29% and 1.47% on MSCOCO retrieval and ImageNet classification, respectively. The results demonstrate

that LoTLIP benefits from long texts and corner tokens, thereby exhibiting a better understanding of the long and short texts.

Implementation of Corner Tokens. In this part, we study the influence of the attention mask mechanism and the number of corner tokens on different downstream tasks as shown in Tab. 3. LoTLIP pre-trained without the pre-defined attention mask encounters performance degradation on most tasks compared to when using the pre-defined attention mask. It indicates that direct interaction between the CLS token and corner tokens limits their ability to aggregate diverse textual features. On short-text-image retrieval and image classification tasks, the performance of LoTLIP improves when introducing more corner tokens. It proves that the corner tokens help with enhancing the short text understanding ability of LoTLIP . When the number of corner tokens is set to more than two, the performance improvement across all tasks is relatively small. Thus, we use two corner tokens in LoTLIP .

5.3 Main Results

We compare LoTLIP with the state-of-the-art methods on downstream tasks involving long texts and short texts. The experimental results are shown in Tab. 4 and Tab. 5, respectively. On 3M data scale, LoTLIP significantly improves the state-of-the-art methods on all tasks. Specifically, LoTLIP

improves the second-best method LiT by 29.99% on average over four long-text-image retrieval tasks. Moreover, LoTLIP improves the second competitor LiT by 6.58% and 10.98% on image classification task and short-text-image retrieval tasks, respectively. It proves that LoTLIP significantly enhances the language-image model for understanding short and long captions by involving long captions in pre-training and incorporating corner tokens in text inputs. It is worth noting that LoTLIP trained with 100M data exceeds all state-of-the-art methods on long-text-image retrieval tasks, even though these methods are pre-trained on a larger scale of data. Concretely, LoTLIP (trained on 100M data) exceeds SigLIP (trained with 12B) by an average of 7.13% over four long-text-image retrieval tasks. Moreover, compared to Long-CLIP, which uses long texts in CLIP for fine-tuning, LoTLIP improves averaged performance by 4.2% on long-text-image retrieval tasks.

6 Conclusion

In this work, we empirically confirm that a key issue arises because training images are typically paired with short captions, which can cause certain tokens to be overshadowed by more salient ones. To address this issue, our initial strategy involved relabeling the data with long captions. However, directly learning from these long captions might lead to degraded performance in tasks requiring an understanding of short text, such as image classification. Subsequently, by incorporating corner tokens to aggregate diverse textual information, we are able to help the model regain its original proficiency in understanding short texts while significantly enhancing its capability to comprehend long texts. We also investigated whether the model could continue to benefit from longer captions and observed a clear trade-off between performance and efficiency. Finally, we validated the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100 million longcaption-oriented text-image pairs. In the task of long-text-image retrieval, our method outperforms the second-best competitor, Long-CLIP, with an improvement of 4.2%.

7 Acknowledgments

This work was supported by National Natural Science Foundation of China (NSFC) under Grants 62225207, and Zhejiang Provincial Natural Science Foundation of China under Grants LD24F020011.

References

J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y . Guo, et al. Improving image generation with better captions. Computer Science. https://cdn.openai.com/papers/dall-e-3.pdf , 2:8, 2023.

M. Byeon, B. Park, H. Kim, S. Lee, W. Baek, and S. Kim. Coyo-700m: Image-text pair dataset. https: //github.com/kakaobrain/coyo-dataset , 2022.

S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In IEEE Conf. Comput. Vis. Pattern Recog. , 2021.

J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al. Pixartalpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 , 2023.

J. Chen, Y. Wu, S. Luo, E. Xie, S. Paul, P. Luo, H. Zhao, and Z. Li. Pixart-{\ delta } : Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.05252 , 2024.

L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. ShareGPT4V: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 , 2023.

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. Adv. Neural Inform. Process. Syst. , 36, 2023.

T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. In Int. Conf. Learn. Represent. , 2024.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog. , 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving CLIP training with language rewrites. Adv. Neural Inform. Process. Syst. , 36:35544-35575, 2023.

R. Garg, A. Burns, B. K. Ayan, Y. Bitton, C. Montgomery, Y. Onoe, A. Bunner, R. Krishna, J. Baldridge, and R. Soricut. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793 , 2024.

H. A. A. K. Hammoud, H. Itani, F. Pizzati, P. Torr, A. Bibi, and B. Ghanem. SynthCLIP: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402.01832 , 2024.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Int. Conf. Mach. Learn. , pages 4904-4916, 2021.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In Int. Conf. Comput. Vis. , pages 4015-4026, 2023.

J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. UniCLIP: Unified framework for contrastive language-image pre-training. Adv. Neural Inform. Process. Syst. , 35:1008-1019, 2022.

Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 23390-23400, 2023.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis. , pages 740-755. Springer, 2014.

D. Liu, H. Zhang, F. Wu, and Z.-J. Zha. Learning to assemble neural module tree networks for visual grounding. In Int. Conf. Comput. Vis. , pages 4673-4682, 2019.

H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Adv. Neural Inform. Process. Syst. , 36, 2023.

Y. Liu, K. Wang, W. Shao, P. Luo, Y. Qiao, M. Z. Shou, K. Zhang, and Y. You. MLLMs-augmented visual-language representation learning. arXiv preprint arXiv:2311.18765 , 2023.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Adv. Neural Inform. Process. Syst. , 35:27730-27744, 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn. , 2021.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Association for Computational Linguistics , 2018.

G. Tan, D. Liu, M. Wang, and Z.-J. Zha. Learning to discretely compose reasoning module networks for video captioning. arXiv preprint arXiv:2007.09049 , 2020.

B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. StableRep: Synthetic images from text-to-image models make strong visual representation learners. Adv. Neural Inform. Process. Syst. , 36:48382-48402, 2023.

J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V. Sharma, and A. Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. arXiv preprint arXiv:2312.08578 , 2023.

H. Wang, Z.-J. Zha, L. Li, D. Liu, and J. Luo. Structured multi-level interaction network for video moment localization via language query. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 7026-7035, 2021.

W. Xiong, J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungta, K. A. Sankararaman, B. Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039 , 2023.

L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu. FILIP: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 , 2021.

P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , pages 67-78, 2014.

Z.-J. Zha, D. Liu, H. Zhang, Y. Zhang, and F. Wu. Context-aware visual policy network for fine-grained image captioning. IEEE Trans. Pattern Anal. Mach. Intell. , 44:710-722, 2019.

X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. LiT: Zero-shot transfer with locked-image text tuning. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 18123-18133, 2022.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Int. Conf. Comput. Vis. , pages 11975-11986, 2023.

B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403.15378 , 2024.

L. Zhao, K. Zheng, Y. Zheng, D. Zhao, and J. Zhou. RLEG: vision-language representation learning with diffusion-based embedding generation. In Int. Conf. Mach. Learn. , pages 42247-42258, 2023.

K. Zheng, Y. Zhang, W. Wu, F. Lu, S. Ma, X. Jin, W. Chen, and Y. Shen. DreamLIP: Language-image pre-training with long captions. arXiv preprint arXiv:2403.17007 , 2024.

8 Appendix

8.1 Data Statistics

As shown in Tab. 6, we report the data statistics of our dataset and other text-image paired datasets. Compared to the other text-image paired datasets, the re-captioned texts of LoTLIP are significantly longer, averaging 136 tokens v.s. 18 tokens. To the best of our knowledge, LoTLIP dataset is the largest dataset consisting of long texts for multi-modal learning. We are continuing to expand the size of LoTLIP by integrating additional MLLMs, as well as gathering more publicly available datasets for long text generation.

8.2 More Experimental Analysis

8.2.1 Influence of Sub-caption Number on LoTLIP and LiT

During the pre-training stage, we randomly select multiple consecutive sub-captions from long texts in LoTLIP as long text inputs. In Fig. 5, we show the influence of the number of sub-captions when the length of text input is limited to 128. With the number of sub-captions increasing, LiT and LoTLIP reaches better performance in both text retrieval and image retrieval on the ShareGPT4V-10k and DCI datasets. There are small improvements when the number of sub-captions gets larger than 3. Moreover, both methods exhibit performance degradation on text-image retrieval and image classification tasks as the number of sub-captions increases. But the performance drop of LoTLIP is even slower. It further indicates that LoTLIP can enhance the understanding of long texts while retaining the short-text understanding ability.

8.2.2 Compare Corner Tokens with Register Tokens

In LoTLIP , we employ learnable tokens, namely corner tokens, to help the model regain its original performance in understanding short texts, while significantly enhancing its capability to comprehend long texts. It is mentioned that registered token [8] also adds learnable tokens in the encoder, which are used to get rid of artifacts in image feature maps. In order to fairly compare corner tokens with register tokens, we implement the same number of register tokens on the text encoder and train the model with the same scale of dataset as LoTLIP . As shown in Tab. 7, the results demonstrate that corner tokens improve the performance of the model for long-text and short-text understanding tasks, which is better than register tokens.

8.2.3 Contrastive Learning without Pre-trained Weights

In Tab. 8, we present the performance of LoTLIP without loading pre-trained weights for text and image encoder on long and short-text related tasks. The results show that directly involving long texts in the pre-training stage can also enhance the performance of the language-image model with randomly initialized weights over all tasks. Besides, LoTLIP further improves CLIP on long and short text-related tasks, when they are both pre-trained with long texts. In our methodology, we opt

1

2

3

4

5

SubCaption Num

45

50

55

60

65

70

75

Long-text-image

ShareGPT4v-LiT

DCI-LiT

ShareGPT4v-LoTLIP

DCI-LoTLIP

(a)

1

2

3

4

5

SubCaption Num

40

45

50

55

60

65

70

Long-text-image

ShareGPT4v-LiT

DCI-LiT

ShareGPT4v-LoTLIP

DCI-LoTLIP

(b)

1

2

3

4

5

SubCaption Num

30

32

34

36

38

40

42

44

46

48

MSCOCO I2T R@1(%)

LiT

LoTLIP

(c)

1

2

3

4

5

SubCaption Num

25

26

27

28

29

30

31

32

MSCOCO T2I R@1(%)

LiT

LoTLIP

(d)

1

2

3

4

5

SubCaption Num

44

45

46

47

48

49

50

51

ImageNet Acc(%)

LiT

LoTLIP

(e)

to load pre-trained weights and fix the image encoder during training, following LiT. This is because LiT is a strong baseline and such a training setting significantly conserves computational resources.

8.2.4 Utilizing Long Captions from Different MLLMs

We use three MLLMs, i.e. , InstructBLIP [7], LLaVA [20], and ShareGPT4V [6], to generate diverse long captions for 100M images. In this part, we conduct experiments to analyze the influence of using long texts generated by different MLLMs in pre-training. As shown in Tab. 9, LoTLIP trained with all generated long texts reach the best performance on average over all tasks. It means that diverse

long texts help language-image pre-trained models to better comprehend texts to align with images. Moreover, LoTLIP trained with long captions generated by ShareGPT4V reaches higher scores on long-text-image retrieval tasks than other MLLMs. This indicates that the long captions generated by ShareGPT4V are of higher quality, which can enhance the long-text comprehension ability of LoTLIP .

8.2.5 Pre-training with different scale of dataset

We compare LoTLIP and LiT on different scales of the pre-training dataset, i.e. , 3M, 12M, 30M, and 100M on long-text-image retrieval, text-image retrieval, and image classification. The results on long-text and short-text centered tasks are shown in Tab. 10 and Tab. 11, respectively. LoTLIP significantly improves LiT on all tasks in both ViT-B/32 and ViT-B/16 under different pre-training settings. It proves that involving long captions in the pre-training stage can help the language-image model to better deal with both long and short-comprehension tasks.

8.3 Visualization

We visualize the attention map of LiT, LiT trained with long texts, Long-CLIP (our implementation), and LoTLIP in Fig. 6. Given the long caption in the first column, the attention visualization map of LiT can only activate regions corresponding to partial objects mentioned in the long caption, e.g. , flat rock. This situation is alleviated after incorporating long captions in LiT pre-training. Benefiting from corner tokens, the highlighted image regions of LoTLIP are well aligned with the given long caption.

An person is washing clothes by scrubbing them on a flat rock with both hands . The flat rock is long enough for both of the person's feet to be side by side. It must be chilly because a mist is rising off the water, and the person is wearing long sleeves. the person probably belongs to a group that believes the head should be covered.

There is a reg and gray bird in a tree. It is perched on a branch and looking to its left side. Around it are other branches that have green leaves. The branches vary in size, but they are mostly tan. There is a blue sky in the background. It is sunny and dappled sunlight is on the plants and the bird.

LiT

LiT+Long Texts

LoTLIP

Long-CLIP

8.4 Limitation

We re-wright the captions of 100M images using three popular open-sourced multi-modal large language models ( i.e. InstructBLIP [7], LLaVA [20] and ShareGPT4V [6]), but we observed hallucination elements in the synthesized long captions. The hallucinations in captions, which do not correspond with the image information, may restrict the full potential of long captions in enhancing the understanding of lengthy texts by language-image pre-trained models.

8.5 Experiment Hardware

To obtain our LoTLIP trained with 100M scale dataset, we apply A100 GPU with 80G memory for training, which costs about 133 GPU days. For models trained on datasets of other scales ( i.e. 3M, 12M, 30M), the training duration decreases linearly with the amount of training data.

8.6 Social Impact

The 100M images used for pre-training are publicly accessible, and the re-annotated long captions are synthesized using public multi-modal large language models on these public datasets, posing no ethical risk in the data source. Our LoTLIP is incapable of generating images and text, thus there is no need for concern regarding the negative social impact resulting from fake, violent, pornographic, or discriminatory content. Moreover, LoTLIP has the potential for positive social impact, considering its excellent image-text retrieval performance, it may serve as a valuable asset in image retrieval libraries in the future.

8.7 Ethic Statement

We have conducted a thorough review to ensure that there has been no violation of the NeurIPS Code of Ethics in this paper.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we explicitly state that our initial attempt is to relabel the data with long captions to facilitate the understanding of long text by the language-image pre-training (LIP) model.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the appendix, we discuss the limitations regarding the the potential harmfulness of the hallucination elements in long captions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [No]

Justification: We primarily demonstrate through experimentation that short captions fail to unlock the potential of LIP in understanding long texts, and that our approach effectively models long captions in a way that aligns with image representation, as described in Sec. 4 and Sec. 5.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In both the description of our method in Sec. 4 and the details of experimental implementation in Sec. 5, we disclose the information needed to reproduce the main experimental results. We will also release our code, model, and the new dataset to facilitate the reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code for this paper requires approval before it can be made open source, hence it is not provided in this submission. However, the code, models, and datasets of this paper will be made publicly accessible after this submission to ensure the reproducibility of the experiments and to foster research progress within the community.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Sec. 5, we explicitly present the implementation details, including the training data, pretraining hyperparameters, the settings of comparison methods, etc.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This paper proposes a foundation model that incurs significant training expenses, so we fixed the random seed and trained it only once. During the evaluation, we also fixed the random seed to ensure the constant results. Consequently, the fluctuation and error in our experimental results are slight.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the appendix, we have specified the GPU type, the memory, the numbers of GPUs required for one training session, and the training duration for our model.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have conducted a thorough review to ensure that there has been no violation of the NeurIPS Code of Ethics in this paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our model has the potential for positive social impact, given its strong imagetext retrieval performance, it may serve as a valuable asset in image retrieval libraries in the future.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The image data used for training is publicly accessible, and the long captions were synthesized using public models on public datasets, posing no security risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code and dataset in the paper. The image data used for training is publicly accessible, and the long captions were synthesized using public models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: We have rewritten captions for 100M images, but the new data requires approval before release. The new data will be made available to the public with our code after submission.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LotCLIP, which enhances CLIP’s capability to understand long texts. It highlights that merely increasing the length of texts (i.e., context length) is not beneficial as it adversely impacts the understanding of short texts (i.e., image classification). To mitigate this trade-off, additional learnable corner tokens are integrated into the text encoder transformer. Moreover, the attention mask is modified so that interactions between corner tokens are restricted, ensuring diversity in the output feature embeddings. The image encoder and text encoder are initialized from pre-trained models and further trained in a LiT manner. LotCLIP is trained on a self-constructed image-text dataset of 100M scale long texts using MLLMs. LotCLIP is evaluated on both long-text retrieval tasks and short-text benchmarks such as image-text retrieval and classification.\n\nSTRENGTHS:\n- This paper addresses the underexplored yet important problem of enhancing CLIP's ability to understand long texts. It introduces a simple modification to the CLIP training framework.\n\n- The 100M scale long caption data will be valuable for training VLMs with an understanding of long contexts.\n\n- Strong empirical results compared to other baselines such as LiT , CLIP, and SigLIP.\n\nWEAKNESSES:\n- [W1] Contributions are unclear. From an architectural perspective, there seems to be no major difference from [1, 2], which introduce additional learnable tokens in the encoders. Is there any special reason or evidence that such a technique is especially beneficial for training with long captions? It is likely to be a general technique that is also helpful for training with short captions. Such learnable tokens, named corner tokens in the paper, are only prepended to the text tokens after the [CLS] token. Corner tokens in different positions, such as after the text tokens, can be further ablated. Would corner tokens also be helpful in vision transformers? More comprehensive analysis around the corner tokens are necessary. \n\n- [W2] Connected to the [W1], the contributions of the data are unevaluable. It only mentions that some image-text pairs from various sources (e.g., CC3M/12M, YFCC15M, LAION, and COYO) are re-captioned using MLLMs such as InstructBLIP, LLaMA, and ShareGPT-4V. No other clear details are provided. Any statistics about the training data are missing. Any details on the MLLMs such as specific architecture and instruction information are completely omitted. For the LAION and COYO datasets, how is a subset selected from the entire scale? Most importantly, no verification step for the extracted long captions is provided. Due to the typical hallucinations of MLLMs, it is unclear how well the obtained long captions align with the original images or original short captions, further complicating the training of VLMs. \n\n- [W3] It is unclear whether the comparison is fair. With the LiT training mechanism, LotCLIP benefits from an ImageNet-pretrained ViT backbone, which shows strong evaluation results on ImageNet compared to other pretrained backbones. It is suggested that other visual backbones pretrained via CLIP or through unsupervised methods such as DINO be tested with LiT training. \n\n- [W4] Training and some evaluation benchmarks seem to overlap, leading to high performance results. For example, ShareGPT has LAION and Conceptual Captions images, which shares the training data. This creates a significant gap in evaluation results compared to DCI, and even between models from the default CLIP and the proposed LotCLIP in ShartGPT evaluation.\n\n- [W5] It is not directly comparable to LongCLIP. Starting from the same pretrained CLIP model, how does LotCLIP perform when fine-tuned on a 1M-scale dataset similar to that used for LongCLIP?\n\n\nIn summary, in the current version, there is no clear evidence of technical contributions and the experimental settings are unclear. \n\n\n---\n\nReferences\n\n[1] Darcet et al., Vision Transformers Need Registers, in ICLR 2024.\n\n[2] Lavoie et al., Modeling Caption Diversity in Contrastive Vision-Language Pretraining, in arXiv preprint 2024.\n\nQUESTIONS:\n- The reasoning behind the naming of the constructed data and model is unclear. Why are they named Dora and LotCLIP?\n\n- In the introduction section, the concept of the corner token first appears, but without supporting explanations, which creates confusion.\n\n- No training details for the baseline methods are provided.\n\nLIMITATIONS:\n- Some limitations on the long-caption data (e.g., hallucinations) are mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo improve the ability of vision-language models (VLMs) for long-text understanding, the paper proposes to relabel the data with long captions, however, direct learning may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, corner tokens are introduced to aggregate diverse textual information, enabling the model to catch up to its original level of short-text understanding yet greatly enhance its capability of long-text understanding. Experiments are performed on a large-scale long caption dataset to demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1) The author points out the phenomenon that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens.\n2) To improve the ability of vision-language models (VLMs) for long-text understanding, the paper proposes to relabel the data with long captions; corner tokens are introduced to aggregate diverse textual information, enabling the model to catch up to its original level of short-text understanding yet greatly enhance its capability of long-text understanding.\n3) Experiments are performed on a large-scale long caption dataset to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1) Lack of details for addressing the limitation for the token length limitation of the text encoder in Sec. 3.4. \n2) For the attention mask part, if the corner tokens can be seen by other text tokens, how does it perform? In Table 3, how the model will the model perform if the corner taken is removed?\n3) The year of references for NeurIPS papers e.g., [10] and [14], is not correct.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nYes. Both the limitation and potential impact are well described in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper addresses a significant gap in current language-image pre-training models, which are typically trained on datasets with short captions. This limitation hinders the models' ability to effectively understand and process long texts. The proposed solution, LotCLIP, introduces methods to enhance long-text understanding without compromising the performance on short-text tasks.

STRENGTHS:
(a) The authors re-captioned 100 million images with long texts using multi-modality large language models.

(b) Introducing the concept of  corner token

WEAKNESSES:
(a) Not very clear why the proposed method is based on CLIP architecture only- could it have been built upon other similar algorithm as well, for example – ALIGN ?

(b) Need to illustrate in detail how this “corner token” learning is actually taking place. 

(c) Should have also reported results on  ALIGN 


(d) The references are not correct/incomplete in some occasions -please cross check all.

QUESTIONS:
Could you please provide a detail diagram on training process encompassing the "corner tokens ?

LIMITATIONS:
No limitations were mentioned by the authors.

***** Raising the final score to 5  from 4********************
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
The paper describes a framework to adapt language-image pre-training models to longer captions. For that, first a new dataset is created with longer captions and second, training is modified to adapt to longer captions. New corner tokens are introduced that are supposed to capture longer dependencies in the text, and the training loss is modified to take into account both short and long text. Experimental results show that the method performs better than other methods for tasks including both long-text retrieval and short-text retrieval.

STRENGTHS:
- A new dataset is created that can be used in the future for improving the ability of the models for long-text pairing with images
- Several modifications are included in the training procedure that allow to take into account long text while preserving the performance with short text. These modification include modifying the text representation with the corner tokens, introducing a specific attention mask and defining a new loss balancing loss for long and short text.
- Experiments show that the proposed method performs better than other existing approaches both in long text and short text retrieval. Experiments includes a detailed ablation study analyzing the impact of the different components of the model.

WEAKNESSES:
It is not clear the difference of the new Dora dataset with the dataset proposed in DreamLIP (reference [34]) where 30M images are also re-captioned with MLLMs. More details on how the Dora dataset has been built will be useful and also some statistics with respect of the length of the captions in the dataset, comparing with the other datasets analyzed in table 1

QUESTIONS:
See above in weaknesses

LIMITATIONS:
There is no specific discussion on the limitations of the method
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12