TULIP: Token-length Upgraded CLIP

## Abstract

Abstract We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions.
Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation.

## 1 Introduction

Obtaining text representations for long captions in vision-language models is an open research challenge.
Within the context of text-only large language models, this problem has been studied extensively (Chung et al., ; Touvron et al., ; Bai et al., ; Gu & Dao, ; Su et al., ; Dubey et al., ; Pawar et al., ; Jiang et al., ; Lieber et al., ), yet, methods for context length expansion have only scarcely made their way into the vision-language domain. For instance, contrastive vision-language models like CLIP (Radford et al., ; Jia et al., ; Li et al., ) are constrained to short input sequences, capped at 77 tokens.
Natively, such models based on Transformers (Vaswani et al., ) process their input as an unordered set of tokens and use positional encoding to preserve order information (Yun et al., ). In particular, CLIP models use absolute positional encodings, which rely on a predefined maximum number of tokens, thereby limiting the input sequence to 77 tokens by design.
We introduce an efficient and generalizable method, TULIP, that upgrades the context window size of CLIP-like models.

Recently, Long-CLIP (Zhang et al., ) highlighted this problem and proposed an approach for unlocking the long-text capabilities of CLIP.
However, their approach focuses on stretching the existing absolute positional encodings, which primarily addresses the issue of the hard token limit but it continues to rely on absolute encodings which inhibit the model’s comprehension of pairwise token relationships.
To address the challenge of comprehensively modeling the pairwise distances between tokens, more flexible positional encoding methods have been proposed (Shaw et al., ; Su et al., ; Golovneva et al., ). For example, relative positional encodings offer an approach that allows the model to capture interactions between tokens more effectively, regardless of their placement in the sequence (Su et al., ). While this method has shown promise in natural language processing tasks, it remains unexplored in vision-language models. This is not surprising, as extending the context length in vision-language models by switching to more flexible positional encodings is computationally costly because of its significant retraining efforts.

Our proposed method upgrades the context window size of CLIP-like models by utilizing relative positional encodings, which are not restricted to a fixed length. Changing the model in this manner normally requires expensive multi-modal retraining. Instead, we propose an adaptation phase to first distill the knowledge of the original CLIP into the new model with relative positional encodings using only caption data.
Such a distillation approach is flexible and can be applied to CLIP-like models constrained by the standard 77-token window, transforming it into a model capable of handling longer captions.
Afterwards, we perform full fine-tuning on the distilled model for a single epoch on 1 million image-caption pairs (Chen et al., ), to further improve the alignment between images and long captions.
After these phases, our TULIP model can ingest captions longer than the usual 77 tokens and we observe improved performance on tasks ranging from cross-modal retrieval to text-to-image generation, compared to (interpolated) fixed token-window baselines.

In addition to our new method, we introduce a new benchmark for long captions adapted from the recently introduced Dense Captioning Images (DCI) dataset (Urbanek et al., ). This benchmark overcomes the limitations of existing retrieval benchmarks which lack diversity, as they focus on specific scenes (e.g., Urban-1K (Zhang et al., )), or are in-distribution datasets with already saturated performance (e.g., ShareGPT4v test set (Chen et al., )). Our results show that evaluating in a true long caption setting is crucial for evaluating contrastive vision-language mode as they unearth an increased performance gap as compared to prior benchmarks.

In summary, our contributions are:
(1) We propose TULIP, the first contrastive vision-language model with relative positional encoding for long captions. (2) We propose a general training procedure to adapt the positional encoding of CLIP-like models using a two-step adaptation encompassing relative position distillation and expansion. (3) We demonstrate improved performance across different cross-modal retrieval and image generation tasks. We introduce a new benchmark Long-DCI for a more comprehensive evaluation of long-caption retrieval tasks.

## 2 Related Work

Position Encodings in Transformer Models.
Transformers, the foundational architecture for many vision-language models, rely on positional encodings to compensate for the lack of inherent positional awareness in their set-based representation.
In the natural language domain, absolute positional encodings (Vaswani et al., ) were initially proposed, where fixed embeddings are added to token embeddings based on their position in the sentence sequence. However, as models have grown larger and more complex, alternative approaches emerged such as relative positional encodings (Shaw et al., ; Press et al., ), randomized positional encodings (Ruoss et al., ), extrapolation techniques (Press et al., ) and positional interpolation (Chen et al., ).
Other works, such as Rotary Position Embedding (RoPE) (Su et al., ) and its variations (Chen et al., ; Peng et al., ), apply relative positional encodings without any modifications to the self-attention mechanism, making it computationally efficient. Another recent approach is Contextual Position Encodings (CoPE) (Golovneva et al., ), which is a more general position encoding technique enabling one to attend to the $i$-th particular word, noun, or even sentence.
In contrastive vision-language models, integrating positional information effectively across modalities remains an unsolved challenge and this is the primary focus of this paper.

Contrastive Vision-Language Models with Long Captions.
Contrastive Vision-language models have made significant strides in aligning visual and textual modalities, driven by the success of large-scale pre-trained models such as CLIP (Radford et al., ), ALIGN (Jia et al., ), BLIP (Li et al., ) and many others (Garg et al., ; Vasu et al., ; ; Cherti et al., ; Sun et al., ). These models leverage contrastive learning techniques to align image and text representations in a shared feature space, enabling robust performance on tasks such as image-text retrieval and zero-shot classification.
However, all these models focus on short, global textual descriptions, often limited by small context windows, such as the 77-token limit in CLIP. Recent research has started considering this limitation, for instance, DCI (Urbanek et al., ) highlights the problem by pointing out that CLIP’s 77-token limit restricts the model from accommodating detailed, dense captions. They also introduce a human-annotated dataset containing dense image descriptions averaging above 1000 words each. Similarly, another work introduces DOCCI Onoe et al. (), a captioning dataset that provides detailed image descriptions that capture
key challenges such as spatial relations, counting, text rendering, and world knowledge. DreamLIP (Zheng et al., ) studies the usage of synthetically generated long captions under the contrastive learning framework but uses subsampled short captions from longer captions instead of processing the complete long captions.
Long-CLIP (Zhang et al., ) engages the challenge more directly, by stretching absolute position encodings through interpolation to enable fine-tuning on long captions. However, interpolation only partially addresses the limitations of absolute encodings when processing longer, complex captions. This is because interpolation merely extends the existing positional information without fundamentally altering its nature or capabilities. These limitations include diminished ability to capture fine-grained relative positions and poor generalization to longer captions (Pawar et al., ). We propose an alternative approach incorporating relative encodings without training from scratch, thereby enabling more effective processing of long captions and the comprehension of the pairwise token relationships therein.

## 3 TULIP

In this section, we introduce our method Token-Length Upgraded CLIP (TULIP). We start by stating the problem setting, followed by introducing the positional encoding swapping and the two-step adaptation procedure: (i) relative position distillation and (ii) relative position expansion.

### 3.1 Problem Statement

Let model $f$ be a contrastive vision-language model, such as CLIP, designed to align text and images in a shared embedding space. The text encoder of $f$, denoted as $f_{T}$ is constrained to processing sequences up to a predefined number of 77 tokens, denoted by $T_{f}=77$.
This is due to the fixed absolute positional encodings $P_{f}\in\mathbb{R}^{77\times d}$, where $d$ is the dimensionality of the embeddings. Given an input sequence $x=[x_{1},x_{2},\dots,x_{n}]$, where $n>T_{f}$, the model truncates $x$ to $x^{\prime}=[x_{1},\dots,x_{T_{f}}]$ losing critical information from the sequence beyond the first 77 tokens.

Our objective is to transform model $f$ into model $g$, without any re-training from scratch, to enable processing sequences of arbitrary length $T_{g}$. In this new model $g$, the token-length constraint of $f_{T}$ is removed, allowing the model to handle inputs of length $T_{g}>77$.

### 3.2 Positional Encoding Swapping

In $f_{T}$, the positional encoding function $P_{f}(i)$ maps each token position $i$ to a vector in $\mathbb{R}^{d}$, within a window of size 77. To overcome this limitation, we redefine the positional encoding function for model $g$, denoted as $P_{g}(i)$ which scales with input length $T_{g}$. This new function allows the model $g$ to process captions of arbitrary length without truncation.

Figure: Figure 1: Swapping the Positional Encoding. We update CLIP models by replacing the absolute positional encoding with relative positional encoding in each transformer block. This modification allows for long caption understanding and better modeling of pairwise token dependencies.
Refer to caption: extracted/5923682/assets/absolute_relative_comparison.png

We implement $P_{g}(i)$ as Rotary Positional encodings (RoPE) (Su et al., ) presented in Figure 1. Unlike traditional absolute positional encodings, where each position in a sequence is assigned a fixed vector, RoPE rotates the embeddings based on the relative distance between tokens.
More specifically, we alter the calculation of the attention weights within the self-attention layers of the text encoder $f_{T}$, which are initially calculated as $\mathrm{softmax}(\frac{q_{m}^{T}k_{n}}{\sqrt{d}})$, where:

$$ $q_{m}=W_{q}x_{m},k_{n}=W_{k}x_{n},$ (1) $$

representing the query and key vectors respectively for the $m$-th and $n$-th tokens in the sequence $x$, and $W_{q},W_{k}$ are their learned projection matrices.
With RoPE, we inject the position information of each token $x_{i}$ into the $q_{m}$ and $k_{n}$ vectors by:

$$ $q_{m}=R_{\Theta,m}W_{q}x_{m},\quad k_{n}=R_{\Theta,n}W_{k}x_{n},$ (2) $$

where $R_{\Theta,m}$ and $R_{\Theta,n}$ represent rotation matrices and $\Theta$ is the rotational frequency associated with the $j$-th dimension of the embedding. The rotational frequency ensures that different dimensions of the token embeddings rotate differently, embedding both absolute and relative positional information into the self-attention mechanism.

### 3.3 Relative Position Distillation

Once the architectural changes are in place, the next challenge is to adapt the text encoder of model $g$ to handle both short and long text inputs while retaining the image-text alignment capabilities of model $f$. We achieve this through knowledge distillation, shown with the first block of Figure 2, where $f_{T}$ acts as the teacher and the text encoder with the new relative encodings as the student $s_{T}$. A key advantage of this approach is that it eliminates the need to retrain the model from scratch despite the introduction of new relative positional encodings. Instead, we distill the knowledge from $f_{T}$, transferring its capabilities to the student model without losing the original alignment performance. This method is not only efficient but also generalizable, making it applicable to any text encoder that requires adaptation to new positional encodings.

Let $x=[x_{1},x_{2},\dots,x_{n}]$ represent an input caption, where $n\leq T_{f}$. Both the teacher model $f_{T}$ and the student model $s_{T}$ encode this sentence into a shared embedding space. Specifically, the teacher model $f_{T}$ aggregates the sequence into a special text token and yields the output embeddings $z_{f_{T}}=f_{T}(x)$. Similarly, the student model yields output embeddings $z_{s_{T}}=s_{T}(x)$.
The distillation loss is formulated as a cosine similarity between $z_{f_{T}}$ and $z_{s_{T}}$, aiming to maximize their alignment:

$$ $\mathcal{L}_{\mathrm{distill}}=\frac{z_{f_{T}}\cdot z_{s_{T}}}{\|z_{f_{T}}\|\| z_{s_{T}}\|}.$ (3) $$

Figure: Figure 2: TULIP training procedure. First, we perform relative position adaptation by distilling the knowledge of the CLIP text encoder into a student text encoder initialized with relative position encodings. This stage uses the first 77 tokens of a long caption (the gray block). The second stage is the relative position expansion, where we fine-tune the distilled text encoder with captions longer than 77 tokens (the combined gray and yellow blocks), together with the vision encoder.
Refer to caption: extracted/5923682/assets/framework.png

This loss function ensures that the student model $s_{T}$ learns from the teacher model $f_{T}$, by leveraging the original model’s ability to align textual and visual information. After the distillation, the student model $s_{T}$ has the capabilities of the teacher model for encoding up to 77 tokens with relative positional encodings. The only missing feature is the ability to process longer context which is addressed next.

### 3.4 Relative Position Expansion

Finally, we expand the context length of model $g$ beyond the original 77-token limit by fine-tuning it with longer captions, shown in the second block of Figure 2.
We start by copying the student model weights into the text encoder of $g$. This is followed by employing Neural Tangent Kernel (NTK)-aware scaled RoPE (bloc97, ), a refined version that adapts to the changing length of the input. In particular, we aim to scale the rotational frequency $\Theta$ by a factor $(\alpha*\frac{T_{g}}{T_{f}})-(\alpha-1)$, where $\alpha$ is a hyperparameter, to accommodate the training of the new higher token positions. The objective behind such scaling is to resolve the problem of losing high-frequency information when interpolating the RoPE embeddings to the higher token positions.
In model $g$, the new positional encodings now support input sentences $x_{g}$ of arbitrary length, $T_{g}>T_{f}$.
We proceed with fine-tuning using the usual contrastive loss as follows: $\mathcal{L}_{\text{}}(x,y)=-\log\frac{\exp(\cos(z_{x},z_{y})/\tau)}{\sum_{y^{
\prime}}\exp(\cos(z_{x},z_{y^{\prime}})/\tau)}$, where $x$ is the sentence, $y$ is the image and $\tau$ is the temperature hyperparameter.
However, to keep the original capability for handling short sentences as well, we jointly optimize two contrastive loss terms for both long and short sentences, as follows:

$$ $\mathcal{L}_{\text{total}}(x_{T_{g}},x_{T_{f}},y)=\lambda\times\mathcal{L}_{ \text{short}}(x_{T_{f}},y)+(1-\lambda)\times\mathcal{L}_{\text{long}}(x_{T_{g} },y).$ (4) $$

The first loss term, $\mathcal{L}_{\text{short}}$, is designed to preserve the model’s ability to handle short captions, aligning with the capabilities of model $f$. The other term, $\mathcal{L}_{\text{long}}$, is introduced to ensure that model $g$ can process longer captions effectively.

By extending the context length in this manner, model $g$ becomes capable of handling both short and long captions, making it more versatile for a wide range of vision-language tasks as we will demonstrate in the experiments.

## 4 Experiments & Results

**Table 1: Long caption cross-modal retrieval comparison on Long-DCI, ShareGPT4V and Urban-1K. TULIP consistently outperforms other CLIP variants across all evaluated datasets and tasks. Note that we adopt the results for CLIP and Long-CLIP from Zhang et al. (), while we fine-tune CLIP (Fine-tuned CLIP) on ShareGPT4V ourselves.**
|  | Long-DCI | ShareGPT4V | Urban-1K |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Img2Txt | Txt2Img | Img2Txt | Txt2Img | Img2Txt | Txt2Img |  |
| ViT-B-16 | CLIP | 35.9 | 33.7 | 78.2 | 79.6 | 68.1 | 53.6 |
| Fine-tuned CLIP | 46.3 | 45.4 | 94.1 | 93.6 | 80.4 | 79.8 |  |
| Long-CLIP | 42.1 | 48.4 | 94.6 | 93.3 | 78.9 | 79.5 |  |
| TULIP (Ours) | 50.2 | 50.6 | 98.6 | 98.6 | 88.1 | 86.6 |  |
| ViT-L-14 | CLIP | 35.0 | 37.0 | 81.8 | 84.0 | 68.7 | 52.8 |
| Fine-tuned CLIP | 51.6 | 50.7 | 95.3 | 95.4 | 78.0 | 76.5 |  |
| Long-CLIP | 54.0 | 46.1 | 95.8 | 95.6 | 82.7 | 86.1 |  |
| TULIP (Ours) | 55.7 | 56.4 | 99.0 | 99.0 | 90.1 | 91.1 |  |

Datasets and downstream tasks. We evaluate TULIP on three downstream tasks: short caption cross-modal retrieval, long caption cross-modal retrieval, and text-to-image generation.
For short caption cross-modal retrieval, we follow Zhang et al. () and evaluate our model on the COCO2017 5k validation set (Lin et al., ) and the full Flickr30k dataset (Plummer et al., ). Similarly, for long caption cross-modal retrieval, we use two datasets namely ShareGPT4V test split and Urban-1K. Both datasets contain 1,000 image-caption pairs, collected by Zhang et al. () and recaptioned using the ShareGPT4V captioning model.

It is important to recognize that long-caption cross-modal retrieval benchmarks come with several limitations. These include a lack of diversity, as some focus on narrowly defined scenes (e.g., Urban-1K), while others consist of in-distribution datasets where performance is already saturated (e.g., ShareGPT4V). To overcome these challenges, we introduce a new benchmark for long captions adapted from the recently introduced Dense Captioning Images (DCI) dataset (Urbanek et al., ). Long-DCI includes 7,000 human-annotated images and long caption pairs, with captions averaging 200 tokens per image. This human-led annotation process ensures diverse and accurate descriptions, avoiding the biases inherent in AI-generated captions like those from ShareGPT4V.

Evaluation metrics.
We report image-to-text and text-to-image retrieval performance using recall as the standard evaluation metric. Our evaluation process is the same for all experiments, including how we handle long input tokens. For each dataset, we choose the best design and settings using a validation set, and then report the final results on the test set.

Training details. Our training procedure comprises two phases: relative position distillation and relative position expansion, both utilize the ShareGPT4V dataset (Chen et al., ) containing 1M image and long caption pairs. During the relative position distillation phase, we truncate captions to the first 77 tokens for both the teacher and student models. We train the student model using cosine loss as the distillation loss function for 20 epochs with a batch size of 640 using the AdamW optimizer (Loshchilov, ), setting the learning rate to 5e-4 with 1000 warmup steps. In the relative position expansion phase, we employ full-length captions without truncation, exposing the model to comprehensive textual details.
The full TULIP model, featuring the new distilled text encoder, is fine-tuned using the NTK approach, with $\alpha$ empirically set to 8.0. For all main experiments, we use 248 number of tokens to match Long-CLIP’s context length for a fair comparison. Note that we can increase the length to more tokens, as shown in our ablations. We perform this fine-tuning stage for a single epoch with a batch size of 1280, a learning rate of 1e-5, and 1000 warmup steps using AdamW. We base our implementations on OpenAI’s pre-trained CLIP-ViT-B-16 and CLIP-ViT-L-14 architectures (Ilharco et al., ). Our code will be made publicly available.

**Table 2: Short caption cross-modal retrieval comparison on COCO and Flickr30k. TULIP shows competitive performance, often matching or exceeding Long-CLIP across different metrics and backbones.**
|  | COCO | Flickr30k |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Img2Txt | Txt2Img | Img2Txt | Txt2Img |  |  |  |  |  |
|  | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 |  |
| ViT-B-16 | CLIP | 51.8 | 76.8 | 32.7 | 57.7 | 44.1 | 68.2 | 24.7 | 45.1 |
| Fine-tuned CLIP | 37.4 | 62.3 | 21.8 | 43.4 | 25.7 | 45.8 | 17.9 | 34.5 |  |
| Long-CLIP | 57.6 | 81.1 | 40.4 | 65.8 | 46.8 | 71.4 | 34.1 | 56.3 |  |
| TULIP (Ours) | 56.8 | 80.3 | 40.7 | 66.1 | 46.1 | 70.8 | 35.2 | 57.4 |  |
| ViT-L-14 | CLIP | 56.1 | 79.5 | 35.4 | 60.1 | 48.5 | 72.6 | 28.0 | 49.3 |
| Fine-tuned CLIP | 37.9 | 63.1 | 23.1 | 45.1 | 26.0 | 46.3 | 17.9 | 34.9 |  |
| Long-CLIP | 62.8 | 85.1 | 46.3 | 70.8 | 53.4 | 77.5 | 41.2 | 64.1 |  |
| TULIP (Ours) | 62.6 | 84.7 | 46.1 | 71.1 | 56.7 | 79.5 | 41.6 | 64.3 |  |

### 4.1 Cross-modal Retrieval Comparison

We evaluate TULIP against the original CLIP, fine-tuned CLIP on ShareGPT4V (Chen et al., ), and Long-CLIP Zhang et al. () for both long-caption (Table 1) and short-caption (Table 2) cross-modal retrieval tasks.
As shown in Table 1, our proposed model outperforms all benchmarks in long-caption cross-modal retrieval across 3 datasets on both image-to-text and text-to-image retrieval, utilizing two different vision backbones: ViT-B-16 and ViT-L-14 (Dosovitskiy et al., ).
On the Long-DCI dataset, which is a more challenging benchmark for all approaches, we can observe that Long-CLIP already shows improvement over the original CLIP and that our TULIP model further improves on this. These results demonstrate the efficacy of TULIP in enhancing CLIP’s capabilities for long-caption cross-modal retrieval, particularly in diverse scenarios.
For short captions, as shown in Table 2, we find that the tailored approach used by Long-CLIP for the first 20 tokens is beneficial for the short caption performance as they outperform CLIP even on short captions. Meanwhile, TULIP can obtain competitive performance without needing to tailor specifically to the first 20 tokens, which demonstrates the flexibility of relative positional encodings across different caption lengths. Overall, the results in Table1 and 2 indicate that TULIP is effective not only for long captions but also for maintaining competitive performance in short caption retrieval scenarios.

### 4.2 Text-to-Image Generation

In this section, we evaluate qualitatively how our proposed procedure enhances text-to-image generation by simply replacing the original CLIP ViT-L-14 text encoder with our TULIP model, in particular using the text encoder. We use Stable Diffusion XL (SDXL) (Podell et al., ) as the image generation backbone. Note that do not perform any additional training of the diffusion model.
As observed in Figure 3, TULIP demonstrates improvements in both long and short caption understanding and modeling of nuanced details.
For example, our version of SDXL + TULIP accurately depicts
“the red tulip inside a wooden box” in the first example. This is not the case with SDXL-CLIP or SDXL-Long-CLIP which generate many tulips in the garden, missing the key detail of a single tulip being inside the box. This observation shows that our model can indeed capture finer details in the description.
The second example shows that our model successfully generates “an old man sits on a rock” even when this detail appears beyond the 77-token limit. On the other hand, both base CLIP and Long-CLIP encoders fail at this. A similar observation is in the third example where the model correctly captures ‘people walking along the streets”.
These examples are direct evidence that our approach can truly capture the meaning of words in longer captions.
For shorter prompts, our model also shows enhanced caption understanding.
For instance, it accurately represents the ferret with a “playful grin” demonstrating the enhanced association of attributes with their corresponding nouns. Even more impressively, it accurately places the ferret inside a “large glass jar”, showing correct spatial understanding - something both CLIP and Long-CLIP fail to achieve. These observations suggest that our TULIP text encoder enhances long caption comprehension, resulting in more accurate and contextually rich image generation across varying prompt lengths. We provide additional results in the appendix.

Figure: Figure 3: Text-to-Image Generation results. We replace the text encoder of SDXL with our own TULIP model. We observe improvements in both long captions understanding and capturing nuanced details, compared to the baselines CLIP-ViT-L-14 and Long-CLIP. Note that the // marks the 77-token boundary in the caption. The words in green indicate visual concepts that are correctly generated by TULIP and are missed by the baselines.
Refer to caption: extracted/5923682/assets/image_generation1.png

### 4.3 Ablation Study

In this section, we carefully analyze various design choices for our proposed method. We examine different positional encoding schemes, investigate the impact of context length, and explore variations in the distillation loss function.

Different types of Relative Positional Encodings
In this experiment, we compare different types of relative positional encodings. We choose a recently introduced Contextual Position Encoding (CoPE) (Golovneva et al., ) for implementing the relative positional encodings in the text encoder. Afterwards, we perform the distillation and context length extension phases using the same dataset and parameters as with RoPE (Su et al., ). In Table 3 we observe that RoPE outperforms CoPE in long-caption retrieval tasks. This is due to RoPE’s strong ability to generalize across varying or extended sentence lengths, even beyond those on which the model was originally trained. In contrast, CoPE struggles to generalize as effectively when sequence lengths increase, as its embeddings are more dependent on the specific context within which they were trained. This explains why CoPE performs similarly to RoPE on the ShareGPT4V test split (same training distribution), but shows a larger performance gap on the Long-DCI and Urban-1k datasets.

The impact of the caption length.
Next, we evaluate the impact of varying context lengths on the performance of RoPE in long-caption image retrieval tasks. To investigate this, we fine-tune only the text encoder during the context length extension phase with different context sizes (which are $n\times 77$): $\{77,154,231,308\}$ tokens while keeping the image encoder frozen. We deliberately freeze the image encoder to isolate the text encoder’s performance, as unfreezing it could potentially mask the text encoder’s limitations in processing longer inputs. Figure 4 presents the model’s performance across three datasets (ShareGPT4V, Urban-1K, and Long-DCI) for cross-modal retrieval tasks. We observe general improvement in performance with increased context length, particularly from 77 to 154 tokens, across all datasets. This improvement is more pronounced for image-to-text retrieval, suggesting that longer contexts enhance text representation and enable more precise alignment with image features. However, we observe a slight performance plateau or minor decline for 308-length sentences, signaling a point of diminishing returns where additional tokens may introduce noise or redundancy. This plateau is likely due to the average caption length in our training data being 174.02 tokens, explaining why performance levels off between 154 and 231 tokens.

**Table 3: Ablation comparing different relative position encodings: RoPE and CoPE in TULIP. RoPE generalizes better across varying or extended sentence lengths, especially on out-of-distribution datasets, namely Long-DCI and Urban-1k.**
|  | Long-DCI | ShareGPT4V | Urban-1K |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Positional encodings | Img2Txt | Txt2Img | Img2Txt | Txt2Img | Img2Txt | Txt2Img |
| CoPE | 50.8 | 49.9 | 98.5 | 97.8 | 86.7 | 82.8 |
| RoPE | 55.7 | 56.4 | 99.0 | 99.0 | 90.1 | 91.1 |

Figure: Figure 4: Impact of the sequence length on cross-modal retrieval tasks. We observe general improvement in performance with increased sequence length, particularly from 77 to 154 tokens, across all datasets and tasks.
Refer to caption: extracted/5923682/assets/context_length.png

Benefit of using cosine distillation loss.
In this section, we vary the distillation loss function used during the relative position adaptation phase and report the results in Table 4. As seen, the cosine loss performs better than other alternatives across different datasets and tasks. We believe this is attributed to its alignment with the normalized embedding space of CLIP models and its scale invariance property. The scale invariance of cosine loss proves particularly beneficial in this distillation setup, where the student model (ViT-L-14 with the added RoPE or CoPE layer) can produce embeddings with different magnitudes compared to the teacher model. This invariance allows the distillation process to focus on transferring the essential directional information of the embeddings, rather than being influenced by potential scale discrepancies introduced by the RoPE or CoPE layer. As a result, cosine loss consistently outperforms other loss functions like L2 and MSE across various datasets (Long-DCI, ShareGPT4V, Urban-1K) and cross-modal tasks such as image-to-text and text-to-image, demonstrating its effectiveness in preserving the semantic relationships learned by the teacher model.

**Table 4: Ablation comparing different distillation loss terms in TULIP. Cosine loss yields the best performance across different datasets and tasks.**
|  | Long-DCI | ShareGPT4V | Urban-1K |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Distillation loss | Img2Txt | Txt2Img | Img2Txt | Txt2Img | Img2Txt | Txt2Img |
| CLIP | 35.0 | 37.0 | 81.8 | 84.0 | 68.7 | 52.8 |
| L2 | 39.4 | 35.8 | 84.8 | 83.9 | 70.2 | 56.3 |
| MSE | 36.4 | 31.8 | 83.0 | 81.3 | 74.0 | 53.9 |
| Cosine | 38.5 | 35.8 | 84.8 | 84.2 | 73.6 | 56.6 |

### 4.4 Additional Analysis

Attention spread visualization. The effectiveness of our model in long cross-modal retrieval tasks and image generation is largely due to its distinct attention distribution patterns. To illustrate this, we visualize the attention scores between the CLS text token and its preceding tokens in the final attention block of the text encoder, as shown in Figure 5. For comparison, we provided attention visualizations for both our TULIP and LongCLIP.
This analysis reveals two key advantages of our model when processing a 248-token caption. First, our model exhibits a more uniform distribution of attention across input tokens, demonstrating its ability to expand the attention scope and effectively aggregate information from a broader range of tokens. This expanded attention field improves the model’s performance in long-caption tasks by capturing and utilizing details from later parts of the caption that other models might overlook. Second, our model shows increased attention to punctuation symbols, particularly commas, which enhances its ability to parse and segment longer texts. Such capability is crucial for better comprehension of complex, multi-caption descriptions.

Caption-image relevance distribution analysis. This analysis investigates where relevant information is distributed within long captions given an image. The aim is to highlight how useful content is spread throughout the whole caption. We analyze 100 randomly selected images with long captions from the ShareGPT4V dataset. For each image-caption pair, we computed the visual embeddings of the image and compared them to the text embeddings of the caption subwindow. We define sliding subwindows of sizes 20, 33, and 55 tokens, moving from left to right with strides of 5, 10, and 15 tokens respectively. We then calculated the cosine similarity between the image encoding and the text encodings of each subwindow, visualized in Figure 6. The figure shows that the similarity scores are distributed across different subwindows, emphasizing the need for models capable of processing longer input sequences. Notably, as window size increases, the similarity patterns become more concentrated and pronounced, suggesting that larger context windows capture more cohesive and relevant information. Additionally, the variability in similarity across different windows highlights the non-uniform distribution of image-relevant information throughout the captions. This reinforces the need to leverage the entire textual sequence when learning image-text representations.

Figure: (a) LongCLIP
Refer to caption: extracted/5923682/assets/longclip_sa_187007.png

Figure: Figure 6: Caption-image relevance distribution analysis across varying window sizes and positions. it shows that image-relevant information is spread throughout captions, emphasizing the need for models to process longer text inputs to capture all pertinent details.
Refer to caption: x1.png

## 5 Conclusion

In this work, we addressed the limitations of CLIP-like models in handling long input sequences. We introduce TULIP, a generalizable method that upgrades the context length beyond the 77-token limit. By leveraging relative positional encodings, our approach enables the effective modeling of pairwise token relationships. Through a two-step training process, we successfully adapt the CLIP-like model to process longer captions without compromising its performance on shorter inputs. Our experiments demonstrate that TULIP considerably improves long-caption performance on cross-modal tasks such as retrieval and text-to-image generation, setting a new standard for vision-language contrastive models that require handling complex, extended text descriptions.

Limitations. Our reliance on ShareGPT4V dataset, synthesized by GPT4V (Achiam et al., ), limits TULIP’s performance to the quality of GPT4V’s long captions. Furthermore, while TULIP is theoretically capable of handling longer contexts due to the nature of the relative positional encodings, its token length is constrained by the average token length of the ShareGPT4V captions.

## Appendix A Appendix

### A.1 Additional text-to-image generation results

Figure: Figure 7: Text-to-Image Generation results.
Refer to caption: extracted/5923682/assets/image_generation_appendix_1.png

Figure: Figure 8: Text-to-Image Generation results.
Refer to caption: extracted/5923682/assets/image_generation_appendix_2.png

Figure: Figure 9: Text-to-Image Generation results.
Refer to caption: extracted/5923682/assets/image_generation_appendix_3.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: None
soundness: 3
strengths: 1. Although the techniques themselves (RoPE and CoPE) are not novel, the paper demonstrates creativity in combining these methods to tackle the limitations of existing models like CLIP. This approach leverages the strengths of both RoPE and CoPE to address the shortcomings of absolute positional encodings, thereby providing a more dynamic and effective way to capture positional relationships in long sequences.

2. The paper is well-written and clearly presented. The authors effectively communicate their ideas and methodologies, making it accessible to readers with a background in machine learning and natural language processing. The use of figures, tables, and examples to illustrate key points is helpful in conveying the technical details of the proposed approach.
summary: The paper addresses the challenge of integrating positional information effectively in contrastive vision-language models, particularly when dealing with long captions. Traditional models like CLIP are limited by short context windows, which restricts their ability to process detailed and dense textual descriptions. The authors propose an approach that combines Rotary Position Embedding (RoPE) and Contextual Position Encodings (CoPE) to better handle long captions without the need for training from scratch. This method aims to enhance the model's ability to comprehend pairwise token relationships and capture fine-grained relative positions in longer, more complex captions. The paper builds upon existing work such as Long-CLIP and proposes a refined technique to address the limitations of absolute positional encodings through interpolation.
weaknesses: 1. The proposed method appears to be an incremental improvement based on existing models like Long-CLIP. The combination of RoPE and CoPE, while potentially effective, does not introduce fundamentally new concepts. Both RoPE and CoPE are not novel and have been introduced in prior works. The primary contribution seems to be the application of these methods to the specific task of processing long captions, which may not be sufficient to claim significant novelty.

2. The paper does not adequately compare its proposed method with other approaches [1] that handle long text descriptions, such as those based on the encoder of T5 model [2]. T5-based models are known for their capability to process long texts effectively. The authors need to clearly articulate the advantages of their approach over T5-based methods and provide empirical evidence to support these claims. Without this comparison, it is challenging to assess the relative effectiveness of the proposed method.

3. The paper lacks a deep theoretical analysis of why the combination of RoPE and CoPE is expected to perform better for long captions. A more detailed theoretical justification or analysis could strengthen the paper by providing a solid foundation for the proposed method's expected performance improvements.

[1] Chang, Huiwen, et al. "Muse: Text-to-image generation via masked generative transformers." arXiv preprint arXiv:2301.00704 (2023).
[2] Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." Journal of machine learning research 21.140 (2020): 1-67.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1) How does the new model affect the compositionality of text inputs as dicussed here? https://aclanthology.org/2023.emnlp-main.301.pdf
2) What do you attribute the improved performance in a shorter context to? In Table 2, TULIP performs a lot better than CLIP.
3) Why are the values in Table 4 lower than those in Tables 1 or 3? If I missed something, can you clarify the caption to avoid confusion?
soundness: 3
strengths: - Significant improvements across cross-modal retrieval and text-to-image generation
- The introduction of a benchmark for long captions is a step towards better evaluation of other research works in this domain
- The innovative approach with relative captioning and distillation has shown improved performance
- The paper writing is easy to follow
summary: The TULIP model enhances CLIP's capability to process long captions by replacing fixed positional embeddings with relative positional encodings, extending the model's inherent context window beyond the 77 token limit. This approach is shown to improve performance on cross modal retrieval and text to image generation tasks. The TULIP training method is 2 stage - relative position distillation from the original CLIP model and finetuning to handle longer captions. Benchmark tests demonstrate TULIP's superior performance in long-caption scenarios. The paper also introduces a new evaluation benchmark, Long-DCI, to better assess long-caption retrieval tasks.
weaknesses: - Switching from absolute to relative positional encodings does improve flexibility with token length but can also introduce challenges in retaining fine-grained positional relationships in shorter contexts
- No specific details about the human annotators
- Needed a more qualitative comparison of how the new approach led to better performance in shorter context
- While CLIP possesses excellent zero-shot capabilities, it suffers from certain limitations in perceptual understanding. A discussion on TULIPs limitations in perceptual understanding would boost the presentation of the proposed approach

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: I would be grateful if, in addition to the points I raised above, the authors could elaborate on the amount of data with long context is needed to train the student model with TULIP, in order for it to perform well on long contexts.
soundness: 3
strengths: - The paper examines the quality of CLIP models in the setting of long text descriptions of images. This setting is of interest to the community - as the context of models grows, the ability to retain information from the entire caption decreases as well, and trying to remedy this is an interesting problem.

- The paper is also clear and easy to understand. I did not have any issues understanding the method presented, or any other part of the paper.

- I also believe that the inclusion of the Long-DCI evaluation dataset is an important contribution to the community. Having benchmark specifically for images with long context captions is extremely important in the advancement of this field.
summary: The authors propose TULIP, a method that allows for extending the capabilities of CLIP beyond the 77 token limit that is often used when training such models. The authors demonstrate that their method improves performance of CLIP trained models, on both traditional image-text retrieval tasks, as well as long-context tasks (one of which is proposed by the authors themselves).
weaknesses: - While the method itself appears useful, I am slightly worried about its novelty. At the end of the day, the model still requires training with longer contexts, so it is not immediately clear to me how different the method is from prior work. I believe that elaborating a bit more on which part of the method proposed by the authors provides the most benefit over prior work such as Long-CLIP would alleviate this concern.

- The paper has an important limitation, in that the setting considered is only CLIP-style models, and not autoregressively trained ones. The latters tend to have much larger context length than the ones considered in this paper. While I understand that direct comparison between the two may not be possible/ideal, having the method being applied to this category of models would greatly strengthen the paper.

- I would also be grateful if the authors could perform evaluation on a task based on the DOCCI dataset [A], which has the same stated goal as the proposed Long-DCI one.

- The experimental evaluation on image generation in Section 4.2 is qualitative rather than quantitative. I believe that the authors could greatly strengthen the conclusions of the paper by evaluating the generated images in a more principled manner (for example, by performing a human preference study on the generated images).

Reference:

[A] Onoe et al., "DOCCI: Descriptions of Connected and Contrasting Images", ECCV 2024

Meta-review: This paper presents TULIP, which aims to enhance CLIP's capability to process long captions by replacing fixed positional embeddings with relative positional encodings, extending the model's inherent context window beyond the 77 token limit. After rebuttal, it received scores of 668. All the reviewers are happy about the paper, commenting that (1) the paper is well-written and clearly presented; (2) though the proposed method is not entirely novel, the application of RoPE and CoPE to tackle the context-length limitation of CLIP is still very interesting and important; (3) results are convincing. Therefore, the AC would like to recommend acceptance of the paper.

Decision: Accept (Poster)