TULIP: Token-length Upgraded CLIP
Authors: Ivona Najdenkoska
Mohammad Mahdi Derakhshani
Yuki M Asano
Nanne Van Noord
Marcel Worring
Cees G. M. Snoek
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Models
CLIP
Position Encodings
Long captioning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a generalizable method, we call TULIP, able to update the token length to any length for CLIP-like models.

Abstract: We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip.

Published as a conference paper at ICLR 2025
TULIP: TOKEN-LENGTH UPGRADED CLIP
Ivona Najdenkoska"
Mohammad Mahdi Derakhshani Yuki M. Asano
Nanne van Noord University of Amsterdam Amsterdam, the Netherlands
Marcel Worring
Cees G. M. Snoek!
ABSTRACT
We address the challenge of representing long captions in vision-language mod- els, such as CLIP. By design these models are limited by fixed, absolute posi- tional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to fixed new to- ken length. Instead, we propose generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by training procedure that (i) distills the original CLIP text encoder into an en- coder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at attps::/githisn wtt.:/cc...aaa.e.e http://oooraeeeoooaaaa////..a
INTRODUCTION
Obtaining text representations for long captions in vision-language models is an open research chal- lenge. Within the context of text-only large language models, this problem has been studied ex-
tensively (Chung et al. 2024] Touvron et al. 2023 Bai et al. 2023 Gu & Dao 2023 Su et al.)
2024 Dubey al. 2024 Pawar et al. [202]] Jiang et al. 2024 Lieber et al. 2024). yet, methods domain. For instance, contrastive vision-lant laiiunla models like CLIP (Radford et al. 2021 Jia et al. 2021 Li
for context length expansion have only scarcely made their way into the
ctt al. 2022) are constrained to short input sequences, capped at 77 tokens. Natively, such models based on Transformers (Vaswani et al. 2017) process their input as an unordered set of tokens and positional encoding to preserve order information (Yun et al.) 2019). In particular, CLIP models use absolute positional encodings, which rely on a predefined maximum number of tokens, thereby limiting the input sequence to 77 tokens by design. We introduce an efficient and generalizable method called TULIP, that upgrades the context window size of CLIP-like models.
Recently, Long- CLIP (Zhang et al. 2024 highlighted this problem and proposed approach for unlocking the long-text capabilities of CLIP. However, their approach focuses on stretching the ex- isting absolute positional encodings, which primarily addresses the issue of the hard token limit but it continues to rely on absolute encodings which inhibit the model's 's comprehension of pairwise token relationships. To address the challenge of comprehensively modeling the pairwise distances between tokens, more flexible positional encoding methods have been proposed (Shaw et al.) 2018 Su et al. 2024 Golovneva et al., 2024). For example, relative positional encodings offer an ap- proach that allows the model to capture interactions between tokens more effectively, regardless of
*Shared first authorship. Corresponding authors: fi.najdenkoska, m.m m.m.derakhshami @uva.nl. The authors can change the order for their purposes. Shared last authorship; order random.

---

Published as a conference paper at ICLR 2025
their placement in the sequence (Su et al. 2024). While this method has shown promise in natural language processing tasks, it remains unexplored in ision-language models. This is not surprising, as extending the context length in vision-language models by switching more flexible positional encodings is computationally costly because of its significant retraining efforts.
Our proposed method upgrades the context window size of CLIP-like models by utilizing relative positional encodings, which are not restricted to a fixed length. Changing the model in this manner normally requires expensive multi-modal retraining. Instead, we propose an adaptation phase to first distill the knowledge of the original CLIP into the new model with relative positional encodings using only caption data. Such a distillation approach is flexible and can be applied to Cike-like models constrained by the standard 77-token window, transforming it into a model capable of han- dling longer captions. Afterwards, we perform full fine-tuning on the distilled model for a single epoch on 1 million image-caption pairs (Chen et al. (023b), to further improve the alignment be- tween images and long captions. After these phases, our TULIP model can ingest captions longer than the usual 77 tokens and we observe improved performance on tasks ranging from cross-modal retrieval to text-to-image generation, compared to (interpolated) fixed ooken-window baselines.
In addition to our new method, we introduce new benchmark for long captions adapted from the recently introduced Dense Captioning Images (DCT) dataset (Urbanek et al. 2024). This benchmark overcomes the limitations of existing retrieval benchmarks which lack diversity, as they focus on specific scenes (e.g.. Urban-1K (Zhang et al.] 2024)). or are in-distribution datasets with already saturated performance (e.g., ShareGiPT4v test set (Chen et al. (003b)). Our results show that evalu- ating in true long caption setting is crucial for evaluating contrastive vision- vision-language mode as they unearth an increased performance gap compared prior benchmarks.
In summary, our contributions are: (1) We propose TULIP, the first contrastive vision-language model with relative positional encodings for long captions. (2) We propose a general training proce- dure to adapt the positional encodings of CLIP-like models using a two-step adaptation encompass- ing relative position distillation and expansion.
(3) We demonstrate improved performance across different cross-modal retrieval and image generation tasks. We define a new benchmark Long-DCI for a more comprehensive evaluation of long-caption retrieval tasks.
2 RELATED WORK
Position Encodings in Transformer Models. Transformers, the foundational architecture for many Dision-language models, rely on positional encodings to compensate for the lack of inherent posi- tional awareness in their set-based representation. In the natural language domain, absolute posi- tional encodings (Vaswani et al. 2017) were initially proposed, where fixed embeddings are added to token embeddings based on their position in the sentence sequence. However, as models have grown larger and more complex, alternative approaches emerged such as relative positional encod- ings (Shaw et al. 2018 Press et al. 202T), randomized positional encodings (Ruoss el al. 2023), extrapolation techniques (Press al. 2022) and positional interpolation (Chen et al. (023cc). Other works, such as Rotary Position Embedding (RoPE) (Su et al.] 2024) and its variations (Chen et al. 2023c Peng al., 2023). apply relative positional encodings without any modifications to the self- attention mechanism, making it computationally efficient. Another recent approach is Contextual Position Encodings (CoPE) (Golovneva et al.,, (024)) which is a more general position encoding technique enabling one to attend to the i-th particular word, noun, or even sentence, In contrastive vasion-language models, integrating positional information effectively across modalities remains an unsolved challenge and this is the primary focus of this paper.
Contrastive LanguaLLag Models with Long Captions. Contrastive Dision-language models have made significant strides in aligning visual and textual modalities, driven by the success of large- scale pre-trained models such as CLIP (Radford et al.) 2021), ALIGN (Jia et al.
2021). BLIP (Li
et al. 2022) and many others (Garg et al. [023] Vasu et al. [2224b] a Cherti et al.
2023 Sun et al.;
2023). These models leverage contrastive learning techniques to align image and text representa- tions in a shared feature space, enabling robust performance on tasks such as image-text retrieval and zero-shot classification. However, all these models focus on short, global textual descriptions, often limited by small context windows, such as the 77-token limit in CLIP. Recent research has started considering this limitation, for instance, DCI (Urbanek et al. 2024) highlights the problem by pointing out that CLIP's 77-token limit restricts the model from accommodating detailed, dense
2

---

Published as a conference paper at ICLR 2025
captions. Similarly, another work introduces DOCCI OOnoe al. 2024 captioning dataset that provides detailed image descriptions that capture key challenges such as spatial relations, counting, text rendering and world knowledge. DreamLIP (Zheng et al., 2024) studies the usage of synthet- ically generated long captions under the contrastive learning framework but uses subsampled short captions from longer captions instead of processing the complete long captions. Long-CLIP (Zhang) et al. 2024) engages the challenge more directly, by stretching absolute position encodings through interpolation enable fine-tuning with long captions. However, interpolation only partially ad- dresses the limitations of absolute encodings when processing longer, complex captions. This is because interpolation merely extends the existing positional information without fundamentally al- tering its nature or capabilities. These limitations include diminished ability to capture fine-grained relative positions and poor generalization to longer captions (Pawar al. 2024). We propose an alternative approach that incorporates relative encodings without training from scratch, thereby en- abling more effective processing of long captions and the comprehension of the pairwise token relationships therein.
TULIP
In this section, we introduce our method Token-Length Upgraded CLIP (TULIP). We start by stating the problem setting, followed by introducing the positional encoding swapping and the two-step adaptation procedure: (i) relative position distillation and (ii) relative position expansion.
3.1 PROBLEM STATEMENT
Let model f be contrastive vasion-language model, such as CLIP, designed to align text and images in a shared embedding space. The text encoder of f. denoted as fr is constrained to processing sequences up to a predefined number of 77 tokens, denoted by T3 77. This is due to the fixed absolute positional encodings PO E IR?7xd, where dl is the dimensionality of the embeddings. Given an input sequence x [1, 42 xn), where n > T, the model truncates x to x' = [.1....... TTy] losing critical information from the sequence beyond the first 77 tokens.
Our objective is to transform model f into model g, without any re-training from scratch, to enable processing sequences of arbitrary length Tz. In this new model g, the token-length constraint of fr removed, allowing the model to handle inputs length T, > 77.
3.2 POSITIONAL ENCODING SWAPPING
In fr, the positional encoding function (i( maps each token position i to vector in Rd, within a window of size 77. To overcome this limitation, we redefine the positional encoding function for model denoted as (2(() which scales with input length Tg. This new function allows the model to process captions of arbitrary length without truncation.
We implement (i) as Rotary Positional encodings (RoPE) (Su et al. 2024) presented in Figure 1 Unlike traditional absolute positional encodings, where each position in sequence assigned fixed vector, RoPE rotates the embeddings based the relative distance between tokens. More specifically, we alter the calculation the attention weights within the self-attention layers of the
text encoder fr, which are initially calculated softmax( I Kn
), where:
Vd
Zm W Wy.mm R; WET..
representing the query and key vectors respectively for the m-th and n-th tokens in the sequence xr, and Wi W& are their learned projection matrices. With RoPE, we inject the position information of each token x; into the Im and k, vectors by:
In Re.m Wyrms hi = Re,n WA
where Re.m and Re,,, represent rotation matrices and θ is the rotational frequency associated with the j-th dimension of the embedding. The rotational frequency ensures that different dimensions of the token embeddings rotate differently, embedding both absolute and relative positional information into the self-attention mechanism.
3

---

Published as a conference paper at ICLR 2025
Transformer Block
layers
Transformer Block
Iayers
768
FEN Block
FFN Block
12.66
H heads
heads
77
Pos. 1
00.1
6:12
6:36
1.24
Multi--eed
Multi-head
attention
attention
+.72
Layerhoom
encoding layer
LayetNorm
enoodlng layer
T, >
ertbedding laye:
Tikes cinbealiing layer
Tokentoed bext
Tolemized text
Figure 1: Swapping the Positional Encoding. We update CLIP models by replacing the absolute positional encoding with relative positional encoding in each transformer block. This modification allows for long caption understanding and better modeling of pairwise token dependencies.
3.3 RELATIVE POSITION DISTILLATION
Once the architectural changes are in place, the next challenge is to adapt the text encoder of model g to handle both short and long text inputs while retaining the image-text alignment capabilities of model f. We achieve this through knowledge distillation, shown with the first block of Figure where fr acts the teacher and the text encoder with the relative encodings ST. A key advantage of this approach is that it eliminates the need to retrain the model from scratch despite the introduction of new relative positional encodings. Instead, we distill the knowledge from fr. transferring its capabilities to the student model without losing the original alignment performance. This method not only efficient but also generalizable, making it applicable any text encoder that requires udaptation to new positional encodings.
the student
Let [z .,,...,,,....................... xn] represent an input caption, where S Ty. Both the teacher model fr and the student model 8T encode this sentence into a shared embedding space. Specifically, the teacher model fr aggregates the sequence into special text token and yields the output embeddings fr fx). Similarly, the student model yields output embeddings žeg = sT(x). The distillation loss is formulated as a cosine similarity between &fz and Zaz, aiming to maximize their alignment:
2fr
Zer
Calstill
Eey
This loss function ensures that the student model sT learns from the teacher model fr, by leveraging the original model's ability to align image and text. After the distillation, the student model sT has the capabilities of the teacher model for encoding up 77 tokens with relative positional encodings. The only missing feature is the ability to process longer context, which is addressed next.
3.4 RELATIVE POSITION EXPANSION
Finally, we expand the context length of model g beyond the original 77-token limit by fine-tuning it with longer captions, shown in the second block of Figure [ We start by copying the student model weights into the text encoder of g. This is followed by employing Neural Tangent Kernel -aare scaled RoPE (bloc97) 2023). a refined version that adapts to the changing length of the input. In particular, we aim to scale the rotational frequency θ by a factor (a +)) (a 1), where a is a hyperparameter, to accommodate the training of the new higher token positions. The objective behind such scaling is to resolve the problem of losing high-freques frequency information when interpolating the RoPE embeddings to the higher token positions. In model g, the new positional encodings now support input sentences Ig of arbitrary length, T2 > Ty. We proceed with fine- tuning using the usual contrastive loss as follows: [(z, y) - log Σ,, (xxppccom.......)//5 where æ is the sentence, y the image and is the temperature hyperparameter. However, to keep the original
exp(ccn(2...,,,/7)
4

---

Published as a conference paper at ICLR 2025
Input caption
Relative Position Distilletion
Expension
The image captures vibrant soona the Elfll Tower Paris, Franco. The
teacher text enooder with atsolute PS
Input image
Fifel Tower,
world-renswmed
Vision encoder
landmark, stands tail the badkground, intricato irnn latiie strunuee Nataament to human inganulty. in the "oreground, camussel adds touch whiney to the scene The cerousel. painted cheerful shades orange while, toppod with red and while striped awringg inviing vasters take spin. The sky above
contaseitve matching
tokin matching
fr
91
long caption
daar true, providing beautful backdinp to the Adding dynamic element the imaga, and yellow The relative posttioss objects photograph, with the cerousel the foreground and the Eittal Tower background.
9T
well-
student text encoder with relative PE
000y
text ensoder intialisee with distilled student model
Figure 2: TULIP training procedure. First, we perform relative position adaptation by distilling the knowledge of the CLIP text encoder into student text encoder initialized with relative position encodings. This stage uses the first 77 tokens of long caption (the gray block). The second stage is the relative position expansion, where fine-tune the distilled text encoder with captions longer than 77 tokens (the combined gray and yellow blocks), together with the vision encoder.
capability for handling short sentences as well, we jointly optimize two contrastive loss terms for both long and short sentences, follows:
(4) The first loss term, Luhort. is designed to preserve the model's ability to handle short captions, align- ing with the capabilities model f. The other term, Liong, is introduced ensure that model g can process longer captions effectively.
Liaal (TT, 2T,, y) X X (Et + +(1 -1 x Liong(IT, V).
By extending the context length in this manner, model g becomes capable of handling both short and long captions, making it more versatile for wide range of vision-language tasks as we will demonstrate in the experiments.
EXPERIMENTS & RESULTS
Datasets and downstream tasks. We evaluate TULIP on three downstream tasks: short cap- tion cross-modal retrieval, long caption cross-modal retrieval, and text-to-image generation. For short caption cross-modal retrieval, we follow Zhang et al. (2024) and evaluate our model on COCO2017 5k validation set (Lin et al. 2014) and the full Flickr30k dataset (Plummer al.) 2015). Similarly, for long caption cross-modal retrieval, we use two datasets, namely ShareGPT4V test split and Urban- K. Both datasets contain 1,000 image-caption pairs, collected by Zhang et al. (2024) and recaptioned using the ShareGPT4V captioning model.
Long-DCI
ShareGPT4V
Urban-
Img2Txt
Txt2mmg
Img2Txt
Txt2Img
Img2Txt
Txt2Img
VIT-L-14 ViT-B-16
CLIP
35.9
33.7
78.2
79.6
68.1
53.6
Fine-tuned CLIP
46.3
45.4
94.1
93.6
80.4
79.8
Long-CLIt
42.1
48.4
94.6
93.3
78.9
79.5
TULIP (Ours)
50.2
50.6
98.6
98.6
88.1
86.6
CLIP
35.0
37.0
81.8
84.0
68.7
52.8
Fine-tuned CLIP
51.6
50.7
95.3
95.4
78.0
76.5
Long-CLIP
54.0
46.1
95.8
95.6
82.7
86.1
TULIP (Ours)
55.7
56.4
99.0
99.0
90.1
91.1
Table Long caption cross-modal retrieval comparison on Long-DCI, ShareGPT4V and Urban-
IK. TULIP consistently outperforms other CLIP variants across all evaluated datasets and tasks. Note that we adopt the results for CLIP and Long-CLIP from Zhang et al. (2024), while we fine- tune CLIP (Fine-tuned CLIP) ShareGPT4V ourselves.
5

---

Published as a conference paper at ICLR 2025
COCO
Flickr30k
Img2Txt
Txt2Img
Img2Txt
Tx12Img
R@D
R@5
R@ 1
R@5
R @
R@5
R @
R@5
VIT-L-14 ViT-B-16
CLIP
51.8
76.8
32.7
57.7
44.1
68.2
24.7
45.1
Fine-tuned CLIP
37.4
62.3
21.8
43,4
25.7
45.8
17.9
34.5
Long-CLIP
57.6
81.1
40.4
65.8
46.8
71.4
34.1
56.3
TULIP (Ours)
56.8
80.3
40.7
66.1
46.1
70.8
35,2
57.4
CLIP
56.1
79.5
35.4
60.1
48.5
72.6
28.0
49.3
Fine-tuned CLIP
37.9
63.1
23.1
45.1
26.0
46.3
17.9
34.9
Long-CLIP
62.8
85.1
46.3
70.8
53.4
77.5
41.2
64.1
TULIP (Ours)
62.6
84.7
46.1
71.1
56.7
79.5
41.6
64.3
Table 2: Short caption cross-modal retrieval comparison on COCO and Flickr30k. TULIP shows competitive performance, often matching or exceeding Long-CLIP across different metrics and model backbones.
important recognize that long-caption cross-modal retrieval benchmarks come with several limitations. These include a lack of diversity, as some focus on narrowly defined scenes (e.g., Urban- 1K). while others consist of mi-iistribution datasets where performance already saturated (e.g., ShareGPT4V). To overcome these challenges, we define new benchmark for long captions adapted from the recently introduced Dense Captioning Images (DCT) dataset (Urbanek et al. 2024). I Long- DCI includes 7,000 human-annotated images and long caption pairs, with captions averaging 200 tokens per image. This human-led annotation process ensures diverse and accurate descriptions, avoiding the biases inherent in Al-generated captions like those from ShareGPT4V.
Evaluation metrics. We report image-to-text and text-to-image retrieval performance using recall as the standard evaluation metric. Our evaluation process is the same for all experiments, including how we handle long input tokens. For each dataset, we choose the best design and settings using a validation set, and then report the final results the test set.
Training details. Our training procedure comprises two phases: relative position distillation and relative position expansion, both utilize the ShareGPT4V dataset (Chen al. 2023b containing 1M image and long caption pairs. During the relative position distillation phase, we truncate cap- tions to the first 77 tokens for both the teacher and student models. We train the student model using cosine loss as the distillation loss function for 20 epochs with a batch size of 640 using the AdamW optimizer (Loshchilov (017)), setting the learning rate to 5e-4 with 1000 warmup steps. In the relative position expansion phase, we employ full-length captions without truncation, exposing details. The full TULIP model, featuring the new distilled text encoder, is fine-tuned using the NTK approach, with empirically set to 8.0. For all main exper- iments, we use 248 number of tokens to match Long-CLIP context length for fair comparison. Note that we can increase the length to more tokens, shown in our ablations. We perform this fine- tuning stage for a single epoch with a batch size of 1280, learning rate of Ie-5, and 1000 warmup steps using Adam W. We base implementations LLIP-ViT---11 14 architectures (Ilharco et al., 2021).
the model comprehensive-textn
OpenAl's pre-trained CLIP-ViT-B. ViT-B-16 16 and
4.1 CROSS-MODAL RETRIEVAL COMPARISON
We evaluate TULIP against the original CLIP, fine-tuned CLIP on ShareGPT4V (Cen et al. 2023b
and Long-CLIP (Zhang et al. 2024) for both long-cat caption (Table I and short-caption ( (Table
cross-modal retrieval tasks. As shown in Table [ our proposed model outperforms all benchmarks in long-caption cross-mode modal retrieval across 3 datasets on both image-to-text and text-to-image re- trieval, utilizing two different vision backbones: ViT-B-16 and ViT-I.-14 (Dosovitskiy et al., 202T).
On the Long-DCI dataset, which is a more challenging benchmark for all approaches, we can ob- serve that Long-CLIP already shows improvement over the original CLIP and that our TULIP model further improves this. These results demonstrate the efficacy of TULIP in enhancing CLIP's ca- pabilities for long-caption cross cross-modal modal retrieval, particularly in diverse scenarios. For short captions, shown in Table 2 we find that the tailored approach used by Long-CLIP for the first 20 tokens is beneficial for the short caption performance as they outperform CLIP even on short captions. Whereas TULIP is able to obtain competitive performance without needing specifically tailor to

---

Published as a conference paper at ICLR 2025
the first 20 tokens, which demonstrates the flexibility of relative positional encodings across differ- ent caption lengths. Overall, the results in Table I [ad that TULIP is effective not only for long captions but also for maintaining competitive performance in short caption retrieval scenarios.
4.2 TEXT-TO-IMAGE GENERATION
In this section, we evaluate qualitatively how method enhances rext-to-image generation by sim- ply replacing the original CLIP ViT-L-14 text encoder in Stable Diffusion XL (SDXL) (Podell et al.) 2023) with TULIP. Note that do not perform any additional training of the diffusion model. As observed in Figure 3 TULIP demonstrates improvements in both long and short caption under- standing and modeling of nuanced details, compared to T5-based models such as PIXART-AIphe
(Chen et al. 2023a) and ELLA (Hu et al. (024), as well as CLIP- VIT-L-11 and Long-CLIP (Zhang al.) 2024). For example, our version of SDXL + TULIP accurately depicts "the red tulip inside wooden box" in the first example. This is not the case with SDXL + CLIP or SDXL Long- CLIP which generate many tulips in the garden, missing the key detail of a single tulip being inside the box. This observation shows that our model can indeed capture finer details in the description. The second example shows that our model successfully generates "an old man sils on a rock " even when this detail appears beyond the 77-token limit. On the other hand, base CLIP, Long-CLIP and PIXART-Alpha encoders fail at this. A similar observation is in the third example where the model correctly captures the "sharp suit" and "moustache" " very nuanced details, unlike the rest of the models. These examples are direct evidence that our approach can truly capture the meaning of words in longer captions. These observations suggest that our TULIP text encoder enhances long caption comprehension, resulting in more accurate and contextually rich image generation across varying prompt lengths. We provide additional image generation examples and human evaluation results in the appendix.
4.3 ABLATION STUDY
In this section, we carefully analyze various design choices for our proposed method. We examine different positional encoding schemes, investigate the impact of context length, and explore varia- tions in the distillation loss function.
Different types of positional encodings. In this experiment, we compare different types of posi- tional encodings, namely absolute and relative ones. We choose recently introduced Contextual Position Encoding (CoPE) (Golovneva al. 2024) for implementing the relative positional encod- ings in the text encoder. Afterwards, we perform the distillation and context length extension phases using the same dataset and parameters as with RoPE (Su et al., 2024). In Table [ we observe that RoPE outperforms CoPE in long-caption retrieval tasks. This is due to RoPE's strong ability to generalize across varying or extended sentence lengths, even beyond those on which the model was originally trained. In contrast, CoPE struggles to generalize effectively when sequence lengths in- crease, as its embeddings are more dependent on the specific context within which they were trained. This explains why CoPE performs similarly to RoPE on the ShareGPT4V test split (same training distribution), but shows a larger performance gap on the Long-DCI and Urban-14 k datasets.
The impact of the caption length. Next, we evaluate the impact of varying context lengths on the performance of RoPE in long-caption image retrieval tasks. To investigate this, we fine-tune only the text encoder during the context length extension phase with different context sizes (which n x 77): {77, 154, 231,308} tokens while keeping the image encoder frozen. We deliberately freeze the image encoder to isolate the text encoder's performance, as unfreezing could potentially mask the text encoder's 's limitations in processing longer inputs. Figure presents the model's performance across three datasets GShareGPT4V, Urban- 1K, and Long-DCI) for modsl retrieval tasks. We observe general improvement in performance with increased context length, particularly from 77 to 154 tokens, across all datasets. This improvement is more pronounced for image-to-text retrieval, suggesting that longer contexts enhance text representation and enable more precise alignment with image features. However, we observe a slight performance plateau or minor decline for 308-length sentences, signaling a point of diminishing returns where additional tokens may introduce noise or redundancy. This plateau is likely due to the average caption length in our training data being 174.02 tokens, explaining why performance levels off between 154 and 231 tokens.
7

---

Published as a conference paper at ICLR 2025
basd models
CLIP-based models
PIXART Alpha
ELLA
SDXL CLIP
SDXL Long-Clip
SDXL + TULIP (Ours)
In quiet garden, with tall green bushes on the left and clear pond on the right, wooden box sits on the grass in the middle. The box slightly open, and inside bright red tulip, standing tall. The sunlight shines on the tulip, making it stand out against the smooth wood the box. light breeze moves through garden, but the tulip stays still, standing tall inside the box.
The painting captures
center, calm lake reflects the sky, its surface rippled only by the gentlest of breezes. The sky above brilliant mix of blues and whites, with fluffy clouds drifting leisurefy across, On the banks of the lake, tall trees stand gracefully, their leaves rustligg in the wind. // The soft light of the setting sun bathes the entire scene in warm glow, creating sense of peace and tranquility. The colors are muted yet vibrant, and the details are captured with precision, giving painting sense of realism while still retaining a dreamlike quality. In the foreground, an old man sits rock, seemingly lost in deep thought or meditation.
illustration from graphic novel. A bustling city street under shine of moon. sidewalks bustling pedestrians enjoying the nightlife. At the corner stall, a young is haggling with the grumpy old vendor. the grumpy vendor, a tall, sophisticated man is wearing a sharp suit, sports // noteworthy moustache is animatedly conversing on his steampunk telephone.
with fiery red hair, dressed in a signature velvet cloak,
Figure 3: Text-to-Image Generation results. We replace the text encoder of SDXL with our own TULIP model. We observe improvements in capturing nuanced details, compared to TS-based mod- els such as PIXART-Alpha and ELLA, as well CLIP-ViT-L-IL and Long- CLI. Note that the marks the 77-token boundary in the caption. The words in green indicate visual concepts that are correctly generated by SDXL. + TULIP and are missed by the baselines.
Long-DCI
ShareGPT4V
Urban-1K 1K
Positional encodings
Img2Txt
Tx12Img
Img2Txt
Tx12Img
Img2Txt
Txt2Img
Absolute
41.9
40.0
96
93.8
72.9
69.4
CoPE
50.8
49.9
98.5
97.8
86.7
82.8
RoPE
55.7
56.4
99.0
99.0
90.1
91.1
Table Ablation comparing different positional encodings such as Absolute, RoPE, and CoPE in TULIP. RoPE generalizes better across varying or extended sentence lengths, especially on out- of-distribution datasets, namely Long-DCI and Urban-1k.
Benefit of using cosine distillation loss. In this section, we vary the distillation loss function used during the relative position adaptation phase and report the results in Table g As seen, the cosine loss performs better than other alternatives across different datasets and tasks. We believe this is attributed to its alignment with the normalized embedding space of CLIP models and its scale invariance property. The scale invariance of cosine loss proves particularly beneficial in this distillation setup, where the student model (ViT-L/14 with the added RoPE or CoPE layer) can produce embeddings with different magnitudes compared to the teacher model. This invariance
8

---

Published as a conference paper at ICLR 2025
(a) ShareGPT4V
(b) Urban-1K
(c) Long-DCI
7989
50.75
$7.30
10:00
11.30
01.00
40.00
$1.40
07.37
97.50
66.72
Acouraty
85.91
Acc Acnurryy
$2.10
17:28
19.00
72.90
23.30
05.81
154
300
77
154
231
305
154
231
508
Concext Longth
Lo
Context Langth
Figure 4: Impact of the sequence length on cross-modal retrieval tasks. We observe general improvement in performance with increased sequence length, particularly from 77 to 154 tokens, across all datasets and tasks.
allows the distillation process focus on transferring the essential directional information of the embeddings, rather than being influenced by potential scale discrepancies introduced by the RoPE or CoPE layer. As a result, cosine loss consistently outperforms other loss functions like 1.2 and MSE across various datasets (Long-DCC DCI, ShareGPT4V, Urban-1K and cross-modal tasks such as image- to-text and text-to-image, demonstrating its effectiveness preserving the semantic relationships learned by the teacher model.
Long-Do DCI
ShareGPT4V
Urban-1K
Distillation loss
Img2Txt
Tx12mmg
Img2Txt
Txt2lmg
Img2Txt
Txt2Img
CLIP
35.0
37.0
81.8
84.0
68.7
52.8
L.2 MSE
39.4
35.8
84.8
83.9
70.2
56.3
36.4
31.8
83.0
81.3
74.0
53.9
Cosine
38.5
35.8
84.8
84.2
73.6
56.6
Table 4: Ablation comparing different distillation loss terms in TULIP. Cosine loss yields the best performance across different datasets and tasks. Note that here we report the performance of the distilled models before the relative position expansion phase.
4.4 ADDITIONAL ANALYSIS
Attention spread visualization. The effectiveness of our model in long cross-modal retrieval tasks and image generation largely due its distinct attention distribution patterns. To illustrate this, visualize the attention scores between the CLS text token and its preceding tokens in the final attention block of the text encoder, as shown Figure 5 For comparison, we provided attention visualizations for both TULIP and LongCLIP. This analysis reveals two key advantages of our model when processing a 248-token caption. First, our model exhibits a more uniform distribution of attention across input tokens, demonstrating its ability expand the attention scope and effectively aggregate information from a broader range of tokens. This expanded attention field improves the model's performance in long-caption tasks by capturing and utilizing details from later parts of the caption that other models might overlook. Second, our model shows increased attention to punctuation symbols, particularly commas, which enhances its ability to parse and segment longer texts. Such capability is crucial for better comprehension of complex, multi-caption descriptions.
Caption-image relevance distribution analysis. This analysis investigates where relevant informa- tion is distributed within long captions given an image. The aim is to highlight how useful content is spread throughout the whole caption. We analyze 100 randomly selected images with long captions from the ShareGPT4V dataset. For each image-caption pair, we computed the visual embeddings of the image and compared them to the text embeddings of the caption subwindow. We define sliding subwindows of sizes 20, 33, and 55 tokens, moving from left to right with strides of 5, 10, and 15 tokens respectively. We then calculated the cosine similarity between the image encoding and the text encodings of each subwindow, visualized in F The figure shows that the similarity scores are distributed across different subwindows, emphasizing the need for models capable of processing longer input sequences. Notably, as window size increases, the similarity patterns become more

---

Published as a conference paper at ICLR 2025
FUR
CIDE
TEA
YOWK
PPay
OFF
SORTANN
(a) LongCLIP
CRURANNG
SUPE
PARMEN
1272
bac
PRIZE
PRACE
EOLDAN
DAR
(b) TULIP
Figure 5: Attention Spread Visualization comparing (a) LongClip and (b) TULIP. Our model achieves uniform attention across tokens, demonstrating superior capabilities in parsing and seg- menting longer texts with precision.
Window Size. 20. a Windows:
Window Size: 20, Wirdows
Window Size: 20. Windows: 15
R Sinierty
las
0. Sinlarty
Window Fosition
Window Position
Window Position
Window Windows,
Window Windows
Window Size: Windows:
Sx Smilanity
0.1
Simllsiity
010
10
10
Window Size: Winchue:
% & Simberray
Window Size: 55,
Windows: 10
Winddww Sizee Windows:
Emmietty
0. mmmterr
10
22
Figure 6: Caption-image relevance distribution analysis across varying window sizes and posi- tions. It shows that image-relevant information is spread throughout captions, emphasizing the need for models to process longer text inputs to capture all pertinent details.
concentrated and pronounced, suggesting that larger context windows capture more cohesive and relevant information. Additionally, the variability in similarity across different windows highlights the non-uniform distribution of image-relevant information throughout the captions. This reinforces the need leverage the entire textual sequence when learning image-text representations.
CONCLUSION
In this work, we addressed the limitations of CLIP-like models in handling long input sequences. We introduce TULIP, generalizable method that upgrades the context length beyond the 77-token limit. By leveraging relative positional encodings, our approach enables the effective modeling of pairwise token relationships. Through a two-step training process, we successfully adapt the CLIP-like model to process longer captions without compromising its performance on shorter puts. Our experiments demonstrate that TULIP considerably improves long-caption performance cross-modal tasks such as retrieval and text-to-image generation, setting a new standard for vision- language contrastive models that require handling complex, extended text descriptions.
Limitations. Our reliance on ShareGPT4V dataset, synthesized by GPT4V (Achiam et al. 2023), limits TULIP's performance to the quality of GPT4V's long captions. Furthermore, while TULIP theoretically capable handling longer contexts due to the nature of the relative positional encod- ings, its token length is constrained by the average token length of the ShareGPT4V captions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303, 08774, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:23009. 16609, 2023.
bloc97.
Awa-eware Scaled RoPE allows LLaMA models have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. https:
/ //www. reddit. cowm///ooaalLLaa http:////ccallaaaaacoootttt///// /ow.///tttt/aaaaaa2///ttttaaaaaa
scaled
rope allows 11 ama wwwels to havee/_aaeee____ 2023.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023b.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint (50.:2336..5555, 2023c.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finttnne lan- guage models. Journal of Machine Learning Research, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. arXiv preprint arXiv:2310.1623 2023.
Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Contextual position en- coding: Learning to count what's important, arXiv preprini arXiv:2405.187/9 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint a5X 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL hrtps: www...o//....2.00 zenodo. 514377477 If you use this software, please cite it as below.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and languaee representation learning with noisy text supervision. In International Conference Machine Learning, 2021.
11

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ing -mmage pre- training for unified vision- laionnlanguage understanding and generation. In International Conference on Machine Learning, 2022.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403. 19887, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
I Loshchilov. Decoupled weight decay regularization, arXiv preprint arXiv:
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, al. DOCCI: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pp. 291 Springer, 2024.
Saurav Pawar, SM Tonmoy, SM Zaman, Vinija Jain, Aman Chadha, and Amitava Das. The what, why, and how of context length extension techniques large language models-a detailed survey. arXiv preprint arXiv:2401.0788 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint 1.Xx::3309000777 2023.
Bryan Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting pegion-o-phraan correspondences for richer image- to-sentence models. In IEEE International Conference on Computer Vision, 2015.
Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthe- sis. ArXiv, abs/2307.01952 2023.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.1 12409, 2021.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter Liu, Sharan Narang, Wei Li, and Yanqi Zhou. Exploring the limits of transfer learning with unified text-to- text transformer. Google, Technical Report, 2019.
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Ben- nani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint ar0iv:2305. 16843, 2023.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. arXiv preprint 0003.0003..03300 155, 2018.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127000 127063, 2024.
12

---

Published as a conference paper at ICLR 2025
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urXiv:2307.0928, 2023.
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Conference on Computer Vision and Pattern Recognition, 2024.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, and Oncel Tuzel. Clip with quality captions: A strong pretraining for vision tasks. arXiv preprint arXiv:2405.0m 089111 2024a.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Confer- ence on Computer Vision and Pattern Recognition, 2024b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances Neural Informa- fion Processing Systems, 2017.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference Learning Reprexentations, 2023. URL pttttps:///penrevviee net forum?
KRLKUUUxxauaX
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to--sequence functions? arXiv preprint 1912 2019.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-Cle CLLI: Unlocking the long-text capability of clip. arXiv preprint 0rXiv:2403.15377 2024.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221, 2022.
Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-image pre-training with long captions. In European Conference on Com- puter Vision 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 QUALITATIVE EVALUATION OF IMAGE GENERATION USING T5-BASEE 25-BASED MODELS, CLIP, LONG-CLIP, AND TULIP
PixArt Exxrt-Alpha++ TT
ELLA + T5
SDXL + CLIP
SDXL Long-Clip
SDXL + TULIP (Ours)
quiet garden, tall green bushes on the left clear pond on right, a wooden box sits on the grass in middle, The box is slightly open, and inside brightt red tullp, standing tall. The sunlight shines the tulip, making stand out against the smooth wood of the box. light breeze moves through the garden, // but the tulip stays still, standing tall inside the box
PixArt-Ai Allhaa T5
ELLA + T5
SDXL CLIP
SDXL Long-Clip
SDXL TULIP (Ours)
The painting captures serene moment in nature. At the center, calm lake reflects the sky, its surface rippled only by the gentlest of breezes. The sky above is brilliant mix of blues and whites, with fluffy clouds drifting leisurely across. On the banks of the lake, tall trees stand gracefully, their leaves rustling in the wind. // The soft light of the setting sun bathes the entire scene in warm glow, creating sense of peace and tranquility. The colors are muted yet vibrant, and the details captured with precision, giving the painting sense of realism while still retaining a dreamlike quality. In the foreground, an old man sits on rock, seemingly lost in deep thought or meditation.
PixArt Alpha TS
ELLA T5
SDXL CLIP
SDXL Long-Clip
SDXL TULIP (Ours)
A futuristic city night, illuminated by neon lights in various shades of blue, pink, and purple, with towering skyscrapers featuring unique geometric designs, flying cars zipping through the alt, // and people walking along the streets below, dressed in sleek, futuristic clothing.
Figure 7: Qualitative results of image generation using text encoders based on T5 (PIXART- Alpha and ELLA), CLIP, Long-CLIP, and TULIP. Our TULIP-based model demonstrates the ability to generate images with nuanced details from captions, offering a plug-and-play solution without requiring costly retraining.
A.2 HUMAN EVALUATION OF TEXT-TO-IMAGE GENERATION
We design a human evaluation study by manually crafting captions and generating images by using Stable Diffusion XL (SDXL) with CLIP and -ULIed text encoders. The resulting images are randomized, and annotators are tasked with selecting the image that best aligned with the prompt (See Figure 9). They were instructed to evaluate alignment based objects, their attributes, and their relationships. A total of 15 annotators reviewed 20 samples. We report the results in Table As can be seen, the win ratios 89% for the SDXL + TULIP model and 11% % for the SDXL CL.IP
14

---

Published as a conference paper at ICLR 2025
AAlpaa 'T5
ELLA + T5
SDXL CLIP
SDXL Long-Clip SDXL + TULIP (Ours)
The sun was high in clear blue sky, making the grass the look even greener it swayed the light breeze. playful dog with soft, golden fur dashed the grass, paws kicking up tiny clumps of earth it Bright red stood tall in nearby garden, their smooth petals glowing the sunlight, adding // burst of color the
PixArt- Alpha TT
ELLA + TS
SDXL CLIP
SDXL Long-Clip
SDXL + TULIP (Ours)
A Illustration from a graphic novel. A bustling city street under the shine of full moon. The sidewalks bustling with pedestrians enjoying the nightlife. the corner stall, a young woman with fiery red hair, dressed in signature velvet cloak, is wearing sharp suit, sports
is haggling with the grumpy old vendor. the grumpy vendor, tall, sophisticated noteworthy moustache animatedly conversing on his steampunk telephone.
PixArt AAlha T5
ELLA
SDXL CLIP
SDXL Long-Clip
SDXL TULIP (Ours)
A mischievous ferret with playful grin squeezes itself into large glass jar, surrounded by colorful candy. The jar sits on wooden table in cony kitchen, and warm sunlight filters through nearby window.// //
Figure& 8: Qualitative results of image generation using text encoders based on T5 (PIXART- Alpha and ELLA), CLIP, Long-CLIP, and TULIP. Our TULIP-based model demonstrates the ability to generate images with nuanced details from captions, offering a plug-and-play solution without requiring costly retraining.
model, reflecting the enhanced alignment capability that we observe in the qualitative comparisons as well. The following instructions were provided to the annotators:
Instructions for annotators:
You are provided with 20 samples, each containing prompt and two images generated by two different Stable Diffusion models. Your task is to compare the two images given the input prompt and choose the you find more aligned with the prompt. Note that the differences might be very nuanced in some cases. Pay attention to the object attributes, relationships, verbs, object counting, etc.
Method
Win rate (%)
SDXL CLIP
11
SDXL. -ULIPP (Ours)
89
Table 5: Human evaluation results. The results reflect the enhanced alignment capability that we observe the qualitative comparisons as well.
15

---

Published as a conference paper at ICLR 2025
Model
Model
Input Prompt
bridge, their stone edge, their beight, shiny surfaces reflecting the overcast light. As the thunderstorm louns closer, the wind picks up, cousing the flowers to bend slightly. The stoem's 's dark elouds create dramatic backdrop, making the bikes and flowers stand out even more against the approaching gloom.
Colorfal Bloeers fall bloom line the base of fall.
petals
drizzle
sold
Model
Model
Input Prompt
A sturdy brown suitcuse with bruss clasps rests on the fooo next to a solt, round bean beg. The suitcase slightly scuffed, showing signs of travel, with a colontul laggage tag hanging from the handle. The hean beg, filled with plush, squishy foom, covered in inviting suitcase
blue fabric, relax. The contrast between hard, structured soft, relaced shape the beg, welcoming atmosphere.
Figure 9: Examples used in the human evaluation study. We provide the text prompt and two randomized images generated by SDXL with different variants of the text encoder (TULIP CLIP).
A.3 EVALUATION COMPOSITIONAL UNDERSTANDING BENCIIMARKS
CLIP-like models often struggle with accurately interpreting the compositional aspects of images, including spatial relationships, numeracy, and complex inter-object attributes. To evaluate TULIP's capabilities these areas, we considered two well-established baselines for multimodal composi- tional understanding: ARO (Yuksekgonul et al. 2023) and VL-Checklist (Zhao et al. 2022). The results in Table indiictee that TULIP consistently outperforms CLIP and surpasses Long-CLIP in 4 out of 7 settings. These findings highlight TULIP's enhanced capability in handling fine-grained captions.
ARO
VL-CCeekllist
Method
VGR
VGA
Flickr
COCO
Obj
Att
Rel
CLIP
59.9
63.1
60.2
47.9
81.1
67.6
61.9
Long-CLIP
64.6
66.6
24.8
23.3
84.3
71.6
63.5
TULIP (Ours)
63.4
66.2
52.3
43,9
85.2
74.3
62.7
Table 6: Comparison across compositional understanding benchmarks, namely ARO and VL- Checklist. We can observe that TULIP consistently outperforms CLIP and surpasses Long-CLIP in 4 out of 7 settings, demonstrating its ability to handle fine-grad fine-grained captions. Note that we use CLIP-ViT-L-I- 14 as vision encoder.
A.4 DOCCI BENCHMARK EVALUATION LONG-CAPTION RETRIEVAL
To additionally evaluate TULIP on long-caption retrieval tasks, we use DOCCI Onoe et al. (024)), which is novel dataset consisting of images paired with long, human-annotated English descrip- tions. We performed experiments to compare our TULIP model to the baselines CLIP and Long- CLIP. We provide the top R@ I results in Table 7 where observe that TULIP obtains the best performance, especially for image-to-text retrieval, demonstrating its efficiency and applicability across more long-caption benchmark.
16

---

Published as a conference paper at ICLR 2025
Img2Txt R@1
Txt2lmg R@1
CLIP
65.7
63.1
Long-CLIP
66.5
78.5
TULIP (Ours)
77.9
79.1
Long caption cross-modal retrieval comparison DOCCI dataset. We observe that TULIP obtains the best performance, especially for image-to-text retrieval, demonstrating its effi- ciency and applicability across one more long-caption benchmark.
Table 7:
A.5 TRAINING DATA AMOUNT FOR TULIP STUDENT MODEL
To analyze the amount of data needed to train the student model, we conduct experiments using 33%, 66%, and the entire Sh dataset (1.2 million image-caption pairs). The results on the retrieval tasks (top R@] I) shown in Table g emphasize the importance of utilizing the full dataset, particularly for improving the performance on -fd-ssiriiittttionn data i.e. Long-DCI and Urban
Long-DCI
Shar
Urban-1K
Amount of data
Img2Txt
Txt2lmg
Img2Txt
Txt2Img
Img2Txt
Txt2lmg
33%
15.2
12.2
88.6
90.0
33.6
36.9
66%
28.7
26.2
94.6
95.5
37.6
45.0
100%
55.7
56.4
99.0
99.0
90.1
91.1
Table Performance comparison across different dataset sizes.
A.6
ADDITIONAL IMAGE GENERATION COMPARISON BETWEEN TULIP AND T5-BASED MODELS
The T5 text encoder (Roberts et al. 2019) is well-known for its ability to handle long captions. To evaluate how our TULIP method compares to T5-based approaches, we conducted additional text- to-image generation experiments. In particular, TULIP-based model and two T5-based models, PIXART-Alpha (Chen et al. (023a) and ELLA (Hu al. 2024). in Figures TO
provide a qualitative comparison between
11 12 14 and 15 This comparison demonstrates that all models perform on par in generating high-quality images from long and nuanced text descriptions. However, TULIP approach stands due to its plug-and-play capability for image generation, requiring no additional fine-tuning, unlike PIXART-Alpha (Chen et al.) (0023)) and ELLA (Hu et al. 2024) This characteristic makes TULIP more versatile and practical compared to the fine-tuning demands of the T5-based counterparts.
A.7 QUALITATIVE EVALUATION OF CROSL RETRIEVAL USING CLIP AND TULIP
Figures 16 and 17 showcase selected results from our qualitative evaluation of cross-modal retrieval tasks. We can observe that our TULIP model excels at capturing fine-graine grained details in both captions and images, outperforming the baseline CLIP.
17

---

Published as a conference paper at ICLR 2025
T5-based models
TULIP model
PixArt-Alpha T5
ELLA T5
SDXL + TULIP (Ours)
Two tall white Christmas trees, adorned with twinkling lights and colorful ornaments, stand in the corner of cozy living room. Beneath the trees, wrapped presents in shiny blue and golden paper and colorful ribbons are stacked in a small mountain.
PixArt-Alpha T5
ELLA T5
SDXL TULIP (Ours)
Colorful flowers in full bloom line the base of a stone bridge, their petals wet from the light drizzle that has begun fall. A pair of yellow gold bikes rest on the stone edge, their bright, shiny surfaces reflecting the overcast light. As the thunderstorm looms closer, the wind picks up, causing the flowers to bend slightly. The storm's dark clouds create a dramatic backdrop, making the bikes and flowers stand out even more against the approaching gloom.
PixArt-Alpha + T5
ELLA + T5
SDXL TULIP (Ours)
juicy double chicken burger, piled high with pickles, onions, and perfectly cooked patty, is placed on wooden table, the bun soft and golden. A pile of fries, crispy on the outside and fluffy on the inside, sits beside the burger, their golden color shining in the light. A bunch of red roses, their petals open and lush, sits in the background, adding a touch of elegance to the scene. The warm, inviting colors of the food contrast beautifully with the deep red of the roses.
Figure 10: Comparison to T5-based models (PIXART-Alpha CCen et al. 2023a) and ELLA Hu
al. 2024 for image generation.
18

---

Published as a conference paper at ICLR 2025
T5-based models
TULIP model
PixArt-Alpha T5
ELLA T5
SDXL + TULIP (Ours)
A tall martini glass with sleek, clear stem holds perfectly mixed drink, the glass gleaming under the soft light. Inside the glass, a single green olive floats at the base, its shiny surface reflecting the light. Nearby, a plate of fresh oysters is arranged neatly, their glossy, wet shells slightly open, revealing the tender meat inside. The contrast between the elegant martini and the rustic oysters creates a striking visual pairing the table.
PixArt-Alpha + T5
ELLA T5
SDXL TULIP (Ours)
A sturdy brown suitcase with brass clasps rests on the floor next to soft, round bean bag. The suitcase is slightly scuffed, showing signs of travel, with colorful luggage tag hanging from the handle. The bean bag, filled with plush, squishy foam, is covered in cozy blue fabric, inviting someone to relax. The contrast between the hard, structured suitcase and the soft, relaxed shape of the bean bag creates a welcoming atmosphere.
PixArt-Alpha + T5
ELLA + T5
SDXL TULIP (Ours)
An enchanted forest at dawn, where glowing mushrooms light up the forest floor, trees with twisted trunks and sparkling leaves reach towards the sky, and magical creatures like fairies and wisps float through the air, while waterfall cascades into crystal-clear pond in the background.
Figure 11: Comparison to T5-based models (PIXART-Alpha (Chen et al. 2023a and ELLA HHu
al. 2024) for image generation.
19

---

Published as a conference paper at ICLR 2025
T5-based models
TULIP model
PixArt-Alpha T5
ELLA T5
SDXL + TULIP (Ours)
A serene beach at dawn, where the waves gently lap against the shore, the sand soft and golden underfoot, and palm trees sway in the light breeze, while seagulls call overhead. lone sailboat drifts on the horizon as the sun rises, casting a golden light across the water.
PixArt-Alpha T5
ELLA T5
SDXL TULIP (Ours)
A bright, full moon hangs high in the sky, its glowing surface casting a soft, silvery light over the landscape. The starry night is dotted with countless sparkling stars, their brilliance forming delicate constellations against the deep navy backdrop. Below, single snowy mountain rises majestically, its peak glistening under the moonlight. The crisp white snow contrasts sharply with the dark sky, creating serene and magical scene
PixArt-Alpha + T5
ELLA + T5
SDXL TULIP (Ours)
freshly baked pizza with cheese and vibrant toppings sits on a wooden table, its crust golden and crisp. Beside it, a glass of deep red wine rests, its smooth surface reflecting the soft glow of candlelight. A small vase white flowers sits nearby, their delicate petals bright and pure against the rustic table setting. The contrasting textures of the warm pizza, the wine glass, and the soft blooms create a harmonious and inviting scene.
Figure 12: Comparison to T5-based models (PIXART-Alpha (Chen et al. 2023a) and ELLA (Hu al. 2024 for image generation.
20

---

Published as a conference paper at ICLR 2025
T5-based models
TULIP model
PixArt-Alpha T5
ELLA T5
SDXL + TULIP (Ours)
A glass vase filled with fresh pink tulips sits in the center of a desk, the long green stems visible through the crystal clear surface. The petals of the tulips are soft, their delicate pink hue adding warmth to the room. Next to the vase, pair of neatly stacked books lies with one slightly ajar, revealing a glimpse of printed pages. The arrangement of flowers and books creates serene, cozy corner ideal for reading or reflection.
PixArt-Alpha T5
ELLA T5
SDXL TULIP (Ours)
The purple cat lounges lazily on sunny windowsill, its bright fur standing out against the light-colored wood. Its oversized sunglasses reflect the sunlight, adding cool, fashionable touch to its relaxed pose.
PixArt-Alpha T5
ELLA + T5
SDXL TULIP (Ours)
The pink tiger stands gracefully in the corner of the house with a green sun glasses, and its vibrant fur shimmering in the sunlight streaming through the window. Its intense gaze blends with the calm atmosphere of the room, surrounded by plants and cozy furniture.
Figure 13: Comparison to T5-based models (PIXART-AIpha CCen et al. 2023a) and ELLA (Hu
ct al. 20244) for image generation.
21

---

Published as a conference paper at ICLR 2025
T5 based models
TULIP model
PixArt-Alpha T5
ELLA T5
SDXL + TULIP (Ours)
A golden sandy beach stretches out under blue sky, with gentle waves lapping at the shore. A few white seashells are scattered along the edge.
PixArt-Alpha + T5
ELLA + T5
SDXL TULIP (Ours)
A birthday cake with creamy white frosting is decorated with blue and purple sprinkles and topped with lit candles. The cake sits table surrounded by white flowers next to white dog.
PixArt-Alphe + T5
ELLA + T5
SDXL TULIP (Ours)
butterfly with orange and black wings flies gently over vibrant flower field. The field is filled with colorful blooms of yellow and pink, their petals swaying in the breeze.
Figure 14:
Comparison to T5-based models (PIXART-Alpha (Chen et al. 2023a) and ELLA Huu al. 2024)) for image generation.
22

---

Published as a conference paper at ICLR 2025
T5-based models
TULIP model
PixArt-Alpha
ELLA T5
SDXL TULIP (Ours)
A green spaceship with glowing red engines flies above a snowy mountain. The mountain's jagged peaks glisten with fresh snow under the soft light of dawn.
PixArt-Alpha T5
ELLA T5
SDXL TULIP (Ours)
white sits on windowsill beside vase filled with bright pink tulips. The cat's eyes are on the petals as sunlight filters through the blue vase.
Comparison to T5-based models (PIXART-Alpha Chen et al. 2023a and ELLA (Hu et al. 20244 for image generation.
Figure 15:
23

---

Published as a conference paper at ICLR 2025
Image
Input Image
Retzieved text CLIP: A worker passes through a cluttered restaurant kitchen. Retzieved text by CLIP: white bowl filled with meat, vegtables and broth.
Retzieved text by TULIP: Two men week the kitchen a resaurant.
Betzieved text by TULIP: An orange bowl filld with lots of noodles and beef.
Input Image
Input Image
Betrieved text by CLIP: Two chill dogs city sidewalk.
Retrieved text by CLIP: A man watches dog prepares to for trisbee. Retrieved text TULIP: dogs the backyard with blowup
round outdoor Retrieved hotdog and bag chips
Figure 16: Qualitative comparison of image-to-text retrieval between CLIP and TULIP. We can observe that our TULIP model can capture the fine-gr ine-grrined details in both captions and images. Note that the red text color indicates visually misclassified concepts in the images, whereas the green color indicates correct ones.
Text: Several children are riding on toy train ride.
Text: Two airplanes flying in the sky above black brdige.
K
Retrieved image by CLIP
Retrieved image by TULIP
Retrieved Image by CLIP
Retrieved Image by TULIP
Text: Two guys playing baseballs, with trees back.
Text: young man body surfing and padding in the water.
Retrieved image by CLIP
Retrieved image by TULIP
Retrieved image by CLIP
Retrieved Image by TULIP
Figure 17: Qualitative comparison of text-to-image retrieval between CLIP and TULIP. We observe that our TULIP model can capture the fine-grained details in both captions and images. Note that the red text color indicates visually misclassified concepts in the images, whereas the green color indicates correct ones.
24


Reviews:
[{"review_id": "7xVkZBrRLF", "reviewer": "Reviewer_XgPN", "summary": "The paper addresses the challenge of integrating positional information effectively in contrastive vision-language models, particularly when dealing with long captions. Traditional models like CLIP are limited by short context windows, which restricts their ability to process detailed and dense textual descriptions. The authors propose an approach that combines Rotary Position Embedding (RoPE) and Contextual Position Encodings (CoPE) to better handle long captions without the need for training from scratch. This method aims to enhance the model's ability to comprehend pairwise token relationships and capture fine-grained relative positions in longer, more complex captions. The paper builds upon existing work such as Long-CLIP and proposes a refined technique to address the limitations of absolute positional encodings through interpolation.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Although the techniques themselves (RoPE and CoPE) are not novel, the paper demonstrates creativity in combining these methods to tackle the limitations of existing models like CLIP. This approach leverages the strengths of both RoPE and CoPE to address the shortcomings of absolute positional encodings, thereby providing a more dynamic and effective way to capture positional relationships in long sequences.\n\n2. The paper is well-written and clearly presented. The authors effectively communicate their ideas and methodologies, making it accessible to readers with a background in machine learning and natural language processing. The use of figures, tables, and examples to illustrate key points is helpful in conveying the technical details of the proposed approach.", "weaknesses": "1. The proposed method appears to be an incremental improvement based on existing models like Long-CLIP. The combination of RoPE and CoPE, while potentially effective, does not introduce fundamentally new concepts. Both RoPE and CoPE are not novel and have been introduced in prior works. The primary contribution seems to be the application of these methods to the specific task of processing long captions, which may not be sufficient to claim significant novelty.\n\n2. The paper does not adequately compare its proposed method with other approaches [1] that handle long text descriptions, such as those based on the encoder of T5 model [2]. T5-based models are known for their capability to process long texts effectively. The authors need to clearly articulate the advantages of their approach over T5-based methods and provide empirical evidence to support these claims. Without this comparison, it is challenging to assess the relative effectiveness of the proposed method.\n\n3. The paper lacks a deep theoretical analysis of why the combination of RoPE and CoPE is expected to perform better for long captions. A more detailed theoretical justification or analysis could strengthen the paper by providing a solid foundation for the proposed method's expected performance improvements.\n\n[1] Chang, Huiwen, et al. \"Muse: Text-to-image generation via masked generative transformers.\" arXiv preprint arXiv:2301.00704 (2023).\n[2] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" Journal of machine learning research 21.140 (2020): 1-67.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGTmS4VaPd", "reviewer": "Reviewer_3TVK", "summary": "The TULIP model enhances CLIP's capability to process long captions by replacing fixed positional embeddings with relative positional encodings, extending the model's inherent context window beyond the 77 token limit. This approach is shown to improve performance on cross modal retrieval and text to image generation tasks. The TULIP training method is 2 stage - relative position distillation from the original CLIP model and finetuning to handle longer captions. Benchmark tests demonstrate TULIP's superior performance in long-caption scenarios. The paper also introduces a new evaluation benchmark, Long-DCI, to better assess long-caption retrieval tasks.", "questions": "1) How does the new model affect the compositionality of text inputs as dicussed here? https://aclanthology.org/2023.emnlp-main.301.pdf\n2) What do you attribute the improved performance in a shorter context to? In Table 2, TULIP performs a lot better than CLIP.\n3) Why are the values in Table 4 lower than those in Tables 1 or 3? If I missed something, can you clarify the caption to avoid confusion?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Significant improvements across cross-modal retrieval and text-to-image generation\n- The introduction of a benchmark for long captions is a step towards better evaluation of other research works in this domain\n- The innovative approach with relative captioning and distillation has shown improved performance\n- The paper writing is easy to follow", "weaknesses": "- Switching from absolute to relative positional encodings does improve flexibility with token length but can also introduce challenges in retaining fine-grained positional relationships in shorter contexts\n- No specific details about the human annotators\n- Needed a more qualitative comparison of how the new approach led to better performance in shorter context\n- While CLIP possesses excellent zero-shot capabilities, it suffers from certain limitations in perceptual understanding. A discussion on TULIPs limitations in perceptual understanding would boost the presentation of the proposed approach", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f8E3NtLw4D", "reviewer": "Reviewer_FB4m", "summary": "The authors propose TULIP, a method that allows for extending the capabilities of CLIP beyond the 77 token limit that is often used when training such models. The authors demonstrate that their method improves performance of CLIP trained models, on both traditional image-text retrieval tasks, as well as long-context tasks (one of which is proposed by the authors themselves).", "questions": "I would be grateful if, in addition to the points I raised above, the authors could elaborate on the amount of data with long context is needed to train the student model with TULIP, in order for it to perform well on long contexts.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper examines the quality of CLIP models in the setting of long text descriptions of images. This setting is of interest to the community - as the context of models grows, the ability to retain information from the entire caption decreases as well, and trying to remedy this is an interesting problem.\n\n- The paper is also clear and easy to understand. I did not have any issues understanding the method presented, or any other part of the paper.\n\n- I also believe that the inclusion of the Long-DCI evaluation dataset is an important contribution to the community. Having benchmark specifically for images with long context captions is extremely important in the advancement of this field.", "weaknesses": "- While the method itself appears useful, I am slightly worried about its novelty. At the end of the day, the model still requires training with longer contexts, so it is not immediately clear to me how different the method is from prior work. I believe that elaborating a bit more on which part of the method proposed by the authors provides the most benefit over prior work such as Long-CLIP would alleviate this concern.\n\n- The paper has an important limitation, in that the setting considered is only CLIP-style models, and not autoregressively trained ones. The latters tend to have much larger context length than the ones considered in this paper. While I understand that direct comparison between the two may not be possible/ideal, having the method being applied to this category of models would greatly strengthen the paper.\n\n- I would also be grateful if the authors could perform evaluation on a task based on the DOCCI dataset [A], which has the same stated goal as the proposed Long-DCI one.\n\n- The experimental evaluation on image generation in Section 4.2 is qualitative rather than quantitative. I believe that the authors could greatly strengthen the conclusions of the paper by evaluating the generated images in a more principled manner (for example, by performing a human preference study on the generated images).\n\nReference:\n\n[A] Onoe et al., \"DOCCI: Descriptions of Connected and Contrasting Images\", ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)