ICLR 2025

Published as a conference paper at ICLR 2025
TULIP: TOKEN-LENGTH UPGRADED CLIP
Ivona Najdenkoska∗Mohammad Mahdi Derakhshani∗Yuki M. Asano
Nanne van Noord
Marcel Worring† Cees G. M. Snoek†
University of Amsterdam
Amsterdam, the Netherlands
ABSTRACT
We address the challenge of representing long captions in vision-language mod-
els, such as CLIP. By design these models are limited by fixed, absolute posi-
tional encodings, restricting inputs to a maximum of 77 tokens and hindering
performance on tasks requiring longer descriptions. Although recent work has
attempted to overcome this limit, their proposed approaches struggle to model
token relationships over longer distances and simply extend to a fixed new to-
ken length. Instead, we propose a generalizable method, named TULIP, able
to upgrade the token length to any length for CLIP-like models.
We do so
by improving the architecture with relative position encodings, followed by a
training procedure that (i) distills the original CLIP text encoder into an en-
coder with relative position encodings and (ii) enhances the model for aligning
longer captions with images. By effectively encoding captions longer than the
default 77 tokens, our model outperforms baselines on cross-modal tasks such
as retrieval and text-to-image generation.
The code repository is available at
https://github.com/ivonajdenkoska/tulip.
1
INTRODUCTION
Obtaining text representations for long captions in vision-language models is an open research chal-
lenge. Within the context of text-only large language models, this problem has been studied ex-
tensively (Chung et al., 2024; Touvron et al., 2023; Bai et al., 2023; Gu & Dao, 2023; Su et al.,
2024; Dubey et al., 2024; Pawar et al., 2024; Jiang et al., 2024; Lieber et al., 2024), yet, methods
for context length expansion have only scarcely made their way into the vision-language domain.
For instance, contrastive vision-language models like CLIP (Radford et al., 2021; Jia et al., 2021; Li
et al., 2022)...