Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models

## Abstract

Abstract Modern retrieval systems often struggle with upgrading to new and more powerful models due to the incompatibility of embeddings between the old and new models. This necessitates a costly process known as backfilling, which involves re-computing the embeddings for a large number of data samples. In vision, Backward-compatible Training (BT) has been proposed to ensure that the new model aligns with the old model’s embeddings. This paper extends the concept of vision-only BT to the field of cross-modal retrieval, marking the first attempt to address Cross-modal BT (XBT). Our goal is to achieve backward-compatibility between Vision-Language Pretraining (VLP) models, such as CLIP, for the cross-modal retrieval task. To address XBT challenges, we propose an efficient solution: a projection module that maps the new model’s embeddings to those of the old model. This module, pretrained solely with text data, significantly reduces the number of image-text pairs required for XBT learning, and, once it is pretrained, it avoids using the old model during training. Furthermore, we utilize parameter-efficient training strategies that improve efficiency and preserve the off-the-shelf new model’s knowledge by avoiding any modifications. Experimental results on cross-modal retrieval datasets demonstrate the effectiveness of XBT and its potential to enable backfill-free upgrades when a new VLP model emerges.

## 1 Introduction

As the volume and variety of data grow exponentially in our multimedia-rich era, developing and maintaining efficient multi-modal retrieval systems becomes increasingly challenging. These systems, which provide data samples relevant to a user’s query, must handle diverse data types, from text and images to audio and video. This growth puts a premium on the scalability and performance of these systems, necessitating continuous advancements in algorithms and technology. Embedding-based deep models for retrieval have emerged as a key solution, transforming high-dimensional data into a lower-dimensional dense embedding space [, , , , , ]. These models capture the semantic meanings of data samples, enabling the quantification of similarities for retrieval.

However, it is important to note that the embedding spaces generated by different deep models are not inherently compatible with each other. This incompatibility restricts the reuse of an existing gallery that has been embedded with an older model when a new, better-performing model is introduced. Consequently, this necessitates “backfilling,” a process where the entire gallery must be rebuilt using the embeddings from the new model. Such a requirement is resource-intensive and time-consuming, posing a significant challenge when building retrieval systems.

Backward-compatible Training (BT) [, , , , ] has been developed to tackle this issue, specifically focusing on image retrieval systems. The main objective of BT is to train a new model from scratch in a manner that ensures its compatibility with an old model that was used to create the existing gallery. A successful BT model must consequently demonstrate that retrieving from the old gallery using a query embedded with the new model outperforms that using an embedding from the old model. This enhancement is crucial for justifying the application of BT.

Figure: Figure 1: A conceptual visualization of Backward-compatible Training (BT, left) and its extension, the Cross(X)-modal version (XBT, right). Circles and squares denote data samples of images and text, respectively. XBT uses Vision-Language Pretraining (VLP) models as baselines to achieve cross-modal, backward-compatible representation learning, allowing the new, improved model to be compatible with the fixed old model.
Refer to caption: x1.png

Taking the BT problem a step further, we propose a new challenging task called Cross(X)-modal Backward-compatible Training (XBT) as shown in Figure 1. Our focus here is on applying BT principles within the realm of cross-modal retrieval, specifically exploring the interaction and compatibility between image and text embeddings of different Vision-Language Pretraining (VLP) models [, , , , ].

To achieve XBT, just like the BT problem, we need to resolve the incompatibility between the embeddings of the old, inferior model and the new, improved model. If we follow most prior practices of training from scratch with an additional compatibility loss term, a substantial number of supervised image-text pairs would be needed. The ideal quantity would be in the hundreds of millions, approximating the amount utilized in the establishment of VLP models. However, accessing the original training samples is often impossible, and training on that scale is prohibitively expensive and impractical. We propose an efficient solution instead: a text-only *pretrained* projection module, $\phi$, to align a given *pretrained* new model’s embeddings with those of the old model.

Our focus is on using only text data to estimate the entire distribution of embeddings in the VLP space. Specifically, we train $\phi$ to learn the similarity between old and new text embeddings in a contrastive manner. By increasing the number of text samples, which is simpler than preparing image-text pairs, $\phi$ can approximate the oracle projection between the intra-modal distributions of both old and new embeddings for texts. Assuming that the intra-modal distribution of texts in the VLP embedding space can mirror that of images, we can simply apply the same $\phi$ to the new image embeddings to synthesize the corresponding old image embeddings.

With these generated synthetic old image and text embeddings, which can be considered as aligned with the new embeddings, we aim to fine-tune the new model. In this process, far fewer supervised image-text pairs are required than in the original training dataset of the VLP. This also allows us to avoid using old model parameters during training, thereby enhancing training efficiency. In addition, we incorporate parameter-efficient fine-tuning schemes [, ] into the new model, which add small trainable parameters and do not harm any of the new model’s original parameters during training. This not only accelerates the new model’s fine-tuning process but also allows for an easy reversion to the original new-to-new retrieval by simply removing the additional parameters.

We demonstrate our approach on various cross-modal benchmarks, highlighting the effectiveness of XBT in cross-modal retrieval protocols. In response to the rapid advancement of VLP models, XBT offers an environmentally friendly solution by enabling backfill-free systems.

Our contributions can be summarized as:

- •
The Cross-modal Backward-compatible Training (XBT) concept is introduced for the first time to solve the backfilling problem that stems from real-world cross-modal retrieval systems.
- •
A novel XBT solution is proposed, which uses a text-only pretrained projection module, $\phi$, to efficiently align the embeddings of new and old models using only text samples.
- •
The proposal is demonstrated on various datasets and protocols, showcasing XBT’s effectiveness in building backfill-free cross-modal retrieval systems.

## 2 Related Works

Backward-compatible Training. The concept of Backward-compatible Training (BT) was first introduced in the study []. This approach influences a new model with the learned classifier of the old model, thereby achieving backward compatibility between the old and new models. However, BT can degrade the original representational performance of the new model. To overcome this, [] proposed aligning class-wise centers presented by the old and new models. Another approach to achieve backward compatibility is to use an additional projection to map the old embedding into the new embedding space by adding a lightweight transformation, as suggested in [, ]. The work of [] further adds an auxiliary feature in preparation for future updates, while [] uses additional dimensionality in the embedding to obtain compatibility. An online strategy that backfills the gallery on the fly is explored in [], and [] addresses the model regression problem. Despite this progress, the scenario considering cross-modal retrieval between image and text, which has many real-world applications, remains unexplored. Our study on XBT in this paper aims to fill this gap.

Vision-Language Continual and Transfer Learning The fields of continual learning [, , ] and transfer learning [, ] bear similarities to backward-compatible representation learning, as all aim to update an existing model to boost performance. In the realm of multi-modal representation learning for cross-modal retrieval, continual learning approaches like [] propose methods to prevent catastrophic forgetting across different modalities. Transfer learning approaches like [] suggest strategies for transferring knowledge from previously labeled categories (source modality) to new, unlabeled categories (target modality). However, our proposed XBT stands apart from these methods as it specifically tackles the challenge of maintaining backward compatibility between old and new models. This unique attribute makes XBT ideally suited for backfill-free retrieval scenarios, where the embeddings of the old model remain unchanged, yet we can still leverage the enhanced performance of the new model.

## 3 Methodology

Our goal is to construct a backfill-free, embedding-based, cross-modal retrieval system using VLP models, which are configured with an image encoder $E^{I}$ and a text encoder $E^{T}$. When a new better performing VLP model $\{E^{I}_{new},E^{T}_{new}\}$ emerges, we aim to ensure its compatibility with the old model $\{E^{I}_{old},E^{T}_{old}\}$ that was used to construct the gallery. To achieve this, we introduce Cross(X)-modal Backward-compatible Training (XBT).
Given an image $x$ and text caption $t$, XBT enables retrieval between a new image embedding $v_{new}=E^{I}_{new}(x)$ and the text embeddings ($w_{old}$), as well as between a new text embedding $w_{new}=E^{T}_{new}(t)$ and the image embeddings ($v_{old}$). We denote backward compatible embeddings as $\bar{v}$ and $\bar{w}$. All embeddings we utilize in this work are l2-normalized.

### 3.1 Criterion for Cross-modal Backward Compatibility

Following the definition of backward compatibility in BT work [], we can construct strict constraints that ensure cross-modal backward compatibility as:

$$ $\displaystyle d(w_{new_{i}},v_{old_{j}})\leq d(w_{old_{i}},v_{old_{j}}), \forall y_{i}=y_{j},d(w_{new_{i}},v_{old_{j}})\geq d(w_{old_{i}},v_{old_{j}}), \forall y_{i}\neq y_{j},$ (1) $\displaystyle d(v_{new_{i}},w_{old_{j}})\leq d(v_{old_{i}},w_{old_{j}}), \forall y_{i}=y_{j},d(v_{new_{i}},w_{old_{j}})\geq d(v_{old_{i}},w_{old_{j}}), \forall y_{i}\neq y_{j},$ $$

where $y_{i}$ and $y_{j}$ represent whether an image and text are paired ($y_{i}=y_{j}$) or not ($y_{i}\neq y_{j}$). The term $d(\cdot,\cdot)$ represents a distance metric in the embedding space, and we choose cosine distance as the baseline. The constraints in Eqn. 1 formally express the requirement that the new embedding must perform at least as well as the old embedding in terms of correctly matching image-text pairs.

However, exhaustively satisfying these constraints is intractable due to the potential for the old embeddings to outperform the new embeddings in certain retrieval cases. As a result, we relax the criteria by defining an alternative evaluation metric as:

$$ $\displaystyle\mathcal{M}(w_{new},v_{old};\mathrm{Q},\mathrm{G})>\mathcal{M}(w_ {old},v_{old};\mathrm{Q},\mathrm{G}),$ (2) $\displaystyle\mathcal{M}(v_{new},w_{old};\mathrm{Q},\mathrm{G})>\mathcal{M}(v_ {old},w_{old};\mathrm{Q},\mathrm{G}),$ $$

where a metric, such as recall, $\mathcal{M}(q,g;\mathrm{Q},\mathrm{G})$ takes query embedding $q$ and gallery embedding $g$ over query set $\mathrm{Q}$ and gallery set $\mathrm{G}$. This criterion suggests that the general performance is enhanced when the query embedding from the XBT-trained new model is used to perform retrieval with the gallery of the old model, compared to the performance of the old model alone. In essence, fulfilling Eqn. 2 indicates that the new model has achieved backward compatibility and can feasibly update without backfilling gallery.

Additionally, it is important to highlight that VLP models function as zero-shot learners. This leads us to define the XBT problem differently from classic BT, which trains separate models for specific image domain retrieval tasks such as ImageNet [] or VGGFace2 []. In contrast, XBT tackles a more challenging task, aiming to bridge a new VLP model with a frozen old VLP model while retaining the new model’s original zero shot capability. In this paper, we therefore assess the performance of XBT using retrieval and classification benchmarks in a zero-shot manner.

### 3.2 Text-only Pretraining

Figure: Figure 2: An illustration of the text-only pretraining of $\phi$ (above), and XBT with $\phi$ (below). Using only text samples, $\phi$ is trained to approximate distribution of old text embeddings from that of new ones. During training, noise $\epsilon$ is injected to input of $\phi$. After pretraining, the same $\phi$ is used to generate both of synthetic old image and text embeddings from the new VLP embeddings and train to learn cross-modal backward-compatible representation.
Refer to caption: x2.png

The vast and diverse image-text pairs used to build VLP models significantly enhance their ability to connect semantically similar visual content and natural language []. However, this creates a complex embedding distribution that is difficult to predict and understand, thereby complicating the XBT process. A straightforward solution is to use a large number of supervised image-text pairs, similar to the approach used when building VLP models from scratch, to estimate the entire distribution of image and text embeddings during XBT.

However, acquiring a sufficient number of accurate supervised pairs is extremely costly. To be far more efficient, we employ a small sized projection module, $\phi$, and train it exclusively with text samples, as shown in Figure 2. We hypothesize that the distribution of text embeddings in VLP models, which is determined by their semantic similarity, is similarly mirrored in the distribution of their corresponding matched images. With this in mind, we aim to train $\phi$ to cover the broad spectrum of the text embedding distribution between the new and old VLP models, making embeddings from the same text sample similar and others dissimilar in a contrastive way:

$$ $\displaystyle\mathcal{L}_{pre}=\mathbb{E}_{t\sim D_{T}}[\mathcal{L}_{c}(\phi(w _{new}+\epsilon),w_{old};\tau_{pre})],$ (3) $$

where $t$ denotes text data sampled from text corpus $D_{T}=\{t_{i}\}^{N_{D_{T}}}_{i=1}$ of $N_{D_{T}}$ text samples, and $\mathcal{L}_{c}(\cdot,\cdot)$ is a standard batch-wise contrastive loss as defined in CLIP [] of temperature $\tau_{pre}$.

To prevent overfitting towards the text domain, which could widen the discrepancy between the image and text embedding representations, we inject noise $\epsilon$ sampled from a Gaussian distribution ($\epsilon\sim N(0,\sigma^{2})$), inspired by the approaches [, ]. Notably, $\phi$ is designed to take the new VLP model’s embeddings, which means that $w_{new}+\epsilon$ is also $l2$-normalized.

During this stage, only $\phi$ is updated, while all encoders remain fixed. By expanding the scale of $D_{T}$, we can effectively replicate the complete text embedding spaces of both new and old VLP models. Additionally, $\phi$ can be applied to new image embeddings to generate synthetic old image embeddings that closely approximate the actual old ones.

### 3.3 Cross-modal Backward-compatible Training

Figure: Figure 3: Our proposed learning process to achieve XBT. Notably, the old VLP model’s encoders are not required in this stage, enhancing efficiency in training.
Refer to caption: x3.png

Training Loss. The objective of XBT is to ensure cross-modal compatibility, specifically between $w_{new}$ and $v_{old}$, as well as between $v_{new}$ and $w_{old}$. For a given dataset $D=\{x_{i},t_{i}\}^{N_{D}}_{i=1}$ of $N_{D}$ supervised image-text pairs, we aim to train the new VLP model encoders, $E^{I}_{new}$ and $E^{T}_{new}$. Note that in our base setting, the text corpus is significantly larger than the supervised dataset, i.e., $N_{D_{T}}\gg N_{D}$.

However, the dimensionality of new and old VLP embeddings may differ, and even if they are the same, they are not be directly compatible. To address this, we apply the pretrained $\phi$ to ensure that the new embeddings match the dimension of the old ones and project into the compatible embedding space, as shown in Figure 2. This can be represented as:

$$ $\displaystyle\phi(v_{new})=\bar{v}_{new},\phi(w_{new})=\bar{w}_{new},$ (4) $$

where $\bar{v}_{new}$ and $\bar{w}_{new}$ denote synthetic old embeddings and $\{\bar{v}_{new},\bar{w}_{new},v_{old},w_{old}\}\subset\mathbb{R}^{K}$, with $K$ denoting the dimensionality of the old VLP model’s embedding.

We then apply XBT loss $\mathcal{L}_{X}$ as follows:

$$ $\displaystyle\mathcal{L}_{X}=\mathbb{E}_{x,t\sim D}[\mathcal{L}_{c}(\bar{v}_{ new},\bar{w}_{new};\tau_{X})].$ (5) $$

Here, $\mathcal{L}_{c}(\cdot,\cdot)$ is the same contrastive loss used in Eqn. 3, with temperature $\tau_{X}$. As shown in Figure 3, the new VLP encoders, $E^{I}_{new}$ and $E^{T}_{new}$, and $\phi$ are trained through $\mathcal{L}_{X}$ in an end-to-end manner. Ultimately, $\bar{v}_{new}$ and $\bar{w}_{new}$ become cross-modal backward-compatible with existing old gallery embeddings, $v_{old}$ and $w_{old}$, as all embeddings are distributed in the compatible space through $\phi$.

Efficient Training. The efficiency of XBT can be attributed to two design choices that *avoid*: (1) any dependence on the old model when conducting XBT, and (2) updating the new VLP model off the shelf. We have already seen how $\phi$ can effectively achieve (1). (2) is simply facilitated by applying small-sized additional parameters, namely Soft prompt [] and LoRA[].

To be more specific, we incorporate the concept of soft prompt tuning, as outlined in [, ]. Soft prompting is applied to the vision encoder $E^{I}_{new}$ by adding trainable prompts as input, along with image patch tokens. We exclude soft prompt tuning on the text side, as we use the text-only pretrained module $\phi$, which requires maintaining the original distribution of text embeddings. We adopt the LoRA strategy [] for the new VLP model’s encoders $E^{I}_{new}$ and $E^{T}_{new}$ to avoid modifying original parameters. These two factors, in conjunction with $\phi$, offer an on-off solution: we can retrieve old samples using additional parameters, and we can easily revert to the original model by removing these parameters for new-to-new retrieval.

Further, regarding $\phi$ during XBT (Sec. 3.3), we only fine-tune the parameters of layer normalization. This is done with the aim of preserving the knowledge learned during the text-only pretraining stage (Sec. 3.2) and facilitating easy adaptation towards the image-text joint representation space.

In the end, our XBT framework offers a parameter-efficient solution that enables rapid convergence with fewer training iterations (a single epoch is sufficient); maintains the new model’s power; and requires far fewer supervised training samples than the scale of VLP model pretraining (approximately 1$\%$ of the level required to train CLIP [] from scratch, which uses a 400M dataset, and 0.2$\%$ of the level required for LAION-2B based CLIP models [], which utilize a 2B dataset). All these help to avoid any full-scale retraining of the new VLP models, paving the way for XBT to simply use new VLP models off-the-shelf as mentioned.

## 4 Experiments

The evaluation of the proposed cross-modal backward compatibility includes image-text zero-shot retrieval. Pretrained VLP models are used as our baseline, with the goal to assess performance in a zero-shot environment by tuning a given pretrained new VLP model that is stronger than the old model. The results highlight the potential of XBT, showing improved performance when old model outputs are matched with the XBT-learned new model.

### 4.1 Setup

Model Training. Our XBT process involves two distinct training stages: text-only pretraining (See Sec. 3.2) and image-text supervised training (See Sec. 3.3). For the text-only pretraining stage, we utilize the text samples from a subset of the 115M filtered web-collected image-text paired dataset from BLIP [], comprising around 67M available pairs (58.2% of the total). We construct a text corpus $D_{T}$ in Eqn. 3 using these text samples. For the subsequent image-text supervised training stage, we use a smaller subset of 4M image-text pairs from the same dataset to construct a supervised dataset $D$ in Eqn. 5. It is important to note that these subsets are significantly smaller than the dataset scale (400M, 2B or more) used to build CLIP models [, ].

Model Evaluation. We validate the effectiveness of XBT with a benchmark comprising three popular image-text paired datasets for cross-modal retrieval evaluation. The first is the nocaps dataset [], which offers a diverse distribution of object names and facilitates a more detailed analysis. We employ the validation split of this dataset, which consists of images each paired with 10 relevant textual captions, totaling 4,500 images and 45,000 captions.

To evaluate XBT at a larger scale, we also include the Flickr [] and COCO [] datasets, which consist of images that are each paired with 5 relevant textual captions. For the Flickr dataset, we use the entire dataset, encompassing 31,783 images and 158,915 captions. For the COCO dataset, we use the validation split, which includes 35,136 images and 175,680 captions. We notate the two datasets as Flickr-31K and COCO-35K respectively for the remainder of the paper.

For retrieval, we utilize all the samples in each dataset for both query and gallery, following the cross-modal benchmark used in [, ]. For evaluation purpose, we adopt recall scores at top K retrieval results (R@K, %) to estimate the cross-modal backward compatibility (See Eqn. 2).

Implementation Details. In our work, we primarily use the popular CLIP models [], based on a Transformer [] backbone, as our baseline VLP models. The models, listed in ascending order of scale and performance, are: CLIP-ViT-B32, CLIP-ViT-L14, and CLIP-ViT-H14. For simplicity, we will refer to them as B32, L14, and H14, respectively. Notably, while B32 and L14 are trained with the same dataset at 400M scale, H14 is trained with a larger dataset at 2B scale. We apply additional LoRA [] parameters for each new model encoder configured as follows: $\text{LoRA}_{\alpha}=16$, $\text{rank}=16$, and $\text{dropout}=0.1$. The proposed projection module $\phi$ consists of three Linear layers with layer normalization (LN) [] and the GELU non-linearity function []. Detailed architecture is as follows: $\texttt{Linear}-\texttt{LN}-\texttt{GELU}-\texttt{Linear}-\texttt{LN}-\texttt{
GELU}-\texttt{Linear}$. Dropout is not applied as it was found to degrade performance empirically. The intermediate hidden dimension of the Linear layer is set to be four times the dimensionality of the output embedding. More details are in Appendix A.2.

### 4.2 Main Results

Table: Table 1: Cross-modal retrieval results on nocaps. Note that for comparison, the B32/L14 case should be compared with B32, and the L14/H14 case should be compared with L14, respectively.

Baseline Comparisons. To simplify notation, we use $w/v$ to denote the retrieval results obtained using $w$ as the query embeddings and $v$ as the gallery embeddings. We establish two protocols: assessing cross-modal backward compatibility with $new/old$ ($\bar{w}_{new}/v_{old},\bar{v}_{new}/w_{old}$), and maintaining the new VLP’s original performance with $new/new$ ($\bar{w}_{new}/\bar{v}_{new},\bar{v}_{new}/\bar{w}_{new}$). All of the above are similarly applied to $v/w$.

To the best of our knowledge, since this paper presents the first method that aims to solve the XBT problem, there are no direct previous works to compare to. Nevertheless, we compare our XBT against three baselines: Full-tune, LoRA-only, and Base, which we elaborate on below.

Naïve solution - Direct backward contrastive learning: The pretrained $\phi$ in Eqn. 3, which maps the new embeddings to the old via text based contrastive learning, constitutes a major contribution of XBT. We therefore would like to compare with a solution that does not utilize such a pretrained $\phi$. A straightforward solution for this is to fine-tune the new VLP encoders ($E^{I}_{new}$ and $E^{T}_{new}$) to be compatible with the old ones by minimizing the following loss:

$$ $\displaystyle\mathcal{L}_{Direct}=\mathbb{E}_{x,t\sim D}[\mathcal{L}_{c}(\bar{ v}_{new},w_{old};\tau_{N})$ (6) $\displaystyle+\mathcal{L}_{c}(\bar{w}_{new},v_{old};\tau_{N})]$ $$

where $\mathcal{L}_{c}(\cdot,\cdot)$ is the same contrastive loss used in Eqn. 5, and $\tau_{N}$ is a temperature. Here, we apply a randomly initialized $\phi$ with the same configuration as XBT for projection, as per Eqn. 4, to generate cross-modal backward compatible embeddings, $\bar{w}$ and $\bar{v}$.

Different training options: Based on Eqn. 6, we consider multiple training setups: (1) full fine-tuning of all trainable components (Full-tune), (2) tuning LoRA parameters-only (LoRA-only), and (3) starting from LoRA-only, adding an extra learnable prompt (Base). Models produced by Base would therefore be akin to XBT without the pretrained $\phi$. For these, we train the $E^{I}_{new}$, $E^{T}_{new}$ and $\phi$ using $\mathcal{L}_{Direct}$ with 4M image-text pairs. Note that, XBT only trains layer-normalization layers including pretrained $\phi$, however, Full-tune, LoRA-only and Base setups are training entire parameters of $\phi$ since it is not trained before. The results of these baselines are presented in Table 1, tested with nocaps dataset. The performance of the original VLP models is also reported for a clear comparison. We highlight our XBT method.

In the context of cross-modal backward compatibility, where recall scores of $\bar{w}_{new}/v_{old}$ and $\bar{v}_{new}/w_{old}$ should be higher than those of $w/v$ and $v/w$ of old model, respectively, Full-tune significantly underperforms compared to the old VLP’s. This indicates that full fine-tuning is not an appropriate solution. While both LoRA-only and Base improve performance over Full-tune, they still fall short of the old VLP’s performance and fail to meet the criterion. In contrast, XBT significantly outperforms these baselines and even surpasses the old VLP’s performance, satisfying the cross-modal backward compatibility for all recall metrics (see Eqn. 2).

In terms of maintaining new VLP’s performance, where recall scores of $\bar{w}_{new}/v_{new}$ and $\bar{v}_{new}/w_{new}$ could be similar to those of $w/v$ and $v/w$ of new model, respectively, all baselines show decent performance in $\bar{w}_{new}/v_{new}$ case but only XBT achieves better performance than new VLP model in $\bar{v}_{new}/w_{new}$ case. These results support that XBT not only maintains the performance of the new VLP model but also enhances it in certain cases. The superior performance of XBT in both cases underscores its promise as a robust solution for cross-modal retrieval tasks.

Table: Table 2: Cross-modal retrieval results on Flickr-31K.

Table: Table 3: Cross-modal retrieval results on COCO-35K.

Large Scale Retrieval. In this experiment, we assess the performance of cross-modal backward compatibility in more practical use cases. We use larger datasets, namely Flickr-31K and COCO-35K, as shown in Tables 2 and 3. These datasets provide more than six times the number of samples typically used in image-text retrieval literature [, ], which make the retrieval process more challenging. We use the second-best performing Base method from Table 1 as a basis for comparison. To validate the applicability of XBT across various VLP models, we utilize three CLIP baselines: B32, L14, and H14. Additionally, we explore different combinations, such as B32/L14 and B32/H14, to further analyze the effectiveness of XBT, beyond the comparisons made in Table 1 (B32/L14 and L14/H32).

When comparing B32/L14 with B32/H14, it is observed that every retrieval cases of improved models (L14 and H14) successfully maintain cross-modal backward compatibility, regardless of their scale. Especially, retrieval performance of B32/H14 is better than that of B32/L14, which demonstrates XBT’s ability to preserve and align original model’s power with old gallery.

Upon examining the results, we consistently observe that XBT outperforms the Base model across all instances and both retrieval scenarios, often by a significant margin. Furthermore, XBT maintains the performance of new VLPs better than Base, achieving even larger margins for $\bar{w}_{new}/\bar{v}_{new}$ and $\bar{v}_{new}/w_{old}$. In summary, XBT consistently demonstrates robust and superior retrieval performance, even in large-scale scenarios.

### 4.3 Further Analysis

Table: Table 4: Ablation study tested on nocaps, with B32 as old and L14 as new model.

Ablation Study. To validate XBT, we conduct further analysis as shown in Table 5. By comparing (a) and (b), we observe that introducing noise during $\phi$ training aids in generalization. The comparison between (c) and (d) demonstrates that the scale of $D_{T}$ is important, supporting our assumption that a sufficient number of text samples can help build a robust $\phi$. (e) outperforms (a), confirming that utilizing more image-text pairs can enhance XBT. The lower performance of (f) also aligns with the notion that the number of image-text pairs is crucial. In (g), when we replace $D$ with CC3M [], from which we can obtain around 2.4M, the performance is similar to (f), suggesting that XBT can be effectively applied with other datasets.

Different VLP models. To further explore the capacity of the VLP model architecture’s generalization of XBT, we evaluate it using BLIP [] Base and Large models. We employ checkpoints of Salesforce/blip-itm-base-coco and Salesforce/blip-itm-large-coco from the Huggingface library. We apply XBT on old and new BLIP models in the same fashion with our CLIP applications, and the results appear in Table 5. From the results, we confirm that XBT provides cross-modal backward-compatibility to the BLIP models too.

Figure: Figure 4: New query vs. Old gallery retrieval results on $nocaps$. B32 as old, and L14 as new model.
Refer to caption: x4.png

Qualitative Results. We facilitate retrieval by utilizing new query embeddings and old gallery embeddings. The results in Figure 4 demonstrate accurate cross-modal backward-compatible retrieval.

## 5 Discussion & Conclusion

Potential Broader Impact and Limitation. This paper highlights our efforts to enhance the field of Machine Learning, particularly in the area of multi-modal embedding-based representation learning. Although our work may have societal implications, we do not believe there are any that require specific emphasis in this context. A potential limitation of the XBT system is that, despite the efficient learning approach reducing the need for image-caption pairs, its performance may still be limited by the quality, diversity, and representational richness of the data during training.

Conclusion. In this paper, we introduced Cross-modal Backward-compatible Training (XBT), a novel task for cross-modal retrieval that focuses on the compatibility between image and text embeddings of different Vision-Language Pretraining (VLP) models. We proposed an efficient solution using a text-only pretrained projection module, $\phi$, to align the new model’s embeddings with those of the old model, thereby enhancing training efficiency. By integrating parameter-efficient training schemes into the XBT framework, we were able to accelerate the model’s training while preserving the original VLP’s zero-shot capabilities. Our approach, demonstrated on various cross-modal benchmarks, effectively builds cross-modal retrieval systems without backfilling, offering an efficient and environmentally friendly solution in response to the VLP improvements.

## Appendix A Appendix

### A.1 Dataset Examples, More Visualization

Figure: Figure 5: A tSNE visualization of 5,000 paired image-text embeddings from COCO [] dataset, using two different CLIP models [], and two different BLIP models []. Five pairs are marked as examples. The distinct distributions of image and text samples in each VLP space are observed.
Refer to caption: x5.png

In Figure 5, we use a t-SNE map to examine the actual distribution of embeddings in the VLP space. It’s evident that the image and text embeddings are distinct. Furthermore, the intra-distribution within both image and text embeddings is similar, suggesting that they are supposed to mirror each other.

### A.2 More Implementation Details

#### Training Recipe

We utilize 8-A100-80GB GPUs to train and evaluate the models. For the text-only pretraining stage in Sec. 3.2, the batch size is set to 8,192 (1,024 batch per GPU) and during this stage, the entire set of trainable parameters of $\phi$ are trained while both VLP text encoders are fixed. Moving on to the image-text supervised training in Sec. 3.3, the batch size is reduced to 1,024 (128 batch per GPU), and in this stage, layer normalization is set to be the only trainable component for all VLP image and text encoders with $\phi$. Temperature hyper-parameters $\tau_{pre}$, $\tau_{X}$, and $\tau_{N}$ are fixed at 2.6592. We employ the AdamW optimizer [] with a fixed learning rate of 1e-4 and a weight decay of 0.01. For soft prompts, we use 10 prompts and apply 100 times larger learning rate, 1e-2. The training iteration is determined by the dataset size, and the entire pipeline is trained for a single epoch. For image augmentation, we begin with a random resized crop, adjusting the image scale between 0.5 and 1.0. Additionally, we apply a random horizontal flip and make random adjustments to the image’s contrast, brightness, and sharpness. To incorporate different perspectives and angles, we modify the image’s translation and rotation. The pretrained weights provided by HuggingFace(^1^11https://huggingface.co/models) [] are applied to baseline VLP models. We employ as: openai/clip-vit-base-patch32, openai/clip-vit-larget-patch14, laion/CLIP-ViT-H-14-laion2B-s32B-b79K.

### A.3 Further Analysis & Discussion

Table: Table 6: Continual learning scenario experimental results with XBT on nocaps. Note that for H14 training, we use L14, which has been previously adapted with B32, and H14 never encounters B14.

Table: Table 7: Computational analysis on baselines. We evaluate with B32 as old and L14 as new model.

#### Continual learning.

In line with the literature on BT works [, ], we set up a continual learning scenario for a sequence of model updates (old model B32, new model L14, and better new model H14) and present the retrieval results in Table 6. Initially, we apply XBT to L14 to ensure compatibility with B32. Subsequently, we apply XBT to H14 to ensure compatibility with the previously learned L32, which is already compatible with B32. For this process, we divide both the image-text pairs and the text-only pretraining train sets into two halves, using each separately for each case. A comparison with the original results reported in Table 6 confirms that our XBT performs well in the continual learning scenario either. It achieves cross-modal backward compatibility while leveraging the power of the improved new model.

#### Computational Analysis.

In Table 7, we calculate the required training cost for each baseline. Despite XBT handling a larger number of training samples, the total training time (Text-only pretraining + XBT) is less than that of the other methods. Furthermore, since XBT does not utilize the old VLP model during training, it significantly reduces the memory load.

#### Research question: image-only pretraining.

There exists a possibility of training $\phi$ using only images, which would replace the text-only pretraining stage. We opted for the text-only scheme because text data is easier to obtain than image data and can provide explicit semantics. These include abstract concepts, logical reasoning, and specific details that may not be visually represented. Text can express nuances of language, sentiment, and context that can be challenging to depict in an image.

However, images can also convey visual semantics that are difficult to describe in words. They can provide a wealth of details about the visual world, including color, shape, texture, spatial relationships, and visual patterns. Images can also convey non-verbal information, such as emotions and actions, that can be difficult to express in text. Therefore, pretraining $\phi$ only with images and comparing it with the text-only approach would be an interesting direction for future research.

Table: Table 8: Zero-shot classification results on ImageNet [], ImageNet-R [], and ImageNet-Sketch []. $\bar{w}$ and $\bar{v}$ are used to compute scores, and accuracy (%) is metric.

#### Research question: Zero-shot Classification.

As we incorporate VLP models, an intriguing research question emerges: How do VLP models, fine-tuned with XBT, perform as zero-shot classifiers? To investigate this, we conduct a zero-shot classification using the text prompt ‘a photo of class name’. As demonstrated in Table 8, XBT outperforms the old VLP in classification performance, though it falls short of the new VLP. Interestingly, we observe that as the number of supervised training samples increases, so does the classification performance. This suggests the potential for XBT-tuned models to function as zero-shot classifiers given sufficient training samples. This opens up a new research direction towards not only achieving backward compatibility, but also comparable performance to zero-shot classifiers.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the weaknesses to check out the questions.
soundness: 2
strengths: - The motivation is meaningful. The paper seeks to avoid the time-consuming updating calculations on the embeddings of a cross-modal retrieval system when a new, improved model is introduced.

- The paper aims to develop a backfill-free cross-modal retrieval system and extends Backward-compatible Training from the vision domain to the cross-modal domain, potentially opening a new research direction.

- The effectiveness of the proposed method is supported by the experimental results.
summary: This paper introduces Cross-modal Backward-compatible Training (XBT), designed to address the challenges of upgrading vision-language models in cross-modal retrieval systems. Traditional methods require costly backfilling, where old embeddings must be re-computed with new models, but XBT aims to eliminate this necessity. The authors propose a text-only pretrained projection module that aligns new model embeddings with those of the old model, significantly reducing the need for extensive image-text pairs. This method enhances training efficiency and preserves the knowledge of the new model while allowing for effective fine-tuning. Experimental results demonstrate the effectiveness of XBT in various cross-modal benchmarks, offering a promising solution for backfill-free upgrades in retrieval systems.
weaknesses: - Poor formulation of the paper: There are incorrect citations in the paper. For example, in Lines 116 and 117, “(BT) was first introduced in the study Shen et al. (2020)” should be “(BT) was first introduced in the study (Shen et al. 2020).” Additionally, in Fig. 2, the captions refer to the wrong figures: "text-only pretraining" is on the left side, while XBT is on the right side, but the paper states "above" and "below."
- The backward-compatible representation seems counterintuitive. If the old model creates a flawed representational space, forcibly mapping the new, more powerful model to the old model's representation space could compromise the new model's representation. Could the authors discuss this issue?
- The baseline only considers the old model. As I understand it, the upper bound of a retrieval system when introducing a new, improved model should be based on the new model, while the old model serves as the lower bound.
- The retrieval performances of CLIP-ViT-B32 on Flickr and COCO are extremely low. According to the original CLIP paper, CLIP-ViT-L14 can achieve 88% R@1 on Flickr's text retrieval, while the performance of CLIP-ViT-B32 reported in this paper is 40.35%, which is over 40% lower than CLIP-ViT-L14. This lead to the experimental results unconvincing.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see weaknesses.
soundness: 3
strengths: 1. The concept of XBT is innovative and addresses a significant practical issue in the deployment of new models in real-world applications. The idea of using a text-only pretrained projection module to align embeddings is interesting and shows potential for efficiency gains.
2. The methodology is well-explained and technically sound. The authors have provided a clear outline of the training process, including the text-only pretraining and the cross-modal backward-compatible training stages.
3. The paper is well-supported by empirical evidence, with experiments on multiple benchmarks demonstrating the effectiveness of XBT in cross-modal retrieval tasks.
summary: The paper presents an approach to address the challenge of model upgrades in cross-modal retrieval systems by introducing Cross-modal Backward-compatible Training (XBT). This approach aims to align the embeddings of new Vision-Language Pretraining (VLP) models with those of the old models, thereby reducing the need for costly re-embedding of data, known as backfilling.
weaknesses: 1. While the paper claims that XBT reduces the need for image-text pairs, the scalability of the approach in very large-scale systems with real-world data distributions is not fully explored. The author only used a source dataset from BLIP (which is a sythetic dataset), a subset from LAION400M will be benificial for this work.
2. The paper briefly mentions the impact of data quality on XBT performance. A more detailed analysis on how noisy or biased training data might affect the embedding alignment could provide deeper insights.
3. The computational analysis is limited to a comparison of training times and memory loads. A more comprehensive analysis, including the trade-offs between accuracy and computational resources, would be valuable.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	Can you show the similarity between new and old representations after φ's transformation for both image and text? Is image similarity lower than text similarity? What is the relationship between similarity and performance? How does noise magnitude in φ affect similarity?
2.	In Table 1, B32 is used as an old encoder model in the original column. Why does B32 show better performance after XBT training compared to the original B32? Can XBT improve the performance of old retrieval systems?
3.	Could the tables be reorganized into three parts to show the retrieval performance of:
-	Old systems
-	New systems
-	Proposed backward-compatible new systems
soundness: 2
strengths: 1.	Comprehensive Experimentation: The paper demonstrates strong empirical validation through extensive experiments across multiple models and datasets. This thorough evaluation effectively showcases the method's generalization capability and robustness across different scenarios.
2.	Novel Perspective: The authors present an innovative approach by proposing that aligning a single modality can potentially lead to cross-modal alignment. This insight offers a fresh perspective on handling multimodal compatibility issues and could inspire new research directions in the field.
3.	Practical Relevance: The work addresses a real-world challenge in deploying updated multimodal systems while maintaining compatibility with existing infrastructure, making it valuable for practical applications.
summary: This paper presents a cross-modal backward-compatible training (XBT) method that enables the deployment of new multimodal encoders while maintaining compatibility with existing retrieval systems. The key innovation lies in its approach to align a single modality to potentially achieve cross-modal alignment, offering a practical solution for real-world system updates. The work addresses an important practical challenge in maintaining backward compatibility while upgrading multimodal systems.
weaknesses: 1.	Some citation formats are incorrect.
2.	Most performance tables are confusing, making it difficult to clearly understand the performance of both old and new retrieval systems.
3.	Lack of several analytical experiments, which are detailed in questions.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why is the text-only pre-training necessary?

2. Why preventing overfitting in the text-only pre-training stage is needed ? Are there any explanations for that?
soundness: 3
strengths: 1. The targeting problem is interesting. And it is might be useful for the real applications.

2. The proposed method achieved expecting performance.
summary: The authors propose a novel approach called Cross-modal Backward-compatible Training (XBT), which extends the concept of Backward-compatible Training (BT) from vision-only to cross-modal retrieval, specifically for aligning Vision-Language Pretraining (VLP) models like CLIP.   A text-only pre-trained projection module, ϕ, is proposed to map new model embeddings to old model embeddings. This module is trained solely with text data, reducing the need for image-text pairs. The paper utilizes parameter-efficient training strategies that improve efficiency and preserve the new model's knowledge by avoiding modifications to it. Experiments on cross-modal retrieval datasets show XBT's effectiveness and its potential to enable backfill-free upgrades when new VLP models emerge.
weaknesses: 1. It seems that the motivation of the text-only pre-training stage is not very clear.

Meta-review: The paper presents an approach to address the challenge of model upgrades in cross-modal retrieval systems by introducing Cross-modal Backward-compatible Training (XBT). This approach aims to align the embeddings of new Vision-Language Pretraining (VLP) models with those of the old models, thereby reducing the need for costly re-embedding of data, known as backfilling.

Strengths:
+ Comprehensive Experimentation: The paper demonstrates strong empirical validation through extensive experiments across multiple models and datasets.
+ Novel Perspective: The authors present an innovative approach by proposing that aligning a single modality can potentially lead to cross-modal alignment.
+ Practical Relevance: The work addresses a real-world challenge in deploying updated multimodal systems while maintaining compatibility with existing infrastructure, making it valuable for practical applications.

Weaknesses:
+ The presentation needs to be further improved, some tables or figures are not clear. The motivation for the text-only pret-training stage needs to be highlighted.
+ More ablation studies are expected. The computational analysis is limited to a comparison of training times and memory loads. A more comprehensive analysis are needed, including the trade-offs between accuracy and computational resources.
+ The experimental results are unconvincing.

Decision: Reject