$\boldsymbol{\lambda}$-Orthogonality Regularization for Compatible Representation Learning
Authors: Simone Ricci
Niccolò Biondi
Federico Pernici
Ioannis Patras
Alberto Del Bimbo
Venue: NeurIPS 2025 poster ()
Keywords: Deep Learning
Representation Learning
Compatible Learning
Primary area: other
TL;DR: We ensure backward compatibility through multiple transformations and a relaxed orthogonality constraint for distribution-specific adaptation.

Abstract: Retrieval systems rely on representations learned by increasingly powerful models. However, due to the high training cost and inconsistencies in learned representations, there is significant interest in facilitating communication between representations and ensuring compatibility across independently trained neural networks.
In the literature, two primary approaches are commonly used to adapt different learned representations: affine transformations, which adapt well to specific distributions but can significantly alter the original representation, and orthogonal transformations, which preserve the original structure with strict geometric constraints but limit adaptability. A key challenge is adapting the latent spaces of updated models to align with those of previous models on downstream distributions while preserving the newly learned representation spaces.
In this paper, we impose a relaxed orthogonality constraint, namely $\lambda$-Orthogonality regularization, while learning an affine transformation, to obtain distribution-specific adaptation while retaining the original learned representations.
Extensive experiments across various architectures and datasets validate our approach, demonstrating that it preserves the model's zero-shot performance and ensures compatibility across model updates.  Code available
at: \href{https://github.com/miccunifi/lambda_orthogonality.git}{https://github.com/miccunifi/lambda\_orthogonality}.

21). Furthermore, the A-Orthogonality Regularization for Compatible Representation Learning
Simone Ricci
Niccolò Biondi 1.2
Federico Pernici 1.2
Ioannis Patras
Alberto Del Bimbo
DINFO (Department of Information Engineering), University of Florence, Italy MICC (Media Integration and Communication Center)
SQueen Mary University of London, UK
Abstract
Retrieval systems rely representations learned by increasingly powerful models. However, due to the high training cost and inconsistencies in learned representa- tions, there is significant interest in facilitating communication between represen- tations and ensuring compatibility across independently trained neural networks. In the literature, two primary approaches are commonly used adapt different learned representations: affine transformations, which adapt well to specific dis- tributions but can significantly alter the original representation, and orthogonal transformations, which preserve the original structure with strict geometric con- straints but limit adaptability. A key challenge adapting the latent spaces of updated models to align with those of previous models on downstream distributions while preserving the newly learned representation spaces. In this paper, we impose relaxed orthogonality constraint, namely X- --Orthogonality regularization, while learning an affine transformation, to obtain distribution-spent adaptation while retaining the original learned representations. Extensive experiments across various architectures and datasets validate our approach, demonstrating that it preserves the model's 's zero-shot performance and ensures compatibility across model updates. orthogonality.
Code available at: httpss//gghubb.cc/mmmuum///a/aa
Introduction
Retrieval tasks are increasingly relevant real-world applications such as face recognition image localization 4 6 and object identification [ 87,8, 9. In image retrieval, gallery of labeled
images is matched to query images to identify related ones, ideally of the same class. Instead of high-dimensional images, retrieval uses low diwensinnional feature vectors obtained from embedding models. Enhancing retrieval performance often involves updating embedding models [T0 11 to leverage more expressive network architectures [22, new training techniques (e.g.. loss functions) or training paradigms
14 [5]. However, neural networks rarely produce compatible features, even when trained on the same data with identical methods and architectures [16). Consequently, matching the features of new queries with those of older galleries can degrade retrieval performance due to incompatibility [15]. To address this, replacing the gallery features generated by the old model with those produced by the new model-a computationally expensive process known as backfilling- is required. The challenge of updating base model while ensuring its backward compatibility and avoiding backfilling has been extensively investigated [7
[
[5
"Corresponding author: simone. Aacci@unifi..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Makes quarry
Gallery
Pihaa)
Aiinnd
b
Genensta
B
)
hold
heow
B
Aligned
Darw
Image
2o 2000
Pold
Pnew
P(haa)
B1 (barv)
Old Model
Model
Overview of the proposed approach for achieving representation compatibility during retrieval system updates, A newly independently trained model is aligned to the old representation space via an orthogonal transformation В1, which preserves geometric structure. A forward transfor- mation F maps the old representations to the backward-aligee aard-aligned space of the model. Only the transformation parameters are optimized during training, while model parameters remain fixed.
Figure 1:
optimal strategy for gallery updates-knnwwn as partial backfilling -as recently begun to receive attention (22).
Architectural changes and additional losses to ensure compatibility can reduce the performance of the updated model [3 24 To address this issue, research has focused aligning the representation of a base model with that of an improved independently trained model using parameter effficient adapters [22 225). On the other hand, the manifold hypothesis [26] 277 suggests that neural networks typically produce latent space representations of identical data distributions that differ primarily by transformation. Consequently, mapping one representation another requires only few parameters, as functionally equivalent models approximate the same latent manifold (28] 29 [27]. Thus, simple
transformation aligning the new representation space to the previous one can provide the backward compatibility of the updated model.
Recent studies have focused affine and orthogonal mappings to adapt the latent space of a base model (source space) to that of another model (target space), using specific data points as reference
Within the plasticity-stabilii paradigm (32, affine mappings offer high adaptability (plasticity) but may alter the source space's configuration [33 34). Conversely, orthogonal mappings maintain the source space's geometric structure (stability). though they offer adaptability different distribution. To preserve the geometric structure of the source space, particularly when more informative than the target space (28 35), while enabling adaptability, we propose novel regularization term. Different from previous work (36), our term constrains transformation to remain within specified proximity the orthogonality condition, controlled by a hyperparameter
30 28
B1
In this paper, we address the challenge of ensuring compatibility between independently trained models by learning different transformations across representation spaces, illustrated in Fig. T Our contributions are summarized as follows:
We propose J-Orthogonality regularization, relaxed orthogonality constraint that retains the original representation space's global structure while enabling slight local adaptations for downstream tasks.
We enhance representation compatibility by employing supervised contrastive loss, which promotes intra-class clustering and inter-model alignment of feature representations, while remaining agnostic model architecture.
We conduct extensive experiments across diverse architectures and datasets, demonstrating that our method not only ensures compatibility between models but also promotes the preservation of the base model's 's latent space geometry, resulting improved accuracy on downstream tasks.

---

We propose novel trchitecture-agnosee backfilling strategy that improves retrieval perfor- mance while optimizing the gallery update process.
2
Related Works
As demonstrated by [16], feature representations from two models even if trained on the same data do not generally coincide, creating costly backfilling in retrieval systems. To avoid this, [5] introduced Backward Compatible Training (BCT), which keeps the old classifier fixed as reference new embeddings align with prior class prototypes. Additionally, they provided formal definition of compatibility between model representations. Subsequent research has expanded this foundation, incorporating additional regularization techniques to better align new representations with previous ones [21 37, 220 (88 [9) and implementing specific architecture design [88 13 40). However, the performance of the updated backward compatiiioo models frequently falls to reach that of models trained independently (23], consequence of the regularization imposed to achieve compatibility. To avoid this, [23] and 24 suggested expanding the representation space to include new classes while ensuring that the representations of old classes remain aligned during updates. To ensure compatibility between models trained independently, mapping- -ased strategies have been developed A1
42 43 Forward Compatible Training (FCT), as detailed by 251, introduces function that aligns embeddings from an older model to those of newer model's lss space, incorporating additional side information for each data point. As noted by (25), the computational overhead of these transformations is minimal compared to the demands of processing images through the embedding model. FastFill 22 improves forward transformation learning by using new model classifier and proposes a Bayesian strategy to optimize the gallery backfilling process leveraging the new model. In contrast, we propose set of transformation functions to ensure not only forward but also backward compatibility during model updates, with a particular focus on the orthogonality property in backward mappings. Additionally, we propose a supervised contrastive loss that promotes intra-class clustering and inter-modality alignment, thereby enhancing adaptation. Finally, we propose a novel gallery backfilling strategy based on distance metric that directly operates on pre-extracted gallery representations, making it agnostic to the underlying architecture.
Method
To achieve compatible representations between independently trained models, we introduce a theo- retically grounded pipeline composed of multiple transformations. First, in Sec. 3.1 we report the definition of compatibility introduced by [15]. In Sec. 3.2 [.33] we introduce a novel backward- ompatibility method, which aligns the new model's representations to those of the previous model using either strict orthogonal transformation or when adapting to downstream task transformation regularized by our proposed X-Orthogonality constraint. Next, in Sec. 3.4, we present forward trasfor- mation learning, which aligns the previous model's representations to those of the newly adapted model via affine more complex transformation, enabling effective gallery set updates. We also apply a supervised contrastive loss (Sec. 3.5) during transformation training to improve alignment between model representations and enhance intra-class cluster compactness, thereby satisfying the compatibility criterion defined in Def. 3.1 Finally, in Sec. 3.6 we propose novel ordering strategy for backfilling the gallery with improved representations in optimized sequence. Throughout methodology, all models serve as fixed feature extractors with frozen parameters, while only the transformation layers are trained.
3.1 Backward-Compatible Representations Definition
The formulation of Backward-Compatiin between representations, introduced by [5] closely related to the concept of latent space communication between different models (30] The formal definition of backward cookpaaidcoopttiblee representations specifies;
Definition 3.1 (Backward-Ce (Backwarr-Comppatiitty. The representation of a model learned at step ls is compat- ible with the representation of distinct model learned at subsequent step t, where k < t, if the following condition is satisfied:
Vi.j: (y: y
d(hi, h²) <
A (u. t = d(b),h)) 2 d(h h3) (1)
3

---

1.0
1-0
0.8
9.6
0
10907
189 10
12
30
85
Angle (Degroes)
99
100
(a) Value of Eq. at different A.
(b) Effect of different at A 6.
(c) KDE of W angles.
Figure 2: Impact of A-Orthogonality regularization on affine transformations. Fig. 2a shows the variation of Eq. 6 for different values of X, demonstrating the influence of the threshold in the regularization. Fig. 2b illustrates the effect of varying a while keeping = 6, highlighting its behavior in the sigmold function. Fig. 2c presents the kernel density estimation (KDE) of angles between the columns of matrix W for different values of X, showcasing the impact regularization orthogonality preservation.
where d(:,-) is distance function and y/ and yj are the class labels associated with the extracted representation vectors h, and hj. respectively. The inequalities Def. 3.1 indicate that the new model's representation, when compared against the old representation, should perform at least as well as the previous model's ' in clustering images from the same class and separating them from those of different classes.
3.2 Backward Transformation
One of the contributions of relative encoding [30 the observation that representation spaces, in practice, often differ only by an angle-prees preeeeepreeriinggg transformation when they share the same similar data semantics. Furthermore, (2 demonstrates that when there is a difference in learned semantics, transformation that preserves both angles and distances--earnne yields superior performance in gross-architecture and cross-modality classification tasks than only angle-preserving mappings. transformation T is defined distances between any two points and the space. Formally, a mapping : R - Rn is an isometry the following condition holds: [[T((a) - T'bbll2 ||a bll2, Ya, E R", where || ll2 denotes the Euclidean norm, or equivalently, general distance metric in other spaces. We leverage this property to achieve backward-compatible representations, aligning the updated model's space with the base model's 's using an orthogonal transformation. This maintains unified representation space across updates, preserving the geometric properties and performance the updated model due to the isometric nature of the transformation.
with Procrustes analysis [44
isometry if it preserves angles and
Given base model t and its updated version d', with & < f, and their corresponding representation vectors h² E TRd and E IR", we learn an orthogonal transformation B_ R.n - R that maps the embedding space of the updated model into the space of the base model. To enforce strict orthogonality, a generic transformation B is parameterized as the matrix exponential of skew-symmetric matrix such that B ef, where the upper triangular entries of P are learnable parameters [5). To enforce alignment between the updated and base representation spaces, we optimize the transformation B1 by minimizing the Mean Squared Error loss between h² and the transformed h':
Ln = B (h') 10113
As the transformation B1 a square matrix, the dimensionalities of the two representation spaces differ, the higher-dimensional feature vector is truncated to match the dimensionality of the smaller representation space.
3.3 X-Orthogonality Regularization
A strict orthogonal constraint (high stability) transformation B might not be ideal when model distributions vary from those on which the adapter trained- the case of private models providing only their extracted embeddings to the user. Imposing such constraint can limit the integration of

---

(a) Source space
(b) Target space
(c) Affine
(d) Orthogonal
(e) X-orthogonality
Figure 3: Effects of affine (Fig. 3c), strictly orthogonal (Fig. 3d). and X-orthogonality (with À 1) regularized (Fig. 3e) transformations trained to align a source representation space (Fig, (3a learned with a LeNet model (embedding dimension 2) on the complete MNIST dataset, with target representation space (Fig. 3b) learned on the first five classes of MNIST using the same architecture.
new, relevant information for downstream tasks. Conversely, an affine transformation (high plasticity)
without geometric regularization may disrupt the updated model's representations [46 47).
As
described in [36], soft orthogonality constraint can applied to the transformation B
R n
R
consisting of weight matrix W E R=**" and a bias term b E R". Previous works 48
49 50] have
proposed constraining the Gram matrix of the weight matrix be close to the identity matrix by minimizing a loss function defined as:
Forth |WTW- -||e
where || ||| denotes the Frobenius norm and W is the weight of the transformation B. This can be interpreted as a weight decay term that restricts the set of parameters to lie close to Stiefel manifold [50] However, this approach does not provide control over the specific level of orthogonality that can be imposed on the transformation.
To this end, we introduce a threshold A, which specifies the desired proximity of the Gram matrix of the weight matrix the identity matrix. A naive solution would be to stop the optimization of the Lorth loss once the Gram matrix of the weight matrix reaches the threshold X, as the loss directly influences the Gram matrix:
min IWTW- T|l s.t. |WTW- T|r 2
This objective can be achieved directly through the use of a Heaviside step function 51 shifted by the parameter X: H(x A) control the degree orthogonality during the minimization process, that effectively deactivating the regularization term in Eq. when the Frobenius norm exceeds the threshold X:
1 This function H offers efficient mechanism to
Lx H(IWW* - I|FF 2) |WWT I|]
However, this approach introduces a discontinuity in the loss function, as highlighted by [53] In particular, their work focuses on evaluating the closeness of these sigmoid functions to the Heaviside step function, providing precise upper and lower bounds for the Hausdorff distance. Building their theoretical and empirical analysis, we propose a smooth modulating function that ensures the effect of the constraint is gradually adjusted, with the penalty becoming more less significant depending the distance from the threshold A. Specifically, we formulate a novel X-Orthogonality Regularization term by optimizing loss function defined
Lx = a (a ([WWT -
A)) |WWT -[]
where (-)) (-) is the sigmoid function, and a is a scaling factor. The sigmoid function acts as a continuous switch that gradually turns the regularization term on and off near the value of X, as shown in Fig. 2a Instead, the scaling factor a controls the steepness of the sigmoid function, which in turn determines how sharply the regularization is activated or deactivated as the value of ||WWT - Ill approaches the threshold A. In Fig. 2b we illustrate different levels of steepness applied to the regularization loss. As increases, its behavior converges more closely to the Heaviside step function.
To further analyze the behavior of the X-orthogonality regularization, we optimize Eq, 6 applied to a transformation B with randomly initialized weight matrix W. As shown in Fig. 2c the kernel density estimation (KDE) of these angles changes based on the value of À used in the regularization. Smaller values of À result in column vectors that become increasingly orthogonal, specifically when
5

---

À = 0 our regularization is equal to Eq 3 Fig. illlssratee the effects of an affine (Fig. Bc), strictly orthogonal (Fig. [d). and X-orthogonality regularized (Fig. (3e] transformations trained align source representation space (Fig. 3a learned on the full MNIST dataset with target representation space (Fig. 3b) learned on the first five classes of MNIST. The toy experiment shows that the X-orthogonal constraint improves alignment by relaxing strict orthogonality while encouraging preservation the source feature space structure, in contrast to an unconstrained transformation.
3.4 Forward Transformation
In addition to a backward transformation that maps the representations of the new model to those of the previous model, it is possible formulate forward transformation F IRd R This transformation maps the representation vector h* E IRd of the previous model to he E IR", the representation of the new model. Since the representation of the new model is superior that of the previous model, the transformation F' should be affine (high plasticity) or multiple projection layers to better adapt to the improved representation. The transformation F is learned by minimizing the Mean Squared Error between the two representations, as ||4"h²)) - h'll following the approach described in [25]. This concept is closely related to latent space communication
30 31 where
d hb) = d h4, T h²) with T is a generic transformation. In previous approaches 25 22
the old representations h are aligned directly with the new he through transformation F, but incompatibility arise between h and F(h*). As mentioned in Sec. 3.2 a backward orthogonal transformation B realigns new representations with the old ones. Instead of adapting old features directly to new representations h', we adapt them to Bi (h'), ensuring a unified alignment across model updates. Furthermore, the transformations F and B can be trained jointly, they utilize the same training data. Consequently, the forward alignment loss in our methodology is defined as:
LF [|F(hh) B1 (
If the extracted representations are derived from a dataset different from the training sets of the two models, as discussed in Sec. 3.3 a X-orthogonal regularized transformation Bx can be employed in place of the strictly orthogonal B1.
3.5 Intra-class Clustering and Inter-Model Alignment
As discussed in Sec. 3.1 the compatibility inequalities defined in Def. 3.1 require not only alignment but also higher concentration of clusters to achieve compatibility. To this end, [22] introduces an additional training loss, Ldiscs that, unlike the influence loss in [15], relies directly on the new model's classifier rather than the old one. However, Laise depends access the new model's classifier and training loss, limiting its applicability, especially when the new model's s architecture is unknown (e.g., embedding vectors from private online models). To overcome this, we propose the use of a supervised contrastive loss, applied directly to representation vectors. This loss requires no classifier or architectural knowledge, it directly leverages representation vectors for alignment and clustering. The supervised contrastive loss (54) minimizes the cross-entropy loss between qi and p;:
K Pa log q:
Leentr
Σ
(
where qi denotes the probability assigned to sample by applying a temperature-scaled softmax over the dot-product similarities between the L22nnrmalized feature h and each other candidate, and P, is the normalized ground indicator distribution that places equal mass on all semantically matching (same-class) candidates and zero all others. Specifically, we utilized a combination of this loss function, where the objective Le is defined as:
Lc Ceonar
(F(n (F(h*). B1 (h')) C (F(h", .ht) h*)
This loss encourages clustering among the adapted representations while also aligning them with those the previous model, thereby promoting intra-class clustering and inter-model alignment of feature representations.
The overall loss function of our framework is defined as a weighted sum of four components: the forward alignment loss LF. the backward aligmment loss LB. the contrastive loss Lc, and the X- A-Othhoonaality regularization term Lx. Formally, the total loss is expressed as:
L = wi LF w2 La + w3 Lo + Lx
(10)

---

Table I: Compatibility evaluation ImageNet K under two scenarios: (a) Extending classes setting, (b) Architecture update setting. For each case (highlighted with different colors), we report CMC- Top1 and mAP metrics.
(a) Extending classes setting. Two models trained in- dependently: daus first 500 classes, and &new on full ImageNetIK, K. Both use ResNet-34 with an embedding dimension 128.
(b) Independently Pretrained Models setting: Two models trained independently the full ImageNet1 K dataset. The first model, chil, is ResNet-18 whereas the second, Srews is ViT-L-16..
Method
Query'Gallery
CMC-Topl
mAP
Method
Query/Gallery
CMC-Topl
mAP
Goal Posl
43.56
Daaa/tass fhotz.
55.62
26.91
Ind. Train.
0.10
0.15
Ind. Train.
Snen:
0.04
0.17
Ouee /nen
61.61
35.69
Orenl (uucn
76.62
56.84
F(chaa) / ddaa
0.10
0.15
F(cras) /Gaad
0.04
0.17
FCT
F [(has)) (d(a))
50.13
FCT
F [(oate (((aa))
59.39
42.65
4now F(
57.21
33.00
Ouow F((aa))
72.54
49.85
F(craa) / /oaa
0.10
0.15
F( doaa
0.04
0.17
FastFill
22
F [Daa)) (Qvas)
50.63
31.48
Fastlill
((osaa)) (dhas)
61.17
46.28
Unew F(
57.21
33.19
Shee F((a))
73.33
52.83
F(chaa)
44.59
26.70
F(crra)) Oran
60.83
40.69
F (Oaa)) F (dua)
$1.46
33.75
F F(haaa) F P(duaa)
61.10
45,91
Ours
B1 1 (Pren) )/ / F(chos)
57.41
34.53
Ours
B3 (fPaew ) F(Coss)
73.53
52.06
(dren) Oois
43,94
25.75
B3 (Oren)) Pote.
65.54
38.55
B. (0 B3 (Pren)
61.61
35.69
B3 (dren (Cron)
76.62
56.84
where W1, W2, and Wt denote scalar weights used to balance the contributions of each term
3.6 Partial Backfilling Strategy
Determining an effective ordering for backfilling samples the forward-adapted gallery set, where F(h²) from the old model is replaced by B1 (h'), is critical for achieving the performance of the new independently trained model as efficiently as possible. However, identifying the optimal ordering of backfilling represents computationally intractable combinatorial problem [22]. To address this challenge, FastFill [22 introduces an ordering inspired by Bayesian Deep Learning. This approach models the alignment error multivariate Gaussian distribution and minimizes the negative ligllikel of this distribution during the training of the mapping function F. However, from retrieval perspective, the most representative instances-those that significantly enhance the separation between distinct classes-ard -re identified as the embeddings closest to their respective class means (55 [56) Accordingly, prioritizing the backfilling of the least informative embeddings will increase the system's performance by reinforcing class distinctions. To this end, we propose a novel method for estimating backfill ordering based directly on the already extracted representation vector F F(h*). First, we calculate the mean representation vector μtc for each class e in the forward- adapted gallery set. Then, we compute a distance metric d of each embedding vector F F(h*) from its corresponding class mean pt. For instance, can the Mean Squared Error, = [h Pell2- Gallery embedding exhibiting the largest distance d from μ are prioritized for backfilling, thereby facilitating the matching with queries generated by the new backward adapted independently trained model B (h').
Experiments
4.1 Image Retrieval Compatibility
Backward compatibility is crucial in retrieval tasks involving gallery set G = ((x,,))) 2 and
query set Q {(x, y:) - each containing Ng and N, images respectively, with associated class labels. A base model indexes the gallery by extracting feature vectors from the images, which are then used to match with vectors from the query set in retrieval tasks. The compatibility definition presented in Def. 3.1 involves computing pairwise distances between all datapoints in the dataset. This process becomes increasingly computationally demanding as the dataset size grows. Then, model updated step t is considered backward-compattee with the base model trained at step & if
7

---

Table Compatibility results for two models pretrained on ImageNetI and adapted to downstream tasks: Oota. a ResNet-18, and &news ViT-L-16, using as backward adapter Bx with À = 12. The ZS column indicates the C performance increase on ImageNetIK, with values in parentheses indicating the increment compared to the newly independently trained model. Each Query/Gallery case is highlighted with different color facilitate comparison of results.
Dutaset
Method
Query/Cialler
CUB
CIFARI00
CMC-Topl
ZS
CMC-Topl
Zs
data/ Coold
44.82
51.13
Ind. Train.
Ohew Ghad
0.4
0.8
Ouen! PPac
71.78
74.08
F F(oaa)) Tous
FCT A5
F F(deaa) F(
51.10
57.35
Orew F(
62.13
69.80
F(eva)) /oaa
0.8
FastFill
2
F(dcaa) / (((daa)
54,50
66.17
Orcu F(
61.49
67.23
51.12
67.29
F F(dcaa):
/ F((aaa)
59.92
67.72
Ours
Bylthen (Pous)
70.72
72.08
Ox(0aaw )f /Oas
60.64
71.85
Ba Ba(n (x(ppw))
75.44 (+3.66)
+0.025
78.23 (+4.15)
+0.112
the Empirical Compatibility Criterion 15 is satisfied:
M ( +g) > M ( 9g), with A <
(1I)
where M denote a performance metric, Φe and Ω represent the extracted gallery and query sets, respectively. Specifically, M ( Φ assesses cross-model retrieval with gallery features from the updated model step & and query features from step k. contrast, M ( 2 +8) refers to same-model retrieval, where both gallery and query features originate from the same model at step k.
Partial Backfilling. Given an ordering π of the images in the gallery set &9, denoted as X Xmy2 ..... X and backfilling fraction 3 [0, 1]. we define the partially backfilled gallery set aT as follows. The first Ng.0 [BNN images in the ordering model, while the remaining images are processed using the old model. Here, N2 denotes the total number of images in the gallery. evaluate different backfilling strategies, we employ the backfilling metric M, introduced in [22), which is defined as: M(55, Φ2, T) = 1. ( 3, Φ8). This metric is the area under the backfilling curve when evaluating performance using M.
processed using the updated
4.2 Evaluation Metrics and Datasets
evaluate performance using two metrics. The Cumulative Matching Characteristics (CMC), which measures top-k retrieval accuracy by computing distances between query and gallery features, considers retrieval successful if least one the closest gallery images shares the query's label. The mean Average Precision (mAP) measures the area under the precision-recans curve across the full recall range [0, 1].
Following prior work model compatibility [15 25
To validate our approach, we utilize the following datasets: ImageNet [57], CIFAR100 (88) and CUB200 [59]. Each dataset's validation/test set serves both the query and gallery, with each query image removed from the gallery to avoid trivial matches during search. The notation 'Query/Gallery' indicates the models used for extracting embeddings in all tables, respectively. CUB200 and CIFARI00 are employed as downstream tasks.
4.3 Extending Classes Setting
In this setting, we update base model by extending the number of classes. We train two models independently: Pota on the first 500 classes and Φnew on all 1000 classes of ImageNet K, both using ResNet-34 architecture with embedding dimension of 128, following Py Torch's standard training

---

Table 3: Extended Classes setting
M
Method
CMC Top
mAP
2015
33.57
PCT
60.49
35.59
Ours
61.20
36.46
0.2
23
150
27
88
100
Table 4: Independently Pre- trained Models setting
0.58
M
Method
CMC-Top1
MAP
0.14
052
FCT 25
73.86
52.02
FastFill
28
75.06
55.34
0.5
23
100
76.59
57.72
Figure 4: Partial backfilling results for the Extending Classes setting (top Figures) of Tab. la and Independently Pretrained Models setting (bottom Figures) of Tab. [b] We use features from the new model &new for the query set (otherwise B (fnew) if trained). For the gallery set, we begin with forward-adapted old features F F(oaaa) and incrementally replace them with new features.
recipe After training the two models independently, adapters are optimized using Adam with a learning rate of 0.001, while keeping the model layers frozen. We compare our method against
FCT (25 and FastFill (22). two mapping methods used to achieve compatible representations. In Tab. [a the performance of each method summarized following the metrics of Sec. [4.2] The results indicate that the new model, &new. is not directly compatible with the old one Oula- Additionally, the two mapping methods, FCT and FastFill, enhance performance across both metrics for the adapted representations of the gallery and query sets. However, these methods achieve backward compatibility with the newly trained model but not with the original one. In contrast, our method aligns the new model with the old one through the orthogonal transformation B This ensures compatibility between the new and old representations and also enhances the performance provided by the forward adapter F. Appendix A provides additional results on Places365 66) dataset.
4.4 Independently Pretrained Models adapted on Downstream Task
Due to escalating training costs, pretrained models are increasingly used, especially for adapting to downstream tasks with local datasets. In this context, employ two mrodels-availabe in the PyTorch aub-pretrained on the ImageNetl dataset: a ResNet-18 with an embedding size of 512, and more advanced Vision Transformer (ViT-L-16) [61] with embedding size of 1024. The ViT model is considered an update over the ResNet-18 due to its enhanced architecture. Tab. [bb] shows adapter training results using the same dataset as the two pretrained models, revealing trend similar to Tab. Ta and demonstrating our method's comparable performance to other baselines, but with compatibility between the updated model and the previous does not require the new model's classifier, relying directly on the extracted embedding vectors. In Appendix B. further validate our method, we apply our approach different architectures used as pretrained models. Instead, in Appendix C we investigate update scenarios involving distribution or objective shifts using CLIP-like [62] models and self-supervised architectures such DINOv2 [63]
Unlike FastFill, approach
Results for compatibility on downstream tasks are reported in Tab. where adapters are trained representations from local datasets (CUB200 CIFAR100) different from the training dataset. Employing a transformation B, with >-Orthogonality regularization, our method enhances local task performance and model compatibility, outperforming the baselines. Results additional downstream datasets (Flower 102 [4]] and Places365) are reported in Appendix [ From Tab. Ta and Tab. Tb) we observe that strict orthogonal transformation, B_, does not result in performance improvements relative to the independently trained model Onew. Conversely, Bx, which provides more plasticity with respect B1, enables the new model to enhance performance in the downstream task. An ablation study on the hyperparameter À is presented in Appendix E and component-wist ablation of the loss terms in Eq. 3.5 is detailed in Appendix

---

4.5 Backfilling Results
In this section, we evaluate our novel backfill strategy discussed in Sec. 3.6 considering the experimental setting detailed in Tab. Ta and Tab. Tb Given that FCT lacks specific backfilling strategy, we employ a random ordering as in [22]. The results, depicted in Fig. Tab. 3 and Tab. 4
demonstrate that our backfilling strategy outperforms the other baselines by certain margin. Notably. Fig. $ illustrates that with less than 50% of the gallery backfilled, we achieve the same performance as the newly independently trained model. In Appendix G provide ablation study using
alternative distance metric to the Mean Squared Error employed in the main experiments.
5
Conclusion
Model compatibility is critical challenge in many large-scale retrieval systems and can hinder system updates when not achieved. this paper, we introduce mapping transformations that align independently learned representations under a unified space, also providing more feature clustering through supervised contrastive loss. We also propose relaxation of the orthogonality constraint to aid adaptation to downstream tasks without compromising the integrity of newly trained independent models. Additionally, we propose novel backfill ordering strategy that enables efficient partial backfilling of the gallery set, achieving the performance of a newly independently trained model with less than half of the gallery backfilled. Our approach demonstrates superior performance compared to previous methods, across the same and different data distributions To contextualize these results, the limitations of the approach are examined in detail Appendix Furthermore, to evaluate its practical utility, analyze its methodological complexity and broader applicability Appendix H
which the models are trained.
Acknowledgments
This paper was partially funded by the project "Collaborative Explainable neuro-symbolic AI for Decision Support Assistant", CAI4DSA, CUP B13C23005640006.
References
[1] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Proceedings of the IFEE conference computer vision and pattern recognition, pages 815-823, 2015.
[2] Weiyang Liu. Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolula, HI, USA, July 21-26, 2017, pages 6738-6746. IEEE Computer Society, 2017.
[3] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4690-4699, 2019.
[4] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Netvlad: Can architecture for weakly supervised place recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 5297-5307. 2016.
[5] Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 726-743. Springer, 2020.
[6] Stephen Hausler, Sourav Garg, Ming Xu, Michael Milford, and Tobias Fischer. Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14141-14152, 2021.
[7] Hyeonwoo Noh. Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proceedings of the IEEE international conference on computer vision, pages 3456 3465, 2017.
10

---

[8] Fuwen Tan, Jiangbo Yuan, and Vicente Ordonez. Instance-level image retrieval using reranking transform- ers. In proceedings the IEEE/CVF international conference on computer vision, pages 12105-12115, 2021.
[9] Bin Yan, Yi Jiang. Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 15325 15336, 2023.
[10] Colin Raffel. Building machine learning models like open source software. Commun. ACM, 66(2):38-40, jan 2023.
[11] Prateck Yadav, Colin Raffel, Mohammed Mugecth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey model moerging: Recycling and routing among specialized experts for collaborative learning. Trans. Mach. Learn. Rex., 2025.
[12] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[13] Niccolò Biondi, Federico Pernici, Simone Ricci, and Alberto Del Bimbo. Stationary representations: Optimally approximating compatibility and implications for improved model replacements. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[14] Jessica Maria Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting- Yao Hu, Chun-Liang Li, Oncel Tuzel, and Hadi Pouransari. MUSCLE: A model update strategy for compatible LLM evolution. In EMNLP (Findings), pages 7320-7332. Association for Computational Linguistics, 2024.
[15] Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards hack hackward-compatible representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2020.
[16] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Yoshua Bengio and Yann LeCun, editors, Feature Extraction:
Modern Questions and Challenges, pages 196 212. PMLR, 2015.
[17] Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang. Wei Xia, and Stefano Soatto. cessttvve-ongruente training: Towards regression frree model updates. In CVPR, pages 14299-14308. Computer Vision Foundation IEEE, 2021.
[18] Niccolo Biondi, Federico Pernici, Mattco Bruni, and Alberto Del Bimbo. Cores: Compatible represen- tations via stationarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-16, 2023.
[19] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopess, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning. pages
PMLR, 2022.
[20] Binjie Zhang, Yixiao Ge. Yantao Shen, Shupeng Su, Fanzi Wu, Chun Yuan, Xuyuan Xu, Yexin Wang, and Ying Shan. Towards universal cackward-compatible representation learning. In IJCAI, pages 1615-1621. ojcai.org, 2022.
[21] Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 9939-9948, October 2021.
[22] Florian Jaeckle, Fartash Faghri, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Fastfill: Efficient. compatible model update. In International Conference on Learning Representations, 2023.
[23] Yifei Zhou, Zilu Li, Abhinav Shrivastava. Hengshuung Zhao, Antonio Torralba, Taipeng Tian, and Ser-Nam Lim. Bt" 2: Backward-compatbbe training with basis transformation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 11229 11238, 2023.
[24] Simone Ricci, Niccolò Biondi, Federico Pernici, and Alberto Del Bimbo. Backward-compattin aligned representations via an orthogonal transformation layer. In ECCV Workshops (17), volume 15639 of Lecture Notes in Computer Science, pages 451-464. Springer, 2024.

---

[25] Vivek Ramanujan, Pavan Kumar Anasosalu Vasu, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. For- ward compatible training for large-scale embedding retrieval systems. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19386-19395, 2022.
[26] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Sociery, 2944)98331 2016.
[27] Minyoung Huh, Brian Cheung. Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In ICML. OpenReview.net, 2024.
[28] Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuelc Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36, 2024.
[29] Marco Fumero, Marco Pegoraro, Valentino Maiorca. Francesco Locatello, and Emanuele Rodolà. Latent functional maps: a spectral framework for representation alignment. In NeurIPS, 2024.
[30] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023.
[31] Valentino Maiorea, Luca Moschella, Marco Fumero, Francesco Locatello, and Emanucle Rodolà, Latent space translation via inverse relative projection. arXiv preprint arXiv:2406. 15057, 2024.
[32] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects, 2013.
[33] Guoliang Lin, Hanlu Chu, and Hanjiang Lai. Towards better plasticity-stability trade-off in incremental learning: A simple linear counector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 89 93-98,22 2022.
[34] Dongwan Kim and Bohyung Han. On the stability-plastinn dilemma of class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20196- 20204, 2023.
[35] Lirong Wu, Zicheng Liu, Jun Xia, Zelin Zang, Siyuan Li, and Stan li. Generalized clustering and multi-manifold learning with geometric structure preservation. In Proceedings of the JEEE/CVF winter conference applicutions of computer vision, pages 139-147, 2022.
[36] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018
[37] Binjic Zhang, Yixiao Ge, Yantao Shen, Yu Li, Chun Yuan, XUYUAN XU, Yexin Wang, and Ying Shan. Hot-refresh model upgrades with regression-free free compatible training in image retrieval. In International Conference on Learning Representations, 2021.
[38] Tan Pan, Furong Xu, Xudong Yang, Sifeng He, Chen Jiang, Qingpei Guo, Feng Qian, Xiaobo Zhang, Yuan Cheng, Lei Yang, et al. Boundary- aware backward-compatin representation via adversarial learning in image retrieval. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 15201-15210, 2023.
[39] Mateusz Budnik and Yannis Avrithis. Asymmetric metric learning for knowledge transfer. In CVPR, pages 2228-823.. Computer Vision Foundation/ JEEE, 2021.
[40] Niccolo Biondi, Federico Pernici, Matteo Bruni, Daniele Mugnai, and Alberto Del Bimbo. C12r: Compati- ble lifelong learning representations. ACM Transactions on Multinedia Computing, Communications and Applications, 18(2s):1 22, 2023.
[41] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory--fficient incremental learning through feature adaptation. In European Conference on Computer Vision, pages 199-715. Springer, 2020.
[42] Chien-Y Yi Wang, Ya-Liang Chang, Shang-T: Ta Vang, Dong Chen, and Shang-Hong Lai. Unified representation learning for cross model compatibility. In 31st British Machine Vision Conference 2020, BMVC 2020. BMVA Press, 2020.
[43] Shupeng Su, Binjie Zhang, Yixiao Ge, Xuyuan Xu, Yexin Wang. Chun Yuan, and Ying Shan. Privacy- preserving model upgrades with bidirectional compatible training in image retrieval. arXiv preprint arXiv:2204.13912 2022.
12

---

[44] Chang Wang and Sridhar Mahadevan. Manifold alignment using procrustes analysis. In Proceedings of the 25th international conference Machine learning, pages 1120-1127, 2008.
[45] Mario Esrcano-Casado and David Martunez-Rubio, Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pages 3794-3803 3803. PMLR. 2019.
[46] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkkskaa et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):35213322 2017.
[47] Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[48] Mehrtash Harandi and Basura Fernando. Generalized backpropagation, etude de cas: Orthogonality, arXiv preprint arXiv: 1,0597, 2016.
[49] Mete Ozay and Takayuki Okatani. Optimization on submanifolds of convolution kernels in cnns. arXiv preprint #xXiv:6600000000 2016.
[50] Lei Huang. Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[51] Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume 55. US Government printing office, 1968.
[52] Sagar Sharma, Simone Sharma, and Anidhya Athaiya. Activation functions in neural networks. Towards Data Sci, 6012)310-316, 2017.
[53] A Iliev, Nikolay Kyurkchiev, and Svetoslav Markov. On the approximation of the step function by some sigmoid functions. Mathematics and Computers in Simulation, 133:223-234, 2017.
[54] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36. 2024.
[55] Björ Barz and Joachim Denzler. Hierarchy-hased image embeddings for semantic image retrieval. In 2019 IEEE winter conference on applications of computer vision (WACV), pages 638-647. IEFF, 2019.
[56] Mikolaj Wieczorek, Barbara Rychalska, and Jacek Dabrowski. On the unreasonable effectiveness of centroids image retrieval, In Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8-12, 2021, Proceedings, Part IV 28, pages 212-223. Springer, 2021.
[S7] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
[58] A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Univ. Toronto, 2009.
[59] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 1011.
[60] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralha. Places: A 10 million image database for scene recognition. IEFF Transactions on Pattern Analysis and Machine Intelligence, 2017.
[61] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
13

---

[63] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research.
[64] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes, 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[65] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tal tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558 3568, 2021.
[66] Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D. Bagdanov. Cross the gap: Exposing the intra-modal misalignment in CLIP via modality inversion. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. Open- Review.net, 2025.
[67] Wenzhuo Liu, Fei Zhu, Longhui Wei, and Qi Tian. C-clip: Multimodal continual learning for vision- language model. In The Thirteenth International Conference Learning Representations, 2025.
[68] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws neural language models. arXiv preprint arXiv:2001.08300 2020.
[69] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 1021 2021.
[70] Gabriele Prato, Simon Guiroy, Ethan Caballero, Trina Rish, and Sarath Chandar. Scaling laws the out- of-distribution generalization of image classifiers. ICMI. 2021 Workshop on Uncertainty and Robustness in Deep Learning.. 2021.
[71] Ethan Cahallero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support our claims compatibility adaptation through experimental validations presented in Sec. and theoretically grounded explanation Sec. 3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the Appendix !
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our manuscript does not provide any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The details of our method are described in Sec. 3 while Sec. 4 provides the hyperparameter settings used to produce the results reported in all tables. Additional ablation studies on the hyperparameters are presented in Appendix I and Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec. provides the hyperparameter settings used to produce the results reported in all tables. Additional ablation studies on the hyperparameters are presented in Appendix E and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Unfortunately, due to limited GPU credits, we were unable to validate our experiments with different seed values. However, we used the same random initialization values (i.c., seed) for all methods, datasets, and network architectures in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: Our approach directly leverages pre-extracted features, requiring minimal GPU usage during training and thereby enabling reproducibility any contemporary GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Our work seems not to have societal impact to date. Anyway, we discussed the positive impact that may have real-workes retrieval systems in the introduction section
(Sec. D
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not to pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the code, data, and models that we used has been properly credited and their license and terms of use respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The present work does not released any new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were used only for the purpose of writing, editing, or formatting purposes and does not impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurij) httt::://eerrpp.ccc///ooeereeeer hct.///omfeeeennne/22////.. for what should or should not be described.
21

---

Table 5: Compatibility evaluation on Places365 under the Extending Classes setting. We use two independently trained ResNet-50 models: Vold trained on the first 205 classes, and &new trained on all classes of Places363.
Method
Query/Giallery
CMC-Topl
mAP
Guaa/ Poad
33.86
15.76
Ind. Train.
Oueu/ Oold
0.21
0.33
Osew/ Unew
37.37
19.11
F( Pould
0.21
0.33
PCT 25
F ( / F(reas)
36.43
19.02
neew
F (Qota)
37.04
18.99
F (oaa) / /eaa
0.21
0.33
FastFill
22
F Pata F F(
39.71
23.98
Guew F (Wols)
38.42
19.94
P (Pota) Dold
38.65
21.88
F (dola) F F(cva))
39.96
26.199
Ours
B (Pae (((oa))
38.50
21.77
B. (dwn)) Oold
35.47
17.98
B1 (Pres) ) B_ (Frew)
37.37
19.11
A Extending Classes Setting Places365
To validate our approach further, we evaluate it using model trained on dataset different from ImageNetIK. K. Specifically, we use a ResNet-50 pretrained on Places205 (from ViSSL) as the old model, and a ResNet-50 pretrained summarizes the performance of each method using the evaluation metrics defined in S.c The results demonstrate that the new model Onew is not inherently compatible with the old model, Pold- Moreover, the adaptation F ( provided by FCT underperforms when compared to the new model alone. contrast, methods that promote better clustering, such as FastFill and our proposed approach, achieve even higher performance than the standalone new model. This improvement arises from leveraging information from both the old and new models, effectively implementing form of knowledge distillation during the learning of the forward adapter. Unlike the baselines, method aligns all adapted representations within a unified representation space, thereby consistently maintaining compatibility with the old model.
Places365 (from CSAILVision) as the new model. Tab.
B Additional Architecture for Independently Pretrained Models Setting
We conduct additional experiments using DenseNet-121 the old model, duid and EfficientNet- B3 as the new model, Onew, both pretrained on ImageNet1K and obtained from the PyTorch Hub. The results of these experiments on the ImageNetI K dataset presented in Tab. 6 Our approach achieves the best performance across all metrics, outperforming the baselines in both cross-model and same-model retrieval scenarios.
C Additional Experiments with DINOv2 and CLIP as Independently Pretrained Models
To investigate update scenarios involving data distribution or objective shifts, we conduct additional experiments using ResNet-18 pretrained on ImageNet K the old model, and both a CLIP [62 pretrained on CC12M [65] dataset and a DINOv2 [33 models. To train both the forward and backward transformations, the ImageNet1K K dataset and the same byperparameters of Tab. [b are used. This setup represents a considerable shift in both data distribution and model objective relative to the new models. Notably, FastFill cannot be applied in this context, as both CLIP and DINOv2 lack classifiers. In Tab. 7a we report the results obtained using DINOv2 as the new, independently trained model. Our approach achieves better results than FCT, further validating its practical applicability to real-world problems.
aaallllatchh4__1111 )) as the new
22

---

Table 6: Compatibility results on ImageNetIK under the Independently Pretrained Models setting. We use two independently trained models: DenseNet- 121 as the old model, Oata, and an EfficientNet-B3 as the new model, &news both pretrained on ImageNet1 and obtained from the Py Torchh Hub.
Method
Query/Giallery
CMC-Topl
mAP
Guaa/ Grad
62.02
32.95
Ind. Train.
Oueu/ Oold
0.11
0.16
Osew/ Gnew
71.60
54.90
F( Ooudd
0.11
0.16
PCT 25
F (oa F(reas)
68.16
53.22
neew (Qota)
70.64
54.63
F (oaa) / /eaa
0.11
0.16
FastFill
22
F Pata / F(
67.76
57.22
Guew F (Vola)
69.47
57.43
P (Poaa) Oold
69.25
50.20
F (dola) F(cva))
69.29
57.36
Ours
B (Pae (((oa))
71.33
57.50
B. (dwn)) Oold
67.23
44.34
B1 (Pres) (Frew)
71.60
54.90
Table 7: Compatibility evaluation involving data distribution objective shifts: (a) ResNet-18 as old model and DINOv2 63 (vit_small Bacplllpochhl41111 dinou2) as the new models; (b) ResNet- 18 as old model and CLIP 662 pretrained on CC12M [65 the model. For each case, we report CMC-Topl and mAP metrics.
(b) CLIP 62). shift the objective function and the data distribution is present between the and new models.
(a) DINOv2 nk
A shift objective function is present hetween the old and new models.
Method
Query/Gallery
CMC-Top1
mAP
Method
Query/Gallery
CMC-Topl
mAP
Outa Doold
(oodd
$5.62
26.91
Cuts Pold Oren Orid
55.62
26.91
Ind. Train.
Oren Orid
0.04
0.17
Ind. Train.
0.04
0.17
Paeu thew
71.92
44.07
Puen ! Phew
44.29
16.15
F(dhaa) Ioaa
0.04
F(drau) fóca F F(crae)) Fleva))
0.04
0.17
FCT 25
F F(rasa) (Oas)FF(ooe (((aa))
59.33
37.53
FCT
42.58
16.93
Ouen F F(ora))
67.97
41.07
Orew F F(oaa)
42.96
16.88
F ((oaa)) /0aa
54.82
32.14
F F(dras) /daa
61.13
41.22
F F(dma) F(oua))
61.30
41.95
F (Chate) Fross))
$7.69
41.08
Ours
Ba (reww of ((laa))
68.74
43.78
Ours
B1 (Duns)
29.26
B1 (dren) doas
58.73
31.50
30.02
16.68
B (frew)) / B (frew)
71.92
44.07
B1 (r(rre)) ).D (Bren)
44.29
16.15
Instead, in Tab. 7b we report the results obtained using CLIP pretrained CC12M the new, independently trained model. this scenario, the pretrained CLIP model exhibits lower retrieval performance ImageNet1 compared ResNet-18. This is well-known limitation of multi- modal training, where intra-modal misalignment can negatively impact the quality of single-modality representations [66]. Specifically, CLIP models are optimized for modal retrieval rather than Lingle-modaiig retrieval tasks, in contrast to DINOv2 or ResNet-18, which are trained exclusively on single modality. This reduction in performance the new model relative the old one causes FCT to degrade the overall retrieval capacity of the system, failing to achieve compatibility, as attempts transform the higher-quality representations of the old model into the power-performing representations of the new model. In contrast, our method introduces an additional loss that encour- ages both intra-class clustering and inter-model alignment feature representations on the specific training dataset. As result, the forward transformation, due to its greater flexibility, improves the performance of the old model's representations. Even in this challenging scenario, our approach outperforms FCT. further validating the robustness of our method.
23

---

Table 8: Compatibility results on Places365 and Flowers102 for two models pretrained on Im- ageNetlK and udapted to downstream tasks: Pata. a ResNet-18 18, and Pnew ViT-L-16, using backward adapter BA with 12. The ZS column indicates the CMC-Topl performance increase on ImageNet K, with values in parentheses showing the increment compared to the newly independently trained model.
Method
Query/Gialery
Places365
Flowers102
CMC-Topl
zs
CMC-Topl
dooal (oldd
22.41
84.35
Ind. Train.
Oven/ Oold
0.20
1.20
Paeu/ Parw
35.15
99.39
F F(dma) /
0.20
1.20
FCT 25
F(oas) F(oas)
28. 17
61
Ouew/ F(
32.12
99.07
F /0os
0.20
1.20
FastFill
F(Qua) (F(aa)
26.38
53.78
Gren! F(lra))
33.04
11.12
F F(ova)) /ou
28.84
83,36
F(oas)) F F(chas)
29.80
89,9 99,90
Bx(Ores) )f ((baa)
33.27
99.41
By (( foaa
98.17
Bx(dwe 1/ BxlOren)
36.38 (+1.23)
+0.38
99.54 (+0.15)
+0.01
D Additional Datasets for Independently Pretrained Models adapted on Downstream Task setting
We further extend our analysis of the Independently Pretrained Models Adapted on Downstream Task setting by including These additions allow us to evaluate our method's 's effectiveness in more challenging scenarios. The results are reported in Tab. g In these experiments, the old model is ResNet-18 and the new model is ViT-L-16, both pretrained on ImageNet- We employ an affine adapter with À = 12. On both additional datasets, our approach consistently outperforms the baseline methods. The proposed A-Orthogonality regularization not only improves retrieval performance but also encourages the adapted new model representation, (x(Qnew), to remain consistent with its original form. As result, retrieval performance on ImageNetI K is preserved.
additional datasets: the larger Places365 and the fine-grained Flowers 102.
the downstream tasks
E
Ablation on the hyperparameter
In our experiments, we select to maximize adaptability to downstream tasks while preserv- ing the pre-trained model's performance on its original training dataset, ImageNetIK. To illus- trate the impact of our approach, Tab. D reports the CMC-Topl scores obtained by applying our proposed X-orthogonal regularizer the new pre-trained model. The results, also reported in
Ablation on
Pideal/Plheti /ildus
25
-18
Ter Shot Sole
CMC-Too1
-1.2
65
Fig. 5 indicate that increasing enhances the performance of the new model's representations on the downstream task.
However, this improvement comes at the ex- pense of reduced performance on the original dataset, as evidenced by a decrease in zero-shot (ZS) scores, particularly pronounced the ab- sence of regularization (A oc). Empirically, we find that setting
11
X-orthogonal regular- ization on CUB dataset. Displayed are the com- patibility metries CUB and the zero-shot (ZS) improvement on ImageNetiK at different value of X. Results correspond to those in Tab.
Figure 5: Ablation
yields the best trade-off across all metrics. (36) optimize a soft
orthogonality constraint, equal to case where
A 0. However, this formulation does not lead to performance improvements and is outperformed by

---

Table 9: Ablation over orthogonal regularization strength À on CUB dataset. Compatibility metrics on the target task and zero-shot (ZS) CMC-Topl gain on ImageNet Parentheses show the increment CMC-Topl over the independently trained new model on CUB dataset.
F(owa)/ (amus)FF(ouun) ((om) Bx(Oww) F(aau)
Bx(dmm)) 3f Bx(dmw)
zs
(strict orth.)
$7.52
66.89
71.78 (+0.000)
+0.000
57.49
66.79
71.54 (-0.241)
-0.001
57.52
66.72
72.00 (+0.224)
+0.008
58,09
68.77
73.07 (+1.294)
+0.028
12
59.92
70.72
75.44 (+3.659)
+0.028
16
59.68
70.21
76.40 (+4.625)
+0.062
22
60.20
69.50
77.89 (+6.109)
-0.318
36
59.32
63.34
78.77 (+6.990)
-3.008
(no reg-)
59.26
62.84
78.89 (47.110)
-3.526
Table 10: Comparison of orthogonal regularization methods with different weight scales w. Compati- bility metrics on the downstream task CUB200 and zero-shot (ZS) CMC-Topl gain ImageNetI Parentheses show the increment in CMC-Topl over the independently trained new model. The last column reports the final value of [W W -.
8 Method
F(doa)/ F(oas)
BA (x(en)))) F/((aa)))
Bx(dheu) (B
| W /||r
SO
57.48
66.79
71.54 (-0.241)
-0.001
0.09
SRIP
57.38
66.57
71.66 (-0.120)
-0.001
0.08
Ours (A 12)
59.92
70.72
75.44 (+3.659)
+0.028
12.05
10
SO
59.11
69.56
74.88 (+3.106)
+0.022
9.50
10
SRIP
58.88
63.58
78.77 (+6.990)
-1.467
29.55
10
Ours (A 12)
59.93
70.70
75.20 (+3.419)
+0.076
12.12
10-2
59.06
63.54
79.06 (+7.283)
-1.444
29.27
10-2
SRIP
59.23
63.42
78.73 (+6.955)
-3.077
35.42
1
Ours (A 12)
59.06
63.54
79.06 (+7.283)
-1.44
29.27
10-3
SO
58.71
62.91
78.78 (+7.007)
-3.162
35.54
10
SRIP
58.83
63.18
78.92 (+7.145)
-3.457
38.63
10
Ours (A 12)
58.71
62.91
78.78 (+7.007)
-3.162
35.54
the use a strictly orthogonal transformation. As discussed in Sec. (3.3] imposing strict orthogonality may hinder the model's ability to incorporate task-specift information. In contrast, our approach relaxes this constraint by introducing a tunable hyperparameter that controls the deviation of the Gram matrix from the identity, allowing greater flexibility while preserving representational consistency.
To further validate our aproach we also study the effect of scalar weight w to the loss contributions of our A-orthogonal regularization compared with two different orthogonal regularizations: Soft Orthogonality (SO) [66 -witch correspond to the spacial case of =0 in our aproach- and Spectral Restricted Isometry Property (SRIP) We test the regularizers across different values of sealar weight: w 1, w 10 w 10 and = 10-3 Additionally, we include column reporting the exact value of ||W W Il| reached by the backward transformation B, the end of training, to indicate the deviation from strict orthogonality
As shown in the Tab. 10 for both SRIP and SO, the final value of |[TTW- T|| is governed by the optimization process and the chosen scalar weight w. Unlike our X-orthogonal regularization, these approaches do not provide direct control over ||W TWT =]]];; smaller contribution of the regularizer to the total loss results in a diminished regularization effect on the backward transformation Bx. When the scalar weight of the regularizer is reduced, the optimization process is unable to fully minimize the regularization term, particularly because competing loss components (such as MSE and the contrastive loss Lş) may favor a non-ortio non-orthogonal transformation. For instance, when = 10 and w = 10 2 the results obtained with SO. SRIP, and our X-orthogonal regularization are comparable to those observed in the case of constraint is entirely ignored. This occurs because, at such a small value of the contribution of the regularizer becomes negligible during optimization. To avoid this issue, in our method we set 1 for the X-orthogonal regularization, thereby ensuring that the regularization term is effectively
oo (see Tab. 9. where the orthogonality
25

---

incorporated into the optimization process during the training of the backward transformation. This ensures that the regularization term achieves the target threshold A, enabling precise control over the
trade-off in the backward transformation and leads to higher representation compatibility on the downstream task, As highlighted by the bold entries in the Tab. [T, our method produces stable results (minor fluctuations are attributable to stochastic optimization) for w 1 and w = 10-1 contrast to SO and SRIP. Conversely, when w is very low (10-² or 10-3), the regularizer cannot be fully optimized, and our method behaves similarly to SO regularization, as introduced constrains (I]W
- F 2 X) influences the minimum of the objective, which is never reached in practice. In contrast, due to its approximate formulation and greater complexity relative to SO, SRIP exhibits an even weaker regularization effect when w is low.
F Detailed Analysis of Loss Term Contributions
In this section, we analyze the contribution of each term to the final loss (Eq. 3.5
optimized during
training. Tab. IT presents the results obtained when the adaptation dataset matches the dataset used to train the models from which the features were extracted, namely ImageNet1K K. In this scenario, We observe that
a strict orthogonal transformation B1 is employed for wiikkwawd-omppatbbiii when used independently. LF ensures compatibility with the representations of the new model but significantly fails to achieve backward compatibility. This behavior highlights a pronounced forward bias inherent to Lr. The backward alignment loss LB alone promotes backward compatibility but degrades forward-adapted representation performance. The contrastive loss Le alone significantly improves inter-model aligmment and intra-class clustering, supporting both backward and forward compatibility. The combination LF Ln + Le achieves the highest overall performance across compatibility scenarios, underscoring the importance of each loss component in maintaining balance between forward and backward trasformation learning.
Tab. 12 shows the impact of these loss terms in a downstream task setting (CUB dataset), where oold is ResNet-18 and &new is ViT-L-16, 16, using A-Orthogonality with À 12. Similar to Tab, II excluding the backward loss LB still yields good forward compatibility but significantly reduces backward compatibility performance. Excluding the contrastive loss Le substantially decreases the adaptation to the downstream task leading to lower (xnwww)) )/ /BBx (énew) values. Using all loss terms LP + LB + Le consistently achieves the best or near-best results in forward and backward compatibility, demonstrating the complementary nature of these terms.
These analyses underline that each loss term contributes uniquely and significantly to achieving comprehensive and model compatibility across various tasks.
Table 11: CMC-Topl (%) on ImageNetI for different loss combinations (V included, x excluded). The setting is the same of Tab. lb where the first model, dola, is ResNet-18, whereas the second, Pnow. is a ViT-L-16.
Losses
Query/Gallery (CMC-Topl %)
LF
Cy
Lc
F(bca)/bau
F(daa)/ F ((das)
B1 (drew) /oaa
B (d )/ F(doa))
B_ B_(dnee)) )/ (shne)
X
59.09
0.04
72.27
76.63
X
0.04
49.34
62.75
0.04
76.63
61.24
58.63
64.97
60.83
76.63
54.18
59.29
62.77
72.46
76.63
V
V
61.25
60.43
65.13
73.44
76.63
8
V
60.85
59.09
65.42
57.90
76.63
V
V
V
60.83
61.10
65.54
73.53
76.63
G Distance metric for Partial Backfilling Ordering
Our proposed partial backfilling strategy is guided by a distance metric d, which measures the dissimilarity between each embedding vector F(h²) and its corresponding class mean μ.. This section investigates the impact of different distance metrics on determining an effective ordering for backfilling images in the gallery set. We compare two distance metrics -Marics-Meen Squared Error (MSE) and Cosine Distance- for ranking images during partial backfilling. The performance of each metric is evaluated under two distinct experimental conditions: the Extending Classes setting (Tab. [3) and
26

---

Table 12: CMC-Topl (%) on CUB for different loss combinations included, excluded). The setting is the same of Tab. 2 where the first model, Pota. is a ResNet- 18, whereas the second, Onew, is ViT-I--16. backward adapter Bx with X 12 is used to adapt the improved model on the downstream task.
Losses
Query/Gallery (CMC-Topl
LB
Le
Fldna)/oau
F(dwa)/ ((
(x(www) //r
H Bx(dne)) (oonn))((ooa (dras)
Bx(Ouew BA (B(rww))
/
0.0
51.72
0.0
63,82
72.14
X
X
0.0
35.27
45.80
0.0
71.91
V
V
X
37.15
47.56
46.56
60.70
69.76
X
X
V
52.79
59.14
58.38
66.46
73.36
X
V
50.43
59.88
58.57
70.13
74.86
X
V
V
53.27
58.66
60.45
59.44
73.12
51.12
59.92
60.64
70.72
75.44
100
Table 13: Extended Classes set-
0.33
ting
0.13
9
Mf
3.45
Method
CMC- Topl
mAP
330
(981
3.10
0.07
Table 14: Independently Pre- trained Models setting
0.33
0.50
Method
CMC-Topl
MAP
055
MSB
76,59
57.72
Dan's Deice
Cusine Distance
76.49
58.18
Figure 6: Different distance metric ablation for our partial backfilling strategy. Results for the Extending Classes setting (top Figures) of Tab. [a] and Independently Pretrained Models setting (bottom Figures) of Tab. lb We use features from the new backward aadadddddaptedd model B (Φnew) for the query set. For the gallery set, we begin with forward-adapted old features F(Qaa)) and incrementally replace them with new features.
the Independently Pretrained Models setting (Tab. 14). MSE computes the Fuclidean distance between feature vectors, capturing both angular and magnitude discrepancies. As shown in Tab. 13 Tab. [4] MSE generally yields robust performance, particularly terms of CMC-Topl. In contrast, Cosine Distance measures the angular distance between normalized feature vectors, emphasizing directional similarity while ignoring magnitude. The results indicate that Cosine Distance achieves slightly better performance in terms of mAP and provides comparable CMC-Topl scores relative to MSE.
H Method Complexity and Broader Applicability
Method Complexity. Our approach requires training only two matrices, resulting in a small number of parameters to optimize. Because method operates solely the extracted embeddings. it does not require any knowledge the underlying models and therefore applicable across different objectives (see Appendix C architectures, and types of learned representations.
In contrast previous methods, which either focus solely on alignment loss without any representation clustering loss (e.g., FCT (25)), require specific architectural components of the pretrained models (e.g., FastFill [22]. which requires access to the classifier of the new model), our approach addresses these limitations. Additionally, while existing baselines provide only forward adaptation, our method designed to achieve both forward and backward compatibility, thereby addressing practical needs that prior works do not meet. For instance:
27

---

B1 (((eww) F((ow) yields higher retrieval values compared to the baselines.
Bi (@new)/ Cold can be achieved exclusively by our method. From a practical standpoint, this allows compatibility be established even before all gallery items are forward-adapted using F.
Since our approach provides a unified representation space, even when the gallery is in a hybrid form (i.e., with some elements already adapted by F and others not), using B1 (1 (Drew) still ensures compatibility. This can not be achieved neither by FCT [25) nor FastFill [22].
The contrastive loss defined in Eq. B relies on the availability of class labels encourage embeddings from the same class to cluster together while pushing apart embeddings from different classes. In scenarios where class labels are not available, Eq. naturally reduces to unsupervised contrastive loss, similar to the objective used for training CLIP models [62]. In this unsupervised setting, we contrast pairs of representations originating from different models, and clustering-singg it cannot byproduct resulting from embedding similarity. Consequently,
be enforced directly-becomeg our approach is flexible and can be applied in both supervised and unsupervised training scenarios, depending on the availability of labels for the downstream task.
Broader Applicability. As is demonstrated in [36). soft orthogonalization has been applied to regularize all the weights of a CNN during training, and could benefit from the increased plasticity offered by our proposed X-orthogonal regularization. While retrieval is the standard scenario for evaluating compatibility [13], our approach is broadly applicable to any task that requires represen- tation adaptation, as it focuses on model alignment and clustering of learned representations. As demonstrated in our downstream task adaptation experiments (see Sec. 4.4), our regularization ap- proach yields improved performance compared to a strict orthogonal constraint, making it a valuable approach in domain adaptation scenarios as well. Furthermore, enforcing geometrical consistency while allowing adaptability has recently been investigated in the context of continual learning for multimodal training [67]. However, the authors of [67 promote this property indirectly through knowledge consolidation loss, rather than by directly applying a regularization constraint. This high- lights both possible future research and the potential applicability of our X-orthogonal regularization across various areas of representation learning.
I Limitations
Our approach relies on the assumption that the new model's embedding space is more expressive (e.g.. higher retrieval accuracy, stronger clustering) than that of the old model. If the updated model not comparable or lower quality, due, for instance, domain mismatch, insufficient training data, or architectural regressions, then both the forward and backward adapters may fail to improve performance or could even degrade compatibility. In many practical systems, this assumption is justified by scaling laws [68
70 7 (i.e., larger models and more data generally yield better feature representations). For downstream tasks adaptation, while our X-orthogonal regularized adapter shows strong performance and compatibility across various retrieval tasks, manual tuning of the orthogonality threshold (A) is needed. The trade-off between preserving the original model's geometry and allowing sufficient plasticity to adapt to new data hinges critically on the choice of X. practice, this hyperparameter could be selected via cross-validation or a small hyperparameter search on held-out portion of the downstream dataset. Although we found that X = 12 provides good balance our experiments (Appendix) [B) different downstream domains (e.g., fine-grained vs. coarse categories) and adapted representations may require different tuning of À to achieve optimal performance, Automating or self-tuning this parameter remains an open challenge.
69
28


Reviews:
[{"review_id": "5FoQhl8drR", "reviewer": "Reviewer_FJf7", "summary": "The paper focuses on the challenge of reconciling representations learned by different neural networks in retrieval systems, particularly given the high training costs and potential inconsistencies. It highlights two common approaches for adapting learned representations: affine transformations, which are adaptable but can distort the original representation, and orthogonal transformations, which preserve structure but are less adaptable. The core problem is how to align the latent spaces of updated models with previous ones on downstream distributions, all while preserving the newly learned representations. To address this, the authors propose a λ-orthogonality regularization applied during the learning of an affine transformation. This technique aims to achieve distribution-specific adaptation while simultaneously retaining the original learned representations. The paper validates this approach through extensive experiments across various architectures and datasets, demonstrating its ability to preserve zero-shot performance and ensure compatibility across model updates.", "questions": "Refer to the weakness.", "limitations": "I am not very familiar with this area, so unfortunately cannot point to any deeper limitations.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nI think the idea of adapting representations is quite interesting as well as the problem is well motivated. Paper is more or less written in a easy fashion, although I've some questions which I point out to in the next section. \n\n## Weakness\n- Can authors add a quick proof of B being orthogonal if it is represented as a matrix exponential of a skew symmetric matrix?\n- For the $\\lambda$ orthogonality constraint, I am not sure on what parameters is it applied. While the equation refers to W, I am not sure where is this W coming from. From what I've understood, it could be the learnable parameters in B, but it is not clear. \n- I am not following the contribution of individual loss terms. Given the best results mentioned in the experiments are when we just apply backward transformation on the new ones, for both query and gallery, can one not get away with just having backward compatibility loss?\n- Table~10 last column seems off, as why should the performance remain same as 76.63 despite heavily changing the  objective functions.\n- For Table~2 why should one see improvement over the independently trained new model when applied the backward compatibility mapping?", "was_revised": true, "final_justification": "I appreciate the rebuttal, and I think the updated manuscript should contain the revised notations that are provided here. That being said, given that I am not an expert in this area, I'd keep my positive rating.", "rebuttal": "We sincerely appreciate the Reviewers’ thoughtful feedback and their recognition of our contributions to representation compatibility through our adaptation approach and its underlying motivation. Below, we address each of the points raised in the review in detail.\n    \n**[Proof of Orthogonality for Exponentials of Skew-Symmetric Matrices]**\n\nWe thank the Reviewer for this suggestion. To clarify why $B$ is orthogonal when reparametrized as the exponential of a skew-symmetric matrix, we provide the following proof.\n\n> **Proof.** Let $P$ be any real skew-symmetric matrix ($P^T = -P$) and define $B$ as its exponential:\n>\n> $$\n> B = e^P = \\sum_{k=0}^\\infty \\frac{P^k}{k!}\\,.\n> $$\n>\n> Since transposition and the matrix exponential commute, it follows that\n>\n> $$\n> B^T = \\bigl(e^P\\bigr)^T = e^{P^T} = e^{-P}\\,.\n> $$\n>\n> Therefore\n>\n> $$\n> B^T B \\;=\\; e^{-P}\\,e^P \\;=\\; e^{-P + P} \\;=\\; e^0 = I\\,,\n> $$\n>\n> showing that $B^T B = I$ and hence $B$ is orthogonal.\n> $\\Box$\n\nThis demonstrates that any matrix of the form $e^{P}$ is orthogonal.\n\n    \n**[Clarification of $W$ and the Learnable Parameters of Transformation $B$]**\n\nThe Reviewer is correct. $W$ denotes the weight matrix of the transformation $B_\\lambda$ (see Lines 148–149): $B_\\lambda: \\mathbb{R}^n \\to \\mathbb{R}^n,\\quad B_\\lambda(x) = W x + b$, where $W \\in \\mathbb{R}^{n \\times n}$ and $b \\in \\mathbb{R}^n$.\n    \nFor clarity regarding the learnable parameters of the backward transformation, we provide additional details for Sections 3.2 and 3.3 below.\n\nIn Section 3.2, we enforce strict orthogonality on the transformation $B_\\perp$ by reparameterizing $W$ as the matrix exponential of a skew-symmetric matrix $P$; thus, the learnable parameters are the upper triangular elements of $P$.\n    \nIn contrast, in Section 3.3, the transformation $B_\\lambda$ uses a general weight matrix $W \\in \\mathbb{R}^{n \\times n}$, and we encourage orthogonality via a a $\\lambda$-orthogonality regularization:\n\n$$\n\\min_{W} \\|W^T W - I\\|_F \\quad \\text{s.t.} \\quad \\|W^T W - I\\|_F \\geq \\lambda\n$$\n\nIf the Reviewer considers it necessary, we are willing to revise the manuscript to avoid any potential confusion by changing the symbol $W$ and referring directly to $W_B$, as it is the transformation subject to the $\\lambda$-orthogonality regularization. Below, we provide an example reflecting this change:\n    \n$$\n\\min_{W_B} \\|W_B^T W_B - I\\|_F \\quad \\text{s.t.} \\quad \\|W_B^T W_B - I\\|_F \\geq \\lambda\n$$\n    \n**[Contribution of Individual Loss Terms and Practical Implications of Learned Transformations]**\n\nWe thank the Reviewer for this insightful question. While it may initially appear that training only the backward adaptation loss (Eq. 2) is sufficient—since it directly encourages alignment of new queries with the old gallery—in practice, this approach is limited. The contribution of each loss term is analyzed in detail in Appendix E of the manuscript. As shown in Tables 10 and 11, optimizing only the backward loss $L_B$ results in a significant drop in performance for the $B(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$ scenario. This degradation arises because the forward transformation $F$ remains untrained when only $\\mathcal{L}_B$ (see Eq. 2) is optimized, which in turn prevents effective retrieval in settings where the transformation $F$ is involved.    \n\nAs noted by the Reviewer, the best results are always obtained when both the gallery and query sets are encoded with the newly trained independent model ($\\phi_{\\text{new}}/\\phi_{\\text{new}}$ or $B_{\\perp}(\\phi_{\\text{new}})/B_{\\perp}(\\phi_{\\text{new}})$). However, this scenario is rarely practical, as extracting embedding vectors for the gallery set is the most computationally expensive operation. This process corresponds to the re-indexing discussed in previous works, which is typically avoided by leveraging representation compatibility. Thus, the case where both gallery and query are re-indexed with the new model serves as an upper bound for achievable system performance following complete re-indexing.\n\nIn practical systems, the query set is usually much smaller than the gallery. Applying the backward transformation $B$ only to the query set enables compatibility with the existing gallery and improves overall system performance, while avoiding the costly re-indexing of the gallery ($B_{\\perp}(\\phi_{\\text{new}})/\\phi_{\\text{old}}$ and $B_{\\perp}(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$). Furthermore, applying the forward transformation $F$ to the already extracted old gallery embeddings introduces minimal computational overhead [25], making it feasible for real-world deployments. This approach also yields better retrieval performance compared to using the untransformed old gallery embeddings directly. Given these considerations, the most practical scenario is $B(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$, which should be directly compared with the $\\phi_{\\text{new}}/F(\\phi_{\\text{old}})$ results reported by the other baselines.\n\nTo better illustrate the relevance of each transformation, we highlight the most informative cases:\n\n- $F(\\phi_{\\text{old}})/\\phi_{\\text{old}}$: Represents the retrieval performance when only the query set is forward-transformed and compared against the old gallery.\n\n- $F(\\phi_{\\text{old}})/F(\\phi_{\\text{old}})$: Indicates performance when both the gallery and query are forward-transformed using $F$.\n\n- $B(\\phi_{\\text{new}})/\\phi_{\\text{old}}$: Shows performance when only the query set is backward-transformed. (It is worth noting that only our approach allows for this scenario.)\n\n- $B(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$: Reflects the performance when the query set is backward-transformed and the old gallery is forward-transformed.\n\nFinally, in the context of downstream task adaptation, even $B(\\phi_{\\text{new}})/B(\\phi_{\\text{new}})$ provides valuable insight. Specifically, it demonstrates an improvement over $\\phi_{\\text{new}}/\\phi_{\\text{new}}$, indicating that relaxing the strict orthogonality constraint enables slight adaptability to the downstream task of the pretrained representation of the new independently trained model while preserving its zero-shot performance on the ImageNet1K dataset.\n    \n**[Unchanged Performance in Table 10 Despite Objective Changes]**\n    \nWe thank the Reviewer for raising this question. As the Reviewer correctly observed, the retrieval performance in Table 10 for the $B_{\\perp}(\\phi_{\\text{new}})/B_{\\perp}(\\phi_{\\text{new}})$ case remains unchanged (76.63), despite significant changes in the objective functions.\n\nThis result is expected and consistent with the backward transformation proposed in our approach. Since $B_{\\perp}(\\cdot)$ is a strict orthogonal transformation—enabled by the exponential reparameterization, as detailed in Section 3.2—this transformation is an isometry, meaning it preserves the geometric structure (angles and magnitudes) of the new model representation. Consequently, the performance of the new independent model is maintained under this transformation regardless of the optimized objective function. This behaviour is also present in Tables 1, 6, and 7, where $B_{\\perp}(\\phi_{\\text{new}})/B_{\\perp}(\\phi_{\\text{new}})$ is comparable to $\\phi_{\\text{new}}/\\phi_{\\text{new}}$. The strict orthogonality is indeed a crucial property of the backward transformation $B$ when the geometric structure of a representation needs to be maintained.\n\nOnly in the setting of downstream tasks, we relax this strict orthogonality with our $\\lambda$-orthogonal regularization, allowing for slight adaptation of the new model representation to the specific task without compromising the original learned representation distribution, as explained in Section 3.3.\n    \n**[Explanation for Improved Performance With Backward Compatibility Mapping in Table 2]**\n    \nThank you for your question. The improvement observed in Table 2 is attributable to the $\\lambda$-orthogonal regularization applied to the backward compatibility transformation $B_\\lambda$. This regularization enables the representation extracted by the new model to effectively adapt to new data, thereby improving its downstream performance.\n\nThis can be attributed to the following reasons:\n- First, our $\\lambda$-orthogonal regularization provides a granular control on the degree of alignment between the new and old model representations. It strikes a balance between strict orthogonality, which may overly constrain the new model representation, and no regularization, which may permit harmful drift in the representation. This controlled flexibility enables the backward transformation to preserve the original learned representation while also facilitating effective adaptation to new data.\n- Second, the backward compatibility mapping, which is also jointly optimized with the contrastive loss described in Eq. 8, encourages both intra-class clustering and inter-model alignment on the downstream task. This leads to better retrieval performance. As shown in the ablation study in Table 11, when the contrastive loss is not included in the optimization, the performance of $B_{\\lambda}(\\phi_{\\text{new}})/B_{\\lambda}(\\phi_{\\text{new}})$ is not always higher than that of $\\phi_{\\text{new}}/\\phi_{\\text{new}}$ (which is 71.78).\n    \nEmpirically, as shown in Table 2 and discussed in Appendix D, applying our $\\lambda$-orthogonal regularization leads to superior adaptation to downstream tasks, compared to both strict orthogonality and the absence of regularization. Table 9 further illustrates the different behaviors of the backward transformation under different values of $\\lambda$. Figure 5 demonstrates that allowing excessive deviation from orthogonality can degrade the learned distribution of the newly pretrained model (e.g., the increment of zero-shot performance on ImageNet drops below zero), while moderate regularization improves downstream retrieval performance."}, {"review_id": "fkJwp1ZkZ4", "reviewer": "Reviewer_SkaR", "summary": "The authors study the problem of embedding model compatibility in vector database search. If a vector database is generated with a model $\\phi_{old}$ but later a model $\\phi_{new}$ is trained, can we still make use of the existing embeddings without backfilling? The authors propose numerous strategies to ensure model compatibility. Their primary contribution is $\\lambda$-orthogonality: during training of the new model, introduce a regularization to ensure that new representations are approximately an orthogonal transformation of the old model features. They also introduce strategies to ensure this does not hinder new model training. They show improvements in the forward and backward compatibility settings.", "questions": "Generally addressing weaknesses would improve my score. Here's an option:\n\nTry applying these approaches to a CLIP model, do compatibility between old ImageNet1k and new CLIP model trained on a small dataset (CC12M). Even if you can't directly do this in the full backward compatibility scenario, I'm interested in how/if the modeling losses restrict new model performance under distribution/objective shift.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n+ Interesting technical contributions:\n\n1. $\\lambda$-orthogonality loss (surprising that this does not damage new model loss too much)\n2. Cluster based approach to enhance alignment as well (similar to BCT)\n\n+ Training forward and backward compatibility at the same time yields a pretty flexible set of options for compatible vector search\n+ Interesting study of partial backfilling along with forward transformations, which adds a good picture of efficient model updates.\n\n\n### Weaknesses\n\n- In comparison to other forward compatibility approaches, which are new model training agnostic, this adds substantially more losses to new model training, potentially limiting utility in practical training pipelines.\n- Not clear how the intraclass clustering approach can be adapted to CLIP-like and other self-supervised learning approaches (e.g. Dino) models, which are very common in image vector databases these days. \n- Not clear how the orthogonal projection would behave in more significant model update scenarios (e.g. a change in distribution/objective, with phi_old = Classifier on ImageNet1k and phi_new = CLIP on CC12M).", "was_revised": true, "final_justification": "Authors addressed my weaknesses. In particular, the CC12M experiment showed improvement in larger domain updates.", "rebuttal": "We thank the Reviewers for their careful evaluation and for highlighting several key strengths of our work. In particular, we are glad that the orthogonality loss and the cluster-based alignment approach—both of which underpin our adaptation method—were found to be interesting. We also appreciate the acknowledgment of our efforts to enable both forward and backward compatibility during training, which yields a flexible set of options for compatible vector search. Additionally, we are pleased that the study of partial backfilling alongside the transformations was noted as providing a comprehensive perspective on efficient model updates. Below, we address each of the weaknesses raised in the review in detail.\n\n\n**[Practical Limitations Due to Additional Losses in New Model Training]**\n\nThank you for raising this significant point. To clarify, we do not train the models themselves, but rather adapters; all backbone models remain frozen (see L103-104), consistent with both FCT and FastFill approaches (we utilize pretrained model weights, downloading them as independently trained models when available—see L295-296 and L884-887). These adapters are implemented as a single matrix, enabling fast training and ensuring practical applicability in real-world scenarios.\n    \nWe will add specific symbols to Figure 1 to distinguish the trainable adapters from the frozen model backbone, thereby avoiding any possible misunderstanding.\n    \nMoreover, while FCT is the first work in this area, as discussed in our paper, it has certain limitations—specifically, its objective loss is solely the MSE between old and new representations. This enforces alignment of the old representation to the new one but does not effectively capture additional information from the adaptation training data. FastFill attempts to address this limitation by introducing an extra loss for improved forward transformation learning; however, this additional loss requires direct access to the new model, particularly its classifier, which is not always possible.\n\nIn contrast, our method operates directly on the embeddings extracted from the networks, making our approach model-agnostic like FCT, but with the added advantage of being able to incorporate extra information from the adaptation data via a contrastive loss that enforces both intra-class clustering and inter-model alignment. Compared to FastFill, our method introduces only a single, simple loss (MSE) for training the backward transformation.\n\n\n**[Applicability of Intr-aclass Clustering and Orthogonal Projection in Modern Self-Supervised (DINOv2) and CLIP-like Models]**\n\nWe thank the Reviewer for encouraging us to investigate update scenarios involving distribution or objective shifts, which will further validate our approach. Our method operates directly on embeddings extracted from any pretrained model; all backbone models remain frozen and are used solely to obtain embedding vectors. As a result, our approach is agnostic to the specific pretraining strategy (i.e., objective function) or dataset (i.e., data distribution).\n\nThe orthogonality constraint imposed on the backward transformation ensures that the representations learned by the pretrained model are preserved during adaptation, thereby avoiding distribution shift, as this transformation is an isometry.\nIn our experiments, we already consider the case where adapters are trained on embeddings generated from a downstream task dataset, whose data distribution typically differs from that of the dataset used for pretraining. Thus, our method can be readily applied to CLIP-like models and other self-supervised architectures such as DINOv2.\n\nHowever, challenges may arise if the new model’s representation quality is insufficient for a specific dataset. As discussed in our Limitation section (Appendix G), our method assumes that the new model’s embedding space is more expressive (e.g., exhibiting higher retrieval accuracy or stronger clustering) than that of the old model. If the updated model is not comparable or has lower representation quality—due, for example, to domain mismatch, limited training data, or architectural regressions—both the forward and backward adapters may fail to improve retrieval performance.\n\nTo empirically investigate this scenario, as requested by the Reviewer, we conducted experiments using a ResNet-18 pretrained on ImageNet1K as the old model, and both a CLIP model pretrained on CC12M and a DINOv2 model (“vit_small_patch14_dinov2”) as the new models. The dataset used to train both the forward and backward transformations was ImageNet1K. This setup represents a considerable shift in both data distribution and model objective relative to the new models. Notably, FastFill cannot be applied in this context, as both CLIP and DINOv2 lack classifiers. For fairness, we used the same hyperparameters across all methods as in the main manuscript.\n\nIn the following Table, we report the results obtained using DINOv2 as the new, independently trained model.\n    \n| Method        | Query/Gallery                              | CMC-Top1 | mAP   |\n|---------------|--------------------------------------------|----------|-------|\n| **Ind. Tr.**  | $\\phi_{\\text{old}}/\\phi_{\\text{old}}$      | 55.62 | 26.91 |\n|               | $\\phi_{\\text{new}}/\\phi_{\\text{old}}$      | 0.04 | 0.17 |\n|               | $\\phi_{\\text{new}}/\\phi_{\\text{new}}$      | 71.92 | 44.07  |\n| **FTC**       | $F(\\phi_{\\text{old}})/\\phi_{\\text{old}}$   | 0.04 | 0.17 |\n|               | $F(\\phi_{\\text{old}})/F(\\phi_{\\text{old}})$| 59.33    | 37.53 |\n|               | $\\phi_{\\text{new}}/F(\\phi_{\\text{old}})$   | 67.97    | 41.07 |\n| **Ours**      | $F(\\phi_{\\text{old}})/\\phi_{\\text{old}}$   | 54.82    | 32.14 |\n|               | $F(\\phi_{\\text{old}})/F(\\phi_{\\text{old}})$| 61.30    | 41.95 |\n|               | $B_{\\perp}(\\phi_{\\text{new}})/\\phi_{\\text{old}}$ | 58.73 | 31.50 |\n|               | $B_{\\perp}(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$ | 68.74 | 43.78 |\n|               | $B_{\\perp}(\\phi_{\\text{new}})/B_{\\perp}(\\phi_{\\text{new}})$ | 71.92 | 44.07 |\n    \n\nAs shown in the Table, our method achieves compatibility even when DINOv2 is used as the new model. This demonstrates the applicability of our approach to models trained with different objectives and datasets.\n\nMoreover, as requested by the Reviewer, we present in the following Table the results obtained with FCT and our approach in the scenario where the old model is a ResNet-18 and the new model is a CLIP model pretrained on CC12M.\n    \n| Method        | Query/Gallery                              | CMC-Top1 | mAP   |\n|---------------|--------------------------------------------|----------|-------|\n| **Ind. Tr.**  | $\\phi_{\\text{old}}/\\phi_{\\text{old}}$      | 55.62 | 26.91 |\n|               | $\\phi_{\\text{new}}/\\phi_{\\text{old}}$      | 0.04 | 0.17 |\n|               | $\\phi_{\\text{new}}/\\phi_{\\text{new}}$      | 44.29     | 16.15  |\n| **FTC**       | $F(\\phi_{\\text{old}})/\\phi_{\\text{old}}$   | 0.04 | 0.17 |\n|               | $F(\\phi_{\\text{old}})/F(\\phi_{\\text{old}})$| 42.58    | 16.93 |\n|               | $\\phi_{\\text{new}}/F(\\phi_{\\text{old}})$   | 42.96    | 16.88 |\n| **Ours**      | $F(\\phi_{\\text{old}})/\\phi_{\\text{old}}$   | 61.13    | 41.22 |\n|               | $F(\\phi_{\\text{old}})/F(\\phi_{\\text{old}})$| 57.69    | 41.08 |\n|               | $B_{\\perp}(\\phi_{\\text{new}})/\\phi_{\\text{old}}$ | 30.02 | 16.68 |\n|               | $B_{\\perp}(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$ | 44.93 | 29.26 |\n|               | $B_{\\perp}(\\phi_{\\text{new}})/B_{\\perp}(\\phi_{\\text{new}})$ | 44.29     | 16.15 |\n    \nIn this scenario, the pretrained CLIP model exhibits lower retrieval performance on ImageNet1K compared to ResNet-18. This is a well-known limitation of multi-modal training, where intra-modal misalignment can negatively impact the quality of single-modality representations [1]. Specifically, CLIP models are optimized for cross-modal retrieval rather than single-modality retrieval tasks, in contrast to DINOv2 or ResNet-18, which are trained exclusively on a single modality.\n\n[1] Mistretta, M., Baldrati, A., Agnolucci, L., Bertini, M., & Bagdanov, A. D. Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion. In The Thirteenth International Conference on Learning Representations.\n\nThis reduction in performance of the new model relative to the old one causes FCT to degrade the overall retrieval capacity of the system, failing to achieve compatibility, as it attempts to transform the higher-quality representations of the old model into the lower-performing representations of the new model. In contrast, our method introduces an additional loss that encourages both intra-class clustering and inter-model alignment of feature representations on the specific training dataset. As a result, the transformation $F$, due to its greater flexibility, improves the performance of the old model’s representations. Even in this challenging scenario, our approach outperforms FCT, further validating the robustness of our method.\n    \nThe intra-class clustering loss defined in Eq. 8 relies on the availability of class labels to encourage embeddings from the same class to cluster together while pushing apart embeddings from different classes. In scenarios where class labels are not available, Eq. 8 naturally reduces to an unsupervised contrastive loss, similar to the objective used for training CLIP models. In this unsupervised setting, we contrast pairs of representations originating from different models, and clustering—since it cannot be enforced directly—becomes a byproduct resulting from embedding similarity. Consequently, our approach is flexible and can be applied in both supervised and unsupervised training scenarios, depending on the availability of labels for the downstream task."}, {"review_id": "x7X1JTAdLP", "reviewer": "Reviewer_tsn4", "summary": "The paper proposes a new method to align the representation spaces of different models. The method leverages a smooth relaxation of the Heaviside function for a soft orthogonality constraint. The authors combine this regularization with 3 other losses: a forward loss, backward loss, and a contrastive loss for clustering. The paper shows that their method improves upon baselines for image retrieval and partial backfilling.", "questions": "1. In Table 1, what does Ind. Tr. mean? Is it expected that CMC and mAP of the last row of Ind. Tr. are the same as the last row of \"Ours\"? \n2. Are there other alternatives to truncating the higher-dimensional features to match like a learned down projection (end of 3.2)?\n3. Are there other applications of representation learning (besides retrieval) where soft orthogonalization could provide improvement? \n4. In Appendix D, L_B seems to be the least important component of the loss and only results in a slight decrease, do you have a sense of why? \n5. Line 921 in Appendix F should be deleted.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The exposition and motivation for the method are clearly explained. There's a nice mix of toy settings and visualizations for showing the intuition of how affine and orthogonal transformations effect representation space. \n2. The main method packages several ideas (soft orthogonal constraints, contrastive clustering, new backfilling) together into a new approach. The experiments are well-executed and support the paper's claims for improving retrieval compatibility and backfilling. The appendix also has a nice set of ablations for each component of the method. \n\nWeaknesses\n1. Some of the presentation of results are a bit unclear and hard to parse. Table 1 in particular would be easier to read if the best results were bolded / highlighted. Throughout section 4, the analysis of the experimental results would be more clear if specific numeric results were referenced (instead of just table numbers). \n2. The general complexity of the method (and addition of a threshold) may make it harder to use in practice. The paper does reference these limitations and provide ablations on the main components. \n3. The general scope of the paper seems quite specific to retrieval tasks; it's a bit unclear how many of the empirical results would be of interest or applicable for broader representation learning.", "was_revised": true, "final_justification": "My final recommended score is a 5 (updated from a 4 before rebuttal) with confidence 3. The initial version of the paper had some problems with clarity of presentation as well as some concerns on method complexity and scope. During the rebuttal, the authors resolved the clarity issues, provided reasonable discussions on practical complexity and broader applicability, and conducted additional experiments on how their method performs under stronger model and distribution shift from ResNet to CLIP. These discussions resolved my main concerns, so I've updated my score.", "rebuttal": "We thank the Reviewer for their thoughtful feedback and for highlighting several key strengths of our work. We appreciate the recognition of our clear exposition and motivation, as well as the use of toy settings and visualizations to illustrate the intuition behind affine and orthogonal transformations. We are glad the integration of soft orthogonal constraints, contrastive clustering, and novel backfilling into a unified approach was noted, along with the strong experimental support and comprehensive ablations. Finally, we value the acknowledgment of our method’s relevance to model update compatibility and its seamless integration into existing adaptation pipelines.\n    \n**[Suggestions for Clearer Presentation of Results in Table 1 and Section 4]**\n\nWe thank the Reviewer for these suggestions to improve the clarity of our manuscript. In the revised version, we will highlight the best results in each table. Furthermore, in Section 4, we will reference specific results directly in the text rather than referring only to the table numbers, as recommended.\n\n**[Practical Complexity and Usability of the Proposed Method]**\n\nWhile our method may initially appear complex, it is, in fact, straightforward to implement in practice. The approach requires training only two matrices, resulting in a small number of parameters to optimize. Moreover, because our method operates solely on the extracted embeddings, it does not require any knowledge of the underlying models and is therefore applicable across different objectives, architectures, and types of learned representations (as demonstrated in the tables referenced in Reviewer SkaR's response).\n\nIn contrast to previous methods, which either focus solely on alignment loss without any representation clustering loss (e.g., FCT), or require specific architectural components of the pretrained models (e.g., FastFill, which requires access to the classifier of the new model), our approach addresses these limitations. Additionally, while existing baselines provide only forward adaptation, our method is designed to achieve both forward and backward compatibility, thereby addressing practical needs that prior works do not meet. For instance:\n\n-  $B_{\\perp}(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$ yields higher retrieval values (i.e., better compatibility) compared to the baselines.\n- $B_{\\perp}(\\phi_{\\text{new}})/\\phi_{\\text{old}}$ can be achieved exclusively by our method. From a practical standpoint, this allows compatibility to be established even before all gallery items are forward-adapted using $F$.\n- Since our approach provides a unified representation space, even when the gallery is in a hybrid form (i.e., with some elements already adapted and others not), using $B_{\\perp}(\\phi_{\\text{new}})$ still ensures compatibility. This is something that neither FCT nor FastFill can achieve.\n  \n\nAs discussed in the Limitations section (Appendix G), the hyperparameter $\\lambda$ can be selected via cross-validation or a small hyperparameter search on a held-out subset of the downstream dataset. While automatic tuning of this parameter remains an open problem and a direction for future work—since we are the first to propose this form of relaxed regularization—the introduction of such a parameter is not uncommon. For instance, in CLIP training, the temperature parameter affects the training process and depends on the training dataset and architectures, yet it is not typically regarded as a limitation.\n    \n\n**[Definition of “Ind. Tr.” in Table 1]**\n    \nThe term \"Ind. Tr.\" stands for \"Independently Trained.\" To avoid any potential confusion, we will change this label to \"Ind. Train\" in the revised manuscript. This row in each table represents the retrieval performance obtained by directly using the independently trained models, without any form of adaptation.\n\n**[Clarification on Identical CMC and mAP Values in “Ind. Tr.” and “Ours”]**\n\nThis outcome is expected and intentional according to our methodology. This occurs specifically when a strict orthogonal transformation $B_{\\perp}$ is applied to the new model representation. Since $B_{\\perp}$ is an isometry, it preserves the geometry (angles and magnitude) of the new model’s embedding space and does not modify any relationships inside the representation. Therefore, the retrieval performance remains identical in both cases, as reflected in the last row of \"Ind. Tr.\" and the last row of \"Ours\".\n    \n**[Alternatives to Feature Truncation for Dimensionality Matching]**\n    \nYes, there are alternatives to truncating the higher-dimensional features to achieve dimensionality matching. One common approach is to apply zero padding to the smaller feature vector in order to match the dimension of the larger one. However, since the padded dimensions do not contain any information, especially in retrieval scenarios where cosine similarity is used, both truncation and padding yield equivalent results.\n\nAnother possibility is to employ a learned down-projection using a rectangular matrix. However, to ensure a strict orthogonality (isometry), the transformation matrix must be square. Furthermore, as noted by [36], enforcing orthogonality on a rectangular matrix through soft orthogonality regularization can lead to suboptimal solutions.\n\n    \n**[Broader Applicability Beyond Retrieval Tasks]**\n\nAs it is demonstrated in [36], soft orthogonalization has been applied to regularize all the weights of a CNN during training, and could benefit from the increased plasticity offered by our proposed $\\lambda$-orthogonal regularization. While retrieval is the standard scenario for evaluating compatibility [1], our approach is broadly applicable to any task that requires representation adaptation, as it focuses on model alignment and clustering of learned representations. As demonstrated in our downstream task adaptation experiments, our regularization approach yields improved performance compared to a strict orthogonal constraint, making it a valuable approach in domain adaptation scenarios as well. Furthermore, enforcing geometrical consistency while allowing adaptability has recently been investigated in the context of continual learning for multimodal training [2]. However, the authors of [2] promote this property indirectly through a knowledge consolidation loss, rather than by directly applying a regularization constraint. This highlights both possible future research and the potential applicability of our $\\lambda$-orthogonal regularization across various areas of representation learning. \n\n[1] Shen, Y., Xiong, Y., Xia, W., & Soatto, S. (2020). Towards backward-compatible representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6368-6377).  \n[2] Liu, W., Zhu, F., Wei, L., & Tian, Q. (2024). C-CLIP: Multimodal continual learning for vision-language model. In The Thirteenth International Conference on Learning Representations.\n\n[36] Bansal, N., Chen, X., & Wang, Z. (2018). Can we gain more from orthogonality regularizations in training deep networks?. Advances in Neural Information Processing Systems, 31.\n    \n**[Analysis of the Impact of $L_B$ in Appendix D]**\n\nTo assess the specific impact of the $L_B$ loss, one should refer to the results reported in the $B(\\phi_{\\text{new}})/\\phi_{\\text{old}}$ column of Table 10 and 11. The $\\mathcal{L}_B$ loss optimizes the alignment between the adapted new representation and the old representation via an isometric transformation $B$. Since the old representation is fixed (pre-extracted with the old model), it serves as a static reference to which the new model's representation is aligned through the optimization of $\\mathcal{L}_B$.\n\nAs the isometric transformation $B$ can only rotate or reflect the embedding space without altering its internal structure, the effect of $\\mathcal{L}_B$ alone is limited compared to the other loss components, which promote intra-class clustering and employ transformations with higher flexibility, as discussed in Section 3.4.\n\nFurthermore, when $L_C$ is combined with either $L_B$ or $L_F$, it can be observed that the backward compatibility performance (i.e., $B(\\phi_{\\text{new}})/\\phi_{\\text{old}}$) is higher when utilizing $\\mathcal{L}_B$ compared to $\\mathcal{L}_F$. This is because alignment to the old representation is explicitly enforced by $\\mathcal{L}_B$, whereas $\\mathcal{L}_F$ does not directly target it. The combination with $\\mathcal{L}_C$ further enhances intra-model alignment and backward compatibility performance.\n\n    \n**[Removal of Line 921 in Appendix F]**\n\nWe will remove the line in the revised version."}, {"review_id": "b7lUDp4QLY", "reviewer": "Reviewer_uQX7", "summary": "This paper addresses the important problem of adapting and aligning representations between independently trained retrieval models, a practical issue for maintaining model compatibility in real-world applications. The authors propose a method that introduces a relaxed orthogonality regularization during the learning of an affine transformation, aiming to balance adaptation flexibility with preservation of the original representation space. They provide extensive experiments across multiple architectures and datasets, demonstrating that the proposed method facilitates cross-model compatibility and maintains zero-shot performance.", "questions": "Could the authors clarify why they chose their specific form of relaxed orthogonality regularization? Have alternative forms like SRIP been considered or tested? Or simply multiply the original regularization term by a coefficient. Would simply lowering the standard orthogonality penalty achieve a similar effect to the proposed approach?\n\nHow does the proposed method compare, both theoretically and empirically, with existing methods such as (deep) canonical correlation analysis, which also aim to align representations while preserving information or distribution? The loss form of CCA is essentially the same as the MSE loss form proposed in this paper.\n\nThe loss functions for backward-compatibility and forward-adaptation seem very similar in terms of implementation. What practical differences arise from separating these objectives, and could one unified framework suffice?\n\nHow sensitive is the method to the regularization coefficient in the orthogonality constraint?", "limitations": "The novelty of the proposed orthogonality regularization is limited, given the existence of many similar techniques in the literature that are not considered or compared.\n\nThe empirical evaluation, while extensive, does not include direct comparison with established alignment and compatibility methods such as (deep) CCA or recent hybrid approaches incorporating autoencoders and regularization.\n\nThe theoretical distinction between backward-compatibility and forward-adaptation losses is not matched by a clear practical difference, reducing the impact of this contribution.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe problem of backward and forward compatibility between model representations is highly relevant, particularly for retrieval systems that must remain usable across model updates.\n\nThe paper provides comprehensive empirical results on a variety of architectures and datasets, strengthening the validity of the findings.\n\nThe proposed method is conceptually simple and can be readily integrated into existing model adaptation pipelines.\n\nWeaknesses:\n\nThe choice of orthogonality regularization and its specific relaxation is not sufficiently justified. There exist several alternative and arguably more principled forms of soft orthogonality constraints (e.g., spectral norm-based regularizations such as SRIP), which are neither discussed nor compared.\n\nThe distinction between their backward-compatibility and forward-adaptation objectives appears to be mainly conceptual; in practice, the loss implementations are almost identical, differing only in the direction of the transformation. This calls into question the novelty and practical benefit of splitting these objectives.\n\nThe paper overlooks substantial related work in representation alignment, especially canonical correlation analysis (CCA), deep CCA, and their regularized or autoencoder-augmented variants, which also aim to preserve information and distributional properties during alignment. The omission of discussion and comparison with these methods limits the positioning of the proposed approach in the broader literature.", "was_revised": true, "final_justification": "My concerns have been resolved.", "rebuttal": "We thank the Reviewer for their insightful comments, which helped us highlight our contributions. We appreciate their recognition of our method’s relevance to representation compatibility in model updates, its conceptual simplicity, and ease of integration. Below we address the questions raised by the Reviewer.\n\n**[Novelty and Justification of the λ-Orthogonality Regularization]**\n\nThe novelty of the λ-orthogonality regularization is that it allows for controlled local flexibility when adapting a pre-trained model to a downstream task. This distinguishes our approach from methods such as the Spectral Restricted Isometry Property (SRIP) and Soft Orthogonality (SO), which do not provide explicit control over the orthogonality constraint.\n\nSRIP [36] is a regularization method that uses the spectral norm to enforce soft orthogonality in a matrix $W$. The SRIP loss is defined as $L_{SRIP}=\\sigma(W^T W - I)$, where $\\sigma(\\cdot)$ denotes the largest singular value. This penalizes the worst-case deviation, i.e., $\\max_i|\\sigma_i(W)^2-1|$. However, $\\sigma$ is a non-smooth function, making the optimization of $L_{SRIP}$ more complex than optimizing a smooth objective. To reduce computational cost, [36] approximates the spectral norm using a two-step power-iteration method, which only provides an approximate solution.\n\nOn the other hand, SO is a much faster alternative, defined as:\n\n$L_{orth}=||W^{\\top}W-I||_F$\n\ni.e., the sum of deviations of all inner products from the identity. Its gradient has a closed form:\n\n$\\nabla_W L_{orth}=\\frac{2W(W^{\\top}W-I)}{||W^{\\top}W-I||_F}$.\n\nSince SO is simple and its gradient is explicit, we use it as a soft orthogonality constraint instead of SRIP. The Frobenius norm aggregates deviations across all inner products, so small changes in individual dimensions only slightly increase $L_{orth}$. This allows the transformation to distribute error across its entries, preserving overall proximity to the Stiefel manifold while still adapting the representation for downstream tasks.\n\nMotivated by this, we introduce a novel, controlled relaxation of $L_{orth}$ by constraining it with a threshold λ defined as:\n$$\n\\min_{W}||W^TW-I||_F\\quad\\text{s.t.}\\quad||W^TW-I||_F\\geq\\lambda\n$$\nThis formulation allows for increased local flexibility modulated by λ, in contrast to other regularizations which do not provide granular control over the proximity to the orthogonality condition.\n  \nWe also compared our λ-orthogonal regularization with SO and SRIP using the experimental setup in Tab. 9 where the represetations of two pretrained models are adapted to a downstream task.\n\n|Method|$F(\\phi_{\\text{old}})/F(\\phi_{\\text{old}})$| $B_{\\lambda}(\\phi_{\\text{new}})/F(\\phi_{\\text{old}})$| $B_{\\lambda}(\\phi_{\\text{new}})/B_{\\lambda}(\\phi_{\\text{new}})$ |ZS|\n|-|-|-|-|-|\n|SO (λ=0)|57.48|66.79|71.54 (–0.241)|–0.001|\n|SRIP|57.38|66.57|71.66 (–0.120)|–0.001|\n|**Ours (λ=12)**|**59.92**|**70.72**|**75.44 (+3.659)**|**+0.028**|\n\nWe observe that SRIP achieves retrieval performance comparable to SO. As detailed in Appendix D, SO corresponds to the case of our proposed λ-orthogonality regularization when λ=0. This further justify the choice of SO over SRIP, as it is faster and does not require any approximatation [36].\n\n[36] Bansal et al., “Can we gain more from orthogonality regularizations in training deep networks?”, NeurIPS 2018.\n\n**[Does Scaling the Standard Orthogonality Penalty Is Equal to the λ‑Orthogonality?]**\n\nThe Reviewer raises a very interesting point. However, the answer is no: simply multiplying the original regularization term by a coefficient that lowers the orthogonality penalty does not achieve the same effect as our λ-orthogonality regularization.\n\nThe key difference lies in the regularization itself. The λ-orthogonality is formulated as the constrained optimization problem:\n    \n$$\n\\min_{W}||W^\\top W-I||_F\\quad\\text{s.t.}\\quad||W^{\\top}W - I||_F\\geq\\lambda\n$$\n\nThis is different from multiplying the soft orthogonality penalty by λ:\n\n$$\n\\min_{W}\\lambda||W^{\\top}W-I||_F\n$$\n\nIn the former case, the minimum of the constrained problem is λ, while in the latter, the minimum will always be 0, even when scaled by λ. Therefore, there is no direct control on the final value of $||W^\\top W - I||_F$ and thus to the trade-off between plasticity and stability of the adaptation of the representation, that is instead demanded to the optimization process.\nFig. 2 empirically shows this effect with a toy problem. It can be observed how λ influences both the angle between the columns of $W$ and the final value of $||W^{\\top}W-I||_F$.\n\n**[Sensitivity of the proposed method on the λ value]**\n    \nThis is examined in Appendix D, where we perform an ablation study on the coefficient λ. Tab. 9 shows results for various λ values, while Fig. 5 demonstrates the trade-off between stability (strict orthogonality) and plasticity (no regularization); increasing λ improves downstream task performance but reduces zero-shot scores, especially when λ=$\\infty$.\n\n**[Relation to CCA and Related Alignment Methods]**\n\nWe thank the Reviewer for relating our method to statistical approaches like Canonical Correlation Analysis (CCA) and deep CCA. However, our approach is primarily inspired by and closely related to Procrustes analysis, which is empirically distinct from CCA as shown in [3]. To clarify our position, we summarize the main differences between Procrustes analysis and CCA.\n\n    \nProcrustes analysis has played an important role in aligning latent spaces of DNNs [4,5,28]. It provides correspondences between latent spaces of different models by estimating an optimal orthogonal transformation $R$ that best matches two paired point sets $X$ and $Y$ [9]. This is done by minimizing the least-squares error $|XR-Y|^2$. In contrast, CCA finds pairs of weight vectors $(a_i,b_i)$ such that the linear projections $U_i=Xa_i$ and $V_i=Yb_i$ are maximally correlated and mutually uncorrelated, producing a series of canonical variable pairs with descending correlations ($\\rho_1\\geq\\rho_2,\\ldots$). Thus, while Procrustes yields a single transformation, CCA returns multiple pairs of canonical variables.\n\nMoreover, their respective objective functions differ: Procrustes minimizes a least-squares distance, whereas CCA maximizes correlation. As a result, the two methods generally yield different optima as CCA is invariant to arbitrary full-rank linear transformations applied individually to $X$ or $Y$, whereas Procrustes is only invariant to joint orthogonal transformations of both sets. CCA and Procrustes coincide only in specific cases of 1-d and when both $X$ and $Y$ are whitened and have the same dimension. However, whitening usually harms performance by removing variance structure that encodes semantic information in representations of pretrained models. Therefore, using CCA to estimate transformation matrices leads to suboptimal retrieval performance.\n\nDirectly estimating the transformation $B$ via Procrustes analysis is, in principle, possible; however, it also limits the applicability of our method. Procrustes analysis yields only orthogonal mappings and is therefore suitable for determining the backward transformation. However, as discussed in Sec. 3.4, the forward transformation requires a degree of plasticity rather than stability (strict orthogonality). Furthermore, while Procrustes analysis is computed in a closed form, our approach necessitates batch-wise optimization as it involves other losses, such as the $L_C$. In the downstream adaptation setting, Procrustes analysis does not permit the slight modifications in the representation that our backward transformation, learned with λ-orthogonality regularization, allows. For these reasons, we adopt the loss functions discussed in Sec. 3.2, 3.3, and 3.4.\n    \n[3] Ding et al., “Grounding representation similarity through statistical testing,” NeurIPS 2021.\n\n[4] Wang & Mahadevan, “Manifold alignment using procrustes analysis,” ICML 2008.\n\n[5] Wang & Mahadevan, “Manifold alignment without correspondence,” IJCAI 2009.\n\n[9] Gower, “Generalized procrustes analysis,” Psychometrika, 1975.\n\n[28] Maiorca et al., “Latent space translation via semantic alignment,” NeurIPS 2023.\n\n**[Separation of Backward and Forward Transformations Objectives]**\n\nThe Reviewer is correct that both backward and forward transformations are learned with the same mean squared error (MSE) loss. However, while learned through the same function, the two objectives yield distinct optimization signals.\n\nIn backward adaptation, only the adapter $B_{\\perp}$ is optimized, aligning the new representation $\\mathbf{h}^t$ to the old one ($\\mathbf{h}^k$). Since $B_{\\perp}$ is an isometric transformation, the geometry of $\\mathbf{h}^t$ is preserved.\n\nIn contrast, forward adaptation updates the old representation $\\mathbf{h}^k$ via adapter $F$ to match $B_{\\perp}(\\mathbf{h}^t)$. Unlike $B_{\\perp}$, $F$ is not constrained to be orthogonal. As a result, the forward transformation enables greater flexibility to adjust the old representation, potentially improving it, as also shown by FCT and FastFill.\n\nAppendix E details the effects of each loss component individually and all their combinations. Jointly optimizing $L_F$, $L_B$, and $L_C$ yields the best performance across compatibility scenarios. As shown in Tab. 10 and 11, using only $L_B$ or $L_F$ affects performance depending on the transformation: optimizing only $L_B$ results in poor performance for $F$, and vice versa. The key novelty of our work lies in learning these two transformations jointly, as prior work focused only on forward adaptation and did not address backward compatibility."}]

Author rebuttal:

Meta-review: This paper tackles the important and practical problem of ensuring compatibility across independently trained neural network representations, with a focus on retrieval systems that require seamless model updates. The main contribution is a $\lambda$-orthogonality, which relaxes strict orthogonality in transformation learning to balance stability and adaptability. The authors propose joint learning of forward and backward transformations, coupled with contrastive clustering, to improve representation alignment. Experiments across diverse architectures and datasets -- including challenging distribution shifts (ResNet $\rightarrow$ CLIP, DINOv2) -- demonstrate the effectiveness of the method.

This paper makes a solid and practically relevant contribution to representation compatibility in retrieval systems. While the relation to prior work and the scope of applicability could be further elaborated in the final revision, the authors have adequately addressed these points during the rebuttal.
Decision: Accept (poster)