Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning
Authors: Ngoc Bui
Menglin Yang
Runjin Chen
Leonardo Neves
Mingxuan Ju
Rex Ying
Neil Shah
Tong Zhao
Venue: ICML 2025 poster (2025)
Keywords: backward compatible training
hyperbolic representation learning
alignment
Primary area: general_machine_learning->everything_else
TL;DR: We propose a backward compatible representation learning method using hyperbolic geometry

Abstract: Backward compatible representation learning enables updated models to integrate seamlessly with existing ones, avoiding to reprocess stored data. Despite recent advances, existing compatibility approaches in Euclidean space neglect the uncertainty in the old embedding models and force the new model to replicate outdated representations regardless of their quality, and thereby hindering the learning process. In this paper, we switch perspectives to hyperbolic geometry, where we treat time as a natural axis for capturing a model’s confidence and evolution. By lifting embeddings into hyperbolic space and constraining updated embeddings to lie within the entailment cone of the old ones, we maintain generational consistency across models while accounting for uncertainties in the representations. To further enhance compatibility, we introduce a robust contrastive alignment loss that dynamically adjusts alignment weights based on the uncertainty of the old embeddings. Experiments validate the superiority of the proposed method in achieving compatibility, paving the way for more resilient and adaptable machine learning systems.

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compattlle Representation Learning
Ngoc Bui
Menglin Yang'
Runjin Chen
Leonardo Neves
Mingxuan Ju
Rex Ying'
Neil Shah
Tong Zhao
Abstract
Orignn
Arrow of Time
bockward-compateee representation learning en- ables updated models to integrate seamlessly with existing ones, avoiding to reprocess stored data. Despite recent advances, existing compatibility approaches in Euclidean space neglect the uncer- tainty in the old embedding model and force the model reconstruct outdated representations regardless of their quality, thereby hindering the learning process of the new model. In this paper. we propose to switch perspectives to hyperbolic geometry, where we treat time as a natural axis for capturing a model's confidence and evolution. By lifting embeddings into hyperbolic space and constraining updated embeddings to lie within the entailment cone of the old ones, maintain generational consistency across models while ac- counting for uncertainties in the representations. To further enhance compatibility, we introduce robust contrastive alignment loss that dynami- cally adjusts alignment weights based on the un- certainty of the old embeddings. Experiments validate the superiority of the proposed method in achieving compatibility, paving the way for more resilient and adaptable machine learning systems.
Veator Datelosse
New
Embeddings Pufe)
Index
Old Model du
New Model Ch.
New Embeddingg
Entaliment Cone
(0)
(b)
Figure I. a) A typical setting of the problem. A large gallery set is embedded and indexed into a vector database using the old model. Updating the model may require re-indexing the entire vector database (backfilling). Backwand- compatible training allows the new model to query and retrieve from the vector database of old embeddings directly. b) Simulating the model evolution hyperbolic space with the entailment cone.
training
numerous applications: from semantic search (Zhu et al., 2023; Chandrasekaran & Mago, 2021) and recommenda- tion systems (Liu et al., 2024; Ko et al., 2022) to retrieval- augmented generation in large language models (LLMs), where vector database is used to find and integrate rele- vant information at inference time (Gao et al., 2023). The ubiquity of pretrained models such as BERT (Devlin, 2018), 8), CLIP (Radford et al., 2021), or GPT-style architectures (Rad- ford et al., 2018) makes these off-the-shelf embeddings read- ily available. Enterprises and researchers alike increasingly rely on these pretrained representations, adapting them to in-house data or tasks for streamlined development and fast deployment (Bommasani al., 2021).
1. Introduction
Representation learning has become an integral part of modern machine learning, offering a way to transform raw data-he it text, images, other modalities-into compact representations or "embeddings" that encode their essen- tial structure (Chen et al., 2022; Bengio et al., 2013; Guo et al., 2019). A high-quality embedding model can unlock
Yet, embedding models do not remain static. Model updates are often necessary to leverage more advanced architectures, integrate new domain knowledge, or address performance limitations (Wang et al., 2024). However, updating these models typically introduces significant challenges. large- scale retrieval pipelines or scenarios where legacy embed- dings have been indexed, these upgrades can be disruptive if older embeddings become incompatible with the new model's representations. Mcanwhile, re-embedding all data not only incurs high computational and operational costs but also raises privacy concerns (Price & Cohen, 2019), es- pecially in applications like face recognition (Chen et al.,
1 Work done during an internship Snap Inc. Yaae University, New Haven, CT, USA "The University of Texas at Austin, TX, USA 'Snap Inc., Bellevue, WA, USA. Correspondence to: Rex Ying <rex.ying yale.edu>, Tong Zhao <tong snsg.@omaa........
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Bockaard Representation Learning
2019) and person re-identification (Cui al., 2024). These issues highlight the need for backward -ackward-compatible represen- tation learning: A model update is considered backward- compatible if embeddings from the new model can interact and retrieve information from embeddings generated by its predecessor (see Figure 1). This property allows old and new embeddings to coexist seamlessly without the need to reprocess sensitive or extensive datasets (Shen et al., 2020).
and use the time-like dimension as natural axis to capture model evolution and the uncertainties inherent in vector embeddings. To capture the dynamic nature of embedding spaces, we employ the entailment loss (Ganea et al., 2018) to enforce a partial-order constraint between the old and new models. This constraint requires the new embeddings to re- main within the entailment cone of their older counterparts, maintaining backward compatibility while embracing the improved expressiveness brought by new data architec- tures. We further introduce robust hyperbolic contrastive loss that dynamically adjusts alignment weights based on the uncertainty levels of old embeddings, enabling more adaptive and reliable alignment across model generations. To the best of our knowledge, this the first attempt to leverage embedding geometry for maintaining consistency in updating models.
While several methods have been proposed for backward- compatible representation learning (Shen et al., 2020; Meng et al., 2021; Ramanujan et al.. 2022; Zhou et al., 2023; Pan et al., 2023; Biondi et al., 2024; Jang & Lim, 2024), they concentrate on Euclidean or hyperspherical spaces powered by the cosine distance. common strategy involves adding an extra loss term to align older and newer embeddings within the same space, compelling the new model to re- construct the old representations regardless of their quality. Consequently, if the old embeddings are suboptimal due to the limited capacity of the old model, the new model's 's ability to learn richer or more accurate representations is constrained. This mismatch between striet compatibility and model evolution becomes especially relevant as mod- els grow larger and more specialized, creating larger gap between old and new representations.
Through extensive experiments across diverse settings, we demonstrate that HBCT effectively enhances backward com- patibility without compromising the performance of new embedding models. Specifically, HBCT improves CMC@ I compatibility of the new model by 21.4% compared to the strongest Euclidean baseline, and yields a 44.8% increase in mAP compatibility.
We argue that hyperbolic geometry is a natural fit for describ- ing model evolution due to its unique properties. Firstly, its exponential growth of areas and volumes with respect to radius makes particularly suited for applications involving continual model updates as new entities or classes emerge. Secondly, recent studies (Khrulkov et al., 2020) show that hyperbolic encoders can effectively capture representation uncertainty, enabling 'free' uncertainty estimation along the time axis. This capability has been successfully applied to enhance performance across various tasks, including image segmentation (Atigh et al., 2022), action recognition (Franco et al.. 2023), and video prediction (Surís et al., 2021). In the backward-compatibl training problem, model updates can be seen as transformations of the embedding space over time, enabling new representations to evolve while main- taining consistency with their previous counterparts. This perspective aligns with the structure of Minkowski space- time (Stein, 1968), where hyperbolic geometry used to describe the space-time curvature (Einstein et al., 2013; Barrett, 2011) and model dynamical systems where relation- ships among particles evolve over time (Araújo & Viana, 2008).
2. Preliminaries
2.1. Riemannian Manifolds and Hyperbolic Geometry
A smooth n-dimensional manifold . is a topological space where every point E .M has a neighborhood Ux S M that is diffeomorphic (i.e., resembles) neighborhood V&= S R.. Each point has an associated tangent space Mx which is an dimensional vector space serving first-order local approximation of M. A Riemannian metric g on M is a smoothly varying collection = (0x)xEM of posi- tive definite bilinear forms (x( -) TxMM x T.MM + R. These bilinear forms determine the curvature at x, which measures how M deviates from the flatness at that point. A Riemannian manifold is the pair (M, g). Euclidean space RD can be viewed as a Riemannian manifold with the stan- dard Euclidean inner product and zero curvature. In contrast, hyperbolic spaces are Riemannian manifolds with constant negative curvature. This negative curvature leads to an ex- ponential growth of areas and volumes with radius, different from the polynomial growth in Euclidean space.
2.2. Lorentz Models of Hyperbolic Geometry
Proposed work. In this paper, we reframe backward- compatible representation learning through the unique lens of hyperbolic geometry and propose a novel frame- Training
There are several models that can represent n-dimensional hyperbolic spaces isometrically, such as the Poincaré ball model (Nickel & Kiela, 2017) or the Klein model (Mao et al., 2024). Following recent work (Desai et al., 2023; Bdeir et al., 2023), we use the Lorentz model (Nickel & Kiela, 2017) for its numerical stability (Mishne et al., 2023).
work, namely Hyperbolic Baucwardd-Copaatiie (HBCT), to align new models with legacy representations. The core idea is to lift the embeddings into hyperbolic space
2

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning
In what follows, we follow the notations and terminology from (Desai et al., 2023).
3. Problem Statement
Following previous literature (Shen et al., 2020; Zhang et al., 2022a), we focus on a typical retrieval setting, where we have large gallery G of items (e.g., images) maintained in a vector database. Each item E g is associated with one class or identity y/ E V' and is encoded into dense vector by an embedding model o( -).. During retrieval, each query q E Q is encoded into a vector embedding, i.e., ((g), and relevant matches are retrieved by comparing with the embeddings in the gallery ((g) = {Δ(x) a E 9)' using a distance function ..) The embedding model is trained dataset D {zz, y} with the base loss function:
Lorentz model embeds d' d-dimensional hyperbolic space in Rd+1 as one-sheeted hyperboloid, where each point x E Rd+1 can be written as [Tiime, Xxpace-]. where Ttime E IR acts like a "time coordinate" and Xspace E Ied like "spatial coordinates". Let ( be the usual Euclidean dot product. The Lorentzian inner product (.,) on Rd+1 is given by
c) (Xspace Yspace)
Frime Utime-
(1)
hyypeeroioonaaypyrroooiic space of curvature K, where An n-dimensional hyperbolic
K> > 0, in the Lorentz model is defined as
_ = fx Rd+11
x) =
1 K.
Ttime > 0). (2)
6+ = =arg min Cbaso D).
(8)
+
Here, we choose the loss function Lasse to be the cross- entropy, following (Shen al., 2020), but could also be any other metric learning loss functions (Kaya & Bilge, 2019; Wang et al., 2018; Chen et al., 2020).
All points x E Ld satisfy Zeime
V
R
| 112.
geodesic between two points is the locally smooth and shortest path connecting two points. Geodesics in L are curves traced by the intersection of L with hyperplanes in Id+ passing through the origin. The distance function between two points x, E Ld is
Let de be the existing encoder trained on an old dataset Dotd and &g is the existing gallery embeddings encoded using Φ. We now want to train a new encoder Φn to replace do subject to the arrival of new data D Pnew 2 Dold, more pow- erful architectures, or new pre-trained weights of existing architectures. Note that new data Duew may contain new identities or classes different from Dotd.
dc(x,y) V cosh (-K
The tangent space Tp td at a point p E Ld consists of all vectors in Euclidean space that are orthogonal to p:
Our goal is to train the new model 6,, in way that it could be directly compatible with the old model de: new queries encoded by on should yield similar or improved retrieval (4)
Tn fz
Rd+1, (p,z)c Ed = 0}.
performance on the existing gallery Φ8 produced by do(-). This property enables backfill-free the old embeddings be gradually replaced by (n as needed-while immediately benefiting from the new model's improved capabilities.
To "move" between the manifold and its tangent space, we use the exponential map expmp -Tpfd - Ld, defined as
(xpm,(2)= cosh(VKi]allaa + sinh (VK (z]lc)
2.
VK|la|ec
(5) Its inverse, the logarithmic map logp Ld + Tp sends
4. Methodology
E Ld back into the tangent space at p:
As mentioned in the introduction, directly foreing new em- beddings to reconstruct outdated representations from the old model can impair the learning process of the new model, because some old embeddings may be poorly represented. To address this, we propose to lift the embeddings to hyper- bolic space, where we leverage the time-like dimension as a natural axis to capture a model's confidence and evolution. Building on this framework, we introduce two uncertainty- aware losses, robust hyperbolic contrastive loss and entail- ment cone loss, designed enforce alignment and partial- order relationship between the old and new models while considering the quality of the old embeddings. The next
(x) cosh fosh-t(-2 (- (p,x))))
(x)),
(6)
V(K x))x
where projp(x) (x) maps a vector in ambient space dd+1 onto the tangent space p
+ (x) u + K Kp up.
These maps allow us to perform Euclidean operations in the tangent space while retaining the underlying hyperbolic structure once we map points back onto Ld Following the literature (Desai et al., 2023), we only use these maps at the origin of the hyperboloid
For simplicity, we abuse notation by letting á to take set of items & as the input and produce vector database containing all embedded items.
1/K T
3

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatibie Representation Learning
where Hw, {hh E [ h) 0} is the decision hy- perplane of the class y parametrized by Wy E Tooh. The term dc Hw,) is the minimum distance from h to the hy- perplane, which also admits a closed form using Euclidean reparameterization trick (Mishne et al., 2023). Since the base training is not our focus, we refer readers to (Bdeir et al., 2023) for full discussion. Our base objective is:
Uncertainty on Training Samples (Payd)
Uncertainty on Gallery Samples (G)
New issntiiss Old iies
New ideniliies
Old identities
10
Dercity
Densit
hasso (h) -oggp(( = =y]b)
0.0
0.2
0.4
0.6
0.0
0.2 0.4
0.6
1- IR,
1- (a,hhi
Figure 2. The distribution of uncertainty measures on hyperbolic embeddings of CIFAR 100 produced by the old model & The old model is R trained with the first 50 classes (old identities) and then used to embed the upcoming 50 classes (new identities) into the vector database. The gallery is unseen samples from the lest set. The curvature parameter K is set 1.0.
Uncertainty Estimation. Existing works (Franco et al., 2023; Atigh et al., 2022) typically use the hyperbolic uncer- tainty estimation in the Poincaré model, defining uncertainty as the f2-norm of hyperbolic embeddings. We adapt this definition of uncertainty to the Lorentz model by leveraging its isometric equivalence with the Poincaré model:
section presents our base hyperbolic encoder with an uncer- tainty measure for its embeddings. Section 4.2 and 4.3 will discuss the entailment and contrastive losses, respectively.
Uncertainty(h) = - hspace
heime
1
sinh( V K |211)
4.1. Hyperbolic Encoder and Uncertainty Estimation
=1 cosh( / K 2/|
VK|zl
We adopt a hybrid Euclidean Euelidean-Hyperbolii model (Khrulkov et al., 2020) to encode images into hyperbolic space. We choose this hybrid model instead of fully hyperbolic model (Bdeir et al., 2023) because it better integrates with any existing Euclidean encoders.
1 Lanh (VKTjzl).
1
(10)
VK
where z is the Euclidean embedding of h before the expo- nential map. This choice of the uncertainty measure results in the same formulation as the Pointcaré uncertainty mea- sure used in Franco et al. (2023) and Atigh et al. (2022). For K 1, the uncertainty is bounded in the range [0. 1].
Let z be the vector embedding of the input æ produced by given encoder (e.g., ResNet (He et al., 2016) or ViT (Doso- vitskiy, 2020)). We apply the exponential map expmo to lift [0, 2] E IR++] from the tangent space at the hyperbolic origin 0 onto the hyperboloid Cd,
Figure 2 shows the distribution of the hyperbolic uncertainty estimates for embeddings produced by ResNet18 trained on the first 50 classes of CIFAR 100. When this old model is used to embed images from the new 50 classes (unseen during its training), the resulting embeddings exhibit higher uncertainty and lower norms | compared to images from the first 50 classes (seen during the training). This obser- vation is consistent with (Atigh et al., 2022), showing the close relationship between an embedding's 's norm and its associated uncertainty in hyperbolic models.
(f())= = expmo (00, x) ))
Here, [0.z] E Rd+11 lies in the tangent space at the origin of the hyperboloid.
For convenience, in what follows, we denote the mappings from the input x to the hyperbolic embedding for the old and new models as:
(x)) x 6up 2o E Rd expmo h, cd ( : x on, Un E ddd expmo, hn E cdd
4.2. Entailment Models Evolution
When training the new model, we aim to keep it consis- tent with the structure discovered by the old model and then refine previously learned relationships. We model this consistency-then-eee refinementn dynamic via the notion of en- tailment cone (Desai et al., 2023; Ganea et al., 2018), which constrains the region where new embeddings can 'descend' from old embeddings in hyperbolic space. In other words, the cone represents the set of permissible embedding adjust- ments as the model evolves.
Training Hyperbolic Encoder. Similar to (Shen et al.,
2020), we train the hyperbolic encoder as a classification task in hyperbolic space by using hyperbolic multinomial logistic regression (Hyperbolic MLR) (Bdeir et al., 2023). The probability of a class y e y given the embedding h can be expressed as:
p(Y = y/h) x exp (sigu((w,
Hw,)). (9)
Formally, the entailment cone at h, in the hyperbolic space is defined by a half-aperture function that diminishes with
4

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning
Comparing
Gradients
1.0
InfoNCE (q
RINCE (q=0.01)
0.0
RINCE (qm0.1)
(q=0.5)
0.4 RINCE (Q=1.0)
0.6
0 ecd Bu ))))
dg Mu)
Sper(h,)
Figure 4. Comparison between InfoNCE and RINCE losses for varying distances between old and new embedding h., and h.. As y increases, the gradient norm decreases for noisy positive pairs (-dc E [-5, -3) and increases for clean positive pairs (-dg E 2-2, O)) For this illustration, we choose one negative sample with the distance de (hn, h() = 4.0 and B = 0.01.
hn
Figure 3. Illustration of the entailment cone. The more certain ho, the narrower the cone defined by h.-
distance from the hyperbolic origin (Desai et al., 2023):
the inherent trade-off between maintaining compatibility and achieving high performance in the new model.
aper(hh.) = sin 1 [K ho.apacell
(11)
To address this issue, we propose variant of RINCE-robust contrastive loss (Chuang al., 2022)-that adaptively adjusts the weight for positive old-new embedding pairs according to the uncertainty of the old embeddings.
where e a small constant (e.g. 0.1) preventing the degen- erate behavior near the origin.
The hyperbolic entailment cone loss penalizes the cases when h, lies outside h,'s entailment cone using the dif- ferences between the exterior angle ext(h, h,) = π Z(O, h., hn) and the aperture (see Figure 3):
Let B batch of images in the training data, RINCE loss for a pair (h,; h,,) E B is given by
1
Ccontrast Chnutrace(hni8 (hni
ho))
q
Centail (h, ho) = max (0, ext hn) aper(ho)),
9
+
B
Σ h
exp( hep(-de(hhe (bm, (b),)
(12)
where the exterior angle is computed by
where 3.q E (0, are hyper-parameters the RINCE loss. As q - 0, RINCE recovers vanilla InfoNCE (Oord et al., 2018) loss. As q - 1, RINCE reduces the weights for noisy pairs (with large distances) and focuses more clean pairs (low distances) Figure 4 for the comparison of the gradient norms for InfoNCE and RINCE with varying
ext(h,, h,)) =co== co
ba,eimme thomsime (K bn)e)
(lho,sp (I (bo.
Here, we penalize h, for each pair (ho hn) it lies outside of the entailment cone dictated by hg We refer readers to (Desai et al., 2023) for full derivations.
The RINCE loss is particularly suitable in the context of compatibility, where the goal is to emphasize the reconstruc- tion of good old embeddings while downweighting bad ones. To achieve this, make an explicit dependence of q to h, for each pair (hn, h,), defining q(h)) Uncertainty (ho). This allows the weights for positive pairs to adjust dynami- cally based on the quality of the old embeddings.
It is worth noting from Eq. (11) that the aperture at h, is inversely proportional to its norm. Therefore, as h, becomes more uncertain (lower norm), its cone 'widens', granting greater freedom for the evolutionary expansion in the new embedding space.
4.3. Uncertainty-aware Contrastive Alignment
Notice that the entailment cone restricts only the spatial coordinate changes of the embedding, without preserving pairwise distance relationships between old and new embed- dings. Meanwhile, contrastive alignment loss (Biondi et al., 2024; Zhang et al., 2022a) or mean distortion (Cui et al., 2024) may hinder the learning of the new model since they naively enforce the new model to reconstruct outdated rep- resentations regardless of their quality. This may exacerbate
4.4. Training Pipeline
The overall loss function to train the updated model is:
L = L + X(Centail + Lcontrest),
(13)
where À is the weight for the alignment regularization. Here, we make no assumption about Cbsse. making it applicable to various training scenarios, including unsupervised and
5

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward- Representation Learning
self-supervised learning, which drive recent advancements in foundation models. We leave further exploration of these settings for future work.
We simulate the arrival of new classes and new architecture. The old model is ResNet18 trained with 50% of classes, and the new model is ViT-B-16 finetuned on the full dataset.
Both:
Numerical Stability. Because the hyperbolic distance grows quickly with respect to the norm |z||, it is often nec- essary to clip or rescale norms to maintain well-conditioned embeddings. We follow (Bdeir et al., 2023) and (Desai et al., 2023) to rescale the norm of z by vd and then clipping Ç to ensure stability. For the old model, we set Ço 1.0 and for the new model, we increase the clipping threshold by S. = So + 0.2 to account for the evolution in the embedding space of the updated model.
We compare our method with other regularization- based methods in Euclidean space, including l2 distance regularization, BCT (Shen et al., 2020), Hot- Refresh² (Zhang et al.. 2022a), AdvBCT (Pan et al., 2023), and HOC* (Biondi et al., 2024).
Baselines,
Metrics. We assess retrieval performance using two metrics: (I) Cumulative Matching Characteristics (CMC@k): This metric measures top-k accuracy by ranking gallery represen- tations based on their similarity to the query representation. A query is considered correctly retrieved if a sample from the same class appears within the top- ranked results. We report CMC top-1 and top-5 scores for all models; (2) Mean Average Precision (mAP): mAP integrates precision and re- call by computing the area under the precision- curve. We use mAP@1 1.0, which represents the average precision over recall values ranging from 0.0 to 1.0.
5. Experiments
We conduct extensive experiments and compare with ex- isting approaches in Euclidean space to demonstrate the effectiveness of our method. We refer to our method as Hyperbolic Backward Compatible Training (HIBCT). The source code is available at htapp:///qthhule com/
snap-
We define metrics to assess the backward backward-compatibee compatibility of an alignment method following (Shen et al., 2020; Pan et al., 2023). Let o. be the base new model trained without any alignment method. The compatibility metric for new model on trained with alignment method is computed by
5.1. Experimental Settings
Datasets. We use CIFAR100 (Krizhevsky et al., 2009) and Tiny-Imagener (I.e& & Yang, 2015), where the test sets serve as hold-out query and gallery sets for evaluating both retrieval performance and compatibility.
M( Q): (9)) M() o.()) Poom = M(6. (Q); o. (9)) ((0.(9):
(14)
Models. We employ two model architectures: ResNet18 (He et al., 2016) and ViT-B-16 (Dosovit- skiy, 2020). ResNer18 is trained from scratch, whereas ViT-B-16 is initialized with weights pre-trained on ImageNet21k (Ridnik et al., 2021). The goal is to replicate the typical practice of obtaining and fine-tuning pre-trained models, which then function external alternatives to replace the currently fine-tuned model.
where -(.:-) is an evaluation metric for the retrieval task, which is either CMC@K or mAP. Since backward- compatibility may come with trade-off in the performance of the new model, we also compare the performance with the base model without alignment o.
Pup (.))))()))
(C)) (15)
Scenarios. We consider four settings to capture a range of real-world use cases:
M(0.(9):0.199)
Implementation Details. For all the models, we use the feature dimension of 128. For ResNet18 model, we use 200 epochs, learning rate 0.1 and for finetuning ViT-B-16, we use 80 epochs with learning rate 0.005. For both models, we use SGD (Ruder, 2016) as the optimizer with a batch size of
Extended data: We randomly select 30% of the training dataset and use it as Dold to train the old model. D Drew will be the the full dataset. The model architecture is ResNet18 and is kept the same for the new model.
Extended class: We extract from the first 50% of classes (~50 classes for CIFAR100 and 100 classes for Tinylmagenet) as Data. Meanwhile, Dnew will be the full dataset. The model architecture is ResNet18 and is kept the same for the new model.
Hoft-Refresh and HOC are two alignment methods that lever- age hyperspherical geometry by using a contrastive objective based on cosine similarity. The key difference between them lies in the selection of negative pairs.
"Note that in (Biondi et al., 2024), the authors proposed using fixed d-simplex classifier to improve compatibility, However, this approach requires predefining the number of dimensions to match the number of classes and maintaining it consistently across model updates, which is usually not practical. Therefore, we adopt only their high-order alignment loss (HOC) while removing the constraint of a fixed classifier.
We assume that we have access to the full training dataset, i.e., Date = D The old model is ResNet18, and the new model finetunes on VIT-B-16 pretrained on Imagenet21k.
New architecture:
6

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatibk. Representation Learning
The comparisons of different backward compatibllity methods. Here, &e indicates that the model produces Euclidean embeddings while of produces hyperbolic embeddings. The columns self and cross refer to the original new-to-new and new-to-old retrieval performance (CMC@ @ I or mAP), respectively. Gray rows are base models without compatibility, and the pink rows indicate the proposed method, Bold indicates the best performance and underline indicates the second best method. When the ablated version of HBCT (i.e., without entailment loss) outperforms the best Euclidean baseline, we highlight it by dashed underline.
Table 1.
CIFAR100
Tiny InyImageNet
Scenario
Old Model
New Model
CMC#1
mAP
CMC81
mAP
self
CRUSS
Pap
Ponn
self
Pap
Prom
zelf
cross
Pup
Picco
self
cross
Pap
0,572
0.672
0.373
0.248
0.716
0.019
0.000
-3,846
0.654
0.072
0.000
-2.195
0.540
0.005
0.000
-2.200
0.420
0.010
0.000
-1.384
0.680
0.645
-0.049
0.510
0.627
0.537
-0.042
0.355
0.538
0.474
-0.004
0.603
0.421
0.311
0.0012
0.362
0.701
-0.014
0.612
0.546
-0.018
0.406
0.702
0.407
0.309
-0.032
0.351
Refiesh
0.718
0.671
0.008
0.693
0.655
0.544
-0.003
0.396
0.545
0.475
0.009
0.610
0.425
0.312
0.011
3.369
Est- datta
AdvECT
0.698
0.681
-0.025
0.700
0.638
0.543
-0.025
0.389
0.493
0.421
-0.088
0.286
0.376
0.287
-0.105
0.227
HOC
0.720
0.670
0,006
0.685
0,656
0.544
0.003
0.396
0.552
0.475
0.022
0.610
0.428
0.313
0.019
0.372
0.576
0.155
0.379
0.235
0.722
0.011
0.000
-3,851
0.651
0.012
0.000
-2.263
0.548
0.002
0.000
-2.230
0.410
0.006
0.000
-1.282
HBCT w/o entail
0.722
0.700
-0.001
0.847
0.653
0.564
-0.002
0.547
0.542
0.542
-0.011
0.966
0.409
0.329
40.002
0.539
HBCT
0.716
-0.002
0.656
0.567
0.001
0.560
0.551
0.007
1.076
0.414
0.334
0.572
0.376
0,310
0.273
0.211
0.716
0.019
0.000
-1,052
0.654
0.019
0.000
-0.85
0.007
0.000
-0:991
0.420
0.008
0.000
.0.971
0.712
0.478
-0.006
0.302
0.653
0.342
-0.002
0.003
0.536
0.307
-0.008
0.127
0.418
0.226
-0.005
0.071
0.695
0.447
-0.029
0.210
0.624
0.328
-0.046
0,054
0.514
0,292
-0.049
0.073
0.401
0.217
-0.046
0.029
0.715
0.498
-0.001
0.360
0,649
0.348
-0.007
0.110
0.534
0.334
-0.012
0.229
0.414
0.231
-0,016
0.094
Ext- class
0,708
01459
-0.011
0.344
0.642
0.337
-0.018
D.080
0.524
0.254
-0.030
-0.770
0.404
0.213
-0.039
0.007
O -HOC
0.713
0.490
-0.003
0.651
0.346
-0.005
0,106
-0.005
0.258
0.416
0.233
-0.011
0.102
0.425
0,330
0.299
0.212
0.002
0.000
0.012
0.000
-0.99
-1.178
0:110
0.007
-1.03
HBCT w/o entail
0.726
0.555
0,006
0.435
0.655
0.388
.0.001
0.176
0.547
0.344
-0.003
0.177
0.404
0.236
.0.014
0.120
HICT
0.722
0.572
00.001
0.495
0.655
0.398
-0.001
0,209
0.553
0,373
0,009
0.295
0,408
0,216
-0.005
0,169
0.714
0,652
0.515
0.425
0,900
0.021
0.000
-3.706
0.879
0.029
0.000
-2.744
0.812
0.002
0.000
-2.039
0.745
0.006
0.000
-1 192292
0.896
0.778
-0.006
0,875
0.728
-0.006
0.337
0.807
0,703
-0.006
0.593
0.732
0.534
-0.017
0.346
04
BCT
0.897
0.830
-0.004
0.623
0.882
0.750
0.003
0.432
0.809
0,757
-0.003
0.794
0.738
0,546
-0.009
0.384
Refresh
0.810
-0.003
0,317
0,876
0.743
-0.003
0.402
0.656
0.731
0.537
-0.019
0.356
New-
AdvBCT
0.891
0.797
-0.011
0.449
0.868
0.737
-0.012
0.374
0.799
0.658
-0.016
0.423
0.722
0,517
-0.031
0,294
Arch
HOC
01895
0.813
-0.006
0.744
-0.002
0.408
0.655
0.728
0.537
-0,023
0.723
0.637
0.546
0.413
0.897
0.009
0,000
4,085
0,882
0.011
0.000
-2.876
0.800
0,002
0.000
-2.099
0.746
0,006
0.000
-1.221
HBCT w/o entail
0.898
0.6915
0.001
40.178
0,882
0.680
-0.001
0.099
0.7936
0.006
0.941
0.751
0.571
0.006
0.475
HBCT
0.893
0.886
00.044
0.937
0.879
0.782
-0.004
0.555
0.811
0.804
0.002
0.982
0.746
0,580
0.000
0,500
0.315
0.281
0.222
0,900
0.018
0.000
-0.716
0.879
0.028
0.000
-0.509
0.811
0.007
0.000
-0.529
0.744
0.005
0.000
-0.125
0.888
0.482
-0.013
0.185
0.857
0.382
-0.025
0.119
0.794
0.451
-0.021
0.321
0,712
0,293
-0.042
0.137
BCT
0.895
0.358
-0.006
-0.051
0,880
0,345
0.002
0,053
0.818
0.101
0.009
0,233
0.750
0.282
0.008
0.115
Hot--Rfrrehh
0.891
0.490
-0.010
0.202
0.867
0 384
-0.013
0,122
0.802
0.474
-0.011
0.365
0.732
0.296
40,015
0,143
Both
AdvRCT
0.878
0.513
-0.025
0,842
0.377
-0.041
0.111
0.802
0.416
-0.010
0.255
0.730
0.288
-0.018
0.126
HOC
0.896
0.445
-0.044
0.116
0.869
0.377
-0.010
0.110
0.798
0.412
-0.016
0,247
0.725
0.287
-0,025
0.126
0.332
0.896
0.025
0.000
-0.831
0.881
0.013
0.000
-0.583
0.834
0.008
0.000
-0.556
0.753
0.006
0.000
-0,384
HBCT entail
01.877
0.188
0.000
-0.487
0.221
0.000
41.204
0.192
-0.201
0.183
40.102
-0057
HBCT
0.897
0.575
0.001
0.324
0.879
0.417
.0.002
0.154
0.812
0.476
-0.003
0.348
0.745
0.313
.0.011
0.184
128. We set the weight decay to 5e-4 and momentum to 0.9,
Table Ablating objectives in HBCT.
and use a cosine annealing schedule to tune the learning rate. The old models are trained only with the base classification loss in Eq. (8), and the new model is equipped with one of the alignment methods. We vary the alignment weight in the range [0, 1.0] and for methods involve contrastive alignment loss, we choose the temperature T E {0.5, 1.0]. For each baseline, we run with 10 combinations of these hy- perparameters and choose the best run that at least does not hurt the performance of the new model. For our method, we fix the curvature K- = 1.0 and we found that performance is consistent across runs, so we fix > = 0.3, r = 0.5, = 0.01 (similar to RINCE) for all the settings. The retrieval distance used for Euclidean models is the cosine similarity distance while Lorentz models is the geodesic distance as in Eq. (3).
Ablation
CMC
mAP
Pup
Poom
Pap
Peom
Hot-Retresh
-0.001
0.36
-0.007
0,110
HBCT w/ RINCE w/ entail
0.001
0.495
40.011
0.209
HBCT w/ RINCE w/o entail
0.006
0.435
-0.001
0.176
-BCT w/ InfoNCE w/ entail
-0.002
0.475
-0.009
0.202
HBCT w/ InfoNCE w/o entail
0.001
0.431
-0.001
0.193
5.2. Empirical Results
Table 1 reports the compatibility performance of competing methods. Notably, HBCT's Prom surpasses other Fuclidean baselines by a substamtial margin in most settings. In partic- ular, HBCT imppro improves CMC@ by 21.4% on average com- pared to the strongest Euclidean baseline, and yields a 44.8% increase in mAP. These compatibility gains can be achieved
7

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compattiee Representation Learning
5
45
1.4
7P%
3.9%%
MM
47%
74
Hltch
Laut
1296
17476
XML
4.
411
19
27 10L
(a) BCT
(b) HotRefresh
(c) HOC
(d) HBCT
Figure 5. The compatibility matrix of different alignment methods after five consecutive model updates. The value at column j and row i quantifies the CMC@1 compatibility Poom where the queries are embedded by the model Φ) and gallery embedded by Øj-
with only minimal compromises to the new model's perfor- mance, as shown by the Pup scores. Meanwhile, among alignment methods in Euclidean space, contrastive- based losses (HOC and Hot-Refresh) show relatively consistent performance. BCT achieves strong compatibility in certain scenarios (e.g., extended data and new architectures) but often entails a sacrifice in the new model's performance.
ity of baseline methods rapidly declines.
5.4. Ablation and Discussion
We also showed in Table I the performance of an ablated version of HBCT where we remove the entailment cone loss. We observe a significant drop in terms backward compatibility in scenarios where we update the architecture to ViT-B-16. However, in some scenarios, HBCT, HBCT without entailment cone loss, also outperforms the best baselines in Euclidean space. We conduct ablation study to further probe this behavior.
Note that we do not directly compare methods based on original retrieval metrics such as CMC@ I or mAP shown in self (new-to-new) and cross (new-to-old) columns in Ta- ble 1, because the 'old model' differs between Euclidean and hyperbolic spaces (Ø vs. <C)), leading inherently dif- ferent retrieval performance. For instance, in extended-class scenarios, hyperbolic encoders often yield better old-model performance than their Euclidean counterparts, whereas Euclidean encoders generally excel in extended-date and new-architecture scenarios. This behavior may stem from hyperbolic representations" stronger ability handle out- of-distribution samples (Li et al., 2024). However, this difference is not the main focus of our paper. Instead, we primarily rely on Peom and Pup. which are calibrated to account for variations in old-model performance.
In the extended-class scenario on the CIFAR 100 dataset, we replace the RINCE loss with a standard InfoNCE loss (us- ing geodesic distance in hyperbolic space as the similarity measure) evaluate its impact. In Table RINCE offers substantial improvement in CMC@ @ 1-based compatibility, while also reducing the sacrifice in mAP's Pup- Interestingly, even HBCT with vanilla InfoNCE on the hyperbolic space can outperform the best baseline in Eu- clidean space. This aligns with our intuition that hyperbolic geometry better captures the evolution of the embedding space by organizing embeddings across models along the time dimension depending on their capabilities. Figure in the appendix illustrates how the embeddings evolve from the old model to the new one. As the model improves, the uncertainty in the gallery embeddings decreases. Additional ablations for hyperparameters are provided in the appendix.
observe that
5.3. Sequential Model Updates
This section investigates the compatibility of alignment methods when applied over multiple model updates. We used the CIFAR-100 dataset for this purpose, segmenting the training data into five sets. Specifically, 20 new classes were incrementally added with each update, leading stages with 20, 40, 60, 80, and ultimately all 100 classes. The base model for the initial two updates (time steps 0 and 1) is ResNet18, which is then transitioned to ResNet50 for the remaining three updates (time steps 2, 3, and 4) to ex- plore architectural shifts. For each method, we report the CMC@ compatibility matrix with Poom for HBCT, HOC, HotRefresh, and BCT. The results Figure show that HBCT provides superior compatibility maintenance with prior models across multiple updates, while the compatibil-
6. Related Work
Compatible Representation Learning. Aligning represen- tations across deep learning models has been widely studied in applications like representation transfer (Li et al., 2015) and model stitching (Bansal et al., 2021) ) to multi-modal foundation models (Fei et al., 2022). retrieval- cations, Shen al. (2020) is the first to introduce the concept of backward-compatible training avoid backfilling the in- dexed embeddings. They enforce compatibility between
appli-
8

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning
trieval (Khrulkov al., 2020; Wei et al., 2024), and infor- mation retrieval (Ganea et al., 2018; Dhingra et al., 201 8). Recent studies have also demonstrated the benefits of hyper- bolic geometry for (multi-modal) foundation models (Desai et al., 2023; Ibrahimi et al., 2024; Pal et al., 2024; Yang et al., 2024). In contrast to these works, we are the first to investi- gate the application of hyperbolic geometry in capturing the evolution of models across updates.
More Unoeraat
(nse
2
7. Conclusion
0
We introduced HBCT, a framework that leverages hyper- bolic geometry to ensure seamless model updates while accounting for representation uncertainty. By constraining embeddings within the entailment cone of old ones and introducing dynamic hyperbolic contrastive loss, HBCT maintains compatibility without hindering expressiveness. Experiments show that our approach outperforms Euclidean- based methods in achieving robust backward compatibility. This work highlights the potential of hyperbolic geometry for scalable model evolution.
-2
-3
(,10)
-4
-4 -2
0 ho
2
4
Figure 6. Visualization of old and new gallery embeddings in CI- FAR We compress 128-dimensional embeddings into a 2- dimensional hyperboloid using UMAP (MeInnes al., 2018) and visualize them in the tangent space ToL . The top histogram is the distribution of the uncertainty estimation for those embeddings.
Limitations and future work. While HBCT is adaptable to various settings, we follow prior work by focusing on supervised learning scenarios. Investigating its effective- ness in self-supervised learning or multi-modal settings is an interesting research direction. Furthermore, to account for embedding space evolution, we gradually increase the clipping norm threshold. This may lead to instability as norms grow (e.g., 10.). A potential solution is to backfill and rescale the time dimension after set number of updates (e.g., . 50 updates), though developing a more principled ap- proach for handling this challenge is an important direction for future work.
old and new models by transferring the learned classifier from the old model influence the new one. While this method facilitates compatibility, it can compromise the ex- pressiveness of the new model. To address this, subsequent studies have explored various techniques to improve the reg- ularization loss, such as contrastive- based alignment( (Zhang et al., 2022a; Biondi et al., 2024), adversarial loss (Pan et al., 2023), and the alignment of class centroids between old and new models (Zhang et al., 2022b). However, these ap- proaches primarily rely on Euclidean geometry, while we shift the focus to hyperbolic geometry and leverage its natu- ral uncertainty estimation to adaptively adjust the alignment.
Acknowledgements
The authors gratefully acknowledge the generous support of Snap Research, the Amazon Research award, and the Samsung Research America award to the lab. Their funding and resources were instrumental in enabling and completing this work.
On a different front, Hu et al. (2022); Wang et al. (2020) pro- posed adding an extra projection to map the old embedding into the new embedding space, while Zhou al. (2023); Ramanujan al. (2022) introduced auxiliary dimensions to increase the expressiveness of either the old or new em- models, recent
Impact Statement
beddings. Building on bockward-compation research has explored dynamic techniques to progressively backfill the gallery in real-time (Zhang et al., 2022a; Jaeckle et al., 2023; Yan et al., 2021). These approaches are orthog- onal and can be integrated into our method.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hyperbolic Representation Learning. Hyperbolic ge- ometry is primarily known for its ability to better capture hierarchical, tree-like structures (Yang et al., 2023; Peng et al., 2021), which enhances performance in various tasks, including molecular generation (Liu et al., 2019), recom- mendation (Yang et al., 2021; Li et al., 2021), image re-
References
Araújo, V. and Viana, M. Hyperbolic dynamical systems. arXiv preprint arXiv:0804.312: 2008. 2
Atigh, M. G., Schoep, J., Acar, E.,, Van Noord, N., and
9

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatibe Representation Learning
Mettes, P. Hyperbolic image segmentation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4453-4462, 2022. 2,
Cui, Z., Zhou, J., Wang, X., Zhu, M., and Peng. Y. Learning continual compatible representation for re-indexing free lifelong person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16614-16623 2024. 2,5
Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model
stitching to compare neural representations. Advances in neural information processing systems, 34:225-20 236, 2021. 8
Desai, K., Nickel, M., Rajpurohit, T., Johnson, J., and Vedantam, S. R. Hyperbolic image-text representations. In International Conference on Machine Learning, pp. 7694-7731, PMLR, 2023. 2, 4. 5, 6, 9,
Barrett, F. The hyperbolic theory of special relativity. arXiv preprint arXiv: / 102.0462, 2011.
Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXi::1810004003 2018.
Bdeir, A., Schwethelm, K., and Landwehr, N. Fully hyper- bolic convolutional neural networks for computer vision. arXiv preprint arXiv:2303.
2023. 2, 4, 6, 13
Bengio, Y., Courville, A., and Vincent, P. Representation learning: review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8):
Dhingra, B., Shallue, C. J., Norouzi, M., Dai, A. M., and Dahl, G. E. Embedding text in hyperbolic spaces. CoRR, abs/1806. 001//036,, 2018. URL ptttp://arxiv...
1828, 2013.
abs/18066 04313.
Biondi, N., Pernici, F., Ricci, S., and Del Bimbo, A. Station- ary representations: Optimally approximating compatibil- ity and implications for improved model replacements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 28793-28804, 2024. 2,5,6, 6,9
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 4,
Einstein, A., Lorentz, H. A., and Weyl, H. The principle of relativity. Courier Corporation, 2013.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021. 1
Fei, N., Lu, Z., Gao, Y., Yang. G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. Towards artificial general intelligence via a multimodal foundation model. Nature Communications, 13(1):3094, 2022. 8
Franco, L., Mandica, P., Munjal, B., and Galasso, F. Hyperbolic self-paced learning for self-supervised skeleton-based action representations. arXiv preprint TrXiv:2333006244 2023. 2,
Chandrasekaran, D. and Mago, V. Evolution of semantic similarity survey. ACM Computing Surveys (CSUR), 54(2):1-37, 2021.
Chen, K., Wu, Y., Qin, H., Liang, D., Liu, X., and Yan, J. R3 adversarial network for cross model face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Ganea, O., Bécigneul, G., and Hofmann, T. Hyperbolic entailment cones for learning hierarchical embeddings. In International conference on machine learning, pp. 1646- 1655. PMLR, 2018. 2, 4,
2019.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.3
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang. H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv: 10997, 2023.
Chen, W., Liu, Y., Wang, W., Bakker, F. M., Georgiou, T., Fieguth, P., Liu, L., and Lew, M. S. Deep learning for instance retrieval: A survey. IEEE Transactions Puttern Analysis and Machine Intelligence, 45(6):7270- 7292, 2022.1 1
Guo, W., Wang, J., and Wang, S. Deep multimodal represen- tation learning: A survey, leee Access, 7:63373-63394, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016. 4,
Chuang. C.-Y., Hjelm, R. D., Wang. X., Vineet, Joshi, N., Torralba, A., Jegelka, S., and Song, Y. Robust con- trastive learning against noisy views. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16670-16681, 2022.
Hu, W., Bansal, R., Cao, K., Rao, N., Subbian, K., and Leskovee, Learning backward compatible embeddings.
10

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward Representation Learning
In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 3018-3028, 2022. 9
Liu, Q., Nickel, M., and Kiela, D. Hyperbolic graph neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc. 2019. URL
Ibrahimi, S., Atigh, M. G., Noord, N. V., Mettes, P., and Worring, M. Intriguing properties of hyperbolic embeddings in vision-language models. Transactions Machine Learning Research, 2024. ISSN 2835- http///pneeveeeeeee/e.e
oc/ppper__ilee/ppper////////// neurips. 10330165543331777111133332233333 pdf.
8856. URL
id-P5D2gfi4Go 9
Liu, Q. Hu, J., Xiao, Y., Zhao, X., Gao, J., Wang, W., Li, Q. and Tang, J. Multimodal recommender systems: A survey, ACM Computing Surveys, 57(2): 2024.
Jaeckle, F., Faghri, F., Farhadi, A., Tuzel, O., and Pouransari, H. Fastfill: Efficient compatible model update. arXiv preprint ar arXiv:2303.04766, 2023. 9
Mao, Y., Gu, J., Werner, M. C., and Zou, D. Klein model for hyperbolic neural networks. arXiv preprint arXiv:2410.168 2024. 2
Jang, Y. K. and Lim, S.-n. Towards cross-modal backward- compatible representation learning for vision- languuge models. arXiv preprint arXiv:2405. 2024.
McInnes, L., Healy, J., Saul, N., and Grossberger, L. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Kaya, M. and Bilge, H. S. Deep metric learning: A survey. Symmetry, 1 11(9):1066, 2019. 3
Khrulkov, V., Mirvakhabova, I.,, Ustinova, E., Oseledets, I., and Lempitsky, V. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4428-6428, 2020. 2, 4, 9
Meng, Q., Zhang, C., Xu, X.,, and Zhou, F. Learning com- patible embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9939- 9948, 2021. 2
Mishne, G., Wan, Z.,, Wang, Y., and Yang. S. The nu- merical stability of hyperbolic representation learning. In International Conference on Machine Learning, pp. 24925 PMLR, 2023, 2,
Ko, H., Lee, S., Park, Y., and Choi, survey of recom- mendation systems: recommendation models, techniques, and application fields. Electronics, 11(1):141, 2022. 1
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Nickel, M. and Kiela, D. Poincaré embeddings for learning hierarchical representations. Advances in neural informa- tion processing systems, 30, 2017. 2
Law, M., Liao, R., Snell, J., and Zemel, R. Lorentzian dis- tance learning for hyperbolic representations. In Interna- tional Conference on Machine Learning. pp. 3672-367600 3681. PMLR, 2019. 13
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: 887.03748, 2018. 5
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Pal, A., van Spengler, M., di Melendugno, G. M. D., Fla- borea, A., Galasso, F., and Mettes, P. Compositional entailment learning for hyperbolic vision-language mod- els. 2024. URL https: //apxiaa..vvv///a///aa org/ abs/2410 06912.
Li, H., Chen, Z., Xu, Y., and Hu, J. Hyperbolic anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17511- 17520, 2024.
Pan, T., Xu, F., Yang, X., He, S., Jiang, C., Guo, Q. Qian, F., Zhang, X., Cheng, Y., Yang, I.., et al. Boundary-aware representation via adversarial learn- ing in image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15201-15210, 2023. 2, 6.
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hoperoft, Convergent learning: Do different neural net- works learn the same representations? arXiv preprint arXiv: 15t:::1110.04440 2015. 8
Li, Y., Chen, H., Sun, X., Sun, Z., L., Cui, L.,, P.S., and Xu, G. Hyperbolic hypergraphs for sequential recom- mendation. In Proceedings of the 30th ACM international conference on information & knowledge management, pp- 988 988-997, 2021.
Peng, W., Varanka, T., Mostafa, A., Shi, H., and Zhao, G. Hyperbolic deep neural networks: A survey. IEEE Trans- actions on pattern analysis and machine intelligence, 44 (12):10023-10044 2021.

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatibbe Representation Learning
Price, W. N. and Cohen, I. G. Privacy in the age of medical big data, Nature medicine, 25(1):37-43, 2019.
Processing, 33:1768-1781, 2024, doi: 10.1109/TP..024. 3371358.
Radford, A.. Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre- training. 2018. 1
Yan, S., Xiong, Y., Kundu, K., Yang, S., Deng, S., Wang, M., Xia, W., and Soatto, S. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14299-14308, 2021. 9
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learming, pp. 8748-8763, PMLR, 2021.
Yang, H., Chen, H., Li, I.., Philip. Y., and Xu, G. Hyper meta-path contrastive learning for nulti--ehaavior recom- mendation. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 787-796. IEEE, 2021. 9
Ramanujan, V., Vasu, P. K. A., Farhadi, A., Tuzel, O., and Pouransari, H. Forward compatible training for large- scale embedding retrieval systems. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19386-19395, 2022. 2,
Yang, M., Zhou, M., Ying. R., Chen, Y., and King, I. Hyper- bolic representation learning: Revisiting and advancing. In International Conference on Machine Learning, pp.
PMLR, 2023. 9
Ridnik, T., Ben-Baruch, E.,, Noy, A., and Zelnik-Manor, L.. Imagenet-21k pretraining for the masses, arXiv preprint arXiv:2104. 10972, 2021. 6
Yang, M., Feng, A., Xiong, B., Liu, J., King, I., and Ying. Hyperbolic fine-tuning for large language models, 2024. URL https: / /awxiviiivvv.////i///aa////aaavv org/ abs abs/2410.. 24010.00100......1
04010.
Ruder, S. An overview of gradient descent optimization algorithms. arXiv preprini urXiv: 04747, 2016. 6
Zhang, B., Ge, Y., Shen, Y., Li, Y., Yuan, C., Xu, X., Wang, Y., and Shan, Y. Hot-refresh model upgrades with regression-allevian retrieval, arXiv preprint arXiv:2201.0972. 0972244 2022a. 3, 5, 6.
compatible training in image
Shen, Y., Xiong, Y., Xia, W., and Soatto, S. Towards backward compatiibee representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 6368-6377 2020. 2, 3, 4, 6,
Zhang, B., Ge, Y., Shen, Y., Su, S., Wu, F., Yuan, C., Xu, X., Wang, Y., and Shan, Towards universal Bekkwadd-oopattinee representation learning. arXiv preprint arXiv:2203.0/583 2022b. 9
Stein, H. On einstein miikkosskim space time. The Journal of Philosophy, 65(1):5-23, 1968. ISSN 0022362X. URL http:/ www. jstorr .eeooooorgg org/ stable /ttp////llle/2222.... 2024512. 2
Zhou, Y., Li, Z., Shrivastava, A., Zhao, H., Torralba, A., Tian, T., and Lim, S.-N. Bt 2: Backward cowward-oompaibleee training with basis transformation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11229-11238 2023. 2.9
Surís. D., Liu, R., and Vondrick, C. Learning the predictabil- ity of the future. In Proceedings of the IEEE/CVF Con-
ference Computer Vision and Pattern Recognition, pp- 12607-12617, 2021. 2
Wang, C.-Y., Chang, Y.-L., Yang, S.-T., Chen, D., and Lai, S.-H. Unified representation learning for cross model compatibility. arXiv preprint arXiv:2008.0482 04821, 2020. 9
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., and Wen, J.-R. Large language models for information retrieval: A survey, arXiv preprint arXiv:2308.0m SSti:2308.001007, 2023.
Wang, H., Wang. Y., Zhou, 7.,, Ji, X., Gong, D., Zhou, J., Li, Z., and Liu, W. Costace: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 22744, 2018. 3
Wang, L., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Wei, R., Liu, Y., Song. J., Xie, Y., and Zhou, K. Explor- ing hierarchical information in hyperbolic space for self- supervised image hashing. IEEE Transactions on Image
12

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatibbe Representation Learning
A. Additional Ablation Studies
In this section, we conduct comprehensive ablation study of HBCT across varying hyperparameters. Unless stated otherwise, all subsequent experiments use the extended-class settings shown in Table 1.
Curvature. In the main paper, fix the curvature value K following existing literature (Bdeir al., 2023). We conducted an ablation study to examine how different curvature values affect compatibility performance. Following (Desai et al., 2023), we also evaluate the compatibility performance when set the curvature value as a learnable parameter, which is adjusted during training with other parameters. The result in Table 3 shows that K E [0.5, 1] yields stable outcomes in both compatibility and the quality of the new model. With the learnable curvature, the new model showed good performance for new-to-new retrieval. However, negatively impacted compatibility performance, particularly cross (new-to-old) retrieval.
Curvature K
self
cross
Pu
P comp
0.1
0.581
0.366
-0.114
0.108
0.5
0.651
0.397
-0.008
0.206
0.7
0.657
0.399
0.002
0.211
1.0
0.654
0.398
-0.003
0.207
1.5
0.604
0.370
-0.079
0.123
learnable
0.660
0.371
0.006
0.124
Table Ablation curvature on HBCT's mAP performance.
Clipping threshold Ç. In the main paper, we set the clipping threshold to Co = 1.0. For the updated model, we raise it to Sn = So + 0.2 to accommodate the evolving embedding space and allow new embeddings greater flexibility to diverge from the old ones. This ablation examines the impact Gn: we hold Co = 1.0 and vary S. E [1.0, 1.3). As shown in Table 4, keeping Sn 1.0 preserves the new model's self-eerformance but harms cross-model compatibility, whereas setting
Sn 1.2 offers the best balance between the two metrics.
So
Sn
self
cross
1.0
1.0
0.7232
0.5525
1.0
1.1
0.7153
0.5719
1.0
1.2
0.7221
0.5705
1.0
1.3
0.7154
0.5588
Table 4. Effect of the clipping threshold Ç.. on the CMC@1 performance of the new model.
Alignment weights. We evaluate HBCT under range of hyperparameter settings. Figure 7 shows that setting À = 0.3 and = 0.5 achieves the best balance between the updated model's self-performance and cross-model compatibility. To assess the relative importance of entailment versus contrastive objectives, we introduce a new weight Aeutail into our loss function. In the experiment shown in Figure 7c, we keep 0.3 fixed and vary Aentail-
= Lasse + X(Centail + Ccontrast),
(16)
Distance function. We compare HBCT under three distance functions geaddesi distance (Eq. (3)), squared Lorentz distance (Law et al., 2019), and the Lorentz inner product (Eq. and we also replace the contrastive loss Lcontrast with mean-distortion loss
Lindist d(h,, hn),
where -) -) is either the geodesic or squared Lorentz distance. As Table 5 shows, all three metrics yield similar HBCT performance. Substituting in Lmdditt neither enhances cross-model compatibility nor preserves the updated model's self- relf-performance, echoing the degradation observed with the &2 alignment loss in Euclidean space.
Classification performance. Throughout the paper, we mainly focus on CMC@k and mAP to focus on the retrieval task. We report the classification accuracy of the alignment methods in comparison with the CMC@) metric in Table 6.
13

---

Learning Along the Arrow of Time: Hyperbolic Geometry for Backwarr-Compattile Representation Learning
Cross Retrieval ImAP)
Trade-off: Retrieval (mAP)
Cross Retrieval (MAP)
0.3
2.0
2.0
04
0.97
3.88
05
Do0s (56
mAR
0:09
0.00
0.
Cross
A85
3,301
6282
33900
0.50
A.7
3.0
3864
0.55
Selt MAP
Selt mAP
Selt MAP
(a) Alignment weight
(b) Contrastive temperature т
(c) Entailment weight Aentail
Figure 7. Comparison self vs, cross retrieval mAP across three hyperparameters.
Distance function
self
cross
RINCE Geodesic
0.6552
0.3982
RINCE Lorentz inner
0.6550
0.3985
RINCE Squared Lorentz
0.6529
0.3979
Mean distortion Geodesic
0.6487
0,3970
Mean distortion Squared
0.6381
0.3923
Table 5. Ablation of different distance functions self and cross mAP performance.
Specifically, we report the cross-classification metric, where the accuracy evaluated for the old classifier head applied the new embedding model.
Self-CMC@1
Cross-CMC@)
Self-Ace
Cross-Ace
BCT
0.695
0.447
76.47
40.66
Hot-refresh
0.715
0.498
77.65
41.20
HOC
0.713
0.490
77.67
41.36
HBCT
0.722
0.572
78.86
42.19
Table 6. Comparison of different methods on CMC@ and accuracy metrics.
14


Reviews:
[{"review_id": "YkipGQWcps", "reviewer": "Reviewer_fRer", "summary": "The paper proposes to leverage hyperbolic geometry for backward compatible learning: setting in which a model is updated and its representation should preserve compatibility with representations from the model before the update. The authors propose a  loss composed of  two terms to (i) constraint the new embedding to lie in the entailment cone of the old embeddings (ii) regularization loss which align weights according to the uncertainty, to preserve performance of the new model.The method is validated using the CIFAR100 and MiniImagenet datasets, analyzing backward compatibility when introducing novel data samples, novel classes or a different architecture. Ablation experiments are performed to quantify the contribution of each term and setting. The method outperforms existing methods which are based on considering euclidean geometry in representation space.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and suggestions. We answer the remaining questions by points as follows.\n\n> Q1: Training could be more expensive with hyperbolic geometry\n\nWe agree hyperbolic models may be more expensive, but our method only adds a simple exponential map to a Euclidean model, incurring minimal overhead. We report the running time of ResNet18 with a batch size of 256 in the table below.\n\n| ResNet18  | Forward only | Forward & Backward |\n|-|-|-|\n| Euclidean | 7.9e-07|  0.0556|\n| Euclidean-Hyperbolic | 8.8e-07 | 0.0630|\n\n> Q2: Applicability could be limited as most methods are not assumed to be trained with hyperbolic embeddings spaces. \n\nOur method is general and can be applied on top of existing Euclidean embedding models.\n\n> Q2: Some questions arise when the geometry of the space is not flat and nor with constant curvature\n\nWe consider Euclidean and hyperbolic geometries because they are most commonly used in existing retrieval systems, mostly because the geodesic distance between two points can be computed efficiently. In some cases, manifolds with non-constant curvature can better represent data, but the geodesic distance will be more computationally expensive, making it difficult to generalize to large-scale retrieval systems.\n\n> Q3: The comparison with these approaches that use hyperspherical geometry (with cosine similarity)?\n\nHot-Refresh and HOC are two alignment methods that employ hyperspherical geometry, as they align models using a contrastive objective based on cosine similarity. Since cosine similarity normalizes embeddings to have unit norm ($| z |_2 = 1$), these methods map embeddings onto a shared unit hypersphere. Pairwise embedding comparisons are then computed using the negative inner product as a distance measure. We will clarify this in the baseline discussion in the revision.\n\n> Q4: How does the similarity measure used in CMC affect results?\n\nWe observe that retrieval performance remains consistent across different similarity measures. Below are the results on CIFAR100:\n- Euclidean-ResNet18:  \n  - Cosine: mAP 0.6543, CMC@1 0.7157  \n  - Euclidean: mAP 0.6543, CMC@1 0.7157\n- Hyperbolic-ResNet18:  \n  - Geodesic: mAP 0.6548, CMC@1 0.7177\n  - Lorentz inner product: mAP 0.6548, CMC@1 0.7177\n- Using hyperbolic distance on Euclidean embeddings (via exponential map):  \n  - Hot-refresh original: \n     - new-to-new retrieval (self): mAP: 0.649485, CMC@1: 0.7147\n     - new-to-old retrieval (cross): mAP: 0.347584, CMC@1: 0.4977 \n  - Hot-refresh -> exponential map \n     - new-to-new retrieval (self): mAP: 0.649482, CMC@1: 0.7147 \n     - new-to-old retrieval (cross): mAP: 0.347583, CMC@1: 0.4978\n\n> Q5: In Table 1 the columns corresponding to self and cross denote the base loss? what is this number referring to, the original model performance?\n\nYes, *self* (new-to-new retrieval) and *cross* (new-to-old retrieval) refer to the original model performance (CMC or mAP). We will explain this explicitly in the paper to avoid future confusion.\n\n> Q6. Experiments larger datasets\n\nWe mainly followed [1, 2] to test on CIFAR100 and Tiny-ImageNet, where hyperbolic geometry is shown to be effective. Given the short time frame for rebuttal responses, extensive experiments on large datasets are beyond our immediate scope but we acknowledge the valuable suggestion and plan to explore larger-scale evaluations in future research to further validate the approach.\n\n> Q7: How does curvature K affect the performance? why the choice of fixing it to 1 for all experiments?\n\nWe set a fixed curvature value following the existing literature [1]. We conducted an ablation study to examine how different curvature values affect compatibility performance. Generally,  $K \\in [0.5, 1.0]$ yields stable outcomes in both compatibility and the quality of the new model. When we also tested the learnable curvature, it showed good performance in the new model but negatively impacted compatibility performance. \n\n|Curvature K|self (mAP)|cross (mAP)|P_up (mAP)|P_comp (mAP)|\n|-|-|-|-|-|\n|0.1 |0.581|0.366|-0.114|0.108 |\n|0.5 |0.651|0.397|-0.008 |0.206 |\n|0.7|0.657 |0.399 |0.002|0.211 |\n|1 |0.654|0.398 |-0.003|0.207|\n|1.5|0.604 |0.370|-0.079 |0.123|\n|Learnable  |0.660 |0.371|0.006|0.124|\n\n> Q8: How impactful is pretraining from scratch assuming hyperbolic geometry? How does this compare to finetuning from a pretrained model assuming hyperbolic geometry as opposed to training from scratch?\n\nIn our experiment, ResNet18 is trained from scratch, while ViT is finetuned from ImageNet21K pretrained weights to reflect a typical usage of open-source models. We do not pretrain ViT from scratch due to the poor convergence on CIFAR100 and TinyImagenet.\n\nReferences:\n[1] Bdeir et al. Fully Hyperbolic Convolutional Neural Networks for Computer Vision, ICLR’24.\n[2] Biondi et al. Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements, CVPR'24.", "claims_and_evidence": "To the best of my judgment,  claims in the paper are validate in the paper. In particular: \n\n(i) They show how hyperbolic geometry is good modeling choice for the problem of backward compatibility, showing previous contains as the entailment loss and the uncertainty estimation naturally fits into this problem. The is further confirmed by the experiments in Table 2. \n\n(ii) The experiments demonstrate how hyperbolic geometry is a better fit than exiting methods based on euclidean geometry.", "methods_and_evaluation_criteria": "The paper compares to many baselines, according to the best of my knowledge.  Concerning the dataset, in the paper (Shen et al 2022) introducing backward compatibility larger models and datasets (e.g. Imdb ) were adopted, so it would be interesting to see if the result scales up to larger datasets.", "theoretical_claims": "No theoretical claims are present.", "experimental_designs_or_analyses": "## Strenghts\n\n - Experiments are sounds and shows that under different setting (sample change, class change, and architectural change) that hyperbolic spaces are effective for the problem of backward compatibility. \n\n- Ablations show the impact of each term in the loss, providing a good understanding of the method.\n\n\n## Weaknesses\n\n- Despite the good performance demonstrated on the proposed benchmark, previous works introducing the problem of backward compatibility (Shen et al 2020) experimented with larger models (e.g. Resnet 100) and in particular larger datasets (e.g. IMDB). This raise some questions on how the performance of the proposed method would scale to this size. \n\n\n- Some details in the table of result could be explained more in depth (e.g. cross and self columns, see questions section)", "relation_to_broader_scientific_literature": "Key contribution of there paper are to demonstrate the importance of characterizing the geometry of representational space with a different metric than the euclidean, i.e. flat. Previous work has highlighted the importance of characterizing different metrics in latent spaces in distinct setting, e.g. generative [2], text [3,4], images (Khrulkov et al., 2020). To the best of my knowledge this is  the first work that tries to advocate for a different geometry to solve the recent problem of backward compatibility, demonstrating an elegant and effective solution.\n\n\nThe work has also important consequences and relation to the field of representation alignment [e.g. 1,5] which could be interesting to relate to. \n\n_[1] Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication.\" ICLR 2023_\n\n_[2] Arvanitidis, Georgios, Lars Kai Hansen, and Søren Hauberg. \"Latent space oddity: on the curvature of deep generative models._\n\n_[3] Gu, Albert, et al. \"Learning mixed-curvature representations in product spaces.\" International conference on learning representations. 2018._\n\n_[4] Dhingra, Bhuwan, et al. \"Embedding text in hyperbolic spaces., ACL_\n\n_[5] Huh, Minyoung, et al. \"Position: The platonic representation hypothesis.\" Forty-first International Conference on Machine Learning. 2024._", "essential_references_not_discussed": "To the best of my knowledge there are no very fundamental related work that has not been cited. It could be  beneficial to include the references in the previous section, although this is not strictly necessary.", "other_strengths_and_weaknesses": "### Strenghts\n\n- *Clarity* the paper is very clear and well written.\n\n- *Originality* the work result in the effective combination existing frameworks for robust contrastive loss (Chuang et al., 2022) , uncertainty properties of hyperbolic geometrical representation spaces (Franco et al., 2023; Atigh et al., 2022) applied to backward compatibility (Shen et al 2022), resulting in an effective and good fitting framework for the problem. \n\n- *Significance*: the problem of backward compatibility I still very open and recent, and of importance for practical application (.e.g industrial deployment of models). The paper demonstrate the importance of correctly characterizing the geometry of representational space, with important practical consequences. Moreover the type of research has possible implication on  diverse fields such as representational alignment, model adaptation, test time training, and out of distribution detection. \n\n\n## Weaknesses \n\n- Limitations in assuming hyperbolic geometry: the hyperbolic geometry assumption could be also limiting in my understanding in two ways: training could be more expensive and applicability of the method could be limited as most methods are not assumed to be trained with hyperbolic embeddings spaces. Also some questions arise when the geometry of the space is not flat and nor with constant curvature.  On the regard see last two questions in the question section.\n\n- A discussion and comparison with methods that use product of spheres seems to miss.", "other_comments_or_suggestions": "I spotted the following typos: \n\n\nLine 270: citation missing: \"unlike previous methods (),\"\n\nLine 312: \"the the\" -> \"the\"", "questions_for_authors": "- How does the similarity measure is used to compare queries to gallery samples in the Cumulative Matching Characteristics metric for retrieval affect results? What happens if one use a different measure (eg. euclidean distance in euclidean based method, or hyperbolic geodesic distance in euclidean based methods, assuming despite pertaining an hyperbolic geometry)\n\n- In Table 1 the columns corresponding to self and cross denote the base loss? what is this number referring to, the original model performance? \n\n- The authors mention approaches that use hypersphere geometry (with cosine similarity). What's the comparison with these methods? \n\n- How does curvature K affect the performance? why the choice of fixing it to 1 for all experiments? \n\n-  How much impactful is pretraining from scratch assuming hyperbolic geometry? How does this compare to finetune from a pretrained model assuming hyperbolic geometry as opposed to train from scratch? Although this is not the direct focus of the paper, the applicability of the method for backward compatible training to pretrained models is an important point in order to apply the method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TucGQqpii5", "reviewer": "Reviewer_eHD4", "summary": "This paper aims to improve backward compatible representation learning by using hyperbolic embeddings instead of Euclidean embeddings. This paper claims that using hyperbolic embeddings achieves greater compatibility with previous models without compromising the performance of new embedding models.\n\nMethods-wise, this paper uses a hybrid Euclidean-Hyperbolic model to encode images in hyperbolic space. This allows better integration of existing Euclidean encoders compared to fully hyperbolic models. The paper also defines an uncertainty measure for Lorentz spaces, based on the analogous uncertainty measure for Poincare spaces. The model uses two auxiliary losses, an entailment cone loss, and RINCE-based loss, which is modified to include the uncertainty measure defined in the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and suggestions. We answer the remaining questions by points as follows.\n\n> Q1: why the Lorentz uncertainty was defined the way it was.\n\nWe define the Lorentz uncertainty using the isometric relation between the Poincaré and Lorentz models. Both uncertainty measures in Poincaré and Lorentz models have the same form with respect to the norm 2 of the Euclidean embedding before the exponential map:\n\n$$\n\t\\mathrm{Uncertainty}(\\mathbf{h}^{\\mathcal{L}}) = \\mathrm{Uncertainty}(\\mathbf{h}^{\\mathbb P}) = 1 - \\frac{1}{\\sqrt{K}} \\mathrm{tanh}(\\sqrt{K} \\|\\| \\mathbf z \\|\\|)\n$$\nBoth the distance to the origin and our uncertainty measure depend explicitly on the Euclidean $\\ell_2$-norm before the exponential map. While prior work uses the unbounded origin distance as a proxy for uncertainty, our bounded [0,1] measure is more interpretable and manageable. We will discuss this explicitly in the revision.\n\n> Q2+3: What is the CMC@1 and mAP performance of the models?… What do the self and cross columns mean in Table 1?\n\nThe *self* (new-to-new retrieval) and *cross* (new-to-old retrieval) columns in the Table refer to the original model performance (CMC or mAP). We thank the reviewer for pointing this out, and we will explain this explicitly in the paper to avoid future confusion.\n\n> Q4: Is there a reason why using the RINCE-based loss sometimes results in a large drop in compatibility?\n\nDuring the experiments, we observed that training with RINCE-based loss sometimes is more difficult to converge in some cases, especially on the Vision Transformer model. This maybe because of the instability of the exponential map when applied to the large norm produced by ViT. However, when training with the entailment cone loss, the training becomes more stable.\n\n> Q5: Given how the norms grow with repeated model updates and the possible resulting instability, how does the performance of this method compare to Euclidean methods after many model updates?\n\nWe conducted an experiment with five consecutive model updates to test the performance of different compatibility methods on the CIFAR100. The training data will be split into five sets, each containing 20, 40, 60, 80, and 100 (all) classes, respectively. For the first two models (time steps 0, 1), we will use ResNet18 as the base model and we will update to ResNet50 at the last three steps (2, 3, 4). For each method, we report the CMC@1 compatibility matrix for HBCT, HOC and BCT. Columns are the encoders for the query, and rows are the encoders for the gallery set. It can be seen that the HBCT can maintain the compatibility with the old model after several updates ($\\phi_1 / \\phi_4$) = 0.32) , while HOC and BCT quickly deteriorate after a few model updates ($\\phi_1 / \\phi_4$ = 0.207 for HOC and 0.14 for BCT).\n\nHBCT\n\n|  $\\phi_o / \\phi_n$    | $\\phi_0$ | $\\phi_1$ | $\\phi_2$ | $\\phi_3$ | $\\phi_4$ |\n|-|-|-|-|-|-|\n| $\\phi_0$ | 0.2016   | | | | |\n| $\\phi_1$ | 0.2295   | 0.3585   ||||\n| $\\phi_2$ | 0.2585   | 0.4309   | 0.4872   |||\n| $\\phi_3$ | 0.2867   | 0.4857   | 0.5525   | 0.6166   | |\n| $\\phi_4$ | 0.3261   | 0.5227   | 0.6051   | 0.6995   | 0.7424   |\n\nHOC\n|    $\\phi_o / \\phi_n$        | $\\phi_0$ | $\\phi_1$ | $\\phi_2$ | $\\phi_3$ | $\\phi_4$ |\n|-|-|-|-|-|-|\n| $\\phi_0$ | 0.1825   | | | | |\n| $\\phi_1$ | 0.17     | 0.3265   | | | |\n| $\\phi_2$ | 0.1954   | 0.3807   | 0.4831   | | |\n| $\\phi_3$ | 0.207    | 0.418    | 0.5357   | 0.6045   | |\n| $\\phi_4$ | 0.1926   | 0.4332   | 0.5831   | 0.6888   | 0.7302   |\n\nBCT\n\n|  $\\phi_o / \\phi_n$      | $\\phi_0$ | $\\phi_1$ | $\\phi_2$ | $\\phi_3$ | $\\phi_4$ |\n|-|-|-|-|-|-|\n| $\\phi_0$ | 0.1818   |          |          |          |          |\n| $\\phi_1$ | 0.1589   | 0.3124   |          |          |          |\n| $\\phi_2$ | 0.1524   | 0.3218   | 0.4456   |          |          |\n| $\\phi_3$ | 0.135    | 0.3299   | 0.4834   | 0.5647   |          |\n| $\\phi_4$ | 0.1497   | 0.3212   | 0.5249   | 0.5939   | 0.7206   |\n\n> Q6: It seems that a possible limitation of the model is that performance may degrade after many model updates.\n\nYes, we have discussed this in the limitations section. Increasing the clipping threshold after each update can lead to numerical instability when it becomes large (e.g., >10). One potential solution is rescaling the time dimension of all embeddings after a certain number of updates (e.g., >20). Another approach is online backfilling, where a subset of items is used to update the gallery set—this can help manage the growing time dimension.\n\nWe hope that we have addressed all your concerns adequately. We have updated the manuscript according to your comments. Please let us know if we can provide any further details and/or clarifications.", "claims_and_evidence": "The high-level intuition of the method is interesting and makes a lot of sense. \n\nThe authors claim that HBCT enhances backward compatibility without compromising the performance of new embedding models. This is supported by the empirical evidence, which suggests that HBCT enhances backward compatibility (best $P_\\mathrm{com}$ scores in Table 1) with little tradeoff in performance of the new models ($P_\\mathrm{up}$ scores near 0 in Table 1).\n\nThe use of the entailment loss and the RINCE-based loss both seem effective according to the ablations (Table 2), but the performance of RINCE is a bit more mixed, sometimes resulting in a large drop in compatibility.", "methods_and_evaluation_criteria": "Overall, the methods and evaluation criteria seem reasonable.\n\nIt is not clear to me why the Lorentz uncertainty was defined the way it was. Perhaps the connection to Poincare uncertainty could be made more explicit in the text. \n\nAlthough not the main focus of the paper, I think it is still important to also report the performance of all models on CMC@1 and mAP directly so readers can assess the tradeoff between compatibility and performance.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Overall the experimental design seems reasonable. However, this paper could benefit from analyses on more datasets than just CIFAR-100 and TinyImageNet.", "relation_to_broader_scientific_literature": "The idea of using hyperbolic embeddings for backwards compatibility is novel and is also a very natural idea.\n\nThe method of this paper is a combination of existing methods. The hybrid Euclidean-Hyperbolic model is based on [1], and the entailment cone loss was first proposed in [2]. The RINCE-based loss is essentially the RINCE loss of [3] with the modification that the $q$ parameter is the Lorentz uncertainty measure. In light of this, the novelty of the paper's methods is quite limited. \n\n[1] Khrulkov, Valentin, et al. \"Hyperbolic image embeddings.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Desai, Karan, et al. \"Hyperbolic image-text representations.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] Chuang, Ching-Yao, et al. \"Robust contrastive learning against noisy views.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses\n\nIn Table 1, the self and cross columns are not explained or referred to in the main text.\n\nIt seems that a possible limitation of the model is that performance may degrade after many model updates. It would be good if the authors could discuss this.", "other_comments_or_suggestions": "Equation 12, the apostrophe in $h_o’$ not defined \nMissing citation (Line 269, second column)", "questions_for_authors": "1. How was the Lorentz uncertainty derived?\n2. What is the CMC@1 and mAP performance of the models?\n3. What do the self and cross columns mean in Table 1?\n4. Is there a reason why using the RINCE-based loss sometimes results in a large drop in compatilibity?\n5. Given how the norms grow with repeated model updates and the possible resulting instability, how does the performance of this method compare to Euclidean methods after many model updates?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePKK2gA94Z", "reviewer": "Reviewer_JxAG", "summary": "The authors propose Hyperbolic Backward-Compatible Training (HBCT), which is essentially an objective for backwards-compatible representation learning in hyperbolic space. HBCT balances the objective of the embedding loss (e.g. cross-entropy on image classification) with a hyperbolic entailment loss that encourages a partial order between old-model and new-model embeddings, dynamically weighted (via RINCE loss) by the old model’s uncertainty for that input. Across a suite of experiments covering a variety of plausible compatibility scenarios (extending data, extending classes, new architecture, and combinations thereof), HCBT performs well.\n\n## Update after rebuttal\nI was satisfied with the authors' responses and am maintaining my weak-accept recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and suggestions. We answer the remaining questions by points as follows.\n\n> Q1. Ablation on hyperparameter sweeping\n\nDue to restricted space, we report the ablation for the clipping threshold. We refer to our response to Reviewer fRer for the ablation with the curvature value. Remaining hyperparameter ablations (alignment weight, entailment cone weight, temperature) are included in the revision.\n\n| $\\beta_n = \\beta_o + \\zeta$ |self-CMC@1|cross-CMC@1|\n|-|-|-|\n| $\\zeta = 0.0$| 0.7232 | 0.5525  |\n| $\\zeta = 0.1$| 0.7153 | 0.5719 |\n| $\\zeta = 0.2$| 0.7221 | 0.5705 |\n| $\\zeta = 0.3$| 0.7154 | 0.5588 |\n\n> Q2. The use of the Lorentz model… it may be the case that this is purely a matter of convenience/numerical stability—if so, I would like the authors to clarify this.\n\nYes, we use the Lorentz model for its numerical stability, following recent work [1, 2]. Its space-time interpretation also fits our context, allowing us to use the time-like dimension to model continual updates and embedding uncertainty.\n\n> Q3. The derivation of Equation 10 is unclear: what is the relationship between this quantity and the distance from the origin of the hyperboloid?\n\nWe define Lorentz uncertainty via its isometric relation to the Poincaré model. Both share the same form, based on the Euclidean norm of the pre-exponential embedding.\n$$\n\t\\mathrm{Uncertainty}(\\mathbf{h}^{\\mathcal{L}}) = \\mathrm{Uncertainty}(\\mathbf{h}^{\\mathbb P}) = 1 - \\frac{1}{\\sqrt{K}} \\mathrm{tanh}(\\sqrt{K} \\|\\| \\mathbf z \\|\\|)\n$$\n\nThe distance from the origin of the hyperboloid is given by\n\n$$\n\td(\\bar{\\mathbf{0}}, \\mathbf{h}) = \\mathrm{cosh}^{-1} (\\frac{1}{\\sqrt{K}} cosh(\\sqrt{K} \\| \\| \\mathbf z \\| \\|)\n$$\nBoth the distance to the origin and our uncertainty measure depend explicitly on the Euclidean $\\ell_2$-norm before the exponential map. While prior work uses the unbounded origin distance as a proxy for uncertainty, our bounded [0,1] measure is more interpretable and manageable.\n\n> Q4. What other metrics can be evaluated? classification performance?\n\nWe report CMC@1 and mAP to focus on retrieval performance. The table shows accuracy and retrieval results for various alignment methods, including cross-accuracy using the old classifier on new features. Accuracy trends generally follow retrieval, though with smaller gaps.\n\n|| Self-CMC@1 | Cross-CMC@1 | Self-Acc | Cross-Acc|\n|-|-|--|-|-|\n|BCT | 0.695 | 0.447  | 76.47  | 40.66|\n|Hot-refresh | 0.715 | 0.498 | 77.65  | 41.2|\n|HOC | 0.713 | 0.490 | 77.67 | 41.36|\n|HBCT | 0.722 | 0.572 | 78.86  | 42.19|\n\n> Q5. How does exponential volume growth relate to continual model updates, and why might it be preferable to polynomially growing spaces like Euclidean space?\n\nWe argue that exponential volume growth in hyperbolic space allows more room to represent new entities with smaller increases in radius, which is advantageous for continual updates. In contrast, Euclidean space requires rapidly growing radii to accommodate new entities.\n\nMoreover, Lemma 1 in [3] shows that in Euclidean space, the probability of new class prototypes aligning with a trained model decreases exponentially with the number of dimensions and classes, leading to an impossibility result for backward compatibility. We hypothesize that hyperbolic geometry with its exponential growth property may mitigate this issue, though a formal analysis is left for future work.\n> Q6. Can this approach be used to train a new model to align embeddings between several different models? Assume we don’t know anything about how they were trained.\n\nAligning a new model to multiple old models is challenging without knowing their training processes. Some models may have incompatible embeddings, enforcing inherent trade-offs in the compatibility. A common solution is to add a projection layer to map embeddings into a shared space [4], which is orthogonal and can be integrated to our method.\n\n> Q7. What happens if you ablate the hyperbolic MLR (i.e. compute loss on Euclidean model, then exp map, then compute the other losses)—does this do better or worse?\n\nWe replaced the hyperbolic MLR with its Euclidean counterpart while keeping the entailment cone and RINCE loss in hyperbolic space. This led to a significant drop in both the new model's performance and its compatibility. This is likely due to conflicting signals between the Euclidean MLR and hyperbolic RINCE loss.\n\n|  | Self-mAP | Cross-mAP |\n|-|-|-|\n|HBCT w/o entail  | 0.435    | 0.388|\n|HBCT  | 0.655 | 0.398|\n|HBCT-EMLR w/o entail | 0.4791   | 0.047|\n|HBCT-EMLR   | 0.4897  | 0.279|\n\n> Q8. Writing structure and  typos\n\nWe reorganized the \"norm control\" and \"overall training\" into a separate section and fixed the typos.\n\nReferences:\n\n[1] Bdeir et al. Fully Hyperbolic Convolutional…, ICLR’24.\n[2] Desai et al. Hyperbolic Image-Text…, ICML’23.\n[3] Biondi et al. Stationary Representations: Optimally…, CVPR’24.\n[4] Linia et al. Asymmetric Image Retrieval with Cross Model...", "claims_and_evidence": "The authors’ claims about their method’s performance on retrieval tasks are solidly backed by experimental results and theoretical justification. Some of their implementation details (e.g. many hyperparameter choices, the specific formula for uncertainty, etc.) are limited. I call these out more specifically in my questions for the authors.", "methods_and_evaluation_criteria": "The authors’ approach to testing their method makes sense to me, though I would like to see classification accuracy included as a metric in their results (see Weaknesses section).", "theoretical_claims": "The authors provide no proofs with their paper. For most of their theoretical claims, the authors defer to the existing literature, which is fine by me. However, certain claims—the form of equation 10, the use of the hyperboloid model, the choice of $\\beta_n = \\beta_o + 0.2$ for clipping—are neither justified with proofs/mathematical intuitions nor tested via ablations.", "experimental_designs_or_analyses": "On the basis of the description in the paper, all aspects of the experimental design appear sound to me.", "relation_to_broader_scientific_literature": "This paper unites the fields of hyperbolic deep learning (particularly the computer vision applications thereof) with backwards-compatible learning. To the best of my knowledge, this is the first paper at the intersection of these two fields.", "essential_references_not_discussed": "To the best of my knowledge, the authors’ references are adequate.", "other_strengths_and_weaknesses": "**Strengths:**\n* **Novelty:** this is the first work (to my knowledge) to combine the fields of backward-compatible representation learning and hyperbolic geometry; the motivation for combining these is compelling. The use of entailment loss as a relaxation of distance-based matching for embeddings is especially insightful.\n* The authors demonstrate strong results on a wide range of retrieval benchmarks. This is an admittedly idealized setting, but it gets at the core differences between their approach and other approaches in the literature. \n* The method is highly general: in particular, the models are not expected to be hyperbolic to start with.\n\n**Weaknesses:**\n* **Many choices are unjustified:**\n  * Most hyperparameter choices are only justified in terms of existing literature. The authors claim to have tried a hyperparameter sweep converging on $\\lambda=0.3, \\tau=0.5, \\beta=0.01$, but do not include the results in the Appendix.\n  * The use of the Lorentz model is somewhat surprising, given that the entailment cone literature generally relies on the Poincare model and the authors make no use of the timelike dimension. I understand that these models are interchangeable, so it may be the case that this is purely a matter of convenience/numerical stability—if so, I would like the authors to clarify this.\n  * The derivation of Equation 10 is unclear: what is the relationship between this quantity and the distance from the origin of the hyperboloid?\n* **Poor structure** throughout the work. For instance:\n  * Sections 2.1 and 2.2 could be combined (there is no need to e.g. define the tangent plane twice); \n  * It is unclear why \"norm control for numerical stability\" and \"overall training pipeline\" are under subsection 4.3; \n* **Limited results:** The authors only test retrieval performance, evaluated via CMC@k and mAP. While I agree these are the most important things to test, what other metrics can be evaluated? In particular, since the authors discuss classification as a training objective, what happens to classification performance?", "other_comments_or_suggestions": "* L115: The Lorentzian norm is defined but not used; what the authors are really interested in is the norm of the spacelike dimensions\n* L288: The reference to “our proprietary training dataset” should be removed from the paper. The paper should only be evaluated on the merits of the actual experiments presented therein. \n* Figure 5: the UMAP embeddings would be more informative if we could see which embeddings are paired with which, e.g. by drawing lines or arrows between old and new embeddings. As this will likely create a lot more visual noise, it may necessitate reducing the total number of points visualized.\n* I caught a couple of typos:\n  * R253: “making it adaptively” should be “adaptive”\n  * R270: empty parentheses\n  * R371: “can be achieved only minimal” should be “can be achieved with only minimal”\n  * L431: “aligns with out intuition” should be “aligns with our intuition”", "questions_for_authors": "* In your paper, you say \"their exponential growth of areas and volumes with respect to time makes them particularly suited for applications involving continual model updates with new entities or classes arrive over time.\" Intuitively, what is the connection between exponential growth in neighborhoods and applications with continual model updates? I could just as easily believe, for instance, that continuous model updates work fine in polynomially-growing vector spaces.\n* Can this approach be used to train a new model to align embeddings between several different models? Assume we don’t know anything about how they were trained\n* What happens if you ablate the hyperbolic MLR (i.e. compute loss on Euclidean model, then exp map, then compute the other losses)—does this do better or worse?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates backward-compatible representation learning, an important aspect of real-world deployment of neural networks. The paper advocates for hyperbolic geometry to address this problem, to deal with uncertainty and model quality. After the rebuttal and discussion phases, all three reviewers vote towards accept. The reviewers find the motivation interesting and the paper novel. They also point out fair concerns regarding the experimental scope of the paper. The rebuttal does a good job in providing additional metrics and ablations, but the AC agrees with the reviewers that the model and dataset sizes remain small-scale. Regardless, the AC find the paper novel and clear. The AC agrees with the unanimous consensus for accept.
Decision: Accept (poster)