Hot-Refresh Model Upgrades with Regression-Free Compatible Training in Image Retrieval
Authors: Binjie Zhang
Yixiao Ge
Yantao Shen
Yu Li
Chun Yuan
XUYUAN XU
Yexin Wang
Ying Shan
Venue: ICLR 2022 Poster (2022)
Keywords: Compatible Representation Learning
Image Retrieval
Model Regression

Abstract: The task of hot-refresh model upgrades of image retrieval systems plays an essential role in the industry but has never been investigated in academia before. Conventional cold-refresh model upgrades can only deploy new models after the gallery is overall backfilled, taking weeks or even months for massive data. In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backfilling the gallery on-the-fly. Compatible training has made it possible, however, the problem of model regression with negative flips poses a great challenge to the stable improvement of user experience. We argue that it is mainly due to the fact that new-to-old positive query-gallery pairs may show less similarity than new-to-new negative pairs. To solve the problem, we introduce a Regression-Alleviating Compatible Training (RACT) method to properly constrain the feature compatibility while reducing negative flips. The core is to encourage the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs. An efficient uncertainty-based backfilling strategy is further introduced to fasten accuracy improvements. Extensive experiments on large-scale retrieval benchmarks (e.g., Google Landmark) demonstrate that our RACT effectively alleviates the model regression for one more step towards seamless model upgrades.

Published as a conference paper at ICLR 2022
HOT-REFRESH MODEL UPGRADES WITH REGRESSION-ALLEVIATIN COMPATIBLE TRAINING IN IMAGE RETRIEVAL
Binjie Zhang 124 Yixiao Ge
Yantao Shen
Yu Li
Chun Yuan't
Xuyuan Xu²
Yexin Wang'
Ying Shan
'Tsinghua University ²ARC Lab, Tencent PCG
>AI Technology Center of Tencent Video
4 "AwS/Amazon AI
'International Digital Economy Academy
{zbj19 90mails, yuancasz) waanguuueedu.... on singaagggyinngchhanggttenttt.tt
ABSTRACT
The task of hot-refresh model upgrades of image retrieval systems plays an es- sential role in the industry but has never been investigated in academia before. Conventional cold-refresh model upgrades can only deploy new models after the gallery is overall backfilled, taking weeks In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backfilling the gallery on-the-fly Compatible training has made it possible, however, the problem of model regres- sionf] with negative flips poses great challenge to the stable improvement of user experience. We argue that it is mainly due to the fact that new-to-old positive query-gallery pairs may show less similarity than new-to-new negative pairs. To solve the problem, we introduce a Regression-Alleviate (RACT) method to properly constrain the feature compatibility while reducing negative flips. The core is to encourage the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs. An efficient uncertainty- based backfilling strategy is further introduced to fasten accuracy improvements. Extensive experiments on large-scale retrieval bench- marks (e.g., Google Landmark) demonstrate that our RACT effectively alleviates the model regression for one more step towards seamless model upgrades$
months for massive data.
Compatible Training
INTRODUCTION
With the rapid development of deep learning, image retrieval algorithms have shown great success in large-scale industrial applications, e.g., face recognition (Sun et al. 20114a b), object re-identification
(Li et al. 2014 Zheng et al.] 2015). image localization (Arandjelovic et al. 2016). Model upgrades are essential to improve user experience in those applications. By properly leveraging massive training data and improved network architectures, the retrieval performances could be well boosted.
Generally, once the model updates, the process of "backfilling" or "re-indexing" is called for offline re-encoding the gallery image features with the new model. The retrieval system can only benefit from the new model after the gallery is overall backfilled, which may cost weeks or even months for billions of industry images in practice. The above process is termed cold-refresh model upgrades.
To harvest the reward of the new model immediately, [hen et al., (2020) introduces the compati- ble representation learning to train the new model with backward compatibility constraints, so that queries encoded by the new model can be directly indexed by the old gallery features. Meanwhile, as the new features and old features are interchangeable with each other, the gallery images can be backfilled on -n-the-fly, and the retrieval performances would be gradually improved to approach the optimal accuracy of the new model, dubbed as hot-refresh model upgrades (see Figure (a)).
'Corresponding authors. Wook done when Binjie, Yantao and Yu are ARC Lab, Tencent PCG. 'Throughout this paper, the term model regression indicates the degradation retrieval performance when the hot-refresh model upgrades proceed.
Thhe code is available wihhtps: ps: /gi thub. http://hhh.cccoooojeeaaaaaaaaaaa

---

Published as a conference paper at ICLR 2022
Features -
Extract feature
Refresh gallery
model New-to-old compatible mAP
New-to-new optimal mAP
Search
Query
Retrieved gallery images
Switch after backfilling finished
mean Avenge Predsion (mAP)
Cold-refresh upgrade
20% Offiine backfiling
Query
Negative Flip
Quick deployment
O
10%
40% Online backfiling
upgrade (ideal)
Hot-refresh
Gallery
Upgrade progress
(a) Pipeline of cold/hot-refresh model upgrades.
(b) Model regression during hot-refresh model upgrades.
Figure 1: (a) The illustration of cold-refresh and hot-refresh model upgrades. (b) The problem of model regression with negative flips in hot-refresh model upgrading procedure of image retrieval systems, where queries retrieved correctly by the old model fail during the upgrade process.
Although existing compatible training methods (Meng et al. 2021 Shen et al., 2020) make it pos- sible to upgrade the model in hot-refresh manner, they still face the challenge model regression Yan et al., 2020) Xie et al.) 2021). which is actually caused by negative flips, i.e., queries cor- rectly indexed by the old model are incorrectly recognized by the new model (see [ We claimed that, during the procedure of hot-refresh model upgrades, the negative flips occur when the new-to-new similarities between negative query-gallery pairs are larger than the new-to-old similarities between compatible positive query-gallery pairs. To solve the problem, a Aegression-Alleviaane
introduce
Compatible Training (RACT) method to effectively reduce the negative flips for compatible representation learning and further improve the hot-refresh model upgrades with faster and smoother accuracy convergence in image retrieval systems.
Specifically, the conventional backward-compatibn regularization is generally designed pull the new-to-old positive pairs closer and push the new-to-old negative pairs apart from each other. To reduce negative flips during the online gallery backfilling, we improve the bockward-compaiiiee regularization by enforcing the new-to-ol, oodd positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs. To further improve the user experience during the model upgrades, we introduce uncertainty-based hot-refresh backfilling strategy, which re- indexes the gallery images following tthe-poor-firss first"" policy. To measure gallery images in a lightweight manner, feed the old gallery features into the new classifier (one fully-connected layer in general), yielding the probability distribution on new classes. Even though ground-truth classes may not exist for the old gallery features, tive old features are expected to have sharper probability vectors given their compatible latent space. Uncertainty estimation (e.g., Entropy- Entropy-based) is utilized here to evaluate the sharpness of the proba- bility vectors, and gallery images with larger uncertainty should have higher priority for backfilling. Experimental results on large-scale image retrieval benchmarks (i.e., Google Landmark v2 (Weyand) et al. 2020), RParis, and ROxford (Radenović et al., 2018) demonstrate the effectiveness of our RACT method in alleviating the problem model regression for hot-refresh model upgrades.
refresh priority of
assume that more discrimina-
Our contributions can be summarized three-fold. (I) We study the model regression problem in hot-refresh model upgrades of image retrieval systems for the first time. (2) cegressioon-alleiiatie
bookward-compatinn regularization is proposed to alleviate the issue of negative flips during com- patible training. (3) An uncertainty-base backfilling strategy is introduced to further improve the user experience with "the-poor-first hot refresh.
2 RELATED WORKS
Compatible Representation Learning. The task of compatible representation learning aims at encoding new features that are interoperable with the old features. It breaks the inherent cold-refresh way of model iteration in the industry the benefit of new models can be immediately reaped in a backfill-free manner, Shen et al. (2020) first formulated the problem of backward-compattee ing and proposed to utilize the old classifier for supervising the compatible feature learning. [Budnik] & Avrithis (2020) made an effort reduce the gap between the large model and small with-
learn-
2

---

Published as a conference paper at ICLR 2022
out the parametric classifier. Wang al. (2020) introduced projection head to transform the old features to be compatible with the new ones. Li et al. (2020b) proposed a framework, namely "Fea- ture Lenses", to encourage image representations transformation-ivvan between performance and efficiency, Duggal et al. (2021) designed comparibiliy--ware neural architecture search scheme to improve the compatibility of models with different sizes. However, since existing compatible algorithms for image retrieval have not investigated the application of hot-refresh model upgrades, the problem of model regression has been overlooked.
To balance the trade-off
Model Regression. Model upgrades play a vital role in improving the performance of industrial applications, where the issue of model regression inevitably harms the user experience. Yan et al. (2020) studied the problem of model regression in image classification, where the negative flips indicate the images correctly classified by the old model but misclassified by the new model. A method, namely focal distillation, was introduced to encourage model congruence on positive sam- ples. Srivastava al. (2020) also investigated the model regression in classification and proposed new metric to evaluate the regression effects. Xie et al. (2021) probed the model regression prob- lem natural language processing. The works mentioned above all focused on the classification tasks. Directly employing their methods on the retrieval tasks leads to sub-optimal performances. Moreover, they did not need to consider hot-refresh gallery backfilling in image classification.
Active Learning. Our uncertainty-basee backfilling strategy is related to active learning. As the real world is teeming with unlabeled data, active learning makes efforts to achieve better perfor- mance with fewer annotated samples by choosing valuable samples actively (Settles) 2009). Active learning has been widely adopted in deep learning tasks, e.g., object detection (Holub et al., 2008), text classification (Zhu et al.] 2008), and image classification (Cao et al. (020).. Though obtaining the priority of samples a common issue between active learning and gallery backfilling, the latter focuses more on enjoying benefits in model upgrades with minimal computational overhead.
Image Retrieval. The task of image retrieval requires searching images of the same content or object from a large-scale database based on their feature similarities, given images of interest. It is fundamental task for many visual applications, e.g., face recognition (Guillaumin et al., 2009 Cui et al. 2013)). person re- re-iddntiiicattoon (Bak & Carr 2017 Liu et al., 2017). image localization (Arand- methods improve the accuracy of image retrieval by casting it as metric learning tasks (Gordo et al.) [016)] Chen et al.) 2018 Brown et al.) 2020), architecture design/search tasks (Zhou et al.) 2019), or httf//smii-uuperveeee
jelovic et al. 2016: Ge et al. 2020b). Deep learning-
learning tasks (Ge et al.] (2020c a). Although many efforts have been made to improve retrieval performance, few people pay attention to the bottleneck of model upgrades in industrial applications.
HOT-REFRESH MODEL UPGRADES OF IMAGE RETRIEVAL SYSTEMS
Given images of interest (referred to as query Q), image retrieval targets correctly recognizing im- ages of the same content or objects from a large-scale database (referred as gallery G). An image encoder d(-) is employed to map the images into feature vectors. Then, the dot product of normal- ized vectors is used for measuring the similarity between query and gallery images for ranking.
3.1 HOT-REFRESH MODEL. UPGRADES WITH COMPATIBLE REPRESENTATIONS
Let dold be the old image encoder that was deployed in the retrieval system and Onew be stronger new encoder which is expected to replace the old one. We denote the evaluation metric for image retrieval, e.g., mean Average Precision (mAP), as M(.-).. The accuracy of the old system is denoted as M(Qula. Gola), where Qoid, Gold are the sets of query and gallery features embedded by Gold. respectively. The new encoder shows an improved accuracy of M Gnew). In conventional cold-refresh model updating pipeline, offline backfilling the gallery images from Gold to Gnew may take months for the massive data in industry applications, reducing the efficiency of the retrieval system upgrades to large extent.
Thanks to the compatible representation learning (Shen et al. 2020), query features encoded by the new encoder (i.e., Qnew) can be directly compared to the old gallery features (i.e., Gola). The re- trieval system can benefit from the new encoder immediately in a backfill-fre manner, significantly accelerating the model iteration at the expense of the optimal performance,
Gola) MM( (new., Guia) < M (neew: Gnew)
3

---

Published as a conference paper at ICLR 2022
Old/new features
Pull closer
Push away
New-to-old positive pair
Regression-free
compatible
Old encoder
New-to-new
negidiive pair
New- oodd
negative pair
Classifier
OOOO
Classification
loss
Figure 2:
The illustration of our Regression Regression-alleviatng Compatible Training (RACT) framework.
Given that the new features and old features are interchangeable due their compatibility, the gallery images can be backfilled on-the-fly progressively improve the retrieval accuracy to achieve the optimal performance of the new model,
M( Onew, Gold) + ..... MMQneww G
80%
... + MM( F U G
50%
oodd
MM(Q
ood
20%
M(Q Gnew
(2)
The above process termed as hot-refresh model upgrades, where 920% indicates the 20% of images the gallery have been re-indexed by the new model (new
3.2 MODEL REGRESSION WITH NEGATIVE FLIPS
Ideally, the performance of the retrieval system should be smoothly upgraded along with the gallery online backfilling, and the user experience would be continuously improved. However, observe the phenomenon of model regression especially in the early backfilling stages, that is, the accuracy degrades when part of the gallery images are refreshed, e.g..
M(Qnew 220% 80%
< M(Quew Gora).
(3)
is actually due the negative flips, where some query images correctly retrieved by the old model are incorrectly indexed by the new model during the upgrade process. There are two main potential factors that cause the negative flips: (1) new-to-old query-galle grryygallrry positive pairs are less similar than new-to-new query-gallery negative pairs; (2) new-to-new positives less similar than new-to-old negative pairs. Given that the new model generally encodes more discriminative features than the old model for the overall performance gains, focus the former in this paper, i.e., there exists a query By that satisfies
(Dnww Daaa (z.p)) (srnew (Iig); Onew (x. ))
is the positive gallery sample, Iy neg is the negative gallery sample, and (.. denotes the cosine similarity of two normalized feature vectors. In the case of Eq. (3). Onew (Igg)
where pos
Qnew
Dotal (a pos) E Gala and &new (I.get ) E 920%
4 METHODOLOGY
We introduce a regression regression-alleviatinn alleviating compatible training approach (Figure 2 for tackling the chal- lenge of model regression in hot-refresh model upgrades of image retrieval systems, which has essential impacts on the user experience but was ignored by previous compatible training methods. Our key idea is to encourage the new-to-old positive pairs to be more similar than both new-to-old negative pairs and new-to-new negative pairs. In addition, an uncertainty-bo to backfill the gallery in "the-poor-first" manner, in order to fasten accuracy improvements.
strategy is proposed
4.1 REVISIT CONVENTIONAL COMPATIBLE TRAINING
Being cast as either the metric learning task (Budnik Avrithis 2020) or the transformation learning task (Wang et al. 2020), the core of existing compatible training methods is to encourage the new feature an image to approach the old features the intra-class images while keeping away from instance discrimination-
the old features of different classes. We treat the compatible training like task (i.e the positive pair is the new and old features of the same instance) for efficient training in large-scale retrieval tasks. To verify that such instance discrimination-like strategy does not.
4

---

Published as a conference paper at ICLR 2022
hurt the retrieval accuracy well the feature compatibility, we conduct experiment by using a pre-sampler for constructing positive pairs of intra-class instances for comparison. As shown in Table of Appendix, similar results can be observed with without pre-sampler. The reason for the similar results might be that the most significant objective of compatible training is to train the new model to map images to same features the old model rather than constraining the distance between positive pairs, according to (Budnik & Avrithis 2020). Note that the negative pair is built by the new and old features of inter-class instances in the mini-batch, i.e., the intra-class samples would not be treated as the negatives in our instance discrimination-lik like compatible training. We denote as the training set and B as the mini-batch Given an anchor image x T, we can formulate the backward-compatible regularization as contrastive loss.
(ep(()ne((x)) (z)) Ccomp(z) (2) = log exp( (z), Pold (z) + kkBBLp(a) exp( (#new (2), O (k))/T)
(5)
where is temperature hyper--arameet and p(x) denotes the set of intra-class samples for (including x) in the mini-batch. With the above regularization, &new is trained to embed new features on the same latent space the old features. However, they ignore the relation between new-old and new-new similarities, facing the risk of negative flips during model upgrading.
4.2
CIAIEESSION-ALEVIAATII COMPATIBLE TRAINING
-alleviaon-lleeiiiinngg Compatibility Regularization. To constrain the feature compatibility and reduce negative flips at the same time, we improve the conventional comppalll loss by adding new-to-new negative pairs extra regularization term. Such an improved backward- compatible loss provides regression-alleviating regularization during training, that is formulated as
cop (x)
exp((Oneu (2), Ded (x))/T)
log
exp((dnew (1)), (ou()))/T)+ Exerppt) exp((1eem (x), dou (b))/T) + exp((see (x), Onew (k)) (k))/T)
In such a way, the new and old compatible positive pairs are trained to achieve more prominent similarities than both new-to-old and new-to-new negative pairs. The issue of negative flips can be alleviated properly when the gallery backfilling proceeds, i.e., the gallery is a mix-up of both old and new features. We do not specifically constrain the new-to-new positive pairs to be closer than new-to-old negative pairs, as the new features are generally trained to be more discriminative than the old ones in order to achieve the model upgrading purpose. Thus, the new-to-new positives are naturally to achieve larger similarities than new-to-old negatives.
Overall Training Objective. Besides the compatibility constraints, we also need the ordinary training regularization for image retrieval tasks maintain the distinctiveness of the new feature representations. Following the strong benchmark in /Weyand et al. (020). we choose the classifica- tion pretext task during training. Specifically, we adopt fully monnecco layer as the classifier, denoted as Wnew f + {l.-... CJ. where C is the number of classes of the new training dataset. The classification loss for the pretext task is formulated as
Lek(x) Lee (Whew &new (I), v(x));
where lee denotes the cross-entropy loss, y(x) is the ground-truth label for x, and Whew &new (x) produces the class logits after a softmax operation. The overall training objective function for our regression-alleviatin compatible training framework is,
(8) where À is the loss weight. New models trained towards the above objectives enable better hot- refresh upgrades of image retrieval systems by properly alleviating the problem of model regression.
C(x) = Las(x) + ACre-comp(x),
4.3 UNCERTAINTY-BASED BACKFILLING
In order to further fasten the accuracy convergence during the hot-refresh model upgrades, we pro- pose to refresh the gallery following "the "the-poor-fin foor-firtf""" priority, i.e., images with non-discriminative features expected to be re-encoded first. Given the massive data in industrial applications, need to quickly measure the distinctiveness of old gallery features in a relatively lightweight way. An uncertainty-bace -based strategy is therefore introduced measure the priority for backfilling the gallery.
5

---

Published as a conference paper at ICLR 2022
We claim that the confidence of class probabilities represents the feature distinctiveness to some tent. Since the old and new latent spaces are directly comparable after compatible training, we can produce the class probabilities of the old gallery features by feeding them into the trained new classi- fier Wiew- A Although the ground-truth classes the gallery images are generally non-overlapped with the new training dataset and even unknown, we can assume that more discriminative features should have sharper class probability vectors. Inspired by the "uncertainty sampling" (Settles) 2009), we evaluate the sharpness the probability vectors using the uncertainty estimation metries. Samples with larger uncertainty scores will be assigned higher priority to be backfilled. We consider three common ways of measuring uncertainty in this paper.
Least Confidence Uncertainty. It considers the gap between 100% and the probability of the predicted class the uncertainty score for each sample, formulated
Uf) = =1 sontmax(wnew (fg))' fg Gold.
where softmax[-] denotes the probability of rank k-th confident class.
Margin of Confidence Uncertainty. It measures the uncertainty of class probabilities by the dif- ference between the top-2 confident predictions,
Ume(fg) = (softmax(wnew (fz)l' softmax[wnew/ (fg)13), fg E Gold
(10)
Entropy- Uncertainty. The above metrics only consider the top-1 or top-2 classes, while the Entropy-based uncertainty measurement takes the overall probability distribution into consideration.
(c(fg) = Σ coftmax[wwedd (fy)l* log(softmax (fg)') fg E Golds
(11)
where the number of classes in new training set.
The introduced two technical parts of our work, i.e., regression-allevaaing compatible training and uncertainty- -bsse backfilling strategy, together contribute to alleviating the model regression prob- lem in hot-refresh model upgrades. The former explicitly regularizes the models to reduce negative flips at training time, while the latter can implicitly reduce negative flips by refreshing the poor features (which show higher risk for negative flips) in priority test time. They be considered as an entire solution (from training to testing) for hot-refresh model upgrades.
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Training Data. Google Landmark v2 (Weyand et al.) 2020). GLDv2 in short, large-scale public dataset for landmark retrieval. Following the setup in (Jeon 2020),
clean subset of GLDv2, dubbed GLDv2-clean, which consists of 1,580,470 samples with 81,313 classes. In this paper, we investigate three different kinds of data allocations for training the old model and the new model, covering most scenarios model upgrades in practice. (1) Expansion: the new training set is expansion of the old training set. We randomly sample 30% of images as the old training set, and use the 100% of images as the new training set. (2) Open-data: the new training data are excluded from the old training data, but they share the same classes. We randomly sample 30% of each class as the old training set, and use the rest 70% of each class as the new training set. (3) Open-class: the new training data and classes are both excluded from the old ones. We randomly sample 30% of classes for the old model training, and images of the rest 70% of classes are utilized for the new model training. The training data details can be found in Table
Old train-set
New train-set
Allocation type
images
# classes
images
classes
Expansion
445,419
81,313
1,580,470
81,313
Open-data
445,419
81,313
1,135,051
81,313
Open-class 472,604
24,393
1,107,866
56,920
Table 1: Three different allocations for the training data, where all the images are sampled from GLDv2-clean. The "expansion" and "open-data" setups share the same old training set.

---

Published as a conference paper at ICLR 2022
mAp old model
mAP of vanilla
mAP of ours
NFR vanilla
NFR of ours
18
18
16.54
16
15.65
16-
16.42
15.48
14
14
15.12
11.90
12
11.40
Model regression
12
12
11.05
10.61
11.65
11.37
#11:004
0%
25%
50%
75%
100%
10
0%
25%
9.91
50%
75%
100%
10
0%
25%
50%
75%
100%
(a) Expansion
(b) Open-data
(c)
Open-clas
Figure 3: The trend of retrieval performance during the hot-refresh model upgrades (R50-R101) on GL.vv-test, in terms of mAP(") and NFR(J). The vanilla models suffer from the model regression significantly, while ours properly mitigate the model regression to some extent. The results of NFR also indicate the effectiveness of our introduced regression-alles -alleviainnni compatible regularization.
Testing Data. We evaluate the trained models on three test sets, including GLDv2-test, Revisited Oxford (ROxford), and Revisited Paris (RParis). GLDv2-test contains 761,757 gallery images and 750 query images. Following the test protocol in (Radenović al. 2018), J. we use the medium testing with 70 queries for both ROxford and RParis. ROxford (Philbin et al. (207)) contains 5,007 gallery images, and RParis (Philbin et al.
2008 consists of 6,357 gallery images.
Evaluation Metric. Mean Average Precision (mAP) is adopted here for evaluating the retrieval performance. Following the evaluation protocols in (Budnik & Avrithis 2020) Weyand et al.) 2020). we use mAp@ 100 for GLDv2-test, and mAP@ 10 for ROxford and RParis. Since mAP measures the overall accuracy with the consideration of both positive flips and negative flips, we define a
metric, namely negative flip rate (NFR), measure the degree of model regression in hot-refresh model upgrades. Note that our NFR is specifically defined for image retrieval tasks, different from the one introduced in Yan al. 2020 Our NFR formulated
1 n n 1
NFR@K
(12)
where 2 denotes the set of queries that correctly recall relative images from the old gallery in top- candidates. 2g is the set of queries that fail be recalled by top-k galleries during upgrading, where the gallery is mix-up of both old and new features. NFR @ is used in this paper.
Architectures. We study two kinds of architecture combinations, (1) R50-R101: ResNet-50 (He al. 2016) for the old model, and ResNet-101 for the new model; (2) R50-R50: ResNet-50 for both old and new models. Due to space limitation, we discuss R50-R101 in the main paper, and the results of R50-R50 can be found in Appendix A.2 Training details can be found in Appendix A.
5.2 CUGGESSSION-LLLVVIATTEI
COMPATIBLE TRAINING
We start with the architecture combination of R50-R101, i.e., ResNet-50 for the old model, and ResNet-101 for the new one. To verify that our regression-alleviatnn compatibility regularization (Eq. (6)) can properly alleviate the issue of model regression, we compare our method to the vanilla compatible training, i.e., Evanilla (z) Les(z) + XCcomp where Lass is defined in Eq. (7). and Lcomp is defined in Eq. (5). Our uncertainty-based backfilling strategy is not used in this section in order to fairly compare with the vanilla settings. The only variable here is the compatibility regular- ization, ie., Leomp for vanilla and Lra for ours. We evaluate the models trained with expansion, open-data and open-class datasets (see Table [] by testing on three different test sets, i.e., GLDv2- respectively. It can be observed that the
test, ROxford, and RParis, as illustrated in Figure 3 models trained with vanilla compatibility regularization (i.e., "vanilla") suffer from the model re- gression significantly. In contrast, the models trained with regularization (i.e., "ours") mitigate the model regression to some extent by properly reducing the negative flips.
gggression-alleviatte compatibility
7

---

Published as a conference paper at ICLR 2022
old
vanillaa
74.75 72.98
64
73.40
68.50
71.72
70
70
64.33
65-10 63.86
(64.28
63.89
0%
25%6
50%
75%
100%
0%
25%
50%6
75%
100%
0%
25%
50%
75%
100%
(a) Expansion
(b) Open-data
(c) Open-class
Figure 4: The trend of retrieval performance during the hot-refresh model upgrades (R50-R101) ROxford, in terms of mAP(1) and NFR()) Evident model regression can be observed for vanilla models, and our method alleviates this issue to a large extent.
of old model
mAP vanilla
of NFR vanilla
86.10
83.61
83.27
83.27
86.04
$3.40
84.54
61..2
80
88.64
81.52
75-
75-
75
79.31
70
50%
75%
100%
70
0%
25%
50%
100% 0%
70
25%
50%
100%
(a) Expansion
(b) Open-data
Open-clas
Figure 5: The trend of retrieval performance during the hot-refiresh model upgrades (R50-R101) on RParis, in terms of mAP(t) and NFR()). Our method consistently alleviates the issue of model regression three different training data allocations. The new-to-old compatible mAP 0% of our model is lower than the old-to-old mAP on the open-data setup, showing the same issue as the vanilla model. mainly due the limited generalization ability of compatible features trained with a sole contrastive loss. Note that in this paper, we aim to study the regressiion-alleviatng regularization rather than advanced compatibility constraints.
5.3 UNCERTAINTY-BASED BACKFILLING
The uncertainty- bassed backfilling strategy is proposed to refresh the gallery following "the-poor- first" priority, in order fasten the accuracy improvements of the image retrieval during hot-refresh upgrading. The uncertainty-based backfilling is plug-and-play with different models. We discuss different backfilling strategies on ROxford, as shown in Figure 6 and draw the following conclu- sions: (1) The Least Confidence and Margin of Confidence generally achieve better performance than Entropy-basee based uncertainty, indicating that the probabilities of top-2 confident classes are more important than the overall classes; (2) The uncertainty-basse -based backfilling strategy can fasten the mAP convergence on top of either vanilla method or our RACT method; (3) Since the uncertainty-baa based backfilling strategy attempts to refresh the poor features (which show higher risk for negative flips) in priority, it can indirectly alleviate the model regression problem. How achieve faster and smoother mAP improvements during backfilling is new task, and first investigated in this paper.
5.4 SEQUENTIAL MODEL UPGRADES
In real-world applications, model upgrades are often conducted sequentially to achieve continuous improvements. To simulate such scenario, we split the training data into 30%, 60%, 80%, and 100% for training the old model, the upgraded model of the 1st generation, the 2nd generation, and the 3rd generation, respectively. As shown in Figure 7 our method can achieve consistent improvements the vanilla baseline in terms of hot-refresh model upgrades (solid line). Another kind of model upgrading mechanism (dotted line) introduced by (Shen et al. 2020)). denoted as "no- refresh" model upgrades, deploys the compatible new model without refreshing the gallery features.
8

---

Published as a conference paper at ICLR 2022
vonilla with Random backfilligg vanilla with Least Conf. backfilling- vonilla with Margin Conf, backfillng-- vanilla with Entropy backfiling ours with Random backfilling
purs with Least Conf. backfilling
ours with Margin Conf. backfilling
curs with Entropy backfilling
(0) Expancioon
Bb Opppp data
10) Open class
Figure 6: Comparisons between random backfilling and our uncertainty- based backfilling on ROx-
ford. Results of R50-R101 are reported in terms of mAP(t). Three kinds of uncertainty studied.
mAP old model vanllia-Hotrefresh
of ourseHot--efresh
vanila--o--refnes Nofresshrrefeehh--
refresh
17
Generation 3 (100% data)
11..0 15.90
Generation (80% data) 15.23 15.00
15.70
Generation (60% data)
13.99
15.00
13.82
11.13
0.428
10.95
8.951
9.91
Figure 7: Sequential model upgrades GLDv2-test (Open-data). Results of R50-R50 are reported in terms mAP(t). "Hot-refresh" the introduced new model upgrading mechanism where the gallery features are refreshed online, and "no-refresh" indicates the backfill-free deployment mech- anism introduced by previous compatible learning methods (Shen et al., (0200).
We observe the considerable performances gaps between hot-refresh model upgrades and the no-refresh model upgrades. It is actually due to the fact that the poor discriminativeness of the old gallery features highly limits the retrieval accuracy of the new model in no-refresh model upgrades.
CONCLUSION AND DISCUSSION
This paper for the first time studies the issue of model regression during the hot-refresh model up- grades for image retrieval systems. We hope it can arouse more public attention to the bottleneck of efficient model upgrades in industrial applications. Despite that the introduced compatibility regularization can properly reduce the negative flips to some extent, there is still a long way to go for completely wiping out the problem of model regression. Specifically, several open questions are worth investigating. (1) In this paper, we only consider the negative flips during the gallery backfilling procedure, however, there may also exist negative flips after the backfilling is finished. (2) We investigate regression-allleiaatnn regularization in the form of a plain contrastive loss. may further help if integrating the regression-alleiiting regularization into more advanced backward compatible losses. (3) We introduce to backfill the gallery in "the-poor "oee-porffist" priority with uncertainty estimation. achieving faster and more stable accuracy improvements.
pegressionnalleviitte
also a new task for model upgrades. Future works are required for
We would also like to discuss the limitations of the introduced method. The raw images of the gallery need to be stored for supporting the hot-refresh model upgrades, which may not be satisfied due to privacy issues. We considered forward-compatible learning (Meng et al. 2021 Wang al.) 2020) as potential solution for dealing with this limitation, which is expected to upgrade the gallery features with the input only the old features rather than raw images.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We demonstrate the core of the algorithm the form of pseudo codes, as shown in Alg. | and
2 The training details and syper--prameters be found Appendix A.1 The datasets are all The full code will
publicly available and the training data splits are carefully illustrated in Table be released upon acceptance.
Algorithm Pscudocode of eeggressionaalleviatee Compatible Training in PyTorch-like style.
momdeel pretrained and fixed old encoder, no gradient model fo. 1a the classitier
modell
wi loss weight, \iambda in the paper tau: temperature, Ytau in the paper
for 1x, targeta) loader: load a mini batch x with samples
x_old = (ug # randemly augmented version x_aow aug(x) snother randomly augmented version
with torch.no_gradie
ald_faat ftd_modn1..frvarrdee old) tensor shape: NxD (x_nex)
new_
classitication loss, Eqn, (7)
cls_logits sew_modei.ffeete feeee feat)
cls_lasa Crondentropylons (rorddntrgyyloossslll......o logit,,,,s,, targets)
select intra-class semples
mosks targets. targets.cxpand/k (N, N) wt.eettargets..xpadd (N, 8) ())
positive logita: Nx1
1_pos bmm www.:eee_..at.veeewl..lelo fed__estt.ierxW...
negative logita:
exciude intra-class samples
___ee__2n am feat._eett.vieee view D), new new_feat feat. view N)) masks+le9
i_agg_n2o =n(nev_feat.
ald_feat. (D, maskselc9
logits: (x(1)28)
logits pos_ i_neg_22s, 1_neg_n2ol, dimml)
regression alleviating compatibllity loas, Sgn. (6)
labels (NI positives the 0-th
re_comp_lese CrossintropyLacc(logtt (lagita/sau, labels)
83D update: model
cis_loss w-ra_ccmp_ion
loss. backwark)) update (aew_model.pparass
omm: hatch matrix maltiplication; mm: matrix maltiplication; cat: concatenation; [): matrix transpose.
Algorithm 2 Pseudocode of Calculating Uncertainty-based Backfilling Order in a PyTorch style.
3 N: the rumber ot images in the gallery number
new classifiex
def cal backft 1 (oddorrrrdlddfoet unsertalnty_eeratteeeerrrrr
with torch.no corch.no_ggad();
logits probs sort (softmax (logits, dim=l), dimel)
+ shape: NxC
if incertainty_strateee
"Seast"; (1) Least Confidance Uncertainty, Equ. (9)
uncertainty. strtyeggttrttgt 'margin': urcartainty
proball,C]
(2) Mazgin of confidence Uncertainty, Ega. (10)
uncertainty_scoree = [probs]:,0] probs[:,11]
inecaraantt__otratcce
'encropy':
(3) Entropy- based Uncertainty, Eqn. (11)
uncertainty_scoe
sum(probs logiprobs + 1e-9), dim-ll
clsa:
randon backtilling
return random. shuffle (N))
return argsort (uncertainty_
$ the-poor-first priority
sum: sum elements: log: logarithu; sort: sor the elements in descending order; argsort: the sorted indices in descending order.
10

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported by NSFC project Grant No. U1833101, SZSTI Grant No. and Tsinghua.
and WD7ZC20008820000555000 the Joint Research Center of Tencent
REFERENCES
Relja Arandjelovic, Petr Gronat, Akihiko Torii, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In IEEE Conf. Comput. Vis. Pattern Recog, 2016.
Slawomir Bak and Peter Carr. One-shot metric learning for person re-identification. In IEEE Conf.
Comput, Vis. Pattern Recog., 2017.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. Eur. Conf. Comput. Vis., pp. 677-694. Springer, 2020.
Mateusz Budnik and Yannis Avrithis. Asymmetric metric learning for knowledge transfer. arXiv preprint arXiv:2006.1633 2020.
Xiangyong Cao, Jing Yao, Zongben Xu, and Deyu Meng. Hyperspectral image classification with convolutional neural network and active learning. IEEE Transactions on Geoscience and Remote Sensing, 2020.
Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In AAAI, 2018.
Zhen Cui, Wen Li, Dong Xu, Shiguang Shan, and Xilin Chen. Fusing robust face region descriptors via multiple metrie learning for face recognition in the wild. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 3554-3561, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog. Ieee, 2009.
Rahul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility heterogeneous visual search. arXiv preprint arXiv:2105.06047 2021.
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for un- supervised domain adaptation on person re-identification. In Int. Conf. Learn. Represent. 2020a.
Yixiao Ge, Haibo Wang, Feng Zhu, Rui Zhao, and Hongsheng Li. Self-supervising fine-grained region similarities for large-scale image localization. In Eur. Conf. Comput. Vis., 2020b.
Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, and Hongsheng Li. Self-paced contrastive learning with hybrid memory for domain adaptive object re-id. In Adv. Neural Inform. Process. Syst., 2020c.
Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. Eur. Conf. Comput. Vis. Springer, 2016.
Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Is that you? metric learning approaches for face identification. In Int. Conf. Comput. Vis. IEFF, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conf. Comput. Vis. Pattern Recog., 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv 2015.
preprint ar
Alex Holub, Pietro Perona, and Michael C Burl. Entropy-based active learning for object recogni- tion. In JEEE Conf. Comput. Vis. Pattern Recog. Worksh. 2008.
SeungKee Jeon. 1st place solution to google landmark retrieval 2020. arXiv preprint 00:::2009.05132 2020.
11

---

Published as a conference paper at ICLR 2022
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In IEEE Conf. Comput. Vis. Pattern Recog., 2020a.
Shaohua Li, Xiuchao Sui, Jie Fu, Yong Liu, and Rick Siow Mong Goh. Feature lenses: Plug- and-play neural modules for transformationiinaaaaan visual representations. arXiv preprini arXiv:2004.05554, 2020b.
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep filter pairing neural network for person re-identification. IEEE Conf. Comput. Vis. Pattern Recog., 2014.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 2017.
Zimo Liu, Dong Wang, and Huchuan Lu. Stepwise metric promotion for unsupervised video person re-identification. In Int. Conf. Comput. Vis., 2017.
Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. In Int. Conf. Comput. Vis., 2021.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, 2007.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Lost in quantiza- tion: Improving particular object retrieval in large scale image databases. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, 2008.
Filip Radenović, Ahmet Iscen, Giorgos Tolias, and Ondřej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In IEEE Conf. Comput. Vis. Puttern Recog,, 2018.
Burr Settles. Active learning literature survey. 2009.
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatible repre- sentation learning. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. An empirical analysis of backward compatibility in machine learning systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, 2020.
Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint adeniifiatioon-vvriiccuttt In Advances in neural information processing systems, 2014a.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In IEEE Conf. Comput. Vis. Pattern Recog., 2014b.
Chien-Yi Yi Wang, Ya-Liang Chang, Shang-Ta Yang, Dong Chen, and Shang-Hong Lai. Unified rep- resentation learning for cross model compatibility. Brit. Mach. Vis. Conf., 2020.
Shuzhe Wang, Zakaria Laskar, Iaroslav Melekhov, Xiaotian Li, and Juho Kannala. Continual leamn ing for image-based camera localization. In Int. Conf. Comput. Vis., pp. 3252-3262, 2021.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In IEEE Conf. Comput. Vis. Pattern Recog.. 2020.
Yuqing Xie, Yi-an Lai, Yuanjun Xiong, Yi Zhang, and Stefano Soatto. Regression bugs are in your model! measuring, reducing and analyzing regressions in nlp model updates. arXiv preprint 105.022000.03048 2021.
Sijie Yan, Yuanjun Xiong, Kaustav Kundu. Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Ste- fano Soatto. Rositive-congrute training: Towards regression-frre -fre model updates. arXiv preprint arXiv:2011.09161. 2020.
12

---

Published as a conference paper at ICLR 2022
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Int. Conf. Comput. Vis., 2015.
Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. Omni-scale feature learning for person re-identification. In Int. Conf. Comput. Vis., 2019.
Jingbo Zhu, Huizhen Wang. Tianshun Yao, and Benjamin K Tsou, Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In P Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), 2008.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 TRAINING DETAILS
The networks are pre-trained on the ImageNet dataset (Deng et al. 2009). The images are resized to 224 224 for both training and testing. During training, random data augmentation is applied to each image before it is fed into the network, including randomly flipping and cropping. We adopt Tesla V100 GPUs for training. and the batch size per GPU is set to 80 for ResNet-50 and 64 for ResNet- 101. For all the experiments, Adam optimizer is adopted to optimize the training model with a weight decay of 0.0001. The initial learning rate is set to 0.01 and is decreased to 1/10 of its previous value every 30 epochs in the total 90 epochs. The old model is trained with only classification loss in Eq. 7 and the new model is trained towards the objective in Eq. (8) The
temperature in Eq. 6 empirically set 0.05, and the loss weight in Eq. (8) set as T.O.
mAP of old model
mAP of varillla
mAP ours
NFR of vanillla
NFR of ours
15.44
16
16
14.42
15.15
14.26
14.40
14.09
11.97
12
12
11.70
10.95
10.76
#11.04
07
9.91
25%
50%
75%
100%
10
0%
25%
99.91
50%
75%
100%
10
0%
25%
50%
75%
100%
(a) Expansion
(b) Open-data
(c) Open-class
Figure 8: The trend of retrieval performance during the hot-refresh model upgrades of the same architecture, i.e., R50-R50. The results are reported GLDv2-test in terms of mAP(f) and NFR(.). We observe that, despite the overall accuracy (i.e., mAP) has gradually increased without regression, vanilla shows significantly larger NFR than ours. Noted that even if the overall mAP has increased, every new mistake on the old positive queries feels like a step backwards, still leading to a perceived regression in experience. Reducing NFR can be important as improving mAP in practice.
mAP old model
of vanilla
mAP ours
NFR of vanilla
of ours
78
78
78
72.78 71.26
73.40
71-
71-
71-
67.34
68.50
64.28
64.38
65, 65.10
64
64.28
64
64.23
63.96
$2.46
57-
57
57-
50
0%
25%
50%
75%
100%
0%
257
50%
75%
100%
0%
25%
50%
75%
100%
(a) Expansion
(b) Open-data
(c) Open-class
Figure 9: The trend of retrieval performance during the hot-refresh model upgrades (R50-R50) ROxford, in terms of mAP(t) and NFR()).
A.2 MODEL. UPGRADES OF THE SAME ARCHITECTURE
Regression-alleviainn Compatible Training. We further evaluate our method when the new model has the same architecture as the old model, egg. ResNet-50 for both old and new models. As illustrated in Figure g the retrieval system with vanilla models is less susceptible to the model regression problem compared to the R50-R101 setup (Figure 3) when testing GLDv2-test. This is because that old and new models the same architecture start from the same weights, i.e., Ima- geNet pre-training. They have natural prediction consistency in the beginning, making the feature
14

---

Published as a conference paper at ICLR 2022
mAp of old model
mAP of vanilla
mAP of ours
NFR of vanilla
NFR ours
90
90
86
85.48
85.92
86
85.03
86
84.89
83.60
84.76
84.28
83.27
82:
83.27
82
81.48
79.68
0.7.65
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
(a) Expansion
(b) Open-data
(c) Open-class
Figure 10:
The trend of retrieval performance during the hot-refresh model upgrades (R50-R50) on RParis, in terms of mAP(1) and NFR()). Our method consistently alleviates the issue of model regression on three different training data allocations. The new-to-old compatible mAP at 0% of our model is lower than the old-to-ole old mAP expansion and open-data setups, showing the same issue as the vanilla model. It is mainly due to the limited generalization ability of compatible features trained with sole contrastive loss.
compatibility easier. Even argue that vanilla models show evident larger NFR than model, indicating an underlying model regression. The results of ROxford and RParis are in Figures 10
Uncertainty-basee Backfilling. We investigate the uncertainty- backfilling on the setup of R50-R50, shown in Figure [T] Backfilling the gallery with uncertainty-base priority effec- tively fasten the performance improvements of image retrieval systems, as well as implicitly alleviate the model regression.
vanillla Random backfilling - vanilla Least backfilling vanilla Margin Conf. backfillling- vanilla Entropy backfilling ouus with Random backfilling ours with Least Conf. backfilling
ours with Margin Conf. backfilling ours with Entropy backfilling
(6) Expension
(b) Open-datt
38
28
14
10 Open-class
20
Figure 11:
Comparisons between random backfilling and our introduced uncertainty- backfill- ing on ROxford, in terms of mAP(1). Results of R50-R50 are reported.
A.3 ADDITIONAL EXPERIMENTS
Analysis Uncertainty-bant based Backfilling. We further investigate the uncertainty-baaeed back- filling with Margin of Confidence metric on RParis and GLDv2-test, shown in Figure 12] where the uncertainty- based backfilling generally achieves faster mAP convergence with either the vanilla models models. Similar conclusions with the ROxford (discussed in Section 5.3 be drawn, i.e., the uncertainty-batt based backfilling strategy can indirectly alleviate the model regression problem since the poor features with higher risk for negative flips refreshed in priority.
15

---

Published as a conference paper at ICLR 2022
vanilla with Random backfilling with Random backfilling
vanilla with Margin Conf. backfilling
ours with Margin Conf. backfilling
55%
75%
100%
25%
704
3805
100%
[a] Open-class on GLDv2
(a) Open-class on RParls
Figure 12: Comparisons between random backfilling and our introduced uncertainty-based backfill- ing on GLDv2-test and RParis. Results of R50-R101 101 are reported in terms of mAP(1).
vanilla with Random backfilling vanilla with Least Conf. backfilling vanilla with Margin Conf. backfilling vanilla with Entropy backfilling with Random backfilling
with Least Conf. backfilling
with Margin Conf. backfilling
with Entropy backfilling
Figure 13: Failure cases of our introduced uncertainty-base backfilling RParis (Open-data).
Results of R50-R101 are reported in terms of mAP(t).
However, we still face some failure cases with the backfilling strategy, as shown in Figure 13 We strategy may not work when the new and old features are not com-
argue that the uncertainty-basee patible enough, since we use the trained and fixed new classifier to produce class probabilities for the old gallery features under the assumption that they lay on the almost same latent space. To fur- ther investigate the phenomenon of Figure 13 we observe that the new-to-old performance (0%) is inferior the old-to-old performance (brown star) in Figure (b) of the main paper, showing the unsatisfactory feature compatibility. More advanced compatible constraints have the potential to solve this issue, but not the core of this paper.
The introduced uncertainty- -based backfilling method is simple but consistently effective in most cases. We acknowledge that the method is not perfect enough to handle all the deployment scenar- ios, i.e., it fails when the feature compatibility is not good enough (generally when the new-to-old retrieval performance is lower than old-to-old retrieval performance). However, we give the first at- tempt totally new task, showcasing the great potential for further work that incorporates model upgrades with ordered gallery images.
The evaluation of compatibility be observed at 0% in all the fig. ures and tables for model upgrades. Specifically, at 0% of model upgrades, the new features of queries need to be directly indexed by the old gallery feature, which an ordinary benchmark for conventional compatible learning (Shen et al.] 2020). To make it clear, we listed the above results in We can observe that our method can achieve better compatible performance than the vanilla baseline, indicating that we do not sacrifice the compatibility for reducing negative flips.
Compatibility Evaluation.
Table 2
Expansion ROxford
Open-data ROxford
Open-class ROxford RParis
Model
GLDv2
RParis GLDv2
RParis
GLDv2
R.50 (old)
9.91
64.28
83.27
9.91
64.28
83.27
11.04
64.40
83.40
R50 (vanilla new)
10.95
64.23
79.68
10.46
62.46
78,96
11.70
63.86
81.48
R:50 (ours new)
11.13
67.34
79.65
10.76
64.38
81.63
11.97
65.10
83.60
R101 (vanilla new)
11.37
65.87
81.52
10.61
63.89
79.31
11.65
63.86
81.52
R101 (ours new)
11.40
66.54
83.75
11.05
64.33
81.64
11.90
65.10
83.61
The evaluation of compatibility in terms mAP, i.e., 0% results hot-refresh model upgrades, where the new features of the queries need to be retrieved by the entire old gallery features. It can be considered as an ordinary benchmark for compatible learning.
Table 2:
16

---

Published as a conference paper at ICLR 2022
A.4 ABLATION STUDIES
Effects of temperature т. The temperature T is empirically set as 0.05 in this work. We discuss different values of in Table A similar increasing trend of retrieval performance upgrades can be observed when т varies from 0.05 to 0.1, while model regression at 20% is observed when T = 0.01. The 100% new-to-new performances are sub-optimal when T 0.1 and T = 0.08. In all, 0.05 is suitable value for the contrastive temperature, and all the experiments in this paper adopt the same temperature value for fair comparison.
r
old
0%
20%
40%
60%
80%
100%
0.1
9.91
11.13(+1.22)
11.44(+0.30)
11.70((0.26)
12.05(+0.35)
12.57(+0.52)
13.28(+0.71)
0.08
9.91
11.24(++.33)
11.65(+0.40)
11.96(+0.31)
12.29(40.33)
12.97(+0.68)
13.73(+0.76)
0.05
9.91
11.13(+1.22)
11.88(+0.75)
12.47(+0.59)
15.14(+0.78) 14.37(+1.05) 12.55(42.01)
13.32(40.85)
0.01
9.91
10.34(+0.43)
10.07(-0.27)
10.54(+0.46)
15.12(+2.57)
15.87(+0.74)
Table 3: Performance of our method with different values of temperature 7 in terms of mAP(f). The results are reported on GLDv2-test with the architectures of R50-R50.
Effects loss weight x. The loss weight À is set as 1.0 in this work for simplicity. We study the ef- fects of different in Table[4 We can observe that the value controls the trade-off between new feature discriminativennss and new-to-old feature compatibility, i.e., models with smaller À achieve better performance at 100% while models with larger achieve better performance at 0%. Despite the start and end points vary from different models, similar increasing trend of the performances during backfilling can be observed.
A
old
0%
20%
40%
60%
80%
100%
0.1
9,91
10.66(+0.75)
10.66(+0.00)
11.28(+0.62)
12.66(+1.38)
14.34(+1.67)
15.64(+1.31)
0,5
9.91
10,94(+1.03)
11.71(+0.77)
12.24(+0.53)
13.19(40.95)
15.07(+0.76) 14.31(+1.12)
1 9.91
11.13(+1.22)
11.88(+0.75)
12.47(+0.59)
13.32(+0.85)
15.14(+0.78) 13.97(+0.81)
14.37(+1.05)
2
9.91
11.10(+1.19)
11.77(+0.67)
12.37(+0.60)
13.16(+0.79)
14.76(+0.79)
5
9.91
11.24(+1.33)
11.79(+0.55)
12.17(+0.38)
12.60(+0.43)
13.34(+0.75)
14.32(+0.97)
Table 4: Performance of our method with different values of loss weight À in terms of mAP((). The results are reported on GLDv2-test with the architectures of R50-R50.
Effects of batch size. We batch size of 80 per GPU for the experiments with ResNer-50. As the batch size may affect the number of negative samples for contrastive loss, we conduct ablation studies by modifying the batch size, as illustrated Table We observe that our method robust to the number of batch size.
b.s.
old
0%
20%
40%
60%
80%
100%
40
9.91
10.98(+1.07))
11.83(+0.85)
12.51(+0.68)
13.53(+1.01)
14.79(+1.26)
15.77(+0.99)
60
9.91
11.05(+1.14)
11.81(+0.76)
12.53(+0.71)
13.43(+0.91)
14.50(+1.06)
15.34(+0.85)
9.91
11.13(+1.22) 11.88(+0.75)
12.47(+0.59)
13.32(+0.85)
14.37(+1.05)
15.14(+0.78)
Table 5: Performance of method with different batch size (b.s.) per GPU in terms of mAP(T).
The results are reported on GLDv2-test with the architectures of R50-R50. We fail to try larger batch sizes due to the GPU memory limitation.
Effects of positive sampler. As mentioned in the method, for more efficient training, we treat the compatible training as an instance discrimination-like task when training with the large-scale retrieval dataset, i.e., GLDv2. Specifically, the positive pair consists the new and old features of the same instance. To verify that the mentioned instance discrimination-lik compatible training does not harm the retrieval performance, we conduct an ablation study by using a positive pre- sampler to randomly sample intra-class sample for each anchor before each iteration. As shown in Table similar results can be achieved by using either the positive pre-sampler or not. The reason for not seeing improvement with the positive sampler might be that, in the task of compatible
17

---

Published as a conference paper at ICLR 2022
Sampler
old
0%
20%
40%
60%
80%
100%
w/
9.91
11.41(+1.50)
12.11(40.70)
12.65(+0.53)
13.41(+0.76)
14.33(+0.92)
15.08(+0.76)
w/o
9.91
11.13(+1.22) 11.88(+0.75)
12.47(+0.59) 13.32(+0.85) 14.37(+1.05) 15.14(+0.78)
Table 6: Compare to the results when adopting positive pre-sampler. The results are reported on GLDv2-test with the architectures of R50-R50, in terms of mAP(1). Using the positive pre-sampler achieves slightly better performance at 0%, but lower training efficiency. Similar final performances can be achieved with either the sampler or not, so we discard the positive sampler for more efficient training with large-scale datasets.
training, the most significant objective is to train the new model to map images to same features as the old model, rather than constraining the distance between positive pairs. Such a phenomenon has also been observed by the related work (Budnik & Avrithis 2020). As shown in Table 3 of (Budnik & Avrithis 2020), the method of "regression" (forces the old feature and new feature of the same image similar using cosine similarity) wins the asymmetric test (cross-model test), compared to other methods that regularize the positive pairs. Moreover, the training dataset, GL.v2,, consists of 1, 1,580,470 samples with 81,313 classes, i.e., 19 samples per class, showing limited intra-class variations, which might also lead to similar results with or without such a positive sampler.
The trade-off between new-to-old negatives and new-to-new new negatives in Lra-comp. We inves- tigate the effects of balancing new-to-old negatives and new-to-new negatives in Lm-comp- Since does not make sense to directly impose weighted factors on the sample distances (i.e., (.-)) in our original loss function (Eq. (6)). first split our original unified contrastive loss into combination of two terms, formulated
exp( (2r), ())((
Crocomp-spii(()))
log exp((enewl) (x), Patel (+))/) + EnEBP exp( (nnee (x), Ouna (k))/T)
(13)
exp( (d)ee (2), dona (z)) (x)))/
+ n log exp((Pree (I), Dau (z)))/T + EkEBrp(e)) (epn((n (x), Prew (b)))/}}*
where 1) is weighting factor. When setting 1) = 1, similar results can be observed as the original setup. When changing 77 from 0.2 to 5.0, we find that a larger 1) may lead to better alleviation of model regression but lower new-to-old compatible performance (0%). We find 7 1.0 better trade-off between feature compatibility and alleviating model regression, showing that our unified version (Eq. (6)) of contrastive loss is reasonable. Detailed results are illustrated in Figure 14
MAP ourse Corn
mAP of our- Curone
mAP of Buus Eirenoperi
5.0
G3.80
$3.00
$2.00 50
$8. 9.
36
70
186
88
70C
206
66
100
78
106
88
90
86
100
10c
105
Figure 14: The analysis of trade-off between new-to-old negative pairs and new-to-new negative compatible regularization. We report retrieval performance during
pairs in our regression-allevian the hot-refresh model upgrades (R50-R50) on ROxford in terms of mAP(1).
Different temperatures for new-to-old negatives and new-to-new negatives in Lra-comp- We empirically set the temperature to 0.05. To further analyze the effects of different temperatures in two types of negative pairs, we conduct experiments as shown in Table 7 Fewer effects can be observed when changing the temperature, so we use 0.05 for both for brevity.
An alternative form of compatible regularizations - triplet loss. In the main paper, we introduce to regularize the compatibility of the learned representations in the form of contrastive loss. An alternative option for formulating the regularization is in the form of a triplet loss, where several triplets combining positives and negatives from old and new models are considered the same time. Specifically, the vanilla version of backward-compatible regularization is formulated as,
Loomp (x)
Σ
(max(()nee (z), (z)) (shew (x), cooad (k)) + m. 0)). (14)
BB p(() AEB))(u))
18

---

Published as a conference paper at ICLR 2022
mAP of old model
mAP of vanilla
mAP of ours
x
NFR of vanilla
NFR of ours
76
85.75
13.95
72
71.80
83.45 83.40
82.94
13.50
68
12.03
65.14
65.13
64.40
#11.04
10
0%
25%
50%
100%
0%
25%
50%
75%
100%
72
0%
25%
75%
100%
(a) Open-class on GLDv2
(b) Open-class on ROxford
(c) Open-class on RParis
Figure 15: Experiments of compatibility constraints in the form of triplet loss. We illustrate the trend retrieval performance during the hot-refresh model upgrades (R50-R50) on GLDv2-test, ROxford and RParis datasets, in terms of mAP(f) and NFR()).
71
old
0%
20%
40%
60%
80%
100%
0.05
0.05
9.91
10.76(+0.85)
11.66(+0.35) 11.31(+0.56)
12.59(+0.93)
13.51(+0.92)
14.09(+0.58)
0,05
0.1
9.91
14.12(+0.69 10.56(+0.65) 11.15(+0.59) 11.52(+0.36) 12.43(40.91) 13.47(+1.00)
10.64(+0.73)
11.14(+0.50)
11.51(+0.36)
12.53(+1.03)
13.43(+0.90)
0.1
0.05
9.91
13.89(+0.42)
Table 7: Performance of our method with different temperature in Eq. O6) in terms of mAP(1). The results are reported on GL.sv2test (Open-data) with the architectures R50-R50, тι and 72 denote the temperature for new-to-old negative pairs and new-to-new negative pairs, respectively.
Sampler
old
0%
20%
40%
60%
80%
100%
w/
9.91
10.76(+0.85)
11.31(+0.56)
11.66(+0.35)
12.59(+0.93)
13.51(+0.92)
14.09(+0.58)
w/o
9.91
10.82(+0.91)
11 11.30(40.48) 11.77(+0.47)
12.59(+0.82)
13.51(40.92)
14.12(+0.61)
Table Compare to the results when adopting a positive pre-sampler for experiments of triplet loss. The results reported GI.Dv-ttest with the architectures of R50-R50, in terms of mAP(f).
where the margin m is empirically set to 0.8 and p(r) denotes the set of intra-class samples for (including 2)) in the mini-batch. The regression-alleviatin compatibility regularization therefore becomes
Lexcomp (x)
Σ
(max(((neu (x), (x)a (Amew (oaa())) m. 0)+
13 ((x)) kEBip(m)
max((gnew (I), (oan (x)) - (Anow (z)) Prew (k)) + m. 0))
(15)
The comparisons between vanilla method and our regression-alleviatin, method in terms of open- class data split on three different datasets can be found in Figure [5] We observe that our method can properly reduce negative flips on GLDv2 and RParis datasets, but fail in ROxford. Both vanilla and ours show inferior performance than the contrastive counterparts, since the unified form of contrastive loss can establish a global structure among new-to-old and new-to-new pairs while dividing them into several triplets cannot.
We also investigate the effects of positive pre-sampler when conducting experiments with triplet loss. As demonstrated in Table 8 similar results can found with or without such sampler, drawing the same conclusion with the experiments of contrastive loss.
A.5 RELEVANT TOPICS
We discuss three relevant topics that also need an old model for training the new model, including compatible learning (Shen al.
2020 Meng et al. 2021 Budnik & Avrithis 2020), knowledge
distillation (Hinton et al. 2015 Li et al. 2020al Yan et al. 2020), and continual learning (Li &
Hoiem 2017 Wang al.
2021
19

---

Published as a conference paper at ICLR 2022
Compare to Compatible Learning. Conventional compatible learning methods (Shen et al. 2021; Budnik & Avrithis, 2020) regularize the feature compatibility by enfore- ing the new-to-old positive pairs to be more similar than new-to-old negative pairs. They did not consider the task of hot-refresh model upgrades thus overlooked the problem of negative flips. We take the sate-offthe-art BCT (Shen al., 2020) an example. The comparisons can he found in Figures [6 and 17 As it only focuses on the backfill-frre model upgrades, it cannot alleviate the negative flips, suffering from the problem of model regression. Note that BCT adopts classification loss as compatible constraints (via feeding the new features into the old classifier), where it is hard to explicitly integrate the regularization of new-to-new negative pairs into the form of BCT (classi- fication) loss. So we use our ingression-alleviatiin compatible loss on top of BCT loss, and achieve state-of-the-ar compatible performance at the same time alleviating model regression.
2020 Meng et al.
Compare to Knowledge Distillation. Although we both regularize the prediction consistency be- tween two different models, we have entirely different purposes. Specifically, distillation aims at training a new model that inherits the knowledge of the teacher (old) model in order to improve the overall accuracy of the new model, and the new model will be deployed individually during infer- ence. In contrast, our task requires cross-model retrieval via learning compatible features the same time reducing negative flips. Well noted that learning new features that are interoperable with old features not equal to inheriting the knowledge of old features. Moreover, we have different exper- imental setups. In knowledge distillation, the old model is generally stronger network with more powerful representation discrimination. In our model upgrading task, the old model is generally a poorer network with inferior representation discrimination. To verify that existing distillation algorithms are inapplicable for hot-refresh model upgrades of retrieval tasks, we compared to related work, namely Focal Distillation (FD) (ann et al.) 2020), which was specially designed for alleviating the model regression of classification tasks via knowledge distillation. As shown in Figures [6 and 17 directly employing Focal Distillation in our task achieves unsatisfactory retrieval performances and is hard to achieve the goal of model upgrades. It is actually due to the fact that it inherits the knowledge of the inferior old model too much, which limits its capability of learning discriminative new features.
Compare Continual Learning. Our work is also related continual learning (Li Hoiem Wang et al. 202T), which aims at training new model that inherits the capability of the old task. We mainly differ in two ways. (1) Application: Hot-refresh model upgrade requires feature compatibility for cross-modal retrieval, i.e. the query is embedded by the new model while the gallery is a mix of both new and old features. Continual learning requires the new model to re- member the capability of the old model task during training. however, the new model is individually deployed without the "interaction" with the old model. Moreover, since continual learning does not need to conduct cross-model deployment, there's no consideration of negative flips. (2) Purpose: Hot-refresh model upgrade requires the new model to outperform the old model and the new fea- tures to be interoperable with old features. Continual learning only requires the new model to inherit the old task knowledge, does not expect the new model to outperform the old model on the old task.
2017
To verify that existing continual learning algorithms are inapplicable for hot-refresh model upgrades of retrieval tasks, we compared to related work, namely Learning without Forget (LwF) (Li & Hoiem 2017). As shown in Figure [8] significant model regression and unsatisfactory retrieval performances are observed. Existing continual learning algorithms cannot tackle the model regres- sion problem of hot-refresh model upgrades.
20

---

Published as a conference paper at ICLR 2022
mAP of old model
OCT
mAP of FD
mAP of ours
MAP of ours+BCT
86-
16-
76-
64.28
Model regression
10
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
70
0%
25%
50%
75%
100%
(a) Open-data on GLDVZ
(b) Open-data on ROxford
(c) Open-data on RParls
Figure 16: Comparison with two related methods, BCT (Shen al.
2020) and Focal Distillation
(FD) Yan et al., 2020). The former one only regularizes the feature compatibility while ignoring model regression, and the latter one specially designed for model regression of classification tasks. We achieve the optimal performance by using our regression-alleviatnng compatible loss on top of BCT. Results of R50-R101 are reported terms of mAP(t).
79
66.69
61-
64.28
64.25
61-
55
0%
25%
50%
75%
100%
59
0%
25%
50%
75%
100%
55
0%
25%
50%
75%
100%
(a) Expansion ROxford
(b) Open-dats on ROxford
(c) Open-class on ROxford
88
u5
$3.60
82-
82:
83.27
82
79
75%
2516
50%
75%
76
25%
(d) Expansion on RParis
(e) Open-data on RParis
(f) Open-class on RParis
Figure 17: Comparison with two related methods, BCT (Shen et al. 2020) and Focal Distillation
(FD) Yan al. 2020). Results of R50-R50 are reported terms of mAP(1).
mAP of old model
mAP of lwf
mAP ours
16
88
84.76
14.08
68.50
13.24
* 83.27
86.32
64.38
67.29
$1.65
12
10.75
10.70
63.06
80.59
10
0%
25%
30%
75%
100%
0%
25%
50%
75%
100%
76
0%:
25%
50%
75%
100%
(a) Open-data on GLDv2
(b) Open-data on ROxford
(c) Open-data on RParis
Figure 18: Comparison with continual learning method LwF ( & Hoiem 2017) GLDv2-test,
ROxford and RParis. Results of R50-R50 are reported in terms of mAP()).
21


Reviews:
[{"review_id": "XtZgduFRsct", "reviewer": "Reviewer_6wRf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n1) The problem is very interesting and is not limited to social network applications. Compatible representations basically decouple learning from matching so that up-to-date features can always be used to match to the gallery-set. This may have several advantages especially in all those applications in which back-propagation is considered too slow to provide up to date features.\n2) Hot-refresh is a novel problem.\n\nWEAKNESSES:\n1) Contrastive loss seems to be a distillation loss because $\\phi_{\\rm old}$ is a fixed network. This should be better discussed.\n2) The paper seems to be divided into two independent parts. The first one is related to the combination of compatible learning with the reduction of negative flips, the second one is about refreshing the gallery. Their dependency seems to be only related because of the final application. This should be discussed.\n3) Contrastive learning usage is not motivated. For example, feature learning through a surrogate classification task is typically performing better than contrastive learning. This is also shown in the landmark dataset paper (Weyand et al., 2020), in which the cosface method is used. Moreover the contrastive effect of the loss seems to be limited being $\\phi_{\\rm old}$ a fixed network. This part is less developed with respect to the rest of the paper. Compatibility is also less evaluated.\n4) Sequential multi-model compatibility in upgrading to novel fresh data is not evaluated. This would have given the paper a lot more value. Even if sequential multimodal compatibility did not perform well, it would have been a great baseline to compare with hot-refresh model update. For example a five step sequential-upgrades with 20%, 40%, 60%, 80% and 100% of the data should be evaluated.\n5) A practical disadvantage of hot-refreshing through resampling is that the original image gallery has to be stored. Compatible learning typically does not require storing the original images. In many applications original images cannot be stored for privacy issues. \n6) The resampling strategy based on discriminative uncertainty seems to require multiple instances of the same classes in the gallery. This could be a strong assumption since the gallery is not a statistically meaningful set like for example the test-set in classification tasks.", "summary_of_the_paper": "The paper presents a method to learn compatible features with low negative flips rate. The paper also introduces a resampling strategy to gradually replace the gallery to ameliorate the performance. Resampling is based on feature uncertainty. The less discriminative a feature is, the sooner it is replaced.", "summary_of_the_review": "The paper introduces a very nice problem with some technical details to be clarified. The reviewer's opinion is that paper is not \"uniformly\" developed, many technical parts would have required more attention.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UUlrbjruUCb", "reviewer": "Reviewer_JYkL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Strengths_\n\n- The presentation of the paper is very good. The paper is well written and the method has been properly motivated. The quality of the figures is also very good and they help a lot to understand the problem, the method, and the experiments.\n- It addresses an interesting problem that has real-world application in industrial image retrieval applications \n\n---\n_Weaknesses_\n\n- Although it addresses a novel problem in image retrieval, I find the novelty of the method itself somehow limited. It's very similar to other methods that sample pairs of features coming from the new and the old model, being the main contribution in this case that new-to-new negative pairs are also considered.\n- I have a couple of comments related to the fact that the authors decided not to use new-to-new positive pairs in the \"regression-free compatible loss\" (Lcomp and Lrf-comp). First of all, what I understand from the main paper is that they decided not to do it because \"it is hard to make sure intra-class instances exist in the same mini-batch without a pre-sampler\", however, adding this pre-sampler is actually easy and trivial, hence it is doable and not a reason why not to consider it in the first place. Secondly, the authors actually include an experiment in the appendix where they introduce a positive sampler and they show that similar results can be obtained by either using it or not. My suggestion here it is two fold: first I would suggest the authors to reference this experiment in the main paper to strength the decision of not including positive pairs. Second, I wondered if the reason of not seeing an improvement when sampling positives pairs in the batch it has to do with the fact that they're seen as negatives (?) in the second term of the denominator in Lrf-comp (k \\in B/x). Could other losses be more suitable for this scenario, eg. a triplet loss where several triplets combining positives and negatives from \"new\" and \"old\" are considered at the same time?\n- I found Figure 6 a bit difficult to \"read\" and to draw conclusions from it. Maybe showing an average per dataset or protocol would be better? Not sure about this though.", "summary_of_the_paper": "The paper addresses the task of hot-refresh model upgrade (ie. the task of deploying a new model into production while the gallery features from an older are still being re-indexed) in retrieval systems and studies the problem of model regression that happens when the similarity of new-feature-to-old-feature positive pairs is lower than new-feature-to-new-feature negative pairs. To solve this the authors propose a compatible training method that takes these flips into account, encouraging the new-to-old pair positives to be more similar than both new-to-old and new-to-new negative pairs (this is nicely illustrated in Figure 2). The paper also introduces an uncertainty-based backfilling (ie. the process of re-indexing the old gallery features with the new model) strategy that tries to follow a poor-first policy using the probability distribution of the training classes yielded by the new classifier.", "summary_of_the_review": "Even though I consider the technical contribution a bit limited I positively acknowledge the fact that they're the fist to study the problem of hot-refresh model upgrade in image retrieval in particular and the problem of model regression.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8pfpBtSZXfI", "reviewer": "Reviewer_QLFh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nS1) First to study the impact of such “hot-refresh” techniques to the image retrieval domain, relying on standard datasets in this area.\nS2) The observation about the reason behind negative flips is novel and interesting – it had not been discussed in previous work. The proposed training method is well-motivated from this observation.\nS3) The proposed training method achieves improved performance in the hot-refresh application compared to previous work, reducing the penalty incurred by negative flips.\n\nWeaknesses:\nW1) The uncertainty-based backfilling approach does not consistently show performance improvements. I am not convinced by this part. In some cases, random backfilling even performs better. The proposed uncertainty-based backfilling method overall feels simplistic.\nW2) Writing could be improved. For example:\nPage 4, “We figure out the reason for the negative flips (...)” – the writing here is a little informal. This part could also be rewritten for more clarity / better explanation. It took me some time to parse why this was the only reason for negative flips – this should be explained better.\nPage 8, in section 5.2. The paragraph mentions the figures, but does not discuss the results shown in them at all.\nW3) The method still shows room for improvement. For example, the lack of pre-sampler to include positive image pairs in the same batch probably hinders performance (the authors justify the lack of that due to it being “hard”, although this is common in deep metric learning, eg for the triplet loss).\n\nMinor details:\nM1) Page 5, below Eq 7: “denotes the cross-entropy” — I think the authors here mean softmax + cross-entropy, right?\nM2) The Oxford and Paris dataset versions used are more commonly referred to as Revisited Oxford (ROxford) and Revisited Paris (RParis). The renaming would make the dataset usage more clear for the readers.\n", "summary_of_the_paper": "The paper proposes a new technique for updating image retrieval models while not requiring full backfill of the index in order to allow deployment. This has the potential to make model upgrades more practical for real-world large-scale applications. The paper is the first to study the impact of such “hot-refresh” techniques to the image retrieval domain. The authors make observations about the reason behind negative flips, which motivate their newly-proposed loss function / training technique. The paper also proposes an uncertainty-based backfilling method to decide the ordering in which images in the index are backfilled. Experiments are conducted on standard image retrieval datasets.", "summary_of_the_review": "This is an interesting paper, targeting an important application, proposing a novel observation and method to improve hot-refresh image retrieval. However, some aspects of the paper are not well-developed (see weaknesses listed above). Due to the latter, my rating is “marginally above the acceptance threshold”.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1JQJlRsaJgP", "reviewer": "Reviewer_Fu6z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nFirst of all, this paper is mostly well-written and well-motivated, even though there are some parts where the readability could be further improved, especially in terms of definition and references on key terms/ideas such as “backfilling” “cold/hot-refresh”, “negative flip” etc. However, this does not impact the overall readability of the paper too much in my opinion.\n\nFor the results, I think they are clearly presented and link to the contributions well, as demonstrated in Figures 3-7 which shows that the new losses and uncertainty-driven backfilling indeed helping the hot-refresh model update process significantly. I particularly appreciate the small figures in Figures 3-5, which shows the progress in NFT v.s. vanilla, as it ties back to the observation of negative flips causing the model regression problem as a main motivation for this work.\n\n## Weaknesses\nFirstly, in terms of novelty, I think this work is ok as neither of these techniques have been used for model updates in image retrieval to the best of my knowledge. However, for uncertainty based backfilling, the authors have also mentioned that the techniques are based on Settles, 2009, which is on active learning. \n\nThe other main contribution is in the Regression-Free Compatibility Regularization (RFCT) in Eq. 6, as mentioned above, alleviates negative flips by diluting the significance of both new-to-old and new-to-new negative pairs in the contrastive loss. This is where I find the contribution could have much more potential, as this simple addition to the loss could have a lot more variety resulting in much more in-depth analysis of the impact of this term - e.g. changing the ratio between the two terms in the loss? Or perhaps different temperatures in each of them? Or even some dynamic tuning of these hyperparameters according to the uncertainty in the backfilling process, which ties the two novelties together. I think there is so much more that can be done to strengthen this novelty, but unfortunately is not presented in this work.\n\nAnother comment I have is the phrase “Regression-free”. If I understand correctly from the authors’ definition, the results in Figure 7 shows that this work is only regression free on the GLDv2 dataset (which the vanilla baseline is already close to regression free), and there is still significant model-regression observed in Oxford5k and Paris6k. I think in this case, claiming the main contribution of this work as “regression-free” is slightly misleading and I would suggest toning down the name to something less strong, e.g. “regression-alleviating”.\n\nLast but not least, not really a weakness here but just a general question, as the authors also took inspiration from active learning, I am wondering how does “hot-fresh” model upgrades compare against other techniques such as continual learning? For example, *Continual Learning for Image-Based Camera Localization* (Wang et. al 2021) also tackles the problem of learning gradually on unseen data, but on the problem of visual localization (which is still quite related to image retrieval). As I am not an expert on active learning / model upgrades, it would be great if the authors could clarify this.\n", "summary_of_the_paper": "This paper tackles the problem of model upgrades in large-scale training of image retrieval models. It proposes a new framework of “hot-refresh” updates with two main techniques: \n1) by alleviating “negative flip” by a regression-free regularization term in the loss and \n2) by prioritizing gallery images with the highest uncertainties during the backfilling process.\n\nThe results suggest improvement across three major retrieval benchmarks compared against baseline model-upgrade methods.", "summary_of_the_review": "Overall, this paper is a good read and does provide useful insights to the proposed problem and potential remedies. \nHowever, I think this works is just so slightly short of passing my threshold for acceptance due to the lack of exploring more on its main contribution, which is the RFCT loss, and potentially giving more valuable information to the community on the synergies between old and new features in the model upgrade process. \nMoreover, the slight overclaim in the \"Regression-Free\" naming requires some extra attention. \nTherefore, I am recommending \"5: marginally below the acceptance threshold\" now, but this decision is marginal and I am willing to consider boosting the score if the authors address my concerns well in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is proposed to deeply investigate the hot-refresh model upgrades of image retrieval systems. The hot-refresh model is very useful since the model can be quickly updated after the gallery is backfilled. To address the model regression with negative flips, this paper introduces a Regression-Alleviating Compatible Training (RACT) method by reducing negative flips. The proposed method has been verified on the large-scale image retrieval benchmark such as Google Landmark. The key contribution of this paper is the new setting targeting an important application in real-world image retrieval systems. However, some of the technical details are not fully explained. Despite these minor concerns, the AC will rate this paper as a poster acceptance based on the overall contributions.
Decision: Accept (Poster)