FastFill: Efficient Compatible Model Update
Authors: Florian Jaeckle
Fartash Faghri
Ali Farhadi
Oncel Tuzel
Hadi Pouransari
Venue: ICLR 2023 poster (2023)
Keywords: Compatible Representation Learning
Image Retrieval
Model Regression

Abstract: In many retrieval systems the original high dimensional data (e.g., images) is mapped to a lower dimensional feature through a learned embedding model. The task of retrieving the most similar data from a gallery set to a given query data is performed through similarity comparison on features. When the embedding model is updated, it might produce features that are not comparable/compatible with features already in the gallery computed with the old model. Subsequently, all features in the gallery need to be re-computed using the new embedding model -- a computationally expensive process called backfilling. Recently, compatible representation learning methods have been proposed to avoid back-filling. Despite their relative success, there is an inherent trade-off between new model performance and its compatibility with the old model. In this work, we introduce FastFill: a compatible model update process using feature alignment and policy based partial backfilling to promptly elevate retrieval performance. We show that previous backfilling strategies suffer from decreased performance and demonstrate the importance of both the training objective and the ordering in online partial backfilling. We propose a new training method for feature alignment between old and new embedding models using uncertainty estimation. Compared to previous works, we obtain significantly improved backfilling results on a variety of datasets: mAP on ImageNet (+4.4%), Places-365 (+2.7%), and VGG-Face2 (+1.3%). Further, we demonstrate that when updating a biased model with FastFill, the minority subgroup accuracy gap promptly vanishes with a small fraction of partial backfilling.

Published as a conference paper at ICLR 2023
FASTFILL: EFFICIENT COMPATIBLE MODEL UPDATE
Florian Jaeckle"
Fartash Faghri Apple
Ali Farhadi Apple
Oncel Tuzel Apple
University of Oxford
Hadi Pouransari' Apple
ABSTRACT
many retrieval systems the original high dimensional data (e.g., images) is mapped to lower dimensional feature through learned embedding model. The task retrieving the most similar data from a gallery set to a given query data is performed through a similarity comparison on features. When the embedding model is updated, it might produce features that are not comparable/ ccm.ptible with features already in the gallery computed with the old model. Subsequently, all features in the gallery need to be re-computed using the new embedding model computationally expensive process called backfilling. Recently, compatible representation learning methods have been proposed to avoid backfilling. Despite their relative success, there is an inherent trade-off between the new model performance and its compatibility with the old model. In this work, we introduce Fast a compatible model update process using feature alignment, and policy based partial backfllingg to promptly elevate retrieval performance. We show that previous backfilling strategies suffer from decreased performance and demonstrate the importance both the training objective and the ordering in online partial backfilling. We propose new training method for feature a alignment between old and new embedding models using uncertainty est estimtiio. Compared to previous works, we obtain significantly improved backfilling results on a variety of datasets: mAP on ImageNet (+4.4%), Places-365 (+2.7%), and VGG-Face2 (+1.3%). Further, we demonstrate that when updating biased model with FastFill, the minority subgroup accuracy gap promptly vanishes with small fraction of partial backfilling"
INTRODUCTION
Retrieval problems have become increasingly popular for many real-life applications such as face recognition, voice recognition, image localization, and object identification. In an image retrieval setup, we have large set of images called the gallery set with predicted labels and a set of unknown query images. The aim of image retrieval is to match query images to related images in the gallery set, ideally of the same class/ /identtt.. In practice, use www-dimensional feature vectors generated by learned embedding model instead of the original high dimensional images to perform retrieval.
When we get access to more or better training data, model architectures, or training regimes we want to update the embodding model to improve the performance of the downstream retrieval task. However, different neural networks rarely learn to generate compatible features even when they have been trained on the same dataset, with the same optimization method, and have the same architecture (Li et al., 2015). Hence, computing the query features with new embedding model, whilst keeping the old gallery features, leads to poor retrieval results due to incompatibility of old and new embedding models (Shen et al., 2020).
"Corr: florian. carranjjacckeeigmaiii
& mpouransari@apple.comm
Work completed during internship at Apple.
Code available at https: wwtpaaww.ww..ppp/a/aaaaaeeaaaeee

---

Published as a conference paper at ICLR 2023
ImageNet--Hl 1k Retrieval CMC- Topl
Model Updaze
FastFill lour)
68.276
SOTAL
Bekee punail beckflling sabfiling
FCT (previous
46,6%
model updaue
Old Scatoree
Time
Old model: in use
Rackfilling
New model
Now fcuuuee
progress
in use
Figure 1: Left: Old and new features for a binary fruits vs. animals classification sctup. Old and new features are somewhat compatible, but have few mismatches shown by red crosses (top). We can improve retrieval performance by backfilling some of the old features to higher quality new leatures in a specific order (bottom). In a partial backfilling scenario the accuracy improvement depends on the order of backfilling. Right: Retrieval performance for ImageNet as a function of time when updating the embedding model with different backfilling strategies. A backfilling curve reaching the new model performance faster is better.
The vanilla. solution is to replace the features in the gallery set that have been generated by the old model with fcatures from the new model. This computationally expensive process carry out backfilling offfine: we carry on using the old
is called backfilling. In practice, gallery features and the old model for queries whilst computing new gallery features with the new model in the background (see Figure 1-right). We only use the new gallery features once we' 'e finished updating the entire set. However, for large scale systems this process is computationally expensive and can take months. In many real world systems, the cost of backfilling is a blocker for model updates, despite the availability of a more accurate model.
This has led rise to the study of compatible representation learning. Shen et al. (2020); Budnik & Avrithis (2021); Zhang et al. (2021); Ramanujan et al. (2022); Hu et al. (2022); Zhao et al. (2022); Duggal al. (2021) all proposed methods to update the embedding model to achieve better performance whilst still being compatible with features generated by the old model (see Figure 1-left-top). Despite relative success, compatibility learning is not perfect: performing retrieval with mixture of old and new features achieves lower accuracies than when we replace all the old features with new ones. In this work, we focus on closing this performance gap. Further, some previous methods degrade the new model performance when trying to make it more compatible with the old model (Shen et al., 2020; Hu et al., 2022) or requiring availability of side-information extra features from separate self-supervised model, (Ramanujan et al., 2022) (which may not be available for an existing system). We relax both constraints in this work.
To benefit from the more accurate embedding model sooner and choaper, we can backfill some or all of the images in an online continuous fashion: We run downstream retrieval tasks on a partially backfilled gallery set where part of the features are computed with the old model, and part of them with the new model (see Figure 1-left-bottom).
In practice, we can consider two scenarios: 1) random order in an extended period of time and want to maximize the average performance during the backfilling process (see Figure 1-right); 2) we have a fixed partial backfilling budget and want to reach optimal retrieval performance after backfilling the allowed number of images (e.g., we can backfill only 10% of the gallery). In both cascs, we want to reach the highest possible performance by backfilling the fewest images possible. We demonstrate that the training objective well as the order by which we backfill images are both crucial. In fact, if we use training losses proposed in the literature and choose a random ordering we may even reduce performance before due to the occurrence of 'negative flips' (Zhang et al., 2021) images that where classified correctly when using the old model but misclassified when using the new one.
we will backfill the entire gallery set with
improvement (see FCT-Random in Figure 2a)
2

---

Published as a conference paper at ICLR 2023
In this work, we propose FastFill: efficient compatible representation learning and online backfilling method resulting in prompt retrieval accuracy boosts as shown in Figure 1-right. In FastFill, we learn computationally efficient holistic transformation to align feature vectors from the old model embedding space to that of new model. The new model is trained independently without sacrificing its performance for compatibility. We propose a new training loss that contains point-to-point and point-to-cluster objectives. We train the transformation model with the proposed loss using uncertainty estimation and obtain significantly improved partial backfilling performance compared to previous works.
We summarize our main contribution below:
We propose FastFill, a new compatible training method with improved retrieval performance at all levels of partial backfilling compared to previous works.
FastFill uses a novel training objective and a new policy to order samples to be backfilled which show to be critical for improved performance.
We demonstrate the application of FastFill to update biased embedding models with minimum compute, crucial to efficiently fix large-scale biased retrieval systems.
On variety of datasets we demonstrate FastFill obtains state-of-the-arl results: mAP on ImageNet (+4.4%), Places-365 (+2.7%), and VGG-Face2 (+1.3%).
PROBLEM SETUP
Image Retrieval an image retrieval system, we have a set of images called the gallery set G with predicted labels, which can be separated into k different classes or clusters. At inference time, we further have a query set Q. For each image in the query set we aim to retrieve an image from the gallery set of the same class. In an embedding based retrieval system, we replace the original D-dimensional data (3 x x w for images) in the gallory and query sets with features generated by trained embedding model @ R + Rd where < D. The model Φ is trained on images æ E 'D a disjoint dalaset from both G and Q. For query image By E 2, the retrieval algorithm returns image æ; from the gallery satisfying:
(single-model retrieval)
arg min (o(())) o(x,)),
(1)
0,
for some distance function D (e.g., l2 cosine distance). In this setup only need to compute the features Φ(œ) of the gallery set once and store them for future queries. We then no longer need be able to access the original images of the gallery. As d D, replacing the images with their features greatly reduces the memory and computational requirement to maintain the gallery and perform retrieval, crucial for private on-device retrieval. Depending on the context, by gallery we may refer to the original images or the corresponding features.
Learning Compatible Representations. For real world applications, we aim to update the embedding model every few months to improve its accuracy 1) to adapt to changes in the world, e.g., supporting people with face mask for a face recognition system, 2) on under- represented subgroups, e.g., new ethnicity, sensors, by adding more data to the training set, or 3) by using enhanced architecture and training optimization. We denote the old and new embedding models by Pold and &new and the training datasets of the two models by Dota and D Dnew, respectively. After a model update, for every new query image we Onew compute the feature, and hence the retrieval task turns in to:
lighting condition, different data capture
(cross-model retrieval)
2; = arg min D(oota(()))( Onew (xq)).
æ,, Eg
RELATED WORK
As Li et al. (2015) realised, different models do not always generate compatible features, even when they are trained on the same dataset. This observation has motivated the rise of the relatively new literature model compatibility: given weaker old model dotd aim
3

---

Published as a conference paper at ICLR 2023
to train new model Onew that has better retrieval performance, as in (1), whilst still is compatible with the old model, as in (2). This way we can improve performance, without having to wait months and incur large computational cost to backfill the gallery. Shen et al. (2020) introduced Backwards Compatible Training (BCT). They add an influence loss term to the new model objective that runs the old model linear classifier generated by the new model to encourage compatibility, Budnik & Avrithis (2021) train a new model by formulating backward-compatible regularization as a contrastive loss. Zhang et al. (2021) proposed a Regression-Alleviae Alleviating Compatible Training (RACT) method that adds a regularization term to the contrastive loss in order to roduce negative flips: these are images that the old model classified correctly but the new model gets wrong. Ramanujan et al. (2022) proposed the Forward Compatible Training (FCT) method that trains an independent new model first and subsequently trains a transformation function which maps features from old model embedding space to that of the new model. Moreover, the transformation requires some oade-information about each data point as input. Iscen et al. (2020), Wang et al. (2020) and Su et al. (2022) also approach the model compatibility problem by learning maps between old and new model embodding spaces. Note that, discussed in Ramanujan et al. (2022) the cost of applying the transformation is negligible compared to running the embedding model on images (hence, the inslant jump in performance in Figure 1-right).
the features
Learning Compatible Predictions. A related line of research focuses on learning compatible predictions (as opposed to compatible features). For the classification task, Srivastava et al. (2020) show that changing the random seed can lead to high prediction incompatibility even when the rest of the learning setup stays constant. Moreover, they observed incompatible points do not always have low confidence scores. Oesterling & Sapiro (2021) aim to achieve better prediction compatible models by ensuring that the new model training loss is smaller than the old model loss on all training samples. Träuble et al. (2021) consider the problem of assigning labels to large unlabelled dataset. In particular, they focus on how to determine which points to re-classify when given new model and how to proceed if the old and new predictions differ from each other. Similar to compatible representation learning, various works in the literature focus on positive-congruent training which aims to reduce the number of negative flips in the prediction setting. Yan et al. (2021) achiove this by focal distillation which puts greater emphasis on samples that the old model classifies correctly. Zhao et al. (2022) argue that most negative flips are caused by logits with large margins and thus try to reduce large logit displacement.
Continual Learning (CL) and Knowledge Distillation (KD). Other less relevant areas of research that focus on learning across domains and tasks include CL and KD. CL (Parisi ct al., 2019; Li & Hoiem, 2017; Zenke ct al., 2017) is related to compatible learning as it also aims to train new model that should keep the knowledge acquired by the old one. However, in the continual learning setup the new model is dealing with a new task, whereas here the task (image retrieval) is fixed. Also, in CL the two models never interact as they are not employed together: as soon as the new model is trained we discard the old one.
In KD (Hinton et al., 2015; Gou et al., 2021) we also train a new model (the student) that inherits some of the capabilities of an old model (the teacher). However, there are two main differences model compatibility sctup. Firstly, KD the teacher model often larger and stronger model than the student, reverse of the setup in model compatibility. Further, the teacher and the student are never used together in practice. Therefore, the features of the two models do not need to be compatible in the KD setting. Zhang et al. (2021) show that neither CL methods nor KD ones are able to achieve good retrieval performance in the model update setting as the features of the old and new models are not compatible.
4 METHOD
4.1 FEATURE ALIGNMENT
Some previous works aim to achieve model compatibility when training the new model, by directly enforcing new features to be aligned to the existing old model features (Shen et al., 2020; Budnik & Avrithis, 2021; Zhang et al., 2021). However, this often greatly reduces the
4

---

Published as a conference paper at ICLR 2023
single model retrieval performance defined in Eq. (1) for the new model, they suffer from the inherent trade-off between maximizing the new model performance and at the same time its compatibility to the weaker old model. Other works have thus proposed learning a separate mapping from old model embedding space to new model embedding space to align their features (Wang et al., 2020; Meng et al., 2021; Ramanujan et al., 2022). Wang et al. (2020) and Meng et al. (2021) both still modify the training of the new model which is undesirable because it can limit the performance gain of the new model and is intrusive and complicated for practitioners. Ramanujan et al. (2022) train the new model and alignment map independently, but to achieve better alignment rely on existence of iddeinffrmatiio (an extra feature generated by an unsupervised model computed for each data point). This is also undesirable because is not applicable to an already existing system without side-information available. We provide results in the presence of such side-information in the Appendix B.2. Here, as a baseline we consider the same feature alignment as in Ramanujan et al. (2022), but without indommnformttio:: new model feature space modcled by an MLP architccture with paramcters 0. Hence, for cross-model retrieval we use ho (z)t instead of @ate(z;) in Eq. (2).
learn a map ha IR + Rd from old model feature space to
4.2 FASTFILL
Despite relative success, all previous methods have limitations and fail to reach full- compatibility: in each case cross-model retrieval defined in Eq. (2) has significantly worse performance than single-model retrieval defined in Eq. (1) with the new model. In order to bridge this gap, (partial) backfilling is required: continuously move from cross model retrieval to single model retrieval by updating the old gallery features with the new model. That is to replace transformed old features ha (x:))( with new features &new (2:) for some or all images in the gallery set (z, E G) (Figure 1); To this end, we propose our new method FistFill, which includes two main contributions: firstly, we change the backfilling curve behaviour for a random policy by modifying the alignment model training and secondly, we introduce better policy that produces the order of samples to update, therefore results in promptly closing the gap between cross-model and single-model retrieval.
We propose a training loss for the aligmment map h to have both high retrieval performance at no backfilling (the instant jump after model update in Figure 1-right) and good backfilling curve, ensuring that we close the gap with new model performance promptly. To achieve this, we include objectives to encourage point-to-poin and point-to-cluster alignments.
The first part of our training loss is the pairwise squared distance between the new features and the transformed old features: Li, (æ:; ho, Pold, Pnew) = [|@ (z:) ho ho()ota(==)) for x E Dreww This is the loss used by FCT (Ramamujn ct al., 2022). Further, given Potd and the new model classifier head, Knew, that the new model has been trained on, we use a discrininative loss that runs Knew on the transformed feat Ldise( y; hgg &atd, Knew) where (z, y) E Dnew a pair of (image, label). Lditc is the same discriminative loss that new model is trained on: in our experiments it is the Cross Entropy loss with Label Smoothing (Szegedy et al., 2016) for ImageNet- 1k and Places-365 datasets, and the ArcFace loss (Deng et al., 2019) for VGGFace2 dataset. This loss can be thought of as the reverse of the influence loss in BCT (Shen et al., 2020) but with one major advantage. Unlike BCT, we do not require old and new models to have been trained on the same classes we are using the new model classifier head rather than the old one. Our proposed alignment loss is combination of the two losses:
Ligt (ho; Polds Pnew, Frneu; R. = Lig + Laise-
fi2 achieves pairwise compatibility, aligning each transformed old feature with the corre- sponding new feature. However, in practice we camnot obtain perfect alignment on unseen data in the query and gallery sets. With Lt2 we encourage to reduce the magnitude of pair-wise alignment error (new();) encourages the error vectors to be oriented towards the same-class clusters of the new model (given by Knew) and away from wrong clusters. Hence, we avoid proximity of cluster of transformed old features to a wrong cluster of new features the mechanism responsible of slow improvements in backfilling.
ho (o)oaa(();) but the direction of error is free. Ldise
5

---

Published as a conference paper at ICLR 2023
We now focus coming up with good ordering to backfill the images in the gallery set (replacing ho (o)ou((); with (x, Finding the optimal ordering for backfilling is hard combinatorial problem and infeasible. We first look at heuristically good (but cheating) ordering: backfill the worst gallery features first. As minimizing the aligmment loss (3) leads to good transformation function,
use it as way measure the quality of the gallery features: we compute the training loss (3) for each image in the gallery set and sort the images in decreasing order. In other words, we backfill the gallery images with the highest loss first as they are probably not very compatible with the new model. We call this method FastFill- FastFil-CChcating. We show in Figure 2a that using Fast File-Chhaiiing results in significantly better backfilling curve than when we backfill using a random ordering.
Unfortunately, we cannot use this backfilling method in practice for images in the gallery as we do not have access to (a;) until we have backfilled the ith image. Instead, we aim to estimate this ordering borrowing ideas from Bayesian Deep Learning. When the training objective is Lig, can model the alignment error by multivariate Gaussian distribution with al covariance matrix", and reduce the negative log likelihood during training. This is similar to the method proposed by Kendall & Gal (2017). Given the old and new models, we train a model dy Rd + R parameterized by that takes the old model feature h(dola as input and estimate logo2. We can then jointly train ho and the by minimizing:
[Io(()aa(a)) - Dnew (x)
1 X
log
(4)
where log (ho(dol(())) and À is hyper- hhper--aranen parameter function of d. Now, we extend this formulation for our multi-objective training in (3). We follow the approximation in Kendall et al. (2018) for uncortainty cstimation in a multi-task sctup where the model has both discrete and continuous output. Further, assunne the same covariance parameter, a², for both of our regression and discriminative objectives (L12 and Lasse). Hence, we can replace Cig in (4) with Lla+disc and get the full FastFill training loss:
C(ha, tho; Gold, Dnews Knew D Dnew) = ((m..))-Dnew Llatdisc
1
log logoo
(5)
0²
In practice, we use a shared backbone for ha and vo. therefore the overhead is minimal. Please see Appendix A.2 for details of architecture, hyper-parameters, and training sctup. In Figure 2b we show that the predicted of by the trained with (5) is good approximation of (1)+diso((:)) for a; E G. In particular, the ordering implied by Liz+dise and a are close. In Figure 2c we show the order of each æ; in the gallery when sorted by (ig+ is well correlated with its order when sorted by of: The Kendall- Tau correlation between two orderings is 0.67 on the ImageNet test set. In FastFill, backfill x,'s 'ss with the ordering implied by their predicted a²'s (from high to low values).
Finally, in Figure 2a we show that FastFill obtains similar backfilling curve the cheating setup using the ordering implicd by cheaply computed of's. Also note that, FastFill even with random ordering obtains significant improvement compared to bascline (FCT with random ordering) due to superiority of our proposed alignment loss. In Section 6 we further demonstrate the importance of our proposed training objective and backfilling ordering.
5 EXPERIMENTS
We now analyse the performance of our method by comparing it against different basclines on variety of datasets. We first describe the metrics we use to evaluate and compare different. compatibility and backfilling methods.
5.1 EVALUATION METRICS
Compatibility Metrics Similar to previous model compatibility works we mainly use following two metrics to evaluate the performance and compatibility of modols. The Cumu- lative Matching Characteristics (CMC) evaluates the top-k retrieval accuracy. We compute
We also considered arbitrary diagonal covariance matrix instead of a scaled identity matrix, and observed no significant improvement.
6

---

Published as a conference paper at ICLR 2023
13
3.0
50000
0000
2.5
11
Fanfiil
20000
Order of 2.
BeadretFCT) Rardon
10000
40-
1.0
20000 40000
Sodtikny
log o
Order of
(b)
(c)
a) Backfilling results for different training objectives and backfilling orderings evaluated on the ImageNet-Iie 1k dataset. The FastFill training objective, defined in Equ. (5), compared to our baseline method (FCT) obtains significantly improved mAP over the course of backfilling when a random ordering usod. The performance is further improved when using the ordering implied by the predicted uncertainties oî. We also compare against WhstFil-Cheating
Figure 2:
which the ordering obtained by computing the training loss (3) on gallery images (hence chealing setup), and show comparable performance. b) We show that the predicted log og the output of thy on gallery images, and log Lig+diin( are well correlated. c) We sort 0i once by of and once by Lty-disc (a;) to get two orderings. Here, for each æ; we plot its order the first ordering vs its order in the second ordering, and observe great correlation (Kendall-Tan correlation-0.67).
the distance between a single query feature and every single gallery features. We then return the i gallery features with the smallest distance. If at least one of the returned gallery images has the same label as the query we consider the retrioval to be successful. We also compule the mean Average Precision (mAP) score, which calculates the area under the precision-recall curve over recall values between 0.0 and 1.0.
Given metric M (either top-k CMC mAP), gallery set 9, and query set Q denote the retrieval performance by (G, Q). After a model update, we use &new to compute features for the query set and Pold to compute those for the gallery set. Similar to previous works, for retricval metrics, we use the full validation/ /tsst set of cach dataset as both the query set and gallery set. When we perform query with an image, we remove it from the gallery set to avoid trivial match.
Backfilling Metrics In partial backfilling we replace some of the old features, here (())) from the gallery with new features, Pnew((;;) In order to evaluate the retrieval performance for a partially backfilled gallery we introduce the following notation: given an ordering Tm...... TT. of the images in the gallery set and a backfilling fraction a E [0, 1], we define Gn, to be the partially backfilled gallery set. We use the new model to compute features for 2, E T|: na] and the old model for the remaining images, that is for all æ; E n v(n, :]. using numpy array notation, where n, = [am]] and n is the size of gallery.
Given metric M, to compare different backfilling and model update strategies during the entire course of model update, we propose the following averaged quantity:
M Q) = Ea- Mo Q)
(6)
M can be any retrieval metric such top-k CMC or mAP defined above. The backfilling metric M correspouds to the area under the backfilling curve for evaluation with M.
5.2 BACKFILLING EXPERIMENTS
We now analyse our method by comparing it to three baselines: BCT, RACT, and FCT. For BCT and FCT we used the official implementation provided by the authors. We re- implemented RACT and optimized over the available at the time of writing. We ran all experiments five times with different random seeds and report the averaged results. We use the la distance for image retrieval.
as the code was not publicly
7

---

Published as a conference paper at ICLR 2023
Feafill
FCT Qway Galey (NuA))
RACT Dany 6057
BCT Quary A/N ley R.M
Oel Medl
Quary Bei
Qsery Oar
Gollory AecAcl
Gallery: Our
Table ImageNet- 1k
Method
top1
m.P
BCT
55.92
37.02
(MC Top
RACT
07-
m03P
FCT
62.80
40.47
Fast
66.49
44.84
Buckiling
Badefllngg
Table 2: VGGFace2
70
CMC Top (15)
60
Method
topl
mAP
mAp
50
RACT
91.01
45.65
85
FCT
95.87
70.26
40
FastFill
96.35
71.57
80
30
20
40
80
100
20
40
80
100
Backfilling (9))
Backfilling (%)
Table Places-363
30-
Method
mAP
CMC 1op
34-
BCT
33.3
15.15
FCT
35.73
16.81
Fast.illl
38.32
19.48
37
Bockiiling
Bockkliin 151
Figure 3: Backfilling results on ImageNet-1k (top), VGGFace2 (middle), and Places-365 (bottom). We use new model features for the query set. For the gallery set we start off using (transformed) old features and incrementally replace them with new features.
ImageNet-lk. We compare FastFill against BCT, RACT, and FCT, three state-of-the-art methods. For RACT backfilling we use random orders. FastFill outperforms all three methods by over 3.5% for CMC-topl and over 4% for mAP (Figure 3, Table 1). Compared to BCT and RACT, FastFill performs significantly better for 0% backfilling (+10%) due to the transformation function. Furthermore, the sigma-based ordering significantly outperforms the random orderings and the cross-entropy one from RACT. We show in Appendix D that FastFill both increases the number of positive flips and decreases the negative flips compared to FCT.
antrop--backilling: since BCT and FCT do not offer policy for
VGGFace2. We show results on the VGGFace2 dataset in Table 2 and Figure 3. FastFill improves the baselines as well as an ablation that combines entropy-basce backfilling with the FastFill transformation by 0.5% CMC-topl and 1.3% mAP. The absolute improvement is relatively small as the gap between 0% and 100% backfilling is not very large, as the transformation already achieves high compatibility.
Places-365. We run experiments on the Places-365 dataset. The results are summarized in Table and Figure FastFill beats FCT with random backfilling by about 2.7% for both metrics. We further report CMC-top5 results in Appendix B.3 for all three datasets.
5.3 UPDATING BIASED MODELS
Embedding models can be biased toward certain sub-groups in their input domain, crucial problem in image retrieval. For example, in face recognition tasks the model

---

Published as a conference paper at ICLR 2023
can be biased toward certain ethnicity, gender, face attribute. It is often due to imbalanced training dataset. Here, we show FastFill can fix model biases efficiently without requiring expensive full backfilling. We train biased old model the VGGFace2- Gender-550 dataset that consists of 50 female and 500 males classes The old model performs significantly better on males identities (90.6%) in the test set than females (68.7%).
We train a new model on a larger gender-balance dataset that consists of 2183 female and 2183 male identities. The new model still has small bias towards male classes (98.33 3% vs 97.1%). Applying the feature aligument trained with FastFill alleviales some of the biases reducing the accuracy gap belween genders to 9.4%. As we show in Figure 4, with only 25% back- filling FastFill reaches the same accuracy gap between genders the new model, whereas random backfilling using FCT requires backfilling the entire gallery set to reduce the bias to the level of the new model. FastFill prioritizes backfilling instances corresponding to female identities which results in a fast reduction in the gender gap compared to previous methods (see Appendix F).
Males Fomales 85
FCT FastFill
8-
Bias Gap (CMC-top1]
6-
Meles Bh Femsles 94.9%
Males 50,35 Femoles 97.1%
25
50
75
100
Backfilling (%)
Figure 4
6 ABLATION STUDY
Here, we perform an ablation study to highlight the different components of our method (Figure 5). We first compare the effect of different training losses (C12, Lditcs tisec with and without uncertainty) when using random backfilling orderings. We show that fis+ greatly improves the backfilling curve. Training with uncertainty has little effect random backfillin. We also show that using a for the backfilling ordering greatly improves performance. We conclude that FastFill (training with +ig+ and uncertainty) obtains the best results for both random and o-based orderings. We show in Appendices C.3 and C.4 that FastFill particularly strong when we encounter new classes at inference time as the relative confidence (used in RACT) between samples from new classes are not as informative.
7 CONCLUSION
Currandom (40.47%)
Model compatibility is a crucial problem in many large-scale retrieval systems and common blocker to update embedding models. In this work, we propose FastFill: an online model update strategy to efficiently close the accuracy gap of previous model compatibility works.
Cands--random (41.95%
(Ei uncert.) (annertt)-ranndon I-rando (40.08%)
( uncertt. suncert))randeee rraadom [42 05%)
(6, Funcert. ) J-sgga (43.61%
(C), die Huncert, siiga (44 84%)
The key idea in FastFill is to perform partial backfill- ing: using a new model we recompute small fraction of the images from a large gallery set. To make the partial backfilling process efficient we propose a new training objective for the alignment model which maps the old model features to those of the new model, and an ordering of the gallery images to perform backfill- ing. To obtain the ordering we train the alignment model using uncertainty estimation and use predicted uncertainties to order the images in the gallery set.
46-
mAP
42-
40-
25
50
75
100
Backfilling (%)
We demonstrate superior performance compared to previous works on several datasets. More importantly, show that using FastFill a biased old model be efficiently fixed without requiring full backfilling,
Figure 5: Ablation Study on Ima- geNet: We compare the effect of dif- ferent training losses and backfilling orderings on the backfilling curves. We report mAP in the parentheses in the legend.

---

Published as a conference paper at ICLR 2023
8 ETHICS STATEMENT
We believe that Fast can have a positive impact in real-world applications by reducing the cost associated with model update, and hence enable frequent model updates.
Machine learning based models are often not completely unbiased and roquire regular updates when become aware of a certain limitation and/or bias. As shown in the paper, when we have a new model with fewer biases than an existing model, FastFill can reach the fairness level of the new model more quickly and cheaply. This motivate frequent model updates to address limitations associated with under represented groups.
As with many machine learning applications, especially ones used for facial recognition, FastFill can be misused. Facial recognition software can be used in many uncthical scttings, so improving their performance can have significant downsides as well upsides.
LIMITATIONS
Our proposed method does not provide a mechanism to decide what percentage of backfilling is sufficient to enable early stopping. Knowing how much backfilling is sufficient would result in saving additional computation.
In some real world applications, the image representations can be used in more sophisticated fashion by the downstream task than the nearest neighbour based retrieval considered in this work. For instance, representations we update from old to new, we also need to retrain the downst model so it is compatible with new feature space. This is not a limitation of Filt in particular, however, in general a limitation of transformation based update methods.
be input to another trained model. Therefore, when
REFERENCES
Mateusz Budník and Yannis Avrithis. Asymmetric metric learning for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238, 2021.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference automatic face & gesture recognition (FG 2018), pp. 67 IEEE, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Procerdings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 4690 4699, 2019.
Rahul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility-wwar heterogeneous visual search. In Proceedings of the IEEE/CVI Conference on Compuler Vision and Pallern Recognition, pp. 10723 10732, 2021.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789 1819, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pollern recognition, pp. 770 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503. 1503.02531, 2(7), 2015.
Weihua Hu, Rajas Bansal, Kaidi Cao, Nikhil Rao, Karthik Subbian, and Jure Leskovec. Learning backward compatible embeddings. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery & Data Mining, 2022.
10

---

Published as a conference paper at ICLR 2023
Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-effiieet incremental learning through feature adaptation. In European conference on computer vision, pp. 699-715. Springer, 2020.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 7482 7491, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint orxiv:1412 14 6980, 2014.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Feature Extraction: Modern Questions and Challenges, pp. 196 212. PMLR, 2015.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine inlelligence, 40(12):2935 2947, 2017.
Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embed- dings. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9939-99999999 99099994,,40 2021.
Nvidia.
Resnet50v1.5 for pytorch, 2019. resnet50v1.5.
URL http::://gituub..om///i/// htt.a///weeeeeeeeeeees////////ae
Martinez M Bertran A Oesterling and G Sapiro. Distributionally robust group backwards compatibility. In NewrIPS 2021 Workshop DistShift, 2021.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Vivek Ramanujan, Pavan Kumar Anasosalu Vasu, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Forward compatible training for large-scale embedding retrieval systems. Proceedings of the IEEE conference on computer vision and pattern recognétion, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 1522 2015.
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatibbe representation learning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognation, pp. 6368 6377, 2020.
Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. An empirical analysis backward compatibility in machine learning systems. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 3272 3280, 2020.
Shupeng Su, Binjie Zhang, Yixiao Ge, Xuyuan Xu, Yexin Wang, Chun Yuan, and Ying Shan. Privacy-presevving model upgrades with bidirectional compatible training in image retrieval. arXiv preprinl ar urxi::2204. 13919, 2022.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference compuler vision and pattern recognition, pp. 2818 22262226, 2016.
11

---

Published as a conference paper at ICLR 2023
Frederik Träuble, Julius Von Kügelgen, Matthäus Kleindessner, Francesco Locatello, Bern- hard Schölkopf, and Peter Gehler. backward-compatiioee prediction updates: probabilistic approach. Advances in Neural Information Processing Systems, 34:116-124 1211, 2021.
Chien-Yi Wang, Ya-Liang Chang, Shang- Ta Yang, Dong Chen, and Shang-Houug Lai. Unified representation learning for cross model compatibility. arviv preprinl arXiv:2008. 04821, 2020.
Sijic Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-congruent training; Towards regression-free free model updates. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 11299-14308, 2021.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987 3995. PMLR, 2017.
Binjic Zhang, Yixiao Gc, Yantao Shen, Yu Li, Chun Yuan, Xuyuan Xu, Yexin Wang, and Ying Shan. Hot-refresh model upgrades with regression-free compatible training in image retrieval. In International Conference on Learning Representations, 2021.
Yue Zhao, Yantao Shen, Yuanjun Xiong, Shuo Yang, Wei Xia, Zhuowen Tu, Bernt Shiele, and Stefano Soatto. Elodi: Ensemble logit difference inhibition for positive-congruent training. arXiv preprint Xiv:2200.. 06265, 2022.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
12

---

Published as a conference paper at ICLR 2023
1. Appendix A describes the experimental set-up for our experiments in the main paper. We detail the dataset set-up (A.1) as well as the training set-up (A.2) used in the main paper.
2. Appendix B reports further experimental results including: experiments focusing architectural changes (B.1) an analysis of Filt and FCT performance when using side-information (B.2), and top-5 retrieval results for the main experiments (B.3).
3. Appendix C focuses on further ablation studies. We first extend the ablation carried out in the main paper on ImageNet- Ik (C.1), before focusing on the Places-365 dataset (C.2). We then demonstrate one of the main strengths of our method when encountering new previously unseen classes during inference; we first show this on ImageNet (C.3) before focuses on a further ablation study, that combines entropy coonfpyccnn based backfilling with FastFill and FCT on VGGFace2 (C.4).
4. Appendix D includes detailed analysis of positive and negalive flips through the backfilling curve for different methods.
5. Appendix includes a more detailed analysis of the correlation of the o² computed by our method with various loss functions.
6. Appendix F contains more detailed analysis on which classes get backfilled first on the VGGFace2-Gender dataset.
A EXPERIMENTAL SET-UP
A.1 DATASET SET-UP
Image/et--k (21). ImageNet-Ik is a large-scale image recognition dataset and the most used subset of ImageNet [ILSVRC]. It was used in the ILSVRC 2012 image recognition challenge. The datasct contains 1000 object classes and a total of 1,281,167 training images and 50,000 validation images. The training dataset is relatively balanced with about 1.2k images per class. The validation set has the same classes as the training set and exactly 500 images per class.
We use a subset of ImageNet-11 1k consisting of the first 500 classes to train the old model; we refer to this as ImageNet-500. We note that the first half of ImageNet-1k 1k contains casier classes than the second half.
Places-365 (32). Places-365 is a large-scale scene recognition dataset that spans 365 classes and contains 1.8 million training images and 36,500 validation ones. The training set contains between 3068 and 5000 images per category and the validation set has exactly 100 images per class. We use the first 182 classes of the training set to create a smaller subset; we refer to as Placcs-182.
VGGFace2 (2). VGGFace2 is large-scale facial recognition dataset. Its training dataset is made up of 8631 classes, each representing a different person, and a total of 3.14 million images. The test set contains 170,000 images corresponding to 500 identities, different to the ones present in the training set. The training set is slightly unbalanced with an average of 362.6, minimum of 57, and a maximum of 843 images per class. The test set contains exactly 500 images per class. During test time, we compute random but fixed subsample of 50 images per class in the test set.
The dataset also has a list of attributes for small number of images in the training dataset. We know the gender of 30,000 images corresponding to 5300 different identities. We create a training and validation set of images for which we know the gender as follows: we disregard all the images in the training set with an unknown gender. We randomly sample 250 female identities and 250 male ones to form the new validation set. As there are more male than female classes remaining, we sample a subset of the remaining male identities to create a balanced training dataset (2183 female and male classes each with about 800,000 images for either (exact numbers depend on the random seed)). We refer to this new subset as We now create unbalanced subset. by sampling 500 male itintities
13

---

Published as a conference paper at ICLR 2023
and 50 female identities from the training set, name it (CCdecc2-Gendde-3550 We use this dataset to train a biased old model.
A.2 TRAINING SET-UP
ImageNet-lk. We train two ResNet50 models (7): the old model (dold) on ImageNct- 500, and the new model (Onew), on ImageNet-lk: We set the embedding dimension of both models to 128. We train both models using the hyper-parameters in ResNet V1.5 Nvidia (17) (SGD optimizer, epochs=100, natchsize=1024, learning rale=1.024, weight decay=3.0517578125 10 momentum=0.875, cosine learning rate decay with 5 epochs of linear warmup). For our method, we jointly train transformation model h, and an uncertainty predictor & on ImageNet-Ik using the objective function defined above (5). We use the same architecture for the MLP architecture for the transformation function as (20) and the same learning set-up (Adam Kingma Ba (13), epochs=80, learning rate=5 * 10-4, cosine learning rate decay with 5 epochs of linear warm-up, freeze of the BatchNorm layers after 40 epochs). For the uncertainty estimator sh we simply use a linear layer from the transformed embeddings space to output single real value. For FCT, train their transformation without side-information exactly as described in their paper and their official code base (https://gg htt///tttubb./// thub. com appplppllfcccttt..e/memees/essme model using the official code implementation (https: (https://githin //githh/ and exper-paramette
For BCT train new /aan//atatnnnnn, ( mentioned in the paper. However, we the embedding dimension to 128 to be directly comparable to the other three models. For RACT, we re-implement the method based on the information provided in the paper as there is no official code release. We set the byper-parametedd the learning rate to get optimal performance. We compare the learning rate schedule used in RACT with the one used in ResNel50V1.5 Nvidia (17). optimizing over the learning rate for each (we try learning rates powers of ten in [1e-5, 1)).
(A = 1 and r == 0.5) as defined in the paper and optimize over
For the retrieval experiments we set both the gallery and query sets to be the validation set of ImageNet- 1k. For BCT and FCT we use random backfilling, as neither paper include a backfilling ordering. For RACT we try both random backfilling and entropy backfilling and pick the better one.
Places-365. We use similar setup to ImageNet experiments: we train two ResNet50 models with an embedding size of 512: the old model on Places-182 and the new one on Places-365, We use the Places-365 validation dataset for retrieval evaluation.
VGGFace2. For the VGGFace2 dataset we train a smaller ResNet18 model on VGGFace2- 863 and a larger ResNet50 one on VGGFace2-8631. Unlike for the other two datasets we use the ArcFace objective (4) to train the two models and the transformation. We use a margin of 0.5 and scale of as proposed by (4). We use the same learning rate schedule as for ImageNet. Unlike for the other two datascts we normalize the features.
14

---

Published as a conference paper at ICLR 2023
B FURTHER EXPERIMENTAL RESULTS
B.1 ARCHITECTURAL CHANGE EXPERIMENTS
We analyse how Fast compares to existing methods when the old and models have different architectures. We run ImageNet experiments where the old model is ResNet- 18, and the new model ResNet-50. Both are trained on the same full Image.Net-lk training set. As before, we repeat the experiments with 5 random seeds and illustrate the results in Table and Figure 6. The overall trends are the same as the other sctups in the original paper. Filt significantly outperforms all baselines.
Table 4: ImageNet- 1k Experiments on architectural changes: The old model a ResNet-18 and the new model ResNet-50.
Method Old model
top1
mAP
54.04
26.41
FastFill
67.77
45.94
FCT
64.91
43.43
RACT
64.80
42.04
BCT
59.26
36.56
FastFill
FCT
RACT
BCT
Old Model
45
CMC Top-1 (%)
40
mAP (%)
60
35
55
30
50
0
20
40
60
80
100
0
20
40
60
80
100
Backfilling (%)
Backfilling (%)
Figure 6: We run experiments on ImageNet to investigale the performance of an architecture change. The old model ResNet-1 18 and the new model ResNet-50; both are trained on ImageNet-Ik, lk.
15

---

Published as a conference paper at ICLR 2023
B.2 SIDE-INFORMATION EXPERIMENTS
PecTri WIP Side Info
FCT wit Side Info
60
CMC X00- (89)
60
46
67
00
63.0
82.5
82.0
42
20
40
90
100
20
60
80
100
0
40
60
100
Backtiliig (961
Backtiiing (%)
Bockhiting 196
Figure 7:
We add side-information to the transformation function trained with either Fist (blue) or FCT (orange). FastFill outperforms FCT on all three metrics: CMC-topl (left), CMC-top5 (middle), and mAP (right).
We train both the FCT transformation model (h) as well the FastFill transformation and uncertainty models (A and vi) with side-information. For the side-information model we train ResNet50 model on ImageNet-500 with SimCLR (3). Otherwise, we use the same set-up as for the main paper ImageNet experinents. We show in Figure 7 that both methods get a performance boost with side-information. But Fill still significantly outperforms FCT three different metries: CMC-topl (67.81% 64.72%), CMC-top5 (84.23% vs 82.35%); and mAP (46.23% vs 42.55%).
B.3 CMC-TOP5
We run experiments using the CMC-top5 retrieval metric. As shown Figure 8 FastFill outperforms all baselines on ImageNet-lk (+2.8%), VGGFace22 (0.1%), and Places-365 (+2.1%) when using CMC-top5 evaluate retrieval performance.
85
98
66.5
CMC Top 5 (96
80
CMC Top (%6)
CMC Top-5 186
96
75-
65.5
FastFill-sgmaa
FastFill
65.0
FCT
70
FCT
FastFill
64.5
RACT
65
BCT
92-
RACT
64.0
75
100
100
Backfill ng (%)
Backfilling (%)
Backfilling (96)
(a)
(b)
(c)
Figure 8: We run experiments using the CMC-top5 retrieval metric on (a) ImageNet-Ike Net-lk, (b) VGGFace2, and (c) Places-365.
16

---

Published as a conference paper at ICLR 2023
C FURTHER ABLATION STUDIES
C.1 ABLATION ON IMAGENET-IK
We elaborate from the ablation study provided in the main paper in section 6 in Figure and Table adding the CMC-topl metric. In particular we show that when training with Edise with or without uncertainty we get poor compatibility performance at 0% backfilling and throughout the backfilling (when measured using mAP). However, adding it to Lt, to get dise improves results for both random and 6² based backfilling for both mAP and CMC-topl.
Table 5: Ablation Study: We compare the effect of different training losses and backfilling orderings on the backfilling curves.
Method
top
mAP
Gia)-oooomm
62.80
40.47
rrandom
65.51
39.45
65.06
41.95
Lt, wun.ert.)-raaaeeeeeeee
62.58
40.08
Cia Jrandrr.)-randoo
64.51
42.05
Ci bencert.)-signae
65.76
43.61
Cdiso + uncert.) uncert.)-sigmaa )-sigma
55.21
27.59
(ig d (uncert))-signt
66.49
44.84
CMC Top-1 (15)
65
60
Top
60
50
Ayranndom
Cas-random
46 se-taandnoon
uuncertt I-random
40
60
80
100
0
20
40
60
100
(8, aunneert (4, (u,+tucerr.)-signn
/vandot J-randmm
Backfiiling (%)
Backfilling
(Fasctuncert) sigma
24
(el- unccert. /-sigma
40
mAP 196)
42
40
20
40
60
80
100
0
20
40
60
80
100
Backfilling (%)
Backfilling (%)
Figure Ablation Study: We compare the effect different training losses and backfilling orderings the backfilling curves.
17

---

Published as a conference paper at ICLR 2023
C.2 ABLATION ON PLACES-365
We provide an ablation study in Figure 10 and Table on the Places-365 dataset. The conclusion is similar to the ImageNet ablation where FastFill improves its ablations using alternative uncertainty measures and significantly improves previous work FCT.
Table 6: Ablation Study on Places-365: We compare the effect of different.
training losses and backfilling orderings the backfilling curves.
Method
Transforma-
Uncertainty
top1
mAP
tion
Estimation
Old Model
29.57
11.39
FCT
X
37.73
16.81
FastFill
Bayesian
38.32
19.48
Fast FissF--Abllttion
Entropy
38.12
18.6
FastFill
FastFill-ablation (entropy)
FCT- PCT-Random
Old
40
38
Top-1 (%)
18
36
MAP (%)
34
14-
32
30
0
20
40
60
80
100
20
40
60
80
100
Backfilling (%)
Backfilling (%)
Figure 10: Ablation Study on Places-365: We compare the effect of different training losses and backfilling orderings on the backfilling curves.
18

---

Published as a conference paper at ICLR 2023
C.3 ABLATION ON IMAGENET-250 тO IMAGENET-500
We provide a further ablation study that compares different ovaluation set-ups focusing different backfilling orderings provide by different. uncertainty estimations. We run experiments where the same classes are used during training and evaluation as well as experiments where we evaluate the compatibility on previously unseen classes. The results are illustrated in Figure 11 and Table 7. In every case we train the old model on ImageNet-250 and the new model on ImageNet-500. For the first first 500 classes for both the query and gallery. This is the same train-test classes sctup. For the second experiment ([500:]) we use the sccond 500 classes for both query and gallery. This is the disjoint train-test classes setup. And finally query and gallery ([:1000)). This setup where half of the train-test classes overlap.
of experiments ([500: }) we use the
all 1000 classes for both
The below results show that Bayesian estimation (FastFill) brings further improved accuracy in the presence of new classes. For example, compared to its ablations, FastFill obtains +1% top-1 improvement when evaluated on the first 500 classes [:500], but gets +2.7% top-1 improvement when evaluated on the second 500 classes [500:] that are unseen during training. Further, when evaluated unscen classes, ablations of FistFill with logits-based uncertainty cstimation gets rosults close to no uncertainty estimation at all as in FCT.
We note that for most real-world image retrieval tasks (as for instance facial recognition, or zero-shot retrieval) the test-time samples (for gallery and query sets) are coming from classes that not present training-time. Using training time classification head to measure uncertainty as in RACT (30) is a major limitation in practice since the relative confidence entropy between the samples from new classes tend be In contrast, our proposed uncertainty estimation method based on feature alignment is explicitly designed to be class agnostic and works well on previously unscen classes. This one of our method's main contributions and novelties.
informative.
Table 7: Ablation Study on ImageNet: We compare the effect of different training losses and backfilling orderings on the backfilling curves. The old model is trained on ImageNet-250 and the new one on ImageNet-500.
Method
Trans-
Uncertainty
forma- tion
Estimation
top1
top1
top1
m MAP
MAP
mAP
[: 500
[500
1000]
500
[500 :]
[: 1000)
Old Model
50.7
12.41
29.32
29.
2.47
14.9
Filt
Bayesian
75.27
17.9
43.0
53.88
4.68
26.44
FastFill-ablatia
Entropy
74.26
15.22
40.78
53.48
4.13
25.74
PastFill-ablation
Margin Conf.
74.21
15.17
40.66
53.21
4.01
25.56
FastFill-ablation
Least Conf.
74.28
15.18
10.74
53.38
4.07
25.65
FCT
71.15
14.94
38.77
48.61
3.7
23.1
RACT
Entropy
66.88
13.65
36.65
41.6
3.3
20.4
19

---

Published as a conference paper at ICLR 2023
FastFill
FastFill-ablation (confidence) (margin)
FastFill-ablation (entropy)
Old Model
RACT
55
75
50
CMC Top-1 (%)
70
mAP (%)
45
65
40
60
55
30
50
20
80
100
40
60
100
Backfilling (%)
Backfilling (%)
FastFill
FastFill-ablation (confidence)
FastFill-ablation (entropy)
Fastrill-ablation (margin)
Old Model
RACT
25
CMC Top-1 (%)
20
(%6
5
15
4
10
3
2
20
80
100
40
60
100
Backfilling (%)
Backfilling (%)
FastFill
PastFill-ablation (confidence) FastFill-ablation (margin)
Fastrill-ablation (entropy)
FCT
Old Model
RACT
45
27.5
CMC Top-1 (%)
25.0
22.5
mAP
35
20.0
17.5
30
15.0
20
80
100
0
40
60
80
100
Backfilling (%)
Backfilling (%)
Figure 11: Ablation Study on ImageNet: We compare the effect of different training losses and backfilling orderings on the backfilling curves. Top: 500: Use the first 500 classes for both query and gallery. This is the same train-test classes setup. Middle: [500:] Use the second 500 classes for both query and gallery. This is the disjoint train-test classes setup. Bottom: [:1000] Use all 1000 classes for both query and gallery. This is setup where half of the train-test classes overlap. These results show that FastFill is particularly strong when we encounter new previously unseen classes during inference time as is common in many real life retrieval settings
20

---

Published as a conference paper at ICLR 2023
C.4 ABLATION ON VGGFACE2
We run experiments on VGGFace2 using confidence and entropy based backfilling (as proposed by Zhang al. (30)) on top of our proposed loss for feature aligument and the FCT baseline. We illustrate the results in Figure 12.
We observe that entropy /confidence based ablation top of FastFill and baseline FCT are quite close to random backfilling whereas our proposed uncertainty estimation significantly improves them. Moreover, FastFill variants utilize a transformation function from old to new feature space that leads to significantly improved model update accuracies compared to RACT which directly enforce compatibility between new and old models (more than 5% on top-1 and 25% on mAP metrics).
We note that the uncertainty estimation measures proposed in Zhang et al. (30) (confidence and entropy) are intuitive and simple to use, but rely on the training time classification head. For most real-world image retrieval tasks (e.g. face recognition, or zero-shot retrieval) the test-time samples (for gallery and query sets) coming from different classes to those during training-time, Using the training-time classification head to measure uncertainty as in Zhang al. (30) is major limitation in practice since the relative confidence/ caffidenne/enrropyy cntropy between the samples from new classes tend to be not informative. In contrast, our proposed uncertainty estimation method based on feature alignment is explicitly designed to be class agnostic and works well on previously unseen classes. This one our method's main contrbbutionsnovelllae
97.5
97.0
95.0
96.5
92.5
FestFill
CMC Top-1 (%)
CMC Top-1 1%)
RectFll-andomm
90.0
Fastrill. east Confidence
96.0
FactFil-Confiddrre Margin
87.5
RacRR Entropy
PCT ndooo
95.5
FastfillL Confidence
85.0
FCTTead Confidence
FostFill Confiidncc Marg
FC Confidence Margin
95.0
Fastrill-nnropyy
RCT-Bandomm
Least Contidence
82.5-
80.0
RACTA east Canfisence
RACT-Rar dom
94.5
FCT-Corlilence Hangin
PCT Encropy
77.5
RACT-Pnt mpy
Margin
100
20
60 Backfilling (%)
40
80
100
Backfilling (%)
(a)
(b)
70
MAp (56
RatRI
Pasifil Randor
60
Fasth Lasse Coristence Morgin
FextFil-Lees Confiderce
MAP (%)
68
Nargin
50
suitill-ntropp
66
RCT-Anddom
40
FCT-eest Donfidence
64
cCmtCoam Margin
30
RXC Luret Corldence
MAC14asserr
cont Hangh
62
BACTS soug
20
40
60
80
100
20
40
Backfilling (%)
Backfilling (%)
(c)
(d)
Figure 12: We run experiments on VGGFacc2 using confidence and entropy based backfilling (as proposed by Zhang el al. (30) on lop our proposed loss for feature alignment and the FCT baseline. We compare different methods using the CMC-topl (a, b) and mAP (c, d) metrics, respectively. In (b) and (d) we further compare against RACT.
21

---

Published as a conference paper at ICLR 2023
D POSITIVE AND NEGATIVE FLIPS ANALYSIS
We analyse the positive and negative flips for FastFill and FCT. A negative flip is an image that gets classified correctly for cross-model retrieval with 0% but gets misclassified when using a E (0, amount of backfilling. Conversely a positive flip is an image that originally gets misclassified but then labelled correctly for some amount of (partial) backfilling, The retrieval performance at a certain backfilling point is a function of the original cross-model retrieval performance 0% backfilling and plus the number of positive minus the negative flips. We show FastFill outperforms FCT by both increasing the number of positive flips with limited backfilling whilst also causing fewer negative flips throughout the entire backfilling curve. We note that even though at 100% backfilling both FCT and FastFill use the same model, the number of flips doesn't 't match as also depends on the cross model performance at 0% backfilling, which is different for the two methods.
neg faiptil
neg neg-flips flips fastrill-sigma pos-flips- fastill--a neg-flips-fct-random pos-flips-cctrandoom
pos-flips-fastfilllsigge
neg-flips-fct-random
pos-flips-cct-randdm
6000
Flips compared to 0% backfilling
Flips compared 0% backfilling
3000
4000
2000
2000
1000
0
-1000
-2000
-2000
-4000
25
50
75
100
25
50
75
100
Backfilling (%)
Backfilling (%)
(a)
(b)
Figure 13: Positive and Negative Flip Analysis on ImageNot. We compare FastFill (blue and orange) and FCT (green and red) on ImageNet using (a) CMC-topl and (b) top-5 retrieval, respectively. The aim is to maximize the number of positive flips (orange and red), whilst minimizing the number of negative ones (blue and green) with as little backfilling as possible.
22

---

Published as a conference paper at ICLR 2023
E
SIGMA ANALYSIS
Further to the analysis done in the main paper above, we show how well o² correlates with different loss functions (fi)+dise((;)) Li, (x;), and (x)) and the orderings that they induce. Our prediction o² most closely correlates with + (æ;) (the two orderings have a Kendall-Tau correlation of 0.67) which is unsurprising it is trained to predict this loss. However, it still predicts both (a,) (Kendall- Tauu correlation of 0.65) and Caise(()) (Kendall Tea correlation of 0.63) well.
When we train our uncertainty estimator to predict Li, (æ;), the correlation with the or- dering induced by Li, (2;) Seendal--Tau-O.T3: increases but at the same time it correlates significantly less strongly with the order based on (x)) (Kondall-Tau=0056)).
When we train with (xi we still get a relative strong correlation with the ordering induced by (xi) [Kendall-Tau-0... but much less so with the Lig (x;, order (Kendall- 0.3=0.34).
2.5
3.0
2.0
2.5
2.0
log Eave (x)
1.5
log y
2.0
1.5
1.0
0.5
1.5
1.0
0.0
1.0
0.5
-0.5
0
1
log of
log
log of
(a)
(b)
(c)
50000
50000
50000
Order of [. (x)
40000
Order of f,((x))
(aase())
40000
30000
30000
30000
20002
20000
20000
10000
10000
10000
0
20000
40000
20000
40000
0
20000
40000
Order of (d)
Order of of
Order of
(e)
(I)
Figure 14: a, b, e) We show that the predicted log of, the output of the on gallery images, and log Lig dise (m; log (2;), and log (a;s, respectively, are are well correlated. c) We sort x; once by of and once by (2x+dec((x)), Li₂ (x;), and Lasee (x;) respectively, to get two sets of orderings. Here, for each æ; we plot its order in the first ordering its order in the second ordering, and observe great correlation (Kendall- Tau correlation=0.67 (0.65, and 0.63 respoctively)).
23

---

Published as a conference paper at ICLR 2023
F BACKFILLING ORDER ANALYSIS- VGGFACE2-CENDEL
We now analyse which types of classes are backfilled first using our FastFill method on the VGGndee2-Gender dataset. We group the classes into minority (female) and majority group (male). As seen in Figure 15 FastFill prioritizes the minority class for carly backfilling which explains the behaviour seen in Section 5.3 in the main paper.
100
100
% of Gallery Set Backfilled
Male
% of Gallery Set Backfilled
Male
Female
Female
80
80
60
60
40
40
20
20
0
0
25
50
75
100
25
50
75
100
Backfilling(%)
Backfilling(%)
(a)
(b)
Figure 15: We compare the backfilling order of the minority (female) and majority (male) classes using (a) FastFill, and (b) FCT-Random. FastFill prioritizes the minority group and backfills the female gallery images first. As a results, FastFill achieves the new model accuracy gap after only 25% of backfilling.
24


Reviews:
[{"review_id": "v6k2VCSxBk", "reviewer": "Reviewer_frJ2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an online backfilling method, FastFill, for backward-compatible retrieval systems. FastFill introduces a new feature alignment loss, L_{disc}, that maps the old feature descriptor to the corresponding cluster in the new feature space. For faster deployment, FastFill proposes to backfill the gallery images on-the-fly by a learned order that indicates the “hardness” of the samples. Compared to previous approaches, FastFill demonstrates better performance for online partial backfilling on three retrieval benchmarks: imagenet1k, vggface2, and places365.", "summary_of_the_review": "The paper proposes FastFill, an online backfilling method for backward-compatible retrieval systems. FastFill is technically sound and demonstrates better performance for online partial backfilling on several benchmarks. However, at this moment, I still have concerns about the feasibility of the online partial backfilling setting.", "strength_and_weaknesses": "Strength\n\n[S1] Technology: the proposed method is technically sound;\n\n[S2] Performance: FastFill demonstrates better performance for online partial backfilling on imagenet1k, vggface2, and places365\n\n[S3] Presentation: the writing is clear\n\n\nWeaknesses\n\nThe paper introduces a new evaluation setting: online partial backfilling. The goal is to deploy the features of the gallery set on-the-fly. FastFill only shows benefits in this setting. However, I’m not sure if online partial backfilling is a feasible setting or of practical interest. \n\ni) If I understood correctly, Figure 1 (right) presents several configurations for image retrieval: (C1) old query features + old gallery features (\\phi_{old}(query) + \\phi_{old}(gallery)); (C2) new query features + old gallery features (\\phi_{new}(query) + \\phi_{old}(gallery)); (C3) new query features + new gallery features (\\phi_{new}(query) + \\phi_{new}(gallery)). Here FastFill is in between (C2) and (C3). Figure 1 (right) seems to indicate that (C2) has a better performance than (C1): 61.8% vs 46.6%. I’m not sure if it is always the case as there may be a large shift in the feature spaces of \\phi_{old} and \\phi_{new}. I would guess the performance of (C1) is in between (C2) and (C3), e.g. a horizontal line between 61.8 and 68.2. In other words, at the early stage of the online backfilling (i.e. (C2)), the updated system may have lower performance than the old one. Therefore, it may not even be deployed. Also, I believe it is necessary to include the performance of (C1) in other diagrams, e.g. Fig.2/3/5.\n\nii) it is also hard to determine if the FastFill model or even (C3) actually has a better performance than (C1) without updating all the gallery features. The unseen query and gallery sets are the “golden” test set that the new model never sees. While we may be able to estimate the performance of the FastFill model or (C3) on this “golden” set in the middle of the backfilling, the estimation may not be conclusive or deserves the risk. As can be seen from Fig. 3 and 5, occasionally there may be performance drops at the last stage of the backfilling (>90%). \n\nAt this moment, I’m not sure the online partial backfilling setting is convincing. This is my main concern.\n\n***\nAfter rebuttal:\n\nMy concern about the configurations (i.e. Weakness i) has been addressed. Regarding the online backfilling setting (i.e. Weakness ii), I agree that it is a more practical problem that may not be expected to be resolved in one research paper. I thereby raise the score to 6. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "OPRvOs6YZo", "reviewer": "Reviewer_gEJP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Image retrieval model can be updated when there is more training data or improved architecture, but it is not straightforward since old and new feature spaces are not compatible, leading to poor retrieval results between new query and old gallery embeddings. A naive solution is to replace the old features by new features from the new model, referred to as backfilling, but this may be computationally expensive and a blocker for model update. Although several compatible learning approaches have achieved compatible feature with the old model, it degrades the new model performance. To alleviate this issue, they consider two practical scenarios; 1) backfill the entire gallery set in random order and aim to maximize the average performance, and 2) backfill a partial gallery set and maximize the performance after partial backfilling. To this end, they train transformation models with uncertainty estimation using bayesian neural network and decide backfilling order based on the uncertainty of alignment loss.", "summary_of_the_review": "This paper address the partial backfilling scenario to alleviate the limitation of existing works. The authors employ bayesian neural network to estimate the uncertainty of alignment error for efficient backfilling strategy. Although it improves the compatible performance, novelty is not strong and analysis and ablative study are insufficient, which makes the results not fully reliable.\n\n===== POST-REBUTTAL =====\n\nMy major concern is about the true effectiveness of proposed method, which mainly consists of FCT and Bayesian-based backfilling update where FCT is borrowed from the existing work. Thus, to convince the readers to use Bayesian uncertainty for backfilling process, I still think the performance gain of FastFill should be analyzed with previous work (confidence-based uncertainty from RACT [1]) under the fair setting (FCT as backbone algorithm). If not, the readers would simply use FCT+confidence rather than FastFill, which is easier to use. However, the performance gain is marginal in the standard setting (0.6%p on ImageNet-1K and 0.2%p on Places-365 in top@1), which makes the overall novelty limited. There are more gains in disjoint setting (2.7%p in top@1 and 0.5%p in mAP on ImageNet-[500:]), but the improvements are still moderate and need to re-write the overall paper with more thorough analysis to argue it as a main claim as there are insufficient experiments. The experimental results are not sufficient to verify the need to employ additional uncertainty head and training objective and the responses do not resolve my main concerns, thus I keep my original rating.\n\n[1] Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval,  Zhang et al, ICLR 2022\n", "strength_and_weaknesses": "### Strength\n- They proposed a compatible learning method for partial backfilling scenario.\n- The experimental results present meaningful performance improvement on three datasets.\n- The writing is easy to read.\n\n### Weaknesses\n- Novelty is weak and comprehensive ablative results are needed. [1] also presents several simple uncertainty-based backfilling strategies based on the classification score, e.g., confidence uncertainty or entropy-based uncertainty. Could you report and compare the results using these strategies on top of baseline and your model? ex) (L_l2+disc)+ confidence uncertainty or (L_l2+disc+uncert) + confidence uncertainty.\n- More detailed analysis is needed. What classes of samples are first backfilled when performing the backfill strategy for each dataset? Is the class distribution uniform? or biased to some classes?\n- Robustness. How is the results when the model architecture is changed, e.g., ResNet-18 to ResNet-50? \n- Model bias section is weak. It presents the results on a single dataset with empirical observation only. There are some biased datasets, like as ImageNet-C, and more thorough analysis is needed to support the argument in that section. \n- There are some strange experimental results but no explanations.\n    - In Places-365, why new model gives worse result than old model in mAP? Even the results of mAP and CMC are conflicted.\n    - In ImageNet and Places-365, why there are negative flips? Is there any ways to do early stopping?\n- Why there is no BCT results on VGGFace2 and Places-365?\n\n[1] Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval,  Zhang et al, ICLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6dvU9TuGDq", "reviewer": "Reviewer_zbYY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a training loss function with a combination of  l2 loss and cross-entropy-based loss for old/new feature transformation module training. To further improve accuracy during replacing old features with new transformed ones, the paper utilizes Bayesian method to estimate the fitting error on the gallery set and backfill the features in error descending order.", "summary_of_the_review": "Overall, I think this paper should be accepted after addressing following concerns, \n\n(1) some details should be explained more In the experiments part, in Table 1, 2, &3, which features are we using here? Old feature? transformed old feature with $h_{\\theta}$? or new features?\n(2)  In Fig 5, and Fig 3. the fig what are the initial points of curves stand for? is that the accuracy of using $h_{\\theta}$ to transform old feature? If so, the numbers should be incorrect.\n(3) Can we test the method on some image retrieval datasets like Google Landmark? I understand that adding experiments is not required and not recommended in rebuttal, but it will be better if we have some results on image retrieval dataset.", "strength_and_weaknesses": "1. The proposed training loss is easy to implement. \n2. Estimating fitting errors of gallery images and re-ordering the backfilling process is novel to compatible learning and easy for practicing.\n3. The paper writing is easy to read.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0bOzVeuw32", "reviewer": "Reviewer_bcVj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of backfilling in retrieval systems, which is the process of updating the features extracted from the images of your database once we have access to a newer and better model: if this is done offline, for very large databases it is a very expensive process and can take very long. The authors state that current methods of online backfilling (in which new computed features as used as soon as they are ready) in order to maintain a high accuracy during backfilling, they try to make features of the newer and older model compatible, and argue that this limits the newer model capability. They propose an online backfilling method that learns a transformation from the older feature space to the newer one (the newer model is trained independently), and a policy to order the samples that need to be backfilled first to keep the highest performance during this online process.", "summary_of_the_review": "The proposed framework for online backfilling looks solid and novel enough. It is simple and easily applicable. The paper also has a thorough experimental analysis and  confirm the claims made in the paper. ", "strength_and_weaknesses": "Pros\n- The paper is well written and the motivations very clear. I like the idea of using the classifier head and a classification loss to guide the error of the alignment towards the same-class clusters and I find it novel. \n- The authors justify their claims with thorough experiments and analysis; it includes a comprehensive evaluation on public datasets where the proposed framework is showed to outperform state of the art methods.\n\nCons\n- I am really curious why partial backfilling in Places-365 results in better performance than after the backfilling is completed. I also wonder if any information in the policy sampling can be used to not only determine the order but also what features should not be updated.\n- I personally think that the title \"Reducing model biases\" of 5.3 is misleading because Fastfill is not helping to reduce the biases of a model but simply \"catching up\" faster to the performance of the newer model, whatever bias/non-bias it has. It is an interesting experiment, but nothing different from the experiments in 5.2 and, in my opinion, slightly oversold. In other words, one can draw the same conclusions from this experiment as from the experiments presented in 5.2, and no newer ones.\n- The name of the dataset use in the experiments in Section 6 is missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster