Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the problem of backfilling in retrieval systems, which is the process of updating the features extracted from the images of your database once we have access to a newer and better model: if this is done offline, for very large databases it is a very expensive process and can take very long. The authors state that current methods of online backfilling (in which new computed features as used as soon as they are ready) in order to maintain a high accuracy during backfilling, they try to make features of the newer and older model compatible, and argue that this limits the newer model capability. They propose an online backfilling method that learns a transformation from the older feature space to the newer one (the newer model is trained independently), and a policy to order the samples that need to be backfilled first to keep the highest performance during this online process.
strength_and_weaknesses: Pros
- The paper is well written and the motivations very clear. I like the idea of using the classifier head and a classification loss to guide the error of the alignment towards the same-class clusters and I find it novel. 
- The authors justify their claims with thorough experiments and analysis; it includes a comprehensive evaluation on public datasets where the proposed framework is showed to outperform state of the art methods.

Cons
- I am really curious why partial backfilling in Places-365 results in better performance than after the backfilling is completed. I also wonder if any information in the policy sampling can be used to not only determine the order but also what features should not be updated.
- I personally think that the title "Reducing model biases" of 5.3 is misleading because Fastfill is not helping to reduce the biases of a model but simply "catching up" faster to the performance of the newer model, whatever bias/non-bias it has. It is an interesting experiment, but nothing different from the experiments in 5.2 and, in my opinion, slightly oversold. In other words, one can draw the same conclusions from this experiment as from the experiments presented in 5.2, and no newer ones.
- The name of the dataset use in the experiments in Section 6 is missing.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to understand, the ideas are novel for this particular problem, and the work is reproducible.
summary_of_the_review: The proposed framework for online backfilling looks solid and novel enough. It is simple and easily applicable. The paper also has a thorough experimental analysis and  confirm the claims made in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an online backfilling method, FastFill, for backward-compatible retrieval systems. FastFill introduces a new feature alignment loss, L_{disc}, that maps the old feature descriptor to the corresponding cluster in the new feature space. For faster deployment, FastFill proposes to backfill the gallery images on-the-fly by a learned order that indicates the “hardness” of the samples. Compared to previous approaches, FastFill demonstrates better performance for online partial backfilling on three retrieval benchmarks: imagenet1k, vggface2, and places365.
strength_and_weaknesses: Strength

[S1] Technology: the proposed method is technically sound;

[S2] Performance: FastFill demonstrates better performance for online partial backfilling on imagenet1k, vggface2, and places365

[S3] Presentation: the writing is clear


Weaknesses

The paper introduces a new evaluation setting: online partial backfilling. The goal is to deploy the features of the gallery set on-the-fly. FastFill only shows benefits in this setting. However, I’m not sure if online partial backfilling is a feasible setting or of practical interest. 

i) If I understood correctly, Figure 1 (right) presents several configurations for image retrieval: (C1) old query features + old gallery features (\phi_{old}(query) + \phi_{old}(gallery)); (C2) new query features + old gallery features (\phi_{new}(query) + \phi_{old}(gallery)); (C3) new query features + new gallery features (\phi_{new}(query) + \phi_{new}(gallery)). Here FastFill is in between (C2) and (C3). Figure 1 (right) seems to indicate that (C2) has a better performance than (C1): 61.8% vs 46.6%. I’m not sure if it is always the case as there may be a large shift in the feature spaces of \phi_{old} and \phi_{new}. I would guess the performance of (C1) is in between (C2) and (C3), e.g. a horizontal line between 61.8 and 68.2. In other words, at the early stage of the online backfilling (i.e. (C2)), the updated system may have lower performance than the old one. Therefore, it may not even be deployed. Also, I believe it is necessary to include the performance of (C1) in other diagrams, e.g. Fig.2/3/5.

ii) it is also hard to determine if the FastFill model or even (C3) actually has a better performance than (C1) without updating all the gallery features. The unseen query and gallery sets are the “golden” test set that the new model never sees. While we may be able to estimate the performance of the FastFill model or (C3) on this “golden” set in the middle of the backfilling, the estimation may not be conclusive or deserves the risk. As can be seen from Fig. 3 and 5, occasionally there may be performance drops at the last stage of the backfilling (>90%). 

At this moment, I’m not sure the online partial backfilling setting is convincing. This is my main concern.

***
After rebuttal:

My concern about the configurations (i.e. Weakness i) has been addressed. Regarding the online backfilling setting (i.e. Weakness ii), I agree that it is a more practical problem that may not be expected to be resolved in one research paper. I thereby raise the score to 6. 
clarity,_quality,_novelty_and_reproducibility: The proposed solution is new. The writing is clear. The authors are suggested to include the performance of (C1) in Fig. 2/3/5 as discussed in the weaknesses section.
summary_of_the_review: The paper proposes FastFill, an online backfilling method for backward-compatible retrieval systems. FastFill is technically sound and demonstrates better performance for online partial backfilling on several benchmarks. However, at this moment, I still have concerns about the feasibility of the online partial backfilling setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a training loss function with a combination of  l2 loss and cross-entropy-based loss for old/new feature transformation module training. To further improve accuracy during replacing old features with new transformed ones, the paper utilizes Bayesian method to estimate the fitting error on the gallery set and backfill the features in error descending order.
strength_and_weaknesses: 1. The proposed training loss is easy to implement. 
2. Estimating fitting errors of gallery images and re-ordering the backfilling process is novel to compatible learning and easy for practicing.
3. The paper writing is easy to read.
clarity,_quality,_novelty_and_reproducibility: Although the proposed training loss is somehow lacking novelty (just a sum of l2 loss and cross entropy loss), it is good for the real-product system to reproduce the method. The paper rises a new problem in compatible learning: backfilling order problem. It is a new direction to look into, and the paper also proposes a practical method to follow.

summary_of_the_review: Overall, I think this paper should be accepted after addressing following concerns, 

(1) some details should be explained more In the experiments part, in Table 1, 2, &3, which features are we using here? Old feature? transformed old feature with $h_{\theta}$? or new features?
(2)  In Fig 5, and Fig 3. the fig what are the initial points of curves stand for? is that the accuracy of using $h_{\theta}$ to transform old feature? If so, the numbers should be incorrect.
(3) Can we test the method on some image retrieval datasets like Google Landmark? I understand that adding experiments is not required and not recommended in rebuttal, but it will be better if we have some results on image retrieval dataset.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Image retrieval model can be updated when there is more training data or improved architecture, but it is not straightforward since old and new feature spaces are not compatible, leading to poor retrieval results between new query and old gallery embeddings. A naive solution is to replace the old features by new features from the new model, referred to as backfilling, but this may be computationally expensive and a blocker for model update. Although several compatible learning approaches have achieved compatible feature with the old model, it degrades the new model performance. To alleviate this issue, they consider two practical scenarios; 1) backfill the entire gallery set in random order and aim to maximize the average performance, and 2) backfill a partial gallery set and maximize the performance after partial backfilling. To this end, they train transformation models with uncertainty estimation using bayesian neural network and decide backfilling order based on the uncertainty of alignment loss.
strength_and_weaknesses: ### Strength
- They proposed a compatible learning method for partial backfilling scenario.
- The experimental results present meaningful performance improvement on three datasets.
- The writing is easy to read.

### Weaknesses
- Novelty is weak and comprehensive ablative results are needed. [1] also presents several simple uncertainty-based backfilling strategies based on the classification score, e.g., confidence uncertainty or entropy-based uncertainty. Could you report and compare the results using these strategies on top of baseline and your model? ex) (L_l2+disc)+ confidence uncertainty or (L_l2+disc+uncert) + confidence uncertainty.
- More detailed analysis is needed. What classes of samples are first backfilled when performing the backfill strategy for each dataset? Is the class distribution uniform? or biased to some classes?
- Robustness. How is the results when the model architecture is changed, e.g., ResNet-18 to ResNet-50? 
- Model bias section is weak. It presents the results on a single dataset with empirical observation only. There are some biased datasets, like as ImageNet-C, and more thorough analysis is needed to support the argument in that section. 
- There are some strange experimental results but no explanations.
    - In Places-365, why new model gives worse result than old model in mAP? Even the results of mAP and CMC are conflicted.
    - In ImageNet and Places-365, why there are negative flips? Is there any ways to do early stopping?
- Why there is no BCT results on VGGFace2 and Places-365?

[1] Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval,  Zhang et al, ICLR 2022

clarity,_quality,_novelty_and_reproducibility: - The authors provided the results on three standard benchmark datasets. The overall writing is clear, but not well structured. There are some missing explanations in method and experiment sections. More detailed analysis to validate the true effectiveness of proposed method is needed.
- There are neither limitation nor ethics statement section.
- This paper focuses on efficient backfilling strategy for online model upgrade, which has not been actively explored yet. However, [1] also present several simple but effective backfilling strategies but the comparisons are missing.
- I cannot evaluate on the reproducibility as there are no code implementations.
summary_of_the_review: This paper address the partial backfilling scenario to alleviate the limitation of existing works. The authors employ bayesian neural network to estimate the uncertainty of alignment error for efficient backfilling strategy. Although it improves the compatible performance, novelty is not strong and analysis and ablative study are insufficient, which makes the results not fully reliable.

===== POST-REBUTTAL =====

My major concern is about the true effectiveness of proposed method, which mainly consists of FCT and Bayesian-based backfilling update where FCT is borrowed from the existing work. Thus, to convince the readers to use Bayesian uncertainty for backfilling process, I still think the performance gain of FastFill should be analyzed with previous work (confidence-based uncertainty from RACT [1]) under the fair setting (FCT as backbone algorithm). If not, the readers would simply use FCT+confidence rather than FastFill, which is easier to use. However, the performance gain is marginal in the standard setting (0.6%p on ImageNet-1K and 0.2%p on Places-365 in top@1), which makes the overall novelty limited. There are more gains in disjoint setting (2.7%p in top@1 and 0.5%p in mAP on ImageNet-[500:]), but the improvements are still moderate and need to re-write the overall paper with more thorough analysis to argue it as a main claim as there are insufficient experiments. The experimental results are not sufficient to verify the need to employ additional uncertainty head and training objective and the responses do not resolve my main concerns, thus I keep my original rating.

[1] Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval,  Zhang et al, ICLR 2022
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an online backfilling method that learns a transformation from the older feature space to the newer one (which is trained separately), and a policy to order the samples that need to be backfilled first to keep the highest performance during this online process.
The authors provided thorough experiments and analysis (with additional benchmarks during response), including a comprehensive evaluation on public datasets with good improvements shown. In particular,  the proposed method significantly improves over RACT which is a relevant method

justification_for_why_not_higher_score: The work is rather specialized (about model update for image retrieval)
justification_for_why_not_lower_score: The paper is well written and can benefit certain group of researchers (esp. image retrieval)
Decision: Accept: poster