Hot-Refresh Model Upgrades with Regression-Free Compatible Training in Image Retrieval

1 INTRODUCTION . With the rapid development of deep learning , image retrieval algorithms have shown great success in large-scale industrial applications , e.g. , face recognition ( Sun et al. , 2014a ; b ) , object re-identification ( Li et al. , 2014 ; Zheng et al. , 2015 ) , image localization ( Arandjelovic et al. , 2016 ) . Model upgrades are essential to improve user experience in those applications . By properly leveraging massive training data and improved network architectures , the retrieval performances could be well boosted . Generally , once the model updates , the process of “ backfilling ” or “ re-indexing ” is called for offline re-encoding the gallery image features with the new model . The retrieval system can only benefit from the new model after the gallery is overall backfilled , which may cost weeks or even months for billions of industry images in practice . The above process is termed cold-refresh model upgrades . To harvest the reward of the new model immediately , Shen et al . ( 2020 ) introduces the compatible representation learning to train the new model with backward compatibility constraints , so that queries encoded by the new model can be directly indexed by the old gallery features . Meanwhile , as the new features and old features are interchangeable with each other , the gallery images can be backfilled on-the-fly , and the retrieval performances would be gradually improved to approach the optimal accuracy of the new model , dubbed as hot-refresh model upgrades ( see Figure 1 ( a ) ) . Although existing compatible training methods ( Meng et al. , 2021 ; Shen et al. , 2020 ) make it possible to upgrade the model in a hot-refresh manner , they still face the challenge of model regression ( Yan et al. , 2020 ; Li & Hoiem , 2018 ) , which is actually caused by negative flips , i.e. , queries correctly indexed by the old model are incorrectly recognized by the new model ( see Figure 1 ( b ) ) . We claimed that , during the procedure of hot-refresh model upgrades , the negative flips occur when * Throughout this paper , the term model regression indicates the degradation of retrieval performance when the hot-refresh model upgrades proceed . the new-to-new similarities between negative query-gallery pairs are larger than the new-to-old similarities between compatible positive query-gallery pairs . To solve the problem , we introduce a Regression-Alleviating Compatible Training ( RACT ) method to effectively reduce the negative flips for compatible representation learning and further improve the hot-refresh model upgrades with faster and smoother accuracy convergence in image retrieval systems . Specifically , the conventional backward-compatible regularization is generally designed to pull the new-to-old positive pairs closer and push the new-to-old negative pairs apart from each other . To reduce negative flips during the online gallery backfilling , we improve the backward-compatible regularization by enforcing the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs . To further improve the user experience during the model upgrades , we introduce an uncertainty-based hot-refresh backfilling strategy , which reindexes the gallery images following “ the-poor-first ” policy . To measure the refresh priority of gallery images in a lightweight manner , we feed the old gallery features into the new classifier ( one fully-connected layer in general ) , yielding the probability distribution on new classes . Even though ground-truth classes may not exist for the old gallery features , we can assume that more discriminative old features are expected to have sharper probability vectors given their compatible latent space . Uncertainty estimation ( e.g. , Entropy-based ) is utilized here to evaluate the sharpness of the probability vectors , and gallery images with larger uncertainty should have higher priority for backfilling . Experimental results on large-scale image retrieval benchmarks ( i.e. , Google Landmark v2 ( Weyand et al. , 2020 ) , RParis , and ROxford ( Radenović et al. , 2018 ) ) demonstrate the effectiveness of our RACT method in alleviating the problem of model regression for hot-refresh model upgrades . Our contributions can be summarized three-fold . ( 1 ) We study the model regression problem in hot-refresh model upgrades of image retrieval systems for the first time . ( 2 ) A regression-alleviating backward-compatible regularization is proposed to alleviate the issue of negative flips during compatible training . ( 3 ) An uncertainty-based backfilling strategy is introduced to further improve the user experience with “ the-poor-first ” hot refresh . 2 RELATED WORKS . Compatible Representation Learning . The task of compatible representation learning aims at encoding new features that are interoperable with the old features . It breaks the inherent cold-refresh way of model iteration in the industry - the benefit of new models can be immediately reaped in a backfill-free manner . Shen et al . ( 2020 ) first formulated the problem of backward-compatible learning and proposed to utilize the old classifier for supervising the compatible feature learning . Budnik & Avrithis ( 2020 ) made an effort to reduce the gap between the large model and a small one without the parametric classifier . Wang et al . ( 2020 ) introduced a projection head to transform the old features to be compatible with the new ones . Li et al . ( 2020b ) proposed a framework , namely “ Feature Lenses ” , to encourage image representations transformation-invariant . To balance the trade-off between performance and efficiency , Duggal et al . ( 2021 ) designed a compatibility-aware neural architecture search scheme to improve the compatibility of models with different sizes . However , since existing compatible algorithms for image retrieval have not investigated the application of hot-refresh model upgrades , the problem of model regression has been overlooked . Model Regression . Model upgrades play a vital role in improving the performance of industrial applications , where the issue of model regression inevitably harms the user experience . Yan et al . ( 2020 ) studied the problem of model regression in image classification , where the negative flips indicate the images correctly classified by the old model but misclassified by the new model . A method , namely focal distillation , was introduced to encourage model congruence on positive samples . Srivastava et al . ( 2020 ) also investigated the model regression in classification and proposed a new metric to evaluate the regression effects . Xie et al . ( 2021 ) probed the model regression problem in natural language processing . The works mentioned above all focused on the classification tasks . Directly employing their methods on the retrieval tasks leads to sub-optimal performances . Moreover , they did not need to consider hot-refresh gallery backfilling in image classification . Active Learning . Our uncertainty-based backfilling strategy is related to active learning . As the real world is teeming with unlabeled data , active learning makes efforts to achieve better performance with fewer annotated samples by choosing valuable samples actively ( Settles , 2009 ) . Active learning has been widely adopted in deep learning tasks , e.g. , object detection ( Holub et al. , 2008 ) , text classification ( Zhu et al. , 2008 ) , and image classification ( Cao et al. , 2020 ) . Though obtaining the priority of samples is a common issue between active learning and gallery backfilling , the latter focuses more on enjoying benefits in model upgrades with minimal computational overhead . Image Retrieval . The task of image retrieval requires searching images of the same content or object from a large-scale database based on their feature similarities , given images of interest . It is a fundamental task for many visual applications , e.g. , face recognition ( Guillaumin et al. , 2009 ; Cui et al. , 2013 ) , person re-identification ( Bak & Carr , 2017 ; Liu et al. , 2017 ) , image localization ( Arandjelovic et al. , 2016 ; Ge et al. , 2020b ) . Deep learning-based methods improve the accuracy of image retrieval by casting it as metric learning tasks ( Gordo et al. , 2016 ; Chen et al. , 2018 ; Brown et al. , 2020 ) , architecture design/search tasks ( Zhou et al. , 2019 ) , or self-/semi-supervised learning tasks ( Ge et al. , 2020c ; a ) . Although many efforts have been made to improve retrieval performance , few people pay attention to the bottleneck of model upgrades in industrial applications . 3 HOT-REFRESH MODEL UPGRADES OF IMAGE RETRIEVAL SYSTEMS . Given images of interest ( referred to as query Q ) , image retrieval targets correctly recognizing images of the same content or objects from a large-scale database ( referred to as gallery G ) . An image encoder φ ( · ) is employed to map the images into feature vectors . Then , the dot product of normalized vectors is used for measuring the similarity between query and gallery images for ranking . 3.1 HOT-REFRESH MODEL UPGRADES WITH COMPATIBLE REPRESENTATIONS . Let φold to be the old image encoder that was deployed in the retrieval system and φnew to be a stronger new encoder which is expected to replace the old one . We denote the evaluation metric for image retrieval , e.g. , mean Average Precision ( mAP ) , asM ( · , · ) . The accuracy of the old system is denoted asM ( Qold , Gold ) , where Qold , Gold are the sets of query and gallery features embedded by φold , respectively . The new encoder shows an improved accuracy ofM ( Qnew , Gnew ) . In conventional cold-refresh model updating pipeline , offline backfilling the gallery images from Gold to Gnew may take months for the massive data in industry applications , reducing the efficiency of the retrieval system upgrades to a large extent . Thanks to the compatible representation learning ( Shen et al. , 2020 ) , query features encoded by the new encoder ( i.e. , Qnew ) can be directly compared to the old gallery features ( i.e. , Gold ) . The retrieval system can benefit from the new encoder immediately in a backfill-free manner , significantly accelerating the model iteration at the expense of the optimal performance , M ( Qold , Gold ) < M ( Qnew , Gold ) < M ( Qnew , Gnew ) . ( 1 ) Given that the new features and old features are interchangeable due to their compatibility , the gallery images can be backfilled on-the-fly to progressively improve the retrieval accuracy to achieve the optimal performance of the new model , M ( Qnew , Gold ) →· · · →M ( Qnew , G20 % new ∪ G80 % old ) → · · · →M ( Qnew , G50 % new ∪ G50 % old ) → · · · →M ( Qnew , G80 % new ∪ G20 % old ) → · · · →M ( Qnew , Gnew ) . ( 2 ) The above process is termed as hot-refresh model upgrades , where G20 % new indicates the 20 % of images in the gallery have been re-indexed by the new model φnew . 3.2 MODEL REGRESSION WITH NEGATIVE FLIPS . Ideally , the performance of the retrieval system should be smoothly upgraded along with the gallery online backfilling , and the user experience would be continuously improved . However , we observe the phenomenon of model regression especially in the early backfilling stages , that is , the accuracy degrades when part of the gallery images are refreshed , e.g. , M ( Qnew , G20 % new ∪ G80 % old ) < M ( Qnew , Gold ) . ( 3 ) It is actually due to the negative flips , where some query images correctly retrieved by the old model are incorrectly indexed by the new model during the upgrade process . There are two main potential factors that cause the negative flips : ( 1 ) new-to-old query-gallery positive pairs are less similar than new-to-new query-gallery negative pairs ; ( 2 ) new-to-new positives are less similar than new-to-old negative pairs . Given that the new model generally encodes more discriminative features than the old model for the overall performance gains , we focus on the former one in this paper , i.e. , there exists a query xq that satisfies 〈φnew ( xq ) , φold ( xposg ) 〉 < 〈φnew ( xq ) , φnew ( xnegg ) 〉 , ( 4 ) where xposg is the positive gallery sample , x neg g is the negative gallery sample , and 〈· , ·〉 denotes the cosine similarity of two normalized feature vectors . In the case of Eq . ( 3 ) , φnew ( xq ) ∈ Qnew , φold ( x pos g ) ∈ G80 % old and φnew ( x neg g ) ∈ G20 % new .

Summary: The paper proposes a new technique for updating image retrieval models while not requiring full backfill of the index in order to allow deployment. This has the potential to make model upgrades more practical for real-world large-scale applications. The paper is the first to study the impact of such “hot-refresh” techniques to the image retrieval domain. The authors make observations about the reason behind negative flips, which motivate their newly-proposed loss function / training technique. The paper also proposes an uncertainty-based backfilling method to decide the ordering in which images in the index are backfilled. Experiments are conducted on standard image retrieval datasets.