Hot-Refresh Model Upgrades with Regression-Free Compatible Training in Image Retrieval

1 INTRODUCTION . With the rapid development of deep learning , image retrieval algorithms have shown great success in large-scale industrial applications , e.g. , face recognition ( Sun et al. , 2014a ; b ) , object re-identification ( Li et al. , 2014 ; Zheng et al. , 2015 ) , image localization ( Arandjelovic et al. , 2016 ) . Model upgrades are essential to improve user experience in those applications . By properly leveraging massive training data and improved network architectures , the retrieval performances could be well boosted . Generally , once the model updates , the process of “ backfilling ” or “ re-indexing ” is called for offline re-encoding the gallery image features with the new model . The retrieval system can only benefit from the new model after the gallery is overall backfilled , which may cost weeks or even months for billions of industry images in practice . The above process is termed cold-refresh model upgrades . To harvest the reward of the new model immediately , Shen et al . ( 2020 ) introduces the compatible representation learning to train the new model with backward compatibility constraints , so that queries encoded by the new model can be directly indexed by the old gallery features . Meanwhile , as the new features and old features are interchangeable with each other , the gallery images can be backfilled on-the-fly , and the retrieval performances would be gradually improved to approach the optimal accuracy of the new model , dubbed as hot-refresh model upgrades ( see Figure 1 ( a ) ) . Although existing compatible training methods ( Meng et al. , 2021 ; Shen et al. , 2020 ) make it possible to upgrade the model in a hot-refresh manner , they still face the challenge of model regression ( Yan et al. , 2020 ; Li & Hoiem , 2018 ) , which is actually caused by negative flips , i.e. , queries correctly indexed by the old model are incorrectly recognized by the new model ( see Figure 1 ( b ) ) . We claimed that , during the procedure of hot-refresh model upgrades , the negative flips occur when * Throughout this paper , the term model regression indicates the degradation of retrieval performance when the hot-refresh model upgrades proceed . the new-to-new similarities between negative query-gallery pairs are larger than the new-to-old similarities between compatible positive query-gallery pairs . To solve the problem , we introduce a Regression-Alleviating Compatible Training ( RACT ) method to effectively reduce the negative flips for compatible representation learning and further improve the hot-refresh model upgrades with faster and smoother accuracy convergence in image retrieval systems . Specifically , the conventional backward-compatible regularization is generally designed to pull the new-to-old positive pairs closer and push the new-to-old negative pairs apart from each other . To reduce negative flips during the online gallery backfilling , we improve the backward-compatible regularization by enforcing the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs . To further improve the user experience during the model upgrades , we introduce an uncertainty-based hot-refresh backfilling strategy , which reindexes the gallery images following “ the-poor-first ” policy . To measure the refresh priority of gallery images in a lightweight manner , we feed the old gallery features into the new classifier ( one fully-connected layer in general ) , yielding the probability distribution on new classes . Even though ground-truth classes may not exist for the old gallery features , we can assume that more discriminative old features are expected to have sharper probability vectors given their compatible latent space . Uncertainty estimation ( e.g. , Entropy-based ) is utilized here to evaluate the sharpness of the probability vectors , and gallery images with larger uncertainty should have higher priority for backfilling . Experimental results on large-scale image retrieval benchmarks ( i.e. , Google Landmark v2 ( Weyand et al. , 2020 ) , RParis , and ROxford ( Radenović et al. , 2018 ) ) demonstrate the effectiveness of our RACT method in alleviating the problem of model regression for hot-refresh model upgrades . Our contributions can be summarized three-fold . ( 1 ) We study the model regression problem in hot-refresh model upgrades of image retrieval systems for the first time . ( 2 ) A regression-alleviating backward-compatible regularization is proposed to alleviate the issue of negative flips during compatible training . ( 3 ) An uncertainty-based backfilling strategy is introduced to further improve the user experience with “ the-poor-first ” hot refresh . 2 RELATED WORKS . Compatible Representation Learning . The task of compatible representation learning aims at encoding new features that are interoperable with the old features . It breaks the inherent cold-refresh way of model iteration in the industry - the benefit of new models can be immediately reaped in a backfill-free manner . Shen et al . ( 2020 ) first formulated the problem of backward-compatible learning and proposed to utilize the old classifier for supervising the compatible feature learning . Budnik & Avrithis ( 2020 ) made an effort to reduce the gap between the large model and a small one without the parametric classifier . Wang et al . ( 2020 ) introduced a projection head to transform the old features to be compatible with the new ones . Li et al . ( 2020b ) proposed a framework , namely “ Feature Lenses ” , to encourage image representations transformation-invariant . To balance the trade-off between performance and efficiency , Duggal et al . ( 2021 ) designed a compatibility-aware neural architecture search scheme to improve the compatibility of models with different sizes . However , since existing compatible algorithms for image retrieval have not investigated the application of hot-refresh model upgrades , the problem of model regression has been overlooked . Model Regression . Model upgrades play a vital role in improving the performance of industrial applications , where the issue of model regression inevitably harms the user experience . Yan et al . ( 2020 ) studied the problem of model regression in image classification , where the negative flips indicate the images correctly classified by the old model but misclassified by the new model . A method , namely focal distillation , was introduced to encourage model congruence on positive samples . Srivastava et al . ( 2020 ) also investigated the model regression in classification and proposed a new metric to evaluate the regression effects . Xie et al . ( 2021 ) probed the model regression problem in natural language processing . The works mentioned above all focused on the classification tasks . Directly employing their methods on the retrieval tasks leads to sub-optimal performances . Moreover , they did not need to consider hot-refresh gallery backfilling in image classification . Active Learning . Our uncertainty-based backfilling strategy is related to active learning . As the real world is teeming with unlabeled data , active learning makes efforts to achieve better performance with fewer annotated samples by choosing valuable samples actively ( Settles , 2009 ) . Active learning has been widely adopted in deep learning tasks , e.g. , object detection ( Holub et al. , 2008 ) , text classification ( Zhu et al. , 2008 ) , and image classification ( Cao et al. , 2020 ) . Though obtaining the priority of samples is a common issue between active learning and gallery backfilling , the latter focuses more on enjoying benefits in model upgrades with minimal computational overhead . Image Retrieval . The task of image retrieval requires searching images of the same content or object from a large-scale database based on their feature similarities , given images of interest . It is a fundamental task for many visual applications , e.g. , face recognition ( Guillaumin et al. , 2009 ; Cui et al. , 2013 ) , person re-identification ( Bak & Carr , 2017 ; Liu et al. , 2017 ) , image localization ( Arandjelovic et al. , 2016 ; Ge et al. , 2020b ) . Deep learning-based methods improve the accuracy of image retrieval by casting it as metric learning tasks ( Gordo et al. , 2016 ; Chen et al. , 2018 ; Brown et al. , 2020 ) , architecture design/search tasks ( Zhou et al. , 2019 ) , or self-/semi-supervised learning tasks ( Ge et al. , 2020c ; a ) . Although many efforts have been made to improve retrieval performance , few people pay attention to the bottleneck of model upgrades in industrial applications . 3 HOT-REFRESH MODEL UPGRADES OF IMAGE RETRIEVAL SYSTEMS . Given images of interest ( referred to as query Q ) , image retrieval targets correctly recognizing images of the same content or objects from a large-scale database ( referred to as gallery G ) . An image encoder φ ( · ) is employed to map the images into feature vectors . Then , the dot product of normalized vectors is used for measuring the similarity between query and gallery images for ranking . 3.1 HOT-REFRESH MODEL UPGRADES WITH COMPATIBLE REPRESENTATIONS . Let φold to be the old image encoder that was deployed in the retrieval system and φnew to be a stronger new encoder which is expected to replace the old one . We denote the evaluation metric for image retrieval , e.g. , mean Average Precision ( mAP ) , asM ( · , · ) . The accuracy of the old system is denoted asM ( Qold , Gold ) , where Qold , Gold are the sets of query and gallery features embedded by φold , respectively . The new encoder shows an improved accuracy ofM ( Qnew , Gnew ) . In conventional cold-refresh model updating pipeline , offline backfilling the gallery images from Gold to Gnew may take months for the massive data in industry applications , reducing the efficiency of the retrieval system upgrades to a large extent . Thanks to the compatible representation learning ( Shen et al. , 2020 ) , query features encoded by the new encoder ( i.e. , Qnew ) can be directly compared to the old gallery features ( i.e. , Gold ) . The retrieval system can benefit from the new encoder immediately in a backfill-free manner , significantly accelerating the model iteration at the expense of the optimal performance , M ( Qold , Gold ) < M ( Qnew , Gold ) < M ( Qnew , Gnew ) . ( 1 ) Given that the new features and old features are interchangeable due to their compatibility , the gallery images can be backfilled on-the-fly to progressively improve the retrieval accuracy to achieve the optimal performance of the new model , M ( Qnew , Gold ) →· · · →M ( Qnew , G20 % new ∪ G80 % old ) → · · · →M ( Qnew , G50 % new ∪ G50 % old ) → · · · →M ( Qnew , G80 % new ∪ G20 % old ) → · · · →M ( Qnew , Gnew ) . ( 2 ) The above process is termed as hot-refresh model upgrades , where G20 % new indicates the 20 % of images in the gallery have been re-indexed by the new model φnew . 3.2 MODEL REGRESSION WITH NEGATIVE FLIPS . Ideally , the performance of the retrieval system should be smoothly upgraded along with the gallery online backfilling , and the user experience would be continuously improved . However , we observe the phenomenon of model regression especially in the early backfilling stages , that is , the accuracy degrades when part of the gallery images are refreshed , e.g. , M ( Qnew , G20 % new ∪ G80 % old ) < M ( Qnew , Gold ) . ( 3 ) It is actually due to the negative flips , where some query images correctly retrieved by the old model are incorrectly indexed by the new model during the upgrade process . There are two main potential factors that cause the negative flips : ( 1 ) new-to-old query-gallery positive pairs are less similar than new-to-new query-gallery negative pairs ; ( 2 ) new-to-new positives are less similar than new-to-old negative pairs . Given that the new model generally encodes more discriminative features than the old model for the overall performance gains , we focus on the former one in this paper , i.e. , there exists a query xq that satisfies 〈φnew ( xq ) , φold ( xposg ) 〉 < 〈φnew ( xq ) , φnew ( xnegg ) 〉 , ( 4 ) where xposg is the positive gallery sample , x neg g is the negative gallery sample , and 〈· , ·〉 denotes the cosine similarity of two normalized feature vectors . In the case of Eq . ( 3 ) , φnew ( xq ) ∈ Qnew , φold ( x pos g ) ∈ G80 % old and φnew ( x neg g ) ∈ G20 % new .

Summary: The paper presents a method to learn compatible features with low negative flips rate. The paper also introduces a resampling strategy to gradually replace the gallery to ameliorate the performance. Resampling is based on feature uncertainty. The less discriminative a feature is, the sooner it is replaced.