Non-Inherent Feature Compatible Learning

The need of Feature Compatible Learning ( FCL ) arises from many large scale retrieval-based applications , where updating the entire library of embedding vectors is expensive . When an upgraded embedding model shows potential , it is desired to transform the benefit of the new model without refreshing the library . While progresses have been made along this new direction , existing approaches for feature compatible learning mostly rely on old training data and classifiers , which are not available in many industry settings . In this work , we introduce an approach for feature compatible learning without inheriting old classifier and training data , i.e. , Non-Inherent Feature Compatible Learning . Our approach requires only features extracted by old model ’ s backbone and new training data , and makes no assumption about the overlap between old and new training data . We propose a unified framework for FCL , and extend it to handle the case where the old model is a black-box . Specifically , we learn a simple pseudo classifier in lieu of the old model , and further enhance it with a random walk algorithm . As a result , the embedding features produced by the new model can be matched with those from the old model without sacrificing performance . Experiments on ImageNet ILSVRC 2012 and Places365 data proved the efficacy of the proposed approach . 1 INTRODUCTION . In recent years , deep learning based methods achieved huge success in various of computer vision tasks , especially for visual searching since they could provide powerful feature representations . In a typical visual search system , the deployed deep learning model extracts the features of both gallery and query images as discriminate representations . During the retrieval stage , gallery images will be ranked based on their feature distances ( e.g . Euclidean distance ) to query images . In conventional approaches , the query and gallery features are generated by the same model . Once the deployed model of retrieval system is updated , the entire set of gallery features need to be ‘ backfilled ’ or ‘ reindexed ’ ( Shen et al. , 2020 ) . As time goes by , the gallery becomes extremely large and ‘ backfilling ’ could be a painful process since millions even billions of images need to be re-processed by the new model , which is computationally expensive . There has to be a new mechanism that processes gallery images and the query image with two different models , while still maintaining the retrieval accuracy . In other words , the new deployed model extracted features should be ‘ compatible ’ to the existing ones without sacrificing accuracy . Such feature compatible learning problem is also named as ‘ Backward-Compatible Training ’ ( Shen et al. , 2020 ) , or ‘ Asymmetric Metric Learning ’ ( Budnik & Avrithis , 2020 ) . Existing approaches for feature compatible learning assumed significant overlap between new and old training sets . In Shen et al . ( 2020 ) , the training set for new embedding model is a superset of the old set . In Budnik & Avrithis ( 2020 ) , the training set for large and small models is the same , which means obtaining new model in an incremental way is not possible . Besides , in Shen et al . ( 2020 ) , the classifier for old model is also needed for computing the influence loss , which is a strong requirement in real applications . As an example , a model deployed in a recommendation system as a black-box API takes images as input and returns the processed features , but the parameters of the model are not accessible . In addition , its classifier and training details are not available , neither does the formula of the loss function . This kind of setting is quite common for various practical reasons in search , recommendation , content understanding and review applications . To address the limitation , we propose an approach for non-inherent feature compatible learning , which only exploits the old model backbone and new training data . Despite the lack of old training data or old classifier , the new model extracts compatible features without sacrificing accuracy . The proposed approach has three contributions including : • Study and formulate the non-inherent setting of the FCL problem for the first time • Establish a baseline with a data-incremental approach , where performance degradation is prevented by regularizing the training process of the new model • Extend the baseline with a random walk algorithm that further improves accuracy The experiments conducted on several standard data sets validated the effectiveness of the proposed approach . 2 RELATED WORK . Our approach is most relevant to feature compatible learning ( Shen et al. , 2020 ; Budnik & Avrithis , 2020 ; Wang et al. , 2020 ) , which has drawn attention from the research community due to the increasing size of gallery sets , and the heavy workload of re-generating gallery features . 2.1 FEATURE COMPATIBLE LEARNING . Shen et al . ( 2020 ) first formulated the ‘ backward-compatible ’ problem by deriving influence loss from an empirical criterion , and solved it by utilizing the old model to regularize the optimization process . In Budnik & Avrithis ( 2020 ) , authors investigated the problem of asymmetric test , where the gallery images are represented by a teacher model and query images are represented by a student model . A pair-based metric for instance-level image retrieval was proposed to achieve the goal . In Wang et al . ( 2020 ) , authors proposed Residual Bottleneck Transformation ( RBT ) blocks for feature embedding transferring . Some previous works ( Li et al. , 2015 ; Yu et al. , 2018 ) discussed the connection between features that learned by different models . However , all methods mentioned above either can not achieve the model compatibility in a data-incremental way or need to exploit old classifier for training . Our proposed method could achieve compatibility without old training data or utilizing any old classifier . 2.2 INCREMENTAL LEARNING . Incremental learning and Life-long learning ( Rebuffi et al. , 2017 ; Li & Hoiem , 2017 ) approaches aimed to stabilize model predictions when updating with new training data . Different from feature compatible problem , incremental learning focuses on maintaining performance on ‘ old ’ classes after introducing ‘ new ’ ones . In Li & Hoiem ( 2017 ) , authors utilized knowledge distillation ( Hinton et al. , 2015 ) and teacher-student models for regularizing features on new data , where model distillation was used as a form of regularization when introducing new classes . In Rebuffi et al . ( 2017 ) , authors proposed to use old class centers to regularize the model learning when new classes were introduced . Most approaches of incremental learning focus on the stability of classifier output , while feature compatible learning try to solve the feature compatibility problem among different models . 2.3 EMBEDDING LEARNING . Our approach is also relevant to the embedding learning problem that optimizes a distance metric to improve discriminative power and robustness of embedded features . There were efforts on designing powerful network architectures ( He et al. , 2016 ; Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ) , discriminative loss functions ( Deng et al. , 2019 ; Wang et al. , 2018 ) , and model optimizers ( Kingma & Ba , 2015 ) , but few of them set FCL as a part of their objectives . 3 APPROACH . This section re-visits the basic criterion for feature compatible learning , and presents the formulation and solutions to the Non-Inherent Feature Compatible Learning problem . 3.1 FORMULATION AND CRITERION FOR FEATURE COMPATIBLE LEARNING . Following the FCL formulation in learning ( Shen et al. , 2020 ; Budnik & Avrithis , 2020 ) , an old embedding model φold trained on old training set Dold maps an image x to a feature vector f = φold ( x ) , f ∈ RKold , where Kold is the feature vector dimension of f . After a period of time , a new embedding model φnew trained on Dnew is obtained . The new embedding model φnew maps the image x into a feature vector f with dimension of Knew , where Kold and Knew are not necessarily equal . In our setting , as explained in Sec.1 , the old training setDold is disjoint with the new training set Dnew ( Dold ∩Dnew = ø ) . In addition the old classifier , training details , and the loss function are not available . The gallery set is defined as G. Since the amount of images in G could be extremely large , the images in G are usually presented as extracted feature vectors by an old embedding model , φold , to save the test time and a query image will be presented as a feature vector with the same embedding model . However , once the embedding model is updated , all the images in G must be presented by the new embedding model φnew , leading to an expensive refresh of the index . To avoid such computation , we need to make it feasible to directly compare G ’ s embeddings produced by φold with the query embedding by φnew . In Shen et al . ( 2020 ) , a strict criterion for feature backward compatible is defined as , d ( φnew ( xi ) , φold ( xj ) ) ≥ d ( φold ( xi ) , φold ( xj ) ) , ∀ ( i , j ) ∈ { ( i , j ) |yi 6= yj } . and , d ( φnew ( xi ) , φold ( xj ) ) ≤ d ( φold ( xi ) , φold ( xj ) ) , ∀ ( i , j ) ∈ { ( i , j ) |yi = yj } , ( 1 ) where the d ( · , · ) is the Euclidean distance metric , xi and xj are the ith query image and jth gallery image respectively , yi and yj are their labels . Such a strict criterion needs every query-gallery pair fulfills the distance requirement , which is not feasible . Alternatively , authors in Shen et al . ( 2020 ) proposed a empirical criterion as the following : M ( φnew , φold ) > M ( φold , φold ) . ( 2 ) where M is a evaluation metric for the corresponding retrieval test set , consisting of gallery set G and query set . M ( φnew , φold ) means to extract query features with new model φnew and gallery features with old model φold , which is defined as cross test . Such a criterion requires the cross test performance between the new and the old model must be better than the test using only the old model , which is defined as self test . To achieve the backward compatibility defined by Eq . 2 , authors in Shen et al . ( 2020 ) proposed as below an influence loss term to add into the loss function for regularizing the new embedding model training : LBCT ( wc , wφ ; Dnew ) = L ( wc , wφ ; Dnew ) + λL ( wc old , wφ ; Dold ) , ( 3 ) where L is the loss function ( normally a softmax-like loss ) , wc , wc old , and wφ denote the classifier weights of the new embedding model , the classifier weights of the old embedding model , and the weights of the new embedding model ’ s backbone , respectively . The influence loss is the second term in Eq . 3 , which fixes wc old and regularizes the feature output of φnew to be compatible with φold , generating relatively small loss on the old classifier . Meanwhile , the new training set Dnew is a superset of Dold . Different from the setting in Shen et al . ( 2020 ) , we do not assume there is overlap between the old and new embedding training set Dold and Dnew . Moreover , the parameters of old classifier wc old are not available , which means we need to make the feature compatibility happen in a incremental way , without knowing the old classifier .

Summary: This work proposes a new problem setting by adding extra constraints to the Feature Compatible Learning problem. The new constraints avoid using old training data and the old model’s parameter when learning a new model. The paper gives a baseline method and its variants for the problem by generating pseudo classifiers to regularize a new model’s learning. The experiments show that the proposed method can satisfy the empirical criterion about success.