Non-Inherent Feature Compatible Learning
Authors: Yantao Shen
Fanzi Wu
Ying Shan
Venue:  (2021)
Keywords: Deep Learning
Feature Learning
Compatible Learning

Abstract: The need of Feature Compatible Learning (FCL) arises from many large scale retrieval-based applications, where updating the entire library of embedding vectors is expensive. When an upgraded embedding model shows potential, it is desired to transform the benefit of the new model without refreshing the library. While progresses have been made along this new direction, existing approaches for feature compatible learning mostly rely on old training data and classifiers, which are not available in many industry settings. In this work, we introduce an approach for feature compatible learning without inheriting old classifier and training data, i.e., Non-Inherent Feature Compatible Learning. Our approach requires only features extracted by \emph{old} model's backbone and \emph{new} training data, and makes no assumption about the overlap between old and new training data. We propose a unified framework for FCL, and extend it to handle the case where the old model is a black-box.  Specifically, we learn a simple pseudo classifier in lieu of the old model, and further enhance it with a random walk algorithm.  As a result, the embedding features produced by the new model can be matched with those from the old model without sacrificing performance. Experiments on ImageNet ILSVRC 2012 and Places365 data proved the efficacy of the proposed approach.

Under review as a conference paper at ICLR 2021
FEATURE COMPATIBLE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The need of Feature Compatible Learning (FCL) arises from many large scale retrieval-based applications, where updating the entire library of embedding vec- tors is expensive. When an upgraded embedding model shows potential, is desired transform the benefit of the new model without refreshing the library. While progresses have been made along this new direction, existing approaches for feature compatible learning mostly rely old training data and classifiers, which are not available in many industry settings. In this work, we introduce approach for feature compatible learning without inheriting old classifier and training data, i.e., Non-Inherent Feature Compatible Learning. Our approach re- quires only features extracted by old model's backbone and new training data, and makes no assumption about the overlap between old and new training data. We propose a unified framework for FCL,, and extend it to handle the case where the old model is black-box. Specifically, we learn simple pseudo classifier in licu of the old model, and further enhance it with a random walk algorithm. As result, the embedding features produced by the new model can be matched with those from the old model without sacrificing performance. Experiments on ImageNet ILSVRC 2012 and Places365 data proved the efficacy of the proposed approach.
INTRODUCTION
In recent years, deep learning based methods achieved huge success in various of computer vision tasks, especially for visual searching since they could provide powerful feature representations. In typical visual search system, the deployed deep learning model extracts the features of both gallery and query images as discriminate representations. During the retrieval stage, gallery images will be ranked based on their feature distances (e-g. Euclidean distance) to query images. approaches, the query and gallery features are generated by the same model. Once the deployed model of retrieval system is updated, the entire set gallery features need to be 'backfilled' or 're- indexed' (Shen et al. 2020). As time goes by, the gallery becomes extremely large and 'backfilling' could be painful process since millions even billions of images need to be re-processed by the new model, which is computationally expensive. There has to be new mechanism that processes gallery images and the query image with two different models, while still maintaining the retrieval accuracy. In other words, the new deployed model extracted features should be 'compatible" the existing ones without sacrificing accuracy. Such feature compatible learning problem is also named Blackward-Compatiin Training' (Shen al. 2020), or 'Asymmetric Metric Learning' (Budnik
conventional
& &Avrithis 2020).
Existing approaches for feature compatible learning assumed significant overlap between new and old training sets. In Shhnn et al.] (2020), the training set for new embedding model is superset of the old set. In Budnik & Avrithis (202)). the training set for large and small models is the same, which means obtaining new model in an incremental way is not possible. Besides, in (Shen et al. (2000), the classifier for old model also needed for computing the influence loss, which is a strong requirement in real applications. As an example, model deployed in a recommendation system as a box API takes images as input and returns the processed features, but the parameters of the model are not accessible. In addition, its classifier and training details are not available, neither does the formula of the loss function. This kind of setting is quite common for various practical reasons in search, recommendation, content understanding and review applications.

---

Under review as a conference paper at ICLR 2021
To address the limitation, propose approach for non-inherent feature compatible learning, which only exploits the old model backbone and new training data. Despite the lack of old training data or old classifier, the model extracts compatible features without sacrificing accuracy. The proposed approach has three contributions including:
Study and formulate the non-inherent setting of the FCL problem for the first time
Establish baseline with a inttminccementll approach, where performance degradation is prevented by regularizing the training process of the new model
Extend the baseline with random walk algorithm that further improves accuracy
The experiments conducted on several standard data sets validated the effectiveness of the proposed approach.
2
RELATED WORK
Our approach is most relevant to feature compatible learning (Shen et al. 2020 Budnik & Avrithis 2020 Wang al., 2020). which has drawn attention from the research community due the creasing size of gallery sets, and the heavy workload of re-generating gallery features.
2.1
FEATURE COMPATIBLE LEARNING
Shen et al. (2020) first formulated the Baccwarr-compatible problem by deriving influence loss from an empirical criterion, and solved by utilizing the old model to regularize the optimization process.
In Budnik Avrithis (2020), authors investigated the problem of asymmetric test, where the gallery images represented by teacher model and query images represented by student model. A pair-based metric for instance-level image retrieval was proposed to achieve the goal. In Wang et al. 2020), authors proposed Residual Bottleneck Transformation (RBT) blocks for feature embedding transferring. Some previous works (Li et al. 2015 Yu et al. 2018) discussed the connection be- tween features that learned by different models. However, all methods mentioned above either not achieve the model compatibility data-incremental way or need to exploit old classifier for training. Our proposed method could achieve compatibility without old training data utilizing any old classifier.
2.2 INCREMENTAL. LEARNING
Incremental learning and Life-long learning (Rebuffi et al. 2017, Li & Hoiem 2017) approaches aimed stabilize model predictions when updating with new training data. Different from feature compatible problem, incremental learning focuses on maintaining performance on 'old' classes after introducing 'new' ones. [n & Hoiem (2017), authors utilized knowledge distillation (Hinton al.) 2015) and teacher-student models for geguaariing features on new data, where model distillation was used as a form of regularization when introducing new classes. In Rebuffi et al. (2017). authors proposed to use old class centers to regularize the model learning when new classes were introduced. Most approaches of incremental learning focus on the stability of classifier output, while feature compatible learning try solve the feature compatibility problem among different models.
2.3 EMBEDDING LEARNING
Our approach is also relevant to the embedding learning problem that optimizes a distance metric to improve discriminative power and robustness embedded features. There were efforts on de- signing powerful network architectures (He et al., 2016 Simonyan & Zisserman, 2014 Szegedy ctt al. 2015). discriminative loss functions (Deng et al. 2019 Wang et (018)), and model opti- mizers (Kingma & Ba 2015), but few of them set FCL part of their objectives.
2

---

Under review as a conference paper at ICLR 2021
3 APPROACH
This section re-visits the basic criterion for feature compatible learning, and presents the formulation and solutions to the non-Inherent Feature Compatible Learning problem.
3.1 FORMULATION AND CRITERION FOR FEATURE COMPATIBLE LEARNING
Following the FCL formulation in learning (Shen et al. 2020 Budnik & Avrithis 2020). an old embedding model Gold trained on old training set Dold maps an image to a feature vector f (.). RKou,, where Kold is the feature vector dimension of f. After period of time, new embedding model &new trained on Dnew is obtained. The new embedding model Onew maps the image x into feature vector with dimension of keww, where Kola and K new are not necessarily equal. In our setting, as explained in S the old training set Dold is disjoint with the new training set D Dew (Doia Dnew = ö). In addition the old classifier, training details, and the loss function are not available.
The gallery set is defined as G. Since the amount of images in G could be extremely large, the images in G are usually presented as extracted feature vectors by old embedding model, oold, to save the test time and a query image will be presented as a feature vector with the same embedding model. However,
the embedding model is updated, all the images in & must be presented by the new embedding model &new. leading to an expensive refresh of the index. To avoid such computation, we need make feasible to directly compare G's embeddings produced by Pold with the query embedding by nnww
In Shen et al. (2020), a strict criterion for feature backward compatible is defined as,
((Qnew((r)) (x,))
(x)) v(i,)) E (i..j)l9: # vz}.
and, d(Quew(Z)), (I,) < d(éola(Z;), (ola(1z))) v(i,)) ((i.j)lu: -yg
where the d(.,,) -) is the Euclidean distance metric, v, and zj are the ,th query image and 5th gallery image respectively, and y/ are their labels. Such a strict criterion needs every query-gallery pair fulfills the distance requirement, which not feasible. Alternatively, authors in [hen al. (2020) proposed a empirical criterion as the following:
M(Oeew dota) > M(oodd dola).
(2)
where M is evaluation metric for the corresponding retrieval test set, consisting of gallery set G and query set. M (onew; dola) means to extract query features with new model Quew and gallery features with old model dold, which is defined as cross test. Such criterion requires the cross test performance between the new and the old model must be better than the test using only the old model, which is defined as self test.
To achieve the backward compatibility defined by Eq. D. authors in (Shnn al. (0200) proposed as below influence loss term to add into the loss function for regularizing the new embedding model training:
LBCT (we, Wo: Dnew) L(we, wT: Dnew) AL old, Wsi Dota),
(3)
where L is the loss function (normally softmax-liks loss), We, We old, and wy denote the classifier weights of the new embedding model, the classifier weights of the old embedding model, and the weights of the new embedding model's backbone, respectively. The influence loss is the second term in Eq. 3] which fixes we old and regularizes the feature output of &new to be compatible with Gold. generating relatively small loss on the old classifier. Mcanwhile, the new training set Dnew is superset of Dota-
al. (020), we do not assume there is overlap between the old and new embedding training set Dold and Dnew Moreover, the parameters of old classifier we old are not available, which means we need to make the feature compatibility happen in a incremental way, without knowing the old classifier.
Different from the setting in Shen et
3

---

Under review as a conference paper at ICLR 2021
treature Extract
Embedding Model
New Training
Psaudo Old Clessifier
-Feature Learn
New Embeddlin Model
Pseudo Old Classifier (lixed)
New Training Dara
New Classifier
Figure I:
Illustration of using averaged features extracted by old model to serve as pseudo classifier. During the training, this pseudo classifier will be not updated by the gradient.
3.2 ACHIEVIIG NON-INIERENT FEATURE COMPATIBLE LEARNING
In this section, will introduce our proposed feature backward training scheme without using old model's classifier We old or old training data Dota-- Although we could not obtain the 'old' classifier on old training data, which essential for influence loss in Eq. we can still try to generate a ppseudo' old classifier. Inspired by Rebutti et al. (017)), Wu et al. (2018) and Xiao et al. (2017), we found that the classifier weights actually act feature embedding centers so that they can generate large score with positive images' features, which suggests that even simply averaging images features processed by dold each class, we could still obtain a reasonable classifier weight for that class, which could be denoted as,
w a- (n)
m7 Σ
Dord (IEs))
where is the image amount in th class, is the i-th image in that class, Pold is the old embedding model. wa-pe (n) could be served as n-th column in the pscudo classifier. In practice, we found that using averaged feature vectors as classifier may not converge, classifier consisted of normed feature vector could lead to much better result,
an- pae (n) w (n) pse
w old
(5)
Iml old (n)ll
Another advantage of the proposed approach is that the old training set is not compulsory to use. The 'pseudo' classifier for Dnew could be generated with the old embedding model easily, no data in Dola is required, then the loss function in Eq. becomes to
LBCT(We, Wa: Dnew) L(we, wai Dnew) AL( an- ws; Duew).
(6)
3.3 RANDOM WALK REFINEMENT FOR PSEUDO CLASSIFIER GENERATION
3.3.1
LIMITATION OF BASELINE APPROACH
The baseline method proposed Section 3.2 (5.2)xxz exploits the old embedding model and new training set for 'pseudo' classifier generation and it can provide us a reasonable result for non-inherent feature compatible learning without 'old' classifier. However, due to mis-labeling or low quality, there are some outlier images in each class, simply averaging the feature vectors extracted by old embedding could not eliminate their negative effects. this section, we will present our random walk approach for refined pseudo classifier generation.
4

---

Under review as a conference paper at ICLR 2021
Feeaure Extract RW Refimement
Old Embeciing Model
New Training Data
Old Classifier
New Embedding Model
RW- Pseudo
Old Classifier (fixed)
New Training Data
New Classifier
Figure 2:
Illustration of using random walk algorithm to refine features extracted by old model before averaged serve column of pseudo classifier. Negative effects of outliers in each class will be reduced. During the training, this pseudo classifier will be not updated by the gradient.
3.3.2 RANDOM WALK FOR CLASSIFIER GENERATION REFINEMENT
Random walk algorithm (Aldous) 1989) has been widely used for ranking system (Page et al. 1999)
or retrieval results refinement (Loy al., 2013 Bertasius et 2017). Random walk is operated on fully connected undirected graph G (V,E), E). where V is the vertice and E is the edge. The similarities between different vertices could be defined as a symmetric matrix S. Each element on the matrix, (i,,j), represents the similarity between vertices i and j (e.g. normalized cosine similarity).
In our context, in n-th class, i-th image's feature fai E RKK could be represented of images' features in one class can concatenated as matrix F, E IR Kmm where m is the image amount of n-th class. The edges, E, among vertices are the similarity scores between different vertices. With notations mentioned above, random walk operation could denoted as,
vertice V.
= F S,
where t is the iteration times. S mxx the normalized similarity matrix between different images, which is denoted as,
(Tp(S(L.i)T)) E (Ap(S 0, i=j
i i
S'i,,))
(8)
where lower temperature T leads to more concentrate probability distribution.
In practice, the refinement by random walk also needs to be weighted with the initial feature matrix F which is computed as,
= + +
(9)
where À E [0, 1] is the weight parameter.
If t tends infinity, the Eq. 9 has converaged close form, which is,
F = X)F XS')
(10)
where I is the identity matrix and (1 denotes matrix inverse operation.
Once the p is obtained, the w pw-nse (n) could be computed with column-wised average pooling

---

Under review as a conference paper at ICLR 2021
m/ Fw (t,
W old
pee
(n)
(11)
m
The refined classifier is also column-wised normalized like Eq. and the influence loss changes to,
5
which is denoted as w old
twn-pse pse
LBCT (We, We; )new) = L(we, Wai Dnew) XL
twn-pee old
Wig Dnww).
(12)
Compared with baseline method introduced in Section 3.1 our proposed random walk refinement approach incorperates more information among images' feature within one class to conduct the pseudo classifier generation, which could largely eliminate the outlier effects. In the experiments, we further show the effectiveness of the proposed method.
4 EXPERIMENTS
We validate the effectiveness of the proposed Non-Inherent Feature Compatible Learning approach on two large public datasets. In this section, We firstly demonstrate the datasets we use and then illustrate the baseline method performance and the effectiveness of the proposed random walk based method. Finally, we analysis the influence of hyperparameter choosing.
4.1 DATASETS AND METRIC
We utilize ImageNet ILSVRC (Deng et al. 2009) and Place365 (Zhou et al., 2017) datasets for training and evaluation. ImageNet dataset contains more than 1.2 million of images with 1,000 classes, Place365 contains about 1.8 million images with 365 classes. We conduct the retrieval process these two datasets' validation sets, which mcans each image will be considered as query image and all other images will be considered as gallery images. The Euclidean distance is used for ranking. The evaluation metric is Top-1 and Top-5 accuracies. We the final global averaged pooled feature (before feed into classifier) for the feature distance computing. When conducting the cross test, both old and new model will be utilized for extracting features and distance metric will be computed with old and new feature. If dimensions of old and new features are different, zero padding will be used for dimension alignment. Following the backward wardd-coompaaiiilit measuring setting in Shen et al. (2020), we consider backward compatibility achieved when the cross test accuracy using new model for queries and old model for galleries surpasses the self test result of only using old model.
4.2 TRAINING DETAILS
We use NVIDIA P40 GPUs for training. For each class on the training set, 30% of training data will be used for old model training and 70% data will be used for new model training. We adopt two widely used backbones, ResNet-18 and ResNet-50 (He al. 2016), serve old model and new model, the output feature dimensions of them are 512 and 2048 respectively. The image input size set to 224 224. Only random crop and random flip are used for data augmentation. We adopt standard stochastic gradient descent (SGD) to optimize the model parameters. The learning rate is set to 0.1 and decreases 10 times every 30 epochs and the training stops after 90 epochs. The weight decay is set to 10 and momentum is 0.9. The weights of two terms in Eq. 3 are equal. The batch size is set to 1024. Both old and new model are random initialized (from scratch). For random walk based refinement, set À to 0.9.
4.3 BASELINE AND PROPOSED APPROACHES ANALYSIS
In this section, we conduct several naive baseline approaches for feature compatible learning without old classifier and old training data. The cross test between old model and these approaches trained new models verified the effectiveness of the method. Besides, surprisingly, the self test of these proposed method even outperform the new model trained without any regularization.

---

Under review as a conference paper at ICLR 2021
Cross Test
Self Test
Old Model
New Model
ImageNet
Places365
ImageNet
Places365
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
Pold 30%
39.6
62.7
28.2
57.2
Pold 30%
new 70%
0.1
0.5
0.0
0.2
Oll 30%
L2
70%
39.4
62.5
28.2
57.3
Oold 30%
pse
30.6
57.6
30.7
60.3
70%
Ool 30%
new 70% Ours)
an- pae
55.0
77.6
36,9
63.8
Oold 30%
Ours)
56.2
77.8
37.4
64.0
new 70%
Puew 70%
58.4
78.7
33.8
62.4
Lig
41.1
64.1
29.0
58.3
new 70%
t- pee 70%
61.3
80.0
35.5
64.4
an- ppse (Ours)
61.9
80.3
38.9
65.7
new 70%
@ new 70%
(Ours)
62.2
80.4
38.2
65.9
Table 1: Baseline and proposed approaches comparison ImageNet Deng al. 20099) and Places365 Zhou et al. (2017) datasets. The self test results of Pold 30% is the lower bound of the comparison.
No regularization between &new and oold We firstly test a simple case which was verified in Shen al. (220)) directly comparing the isolatedly trained model (denoted as Pold 30%). As illustrated in Section 4.2 we use 30% of training data in each class train Pold 30% and 70% of data is used for training Pnew 70%- For aligning their output feature dimensionality, we directly use zero-padding for Oold 30%. 's feature output to pad it from 512 to 2048. The results on ImageNet and Places365 are shown Table such directly comparison an epic failure, which has already been verified in Shen et al. (020).
model (denoted as &new 70%) and old
L2 regression between σnew and Cold output feature simple baseline that could be proposed is using L2 loss to minimize the output features' Euclidean distance between 'old' and 'new" model, which is denoted as ø²2 70% in Table Such simple baseline could not meet the requirement of the feature compatible learning. The possible reason for L2 loss failure is that L2 loss only focuses on decreasing distance between feature pairs from the same image, however, it ignores the distance restriction between negative pairs.
Trained pseudo classifier with new data and dotd output feature Another baseline we compared using old model backbone to process new training data for obtaining the corresponding features, then these features will be fed into a linear layer. The output of this linear layer will be sent into softmax layer and optimized with cross cross-entropy loss and new training data label. The loss function could be denoted as,
L -=--- E log( -V
exp(w" T f((L,]) Σ, excl ADD
(13)
where is the i-th image's feature processed by dold 30% L; is the corresponding label, N is the new training data amount, w is the linear classifier weight.
With minimizing the loss denoted in Eq. 13
obtain linear classifier which produces relatively small loss with fixed old backbone Pold 30% on new training data, which could be used in influence loss (Eq. B to
as Wold. The 'new" model trained with such regularization is denoted test result in Table shows that although it is not total failure, but they can not beat the self test result of dold 30%, which is the basic requirement for feature compatible learning.
ot pse 70%. the

---

Under review as a conference paper at ICLR 2021
ImageNet
Old Model
New Model
T
top-1
top-5
1.0
0.1
55.7
77.8
1.0
0.5
55.9
77.9
Oold 30%
1.0
0.9
55.7
78.0
new 70%
0.1
0.9
56.1
77.9
0.05
0.9
56.2
78.1
Table 2: The hyperparameter analysis about softmax normalization temperature and random walk weight parameter.
Feature averaging pseudo classifier As illustrated in Section 3.2 fier with averaged feature representation of all images in each class on new training dataset. Then we utilize such pseudo classifier to regularize the new model training as shown in Eq. 6 The model is denoted as @ gan-pese peee new 70%% As illustrated in Table 1 such simple approach could actually satisfy the feature compatible learning requirement, which surpasses self test performance of using Pold 30% by 15.4% in terms of top-1 accuracy on ImageNet dataset and 8.7 % on Places365 dataset.
we could make pseudo classi-
Feature averaging pseudo classifier with random walk refinement Following the algorithm in- conduct random walk refinement for generating classifier, we set tem- rwn- pse perature parameter T' in Eq. to 0.05 and the À in Eq. 7 to 0.9. Then the refined classifier w old will be served as old classifier in Eq. 12 The trained new model with such regularization denoted pw pse As illustrated in Table the proposed random walk refinement approach not only fulfill the requirement of feature compatible training, but also outperforms all compared methods.
troduced in Section 3.3
as @ new 70%.
Better self test performance In Table we show that with our proposed method, the self test pat-pse results of e &new 70% and &hew 70% even surpass the performance of dnew 70% which means with the proposed approach, the feature could be improved with the information that provided by old model.
pss- -wn-ppee
4.4 HYPERPARAMETERS ANALYSIS FOR RANDOM WALK
Temperature 7 setting for softmax normalization We firstly investigate the influence of softmax normalization temperature in Eq. 8 High temperature in softmax function leads to a softer distribu- tion, while based on our analysis Section 3.3] there some noisy images in each class. We need to make the averaged feature more concentrate on the non-noisy images, so the temperature of the softmax should be low. As illustrated in Table such assumption verified. Lower temperature provides better results.
Effect of choosing different À Another investigation is about the choice of À in Eq. [0] larger means the larger weight for random walk refined feature and less for original feature. As shown in Table 2 larger À will provide slightly better performance.
CONCLUSION
In this paper, we investigate several approaches to achieve non-inherent feature compatibility, which aims at achieving feature compatibility without utilizing old training data and old classifier. We con- duct comparison experiments on two large public datasets. Compared with other simple baselines, approach achieved better performance with limited offline computing cost increasing, Further- more, the proposed method could even improve the feature learning and achieves better self test result compared with model trained without any feature compatible regularization. In the future, we would like to investigate more about the regularization scheme for filling up the accuracy gap between cross test and self test.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
David J Aldous. Lower bounds for covering times for reversible markov chains and random walks on graphs. Journal of Theoretical Probability, 2(1):91-100, 1989.
Gedas Bertasius, Lorenzo Torresani, Stella X Yu, and Jianbo Shi. Convolutional random walk networks for semantic image segmentation. In Proceedings the Vision and Pattern Recognition, pp. 858-866, 2017.
the IEEE Conference on Computer
Mateusz Budnik and Yannis Avrithis. Asymmetric metric learning for knowledge transfer. arXiv preprint arXiv:2006. 210231006.,33,,, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4690-4699, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.022 2015.
Diederik Kingma and Jimmy Lei Ba. Adam: A method for stochastic gradient descent. In ICLR: International Conference on Learning Representations, 2015.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hoperoft. Convergent learning: Do different neural networks learn the same representations? In NIPS, pp. 196-212, 2015.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on patlern analysis and machine intelligence, 40( 40(12):2935-2947 2017.
Chen Change Loy, Chunxiao Liu, and Shaogang Gong. Person re-identification by manifold ranking. In 2013 IEEE International Conference on Image Processing, pp. 3567-3571. IEEF, 2013.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
Sylvestre-Alvise Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward colppaall repre- sentation learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409. 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Chien-Y Yi Wang, Ya-Liang Chang, Shang-Ta Yang, Dong Chen, and Shang-Hong Lai. Unified rep- resentation learning for cross model compatibility. arXiv preprint arXiv:2008. (482), 2020.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5265-5274, 2018.

---

Under review as a conference paper at ICLR 2021
Zhirong Wu, Yuanjun Xiong. Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Tong Xiao, Shuang Li, Bochao Wang. Liang Lin, and Xiaogang Wang. Joint detection and identi- fication feature learning for person search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3415-3424, 2017.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2018.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
10


Reviews:
[{"review_id": "aLqm_ntatw5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis work proposes a new problem setting by adding extra constraints to the Feature Compatible Learning problem. The new constraints avoid using old training data and the old model’s parameter when learning a new model. The paper gives a baseline method and its variants for the problem by generating pseudo classifiers to regularize a new model’s learning. The experiments show that the proposed method can satisfy the empirical criterion about success.\n\n[Strengths]\n1. The constraints added to the feature compatible learning are sounded and may have practical value.\n2. The writing of this paper is satisfactory. The clarity is good and easy to follow. Both the related works, motivations, and technical details are clearly introduced.\n3. The experiment has a reasonable set of baselines for the problem. The ablation study clearly shows how the proposed enhancement contributes to performance gain.\n4. Section 3.3.2 is a new interesting way to combine the feature representation for a group of data.\n\n[Weaknesses]\n1. Figures 1 and 2 do not help understand the method’s overview nor the problem setting.\n2. The gain of the RWN classifier is not significant. There are other simple ways to remove the outliers for computing the mean representations. One example is to ignore the top 10 percent of the data based on the initial mean and variance, then re-compute the mean representation. The RWN should be compared with those simple baselines.\n3. There are several minor typos. Please have proofreading.\n\n[Questions]\n1. The gain of RWN looks small in Table 2. What is the variance of Table 2?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V36hKpLMIRe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses an interesting problem in retrieval system - compatible features learning. Given the old feature extractor and a new dataset, the objective is to learn a new feature extractor, so that the features extracted by two (old and new) feature extractors are comparable to each other. In the proposed setting, the old dataset (including its statistics), old classifier, and the parameters of the old model are not available.\n\nThe presentation of this paper is clear, including the problem description and basic idea of the methodology. The construction of the pseudo old classifier is well motivated and reasonable. The performance improvements are significant based on the presented results.\n\nHowever, I still have some concerns.\nFirst, it is unclear that why random walk provides benefit to obtaining a better feature matrix.\nBased on eq 10, each column of the new feature matrix (left side) is the linear combination of the columns of the old feature matrix. Subsequently, the feature matrix is averaged to obtain the pseudo classifier. So intuitively, the pseudo classifier is obtained by weighted averaging of the feature matrix, where the weights are learned by random walk from their similarity scores.\nGiven these facts, more explanations are needed for why random walk could produce the beneficial weights.\n\nSecond, the formulation of the loss function (L) is not given. (eq 3), which is an important detail.\n\nThird, there is no experimental comparison with other works. So it is hard to see the contribution of this paper.\nI understand that this is a novel problem, which may not have prior work to compare with. However, to make the results more convincing, the proposed methods can be used to other similar settings for evaluation, e.g. the setting in Shen et al. 2020.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MSJvsmBYeD8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with an interesting problem of feature compatible learning that the features produced by new model should be compatible with old features. The proposed method uses nearest class–mean classifier instead of linear classifier. Random walk is applied to refine the class means. The proposed method is compared with several baseline methods and shows good performance.\n\n\nPro:\n1.\tThis paper deals with an interesting problem of feature compatible learning. \n2.\tA new experimental setting is presented.\n3.\tA simple method based on nearest class–mean classifier is proposed and shows good performance.\n\nCons:\n1.\tSome incremental learning methods also deal with the feature compatible problem when old features are stored for rehearsal. The paper lacks the review of these methods.\n2.\tThe regularization term in Eq (6) and (12) is not defined. \n3.\tIt needs more details to explain the two important settings (cross test and self test) and the experimental protocols.\n4.\tAs the proposed method needs to compute the similarity graph for data in every class. How much extra time and memory does it take to implement the random walk refinement?\n5.\tMore experimental results, e.g. visualization, should be given.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2-ZsLE5i4dU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I spot a couple of key issues with the proposed method:\n\n1) Misinterpretation of the previous study by Shen et al. (CVPR 2020): As presented in Eq (8) of that CVPR paper, the dataset where the influence loss is applied is NOT necessarily the old training set; It  can be the new training dataset. Also, their results show that using new training set for the influence loss is slightly better than using old training set. This makes the statement that \"previous works rely on old training\" invalid. So Eq (3) of this submission is not complete and somewhat misleading.\n\n2) Shen et al. (CVPR 2020) only use the classifier but not the feature model. Although this proposed work does not use the old classifier, the feature model is instead used to estimate the classifier weights. So, it is not true to claim that \"the old model is a black-box\" for the proposed method; Just use a different part of the old system. Besides, in deep classification model, the feature vectors and classifier weights are closely related and both reveal similar amount of information.  In particular, if a cosine classifier is used, the two can be exchanged to each other. So, this proposed model is actually leveraging the same amount of information from the old system, as compared to Shen et al. (CVPR 2020).\n\nOverall, the two points above would make me to conclude that this work comes with fundamental flaws/drawbacks.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with a problem of feature compatible learning, where the features produced by new model should be compatible with old features. As pointed out by the reviewers, there are several weaknesses with this paper: (a) the novelty is not strong enough, (b) the experimental results should be better explained and be more thorough, (c) the formulation is not well motivated.
Decision: Reject