Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: I spot a couple of key issues with the proposed method:

1) Misinterpretation of the previous study by Shen et al. (CVPR 2020): As presented in Eq (8) of that CVPR paper, the dataset where the influence loss is applied is NOT necessarily the old training set; It  can be the new training dataset. Also, their results show that using new training set for the influence loss is slightly better than using old training set. This makes the statement that "previous works rely on old training" invalid. So Eq (3) of this submission is not complete and somewhat misleading.

2) Shen et al. (CVPR 2020) only use the classifier but not the feature model. Although this proposed work does not use the old classifier, the feature model is instead used to estimate the classifier weights. So, it is not true to claim that "the old model is a black-box" for the proposed method; Just use a different part of the old system. Besides, in deep classification model, the feature vectors and classifier weights are closely related and both reveal similar amount of information.  In particular, if a cosine classifier is used, the two can be exchanged to each other. So, this proposed model is actually leveraging the same amount of information from the old system, as compared to Shen et al. (CVPR 2020).

Overall, the two points above would make me to conclude that this work comes with fundamental flaws/drawbacks.

review_title: This paper considers an interesting problem called feature compatible learning, that is, learn a new and better feature embedding model without the need for updating existing feature embedding. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary]
This work proposes a new problem setting by adding extra constraints to the Feature Compatible Learning problem. The new constraints avoid using old training data and the old model’s parameter when learning a new model. The paper gives a baseline method and its variants for the problem by generating pseudo classifiers to regularize a new model’s learning. The experiments show that the proposed method can satisfy the empirical criterion about success.

[Strengths]
1. The constraints added to the feature compatible learning are sounded and may have practical value.
2. The writing of this paper is satisfactory. The clarity is good and easy to follow. Both the related works, motivations, and technical details are clearly introduced.
3. The experiment has a reasonable set of baselines for the problem. The ablation study clearly shows how the proposed enhancement contributes to performance gain.
4. Section 3.3.2 is a new interesting way to combine the feature representation for a group of data.

[Weaknesses]
1. Figures 1 and 2 do not help understand the method’s overview nor the problem setting.
2. The gain of the RWN classifier is not significant. There are other simple ways to remove the outliers for computing the mean representations. One example is to ignore the top 10 percent of the data based on the initial mean and variance, then re-compute the mean representation. The RWN should be compared with those simple baselines.
3. There are several minor typos. Please have proofreading.

[Questions]
1. The gain of RWN looks small in Table 2. What is the variance of Table 2?

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper deals with an interesting problem of feature compatible learning that the features produced by new model should be compatible with old features. The proposed method uses nearest class–mean classifier instead of linear classifier. Random walk is applied to refine the class means. The proposed method is compared with several baseline methods and shows good performance.


Pro:
1.	This paper deals with an interesting problem of feature compatible learning. 
2.	A new experimental setting is presented.
3.	A simple method based on nearest class–mean classifier is proposed and shows good performance.

Cons:
1.	Some incremental learning methods also deal with the feature compatible problem when old features are stored for rehearsal. The paper lacks the review of these methods.
2.	The regularization term in Eq (6) and (12) is not defined. 
3.	It needs more details to explain the two important settings (cross test and self test) and the experimental protocols.
4.	As the proposed method needs to compute the similarity graph for data in every class. How much extra time and memory does it take to implement the random walk refinement?
5.	More experimental results, e.g. visualization, should be given.

review_title: An interesting problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses an interesting problem in retrieval system - compatible features learning. Given the old feature extractor and a new dataset, the objective is to learn a new feature extractor, so that the features extracted by two (old and new) feature extractors are comparable to each other. In the proposed setting, the old dataset (including its statistics), old classifier, and the parameters of the old model are not available.

The presentation of this paper is clear, including the problem description and basic idea of the methodology. The construction of the pseudo old classifier is well motivated and reasonable. The performance improvements are significant based on the presented results.

However, I still have some concerns.
First, it is unclear that why random walk provides benefit to obtaining a better feature matrix.
Based on eq 10, each column of the new feature matrix (left side) is the linear combination of the columns of the old feature matrix. Subsequently, the feature matrix is averaged to obtain the pseudo classifier. So intuitively, the pseudo classifier is obtained by weighted averaging of the feature matrix, where the weights are learned by random walk from their similarity scores.
Given these facts, more explanations are needed for why random walk could produce the beneficial weights.

Second, the formulation of the loss function (L) is not given. (eq 3), which is an important detail.

Third, there is no experimental comparison with other works. So it is hard to see the contribution of this paper.
I understand that this is a novel problem, which may not have prior work to compare with. However, to make the results more convincing, the proposed methods can be used to other similar settings for evaluation, e.g. the setting in Shen et al. 2020.


review_title: good topic and straightforward solution, but need more explanations and comparisons

Meta-review: This paper deals with a problem of feature compatible learning, where the features produced by new model should be compatible with old features. As pointed out by the reviewers, there are several weaknesses with this paper: (a) the novelty is not strong enough, (b) the experimental results should be better explained and be more thorough, (c) the formulation is not well motivated.
Decision: Reject