Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes to apply residuals to the anchors as representations for inputs. Compared with the features directly extracted from inputs, the residuals provide the relative information with a learnable codebook, which can alleviate the overfitting problem. Besides, an additional branch is included for the consistency between representations from different codebook.
strengths_and_weaknesses: Strong  
1.	Adopting residuals for generating features is novel for deep metric learning.   
2.	The ablation study shows that a consistency constraint is helpful for learning representations.  
3.	The experiments is sufficient on benchmark data sets.


Weak  
1.	The motivation of introducing residuals for better generalization on unseen classes is not convicting. Note that the anchors are also learned from the training data, which cannot work as reference points for unseen classes. For example, according to the example in Line 57, the anchor points will only contain daytime faces from training while the residuals of night-time faces to those anchors cannot capture the appropriate information.  
2.	The additional branch increases the training cost, which should be discussed.  
3.	The performance of MS in Table 5 is degenerated compared to the original paper, which reports the similar performance without the proposed method. Since many previous works have BN-Inception as the backbone, it is better to include the results with the same backbone and same dimension of features for a fair comparison.  

questions: My major concerns are about motivation and the fair comparison issue as listed in weakness.
limitations: Compared baseline methods, the additional branch for consistency constraint may require more computational resources/running time.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a coded residual transform (CRT) to improve generalization of  deep metric learning method. They use a set of learned anchors to encode the image embeddings. The proposed CRT obtains state-of-the-art performance on four datasets including CUB-200-2011, Cars-196, Standard Online Products and In-shop Clothes Retrieval. 
strengths_and_weaknesses: Strengths:

1. The authors proposed a coded residual transform (CRT) to improve generalization for deep metric learning. 
2. The authors utilize a set of earned anchor features to encode the image embeddings. The anchor diversity loss, CRT consistency loss and MS-loss [11] is used for optimization. 
3. The experiments and ablation studies are solid. 

Weaknesses:

The authors should discuss and compare margin-based softmax [a][b][c] in the sections of  related work and experiments. Although the CRT is different from margin-based softmax, the insight is similar. The parameters W in softmax can be treated as the anchor features. 

[a] Liu, Weiyang, Yandong Wen, Zhiding Yu, and Meng Yang. "Large-Margin Softmax Loss for Convolutional Neural Networks." In ICML. 2016.

[b] Wang, Hao, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. "Cosface: Large margin cosine loss for deep face recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274. 2018.

[c] Deng, Jiankang, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. "Arcface: Additive angular margin loss for deep face recognition." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699. 2019.
questions: 1. As stated in weakness, the insight of margin-based softmax is similar to CRT. For Eq. (3) and Eq. (4), why not to use softmax formula such as n-pair loss [a] ?
2. The details of batch samplers for training should be clarified in the experiment settings. 
3. For the total loss function (Line 205), is MS-loss necessary? Other metric learning loss such as Siamese loss, triplet loss and n-pair loss should be compared. 

[a] Sohn, Kihyuk. "Improved deep metric learning with multi-class n-pair loss objective." Advances in neural information processing systems 29 (2016).
limitations: See weaknesses and Questions.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript proposes a new framework for generalisable deep metric learning. The main idea is to learn a collection of anchors and get the final representation by transforming the original embedding vectors into their distances to the anchors, weighted by their correlation with said anchors. These distances are then aggregated by anchor-wise projection and averaging. The model is trained with Anchor Diversity Loss (minimizes anchors’ cross-correlations) and Multi-Similarity loss, which is a recent metric learning loss [11].

strengths_and_weaknesses: The proposed framework is sound, intuitive and the paper is easy to follow. The presented performance results are also very promising, as is the empirical support to the authors’ claims of the generalisation capability of the model (Lines 206-219). I also believe similar or modified versions of this framework could be used in the future, which makes this a good contribution. 

I’ve found Table 4 to be confusing to parse given that the numbers presented here do not match with their published versions (I’ve only checked Hyp-DeIT); the authors acknowledged the discrepancy and attributed it to different settings and results that are hard to reproduce under compute constraints, which I’ve found to be fair compromises given that they have also designed the experiments in Table 5, where comparison is made fair by comparing against baseline versions of a large set of backbones. 

The ablation study is complete enough and it is good to know that performance is not too sensitive to weight sharing and the number of anchors in the first embedding branch. I’ve just found it hard to parse the permutations described in Lines 297-308 and presented in tables 8-9; I don’t know what’s the number of anchors in the first branch for Table 9, or why the weights are shared for this experiment (are they not shared for Table 8?) and it raises the question of whether the weights are shared or not for the other experiments as well since the performance is comparable and the gain in compute use is likely considerable.

questions: Does the backbone get trained as well or are the weights frozen?
Was the choice of dimensionality made out of compute restrictions? I ask since other papers have used higher dimensionalities such as 384 (for ViT based models) or 512.
Related to the previous question, how did you change the output dimensionality for the ViT based methods such as Hyp-DeiT? Did you train those from scratch with an added bottleneck layer?
It is not clear to me if the second embedding branch also uses a second backbone. Does it use another backbone and is it trained separately or do they share backbones?
limitations: Since this is an image retrieval paper that makes use of common benchmarks, I do not believe there is a negative social impact to it beyond the environmental cost of training deep neural networks. 

The authors only present one limitation in their conclusion: that the anchors are learned from the training set and therefore may not be the best possible anchors. I believe another limitation to be the need for two embedding branches, since in the non-sharing configuration it incurs in increased memory and compute usage (this is a lesser issue if the backbone is not duplicated)
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper revives a classic image retrieval technique, VLAD, with modern Deep Network and show promising results. In short for the main idea of the paper, the proposed code residual transform (CRT) method maps the feature map into anchor pools and output residue features to represent an input image. In table 5, CRT outperforms the baseline method by a clear gap across multiple datasets. Instead only providing the number to beat the SOTA, the author also provide additional evidence such as feature visualization and embedding space density metric to support the proposed method.
strengths_and_weaknesses: Strengths
Originality: The proposed method revives a classic image retrieval technique, VLAD, with modern Deep Network is novel
Quality: The submission is technically sound. The anchor pool is well visualized in Figure 3(But still some flaw, see later). Additional metric such as table 1 embedding space density which helps understanding where is gain coming from with the proposed method
Clarity: The submission is clearly written and well organized.
Significance: Reviving a classic image retrieval technique with modern techniques is significant.

Weaknesses
Clarity: Figure 1 and Figure 2 should be merged together. Figure 1 is clear enough for presenting the idea, but figure 2 is not. 
Clarity: Please use another terminology for the "anchor" in the main paper. The term anchor is already used in the triplet loss. It will confuse readers who are familiar with DML.
questions: 1. In the figure3 visualization, can you explain the red dots on the top right corner of image in the end of first row? 
2. Would you like provide the correlation/similarity matrix for the anchor pool? It is supposed to be independence as much as possible as show in Eq 3. But as shown in figure 3, different attentions are kind of overlap. 
3. can you provide the sampler detail? what is the batch size? how many classes in a batch?

limitations: N/A
review_title: null

Meta-review: metareview: This paper proposes a coded residual transform for deep metric learning, aiming to improve the generalization ability of metric learning to unseen classes. Four expert reviewers assessed this paper, with preliminary reviews at odds. After author rebuttal, some reviewers acknowledged the rebuttal by increasing the score, but one reviewer still held major concerns including the vague motivation and unconvincing evaluation. AC read the paper itself as a neutral referee, and considered all reviewing material. AC's take on the paper is as follows.

- The motivation of using coded residual to improve generalization to unseen classes lacks a sufficient elaboration, either theoretical or technological -- showing the motivation with a toy example is not strong. Also, there is no clear connection between the coded residual and the generalization property in Roth et al. [22] -- again, only a partial result in Table 1 is not enough.

- The algorithmic contribution is slightly below the bar of NeurIPS. By toothing apart each component, only Eq. (1) regarding the coded residual is somewhat novel to me. Eq. (2) is a common feature combination method, similar to concat, element-sum, or the aggregation used in GNNs and PointNet. Eq. (3) is a common criterion in linear discriminant analysis. Eq. (4) is something like the kernel matrix, which is of O(N^2) complexity, very time-consuming to compute for deep learning. In all, the technical novelty is relatively slim while the practical value in terms of efficiency is not high.

- Writing is problematic in some way. Authors try to explain the motivation of coded residual but in fact the idea was firstly proposed in the classic method VLAD [49]. Perhaps the new thing is that authors also try to explain that the coded residual is more generalizable to unseen classes, but this is less elaborated. Authors shall clearly credit to VLAD and limit their own contribution to "generalizing to unseen classes".

Nonetheless, AC feels that the idea of coded residual is interesting in the metric learning context, and authors are suggested to work forward for a stronger approach that generalizes learned metrics to unseen classes, which is important for practical open-world applications. After discussion between AC and SAC, their opinion on this paper was somewhat in disagreement. SAC suggested that by taking the reviews as well as the scores (8/7/6/4) into consideration, the paper should be accepted, and the negative points above can be addressed by the authors in the next version. AC revised the metareview from (Borderline) Reject to (Borderline) Accept accordingly.
confidence: Less certain
Decision: Accept