Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Paper presents a method to perform top-k reranking on a (image) retrieval system. The method uses external data to learn, in a supervised manner,  a reranking module that produces contextualized representations (by means of self attention) of the top-k results, which are then used to rerank them. 
The method seems inspired by the recent LattQE [16], that also uses a reranking module based on self attention and trained in a supervised manner. However, I see this paper has relevant contributions / differences:
- The proposed method uses "affinity" representations as the input of the reranking module, that probably capture better the similarity space.
- The form of supervision is different, computing a contrastive loss only amongst the top-k results. 
- In addition, a "fidelity" loss and a different form of data augmentation is proposed.
main_review: In general, I enjoyed reading this paper. 

- In terms of novelty, I think the paper is somewhat incremental from the work of [16]. However, I think enough novel technical contributions are proposed, making it, in my opinion, barely above the threshold.

- In terms of clarity, the paper is generally easy to follow, although there are some technical details I could not understand completely, I'm asking some clarifying questions to the authors below.

- The proposed method and decisions seem reasonable as well. There are some decisions I find somewhat surprising though, again, I will ask clarifying questions below.

- In terms of significance / quantitative evaluation, the method clearly outperform existing approaches by a significant margin on two standard benchmarks (plus augmented versions). The model achieves good results both on rOxford and rParis using the same model, which is not common (typically, models that work on one dataset tend to fail in the other due to differences in statistics within the datasets, for example number of relevant images per query). On the negative side, I wish the authors could have reported results on more datasets (e.g. Instre). I've seen many papers accepted reporting results only on rOxford and rParis, so this is not grounds for rejection, but I wish the authors would consider including those results as well. Also, Pytorch code was provided, which will help other researchers build on top of this work. Thank you authors for making the code available.

On the negative side, it seems to me (and please authors correct me if I'm wrong) that some hyper parameters are validated directly on test (see, e.g. Figure 2, where Roxf hard is used). This is a barely-discussed, dark standard practice in the retrieval community. I am not going to strongly penalize the authors for this when essentially every other paper gets away with it. However, I'd urge the authors to reconsider validating the hyper parameters in rSfM120k or other dataset (and perhaps add the results validating in rOxf in the supplementary material)

Questions for authors:

- It was unclear under which circumstances different features were used. Was this used only to augment the training data by generating different lists of top-k results, or was it also used to generate different affinity features (in the sense of the dot product scores being different because the features are different, not in the sense of the L anchors being different). Also, does this happen only at training time, or also at test time?

- I understand that no form of database augmentation (e.g. DBA) is happening, is that correct?

- Can you elaborate a bit more about the need for the MSE loss? It is unclear to me why we'd want the refined features to be similar to the original features (condition on an MLP transformation). Seems like an adhoc decision.

- Did you experiment training this reranking model in GL18 (since that's where the features seem to be coming from)? Not asking the authors to do this experiment if they didn't, just wondering if they actually did it and if there's any insight (e.g. performing worse because of more overfit).
limitations_and_societal_impact: - Paper describes some limitations (particularly, if the first round of results is bad, reranking will not work well).

- Paper does not have a societal impact section. Although I understand this is not as obvious here as in other problems (e.g. face recognition), I think authors could have made an effort here. For example, is it possible that when trying to retrieve images with very very few relevant items (because they are not classic western-centric landmarks), the reranking actually makes the results worse?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a new re-ranking method for image retrieval. It proposes to learn an affinity vector for each of the top-ranked candidates w.r.t a set of anchor images, using a transformer encoder. These affinity vectors are then used to re-estimate the similarity scores between the top-ranked candidates and the query image. The proposed method is evaluated on the Revisited Oxford and Paris datasets and shown to be superior or complementary to other query expansion/diffusion based reranking methods.
main_review: ------------------------------------------------------------------------------------------------------------------

Strengths

[S1] The paper proposes to aggregate the contextual similarities between the query image and the top-ranked candidates for image reranking using a transformer based model. The proposed method is technically sound. Compared to other reranking methods, the proposed method is also more flexible in the sense that only the affinity vector for each image is used. Therefore, the training features and testing features can be from different feature extractors. It also enables cross-feature based data augmentation.

[S2] The paper is easy to follow.

[S3] The proposed method is shown to be superior or at least complementary to state-of-the-art query expansion/diffusion based reranking methods. Ablations on several components (e.g. # of images to be reranked, # of anchor images, data augmentation, etc) are also included.

------------------------------------------------------------------------------------------------------------------
Concerns

[C1] For the comparisons shown in Table 5, I wonder if the scores of the baselines are from their original papers or re-training/re-testing/re-implementation? As the numbers for some baselines appear to be inconsistent with those reported in their original papers. For example, the mAP for GSS[22] on RParis Medium is {92.4}/{88.9} in {the original paper}/{this paper}.

[C2] For the comparisons shown in Table 5, I wonder if the proposed method and the baselines are evaluated under similar settings. For example, i) are they using the same initial ranking lists? ii) are they using the same feature extractor, i.e. R-GeM? This is important for fair comparisons and justifying the real advantage of the proposed method.

Given the concerns listed above, I’d like to rate the paper as “marginally below the acceptance threshold” at this moment.

------------------------------------------------------------------------------------------------------------------
Minor comments

[M1] Table 5, DSM [9] → DSM [40]

-------------------------------------------------------------------------------------------------------------------
Post rebuttal

I have read the reviews and the authors' replies. In general, I think the idea of using only the affinity vectors for reranking is interesting. It makes it possible to reuse the pretrained reranking model for different feature extractors. The authors also addressed my concerns on the evaluation: if all the methods were tested under the same setting. Therefore, I raise my score and vote for accepting the paper.




limitations_and_societal_impact: I don't think there will be any potential negative societal impact of the paper. For the limitations of the paper, please refer to the "concerns" part of the main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a re-ranking algorithm for image retrieval, capable of improving the ranking results of an off-the-shelf similarity-based retrieval system, by representing each query by an affinity vector that measures similarities to a set of anchor images, and then refine this representation by aggregating it with contextual information obtained from other top-ranked retrieval results using a transformer architecture. The refined representations of the top-K images can then be used for re-ranking. 

The paper shows improvements both in precision and time complexity over previous re-ranking algorithms. The time complexity improvements are mostly due to the introduction of a fixed set of anchor images, which avoids context affinity features that scale linearly with the number of top-k retrievals considered. The precision improvements are a result of a learnable context aggregation module (in the form of a transformer) and the data augmentations used for training it.

Experiments on several benchmarks show healthy improvements over prior re-ranking procedures.
main_review: The paper introduces a re-ranking algorithm for content-based image retrieval. As mentioned above, the proposed procedure yields performance improvements both in terms of precision and computational complexity. These improvements were shown in several benchmark datasets. The proposed method is interesting as it seeks to *learn* how to best aggregate and re-rank the outputs of a similarity-based retrieval system.

I am overall positive about the paper. Nevertheless, I hope the authors can clarify/address the two following points. I will adjust my final recommendation based on that.

The paper claims to introduce a new data augmentation technique for training the re-ranking model. However, this seems to be using different CNN backbones to generate multiple features for each image, and treat these features as augmentations. This technique does improve performance significantly. However, I'm wondering if the improvements are mostly due to the use of multiple backbones, as in the case of ensembles (often used for competitions)? Is the proposed method using all backbones at test time, or only one? if only one, which one? if not, how did you ensure a fair comparison to other methods (since it would be similar to an ensemble)?

The fixed number of anchors reduces the time complexity of the proposed procedure without affecting its precision. Table 2 also shows that the method is not too sensitive to this parameter. However, I'm concerned whether this is because evaluations are being done on datasets with a relatively low number of concepts, and so a small set of anchors can cover all the concepts. Would there be a significant performance hit if the method was applied to datasets with a large number of concepts like ImageNet? Would the method need a much large number of anchors in order to maintain high performance? And if so, wouldn't the computational gains be diminished?

-----

Post rebuttal:
Thanks to the authors for the rebuttal, and the other reviewers for their insightful reviews. After carefully considering all inputs on the paper, I will increase my rating to 7. The paper makes progress in content-based image retrieval, both in terms of precision and computational complexity, by improving upon prior re-ranking algorithms. The authors also addressed my main concerns about the fairness of their comparisons. I strongly suggest the authors to clearly articulate in the paper how the algorithm operates during deployment (in order to avoid such confusions in the future).

limitations_and_societal_impact: This paper aims to improve image retrieval (an established research area). As such, it does not introduce new negative societal impacts.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, a visual re-ranking method is proposed. This method leverages the contextual similarity of the top-K retrieved images by comparing the original features with a set of anchor images, which is further refined by a transformer encoder. The refined features are then used to re-rank the top-K ranking list.

Moreover, a new data augmentation scheme is designed by employing different feature extractors, which enlarges the feature variety in the ranking list and improve the robustness of the re-ranking model.

Comprehensive experiments have been conducted on four benchmarks, demonstrating clear improvement and strong robustness.
main_review: ### Originality 
A new re-ranking method is proposed in this paper, the proposed method differs to previous works by extracting contextual similarity from top-K images and refining them with an attention-based model. Different kinds of re-ranking methods have been included in the related work, but the discussion about the relationship between the proposed work and previous works is missing.

### Quality
The proposed method is clearly described, the contribution of each component is well demonstrated in the ablation studies. Comparison with state-of-the-art methods have also been given to support the effectiveness and generality of the proposed method. Time and memory complexity comparison is provided.

### Clarity
Overall, the paper is super clear and well structured. A few minor changes could be done to resolve the following issues:
* (**L154-156**): what is the dimension of the learnable projection matrix $W_M$, and what is the size of vector $h_i$ ?
* (**L163**): what is $S'$ and how do we initialize it ?
* (**Table-5**): what does the entry of *Affinity Feature* means? And maybe merge the second block and the third block of QE methods by having two lines for each QE such as AQE and AQE + DBA.
* (**Temperature parameter $\tau$**): seeing the importance of this parameter in the contrastive loss, maybe include an ablation study of this hyper-parameter to show how robust is the method against this hyper-param.

### Significance
The well-designed and easy-to-use re-ranking method could benefit many related applications. Besides, the usage of affinity feature and attention-based encoder could help the community understands better the effectiveness of transformer-based architecture in retrieval tasks.


limitations_and_societal_impact: Limitations and potential negative societal impact of this work is discussed in the supplementary materials.
review_title: null

Meta-review: The paper introduces a new re-ranking method for image retrieval. It proposes to learn an affinity vector for each of the top-ranked candidates w.r.t a set of anchor images, using a transformer encoder. These affinity vectors are then used to re-estimate the similarity scores between the top-ranked candidates and the query image. The reviewers found the work to be technically sound, have novelty, and produce good results. There were some concerns about the fairness of the evaluations, and many other questions of detail, which the authors have addressed effectively in their rebuttal. 
Decision: Accept (Poster)