Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: = Summary 
This paper presents a simple method that draws visual attention of deep embedding networks for metric learning. It basically follows the class attention mapping strategy based on global pooling operation [Zhou et al., CVPR 2016], but extends the original version to point-specific attention which is novel and enable interesting applications on image retrieval. In addition, the proposed method seems also independent of the loss function used for metric learning, thus it can be applied to most of existing deep embedding networks to understand their behaviors in a qualitative manner. 


= Decision
My current decision is officially "weak reject" but "borderline" in my mind. The major concern of mine is its weaknesses in clarity and technical novelty. However, I still believe this submission is valuable since it addresses a relatively new and timely topic, the proposed method is simple yet effective, and the applications of point-specific attention (i.e., "cross-view pattern discovery" and "interactive retrieval") are all interesting and practically useful. If the clarity issues are all clearly addressed, I would upgrade my rating. 


= Comments
[Pros]
1) The motivation and implementation of the point-specific attention are convincing. 
2) The point-specific attention enables not only image-to-image retrieval, but also more elaborate understanding about a pair of images and their similarity.
3) The proposed technique is model-agnostic and loss-agnostic, thus can be applied to most of existing deep embedding networks for image retrieval. Also, the proposed technique does not degrade the retrieval performance.
4) The proposed technique is simple yet effective in multiple applications, most of which are practically useful and have great impact. For example, the weakly supervised localization is an essential step towards many weakly supervised approaches for higher-level recognition tasks like semantic segmentation, and the interactive retrieval will allow us to build more realistic and useful image retrieval systems.

[Cons]
1) The proposed technique itself is not new but a straightforward extension of an existing work [Zhou et al., CVPR 2016].
2) The manuscript is not crystal clear.
- The name of the proposed point-specific attention (i.e., "Partial attention") is misleading.
- The way to compute the point-specific attention map is not clearly described in Section 3.
- It is hard to understand the contents in Figure 6 as they are not clearly illustrated. 
- The experimental and implementation details of the last two applications are not given. For example: (cross-view pattern discovery) how to compute the angle error, and how much the proposed technique is sensitive to the position selected on the query image, (interactive retrieval) how to compute the similarity between images given a specific region of interest on query.
3) More qualitative results on the last two applications should be presented, even in an appendix, to convince future readers. Especially, in the case of "interactive retrieval", more results are demanded as quantitative performance analysis seems not straightforward.


= Post-rebuttal review
The rebuttal resolves my major concerns and the manuscript has been carefully revised accordingly. So as I promised in my original review, I upgrade the score to weak accept. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes a visualization method for deep metric learning, which derived by analyzing the inner product of two globally averaged activations. The proposed method can generate an overall activation map which highlights the regions contributing most to the similarity. Also, it can generate a partial activation map that lights the regions in one image that have significant activation responses on a specific potion in the other image. The authors also analyzed the linearly of the fully connected layers and global max pooling. These contributions make the applicability of CAM to many CNN architectures. Further, the metric learning architecture is extended to Grad-CAM map, and the problem of Grad-CAM map is pointed out. To the best of my knowledge, these contributions are novel, and derivations seem to be correct. 

Experiments on weakly-supervised localization, model diagnosis, and the applications of the proposed decomposition model in cross-view pattern discovery and interactive retrieval are promising. 

Overall, this paper is well written, and contributions are good. 

Minor problems.  
In Sec.1 and Sec.2.2, the authors wrote the Grad-CAM has been used for visualization of re-ID (Gordo & Larlus (2017)). However, this paper seems to be not the works of Grad-CAM nor re-ID. 

In my understanding, Decomposition+Bias is a more accurate model than Decomposition. 
In the experiments of the Sec5.1 and 5.2, the performances of Decompsotion+Bias are lower than Decomposition. However, there are no explanations for this reason. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work proposes a novel approach for visualizing the predictions of neural network models on pairwise tasks, e.g. predicting whether two images are similar. The authors show that to see similarity between two images down on the pixel/region level, it is not sufficient to apply methods like Grad-CAM which do not aim for decomposition. Instead, the authors' approach namely targets decomposition, and shows the benefit of decomposition through intuitive examples and qualitative results. The authors also quantitatively show the benefit of their method. First, they measure the performance of their method vs CAM and Grad-CAM, on the weakly supervised localization (WSL) task. They also show how their method reveals the disadvantages of standard triplet loss compared to a recent metric learning loss. 

My concerns:
1) While showing performance on WSL is appealing as it allows for a way to quantitatively evaluate the method, I wonder if there are other ways to evaluate, that explicitly measure the quality of the proposed technique for allowing interpretability and understanding of the base model's performance. 
2) The Triplet vs MS experiment is interesting, but I'm not sure this is the most convincing way to show that this proposed visualization technique is better than something else. Just because something shows one method is worse than another, doesn't mean that this better/worse assessment is accurate. Further, how would Grad-CAM do on the same task?
3) The retrieval experiment only shows qualitative results, and again, no baseline is compared. 
4) Similarity is a relative judgement; it's hard to say if two items are similar, but easier to say if A and B are more similar than A and C. It seems the proposed method doesn't consider negatives, which is perhaps a limitation.
review_title: Official Blind Review #1

Meta-review: This submission proposes a method for providing visual explanations for why two images match by highlighting image regions that most contribute to similarity.

Reviewers agreed that the problem is interesting but were divided on the degree of novelty of the proposed approach.

AC shares R1’s concern that localization accuracy is not satisfactory as a quantitative measure of the quality of the explanations. In particular, it pre-supposes what the explanations ought to be, i.e. that a good explanation means good localization. A small user-study would be more convincing. A more convincing evaluation would also include a study of explanation of image pairs with different degrees of similarity (e.g. images that are dissimilar as well as images with the same object).

AC also shares R2’s concern about the validity of the model diagnosis application. This discussion also relies on the assumption that better localization of the whole object means a better explanation. Further, the highlighted regions in Figure 5 are very similar. Once again, a user study would help to indicate whether these results really do improve explainability.

Reviewers also had concerns about missing details and, while the authors did improve this, key details are still missing. For example, the localization method that was used was only referenced but should be described in the paper itself.

Given that several concerns remain, AC recommends rejection.

Decision: Reject