Why do These Match? Explaining the Behavior of Image Similarity Models
Authors: Bryan A. Plummer
Mariya I. Vasileva
Vitali Petsiuk
Kate Saenko
David Forsyth
Venue:  (2020)
Keywords: explainable artificial intelligence
image similarity
artificial intelligence for fashion

Abstract: Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings. Recent work has primarily focused on explaining models for tasks like image classification or visual question answering.  In this paper, we introduce an explanation approach for image similarity models, where a model's output is a score measuring the similarity of two inputs rather than a classification.  In this task, an explanation depends on both of the input images, so standard methods do not apply. We propose an explanation method that pairs a saliency map identifying important image regions with an attribute that best explains the match.  We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition. Our approach's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2.

Under review as a conference paper at ICLR 2020
WHY DO THESE MATCH? EXPLAINING THE BEHAV- IOR OF IMAGE SIMILARITY MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Explaining deep learning model can help users understand behavior and allow researchers to discern its shortcomings. Recent work has primarily focused on explaining models for tasks like image classification or visual question answering. In this paper, we introduce where a model's output is a score measuring the similarity of two inputs rather than classification. In this task, explanation depends both of the input images, so standard methods do not apply. We propose an explanation method that pairs saliency map identifying important image regions with attribute that best explains the match. We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition. Our approach's 's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2.
explanation approach for image similarity models,
INTRODUCTION
Many problems in artificial intelligence that require reasoning about complex relationships can be solved by learning some feature embedding measure similarity between images and/or other modalities such as text. Examples of these tasks include scoring fashion compatibility (Han et al.) 2017b Hsiao & Grauman| 2018: Vasileva al., 2018), image retrieval (Kiapour et al. [015 Rade- novi et al. 2018; Yelamarthi et al. 20188). or zero-shot recognition (Bansal et al.) 2018 Li et al.] 2018b] Wang et al. 2018). Reasoning about the behavior of similarity models can aid researchers in identifying potential improvements, show where two images differ for anomaly detection, pro- mote diversity in fashion recommendation by ensuring different traits are most prominent in the top results, or simply help users understand the model's predictions which can build trust (Teach & Shortliffe 1981). However, prior work on producing explanations for neural networks has primarily Tocused on explaining classification models (e.g. (Fong & Vedaldi 2017 Nguyen et al., 2016 Pet- siuk et al. 2018 Ribeiro et al. 2016 Selvaraju et al. 2017] Zeiler & Fergus) 2014)) and does not directly apply to similarity models. Given single input image, such methods produce saliency map which identifies pixels that played a significant role towards a particular class prediction (see Figure Dh for example). On the other hand, a similarity model requires at least images produce score. The interaction between both images defines which features are more important, so replacing just one of the images can result in identifying different salient traits.
Another limitation of existing work is that the saliency alone may be insufficient as an explanation of (dis)similarity. For image pairs where similarity determined by the presence absence of an object, a saliency map may be enough to understand model behavior. However, when we consider the image pair in Figure [].. highlighting the necklace as the region that contributes most to the similarity score is reasonable, but uninformative given that there are other objects in the image. Instead, what important is the fact that the necklace shares a similar color with the ring. Whether these attributes salient parts are better fit domain (i.e. attributes for e-commerce imagery vs. saliency for natural imagery), but instead by the images themselves. For example, an image be matched form because of a shirt's collar (salient part), while two images of animals can match because both have stripes (attribute).
explanation is not determined by the image
Guided by this intuition, introduce Salient Attributes for Network Explanation (SANE). Our approach generates a saliency map to explain a model's similarity score, paired with an attribute explanation that identifies important image properties. SANE is a "black box" method, meaning it

---

Under review as a conference paper at ICLR 2020
(a) Prior work
(b) Our work
Itx call
Iaage casutlee
Howiste to say?
Cer Cataloation Expaationn
Theyrr De JAcIuI
Existing explanation methods focus on image classification problems (left), whereas we explore explanations for image similarity models (right). We pair saliency map, which identifies important image regions, but often provides little useful information, with an attribute (e.g., golden), which is more human-interpretable and, thus, better explanation than saliency alone.
Figure 1:
can explain any network architecture and only needs to measure changes to a similarity score when provided with different inputs. Unlike a standard classifier, which simply predicts the most likely attributes for given image, our explanation method predicts which attributes are important for the similarity score predicted by model. Predictions are made for each image in pair, and allowed to be non-symmetric, e,g., the explanation for why the ring in Figure [ matches the necklace may be that it contains "black", even though the explanation for why the necklace matches the ring could be that it is "golden." " A different similarity model may also result in different attributes being deemed important for the same pair of images.
SANE combines three major components: an attribute predictor, a prior on the suitability of each attribute an explanation, and a saliency map generator. Our underlying assumption is that least one of the attributes present in each image should be able to explain the similarity score assigned to the pair. Given an input image, the attribute predictor outputs a confidence score and activation map for each attribute, while the saliency map generator produces regions important for the match. During training, SANE encourages overlap between the similarity saliency and attribute activation. At test time, we rank attributes as explanations for an image pair based on a weighted sum of this sttiibute-saleenyy map matching score, the explanation suitability prior of the attribute, and the likelihood that the attribute is present in the image. Although we evaluate only the top-ranked attribute in our experiments, in practice more than one attribute could used to explain similarity score. We find that using saliency maps supervision for the attribute activation maps during training not only improves the attribute-salient matching, resulting in better attribute explanations, but also boosts attribute recognition performance using standard metrics like average precision.
We evaluate several candidate saliency map generation methods which are primarily adaptations of "black box" approaches that do not rely on particular model architecture require access to network parameters to produce a saliency map (Fong & Vedaldi 2017 Petsiuk et al. 2018, Ribeiro al. 2016; ZZiler & Fergus 2014). These methods generally identify important regions by mea- suring a change in the output class score resulting from some perturbation of the input image. Sim- ilarity models, however, typically rely on learned embedding space reason about relationships between images, where proximity between points or the lack thereof indicates some degree of cor- respondence. An explanation system for embedding models must, therefore, consider how distances between embedded points, and thus their similarity, change based on perturbing one or both of the input images. We explore two strategies for adapting these approaches to our task. First, we manip- ulate just a single image (the one we wish to produce an explanation for) while keeping the other image fixed. Second, we manipulate both images to allow for more complex interactions between the pair. See Section 3.2 for additional details and discussion on the ramifications of this choice.
Our paper makes the following contributions: 1) we provide the the first quantitative study of ex- plaining the behavior of image similarity models; 2) we propose novel explanation approach that combines saliency maps and attributes; 3) we validate our method with metrics designed to link our explanations to model performance, and find that it produces more informative explanations than adaptations of prior work to this task and also improves attribute recognition performance.
RELATED WORK
Saliency- Explanations. Saliency methods can generally be split into "white box" and "black box" approaches. "White box" methods assume access internal components of neural network,
2

---

Under review as a conference paper at ICLR 2020
(a) SANE Model
(b) Attribute Explanation Selection
actvvatio mapa
Query image
adtribute
global average leyer
attibeue
prod
lenoy
activation makdiiing loss
saliency generalor
attribtt ranking
"sequin"
gerwnstrr
inage sim similally
sallency
imagee
witriburle oxgi Ingiaaation prrtor
reference imagee
mocel
simiority
model
Figure 2: Approach Overview. During training, each saliency map produced by the generator is encouraged match at least one ground truth attribute's activation maps. Then, at test time, we rank attribute explanations by how well the saliency and attribute activation maps match, along with the likelihood of the attribute and its explanation suitability prior. Note that we assume the image similarity model has been pretrained and is kept fixed in all our experiments
either in the form of gradients or activations of specific layers (e.g. (Cao al., 2015 Chang et al.
2019 Nguyen et al. 2016 Selvaraju et al., [017] Simonyyn et al. 2014] Yosinski et al. 2015
Zhang al. 2016; Zhou et al. 2016)). Most of them produce a saliency map by using some version of backpropagation from class probability to an input image. In contrast, "black box" approaches require no knowledge of the internals (e.g. weights, gradients) of the models. These methods obtain saliency maps by perturbing the input in predefined way and measuring the effeet of it the model output, such as class score. We adapt and compare three "black box" " and one "white box" methods for our saliency map generator in Figure 2 "Black box" approaches include Sliding Window (Zeiler & Fergus| 2014)). which masks image regions sequentially, and Randomized Input Sampling for Explanations (RISE) (Petsiuk et al. 2018), which masks random sets of regions. Both measure the effect removing these regions have on the class score. LIME (Ribeiro et al. (016) first obtains a super-pixel representation of an image. Super-pixel regions are randomly deleted, and their importance is estimated using Lasso. "White box" Mask (Fong & Vedaldi (017) learns saliency map directly by using different perturbation operators and propagating the error to a low resolution mask. Although there is some limited work which adapts some saliency methods to the image similarity setting (e.g.: Gordo & Larlus (2017). they present qualitative results only, ie., these methods are not evaluated quantitatively on their explanation accuracy done in our work.
Natural Language and Attribute-based Explanations. Instead of producing saliency maps, which can sometimes be difficult to interpret, researchers have explored methods of producing text-based explanations. These include methods which justify a model's answer in the visual question an- swering task (Huk Park et al. 2018: Li et al.) 2018a), rationalize the behavior of self-driving or describe why a category was selected in fine-grained object classifi- attribute explanations describing why two images are similar or dissimilar as guidance for image clustering. Our approach could be used automatically generate these explanations rather than relying on human feedback. Several works exist which learn attribute explanations either to identify important concepts (Kim) al., 2018a| Bau al.) 2017; Fong & Vedaldi 2018) or to justify a model's 's decision by pointing to evidence (Anne Hendricks et al.) 2018). Kim et al. (2018a) learns a concept activation vector that separates examples with an attribute against examples without it, and then scores the sensitivity of attributes based on how often directional derivative changes the inputs towards the concept. However, all these methods were designed to explain categorical predictions rather than similarity models. To the best of our knowledge, ours is the first work which uses attribute explanations in the image similarity setting.
vehicle (Kim et al.]|2018b cation (Hendricks al. 2016). Lad & Parikh| (204)) used
3 SALIENT ATTRIBUTES FOR NETWORK EXPLANATIONS (SANE)
We are given a fixed model that predicts the similarity between two images, and must explain why query image is similar to a reference image. While typical models for predicting similarity are learned from data, using an embedding method and a triplet loss, our approach is agnostic as to how the model being explained is built. Our method consists of three components: the attribute expla- nation model (Section B.I), the saliency map generator (Section 3.2). and an attribute explanation suitability prior (Section 333) Although train CNN to produce attribute predictions, the image
3

---

Under review as a conference paper at ICLR 2020
similarity model we wish to explain is kept fixed. At test time, recovers saliency map for the match from the query image in a pair, then uses the attribute explanation model and attribute suitability prior to rank each attribute's 's ability to explain the image similarity model.
3.1 ATTRIBUTE EXPLANATION MODEL
Suppose we have access to pairs of images Tg). Here, I, denotes a reference image and Tg a query image. We wish to obtain an explanation for the match between I, and T,. Associated with each pair is a saliency map m, produced by a saliency map generator (described in Section 3.2 We need simply swap the query and reference images compute saliency map for m,, which would also likely result in a different saliency map than mq. Finally, assume we have access to A.
binary attribute annotations 04, i = 1, A. and let agt E {0, 1] be the set of ground truth attribute annotations for given query image. If no attribute annotations are provided, an attribute discovery method could be employed (e.g., (Han et al. 2017a] Vittayakorn et al. 2016)). We explore using an attribute discovery method in the appendix.
Our attribute explanation model produces confidence scores à E RA for Ig. Unlike standard attribute classifier, however, our goal is not just predict the most likely attributes in I,. but rather to identify which attributes contribute the most to the similarity score s(rr [4) produced by the similarity model we wish to obtain explanations for. accomplish this, the layer activations for attribute a; before the global average pooling layer is defined as attribute activation map n,- This attribute activation map represents a downsampled mask of an image that identifies prominent regions in Ig for that attribute. We encourage at least ground truth attribute's activation map for image Ig to match saliency map my as a form of regularization (see Figure 2 for an overview). Our underlying assumption is that least one of the ground truth attributes of I should be able to explain why I is similar to Ir. Thus, at least one of the attribute activation maps n, should closely resemble the saliency map for the match, my.
Each attribute confidence score is obtained using a global average pooling layer on its attribute activation map followed by softmax. The attribute explanation network is trained using Huber loss (Huber 1964), sometimes referred to as smooth & loss, which helps encourage sparsity in the predictions. More formally, given set of confidence scores and attribute labels agt. our loss is,
1 (agt a)² for lag - al S (agt à otherwise.
Hubeer (â, age)
Note that multiple attributes can be present in the image; and that this loss operates attributes, attribute activation maps. Since the confidence scores sum to one (due to the softmax), we scale binary label vector by the number of ground truth attributes Agt (e.g., if there are four attributes for an image, its label would be 0.25 for each ground truth attribute, and zero for all others).
Leveraging saliency maps during training. We explicitly encourage our model to identify at- tributes which are useful in explaining the predictions of an image similarity model by finding which attributes best describe the regions of high importance to similarity predictions. To accomplish this, first find a set of regions that may be important to the decisions of image similarity model by generating a set of K saliency maps M, for up to reference images that are similar. For the image under consideration, to each ground truth attribute. Then, for each saliency map we find its best match in Ngz, We match saliency maps to attributes rather than the other way around since all annotated attributes are necessarily relevant to the explanation of s(I,, Ig). We use & loss between the selected attribute activation map and saliency map, i.e..
also construct set of attribute activation maps Ngt corresponding
Lhm K.
Σ min [m nl/2-
AnENg
YIEM,
Combined with the attribute classification loss, our model's complete loss function is:
Leooat = L.Huber + XL
(3)
where À is scalar parameter. See appendix for implementation details and parameter values.
4

---

Under review as a conference paper at ICLR 2020
3.2 SALIENCY MAP GENERATOR
A straightforward method of producing a saliency map with a "black box" method is to manipulate the input image by removing image regions and measuring its effect on the similarity score. If a large drop in similarity is measured, then the region must be significant. If almost no change was measured, then the model considers the image region irrelevant. A saliency map is generated from this approach by averaging the similarity scores for each pixel over all instances where it was removed from the imput. The challenge is determining the best way of manipulating the input image to discover these important regions. These "black box" methods assume you don't 't have access to the underlying model's parameters to compute gradients, and as our experiments will show, can also produce saliency maps that are often comparable or better than some "white box" methods which leverage these gradients. We compare four salieney methods: Sliding Window (Zeiler Fergus) 2014). LIME (Ribeiro et al. 2016), "white box" Mask (Fong & Vedaldi 2017), and RISE (Petsiuk et al. 2018). We now describe how we adapt these models for our task; additional details on each method are in the appendix.
Computing similarity scores. Each saliency method we compare was designed to operate on a single image and measures the effect manipulating the image has on the prediction of specific object class. However, an image similarity models predictions are made for two or more images. Let us consider the case where we are just comparing two images, a query image (i.e. the image want to produce an explanation for), and a reference image, although our approach extends to consider multiple reference images. Even though we do not have access class label, we can measure the effect manipulating an image has on the similarity score between the query and reference images. Two approaches are possible: manipulate both images, or manipulate only the query image.
Manipulating both images would result in NM forward passes through the image similarity model (for N, M the number of query and reference image manipulations. respectively), which is pro- hibitively expensive unless M << N. But we only need an accurate saliency map for the query image, so we set M << N in our experiments. There is another danger: for example, consider two images of clothing items that are similar if either they both contain do not contain a special but- ton. Masking out the button in one image and not the other would cause a drop in similarity score, but masking out the button in both images would result in high image similarity. These conflicting results could make accurately identifying the correct image regions contributing to a score difficult.
The alternative is to manipulate the query image alone, i.e. keep a fixed reference image. We evaluate the saliency maps produced by both methods in Section 4.1
3.3 SELECTING INFORMATIVE ATTRIBUTES
At test time, given a similarity model and a pair of inputs, SANE generates a saliency map and selects attribute to show to the user. We suspect that not all attributes annotated for dataset may prove to be useful in explaining every image similarity model. Thus, we take into account each attribute's 's usefulness in explaining predictions made by similarity model by learning concept activation vectors (CAVs) (Kim et al. 2018a) over the image similarity embedding. These CAVs identify which attributes are useful in explaining layer's activations by looking whether at- tribute positively affects the model's predictions. CAVs are defined as the vectors that are orthogonal to the classification boundary of a linear classifier trained to recognize an attribute over layer's ac- tivations (i.e. the image similarity model's embedding). Then, the sensitivity of each concept to an image similarity model's predictions (i.e., the TCAV score) is obtained by finding the fraction of features that were positively influenced by the concept using directional derivatives computed using a triplet loss with a margin of machine epsilon. Note that this creates a single attribute ranking over the entire image similarity embedding (i.e., it is agnostic to the image pair being explained) which we use as an attribute explanation suitability prior. Finally, attributes are ranked as explanations using a weighted combination of the TCAV scores, the attribute confidence score å, and how well the attribute activation map n matches the generated saliency map mg. l.e.,
c cmm à n, TCAV) áiâ + Φ2 deas (m, n) + 63 TCAV,
where dam denotes cosine similarity, and Φ1-3 are scalars estimated via grid search on held out data.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
Datasets. We evaluate our approach using two datasets from different domains to demonstrate its ability to generalize. The Polyvore Outfits dataset (Vasileva et al. 2018) consists of $65,054 fashion product images annotated with 205 attributes and composed into 53,306/10,000/5..0
train/test/validation outfits. Animals with Attributes 2 (AwA) (Xian et al. 2018 consists of 37,322 natural images of 50 animal classes annotated with 85 attributes, and is split into 40 animal classes for training, and 10 used at test time. To evaluate our explanations we randomly sample 10,000 ground-truth (query, reference) pairs of similar images for each dataset from the test set.
Image Similarity Models. For the Polyvore Outfits dataset we use the type-aware embedding model released by Vasileva al. (2018). This model captures item compatibility (i.e. how well two pieces of clothing go together) using a set of learned projections on top of general embedding. each of which compares a specific pair of item types (i.e. a different projection is used when comparing a top-bottom pair than when comparing top-shoe pair). For AwA we train feature representation using a 18-layer ResNet (He et al. 2016) with a triplet loss function that encourages animals of the same type to embed nearby each other For each dataset/model, cosine similarity is used to compare an image pair's 's feature representations.
4.1 SALIENCY MAP EVALUATION
Metrics. Following Petsiuk et al. (2018), evaluate the generated saliency maps using insertion and deletion metrics which measure the change in performance of the model being explained as pixels are inserted into blank image, deleted from the original image. For our task, we generate saliency maps for all query images, and insert or delete pixels in that image only. If a saliency map correctly captures the most important image regions, we should expect a sharp drop in performance as pixels are deleted (or sharp increase as they are inserted). We report the area under the curve (AUC) created as we insert/delete pixels at a rate of 1% per step for both metries. We normalize the similarity scores for each image pair across these thresholds so they fall in [0-1] interval.
Results. Table compares the different saliency map generation methods on the insertion and dele- tion tasks. RISE performed best on most metrics, with the exception of and LIME doing better on the deletion metric on AwA. This is not surprising, since LIME learns which super-pixels contribute to similarity score. For AwA this means that parts of the animals could be segmented out and deleted or inserted in their entirety before moving onto the next super-pixel. On Polyvore Outfits, however, the important components may be along the boundaries of objects (e-g- the cut of dress), something not well represented by super-pixel segmentation. Although Mask does not perform as well as other approaches, it tends produce the most compact regions of salient pixels as it searches for a saliency map with minimal support (see the appendix for examples). Notably, we generally obtained better performance when the reference image was kept fixed and only the query image was manipulated. This may be due to the issues from noisy similarity scores as discussed in Section 3.2 and suggests extra care must be taken when manipulating both images.
4.2 ATTRIBUTE PREDICTION EVALUATION
Metrics. To begin, we report the overall performance of SANE using mean average precision (mAP) on the standard task of attribute recognition computed over all images in the test set. Two additional metrics are used to evaluate our attribute explanations using the (query, reference) image pairs used in the saliency map experiments which are similar to the evaluation of saliency maps. Given the set of attributes we know exist in the image, similarity score using Eq. ( and then see the effect of deleting that attribute from the image has on the similarity score. Similarly, we select the attribute which best explains the similarity score from those which aren't I present in the image and measure the effect of inserting the attribute has on the similarity score. Intuitively, if an attribute was critical for an explanation, then the similarity score should shift more than if different attribute was selected. Scores for these metrics are expressed in terms of relative difference, so higher numbers are always better regardless of whether we are inserting or deleting an attribute. Scores are also normalized so the upper bound is 1.
select which attribute among them best explains the
Upon acceptance, we will make our embedding model for the AwA dataset publically available.

---

Under review as a conference paper at ICLR 2020
Table I: Comparison of candidate saliency map generator methods described in Section 3.2
We
report AUC for the insertion and deletion metrics described in Section 4.1
Polyvore Outfits
Animals with Attributes 2
Method
Fixed Reference?
Insertion (1) Deletion (4)
Insertion (1) Deletion (4)
Sliding Window
Y
60.2
53.6
76.9
76.8
LIME
Y
58.4
55.4
77.0
71.2
Mask
Y
59.4
53.3
74.5
77.3
RISE
Y
64.3
49.5
77.4
75.1
Sliding Window
N
59.6
54.3
77.6
76.3
Mask
N
58.9
54.6
75.8
78.4
RISE
N
61.4
51.9
77.7
74.0
To insert or remove an attribute, find representative image in the test set which is most similar to the query image in all other attributes. For example, let us consider a case where we want to remove the attribute "striped" from a query image. We would search through the database for the image which is most similar in terms of non-striped attributes, but which hasn't been labeled as being "striped." " These attribute comparisons are based on the average confidence for each attribute computed over the three attribute models we compare in Table 2 On the Polyvore Outfits dataset we restrict the images considered to be of the same type as the query image (i.e., if the query image is a shoe, then only shoe can be retrieved). After retrieving this new image, compute its similarity with the reference image to compare with the original (query, reference) image pair. Examples of this process can be found in the appendix.
Compared methods. We provide three baseline approaches: a random baseline, a sample attribute classifier (i.e. no attribute activation maps), and modified version of FashionSearchNet (Ak al., 2018), an attribute recognition model which also creates a weakly-superrised attribute activation map for comparison. Additional details these models can be found in the appendix.
Results. Table 2 compares the performance of the compared attribute models for our metrics. Our attribute explanation metries demonstrate the effectiveness of our attribute explanations, with our model which matches saliency maps and includes TCAV scores getting the best performance on both datasets. This shows that when we "insert" "delete" the attribute predicted by SANE from the image, it affects the similarity model's score more than the baselines. Notably, our approach outperforms SearihnS + Map Matching, which can be considered weakly-supervised version of SANE trained for attribute recognition. The second line of Table 2 reports that TCAV does well on the insertion task, but poorly on the deletion task. This is due, part, to the fact that other models, including SANE, are trained to reason about attributes that actually exist in an image, whereas for insertion the goal is to predict which attribute that isn't I present in the image would affect the similarity score most significantly. Thus, using bias towards globally informative attributes (i.e., TCAV scores) is more useful for insertion. Finally, training SANE to produce explanations leads 2% improvement simple attribute classifier while using the same number of parameters. SANE also outperforms SeshhonSearchNet, which treats localizing important image regions latent variable rather than using saliency maps for supervision.
the standard attribute recognition task measured in mAP over
We conducted small scale user study to verify humans found our explanations useful. We showed the users the saliency map explanations for image pairs or the attribute explanations, and asked if they felt the explanations helped them understand the model's behavior. We obtained at least 25 responses for each question, where 56% of subjects found the saliency maps useful and 72% found the attributes useful on the AwA dataset, while on Polyvore Outfits these are 87% and 79%, respectively. This verifies that most users thought our explanations were useful, which could help build trust, while also showing how the different explanations are ideal different circumstances.
We provide qualitative examples of our explanations in Figure Examples demonstrate that our explanations pass important sanity checks. The explanation attribute well-correlated with the localization of important pixels in the saliency map for each pair. Notice that "golden", "striped" and "printed" in the first two columns of Figure 3 are sensibly localized, and are also reasonable explanations for the match, while more abstract explanation like "fashionable" is linked to the high heel, the curve of the sole, and the straps of the shoe. Note further that the explanations non-trivial: they more often than not differ from the most likely attribute in the query image,
7

---

Under review as a conference paper at ICLR 2020
Table Comparison of how attribute recognition (mAP) and attribute explanation (insertion, dele- tion) metrics described in Section 4.2 are affected for different approaches. We use fixed-reference RISE our saliency map generator for both datasets, Higher numbers are better for all metrics.
Animals with Attributes 2 MAP Insertion (f) Deletion (1) mAP Insertion (1) Deletion (1)
Polyvore Outfits
Method
Random
71.7
73.3
41.2
48.9
TCAV (Kim et al.
2018a)
75.3
74.3
47.6
49.7
Attribute Classiher
28.8
72.8
74.8
65.1
41.9
50.0
Seerhhn (Ak et al. 2018
29.6
73.2
74.6
66.3
41.8
50.3
FashionSearchNet + Map Matching
72.8
75.2
42.8
50.8
SANE
30.9
73.5
74.9
67.3
42.4
50.2
SANE + Map Matching
73.7
75.7
44.1
51.3
SANE + Map Matching TCAV (Full)
76.8
76.2
48.9
51.6
Explanation
Reference
Explraatio
Reference Image
Quary Imaga
Explanation
Reference Image
Query Image
Explanation
Image
"prited"
"fonger
"naachv""
lohed*
"ashionabie"
"beat"
'shan"
"colitary"
"pakshee"
Figure Qualitative results of our attribute explanations for pairs of examples on the Polyvore Outfits and the AwA datasets. The attribute predicted explanation for each reference-quee querry match shown below the saliency map. The most likely attribute for the query image as predicted by our attribute classifier is shown directly underneath it.
as predicted by standard attribute classifier. In other words, our explanation model is utilizing information from each pair of images and the saliency map characterizing the match produce sensible, interpretable explanation.
Figure shows example of how directly removing the attribute predicted as the explanation can affect similarity (possible here because the attribute is color.) Here we see that when we modify the white dress to different color, the similarity score drops significantly. The only exception when we make the dress the same color (black) the attribute explanation of the pants it is being compared to. This demonstrates in a causal way how our predicted attributes can play a significant role in the similarity scores.
CONCLUSION
In this paper we introduced SANE, a method of explaining an image similarity model's behavior by identifying attributes that were important to the similarity score paired with saliency maps indicating import image regions. We confirmed humans believe our explanations are useful for explaining a model's behavior, which could help build trust, to supplement automatic metries. In future work we believe closely integrating the saliency generator and attribute explanation model, enabling each component take advantage of the predictions of the other, would help improve performance.
Refference
Query Image
Replocing Color
Image
Attribute Explanation:
"black"
"white"
0.15
.0.08
.0.34
Figure 4: Example of the effect replacing the attribute used as an explanation of the model's behavior has on image similarity score (higher score means items more compatible).
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Kenan E. Ak, Ashraf A. Kassim, Joo Hwee Lim, and Jo Yew Tham. Learning attribute representa- tions with localization for flexible fashion search. In CVPR, 2018.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explana- tions. In ECCV, 2018.
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, 2018.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Toralla. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, 2017.
Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, Deva Ramanan, and Thomas S. Huang. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In ICCV, 2015.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In ICLR, 2019.
Ruth Fong and Andrea Vedaldi. Net2vec:
Quantifying and explaining how concepts are encoded by filters in deep neural networks. In CVPR, 2018.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In ICCV, 2017.
Albert Gordo and Diane Larlus. Beyond instance-level image retrieval: Leveraging captions to learn global visual representation for semantic retrieval. In CVPR, 2017.
Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. Automatic spatially-aware fashion concept discovery. In ICCV, 2017a.
Xintong Han, Zuxuan Wu, Yu-Gang Jiang, and Larry S. Davis. Learning fashion compatibility with bidirectional Istms. In ACM MM, 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In ECCV, 2016.
Wei-Lin Hsiao and Kristen Grauman. Creating capsule wardrobes from fashion images. In CVPR, 2018.
Peter J. Huber. Robust estimation of a location parameter. Annals of Statistics, 53(1):73-101, 1964.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Mareus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In CVPR, 2018.
M. Hadi Kiapour, Xufeng Han, Svetlana Lazebnik, Alexander C. Berg, and Tamara I.. Berg. Where to buy it: Matching street clothing photos to online shops. In ICCV, 2015.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICMI., 2018a.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In ECCV, 2018b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Shrenik Lad and Devi Parikh. Interactively Guiding Semi-Supervised Clustering via Attribute-based Explanations. In ECCV, 2014.

---

Under review as a conference paper at ICLR 2020
Qing Li. Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. In ECCV, 2018a.
Yan Li, Junge Zhang, Jianguo Zhang, and Kaiqi Huang. Discriminative learning of latent features for zero-shot recognition. In CVPR, 2018b.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In NIPS, 2016.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018.
Filip Radenovi, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In CVPR, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In KDD, 2016.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop. 2014.
Randy L. Teach and Edward H. Shortliffe. An analysis of physician attitudes regarding computer- based clinical consultation systems. Computers and Biomedical Research, 14(6))542-558, 1981.
Mariya I. Vasileva, Bryan A. Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David Forsyth. Learning type-aware embeddings for fashion compatibility. In ECCV, 2018.
Sirion Vittayakorn, Takayuki Umeda, Kazuhiko Murasaki, Kyoko Sudo, Takayuki Okatani, and
Kota Yamaguchi. Automatic attribute discovery with neural activations. ECCV, 2016.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. TPAMI, 40(8), 2018
Sasi Kiran Yelamarthi, Shiva Krishna Reddy, Ashish Mishra, and Anurag Mittal. A zero-shot frame- work for sketch based image retrieval. In ECCV, 2018.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop, International Conference on Machine Learning (ICML), 2015.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Jianming Zhang, Zhe Lin, Shen Xiaohui Brandt, Jonathan, and Stan Sclaroff. Top-down neural attention by excitation backprop. In ECCV, 2016.
B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba. Learning Deep Features for Discrim- inative Localization. In CVPR, 2016.
A CANDIDATE SALIENCE MAP GENERATOR DESCRIPTIONS
In this section we provide additional details about each of the candidate saliency map generation methods used in our paper. We split these approaches into two groups: methods which analyze behavior solely through input manipulation (described Section and those which use an opti- mization procedure to learn some parameters in combination with input manipulation (described in Section A.2.. Please see Section 3.2 of our paper for a description of how these methods are adapted to our task. We also provide runtime comparison of each approach in Table 3 A qualitative com- parison between saliency map generators on the Polyvore Outfits and AwA datasets provided in Figures and
10

---

Under review as a conference paper at ICLR 2020
A.1 SALIENCY MAPS BY INPUT MANIPULATION
A straightforward approach to producing a saliency map is to manipulate the input image by re- moving image regions and measuring the effect this has on the similarity score. If a large drop in similarity is measured, then the region must be important to this decision. If almost no change was measured, then the model considers the image region irrelevant. The saliency map is generated from this approach by averaging the similarity scores for each pixel location over all instances where it was removed from the input. The challenge then is to determine how to manipulate the input image to discover these important regions.
Sliding Window (Zeiler & Fergus, 2014). The first approach to removing regions of an image we shall discuss is a sliding window, where regions are sampled regularly across an image. There direct tradeoff, however, with how densely frames are sampled and the computational time it takes to do forward pass through the network for each manipulated image. If frames are not densely sampled to enable an efficient solution, then wouldn't It be able to localize important regions accurately. If regions are too densely sampled then removing them might not make enough of difference the similarity score to take measurements accurately.
RISE (Petsiuk et al., 2018). This method uses Monte Carlo approach to generate saliency maps. A set of NV random binary masks of size h x w is sampled where each element is independently set to I with probably p, and all other elements are set to O. Typically these masks are much smaller than the input image, so they are upsampled using bilinear interpolation. This produces small continuous regions within the upsampled mask that can be used to manipulate the input image. To remove the fixed grid structure the masks are upsampled to larger than image size and then cropped randomly. Although this approach does require a significant number of random masks (we found 2,000 to be sufficient in our experiments), we found this approach significantly outperforms using a sliding window that samples a similar number of masks on our task.
A.2 LEARNED SALIENCY MAPS
We shall now discuss methods which combine input manipulation with an optimization procedure used to directly learn a saliency map. As in Section [A.]] we compare generating saliency maps for single query image at time using a fixed reference image well as generating saliency map by manipulating both the query and reference images.
LIME (Ribeiro et al., 2016). Rather than masking regions without any concern over the continuity of region, this approach to generating saliency maps operates over superpixel segmentation of an image. Images are manipulated by randomly deleting superpixels in the image. After sampling N manipulated inputs, the importance of each superpixel is estimated using Lasso. Finally, important regions are selected using submodular optimization.
Mask (Fong& & Vedaldi 2017). In this approach a low resolution saliency map is directly learned us- ing stochastic gradient decent and upsampled to the image size. Instead of manipulating an image by just deleting regions as in other methods, two additional perturbation operators are defined: adding Gaussian noise and image blurring. To help avoid artifacts when learning the mask a norm is used in addition to an L1 regularization to promote sparsity. This approach removes the reliance on superpixels and tends to converge fewer iterations than LIME, although it is consider- ably slower in practice than other approaches (see Table 3 That said one advantage it does have
over other approaches the ability to learn the salience map for both the query and reference image jointly (which we take advantage of when we are not using a fixed reference image).
B ADDITIONAL. EXPERIMENTAL. OR IMPLEMENTATION DETAILS
B. SALIENCY MAP GENERATOR DETAILS
Sliding Window. When manipulating the inputs of the reference image, we apply 625 occlusion windows each covering a square region of about 12% of image area. When manipulating both images apply 36 occlusion windows to the reference image.
11

---

Under review as a conference paper at ICLR 2020
Reference Image
Query Image
Sliding Window
LIME
Mask
RISE
Figure 5: Qualitative examples comparing the saliency map generator candidates on the Polyvore
Outfits dataset.
12

---

Under review as a conference paper at ICLR 2020
Reference Image
Query Image
Sliding Window
LIME
Mask
RISE
Figure 6: Qualitative examples comparing the saliency map generator candidates on the Animals
with Attributes dataset.
13

---

Under review as a conference paper at ICLR 2020
Table 3: Runtime comparison of the compared saliency generation methods and how using fixed reference image, or manipulating both the query and reference images affects performance.
Method
Fixed Reference?
Time(s)
Sliding Window
0.2
LIME
Y
1.2
Mask
4.1
RISE
Y
0.3
Sliding Window
2.5
Mask
N
7.2
RISE
N
5,8
RISE. For both datasets we randomly sample 2,000 random masks upsampled from 8 x mask with the probability of preserving region of 0.5. When manipulating the inputs of the reference image, we generate 30 random masks.
LIME. We generate LIME saliency maps using 1000 samples.
Mask. We learn 14 14 perturbation mask for both datasets. We train the mask for 500 iterations using Adam Kingma & Ba 2015) with learning rate of 0.1.
B.2 COMPARED ATTRIBUTE METHODS
In addition to random baseline, we provide two for comparison to our model for our attribute experiments in Section 4.2 First, we train simple attribute classifier using the same architecture as SANE, but where the attribute activation maps are not regularized with saliency maps. Second, we use modified version of FashionSearchNet (Ak et al. 2018), which designed for fashion search using attribute information. This network uses an attribute activation map to identify and extract a region of interest for each attribute. These extracted regions are fed into branches consisting of three fully connected layers which trained for both attribute classification and image retrieval. We remove the image retrieval components For all attribute models, we use 50-layer ResNet base image encoder He et al. (2016) and the last convoluntional layer of the network the same number of channels as the number of classes. This provides a simple baseline and a model with a generic weakly-supervised attribute activation map for comparison.
B.3 SANE DETAILS
Due to its efficient (see Table 3 and overall good performance (see Table I we selected the fixed- reference RISE our saliency map generator. For each training image, we sample up to five sim- ilar images using the ground truth annotations of each dataset and generate saliency maps using each sampled image the reference image. We train our attribute model for 300 epochs using Adam (Kingma & Ba 2015) with a learning rate of 5e and set À = 5e in Eq. 3 from the pa- per. After each epoch, we computed mAP on the validation set and kept the best performing model according to this metric. Additional qualitative examples for explanations produced by our SANE model on the Polyvore Outfits and AwA datasets are provided in Figures 7 a
We provide an example of the attribute deletion process in Figure 9 After identifying an attribute to remove in an image, we search for the most similar image the input from a database that doesn't contain the input attribute. Image similarity is computed over the attribute space, i.e., we want to keep the predictions of each attribute the same, and only vary the target attribute. For Polyvore Outfits, we only consider images of the same type (i.e., so tops can only be replaced with other tops). To ensure we don't 't bias towards a single attribute model, average the predictions made by each attribute model in our experiments (Attribute Classifier, FashionScarchNet, and SANE). We on the left side of Figure 9 that some attributes like colors are largely retained when the attribute has to do with non-color based attribute. On the returned AwA images the right side of Figure 9 we see how some attributes can lead to significant changes in the images or almost none at all depending on the attribute selected to remove.
In Section 3.3 we discuss how we estimate how likely each attribute is a "good" explanation in held- out data. This is used prior bias our attribute selections towards attributes that known
14

---

Under review as a conference paper at ICLR 2020
Reference Image
Query Image
Reference Image
Query Image
Explanation
Explanation
"golden"
"strappy"
"high"
"knitted"
"colorful"
"suede"
"gladiator"
"tasseled"
"floral"
"vintage"
"beachy"
"nude"
"retro"
"leather"
"fashionable"
"rod"
"cashmere" "
"monogram"
"casual"
"sleeveless"
Figure 7: Additional qualitative examples of our SANE explanations on the Polyvore Outfits dataset.
15

---

Under review as a conference paper at ICLR 2020
Reference Image
Query Image
Reference Image
Query Image
Explanation
Explanation
"claws"
"stalker"
"swims"
"tail"
"smart"
"hairless" "
"forager"
"black"
"fish"
"panda"
"solitary"
"hands"
"strong"
"fierce"
"claws"
"forager"
"domestic"
"tree" *
"smelly"
'timid"
Figure 8: Additional qualitative examples of our SANE explanations on the AwA dataset.
16

---

Under review as a conference paper at ICLR 2020
Input Image
Attribute to Remove
Returndd Image
Input Image
Attribute Remove
Returned Image
"studded"
"walkss"
"lace"
"active"
"silver"
"fast"
Figure 9: Examples of the attribute deletion process used to evaluate how good an attribute as explanation. We measure the similarity of the input image and some reference image as well as between the returned image and the reference image. If large change in similarity is measured then the attribute is considered "good" explanation. If similarity stays about the same, the attribute is considered a "poor" explanation, e.g., trying to remove "active" from the pandas on the right.
longneck
bush slow newworld
solitary
mountains
other
nocturnal
toughskin
vegetation
grazer
fierce
plains
group
chewteeth
tail
forager
gray
domestic
oldworld
patches
agility
black
bulbous
big
smart
active
ground
timid
hooves
Figure 10: The likelihood each attribute in the AwA dataset identified the best attribute for an image pair on held-out data.
17

---

Under review as a conference paper at ICLR 2020
Table 4: Discovered attribute explanation performance comparison using the full SANE model.
Polyvore Outhits
Animals with Attributes 2
Attribute Types
Insertion (t) Deletion (1) Insertion (1) Deletion (1)
Random
71.7
73.3
41.2
48.9
Full Frame Discovery
75.7
75.0
47.9
50.5
Patch Discovery
76.1
75,4
48.2
51.1
Supervised Attributes
76.8
76.2
48.9
51.6
be good attribute explanations. In Figure T0 we show the ground truth bias for the attribute detection task according metrics for the AwA dataset. Note, however, that this prior would change for a different image similarity model. For example, if the image similarity model was more biased towards colors, then would expect see the likelihood for "black," "brown," and "gray" to increase.
C DISCOVERING USEFUL ATTRIBUTES
For datasets without attribute annotations, or those where the annotated attributes doesn't 't cover the extent of the visual attributes present the dataset (i.e. there are many unannotated attributes) we propose method of discovering attributes that are useful for providing model explanations. An attribute that is useful for explantions would commonly appear the high importance regions of saliency maps. When generating saliency maps for query image, if many reference images attend to the same region the query image then is likely they are all matching to for similar reasons (i.e. there may be some attribute that they share which matches the query). Given this observation, we discover attributes using the following saliency-based procedure:
1. Obtain K similar images for query image q using k-NN.
2. Generate a saliency map over for each of the similar (reference) images.
3. Keep only those reference images which have their saliency peaks in the most common location (such as a unit square in a 7 x grid) and pick top N of them that have the highest similarity.
4. For each reference image, generate its saliency map with q and crop 30 30 patch around the peak saliency region in the reference image.
5. Upsample all the generated patches to full image resolution and get their embeddings.
6. Cluster the patches produced for multiple queries q. Each cluster represents an attribute. multiple patches were extracted from image and they got assigned to different clusters, this image would labeled with multiple attributes.
Figure Iaa illustrates the clustering produced by this procedure for set of queries from Polyvore Outfits dataset.
To evaluate this approach we compare to randomly assigning images to clusters and to clustering based their own embeddings, disregarding the saliency of image regions (Figure lTb). Saliency- based attribute discovery works best among the three unsupervised methods for Polyvore Outfits data, but full-frame clustering outperforms it for the AwA dataset (Table 4 We suspect the full frame clustering works better for AwA since it considers the background more than the patch-based method (Polyvore Outfits image's 's typically have white backgrounds). In addition, our discovered attributes would likely be noisier due to the similarity model focusing on the background patches in some images as well. Although our initial results are promising, attempting to discover attributes useful for explanations warrants additional investigation.
18

---

Under review as a conference paper at ICLR 2020
(a) Patch-based clustering
(b) Full-frame clustering
Figure 11: Six clusters defining the attributes for two approaches to attribute discovery. (a) Each image assigned list clusters that have patches from this image. Clustering is per- formed on salient patches.
(b) Each image is assigned one of the clusters as an attribute. Clustering is performed on full-frame images.
19


Reviews:
[{"review_id": "S1gkMmh1qH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview/Contribution:\n====================\nThe paper proposes an explanation mechanism that pairs the typical saliency map regions together with attributes for similarity matching deep neural networks. The authors tested their methods on two datasets, i.e. Polyvore Outfits (a clothing attributes dataset) and Animals with Attributes 2 (a dataset of attributes for animals).\n\nOverall, the paper has merit to be accepted to the conference with the following strengths and weaknesses. I suggest to the authors to address the weaknesses pointed out to make the paper more stronger, especially adding few more attributes datasets such as person attributes datasets as noted below in the weakness section.\n\nStrength:\n========\n- The paper is written clearly and is easy to understand. I have seen the additional results and visual comparisons in the supplemental material and it was useful, albeit a bit longer.\n\n- Explanations have the potential to make decisions made by a deep neural model transparent to end users among other benefits especially for sensitive applications such as healthcare and security. Explaining decisions made by similarity matching models has many applications including person attribute recognition and person re-identification for surveillance scenarios [1]. So, in this respect, this paper is relevant to the target audience.\n\n- There is a bit of confusion between explanation and interpretation of decisions made by deep neural network models in the explainable AI literature and in most cases the two are used interchangeably. Hence, saliency maps are considered as explanation on their own by many. Combining saliency map based interpretations together with higher level concepts such as attributes has the potential to generate more realistic explanations of the decisions. The authors made this point at the second paragraph of the introduction. \n\n- Fig. 1 (b) also is a clear example of the kind of explanations generated using a template with the key attribute in question accompanied by the visual saliency map interpretation.\n\n- Fig. 2 clearly shows the overall proposed method and the attribute ranking based on the attributes explanation prior and the match between the saliency map and attribute activation maps.\n\n- The attribute ranking and selection method of informative attributes using combinations weighted TCAV and cosine similarity between the attribute activation map and the generated saliency map is novel.\n\nWeakness:\n===========\n- Applications of such a combined explanatory system don’t seem to be highly motivated in the introduction. I suggest the authors discuss more of the image similarity based applications and less on the discussion and heavy citation of generalized deep neural networks.\n\n\n- The forms of the two loss components are both variants of l_{1} and l_{2} standard losses and they could be subject to issues with the standard variants of the l_{1} and l_{2} losses such as lack of translation and other transformation invariances. Hence, it would have been more useful to give the reasoning for the selection of the losses employed compared to other similarity and divergence based losses that are less sensitive to such variations.\n\n- Similar to the above point, the choice of cosine similarity to compare match b/n attribute activation maps and saliency maps seem arbitrary. The method is described well but why cosine similarity was chosen in terms of its benefits compared to other similarity metrics is not that clear.\n\n- Evaluation on more datasets such as person/pedestrian attributes datasets would have demonstrated the generalizability of the proposed method across multiple practical domains. As such, I would suggest the authors test their method on at least one person/pedestrian attributes dataset such as PETA, Market1501, etc.\n\n- Although Fig. 1 (b) motivated a more practical high level explanation, in the results section, the attribute explanations are reduced to just the selected attribute that matched with the saliency well. Human-like concise attribute-based high level explanation just like the example given in Fig. 1 (b) would have made the paper stronger. Even if NLP is beyond the scope of this paper, a simple template based explanation that incorporated the selected/matched attribute would have been more effective.\n\n- The results are too concise and a few ablation results on different losses etc. could have helped. There is too many qualitative results especially in the supplementary.\n\n1) Bekele, E., Lawson, W. E., Horne, Z., & Khemlani, S. (2018). Implementing a Robust Explanatory Bias in a Person Re-identification Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 2165-2172).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJl17Yad5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces SANE, a new approach for explaining image similarity models by combining a saliency map generator and an attribute predictor. In this way, the method is not only able to highlight what regions contribute the most to the similarity between a query image and a reference image, but also predict an attribute that explains this match. During training, SANE jointly optimizes the attribute prediction of the query image and maximizes the overlap of the saliency map of the image similarity and the attribute activations.\n\nI think the paper addresses a very interesting problem that has been commonly overlooked. There are many recents works on the explainability of neural networks for images classification and other similar tasks, but very few have addressed this problem for image similarity. It is also novel and interesting the addition of an attribute predictor in the system which provides additional information that cannot be captured by the saliency map alone. Finally, the paper also does a big effort presenting a quantitative study of how SANE is able to explain similarity models.\n\nHowever, I would also like to raise a couple of issues/questions regarding the method and its technical contribution:\n\n- I would suggest the authors to include a brief explanation of the architecture used for the attribute predictor since it will help to understand how the attribute activation is computed. I am assuming that a Fully-Convolutional Neural Network is being used, where the output of the last convolutional layers has as many channels as the numbers of classes. Is this correct?\n\n- Why using softmax + L1 loss to train the multi-attribute predictor? Aren't there other activations and losses better suited for multi-label classification, such as sigmoid + binary cross entropy loss, where there's no need to divide the ground-truth labels?\n\n- In order to match image similarities with attribute descriptions the authors propose matching similarity saliency maps with attribute map activations. This is done by first computing a saliency map for the similarity between a query image and a reference image, computing the activation maps of the ground-truth attributes of the query image, then finding the attribute activation that best matches the saliency map, and finally minimizing the distance between the saliency map and the attribute activation using an L2 loss (cf last paragraph Section 3.1). My first question is: how is the best matching attribute match? I missed this explanation in the paper and, to my understanding, this is a very crucial step. My second concern is that I don't see why the attribute activation map should be matched with the similarity saliency map since not all the regions highlighted in the similarity map might describe the attribute. For example, if we're comparing two images  containing a jacket and both contain the attributes \"zipper\" and \"black\", the saliency map might highlight regions of the zipper and where the black color is present, but the activations of the attribute \"black\" should not be enforces to match the regions of the zipper. Could the authors explain the intuition behind this design choice?\n\n- A final minor comment: as I mentioned before, the introdution of attributes in the explanation process is a very interesting contribution since they provide the user an explanation that is a step closer to a description in natural language. However, this comes at the price of needing attribute annotations at training. In order to overcome this problem, the authors suggest using an attribute discovery method when no attribute annotations are provided. My question therefore is: how are these automatically discovered attributes gonna be useful in order to provide a description, given that they are not associated with any word or concept?\n\n\nAlthough the paper proposes a very interesting approach for explaining image similarity models, I also have some concerns that I think should be addressed before its acceptance. Therefore, my initial recommendation is weak reject.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJeHgcM6KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I Summary\nThis paper proposes a novel method for image similarity models explanation, introducing Salient Attributes for Network Explanation (SANE). The method identifies attributes that contribute positively to the similarity score, thus explaining the important image properties, and pair them with a generated saliency map unveiling the important regions of the image. The method combines three major components: \n- An attribute explanation model\n- A saliency map generator where three \"black box\" algorithms are tested (sliding window, RISE, and LIME) and one \"white box\" (Mask)\n- An attribute explanation suitability prior is computed by the weighted combination of the TCAV scores of an attribute, its confidence score and the matching of its activation map with the generated saliency map\n    \nUsing the saliency maps as supervision for the attribute activation maps seems to improve attribute explanations. The obtained explanations help users understand the model's predictions and build trust.\n    \n\nII Comments\n\nOverall the paper is well written and presents an interesting method for explaining image similarity models. However, from a writing perspective, it can be hard to follow as the paper lacks story-telling as to why such or such methods were chosen/implemented.\n\n1. Content\n- While this work is conceptually interesting, the technical novelty and contributions don't stand out as much as they could. What is the context in improving image similarity explainability? I believe examples in industry or medical could be found to highlight the story of the paper. Why a method is used over another? (TCAV, Mask etc, what lead to this choice?)\n- In 2. Related work, Saliency-based explanations, the paper refers to white-box models but does not offer explanations as to why Mask is chosen over other methods (gradcam, guided backprop etc).\n- In 3.3 TCAV is mentioned, as far as I know, the method works with concepts as images against random images. Here attributes are used as the concepts, how are the random counterparts selected? Moreover, the section on TCAV should be in the related work, whereas how it is used for this specific case would be described in 3.3. \n- In eq 4, â is mentioned but s is used.\n- In 4.2 there is a small user study to verify if the explanations were useful, the study is a nice addition, I really like this kind of results! It would be even more interesting if it compared the results with other baselines.\n- The \"discovering attributes\" part in the appendix is promising, this is something that could be referred to in the conclusion.\n\n2. Writing \n- Intuitive and well-described explanations are given in most paragraphs with examples (3.2, Manipulating similarity scores, the button example) which give a good understanding of the problem. This led to a better comprehension of the challenges.\nSmall typos, did not impact the score:\n- section 3. l 7 explanation -> explain\n- section 4.2, results, l 5 effects -> affects\n\nIII Conclusion\nThe idea is interesting and seems to yield good results, especially in the appendix with the discovering attributes methods. The paper could sell itself a little better with more context/applications where it could be used.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an explainability method for deep visual representation models that have been trained to compute image similarity. 

Strengths:
-The paper tackles an important and overlooked problem.
-The proposed approach is novel and interesting.

Weaknesses:
-The evaluation is not convincing. In particular (i) the evaluation is performed only on ground-truth pairs, rather than on ground-truth pairs and predicted pairs; (ii) the user study doesn’t disambiguate whether users find the SANE explanations better than the saliency map explanations or whether users tend to find text more understandable in general than heat maps. The user study should have compared their predicted attributes to the attribute prediction baseline; (iii) the explanation of Figure 4 is not convincing: the attribute is not only being removed. A new attribute is also being inserted (i.e. a new color). Therefore it’s not clear whether the similarity score should have increased or decreased; (iv) the proposed metric in section 4.2 is flawed: It matters whether similarity increases or decreases with insertion or deletion. The proposed metric doesn’t reflect that.
-Some key details, such as how the attribute insertion process was performed, haven’t been explained. 

The reviewer ratings were borderline after discussion, with some important concerns still not having been addressed after the author feedback period. Given the remaining shortcomings, AC recommends rejection.
Decision: Reject