Analysis of Transformers for Medical Image Retrieval

Authors: arvapalli sai susmitha
iit  kanpur
india  vinay
p  namboodiri
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 11:00:13

Abstract: This paper investigates the application of transformers to medical image retrieval. Although various methods have been attempted in this domain, transformers have not been extensively explored. Leveraging vision transformers, we consider co-attention between image tokens. Two main aspects are investigated: the analysis of various architectures and parameters for transformers and the evaluation of explanation techniques. Specifically, we employ contrastive learning to retrieve attention-based images that consider the relationships between query and database images. Our experiments on diverse medical datasets, such as ISIC 2017, COVID-19 chest X-ray, and Kvasir, using multiple transformer architectures, demonstrate superior performance compared to convolution-based methods and transformers using cross-entropy losses. Further, we conducted a quantitative evaluation of various state-of-the-art explanation techniques using insertion-deletion metrics, in addition to basic qualitative assessments. Among these methods, Transformer Input Sampling (TIS) stands out, showcasing superior performance and enhancing interpretability, thus distinguishing it from black-box models.

Main paper: 
 introduction: 
 Over the past decade, the intersection of medical imaging and deep learning has witnessed significant advancements, addressing challenges in managing vast datasets, as highlighted by Hwang et al. (Hwang et al., 2012) . Content-based medical image retrieval (MIR) has emerged as a crucial tool, aiding clinicians in recognizing related medical images and recalling prior cases during diagnosis (Agrawal et al., 2022) . Traditionally, MIR has heavily relied on convolutional neural networks (CNNs), as evidenced by (Shetty et al., 2023) and (Qayyum et al., 2017) . Despite their effectiveness in feature extraction and similarity identification, they pose challenges due to their inability to effectively capture long-range dependencies and the lack of interpretability, known as the "black-box" problem (Hu et al., 2022) .Our work addresses these challenges in medical image retrieval by adopting Vision Transformers (ViTs) (Dosovitskiy et al., 2021) . ViTs, as explored in (El-Nouby et al., 2021) , offer superior performance, excelling at capturing long-range dependencies and relationships between distant image regions through their multi-head attention mechanism (Zuo et al., 2022) . Inspired by (El-Nouby et al., 2021) work we experiment with Vision Transformers employing contrastive learning and regularization, comparing their performance with convolutional baselines using both cross-entropy loss and contrastive loss under similar settings and also vision Transformers with cross-entropy loss. Our findings demonstrate the superiority of Vision Transformers over these baselines for all datasets. Recognizing the significance of model explainability in clinical applications, We apply various state-of-the-art eXplainable AI (XAI) techniques tailored to Vision Transformers. Beyond simple qualitative comparisons, we conduct quantitative evaluations of saliency maps using insertion and deletion methods. The primary contributions of our work are as follows:• We provide a systematic analysis of vision transformer architectures (Section 4.2) and hyperparameters/variants (Section 4.3). Our findings consistently highlight the superiority of transformers with contrastive loss over convolutional baselines. These also outperform vision transformers and convolution baselines using cross-entropy loss across all datasets.• Analysis of various state-of-the-art explanation techniques for Vision Transformers (Section 4.4). We quantitatively evaluate the resulting saliency maps using insertion and deletion methods. 
related works: 
 Content-based Medical Image Retrieval (MIR) plays a crucial role in enhancing diagnostic reliability for radiologists by retrieving pertinent medical cases that resemble a provided image. The early MIR methods relied on basic features and struggled to capture intricate relationships within medical images. The "semantic gap" between features and actual content led to inaccurate retrieval (Xuan et al., 1995) (Zhang et al., 2008) . Convolutional neural networks (CNNs) revolutionized MIR and bridged the semantic gap by surpassing hand-crafted techniques (Sklan et al., 2015) .In the context of COVID, Kvasir, and ISIC datasets, studies (Tschandl et al., 2019) , (Shetty et al., 2023) , and (Agrawal et al., 2022) leveraged pre-trained CNN architectures like ResNet, VGG, Densenet for medical image retrieval. Recently, (Ahmed et al., 2023) proposed a novel relative difference-based similarity measure (RDBSM) for improved retrieval. Further, ( Öztürk et al., 2023) have introduced an opponent class adaptive margin (OCAM) loss for S-bit hash code generation in image retrieval. CNN-based architectures have been extensively employed in the majority of studies published in the literature. However, the limitations of CNNs in capturing long-range dependencies prompted the exploration of vision transformers (ViTs) (El-Nouby et al., 2021) . Inspired by the capabilities of ViT in general computer vision, we employ them with contrastive loss and differential entropy regularization for the task of medical image retrieval.Few recent studies have explored various ViT-based approaches for Medical Image Retrieval. (Trinh and Nguyen, 2021) present a Mixer-MLP-based MIR for endoscopic images. (Thakrar et al., 2023) use a modified ViT for content-based image retrieval in chest X-rays, employing binary cross entropy and L 1 loss for training. (Gupta et al., 2023) propose denselink-search for efficient nearest neighbours in medical image retrieval. (Manzari et al., 2023) have introduced MedViT, a hybrid model combining ViTs and CNNs for Medical Image Classification. However, these methods may lack interpretability, acting as black boxes in decision-making processes. (Hu et al., 2022) address interpretability concerns with their X-MIR method for CNNs, employing deep metric learning and similarity-based saliency maps for visual explanations of retrieved images. In our work, we undertake a benchmark-ing study to analyse the various architectures and loss-functions and further analyse the interpretability of these methods.Explaining Vision Transformers (ViTs) presents challenges, as traditional attention weight-based methods designed for CNNs are inadequate due to the distinctive nature of ViTs with multiple attention heads and encoder blocks (Serrano and Smith, 2019) (Stassin et al., 2023) . To address this, various ViT-specific explainability methods are explored. Attention Rollout (Abnar and Zuidema, 2020) , Chefer 2 (Chefer et al., 2021) , Transition Attention Maps(TAMS) (Yuan et al., 2021) , Bidirectional Transformers (BT) (Chen et al., 2022) , ViT-CX (Xie et al., 2022) , and Transformer Input Sampling (TiS) (Englebert et al., 2023) contribute uniquely to understanding ViT decision-making, providing insights into attention, gradients, and perturbation-based explanations. The vision transformer (ViT), introduced by (Dosovitskiy et al., 2021) , introduced tokenization of image patches for transformers, by transforming input images into a sequence of 2D patches (e.g., 16x16). These patches undergo a learnable linear projection, resulting in token embeddings. A special learnable [CLS] token is added at the sequence's beginning and serves as a global representation. The transformer encoder block consists of L layers, each of which is composed of two sub-layers: a multi-headed self attention (MSA) layer and a multi-layer perceptron (MLP) layer. The resulting global representation, derived from the [CLS] token output, is used for subsequent processing. 
method: 
 As shown in Figure 1 , we use transformers for content-based MIR.In the training phase, we start with the pre-trained Vision Transformer (ViT) model and then fine-tuned for each dataset with metric learning, specifically employing a contrastive loss. Through this process, the model acquires feature embeddings i.e the cls token acts as the global image descriptor. The process maps similar images into a common feature space and cosine similarity is used to retrieve the closest images. A cross-batch memory (Wang and Isola, 2020 ) is used along with differential entropy regularization. The use of cross-batch memory enables to reduce the dependency on mini-batch for obtaining informative negative pairs. Using a cross-batch memory we are now able to increase the number of hard negatives without incurring significant computational overhead. In the testing phase, we deploy the trained ViT model to generate feature embeddings for each query image in a separate test dataset. These embeddings are then utilized to rank database images, enabling efficient medical image retrieval. Following this, performance metrics such as precision, recall, and F1 score are calculated to assess the efficacy of the retrieval process.In our study, we explore three model variants: MIRDeiT small, MIRViT small, and MIRViT base. MIRDeiT small adopts the DeiT small (Embedding length is 384) (Touvron et al., 2021) architecture with a 16x16 patch size and an image size of 224x224 pixels. Similarly, MIRViT small follows the ViT small architecture (Dosovitskiy et al., 2021) with matching patch and image dimensions (Embedding length is 384). In contrast, MIRViT base is based on the larger ViT base architecture (Embedding length is 768). All these models and the CNN models used for comparison are imported from the timm PyTorch library. The total loss (L) used for training is a combination of the contrastive loss and the differential entropy regularization given as: L = L contr + λL KoLeoThe contrastive loss (L contr ) encourages similarity among samples with the same label and dissimilarity among samples with different labels. Mathematically, it is expressed as:L contr = 1 N N i   j:y i =y j 1 -z T i z j + j:y i ̸ =y j z T i z j -β  Here, z i represents the l 2 -normalized embedding vector of sample i, N is the total number of samples, and y i is the label of sample i. The margin β prevents the training signal from being dominated by easy negatives. Only negative pairs with a similarity higher than a constant margin β contribute to the loss. The representations z i are assumed to be l 2 -normalized, making the inner product equivalent to cosine similarity. Simultaneously, L KoLeo is the differential entropy loss. It serves as a regularizer (Sablayrolles et al., 2018) and is based on the (Kozachenko and Leonenko, 1987) differential entropy estimator. It prevents representations of different samples from being close by increasing their distance from positive examples and hard negatives. Mathematically, L KoLeo aims to maximize the distance between each point and its nearest neighbor:L KoLeo = - 1 N N i=1 log(ρ i )Here, ρ i represents the minimum distance between the embedding vector z i and any other embedding vector z j (where j ̸ = i). The regularization term is then used with a weighting coefficient λ.During testing, each image in the test dataset, as depicted in Figure 1 , acts as a query, with the top k retrievals extracted for each query. The learned embedding network processes input images, denoted as x, producing embedding feature vectors z(x). This process applies to both query images (q) and retrieved images (r), resulting in feature vectors z q and z r , respectively. To rank the retrieved images, a similarity score s between z q and z r is computed using cosine similarity: s(z q , z r ) = zq•zr ∥zq∥∥zr∥ . Standard image retrieval metrics, such as mean average precision (mAP), mean precision (mP@K), and Recall (R@K), are calculated for evaluation.While vision transformers (ViTs) employ attention mechanisms, relying solely on raw attention is considered insufficient for comprehensive explanations. This raw attention overlooks the value component and emphasizes the query and key elements (Jain and Wallace, 2019; Serrano and Smith, 2019) , This has led to the development of methods specifically made for ViTs. For instance, attention rollout (Abnar and Zuidema, 2020) combines attention heads and an identity matrix for residual connections. Chefer 2 (Chefer et al., 2021) offers a generic explanation method for transformers, using gradients and identity matrices for attention score computation. TAMs (Yuan et al., 2021) model representation evolution as a Markov chain, yielding class-specific explanations with integrated gradients. Bidirectional Transformer (BT) involves element-wise multiplication of Reasoning Feedback and Attention Perception (Chen et al., 2022) , providing saliency maps for token (BT-T) and head (BT-H). ViT-CX (Xie et al., 2022) avoids direct dependence on attention weights, using perturbation masks derived from patch embeddings. Transformer Input Sampling (TiS) (Englebert et al., 2023) instead masks tokens before their introduction into a transformer, improving interpretability and reducing the number of tokens. We have compared a number of the above explainability methods to obtain explainable medical image retrieval and provide results for the same in section 4.4. 
datasets: 
 In our comprehensive study, we utilize three datasets for medical image analysis. The curated COVID-19 Chest X-Ray Dataset (Sait et al., 2020) includes 1281 COVID-19 Xrays, 3270 Normal X-rays, and 4657 pneumonia X-rays (viral and bacterial). Our focus is on overall pneumonia classification. The ISIC Skin Lesion Dataset (Codella et al., 2018) comprises images of benign nevi, seborrheic keratosis, and melanoma (non-cancerous and malignant). This dataset has 2,750 images. The Kvasir-V2 dataset (Pogorelov et al., 2017) contains 8,000 annotated endoscopy images, categorized into eight classes by experienced endoscopists based on anatomical landmarks, pathological findings, or specific endoscopic procedures.We adopted the same training details as outlined in (El-Nouby et al., 2021) , the optimization of these models employs the AdamW optimizer with a learning rate of 3 × 10 -5 , weight decay of 5 × 10 -4 , and for 10k iterations. Contrastive loss margin (β) is set to 0.5. In the absence of regularization (λ = 0) and with differential entropy regularization, different variants of λ (λ = 0.3, 0.7) are employed. Standard data augmentation techniques are applied, including resizing images to 256 × 256, random cropping to 224 × 224, and random horizontal flipping. The dynamic offline memory queue aligns with the dataset's size. In the case of cross-entropy, similar optimizer and iteration settings were used, applying basic cross-entropy loss for classification.In our diverse evaluation across ISIC, COVID, and Kvasir datasets, as detailed in Table 1, we tested traditional CNNs (Densenet121, Resnet50), various vision transformers (DeiT small, ViT small, MedViT), and the MIRViT variants. MIRViT small consistently outperforms CNNs, other vision transformers, and even MedViT(CNN-Transformer Hybrid) in medical image retrieval, demonstrating its efficacy.In Table 1 , to maintain conciseness, the results for ISIC are derived from λ = 0.3 for MIRViT small and λ = 0 for MIRDeiT small, while for Kvasir and COVID, MIRViT small is based on λ = 0.7 and MIRDeiT small is based on λ = 0.3 and λ = 0.7 respectively.The above results are based on three experimental runs. The complete set of results of the model combinations are provided in Figure 2 .Model Loss R@1 R@5 R@10 mAP mP@1 mP@5 mP@10 MIRViT small's superiority is evident, for example for the ISIC dataset, achieving higher recall and a significant 13.24% increase in mAP compared to Resnet50. It also outshines vision transformers trained with cross-entropy loss, emphasizing its precision in top-k scenarios, with a notable 7.64% surge in mAP compared to DeiT small. Similarly, across the COVID and Kvasir datasets, MIRViT small outperforms convolution-based methods and transformers using cross-entropy losses, showcasing versatility in diverse medical imaging contexts. MedViT, while excelling in classification, lags in retrieval. In summary, vision transformers, especially MIRViT small, exhibit good potential in medical image retrieval. Their consistent superiority in recall, precision, and mAP underscores their effectiveness. Use of contrastive loss, with or without differential entropy regularization is beneficial.The assessment of image retrieval results in Figure 2 consistently shows the dominance of MIRViT small over MIRDeiT small and MIRViT base across diverse datasets, including ISIC, COVID, and Kvasir. In the ISIC dataset, MIRViT small consistently outperforms other configurations, delivering optimal performance at λ = 0.3. For the COVID and Kvasir datasets, MIRViT small at λ = 0.7 achieves the highest mAP, closely followed by λ = 0.3 with a small difference. Notably, MIRViT base does not emerge as the top performer for all datasets, indicating that a 384 embedding length performs well for retrieval on these medical datasets.In conclusion, MIRViT small with λ = 0.3 emerges as an optimal choice for medical image retrieval tasks. To assess visual explanations, we employ insertion and deletion casual metrics in image retrieval, gauging how well the generated explanations capture the causes behind predictions. We measure changes in image similarity as a result of changes to the retrieved image. The insertion metric measures increased image similarity by starting with a blurred version of the original retrieved image and gradually revealing pixels from highest to lowest relevance. Conversely, the deletion metric assesses the decline in image similarity we gradually mask out pixels on the retrieved image with a constant gray value from highest relevance to lowest based on the computed saliency map. We then compute the similarity score s between the query image q and perturbed versions of the retrieved image r (either in the form of insertion onto a blurred image or deletion using a constant gray value) s(z q , z r) = (max(0, zq•z r ∥zq∥∥z r∥ )).To rectify non-negative outputs, all similarity values are adjusted to a minimum of zero. The area under the curve (AUC) is used to measure the effectiveness of saliency maps. Higher AUC values are preferred for insertion, and lower AUC values are desirable for deletion.We implemented all explainability methods using the same hyperparameters as in (Englebert et al., 2023) . In the saliency maps, vibrant (red) regions signify the primary focus, while cooler (blue) areas have less impact. The maps are rescaled using bilinear interpolation to align with the input image resolution.Chefer2 and TIS consistently emerge as strong performers Table 2 , shows top-tier scores across datasets and metrics. However, the qualitative evaluation through visual explanation maps shown in Figure 3 suggests that TIS is qualitatively better. On the other hand,  
conclusion: 
 In conclusion, our study presents an analysis of various architectures and parameters for transformers, along with the evaluation of explanation techniques. MIRViT small emerges as the top-performing model across varied datasets ISIC, COVID, and Kvasir. Our exploration of loss functions underscores the limitations of simple cross-entropy and the effectiveness of the contrastive approach. Further, our analysis of state-of-the-art eXplainable AI methods suggests Transformer Input Sampling (TIS) as being better. In future, we intend to further explore various advances for these algorithms for the medical image retrieval task. As can be observed from the table 3, the MIRViT small and MIRDeiT small have fewer parameters than ResNet50 and have improved performance. DenseNet model is more compact, but in general, the parameter settings for other small and base models are comparable. 
 
 Appendix:  



Reviews: review: paper_summary: paper_summary: The paper performs a benchmarking analysis of three Vision Transformer (ViT) models for medical image retrieval across three datasets encompassing chest x-ray, dermoscopy, and endoscopy images. It compares contrastive loss against cross-entropy loss and highlights the superiority of transformers trained with contrastive loss. Furthermore, the study delves into seven model interpretability techniques, providing both quantitative and qualitative evaluations, ultimately identifying Transformer Input Sampling as the most effective approach.
main_review: main_review: I am very grateful to the authors for responding positively to my review comments and making the appropriate changes in the original manuscript. I believe the revised version has been improved, and I updated my rating accordingly.
strength_weakness: strength_weakness: 1. Figure 1 illustrates the input of both the query image and the database image to ViT simultaneously, yet lacks explicit explanation, leading to confusion about the workflow.
   
2. The method description should provide further clarification on the cross-batch memory. Specifically, how does this mechanism facilitate the mapping of similar images into a common feature space?

3. Section 4.3 concludes that MIRViT_small with λ = 0.3 emerges as an optimal choice for medical image retrieval tasks. However, this setup does not consistently yield the best performance across all three tasks, rendering the argument less tenable.

4. The introduction to the insertion and deletion causal metrics lacks clarity regarding why and how these metrics quantify the explainability of the techniques.
questions: questions: The question listed in the Weaknesses needs to be clarified.
paper_summary: paper_summary: This paper investigates various vision transformers that incorporate contrastive learning for medical image retrieval. The authors conducted a systematic analysis of vision transformers and their hyperparameters, showcasing the superiority of transformers with contrastive loss over convolutional baselines. They demonstrated that the contrastive loss aids in outperforming vision transformers and convolutional baselines that rely solely on cross-entropy loss. The authors evaluated their methods across three distinct and diverse datasets: ISIC 2017, COVID-19 chest X-ray, and Kvasir.
main_review: main_review: The authors have satisfactorily addressed all of my questions and have made necessary adjustments in their experiments. However, I believe that exploring other variants of Swin Transformers would significantly enhance the strength of their paper.
strength_weakness: strength_weakness: 1. The authors could extend their investigation to include different variations of Swin Transformers for the Medical Image Retrieval task.
2. It would be intriguing to observe the impact of the formulated contrastive loss on Swin Transformers
3. It is not evident from the manuscript how many times each experiment was repeated. Demonstrating statistical significance across multiple runs would enhance the paper.
4. The authors could illustrate the convergence speed of the model using contrastive loss versus cross-entropy loss;
questions: questions: Please see the points included in the section "Weaknesses".
paper_summary: paper_summary: The study explores the use of transformers in medical image retrieval. Results from experiments conducted on three datasets indicate that a transformer trained with contrastive loss surpasses convolution-based methods and transformers trained with cross-entropy loss. Additionally, a quantitative assessment of several state-of-the-art explanation techniques is performed, revealing superior performance of TIS.
main_review: main_review: I would be grateful for the authors' feedback. most of my concerns have been addressed, with the exception of the issue regarding novelty. I am confident that the readability of the revised version has been improved, and I believe that the community stands to gain valuable insights from this work. Therefore, I have decided to adjust my rating accordingly.
strength_weakness: strength_weakness: • Despite being an analysis paper focused on the backbone, the reviewer remains apprehensive about its novelty. 

• The pretraining and finetuning strategy lacks clarity.

• Figure 1 lacks clarity in its presentation. The inclusion of both training and testing phases is confusing. 

• The presentation of results in Table 1 is unclear, and the comparison between different backbones may not be equitable.

• More backbones such as Swin Transformer and CNN-Transformer hybrid backbones shall be included and evaluated.
questions: questions: Please provide a detailed discussion of this paper’s novelty, and please respond to the questions in the Comments section.

Comments: comment: We thank the reviewer for their comments and suggestions. Below are the clarifications for their comments:

1) The primary contributions of the paper include the analysis of transformers with contrastive loss for medical image retrieval and the evaluation of explanation methods. While these methods are well-established for classification on datasets like ImageNet, their application in medical image retrieval settings is not so well analyzed. With this work, we seek to provide strong comparable baselines that can be further used by the community. As is evident, we are not advocating a new loss function or architecture in this paper. However, we believe the community does benefit from the benchmarks and analysis of techniques that we provide. We intend to provide these models, code, and analysis techniques as open source to enable the community to benefit further from this work on acceptance.

2) The suggestions by the reviewer have been incorporated into the paper (page 4 highlighted in red). Thanks for the suggestions.

3) We adopt the CLS token embedding for both training and testing. Specifically, the CLS token for each query image in a separate test dataset is computed and used to compute the cosine distance to the retrieval set of images. 

4) The complete set of results for the combinations is provided in Figure 2. In Table 1, to maintain conciseness, the results for ISIC are derived from λ=0.3 for MIRViT_small and λ=0 for MIRdeiT_small, while for KVASIR and COVID, MIRViT_small is based on λ=0.7 and MIRdeiT_small is based on λ=0.3 and λ=0.7 respectively. It is worth noting that, as depicted in Figure 2, adopting λ=0.3 across all settings would not be detrimental.  In general, we would recommend using cross-validation for practical use in clinical settings. We have expanded our explanation on page 6 (highlighted in red) for better understanding.

5) As recommended by the reviewer, we have now included the number of parameters for each architecture in the appendix, page 13 as Table 3. As can be observed from the table, the MIRViT_small and MIRdeiT_small have fewer parameters than ResNet50 and have improved performance. The DenseNet model is more compact, but in general, the parameter settings for other small and base models are comparable. These results are consistent with classification performance results.

Also as recommended by the reviewer, it would indeed be beneficial to incorporate comparisons with several additional transformer baselines, including the Swin Transformer. However, we have access to limited computational resources and therefore, if we obtain access to sufficient computational resources, we will definitely include Swin transformer and additional transformer baselines. We believe that the present usage of ViT and deiT does provide an indication of the performance capabilities of transformers for the task of medical image retrieval.
We thank the reviewer for their comments and suggestions and their appreciation of the work. We provide clarifications below for the points raised and we have also amended the paper based on these points.

1) Thanks to the reviewer for appreciating the work and recommending the analysis using Swin transformers. We intend to incorporate this analysis in the future. At present, due to computational bottlenecks, we have not been able to do the same.

2) Yes, indeed, we find the idea interesting. As mentioned, we plan to explore the analysis using Swin transformers in the future. We appreciate the suggestion to compare the contrastive loss using Swin transformers.

3) Thank you for raising this point. We have run each experiment three times. In the revised version we provide the mean and standard deviation for each experiment in Table 1. As can be observed there is less standard deviation.

4) The models using cross-entropy loss have converged around 2000 epochs and contrastive loss have converged around 6000 epochs.
Thanks to the reviewer for their comments and suggestions. We have revised the paper based on the same and we provide the response below for the comments:

1) As recommended by the reviewer, we have now revised Figure 1 to clearly depict both the training and testing phases separately. The updated figure is now presented on page 3. Moreover, we've added further explanations about the training and testing processes, highlighted in red on page 4.

2) As requested by the reviewer, we have expanded the explanation of cross-batch memory usage on page 4. Essentially, incorporating cross-batch memory enables us to reduce the dependency on mini-batch for obtaining informative negative pairs. With the implementation of cross-batch memory, we can now increase the number of hard negatives without incurring significant computational overhead.

3) As the reviewer has correctly identified, λ = 0.3 is not always the optimal choice. The other parameter that did do well was λ =0.7 which however did not do well on the ISIC dataset. As a recommendation for practical usage in a clinical setting, we would always advocate a cross-validation of the parameter setting on the desired actual dataset and the analysis provided is mainly an indicative recommendation for initial consideration.

4) As recommended by the reviewer, we have now expanded the explanation of these metrics on page 7 highlighted in red. The insertion and deletion metrics involve intervention-based analysis. We modify the samples and observe the difference in performances. For instance,  when we blur part of an image, we can gauge the significance of the unblurred regions for retrieval. Qualitative examples of the resulting maps are presented in Figure 4 on page 9. While the analysis methods  were originally introduced for classification in Petsiuk, Vitali et al. “RISE: Randomized Input  Sampling for  Explanation of Black-box Models.” ArXiv abs/1806.07421 (2018)., we illustrate its utility for retrieval analysis.