HARP: Unsupervised Histopathology Artifact Restoration

Authors: moritz  fuchs
mirko  schöber
niklas  woltering
marie-lisa  eich
leonille  schweizer
anirban  mukhopadhyay
s  fuchs
m  kumar
n  schöber
m.-l  woltering
l  eich
a  schweizer
  mukhopadhyay
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 17:35:25

Abstract: Histopathological analysis, vital for medical diagnostics, is often challenged by artifacts in
sample preparation and imaging, such as staining inconsistencies and physical obstructions.
Addressing this, our work introduces a novel, fully unsupervised histopathological artifact
restoration pipeline (HARP). HARP integrates artifact detection, localization, and restoration
into one pipeline. The first step to make artifact restoration applicable is an analysis
of anomaly detection algorithms. Then, HARP leverages the power of unsupervised segmentation
techniques to propose localizations for potential artifacts, for which we select the
best localization based on our novel inpainting denoising diffusion model. Finally, HARP
employs an inpainting model for artifact restoration while conditioning it on the artifact localizations.
We evaluate the artifact detection quality along with the image reconstruction
quality, surpassing the state-of-the-art artifact restoration. Furthermore, we demonstrate
that HARP improves the robustness and reliability of downstream models and show that
pathologists can not tell the difference between clean images and images restored through
HARP. This demonstrates that HARP significantly improves image quality and diagnostic
reliability, enhancing histopathological examination accuracy for AI systems.

Main paper: 
 introduction: 
 Histopathological analysis stands at the forefront of diagnostic medicine, informing critical decisions with life-altering implications. Yet, the reliability of such analysis is often compromised by artifacts introduced during sample preparation and imaging, ranging from staining inconsistencies (Tellez et al., 2019) to physical obstructions like folds and blood cells (Kanwal et al., 2022) . These artifacts can distort the data, leading to diagnostic inaccuracies of the employed AI (Schömig-Markiefka et al., 2021; Wang et al., 2021b) .While in clinical practice, whole slide images (WSI) can be rescanned to address these issues, the restoration of corrupted histopathological images with an AI model offers an effective alternative to improve image quality without such labor and time-consuming process. Recent methods have ventured into this territory with supervision-heavy approaches, demanding extensive manual input (He et al., 2023) or supervision (Dahan et al., 2022; Ke et al., 2023) on patches containing artifacts in order to restore a WSI. Approaches that explicitly learn each artifact type's appearance by supervision will struggle with unseen artifacts and fail to restore the image (He et al., 2023) , especially where unseen artifacts are common and manual intervention is costly. These supervisions usually render such approaches unreliable and labor-intensive when pursuing a holistic pipeline. To the best of our knowledge, there has not been a holistic, unsupervised approach that detects the artifacts and performs image restoration in one pipeline. Recognizing this gap, we introduce a fully unsupervised Histopathological artifact restoration pipeline (HARP) that deploys the three steps depicted in Figure 1 , which are essential for a clinical workflow for computational pathology: artifact detection, artifact localization, and artifact restoration.In order to make HARP useable in the clinical workflow, the first step is to reliably detect artifacts. Many studies in recent years have developed unsupervised anomaly detection methods, which are able to identify unusual images (Wang et al., 2021a; Yu et al., 2021; Zavrtanik et al., 2021) . We evaluate anomaly detection methods with the AnomaLib framework (Akcay et al., 2022) on histopathology images with realistic and proven synthetic artifacts from Stieber et al. (2022) . In the second step, we require the localization of the relevant artifacts. This step is crucial for the applicability of our pipeline to generalize to many different artifact types without requiring knowledge of them. Based on the original image, we generated multiple localization masks of artifacts by leveraging pre-trained knowledge of SAM (Kirillov et al., 2023) and clustering with DBSCAN (Ester et al., 1996) . As we train a diffusion model to restore the images, we leverage it to generate an activation map, which we deploy to rank the top 5 masks. In the final step, we conduct the artifact restoration. We generate a restored image for each localization by conditioning our diffusion model to inpaint the image based on the localization mask. Recent works (He et al., 2023) have shown great results on artifact inpainting with the RePaint (Lugmayr et al., 2022) and manually annotated artifacts. This approach has the limitation of being computationally heavy, rendering it unsuitable for a clinical workflow. HARP leverages our novel inpainting denoising diffusion model, incorporating the condition input at every step, which reduces the required computational time. Lastly, we select the final image with the artifact detection method. In summary, our contributions are threefold. We evaluated existing anomaly detection methods for histopathological artifacts. Secondly, we develop a (I) novel conditional image inpainting denoising diffusion model. Further, we demonstrate its capability for (II) artifact localization and restoration, and we evaluate the (III) impact of HARP on downstream model performance. Lastly, we evaluate the restored images by conducting a (III) study with pathologists. 
related work: 
 In the niche of histology imaging, there have only recently been efforts to restore artifacts, in contrast to the extensive work in general computer vision where object removal and image inpainting is a long-standing topic (Zhang et al., 2023) . In standard computer vision applications, techniques for object removal, such as RePaint (Lugmayr et al., 2022) , Palette (Saharia et al., 2022) , DDRM (Kawar et al., 2022) , or GAN-based networks (Zhang et al., 2020) , have been employed to address issues like shadow elimination and image blurring. These methods, however, typically hinge on certain prerequisites: target predictability, pre-defined object masks, clean image pairs, or extensive datasets (Xiang et al., 2023) .Despite these advancements, these methods have limited direct application for histopathological image restoration, and data scarcity and annotation costs are rampant issues in medical imaging. In recent years, many papers have tackled the topic of stain normalization (Faryna et al., 2021; Wagner et al., 2022) and data augmentation (Tellez et al., 2019) in order to improve model generalization. However, the domain of artifact restoration in histopathological images has only seen the emergence of the first supervised methodologies (Dahan et al., 2022; Ke et al., 2023; He et al., 2023) , requiring considerable manual input or supervision on artifacts for effective WSI restoration. While these approaches represent innovative strides, they are often impractical for real-world application due to their inability to cope with unseen artifacts or require costly manual input. Consequently, their assessments lack real clinical settings complexities and variability. These inherent limitations undermine the reliability and are impractical in clinical practice. HARP addresses this gap by proposing an approach that focuses on preserving critical features, thereby ensuring that the artifacts are restored without compromising the fidelity of the histological images. 
method: 
 HARP involves three steps that we are explaining in order. First, we give a brief explanation of the artifact detection. Then, we explain the training of the novel inpainting diffusion model, which we leverage for artifact localization and artifact restoration.Artifacts Detection: In HARP, the initial step is the efficient and reliable detection of artifacts, which is crucial for effective restoration in clinical workflows. Utilizing AnomaLib, we explored various anomaly detection methods tailored for histopathology images. Among these, we identified FastFlow (Yu et al., 2021) , which employs a Vision Transformer (ViT) Encoder to conduct normalizing flow on latent representations, as the most suitable for our needs. This method demonstrated superior performance in artifact detection and was easily integrated into our pipeline. The implementation of this method marks the beginning of our artifact detection phase in HARP, as illustrated in Figure 2 , setting the stage for accurate histopathological image restoration and enhancing diagnostic reliability.Training the Conditional Diffusion Model: The purpose of conditional generative models is to estimate the data distribution p(y|x), where x ∈ [0, 1] H×W is a conditional image and y ∈ [0, 1] H×W is the target, with height H and width W . Given this goal, Denoising Diffusion Models (DDMs) operate by reversing the process of gradually introducing Gaussian noise into image samples y 0 ∼ p(y 0 ). After a series of T = 1000 diffusion steps, the resulting sequence y 1 , ..., y T converges towards a Gaussian noise profile, particularly as T approaches infinity. Given a well-calibrated variance schedule β 1 , ..., β T ∈ (0, 1) T , small steps and large T , we train a denoising model f θ to reverse each step in this sequence. The reverse distribution is defined by the conditional: p θ (y t-1 |y t , x) := N (y t-1 ; µ θ (x, y t , t), Σ θ (x, y t , t)).Here, µ θ and Σ θ are predictions given the model parameters θ. To condition our denoising model f θ for inpainting, we randomly mask out an area x := y T * m + y 0 * (1 -m), with a binary mask image m ∈ {0, 1} H×W . With p(y T |x) being a known distribution, one can initiate the reverse process by sampling y T ∼ N (0, I) and iteratively applying the denoising model, thereby transforming it back into the conditional data distribution. The training of f θ is simplified to the following loss (Ho et al., 2020) :E y 0 ∼p(y 0 ),ϵ∼N (0,1),m,t [∥f θ (x, ( √ ᾱt y 0 + √ 1 -ᾱt ϵ) * m + y 0 * (1 -m) ỹt , t) -ϵ ∥ 1 ] (1)with ϵ ∼ N (0, I), and the factorization α t := 1 -β t and ᾱt := t s=1 α s . The novel approach of incorporating x in each step of the denoising process is critical for our model's performance and allows for fewer RePaint (Lugmayr et al., 2022) cycles during artifact restoration.Artifact Localization: In order to localize potential artifacts without supervision, we gather an activation map of our novel denoising diffusion model f θ when conditioned on the entire noised image x := ȳT . If we noise an artifact image ȳ up to 900 steps and denoise again, all the major features tend to stay the same, but minor details on known structures change slightly. As our model does not know artifacts, minor details do not change on artifacts. Using this, we calculate the reconstruction error: ∥ȳ 0 -y 0 ∥, which aggregate as an activation map over 25 denoising repetitions. We prompt SAM (Kirillov et al., 2023) and DBSCAN (Ester et al., 1996) to generate object localizations on the artifact image y 0 . We remove masks that are too large(> 60%) and small(< 0.4%), as reconstruction would be unfeasible or not worthwhile, and sort out duplicated masks. Then, we score each mask by aggregating the activations over the mask area and select the top 5 binary masks m 0 , ..., m 4 based on the lowest activation received. Finally, we dilate the masks to smooth the inpainting boundary of artifacts. Most importantly, this is -to the best of our knowledge -the first method to localize histological artifacts without any supervision.Image Restoration: In order to restorate an artifact image ŷ0 , we condition on an image x := ŷT with ŷt := ŷt * m + ŷ0 * (1 -m), where m is one of the previously determined artifact localizations. To harmonize the masked area with the image, we apply a denoising procedure to our novel inpainting denoising diffusion model similar to RePaint (Lugmayr et al., 2022) with resampling = 3 and jumpsampling = 10 to generate the restorated and artifact-free image ỹ. The best reconstruction from the 5 gets chosen by FastFlow (Yu et al., 2021) , having the least probability of containing an artifact. 
experiments and results: 
 In this section, we start by introducing the dataset, artifacts, and backbones. Second, we evaluate artifact detection methods for each artifact. Next, we quantify the restoration qualities of our model with ground truth artifact masks and our localization maps. The most important ability for artifact restoration is its application in the clinical workflow for computational pathology. For this reason, we evaluate a state-of-the-art downstream model on artifact images with and without the use of HARP and conduct a user study to determine whether there is a difference between a clean image and the outputs of HARP.Dataset and Artifacts: We evaluate all methods of the Breast Cancer Semantic Segmentation dataset (BCSS) (Amgad et al., 2019) , for which we use an FCN8 architecture proposed in the paper for the downstream task evaluation. While BCSS contains a multitude of labels, we focus on the four predominant classes (% of labels): tumor (45%), stroma (17%), lymphocyte-rich tissue (35%) and necrosis (3%). We train our inpainting denoising diffusion model and the downstream task model using 11075 training, 1031 validation and withhold 1000 test patches. The diffusion model backbone is based on the guided diffusion model from github.com/Janspiry/Palette-Image-to-Image-Diffusion-Models. From each region of interest within a WSI, we sample random crops of size 600x600 with 0.24 mpp, which we resample to 256x256. We train our models using an NVIDIA RTX 4090. Our code is available here: github.com/MECLabTUDA/HARP . Leveraging previous works (Stieber et al., 2022) , we adopted the following artifacts: dark spots, fat drop, squamous epithelia, threads, blood cell and blood group, compression, cuts, overlap and folding, which have been shown to be realistic and detrimental to downstream performance by (Wang et al., 2021b; Babendererde et al., 2023) . We generate 100 samples for each artifact.Artifact Detection: The first step to deploying artifact restoration efficiently in a clinical workflow is to detect artifacts reliably. We evaluate various methods from Anoma-Lib (Akcay et al., 2022) on the 4 local artifacts from Schömig-Markiefka et al. ( 2021), which can be found in the appendix. From these methods, we chose the best three methods: DRAEM (Zavrtanik et al., 2021) , FastFlow (Yu et al., 2021) and STFPM (Wang et al., 2021a) and validate them using the validation set to determine the quantile for which each method keeps 95% of normal samples (dashed lines). Figure 3 shows how this affects the detection of all artifact types. We select the FastFlow method for the pipeline, as it only misses most artifacts from squamos epithelia and has the best accuracy with 85.8%. DRAEM and STFPM are strong competitors; DRAEM misses most of the artifacts for squamous epithelia, cut, and overlap. DRAEM average accuracy is 83.9%. STFPM, on the other hand, misses squamos epithelia, thread, and cut artifacts with an average accuracy of 74.9%.Artifact Restoration Quality: Now, we want to assess the image quality generated by the artifact restoration model and the full HARP pipeline. As HARP is, to the best of our knowledge, the first fully unsupervised method, we compare to three methods that either require artifact images during training like AR-CycleGAN (Ke et al., 2023) or manual input by providing localization masks like ArtiFusion (He et al., 2023) and DDRM (Kawar et al., 2022) . To provide localizations reliably to all models, we leverage the ground truth segmentation masks of the artifacts from Stieber et al. (2022) . We train ArtiFusion and DDRM on the BCSS with the same backbone architecture as our model. Further, as AR-CycleGAN requires a dataset representing artifacts, we used the original dataset and deployed it to our test cases. We evaluate 100 artifact images per artifact type in Table 1 and 2 for Fréchet Inception Distance (FID ↓) and Peak Signal Noise Ratio (PSNR ↑), where the arrow indicates the direction of improvement. Our artifact restoration model performs best for 13 out of 20 results, and DDRM offers minimal improvements on 5 of the metrics and ArtiFusion on 1. Ours has a reduced runtime of 18.6 sec. per image vs. 30.9 sec. for ArtiFusion and 37.4 sec. for DDRM. AR-CycleGAN fails to generalize to the unseen artifact domains, demonstrating supervised training limits. When looking at our fully unsupervised HARP method, we see that it can even improve on the dark spot artifact, which is likely due to a better mask. Evaluating the artifact localizations and the ground truth masks by DICE, HARP achieves 54.5%. However, HARP limitations are when the localizations are suboptimal, e.g., for blood cells and groups. This is likely due to one of three reasons: First, the artifact detection failed. Second, these artifacts are often spread out over the whole image, contrary to other artifacts. Third, the training set likely contains some artifacts, leading to a reproduction of the same artifacts in the image. However, when the segmentation mask is appropriate, the results are excellent, as supervised results show. The key area for improvement lies in the quality of unsupervised localization masks, as they significantly contribute to the artifact restoration process. Downstream Application: To evaluate the usability of HARP for the clinical workflow of computational pathology, we evaluate the segmentation performance of the downstream model using clean images, artifact images, artifact images excluding artifact segmentations, and restored images with HARP. These images all have the same underlying image and segmentation mask from the test, for which we calculate the DICE score per class and the average. It is important to note that AI-generated images pose a risk for accurate diagnoses, similar to undisclosed deepfakes; therefore, we ensure transparency in our process by excluding these contents using HARP's artifact localization. As seen in Table 4 , the performance of the artifact images significantly decreases compared to the originally clean images. HARP is able to recover the artifact images and effectively reduces the performance drop introduced by the artifacts by 48%, which makes the downstream model more robust and reliable for the daily clinical workflow.User Study: Finally, we conducted a user study with four pathologists on 50 image pairs as a visual turning test on the produced image quality. One of the images from the pair is a normal image from the training distribution, and the other is an image from the test distribution augmented with an artifact and then processed with HARP. We use 5 82.2 ± 0.5 82.0 ± 0.9 78.5 ± 2.3 69.3 ± 2.8 78.0 ± 1.0 artifact images from each of the 10 artifact types. The pathologists were given instruction to conduct the study on 256x256 images with 100% scale and not to zoom to avoid image interpolation artifacts from the preprocessing of all images affecting the study. The study was timed in order to ensure that participants followed a standard clinical workflow. We calculate the Matthews correlation coefficient (MCC) for each participant and give the number of falsely classified samples. The participants achieved the following scores: -0.071 MCC (27/50), -0.159 MCC (30/50), 0.239 MCC (20/50), and 0.296 MCC (18/50) with the times 7:34 min, 5:50 min, 4:45 min, and 7:00 min, respectively. All our participants found it impossible to tell the real difference between images, as our results suggest, at best, there is a weak positive correlation by chance. This further demonstrates the potential of HARP and the risks of not disclosing generated content for the clinical workflow. We give more results and a sample of five images from the study in the appendix. Dangers and Impact: Generative AI can not go unlabeled -as the EU AI Act (EU, 2021) suggests. In pathology, Generative AI risks misleading the diagnosis done by pathologists and other AIs, which we showed with our user study. Therefore, we exclude the artifact localization masks in the downstream evaluation and recommend highlighting them. Nonetheless, computational pathology has the promise of saving time and increasing diagnostic accuracy for patients, for which HARP is a supportive structure to ensure reliability. Further, it has the dual benefit of improving the image quality without rescanning. 
conclusion: 
 In conclusion, our work presents the Histopathological Artifact Restoration Pipeline (HARP), a novel and fully unsupervised approach -the first of its kindfor restoring artifacts in histopathological images. HARP efficiently integrates artifact detection, localization, and restoration into one seamless clinical workflow for computational pathology. Our work significantly enhances image quality and diagnostic accuracy by leveraging advanced artifact detection, deploying a novel unsupervised artifact localization technique, and presenting a new state-of-the-art inpainting denoising diffusion model. We evaluate the artifact detection quality along with the image reconstruction quality, surpassing the state-of-the-art artifact restoration methods. Furthermore, we demonstrate that HARP can improve the robustness and reliability of downstream models. Finally, we show that pathologists can not tell the difference between clean images and images processed through HARP. This breakthrough in medical image processing holds immense potential for improving AI systems' reliability in histopathological examinations.Table 5 : DICE scores between the ground truth (GT) mask from (Stieber et al. 2022) and our unsupervised localizations for each type of artifact. We further compare HARP localizations undilated (u.) with the GT masks to assess whether our initial localization under-segments the GT; as a result, we see that the dilation in HARP is necessary. The DICE only decreases significantly for dark spots. However, we see in where one is from the original training distribution and one is a test sample that had an artifact, which got removed from HARP. We then asked the participants to label which image had an artifact and went through the pipeline. Spoiler: Right answers from left to right: Bottom, Top, Top, Bottom, Top Figure 5 : Qualitative samples from HARP for each artifact: We selected random samples from the HARP. For the blood cell and group artifacts, it clearly shows the failure case, when not everything is selected by the localization mask, usually due to a lack of a better localization proposal. The Air Bubble case is a rare sample that has a natural fold/compression artifact that got segmented with the air bubble and removed. 
 
 Appendix:  


Decision: The paper describes an effective pipeline for artifact removal in histopathology images. The reviewers see merit in the work, but also raise questions regarding the methodological novelty and the practical application of the work. The authors have actively engaged during the rebuttal period, addressed some of the reviewers' concerns and adapted their paper accordingly.

Reviews: review: paper_summary: paper_summary: In this paper, the authors proposed HARP, an unsupervised histopathological artifact restoration pipeline. This framework detects, localizes and restores the image with artifacts. With ground truth masks, their method outperforms the SOTA methods. Although in a fully unsupervised manner, it still fall behind current supervised methods, it is very interesting and useful in practice.
main_review: main_review: The authors proposed a pipeline to conduct artifact restoration in a unsupervised manner, which is very interesting to me. The rebuttal solve most of my concerns and I think this is a good paper for MIDL.
strength_weakness: strength_weakness: To my understanding, the performance of the proposed method heavily depends on the quality of the 1st step (artifact detection) and 3rd step (artifact localization). 
1. In the 1st step, the authors used a general anomaly detection method for artifact detection. It is a general method, I have a concern about it performance in the pathology domain.
2. Some parts in the 3rd step are unclear to me.
questions: questions: As stated in the Weaknesses and major concerns in the Detailed Comments.
paper_summary: paper_summary: The paper presents a pipeline for unsupervised artefacts detection in histopathology images, including artefact localisation and restoration with pinpointing.
main_review: main_review: The paper has limited methodological novelty. I find the advocated use of restoration for artefacts in histopathology problematic (particularly without discussion of limitations and dangers) and perhaps unnecessary.
strength_weakness: strength_weakness: The pipeline is a combination of several existing methods, without much methodological novelty. My main concern is the artefact restoration process, which in my opinion is unnecessary and constitutes solving a more difficult task (restoring the image/eliminating the artefacts) than needed. As histopathology images are vast, an area with an artefact once detected can simply be ignored/not processed in downstream processing. The process of restoration is dangerous as it can introduce non-existent information or remove important information (in the case of of a false positive).
questions: questions: The dangers and limitations of the "restoration" component of this pipeline should be discussed. 

A better baseline should be presented that compares the methods to simply ignoring detected areas with artefacts.
paper_summary: paper_summary: this is a paper about histopathology image artifact restoration. Unsupervised learning methods are used for artifact localization first. Then novel diffusion based inpainting method is used for restoration. Good (SOTA) results were obtained in the experiments. Authors showed that the proposed model, HARP, also improves robustness and reliability of downstream tasks too. Pathologists were not able to tell the difference between clean and restored images.
main_review: main_review: paper has a good rebuttal period, many questions were answered positively, figures were not informative and authors promised to improve them, not sure how to check them before giving final score, 

overall,  since it is not super innovative, I think it is a weak accept, fair conference paper.
strength_weakness: strength_weakness: - inpainting with diffusion is not totally new as it has been already used in Computer Vision applications, perhaps incrementally new in medical imaging.
- colors in the figures are making it hard to understand and depict what is going on.
- too much bold faces make the reading difficult.
- comparisons can be strengthened but I dont see this as a major problem.
questions: questions: - how does this inpainting affect the quantification ? are those region represented in the final quantitative results in clinical value? or they are skipped due to artifacts? we can't rely on those regions, right?
paper_summary: paper_summary: This paper proposes a series of unsupervised methods for detecting and removing artifacts in histopathology images.

Pipeline: Anomaly detection using FastFlow --> artifact localization using the difference from the original vs corrupted-and-then-denoised output, followed by segmentation of the localization heatmap --> image restoration using the artifact image masked with the predicted segmentation.

Results are compared against baselines using image quality metrics and downstream segmentation performance. The restored images are visually compared against clean images by pathologists.
main_review: main_review: Thank you to the authors for the detailed responses.
It seems that some of the responses have been added to the manuscript, but some have been left as responses here in the review comments. Many of the questions arose from incomplete or hard-to-find info from the manuscript, so I believe it could have been better for these comments to be reflected in the manuscript itself.
I still feel a bit reserved about the concatenation of input alone being a noteworthy novelty, but I understand the argument that the combination of all other components can be an important contribution as well. Given the responses here and the amount of changes in the manuscript, I will slightly increase my rating to borderline accept.
strength_weakness: strength_weakness: In Tables 1 & 2, HARP seems to perform worse than baselines for most artifact types. The best performance is achieved by "Ours", which if I understood correctly, uses the ground-truth segmentation and then performs the rest of the restoration. This seems to speak more about the effectiveness of the diffusion model for restoration rather than the unsupervised aspect of HARP.

The novelty seems to be mostly in the application of a collection of existing methods rather than a new method. FastFlow, RePaint, SAM/DBSCAN are all existing methods. The claimed novelty on the "novel diffusion model" is unclear -- there's only one sentence mentioning "incorporating $\tilde{x}$ in each step of the denoising process". This sounds like a simple input channel concatenation.

Some explanations should be improved. Details are in the comments below.
questions: questions: Any responses to the weaknesses and detailed comments would be good for the rebuttal.

Comments: title: Answer to Reviewer DNG7 Comment
Answer to jxzF comments
Answers to jxzF
Answers to yaJT
Answers to DNG7
Answers to ukLv
comment: Thank you for your input. The confidence intervals indeed represent the mean ± standard deviation. We've validated the improvement from inpainting over just excluding artifacts localizations of HARP with a significant two-sided t-test (p=4.89e-9). 

The remaining concerns seem to stem primarily about difference in opinions. We believe we have adequately addressed these within the limitations section of our paper and do not foresee additional revisions in this area.

We appreciate your engagement and are open to further discussion.
I would like to thank the authors for addressing some of my comments and remarks. It is not clear what the added "confidence intervals" are, I assume it is mean +- std? If that is the case, I would argue that there is no substantial difference between processing the slides with the detected artefacts excluded and inpainting the artefacts with the proposed method. 

According to the authors using the inpaintin has the additional benefit of making the viewing more pleasant for pathologists, but I would argue that pathologists would prefer to see the original image, and certainly not an inpainted image with the inpainted regions annotated as suggested. This would be an unnecessary complications, and a potentially dangerous one.

Because of this I will keep my original assessment unchanged.
thanks for the responses
# Comments:
### “It is puzzling how, even though HARP did not seem to perform very well in terms of quantitative metrics, the user study seems to indicate that pathologists could not tell the difference between clean vs restored images. Is there a good explanation for this phenomenon? Are the metrics suboptimal for determining the quality of restoration? Were the average metrics skewed due to a few badly restored images?”
We agree this is a problem of the metrics images that did not detect, or artifacts were only partially inpainted received awful scores, severely impacting the average. This can be seen especially for blood-related artifacts. For the user study, we only selected images that had artifacts detected and therefore were inpainted. However, missing minor parts of the artifacts, e.g. for blood cells, might have gone unnoticed by the clinicians as the augmentation looks quite realistic. This also shows a limitation of the study, clinicians are trained to classify/diagnose patients and ignore the artifacts, which might have caused them to miss such instances. However, some likely showed a minor correlation because of such leftover artifacts.
### “How are the masks generated for training the diffusion model?”
Thank you for the excellent question. It is a combination of diagonal mask, box mask, free-form stroke mask and mask covering the entire image. 10% of the time diagonal mask that covers have the image from random corner. 10% of the time, a box mask that covers a random corner of the image. 10% of the time mask that cover the entire image. 70% of the time, union of box mask at a random spot of the image and free-form stroke mask from. The free-form Image inpainting uses gated convolution. Mask covering the entire image is to make sure the diffusion model still learn the entire image, which is required as we wanted to use the same diffusion model to do reconstruction of the entire image for artifact localization. The entire implementation will be available in a GitHub repository upon acceptance.
### “Labels are missing for Fig. 2-III, the two inputs into the diffusion model. It's hard to decipher from the text and the rest of the figure what those images are supposed to represent (e.g. is one of them supposed to be $\bar{x}:=\bar{y}_T$? Wouldn't that be just random noise?).”
Thank you for this comment; we improved the figure to make this clear. To your question: yes, that would be just random noise, as we have no prior on the location of the artifacts. One could potentially integrate a region proposal here to improve the activation map and therefore the selection of potential artifact localizations, as we anticipated this change for the future we opted to use the more abstract form of writing, but made it now more explicit.
### “The rationale behind using the reconstruction error as "activation map" is unclear. Is this from an empirical observation? Am I understanding correctly that artifacts will have lower error? Why is reconstruction better for artifacts if the reconstruction is performed to inpaint real images? Also, if other studies have used similar approaches, there should be a citation.”
We agree that the reconstruction map is not a real activation map as known from CAM or similar methods. However, the activation map encodes how much the diffusion model could accurately reproduce-- what is known to the model– that's why we see it as an activation map over time in the diffusion process. This is also why artifacts usually have a lower score in reconstruction error, as the model does not know how to alter such content, they are mostly not represented in the training data. For known contents, like cells and stroma etc., the diffusion model builds a more robust representation of them from the training, making the diffusion model to propose just very slight alterations. In order to increase this effect, we aggregate the activation map over 25 passes to improve the consistency. Concept-wise, this is close to anomaly detection methods like FastFlow and Draem; however, they do not use diffusion models and therefore do not have the time to unroll the denoising objective. If there are better citations that are closer to what we are doing, please bring them to our attention.
### “"we ensure transparency in our process by excluding these contents using HARP’s artifact localization" - what are the "contents" being excluded in this sentence?”
We exclude the inpainted areas that we determined with our artifact localization. In order to assess the value of the artifact restoration, we added results only excluding these areas without restoring them, and we see that the artifact restoration has a significant effect on the downstream model performance. We hope this helped to clarify your question.

Thank you very much for your detailed review, we hope we could clear up all of your questions. Feel free to ask additional questions when we left something untouched.
# Weaknesses:
### “In Tables 1 & 2, HARP seems to perform worse than baselines for most artifact types. The best performance is achieved by "Ours", which if I understood correctly, uses the ground-truth segmentation and then performs the rest of the restoration. This seems to speak more about the effectiveness of the diffusion model for restoration rather than the unsupervised aspect of HARP.”

As there exists no fair comparison to HARP, we opted to compare all on fair grounds by giving them the ground truth segmentation masks. In this case, our method -- the Artifact restoration model of HARP --  clearly outperformed all the comparisons resulting in  new SOTA, while being computationally more efficient. In a real world application, no supervised method would get a ground truth segmentation mask to base their inpainting on. As there are no comprehensive datasets that include a high diversity of annotated natural artifacts, we decided to make our model HARP fully unsupervised. The results of this unsupervised approach should be seen as something with no direct comparison as other method like ArtiFusion would require to inpainting the full image, which is clearly a loss of valuable that can be reliably examined without the potential of misclassification based on some generative AI.  As we found the only two reproducible methods that produced somewhat decent results from the histopathology domain, we adapted DDRM to our problem to improve the number of comparisons. We think this field clearly deserves more attention and further research, that’s why we make our code accessible after the acceptance.
### “The novelty seems to be mostly in the application of a collection of existing methods rather than a new method. FastFlow, RePaint, SAM/DBSCAN are all existing methods. The claimed novelty on the "novel diffusion model" is unclear -- there's only one sentence mentioning "incorporating in each step of the denoising process". This sounds like a simple input channel concatenation.”

Thank you for your valuable feedback on our manuscript. I would like to address the concerns raised regarding the novelty. There has **not** been an **unsupervised artifact localization method in histopathology**, yet. Further, **our diffusion model integrates the addition of $\tilde{x}$** to be concatenated to the input, which allows us to **reduce time-consuming procedures like RePaint and gain improved results**. This last novelty alone would be enough improvement in itself so that other people would justify a standalone paper. Further, our method also outperforms all other SOTA supervised methods. However, we do not like publishing minimal publishable novelties. We add **our novel artifact localization** and artifact detection to evaluate the first **truly unsupervised approach for artifact restoration**. That's why it might seem like a lot is going on that has been already done, which is only to show the full potential of your novel approach.
# Weaknesses:
### “inpainting with diffusion is not totally new as it has been already used in Computer Vision applications, perhaps incrementally new in medical imaging.”
Certainly, the use of inpainting with diffusion models is a well-established technique in the field of Computer Vision. The application in medical imaging has it own intricacies and specific requirements to potentially offer new challenges and opportunities for this technology. However, we do propose novelties that have shown specific application to medical images. Therefore, the impact to pathology could be substantial and certainly worthy of exploration and publication.
### “colors in the figures are making it hard to understand and depict what is going on.” and “too much bold faces make the reading difficult.”
Thank you for your feedback, we do like these colors as they are easy to recognize and associate with the different parts of HARP. However, we try to adapt the figures more easily readable, including a high dpi in order to allow for zooming in on close for better reading. Further, we revised the text to contain significantly less bold faces in order to make reading easier. We hope you like the changes.
### “comparisons can be strengthened but I dont see this as a major problem.” and “how does this inpainting affect the quantification ? are those region represented in the final quantitative results in clinical value? or they are skipped due to artifacts? we can't rely on those regions, right?”
As requested by other reviewers, we do include the performance of the downstream task only excluding the artifact localizations without restoring them in order to better assess the value of artifact restoration alone. Further, we added the variance to the downstream task in Table 3. These results show that the artifact restoration improves the downstream model even though all correspondingly inpainted pixel are excluded in the evaluation.

Generative AI should never go unlabeled – as the EU AI Act suggests. Especially in pathology, Generative AI poses the risks of misleading the diagnosis done by pathologists and other AI systems, which we showed with our user study. That’s why we exclude the artifact localization masks from the downstream evaluation and would recommend highlighting them with a transparent overlay when presented to the pathologist in order to make sure they see which parts have been altered by an AI. In case of doubt, a pathologist should switch to the original image taken by the scanner or even look at the raw sample. Nonetheless, computational pathology has the promise of saving time and increasing diagnostic accuracy for patients, for which HARP could be a supportive structure to ensure the reliability of such an AI system. Further, it has the dual benefit of improving the image quality for the pathologist viewing an image without the need to rescan/remount the sample. We included a shorter version of this paragraph as discussion of this problem.  However, we encourage future research into this topic in order to improve the reliability of restoration, more efficient and broadly applicable tools in order to achieve a trustworthy AI that revolutionizes the daily clinical routine and that’s why we make our code available upon acceptance.

Thank you very much for your review, we are very delighted about your feedback and interesting questions.
# Weaknesses
### “The pipeline is a combination of several existing methods, without much methodological novelty.”
Thank you for your valuable feedback on our manuscript. I would like to address the concerns raised regarding the novelty. There has not been an unsupervised artifact localization method in histopathology, especially  one that leverages diffusion models with pre-trained knowledge of a foundation model. Further, our diffusion model integrates the addition of x tilde to be concatenated to the input, which allows us to reduce time-consuming procedures like RePaint and gain improved results. Our diffusion model outperforms all SOTA supervised methods. Further, we add our novel artifact localization and artifact detection to evaluate the first truly unsupervised approach for artifact restoration. 
### “My main concern is the artefact restoration process, which in my opinion is unnecessary and constitutes solving a more difficult task (restoring the image/eliminating the artefacts) than needed. As histopathology images are vast, an area with an artefact once detected can simply be ignored/not processed in downstream processing.”
While we agree that artifact restoration is a more difficult task to do, but it is necessary to improve the overall viewing experience of the pathologist.  The alternative of rescanning/remounting the sample is too time-consuming to be a viable option for clinical pathologists under time pressure. We also added results for the downstream task when we do not inpaint the artifact localization and just exclude it from evaluation, and the restoration has a clear positive impact on the downstream model, further establishing the benefits.
# Comments
### “Confidence intervals should be included in Table 3, otherwise it is not possible to assess the performance of the methods relative to each other. A true baseline comparison would be a method that ignores artefact areas in a downstream tasks.”
Thank you, that's an excellent idea, we added variances and the results by just ignoring the artifact localization of the artifact image in table 3.
### “While I appreciated the informal tone of the caption of Figure 4, perhaps it is not very suitable for a scientific papers and the authors should reconsider.”
Thanks, we removed the “encouragement to try it out yourself”. We will include the full study in the GitHub repository up on acceptance.
# Questions
### “The dangers and limitations of the "restoration" component of this pipeline should be discussed.” and also the weakness: “The process of restoration is dangerous as it can introduce non-existent information or remove important information (in the case of of a false positive).”
Generative AI should never go unlabeled – as the EU AI Act suggests. Especially in pathology, Generative AI poses the risks of misleading the diagnosis done by pathologists and other AI systems, which we showed with our user study. That’s why we exclude the artifact localization masks from the downstream evaluation and would recommend highlighting them with a transparent overlay when presented to the pathologist in order to make sure they see which parts have been altered by an AI. In case of doubt, a pathologist should switch to the original image taken by the scanner or even look at the raw sample. Nonetheless, computational pathology has the promise of saving time and increasing diagnostic accuracy for patients, for which HARP could be a supportive structure to ensure the reliability of such an AI system. Further, it has the dual benefit of improving the image quality for the pathologist viewing an image without the need to rescan/remount the sample. We included a shorter version of this paragraph as discussion of this problem.  However, we encourage future research into this topic in order to improve the reliability of restoration, more efficient and broadly applicable tools in order to achieve a trustworthy AI that revolutionizes the daily clinical routine and that’s why we make our code available upon acceptance.
### “A better baseline should be presented that compares the methods to simply ignoring detected areas with artefacts.”
We added the results in Table 3 of only excluding the artifact localization from evaluation and not applying the artifact restoration. This shows that the artifact restoration still comes with an added benefit for the downstream model. Further, it has the dual benefit of improving the image quality for the pathologist viewing an image without the need to rescan/remount the sample. However, this restoration should be made transparent to the pathologists, as discussed before.

Thank you very much for your review, we are very delighted about your feedback and interesting questions.
# Weakness 1 and 1. Major concern, artifact detection
We agree that not all methods from natural images translate well to medical images. However, we trained 10 anomaly detection methods only on pathology images and evaluated them in Table 4. We further evaluate the three best extensively in Figure 3. We think the results show that these methods can definitely work, and we do not find any anomaly detection method that was specifically developed for histopathology that includes techniques that are not already present in more generalizable methods evaluated by us. Nonetheless, we believe a more tailored approach for pathology could definitely improve upon them in the future. Regarding approaches like HistoQC, they often require a massive amount of manual labor to select the features that can detect the artifacts. Further, approaches that train with supervision on the artifacts will fall short for unobserved artifacts like the AR-cycleGAN in the restoration case, which only had information on folds and overlaps, which we outperform severely.
# 2. Major concern, artifact detection
We are thankful for your concern about the generation of artifacts into images without any artifacts. Our short answer is: No, they are usually not even detected to contain an artifact. HARP actually has four layers of security built into it to prevent such situations. First, we calibrate so that only 5 % of images from the validation set would get into the pipeline, in that 5 % are likely also images that contain natural artifacts. Hence, it is unlikely that an image could be inpainted without actually having an artifact. Second, we did not observe a case where an image got an artifact inpainted was selected as the final result, as our approach is conservative about selecting which inpainted image would be selected – hence the 5 artifact localization proposals. Thirdly, even when there would be such an image, it is very unlikely that an artifact would be too introduced to that image as the diffusion model was trained to restore artifact free images. Even when our diffusion model would inpaint an artifact, a newly introduced artifact would be ranked much lower by the final anomaly detection than others that reproduced tissue close to the original image -- we would only select the image considered to be the most natural. Lastly, we would only display inpainted areas to pathologists with an indication that the region was generated by an AI, as required by laws like the EU AI Act. Having built in so many layers of security, we are very confident that it would not introduce new artifacts into images that had none to begin with. However, when there are already artifacts we could observe images that were inpainted with additional artifacts in some of the 5 localization maps proposals, but those would be ranked much lower and not selected as the final image. For these reasons, we think it would not affect the downstream task, which shows in our downstream experiment.
# Weakness 2, 1. + 2. Major concerns, artifact localization
We use the same diffusion model for inpainting, to reconstruct complete images. If an image contains an artifact, the part with the artifact is expected to have poor reconstruction, as it deviates from the learned pattern. Further, we observed this for every instance of diffusion model we trained in the process of writing this paper, so we are fairly confident that it works as long as the artifacts are not significantly represented in the training dataset.

Now, the BCSS dataset we trained on is a very clean dataset, even though it has some artifacts – mostly blood related artifacts — but they are minimal, so the model is less likely to know how to reproduce this. We only notice a slight alleviation for the blood cells in the activation map, which likely also causes them to be among the hardest artifacts to restore. However, this could be improved in two ways:
- Improved manual cleaning of the dataset.
- Active unlearning of artifact features.

To the best of our knowledge, both methods would introduce some element of supervision, which we want to avoid as annotation costs are high or would make our model not an unsupervised approach, which we see as important to generalize to unseen artifacts.
# Weakness 2 and 1. Minor concern
We prompt SAM with 32x32 grid points on the entire image. We will make the entire code accessible on GitHub upon acceptance in order to ensure reproducibility. Disjointed artifacts will sometimes be segmented separately, as it is the case for scattered blood cells. For cases like this, the image could go through the pipeline multiple times, until the final artifact detection is below the anomaly threshold.
# 2. Minor concern:
Thank you for noticing, and we updated the PDF accordingly. It is the internal anomaly score of each method that gets bound to [0...1] by Anomalib, where 1 is a perfect anomaly and 0 is perfectly represented as in distribution.

Thank you very much for your review.