Venue: MIDL 2023 Conference

Reviews:

final_score: rating: 5: Strong accept
review_content: summary: This paper performs a thourough benchmark of different methods for image-to-image translation, in the context of histopathology staining (back and forth between MT and H&E (full disclaimer: I have no idea what those are)).

The authors do a (I think) extensive coverage of the litterature over time (spanning 20 years) and implement 12 methods in a common framework (to be released publicly). The evaluation is made on a dedicated datasets, and completed with an evaluation of performances in a typical use-case, plus another evaluation by clinicians (to spot the "fake" images).
strengths: - Extensive coverage of the existing methods, including non-deep-learning/GAN based methods
- Dedicated and consistent dataset collected for the purpose of the study
- Thorough evaluation on multiple metrics
weaknesses: The experiments could benefit from multiple runs, and maybe some pairwise statistical test, to see if the difference between methods are statistically significant.

--- 
Not a weakness of the paper per se (it is a well executed benchmark), but with the overall field of image-to-image translation, outside of artistic applications: I do not see the point. See my detailed comment in the "questions to address in rebuttal". This will _not_ affect my final rating, but I genuinely wish to engage in this discussion, as so far I have always been quietly dubious about such applications.
detailed_comments: Misc:
- Tables could benefit from standard deviation information
- Maybe indicating if the difference between CycleGAN/CUT, CUT/MUNIT, MUNIT/StainGAN (and so on in Table 1) are statistically significant, would be good.
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper provides a comprehensive evaluation of 12 different stain transfer methods for pixel-to-pixel translation in histopathology. The evaluation is based on a large dataset of over 1000 images. Both quantitative and pathologist assessments are included in the paper for a comprehensive comparison.
strengths: 1. The paper is very well-written. The structure is clear and easy to follow.
2. Not only quantitative evaluation on stain transfer is performed, but also the effect of stain transfer on downstream classification tasks as well as pathologist assessment are presented. This provides a comprehensive evaluation on the 12 stain transfer methods.
weaknesses: Stain transfer is an important preprocessing step in computational histopathology that helps to address the domain gap. It is crucial to assess its impact on subsequent tasks. This study aims to compare different stain transfer algorithms, therefore, it would be beneficial to include additional comparisons with downstream tasks such as histopathology image diagnosis. Furthermore, as shown in Table 2, many stain transfer algorithms result in a decline in performance. It is suggested to include an analysis of why these algorithms are not effective. In other words, the factors that contribute to the success of StainGAN and UNIT should be discussed.
detailed_comments: Please refer to the comments in the Weakness section.
recommendation: Poster
review_title: null

final_score: rating: 3: Borderline
review_content: summary: The paper categorizes histopathology image2image translation applications into three families: 1) image normalization/augmentation, 2) stain transfer, and 3) virtual stain transfer. After claiming that benchmarking has only been done for 1), they proceed to assess and compare lots of image2image techniquers in this context, for a particular dataset and two stains.
strengths: * The list of I2I techniques that have been implemented and assessed is impressively long.
* Two pathologists were involved in human evaluation
* A general, very popular, and well-accepted technique (cycle-gan) perform better than specialized approaches, which I find consistent with a stream of literature showing that many resarch advances are brittle and probably overfitted to certain datasets.
* The paper may have practical interest for practitioners in the area
weaknesses: The main problem I find is in the motivation to carry out this analysis. There is a distinction into three families of problems, but I am not sure I agree what the way that distinction has been laid out. Note that the entire paper is based on that taxonomy, since the authors state that no one has benchmarked two of the subcategories, and they are doing so. Please see below for details about this point.
detailed_comments: This benchmarking paper is based upon a division of I2I applications in histopathology into three families, as explained in the intro. According to the authors, we have:

1) Image Normalization & Image Augmentation: normalization is used in inference time, augmentation during training. Note that for illustrating normalization, they cite the paper (BenTaieb and Hamarneh, 2018), with the title of "Adversarial stain transfer for histopathology image analysis".
2) Stain Transfer, which they describe as methods to re-stain a slide so that it enables quantitative analysis of different structures and tissues, without having to re-acquire.
3) Virtual staining, which is the same as 2) but starting from unstained samples.

If I understand correctly, as opposed to 1), when we do 2) or 3) our goal is not to generate new training data, or adapt test samples at inference time, but rather enable pathologists to read the image in different stains, is this right? 

If my understanding is correct, then:
* Shouldn't we have an evaluation in which we try to understand if re-stained (translated) images actually allow to distinguish structures that were undistinguishable with the "old staining"? Running some quantitative analysis like as a histopathologist would do, instead of looking at SSIM, WD, and FID? 
* Why does the paper have a section on computer-aided diagnosis? In section 4 the authors adapt in test time test samples and improve the diagnostic, but doesn't this fit in the Image Normalization category inside 1) above?

----

I have some other minor commments, please see below:

- I think it would be a good idea for the sake of readability to add in between parenthesis and maybe with a smaller fontsize, after each number in Table 1 the corresponding rank 1st-12th.
- I find section 4 to be a bit poorly written; what is the system here, maybe give a short description? Can we really average all small tiles' inflammation scores into a single slide-level score in the [0,2] range, and this is meaningful? I mean, don't the score 0 tiles overwhelm the other ones in the averaging? Also, is the MSE computed at the tile or at the slide level? Finally, why couldn't Pix2Pix and UTOM handle 300x300 image sizes? it does not sound like a lot to deal with, right?
-"first Wasserstein Distance" -- remove "first"?
- The reference to (Mirza&Osindero 2014) in the intro to illustrate "recent success" does not feel too recent, it's been nine years already, lots of things have happened...
recommendation: Poster
review_title: null

final_score: rating: 3: Borderline
review_content: summary: This paper compares three traditional and nine Generative Adversarial Network-based H&E to MT stain transfer approaches whose quality performance has been quantitatively assessed. First, the authors evaluate the output of the models with Structural Similarity Index (SSIM), Fréchet Inception Distance (FID), and first Wasserstein Distance (WD) by comparing it with its corresponding reference image for the whole dataset. Secondly, the reference MT stained tissue images and synthetically-produced tissue images are fed into a stain-level classifier, and Mean Squared Error (MSE) is calculated by using the outputs of the classifier. Finally, a reader's study is performed on the outputs of CycleGAN and MUNIT to investigate whether the synthetic images are indistinguishable from the authentic ones.
strengths: 1. The paper is generally well-written
2. Code to be shared with the community 
3. Evaluated on a downstream stain tissue classification task
4. A reader's study has been performed to validate that the produced images by the CycleGAN model are as satisfactory as the authentic images
weaknesses: 1. Only GAN-based approaches have been compared. 
2. Assumptions about some of the experiments have not been stated. 
3. Some additional metrics are required to analyze the classification performance.

detailed_comments: 1. Can you explicitly state the contributions of your research paper? 
2. It is not surprising that CycleGAN and Pix2Pix tend to perform better than the other methods in terms of image quality, and there are existing research papers already demonstrating that (de Haan et. al., 2021) and (Salehi et. al., 2021). It would be more interesting to see whether it is possible to improve these results with recent generative models for I2I, such as Latent Diffusion Models (Rombach et. al., 2021). 
3. Sufficient selection of metrics in terms of measuring quality, but cannot say the same thing for the classification tests.
4. I couldn’t understand the trick you did on Pix2Pix and how using that trick helped you to overcome the aligned image prerequisite. Can you elaborate on that a bit further?
5. Are you planning to experiment on other stain types, such as periodic acid-Schiff and Jones silver stain? 
6. Do you have the ground-truth inflammation scores for the 1300 image tiles used for testing the artificially stained tissues? If so, can you provide the changes in the classification accuracy/precision/recall/$F_1$-score? If not, I assume that you calculate the MSE of the model outputs between authentic and synthetic stained tissues, and I would like to ask how you can be sure that Heinemann et al.'s system is working effectively for your testing split. 
7. Can you mention the years of expertise of the pathologists in Section 5?
8. Are you also planning to open-source the dataset alongside the source code?
9. What is the reason for you to select MUNIT and CycleGAN for the pathologist assessment? 

- Kevin de Haan, Yijie Zhang, Jonathan E. Zuckerman, Tairan Liu, Anthony E. Sisk, Miguel F. P. Diaz, Kuang-Yu Jen, Alexander Nobori, Sofia Liou, Sarah Zhang, Rana Riahi, Yair Rivenson, W. Dean Wallace, Aydogan Ozcan. “Deep learning-based transformation of H&E stained tissues into special stains”. Nature Communications 12 (2021).
- Pegah Salehi, Abdolah Chalechale. “Pix2Pix-based Stain-to-Stain Translation: A Solution for Robust Stain Normalization in Histopathology Images Analysis”. MVIP 2020. 
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer . “High-Resolution Image Synthesis with Latent Diffusion Models”. CVPR 2022.

**Reason for the change in rating**: Improvement in writing, making satisfactory explanations and the share of code. However, I am still feeling unconvinced by how the use of only GAN-based generative models makes this study interesting, considering that the development of generative models is an active research area.
recommendation: Poster
review_title: null

Meta-review: metareview: The authors propose a comprehensive evaluation of 12 different stain transfer methods for pixel-to-pixel translation in histopathology. The authors well-performed this comparison. I think that such comparative study is important for the field.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)