Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 4: Weak accept
review_content: summary: In this paper, the authors tackle three problems in medical image classification: limited dataset, lack of region-based supervision and domain shift. For this purpose, the data is augmented using previously proposed approach Snap-Mix, a new patch embedding technique is employed that considers the two patches each with the highest and lowest CAM activations and a voting method is investigated that constructs multiple masked images based on the CAM score. The approach is evaluated for the classification of COVID-19 in chest CT images using three datasets.
strengths: - The authors provide through validation of the proposed method using three large covid-19 chest CT datasets. The method is compared with previously proposed approaches and the parameter sensitivity is investigated in details. The results show that the proposed approach results in improved performance for all three datasets.
- The paper is easy to follow and the ideas are supported by visualizations.
- Reproducibility aspect: Implementation is made available in public.
weaknesses: - Paper lacks clarity on several important aspects: distribution of dataset into train/test/validation, how information from 2D slices is merged to finally obtain patient-level image classification etc.
- The formatting of the paper needs some improvement (For example, repositioning figures and tables, avoiding casual writing etc. For more explanation, please see in the section below).
detailed_comments: - How does theta value of 0.2 in the test time augmentation relate to a pseudo-probability of 0.5?
- For the CAM based test time augmentation, the highest CAM activated regions are masked out considering that the activations come from artefacts. But otherwise, the most significant part of the image where the network decision for the classification comes from is being ignored. Probably the motivation needs some explanation.
- Section 3.3 / Conflicting information: First it is mentioned that image is classified k+1 times by masking one out of k different patches. Later, it is explained that "in the second, we mask out the two patches with the highest CAM scores; and so on".
- Avoid casual sentences: "in which performance levels are in the high 70’s percentage", "we reduce the error rate by one and a half or more, depending on the measurement error" (what is meant by "depending on the measurement error"?)

Minor comments:
- Maintain uniformity: state-of-the-art vs state of the art, Site-A vs SITE-A, Snap-Mix vs Snap-Mix
- Change "which have been shown" to which have shown
- Not clear, maybe needs rephrasing: "The first two datasets are trained together on the literature"
- Dataset A: It is mentioned that "The 1,230 negative samples are affected by other lung diseases". Are all 1,230 negative samples affected by other lung diseases? Otherwise, the sentence needs to be rephrased.
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: Authors present their method for binary image classification, in which they evaluate the addition of three 'explainability-based' components: data augmentation via SnapMix, a loss function based on contrastive patch embedding, and test time augmentation. All three components are based on the CAM method (class activation mapping). The results show that adding the three components does improve classification of covid in CT images, when compared to both results reported in literature and baseline implementations.
strengths: The paper is clear and, personally, I think the CAM based additions to the image classifier are of interest. Authors included a clear description of the methods; and exhaustive results from both literature and their own baseline experiments. They included quite a large amount of data for their experiments.
weaknesses: Personally, I think COVID classification on medical images is not a useful task. Detection of COVID is easily done with nasal swabs (pcr or self-test), which is also much cheaper than acquiring medical images. There is no added value in COVID detection on imaging. I think authors should include a more clear rationale for doing this specific task: do they actually want to improve covid detection or is this 'just' a dataset to demonstrate their method on?

Another weakness is the use of 2D image slices, instead of using actual 3D volumes. I do not understand why authors do not use 3D imaging, since they use CT images? CT imaging is by nature 3D, but authors reduce this to a 2D image classification task. However, it is not clear why and how the 2D slices were selected from the 3D volumes? Why is the method not applied in 3D? How are the multiple 2D classifications turned into a final classification on the patient level? 

It is also not clear how the dataset was split into training / validation / test sets. Especially information about the validation set is missing. It seems that some of the experiments and parameter-tuning (e.g. in Figure 3) are performed on the test set, in stead of the validation set. This should be clarified in the manuscript.
detailed_comments: I think the title of the manuscript should be more descriptive.

I would suggest to include exact numbers / results in the Abstract. Something like 'a gap twice as large' is not very exact. Also, I think a little bit more information on the methods can be included in the Abstract.

In the last paragraph of the Introduction, I would suggest to include the exact numbers. Not vague quantifications like 'in the high 70's percentage' or 'reduce ... by one and a half or more'. 

In the Experiments, please include a clear description on the 2D / 3D nature of the data and how slice-selection choices were made.

Please make a separate Results section, in stead of including the results in the Experiments section. Also consider a separate Discussion section, so that the Results section remains clear and factual.
recommendation: Oral
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper adapts the SnapMix data augmentation method to improve classification of COVID-19 CT image slices. The method appears highly successful at improving over the presented baselines using several open datasets. Methods are presented in sufficient detail, but the dataset and validation approach are less clearly articulated.
strengths: * The primary contribution of this method is application of the SnapMix data augmentation approach with a contrastive loss function and a voting fusion procedure. 
* Extensive baseline procedures are examined and substantial improvements are seen. 
* The method is highly competitive with the state of the art. 
weaknesses: * There is limited methodological novelty. 
* A statistical assessment of paired differences was not performed. 
* No discussion of the 3-D nature of CT images was present in the data section. 
* For methodological development, the paper falls short as it focuses on a single task. Hence, I have categorized this paper under validation / application. 
recommendation: Poster
review_title: null

Meta-review: metareview: Three reviewers have reviewed the work. While not all reviewers are convinced about the clinical value of COVID detection in CT scans, they recognize that the work has methodological contributions, and the strength of the quantitative results is clear. The use of 2D slices instead of 3D CT volumes is mentioned as a limitation. The work is recommended for presentation as a poster.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)