Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a transformer based model for the classification of whole slide histopathology images. The method first extract ResNet features locally across WSIs, then apply a transformer based model on top of the features. The authors also propose a Pyramid Position Encoding Generator (PPEG) module for transforming local features and encode positional information. The proposed framework is general and can be applied to other non-medical image analysis tasks.

The proposed method achieves better result compared to recent baselines.
main_review: The theorems give some analysis on why multi-head self-attention layers may be good for this application. However, conclusion of "smaller information entropy" is not exciting. To some extent, it is almost straightforward to think that adding cross-patch correlation is beneficial. The paper is somewhat hard to follow. The authors introduced terms like TPT and PPEG before these terms were explained.

I have a couple of concerns on experiments:
1. The authors claim that the multi-head self-attention layer models the correlation across patches. To me, the convolutional layers in the PPEG module also mix features across patches, and thus, uses spatial patterns (correlation) across patches. It is not clear to me if simply stacking multiple PPEG modules would perform similarly, compared to the proposed approach. It would be great if authors can show the effectiveness of the attention layers via ablation studies.
2. The authors compared their proposed PPEG against commonly used positional encoding methods such as sin-cos. It is not clear to me if the comparison is fair. First, without any non-linearity layer between self-attention layers, the self-attention layers tend to converge to a trivial state of all tokens having the same feature vector. Thus, if the authors use sin-cos as the baseline method, the authors should explain if they still put some layers with non-linearity in the place of PPEG. Second, PPEG introduces computation and increases model capacity, where as sin-cos barely does. It is not clear if the performance gain is due to increased model capacity, or due to the intended mechanism.

Minor problems:
1. What does "TPT" stand for?
2. "discraded" should be "discarded" in the caption of Figure 3.
limitations_and_societal_impact: The authors can show failure cases and discuss possible improvements.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 	The authors propose a Transformer-based MIL method for WSI classification.  
The method is designed to avoid the IID assumption inherent to standard MIL inference methods (e.g. mean, max pooling). Secondly, they propose a positional encoding module for patches in WSIs based on CNNs to incorporate spatial information into the prediction, along with semantic information learned in the MSA layers. In the ablation study, they demonstrated that PPEG model performed better than conventional sin-cos positional encoding. They used three WSI classification datasets from open sources with previously benchmarked results. The authors were able to achieve SOTA results on each task. 

main_review: Originality: original work

Quality: very well done, figures, tables, algorithms look excellent

Clarity: The paper is dense and the authors attempted to cover a lot of ground. The writing is clear for the space alotted. 

Significance: Paper is significant for moving WSI classification forward by applying Tranformers to MIL.  

Strengths:
This paper is the natural extension of previous RNN-based and attention-based MIL methods for WSI classification in computational pathology. The application of Transformers to patch-based WSI classification is a natural one given the recent success of Transformers for NLP (patches-as-token). 
The authors selected three representative WSI datasets for model testing.
Figure 2 is excellent and provides a clear illustration of the difference between the MIL inference methods using an explicit pooling matrix. 
The authors are commended for stating explicitly, and addressing, that an IID assumption for patch-based WSI classification is incorrect and should be avoided in the development of models moving forward.   

limitations_and_societal_impact: Weakness:
One major problem, or point of misunderstanding, is how the ‘Spatial Restore’ step in Algorithm 3 PPEG processing flow is related to the actual spatial relationships between patch locations in the WSI. The input order of patches into the MSA module is unclear from the paper, (random, sliding window, etc. ?), but there is no ‘natural’ square in the WSI. Yet in processing the patch tokens (H_f), an artificial 2-d square feature map is generated. With the patch tokens now populating the ‘pixels’ in the feature maps, adjacent patch tokens/pixels are not necessarily locally connected and ought not to belong in the same convolutional receptive field. I assume this is what the authors mean by ‘context information', however, it is unclear how the PPEG module provides the appropriate patch-level spatial context within the WSI. Different sized convolutions do not address this problem. 

If the above criticism is correct, does the model perform differently when patch tokens are randomly shuffled versus structured sliding-window patching (semi-ordered)? If they do not, then PPEG is learning to perform some other function than incorporating spatial/context information. 

Adding attention maps from CLAM versus ABMIL versus TransMIL to Figure 6 would be constructive. As proposed, it is unclear what is actually driving the boost in performance: better spatial/bypass attention or correlative/self-attention.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a transformer based approach for WSI classification, correlation among instances are considered to address the iid assumed by previous methods. The methods were tested on three public datasets, i.e. Camelyon16, TCGA-NSCLC and TCGA-RCC datasets, and the results suggest that the performance for binary and multiple category classification is better than state of the art. 
main_review: 
The idea of using transformr to process those instances for WSI classification seems to be an interesing idea and a good application for transformer. The authors shall include the previous work "Patch Transformer for Multi-tagging Whole Slide Histopathology Images" and discuss the main differences.  The experiments shall also include this work for comparison, as this is the closest work.

For WSI images, while image-level classfication is useful, it is also important to mark those cancer cells such that the classification can be justified. For a WSI (bag) classified as carconoma, losts of instances (patches)  included in the bag might be normal cells. How do these cells interfere with the classification results?  Can the proposed approach (transformer, correlated MIL) mitigate such issues? More detailed analysis on this problem are expected.

Minor problems:
1.	There are some spelling errors through the manuscript, e.g. the title “Classication” should be “Classification”. Please check the manuscript carefully.
2.	For the related work, the author only listed some literatures, but without giving a discussion about the strength and weakness of the existing works.
3.	In Table 1 we can see that, besides the CAMELYON16 dataset, the baseline MIL-based methods showed much lower performances than max-pooling. Please give some discussions about the reason.
4.	For ablation study, The Table 2 and Fig. 5 were not mentioned in the manuscript. What do the values stand for in Table 2 and Fig. 5? Why giving the detailed discussion in Appendix? I suggest move the discussion to the main manuscript.
5.	For Fig.6, What is the purpose to show the zoom-in view of heatmap? I cannot see anything special in this area.
6.	For Fig. 7, the initial accuracy of MIL-based baseline model were higher than the converged models, especially for the NSCLC dataset, why?

limitations_and_societal_impact: n/a
review_title: null

Meta-review: The paper has an interesting application (if rather straightforward) of transformers on Multiple Instance Learning for classification of Whole slide images. The authors successfully responded to the reviewer criticisms. All reviewers recommend acceptance. 

Authors are encouraged to include the additional experiments and explanations to the final version
Decision: Accept (Poster)