Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning
Authors: Saarthak Kapse
Srijan Das
Jingwei Zhang
Rajarsi R. Gupta
Joel Saltz
Dimitris Samaras
Prateek Prasanna
Venue: Submitted to ICLR 2023 (2023)
Keywords: Computational pathology
Cell segmentation
Self supervised learning
Vision Transformer
Sparse attention

Abstract: In this work, we develop Di-SSL, a Diversity-inducing Self-Supervised Learning technique for histopathology image analysis. SSL techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective rep- resentations without any human supervision. Lately, computational pathology has also benefited from the resounding success of SSL. In this work, we develop a novel domain-aware pretext task to enhance representation learning in digital pathology. Our analysis of vanilla SSL-pretrained models’ attention distribution reveals an insightful observation: sparsity in attention, i.e, models tends to localize most of their attention to some prominent patterns in the image. Although atten- tion sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal in digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially intermixed biological com- ponents. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we first leverage cell segmenta- tion to densely extract multiple histopathology-specific representations. We then propose a dense pretext task for SSL, designed to match the multiple correspond- ing representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversi- fication in attention for capturing context rich representations. Through quantita- tive and qualitative analysis on multiple slide-level tasks across cancer types, and patch-level classification tasks, we demonstrate the efficacy of our method and observe that the attention is more globally distributed. Specifically, we obtain a relative improvement in accuracy of up to 6.9% in slide-level and 2% in patch level classification tasks (corresponding AUC improvement up to 7.9% and 0.7%, respectively) over a baseline SSL model.

Under review as a conference paper at ICLR 2023
ATTENTION DE-SPARSIFICATION MATTERS: INDUC- ING DIVERSITY IN DIGITAL PATHOLOGY REPRESEN- TATION LEARNING
Anonymous authors Paper under double-bling blind review
ABSTRACT
In this work. develop DiRL, Diversity-inducinn Representation Learning technique for histopathology image analysis. SSL techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective rep- resentations without any human supervision. Lately, computational pathology has also benefited from the resounding success of SSL. In this work, develop novel prior-guided pre-training strategy based on SSL to enhance representation learning in digital pathology. Our analysis of vanilla SSL-pretrained models' at- tention distribution reveals an insightful observation: sparsity in attention, i.c, models tends to localize most of their attention some prominent patterns in the image. Although attention sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially in- termixed biological components. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we first leverage cell segmentation to densely extract multiple histopathology-specieee resentations. We then propose a prior-guided dense pretext task for SSL, designed match the multiple corresponding representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversification in attention for capturing context rich rep- resentations. Through quantitative and qualitative analysis on multiple leide-level tasks across cancer types, and patch-level classification tasks, we demonstrate the efficacy of our method and observe that the attention is more globally distributed. Specifically, we obtain a relative improvement in accuracy of up to 6.9% slide- level and 2% in patch level classification tasks (corresponding AUC improvement up to 7.9% and 0.7%, respectively) over a bascline SSL model.
rep-
INTRODUCTION
Computational pathology is a rapidly emerging field that aims at analyzing high resolution images of biopsied resected tissue samples. Advancements in computer vision and deep learning has enabled learning of the rich phenotypic information from whole slides images (WSIs) to understand mechanisms contributing to disease progression and patient outcomes. Acquiring crop-level local- ized annotations for WSIs is expensive and often not feasible; only slide-level pathologist labels are usually available. In such scenario, weak supervision a commonly utilized strategy, where crops are embedded into representations in the first stage, followed by considering these WSI-crops' repre- sentation as a bag for multiple instance learning (MIL). Now the question remains, how do we learn model to effectively encode the crops into rich representations? Traditionally, ImageNet (Krizhevsky ct al. 2017) pre- trained neural networks are utilized to extract the representations (Lu et al. |02200022 Ib; Lerousseau et al. 2021 Shao et al.) 2021). However ImageNet and pathology datasets are composed of different semantics; while the former contains object-centric natural images, the later consists of images with spatially distributed biological components such cells, glands, stroma, etc. There- fore, to learn domain-specin features of WSI-crops in the absence of localized annotations, various
self-supervised learning (SSL.) techniques recently gaining traction (Ciga et al. 2022 Stacke

---

Under review as a conference paper at ICLR 2023
View 1
View 1
Natural Image
Pathology Image
View
Vanilla SSL [Sperse)
View
Vanilla-SSL (Sparse)
Ours [Diversifled
(a)
[b]
Figure 1: Diversification of attention for encoding dense information in digital pathology. View and View are two augmented views of the input image. a) Illustration of attention map from model pre-trained
ImageNet using vanilla SSL. b) Attention map of model pre-trained on histopathology dataset with vanilla SSL, and with our proposed pre-training strategy. In both nat- ural imaging and digital pathology, vanilla SSL pre-training creates sparse attention maps, i.e., attends largely to only some prominent patterns. Although attention sparsification can be beneficial natural image tasks such as object classification, this could be sub-optimal for encoding represen- tations in digital pathology as it leads to loss of important contextual information. Through a more diversified attention mechanism, DiRL encodes dense information critical to non object-centric tasks.
et al. 2021 Boyd et al. ) SSL models pre-trained on histopathology datasets have been shown to be effective downstream classification tasks when compared those trained on ImageNet.
To further analyze the role of SSL in computational pathology, we pre-trained a vision transformer (Dosovitskiy et al., 2020) on various WSI datasets using vanilla SSI. (Caron et al. 202T). ). In-depth analysis of the pre-trained models' attention maps on WSI-crops led us to a striking observation: sparsity in attention maps. The model tends to localize most of its attention to a small fraction of regions, leading to sub-optimal representation learning. To further validate our observation, we visualized the attention maps of self-supervised ImageNet pre-trained model on natural images (see Fig. [. Similar observations led data. We believe that sparsity in attention might potentially benefit the performance in some natural imaging tasks such object classification. This stems from the fact that during SSL, the model is tasked to match the two views, optimizing which leads the model to focus on the prominent patterns. For example, in Fig. [Ta. for an object-centric ImageNet example, since the prominent pattern is the object (eg. bird) itself (Yun et al., 2022), the model tends to center its attention towards the object, thus benefiting numerous downstream applications (for eg., bird classification). In contrast, WSI-crops are not object-centric, rather they constitute spatial distribution of complex structures such as cells, glands, their clusters and organizations, etc, see Fig. [Tb).. Encoding this dense information available into a holistic representation demands the model to focus more diversely to various histopathology primitives and not just to specific ones. Conversely, the vanilla SSL model pre-trained on histopathology only sparsely attends to the important regions (Fig. [[b)), i.e.. there is inadequate diversity in attention. We hypothesize that this sparsely attending model could result in encoding sub-optimal representations, as
conclude that this is property of SSL rather than of
context-rich details often ignored.
To address this issue of inadequate attention diversity, we propose DiRL, a invverity-induccin pre- training technique, tailored enhance representation learning in digital pathology. Each WSI-crop consists of two regions: cellular regions (one containing cells) and non-cellular regions (containing no cells). We leverage an off-the-shelf cell segmentation pipeline identify these regions. This domain-specific knowledge is then utilized to extract region-level representations separately for the cellular and non-cellular regions. We further propose encode the inter- and intra-spatial in- terplay of two regions. This biologically-insee inspired step (Saltz et al., 2018 Fassler et al. 2022) is achieved through a transformer- based disentangle block to encode the self-interaction within the re- gions, and cross-interaction between both the regions, termed as disentangled representations. In contrast to vanilla SSL frameworks that leverage one image-level representation for WSI-crop, our prior-guided representation learning framework leverages histology-speciff domain knowledge to densely extract a set of region-level and disentangled representations. We then task our framework to match all the corresponding representations between the views. We hypothesize that optimizing this dense matching objective between the views would encourage the model to diversify its attention to various regions; matching assorted representations would then enforce the model to explore diverse
2

---

Under review as a conference paper at ICLR 2023
image-regions relevant for each such representations. We validate this hypothesis through consis- tent improvements in performance on multiple downstream tasks such as slide-level and patch-level classifications. Our qualitative analysis attention distribution of the pre-trained models reveals that our DiRL. framework can effectively de-sparsify attention, thereby learning global context-rich representations, unlike existing methods.
To summarize our main contributions, we:
Demonstrate that attention sparsification in self-supervised learning may lead to learning sub-optimal representations in digital pathology classification tasks.
Design a novel hanced representations for digital pathology.
pretext task to de-sparsify attention maps and achieve en-
Demonstrate the efficacy of our DiRL through slide-level and patch-level classification tasks on four WSI datasets and two patch datasets.
RELATED WORK
In this section, we briefly discuss vision transformers, SSL and its dense counterpart, and their ap- plication in computational pathology.
Vision transformers. Inspired by the success of self-attention modules in language models
(Vawwani al.] 2017). vision transformers (ViTs) Dosovitskiy et al. (2020); Liu et al. (2021): > Tou- vron et al. (2021); Xu et al. 2022); ;
Ali et al. (2021); Tu et al. (2022) have been have been pro-
posed to exploit non-local spatial dependencies in the imaging domain. Recent studies Wang et al.
2202bb Chen Krishnan 2002);; Chen et al. 2022); Stegmüller et al. (2022): Gao et al. (202 )
Chen al. (2021) have demonstrated the promise of transformer-ban based architectures in modeling histopathology imaging for cancer diagnosis and prognosis. However, to the best of our knowledge, no existing work has leveraged the flexibility of attention mechanism in transformers to instill the biology-relevann domain knowledge into vision transformers. For example, interaction between con- cepts/primitives such as tumor nuclei and stroma or between lymphocytic cells plays an important role in disease pathophysiology and treatment outcome. Our proposed method takes step in this direction through domain-driven pretext task.
Image-level SSL aims at learning visual representations through different pretext tasks. Contrastive and non-contrastive methods such as [Chnn et al. (2020); He et al. (2020); Caron et al. (2021). I). have shown tremendous potential in learning robust and rich representation in natural imaging. Building upon them, studies such as Ciga et al. (2022); Stacke et al. (2021): y Li et al. (2021)) Kapse et al.
(022): Boyd et al. (2021): Chen & Krishnan 2022 Kurian et al. (2022) have explored SSL pre-
training in histopathlos logy image analysis.
Region-level SSL aims further boost information encoding through dense pre-training techniques such as [i et al. (2021b); Yun et al.] (2022); |Wang et al.] (2021a). These techniques impose addi- tional constraints to match 1) region-level correspondences across the two views of the data or 2) neighbor-level intra-view correspondences within the data. Studies such as Wen et al. (2022); Yang or clustering based regions in self-supervision to enhance representation learning. However, the goal of these studies to mainly improve the transfer performance for dense-preed predecereiitiion tasks such as object detection and segmentation. In contrast, we tailored a dense pre-training strategy in histopathology to enforce the model to focus on diversc-regions thus diversifying model's ' attention. This diversified atten- tion encourages the model in effectively encoding the complex information about various histology components, thereby augmenting classification performance.
al. (2022): Hénaff al. (202T) have explored utilizing segmentation-
3 PROPOSED METHOD
In this section, we first describe naïve vision transformer framework for Whole Slide Images (WSIs). This is followed by explaining how cell segmentation can be used as a prior in pre-training for WSIs. Next, present the extraction of region-specific representations using proposed cell- back pooling and disentangle block. Finally, we present DiRL, our diversity-induss technique, to lear discriminative features for WSI patches which are subsequently leveraged by multiple instance learning (MIL) framework for downstream classification tasks. An overview of the proposed architecture is shown in Fig. a).
pre-training
3

---

Under review as a conference paper at ICLR 2023
Region- level Representations
Disentangled Representations
Prior black
.1.1..111
Collular region
Not-celular region
Prior block
Prior-block
Prior-block
Average pooting
Cell prior mask
interaction
interaction
M- Cel
00-1
Distarded In inference
Disentangle Block
Back tokens
Transformer Encoder
Cell centroid map
Cell prior
0000-1
Linear Projection of Rlttenndd Patches
WSI-crop
(b)
Figure 2: Overview of the proposed DiRL framework: a) WSI-crop is patchified and fed into a linear projection layer followed by transformer encoder. The output is fed to Prior-block and to a Disentangle block. The Prior-block pools region-level representations separately for cellular and non-cellular regions. The Disentangle block encodes spatial interplay between the two regions followed by prior-block to extract region-level disentangled representations. b) Cell priors Pe and 1-P pool the tokens associated with the cellular and the non-cellular regions, respectively, followed by average pooling to extract region-level features. c) Cell segmentation from WSI-crop followed by extraction of cell centroid map. Cell prior mask is generated by discretizing the cell centroid map into patches. Cell prior vector Pe is then produced from the cell prior mask. After pre-training, modules above the red dashed line discarded in the inference stage.
Preliminaries. For an understanding of the primary components in transformer such as MSA (Multi-head Self-Attention), LN (Layer Normalization), and MLP (Multi-Layer Perceptron), we refer the readers Vaswani 2017).
3.1 VISION TRANSFORMER FOR WSI
From each WSI, W, w1, W2, wn crops are extracted, where is variable for each W. Each wi then decomposed into patches X = [X]_ X² ..... X*] Ruxxxpx3, where (p. p) is the spatial
size of each patch. Each patch is transformed into a token using a shared linear projection layer,
(1) where are convolutional filters operating on each patch with number p X p size filters, thus extracting d dimensional feature vector for patch. This is followed by adding 1D learnable position embedding as in Vaswani et al. (2017). The transformer block models the relationship between the tokens using multi- head self-attention block:
To | X'E, X
Thd T MSA(LN(TT)));; T + MLP((NN(T))
where is index of the tth block transformer encoder, composed of L stacked transformer blocks. Thus in each block, these tokens interact with each other learn representations for each w.. The resulting Tr of dimension (n, d) is average pooled across all the n tokens to compute the image-level representation of dimension (1, as shown Fig. ()).
3.2 CELL SEGMENTATION DOMAIN PRIOR
Each WSI-crop W, consists of two regions, one containing cells and the other without cells. There has been substantial advancements in deep learning research pertaining cell segmentation; this stems from the important role of image analysis and machine learning algorithms in visual inter- pretation of cellular biology (morphology and spatial arrangement) in digitized pathology scans (Lu et al. 202 a Shaban et al. 2022 Ding et al. 2022). Identifying the cellular and non-cellular regions in W: can be achieved by exploiting the cell segmentation output as prior via techniques such as Sa- hasrabudhe et al. p: Hou et al. 2020);; Vahadane & Sethi (2013). Following cell segmentation, the centroids are extracted to yield the cell centroid map. a binary map (C) of values zeros and ones, with Ci 1 if centroid of any cell is present (i,j) pixel in WSI-crop. We term this cell prior
4

---

Under review as a conference paper at ICLR 2023
uxd
axd 1xd.
10:01
Ixd
1xd
X0XC F311 3F
RAM uxd
1xd
(q()
Mai
(qiT)
J1
11:11-1 LILLR
B
ArAnn Korta DN 10
Attention Matrix (1t
Attention Matrix
(2
Monal
Jerms
by average pooling
(b) followed by prior based pooling
Disentangled gleed MSA followed by prior based pooling
Figure 3: a) Illustration of the n attention matrix from the last layer of the transformer encoder, where 9, k; are projections of tokens in transformer block. Matrix multiplication of attention ma- trix with value v, followed by average pooling across all the tokens generates the representation (f) for wi. b) Cell prior Pe is utilized to separately pool cell tokens and back tokens to extract region-level representations. c) Tokens from the L layer are interacted in the transformer-basee disentangle block, forming the attention matrix. Attention masks Mael/ and Meross added with the attention matrix, generating desired matrices for disentanglement. Note that denotes softmax activation. Matrices then multiplied with e followed by prior-based pooling, thus extracting four representations encoding spatial interplay in to For clarity, the cellular and non-cellular region patches (ne and ni respectively) are arranged in separate groups.
mask shown in Fig. c).. Since cell segmentation is routinely used in computational pathology, we use off-the-shelf, well established cell segmentation pipelines instead of training new model. To be coherent with ViT, is decomposed into n patches patch is transformed as follows: C MaxPool(C") i.e., if the patch C contains one more centroids, it becomes one, else it remains zero. Thus, the cell prior mask is downsampled into a binary vector of dimension (n, 1), denoting the presence of cells in each patch of wy. We term this vector as cell prior (P.), which is invoked to extract the region-specifie representations for each wy
[C' C² .....CC) Cm)l of size (p.p). Each
3.3 PRIOR-BLOCK FOR CELL-BACK POOLING
Following L stacks of transformer blocks, set of tokens Tr. E Rixd is fed to prior-block. In this block, the tokens can be categorized into (a) cell tokens, implying the tokens whose input patches contain at least a cell and (b) background or back tokens, whose input patches do not capture any cells. The cell and back tokens are separately encoded to represent region-level features from the cell prior Pa follows:
P. Pc fo= (1 PP)
fo
(3)
E1-Pe
fe is the average pooled representation of all the cell tokens, i.c. representation of the cellular region. fo the average pooled representation of all the back tokens, representing non-cellular regions. In the process, Cell-Back Pooling is exploited in the prior-block to extract two region-level representations as shown in Fig. 3).
3.4 DISENTANGLE BLOCK
We take step towards obtaining region-level representation by proposing transformer block for disentangling the cellular and non-cellular regions. This disentanglement is performed to encode self-interaction in each region and cross-interaction between the two regions. To accomplish this disentangiement, we devise two attention masks, Maelf and Moross, each of dimension (n x n)) as shown in Fig. c).. The goal of Maetf f is to only allow token interaction between the same regions, iee, cell-cell and back-back. In contrast, Moross allows tokens to interact between the different regions (cell-backs The masks Mself and Meross are computed as:
00 Po Poli). (1.o (0.
if if + Pa(i) or i= oc, otherwise
0c, otherwise
where indices i.j E {1, Recall that in transformers, tokens are projected into three embed- dings q, k, and the output of MSA block is computed weighted sum of the values v, where
5

---

Under review as a conference paper at ICLR 2023
the weight assigned to each value is determined by self- self-attention operation softmax(gk") Unlike standard MSA in wi, where all the tokens from cellular and non-cellular regions interact with each other through self-attention (see Fig. B)), we propose to disentangle the interactions between the regions. f and Meross are linearly combined with the attention matrix to obtain disentangled self-attention matrices follows:
MSAcets = softmax(qk" + Moetr); MSAcross = softmax(gk' + Meross) (5)
Note that attention masks Manty and Meross are linearly combined before the softmax activation to ensure that the sum each row the self-attention matrix remains one. The disentangle block operates at the output of the transformer encoder TL as:
Teels = Tr + MSA (T.((T))):
Taals -Toos + MMMP(LN(TTTLL sOoLs)
(6)
T Terose Tr. + MSAcroas (LN():));
Teroas =Toross +MLP(LN(T (Tcoos))
Finally, similar to 3.3 the region-level features are encoded using the cell prior P. using:
Pc) fec = Teels fus = -1-P. Po for = P.
Pe foo = (1 Pa) Teross
(8)
Po
Σ1 Po
Thus, the prior-based pooling Taels and Teross results in four disentangled representations fee: fus. feb. and fae. encoding the spatial interplay between the cellular and non-cellular regions. Thus, for each WSI-crop w2 encode six representations: two region-level representations using cell-back pooling, and four disentangled representations using disentangle block. Our prior griddd pre-training framework operates on these six representations pre-train the model.
3.5 DIVERSITY-NDUUCIS PRE-TRAINING FOR WSI
In this section, we formulate our eveesity-induccin representation learning (DiRL) using a widely used SSL framework for pre-training histopathology data: DINO (Caron et al. 2021). However, in practice, our pre-training technique can be integrated with any pairwise SSL framework (Li et al.) 2022]. as demonstrated Appendix A. DINO consists of student and teacher branch, where the teacher is a momentum updated version of the student, thus both having same architecture (models). Different views of the input image are fed to both the branches to encode them into image-level representations. A projection head is applied on top of these representations with softmax activation. SSL performed by matching the student's s output with the teacher's probability distribution through cross-entropy loss, LCE, In contrast to vanilla DINO, DiRL yields six feature vectors from each branch (see Fig. 2}. Therefore, the loss function is modified as:
CCE = >1 (L + CC + x2 x (C + + + +CC + +CC. CCE)
where LCE is the cross-entropy loss between projection of representation fe of student and that of teacher branch. Likewise, the projected distribution of all other corresponding representations from student and teacher are matched. This linear combination of losses encourages the framework to perform a dense matching of the region-level and disentangled representations of the augmented views. Consequently, the dense matching promotes the model globally diversify the attention map (refer to Fig Fig [2].
We propose another variant of DiRL, without the disentangle block, i.c. similarities of only projec- tion distribution of fo and fo are maximized between the views. We name this variant as Cellback. Following the pre-training, only linear projection layer, position embedding, and the transformer encoder of the teacher are retained. This pre-trained ViT extracts the average pooled feature repre- sentation for all w; belonging to WSI W. to generate feature matrix of dimension (N, d), where N is variable number of WSI-crops for each W. Note that the prior is used only at pre-training. Finally, MIL operates over this matrix for WSI slide-level analysis, as discussed next.
3.6 MULTIPLE INSTANCE LEARNING FOR LLIDE-LEVER TASKS
Multiple instance learning (MIL) is widely used method in WSI slide level analysis. We refer the readers to lsse et al.] (2018) Lerousseau et al. 2 Shao et al. (202I): Lu et al. (2021b) for an overview. We adopted DSMIL (Li al. (02 framework for this work. Following pre- pre-training, the pre-trained model used to extract features for WSI-crops in W. The MIL model takes these features input bag, optimizing the model weights through slide-level label supervision.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS AND RESULTS
Our pre-trained models are evaluated for both slide-level and patch-level classification tasks. As Baseline, we pre-trained vision transformer with DINO (Caron et al. (021), a vanilla self- supervised framework, which optimizes the similarities between two views through just one image- level represetttiooo per view. This is compared pre- pre-training with our proposed DiRL and Cellback frameworks. The encoders in our frameworks are implemented with both ViT-Tiny (ViT-T, d = 192) and ViT-Small (ViT-S, d = 384) consisting of 5M and 22M parameters, respectively.
Dataset and tasks: For slide-level classification, we use the following datasets: 1) TCGAA-Lug (AI- bertina et af. 20T6; Kirk et al. 2016) at xx, x 2) TCGA-BRCA (Lingle al., (016) at 5x, TCGA- PAAD (Raphacl et al. 2017) at 5x, and 4) BRIGHT (Brancati et al., 202 at 10x. Note that our proposed pre-training is performed separately for each dataset followed by evaluating them for slide-level classification. This classification task comprises 1) TCGA-Lung: Lung Adenocar- cinoma (LUAD) versus Lung Squamous Carcinoma (LÚSC), 2) TCGA-BRCA: Invasive Ductal (IDC) verses Invasive Lobular Breast Carcinoma (ILC), TCGA-PAAD: Basal versus Classical Pancreatic Ductal Adenocarcinoma, 4) two sub-tasks in BRIGHT: 3-class WSL-classification (non- cancerous, precancerous, and cancerous), and 6-class WSL-classification, termed as BRIGHT (3) and BRIGHT (6), respectively.
For patch-level classification, evaluations are performed on Chaoyang (Zhu al. 2021) and MHIST (Wei et al. (02T) datasets, which contain localized annotation at crop-level. MHIST consists of two classes of colon cancer, whereas Chaoyang contains four classes of colon cancer.
Note that, for generating cell prior Po, we employed Ho for TCGA-Lung (Graham et al., 2019) and due to computational limitations we employed Cellpose (Stringer al.) (202]] for the other three WSI datasets. Further details on implementation and dataset splits are provided in A.1
4.1 SLIDE-LEVEL AND PATCH-LEVEL CLASSIFICATION
Slide-level classification: In Table D. we show the slide-level classification results on the four datasets with tiny and small ViT models pre-traine trained using Baseline, Cellback (DiRL without dis- entangle block), and DiRL frameworks. It may be observed that for the Lung, BRCA, and PAAD datasets, DiRL consistently surpasses the Baseline (up to 6.9% relative accuracy gain) and Cellback (up to 3.3% relative accuracy gain) models for both ViT-T and ViT-S architectures. In all cases, both DiRL and Cellback considerably outperform the Mmilll-DINO Baseline (accuracy and AUC). Interestingly, DiRL-T performs even better than Baseline-S in BRCA and PAAD, substantiating the importance of efficiently encoding diversified information even in smaller feature embedding (d- = 192) in DiRL-T against inefficiently and sparsely encoding into a larger feature embedding (d 384) VIT-S. This paves the direction for efficiently encoding -nnraiiiifooattionn in smaller models. In the BRIGHT dataset, it is observed that Cellback-S achieves the best performance for both the sub-tasks. We anticipate this might be due to the higher magnification (10x) of this dataset compared to the others (5x). Consequently there is much less contextual information in WSI- crops for BRIGHT dataset, resulting in most regions being either cell-dominant or non-cell dominant (see Fig. 14 in the Appendix). This potentially makes encoding spatial interplay especially noisy due to lower co-occurence the different concepts in a WSI-crop, thus hurting pre-training with disentangle block in DiRL (discussed in more detail in Appendix A.55) Additional comparisons with SOTA SSL methods are provided in the Appendix A.2
Patch-level classification: For evaluating the generalizability of the learned representations, we use BRCA pre-trained models and fine-tune them on MHIST (Wei et al.) (2021) and Chaoyang (Zhu et al.) (021 datasets (because of visual similarities between breast and colon cancers (Bremond jet al.] (984)). In Table [ we report the 5-fold cross valida-
Table Results for seide-level classification tasks. ViT-Tiny, and S denotes ViT-Small.
denotes
Datnsct Metre
Lang Acc, AUC
BRCA Acc, AUC
PAAD Acc, AUC
BRIGHT (3) Acc, ALC
BRIGHT (6)
AUC
Acc, AUC
Baseline-T
0.895, 0.959
0.396, 0.944
0.768, 0.561
0.625, 0.841
0.512, 0.764
Cellback-T
0.906, 0.961
0.396, 0.938
0,303, 0.713
0.650.0.888
0.512, 0.793
DiRL-T
0.906, 0.965
0.927, 0.957
0.821, 0.708
0.650, 0.859
0.500,
RoselinesS CallbackeS
0,91,00967
0.806, 0:947
0,831 0.704
0.637,
0.475,
0.921, 0.966
0.955
0,821, 0.719
0.675,
DiRL-S
0.927, 0.973
0.927. 0.950
0.839, 0.733
0,650,0.852
0.500, 0.799

---

Under review as a conference paper at ICLR 2023
tion accuracy and AUC the official test set. We observed that both our models (Cellback and DiRL) outperform the Baseline on the two datasets using ViT-T and VIT-S backbones. Relative the Baseline, DiRL improves the accuracy by 1.4 - 2% on MHIST and 0.8 1.1% on the Chaoyang dataset. Similarly, it also improves the corresponding AUCs by 0.4 - 0.7% and 0.3 0.4%, respectively.
Comparison of DiRL with other dense pre-training methods:
Note that our pre-training aligns with dense pre-training literature as we perform dense matching across two views through region- level and disentangled representations instead of just matching through one image-level representation. For fair comparison, we re-implement dense pre-training methods closely related to our research: 1) SelfPatch (Yun et al. (022) and 2) VVV (Li et al., 202Tb). In addition to image-level matching as in DINO, Self- Patch enforces invariance against each patch/token and its neigh- bors, whereas VsV enforces matching between all the corre- sponding match-based tokens across views. Note that we use ViT-S as the encoder for both the techniques. In Table we showcase the results for SelfPatch and Es for slide-level classification tasks all four datasets.
Table 2: Results for crop-level classification tasks.
Dataset Metric
MHIS Ace, ATC
Chaoysnn Ace, ADC
Baseline-T T
0.857. 0,935
0.819,
Cellback-T
0.868. 0.934
0.825, 0.942
DIRL-T
0.884, 0,942
0,828. 0.945
Baseling-S
0,885 0.945
0,830, 0.946
Callback-S
DIRL-S
0.897, 0.949
0.837, 0.950
We find that our DiRL-bbased models perform on par with Es- ViT on BRIGHT and BRCA dataset, and significantly outper- form on the Lung and PAAD datasets. Whereas SelfPatch per- forms significantly worse in most tasks, possibly because neigh- borhood token invariance hardly exists in pathology images unlike for well-defined objects in natu- ral images. Thus, our nomain-inssiiee dense matching in DiRL shows consistent improvements for slide-level classification compared to the other densely pre-trate pre-trained models.
Table 3: Comparison of DiRL with existing dense pre-training SSL methods (SelfPatch and EsVIT).
Dataset
Lung AUC
BRCA
PAAD AUC
BRIGHT (3)
BRIGHT (6)
Mitric
AUC
Apc, AUC
Baseline S
0,911, 0.967
0,896, 0.947
0,821, 0,701
0.637, 0.835
0,475,0.787 0.787
SattFacch-8
0.984
0.886, 0.926
0,821, 0.728
0.683
0,362,
ESTTS
0.916, 0.967
0.927, 0.954
0,768, 0.431
0.675, 0.857
0.525, 0.750
CallbacksS
0.921, 0.966
0.927, 0.955
0,821, 0.719
0.513
DiRL-S
0.927, 0.973
0.927, 0 0.950
0.839, 0.733
0.650, 0.852
0.500, 0 759
4.2 ANALYSIS OF LEARNED ATTENTION
Here we demonstrate the de-sparsification of the learned attention of our DiRL pre-trained models. Recall that the aggregated attention associated with a token is represented by the sum of all values across its corresponding column in the n x self-attention matrix. The sum of the aggregated attention values of all tokens (n) should be
BRCA
n. Due to this constraint, if the model attends to some tokens with
Havorn Lor of oohero
back
high attention values, then the atten- tion value associated with other to- kens are reduced significantly. In Fig. 4 we plot the distribution of aggre- gated attention values of tokens from the last layer of the transformer en- coder pre-trained by Baseline (vanilla DINO), Cellback, DiRL, Es ViT, and SelfPatch. We then split the aggre- gated attention values in three bins: 0-0.5, 0.5-2, >2. The 0-0.5 and > bins indicate sparse attention learned through low and high concentrated attention values, respectively. Whereas the 0.5-2 range is the desired bin with moderate attention values that would lead to a de-sparsified attention map (and hence, optimal encoding of context-rich information). Plots of attention distribution of the tokens are illustrated individually on the test sets for Lung and BRCA. Plots for PAAD and BRIGHT are illustrated in Appendix A.6
Altantion
Figure 4: Attention distribution plot. The second bin (0.5- 2) is the desired one. Here, the percentage values show the fraction of tokens with attention values in the desired range. The baseline method has a higher fraction of tokens in lower range and higher range sparse bins, which is not ideal for digital pathology applications.
8

---

Under review as a conference paper at ICLR 2023
The plots show that Baseline trained with vanilla DINO has around 20-30% tokens in the lower range sparse bin (0-0.5) and around 8-10% in higher range sparse bin (>2).. Whereas for our Cellback and DiRL models, fewer than 5% tokens lie in the lower range sparse bin, while 3-5% lie in higher range sparse bin. importantly, our models yield significantly more diversified attention with more than 90% tokens in the desired bin (0.5-2) compared to that of 65% tokens for the Baseline. Interestingly, SelfPatch is able to diversify the transformer attention well, avoiding the sparse bins. However it still performs 2-20% lower than our models on various slide-level classification tasks, This might be due to the neighbor invariant self- sauerrvrrriiion (refer to Yun et al. (2022) being noisy in histopathology domain (as discussed in 4.1). VsV consistently contains 10- 15% more tokens compared to the Baseline in the desired bin. However it still contains much more tokens in the sparse bins compared to DiRL and Cellbuck. These observations justify our premise that dense matching could diversify the attention, which is crucial for learning representations for histopathology.
In Fig. 5 we visualize the attention overlay from models pre-trained using Baseline vanilla DINO, and our proposed DiRL and Cellback. The regions containing tumor cells are outlined in white, while those with necrosis and immune cells are outlined in yellow and green, respectively. is evident that the Baseline model is sparsely attending the WSI crop, often ignoring crucial tumor cell-dominant regions. In contrast, our models are able to globally diversify attention. Bar plots show that almost all tokens have moderate attention values ranging from 0.5 to 2 in DiRL. In contrast, Baseline has large number of tokens having very low attention (<0.5). Note that all the attention values >> are clipped to I for visualization. Additional visualizations are provided A.6
WSI Gros
Basclire
Co Back
QRL
Attertin detributior
Figure 5: Attention visualization. Depicts the sparse attention by Baseline, and its subsequent de- sparsification by our methods representative lung cancer patch (detailed in(A.6). Bar plot shows the percentage tokens in the three bins. Baseline contains greater than 30% of tokens' attention values in the sparse bins; comparatively method contains fewer than 10%.
4.3 ABLATION STUDIES
Here study the utility of various components proposed in our frame- work. We perform our ablations the Lung cancer dataset.
04y-
0.44
Data efficiency: We investigate the effect of pre-training with different fractions (20-100%) of total training data. As seen in Fig. [6] the gain in both AUC and accuracy around 6% when DiRL-based models are pre- trained with significantly less data (20% data). These empirical findings show the importance of DiRL especially in low data regimes, for e.g. rare cancers. Other ablations including: 1) Baseline with an additional layer, 2) Effect of cell segmentation pipeline, and 3) Adaptation of DiRL with SimCLR SSL, are discussed in A.3
085
Collack
570
082
Figure 6: Data efficiency study. Illustrates the effect of pre-training with different amount of training data.
5 CONCLUSION
In this work, we present a crucial requirement of tailoring SSL techniques for digital pathology through our insightful observation about the sparsity of attention. We proposed DiRL. framework that densely encodes pathology characteristics and uses them in a dense matching objective for prior-guided pre-training. Through qualitative analysis, we showed that DiRL de-sparsifies the at- tention map. thus boosting the capabilities to encode diverse information in complex histopathology multiple slide-level
imaging. This corroborated by consistent performance improvement and patch-level classification tasks by DiRL. We believe our work opens exciting avenues toward utilizing domain-specific concepts and instilling this domain knowledge in neural networks

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yousef Al-Kofahi, Wiem Lassoued, William Lee, and Badrinath Roysam. Improved automatic de- tection and segmentation of cell nuclei in histopathology images. IEEE Transactions on Biomed- ical Engineering, 57(4):841-85, 2009.
B Albertina, M Watson, C Holback. Jarosz, S Kirk, Y Lee, and J Lemmerman. Radiology data from the cancer genome atlas lung adenocarcinoma [tega-luad] collection. The Cancer Imaging Archive, 2016.
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeck, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
Anil Bhattacharyya. On measure of divergence between two statistical populations defined by their probability distributions. Bull. Calculta Math. Soc., 35:99-109, 1943.
Joseph Boyd, Mykola Liashuha, Eric Deutsch, Nikos Paragios, Stergios Christodoulidis, and Maria Vakalopoulou. Self-supervised representation learning using visual field expansion on digital pathology. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 647, 2021.
Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio, Giosuè Scognamiglio, Guil- laume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, et al. Braes: A dataset for breast carcinoma subtyping h&e histology images. arXiv preprint arXiv:21 2021a.
Nadia Brancati, Giuseppe De Pietro, Daniel Riccio, and Maria Frucci. Gigapixel histopathological image analysis using attention-basse neural networks. IEEE Access,
2021b.
Alain Bremond, Pierrette Collet, René Lambert, and Jean-Louis Martin. Breast cancer and polyps of the colon: A contrlltrol study. Cancer, 54(11 ):256622255555 1984.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Haoyuan Chen, Chen Li, Xiaoyan Li, Ge Wang, Weiming Hu, Yixin Li, Wanli Liu, Changhao Sun, Yudong Yao, Yueyang Teng, et al. Gashis-transformen transformer: A multi-scale visual transformer approach for gastric bistopathology image classification. arXiv preprint arXiv:2 14528, 2021.
Richard J Chen and Rahul G Krishnan. Self-supervised vision transformers learn visual concepts in histopathology. arXiv preprint arXiv:2203. 00585, 2022.
Richard Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Kr- ishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16144-1615. 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Ozan Ciga, Tony Xu, and Anne Louise Martel, Self supervised contrastive learning for digital histopathology. Machine Learning with Applications, 7:100198, 2022.
James A Diao, Jason K Wang, Wan Fung Chui, Victoria Mountain, Sai Chowdary Gullapally, Ram- prakash Srinivasan, Richard N Mitchell, Benjamin Glass, Sara Hoffman, Sudha K Rao, et al. Human-intcrpreet interpraante image features derived from densely mapped cancer pathology slides pre- dict diverse molecular phenotypes. Nature communications, 12(1):1-15, 2021.
Ruiwen Ding, Prateek Prasanna, Germán Corredor, Cristian Barrera, Philipp Zens, Cheng Lu, Priya Velu, Patrick Leo, Niha Beig, Haojia Li, et al. Image analysis reveals molecularly distinct patterns of tils in nsele associated with treatment outcome. NPJ precision oncology, 6 6(1):1-15, 2022.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
Danielle Fassler, Luke A Torre-Healy, Rajarsi Gupta, Alina M Hamilton, Soma Kobayashi, Sarah C Van Alsten, Yuwei Zhang, Tahsin Kurc, Richard A Moffitt, Melissa A Troester, et al. Spatial characterization of tumor-infitrating lymphocytes and breast cancer progression. Cancers, 14(9): 2148, 2022.
Zeyu Gao, Bangyang Hong, Xianli Zhang, Yang Li, Chang Jia, Jialun Wu, Chunbao Wang, Deyu Meng, and Chen Li. Instance-based vision transformer for subtyping of papillary renal cell carci- noma in histopathological image. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 299-308. Springer, 2021.
Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 58:1518:15:0 18:13, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-superviee learning. Advances in neural information processing systems, 33:21271 -21284, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Olivier Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10086-10096, 2021.
Le Hou, Rajarsi Gupta, John S Van Arnam, Yuwei Zhang, Kaustubh Sivalenka, Dimitris Samaras, Tahsin M Kure, and Joel H Saltz. Dataset of segmented nuclei in hematoxylin and eosin stained histopathology images of ten cancer types. Scientific data, 7(1):1-12, 2020.
Maximilian llse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learn- ing. In International conference machine learning, pp. 2127-2136. PMLR, 2018.
Saarthak Kapse, Srijan Das, and Prateek Prasanna. Cd-net: Histopathology representation learning using pyramidal context-detail network. arXiv preprint arXiv:2203.15072 2022.
Kirk, Y Lee, P Kumar, J Filippini, Albertina, M Watson, and J Lemmerman. Radiology data from the cancer genome atlas lung squamous cell carcinoma [tega-luse] collection. The Cancer Imaging Archive, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-99) 90, 2017.
Nikhil Cherian Kurian, S Varsha, Akshay Bajpai, Sunil Patel, and Amit Sethi. Improved histology image classification under label noise via feature aggregating memory banks, 2022 IEEE 19th International Symposium Biomedical Imaging (ISBI), pp. 1-5. IEEE, 2022.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lec. i- mix: A domain-agnoste strategy for contrastive representation learning. arXiv preprint arXiv:2010.08880 2020.
Marvin Lerousseau, Maria Vakalopoulou, Eric Deutsch, and Nikos Paragios. Sparscconvmil: Sparse convolutional context-aware aware multiple instance learning for whole slide image classification. In MICCAI Workshop Computational Pathology, pp. 129-139. PMLR, 2021.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318-14322 2021a.

---

Under review as a conference paper at ICLR 2023
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self--sue vision transformers for representation learning. arXiv preprint arXiv:2106.09700 097885, 2021b.
Xiang Li, Jinghuan Shang, Srijan Das, and Michael S. Ryoo. Does self-supervised learning ally improve reinforcement learning from pixels?, 2022. URL httph:t//.......orttt org/abs///// 2206. 05266
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory,
1991.
W Lingle, BJ Erickson, MI. Zuley, R Jarosz, E Bonaccio, Filippini, and N Gruszauskas. Radiology data from the cancer genome atlas breast invasive carcinoma [tega-brca] collection. The Cancer Imaging Archive, 10:K9, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
David G Lowe. Distinctive image features from scale-invan invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Cheng Lu, Can Koyuneu, German Corredor, Prateek Prasanna, Patrick Leo, Xiang Wang, An- drew Janowezyk, Kaustav Bera, James Lewis Jr, Vamsidhar Velcheti, et al. Feature-driven local cell graph (flock): new computational pathology-based descriptors for prognosis of lung cancer and hpv status of oropharyngeal cancers. Medical image analysis, 68:101903, 2021a.
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021b.
Niccolò Marini, Marek Wodzinski, Manfredo Atzori, and Henning Müller. A multi-task multiple instance learning algorithm to analyze large whole slide images from bright challenge 2022. In 2022 IEEE International Symposium Biomedical Imaging Challenges (ISBIC), pp. 1-4. IEEE, 2022.
Hans Pinckaers, Bram Van Ginneken, and Geert Litjens. Streaming convolutional neural networks for end-to-end learning with multi-megapixel images. IEEE transuctions on pattern analysis and machine intelligence, 2020.
Benjamin J Raphael, Ralph H Hruban, Andrew J Aguirre, Richard A Moffitt, Jen Jen Yeh, Chip Stewart, Gordon Robertson, Andrew D Cherniack, Manaswi Gupta, Gad Getz, et al. Inte- grated genomic characterization of pancreatic ductal adenocarcinoma. Cancer cell, 32(2):185- 203, 2017
Mihir Sahasrabudhe, Stergios Christodoulidis, Roberto Salgado, Stefan Michiels, Sherene Loi. Fab- rice André, Nikos Paragios, and Maria Vakalopoulou. Self-supervised nuclei segmentation in histopathological images using attention. In International Conference on Medical Image Com- puting and Compute--Asiisted Intervention, pp. 393-402 Springer, 2020.
Joel Saltz, Rajarsi Gupta, Le Hou, Tahsin Kure, Pankaj Singh, Vu Nguyen, Dimitris Samaras, Ken- neth R Shroyer, Tianhao Zhao, Rebecca Batiste, et al. Spatial organization and molecular corre- lation of tumor-infitrraiin lymphocytes using deep learning on pathology images. Cell reports, 23(1):181-193, 2018.
Muhammad Shaban, Shan Ahmed Raza, Mariam Hassan, Arif Jamshed, Sajid Mushtaq. Asif Loya, Nikolaos Batis, Jill Brooks, Paul Nankivell, Neil Sharma, et al. A digital score of tumour- associated stroma infiltrating lymphocytes predicts survival head and neck squamous cell car- cinoma. The Journal of Pathology, 256(2): 174-185, 2022.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Ad- vances in Neural Information Processing Systems, 34:2136-2147, 2021.
12

---

Under review as a conference paper at ICLR 2023
Karin Stacke, Jonas Unger, Claes Lundström, and Gabriel Eilertsen. Learning representa- tions with contrastive self-supervised learning for histopathology applications. arXiv preprint arXiv:2112.057600 2021.
Thomas Stegmüller, Antoine Spahr, Behzad Bozorgtabar, and Lean-Philippp Thiran. Scorenet: Learning non-uniform attention and augmentation for transformer saasformer-baeedd histopathological image classification. arXiv preprint arXiv:2202.077570, 2022.
Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu. Cellpose: a generalist algorithm for cellular segmentation. Nature methods, 18(1):100-100) 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.01697, 2022.
Abhishek Vahadane and Amit Sethi. Towards generalized nuclear segmentation in histological im- ages. In 13th IEEE International Conference Biolnformatics and BioEngineering, pp. 1-4. IEEE, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supervied visual pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3024-3033, 2021a.
Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang. Jing Zhang, Junzhou Huang, Wei Yang, and Xiao Han. Transpath: Transformer basedd self-supervised learning for histopathological image classification. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 186-195. Springer, 2021b.
Jerry Wei, Arief Suriawinata, Bing Ren, Xiaoying Liu, Mikhail Lisovsky, Louis Vaickus, Charles Brown, Michael Baker, Naofumi Tomita, Lorenzo Torresani, et al. A petri dish for histopathology image analysis. In International Conference Artificial Intelligence in Medicine, pp. 11-24.
Springer, 2021.
Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXi::200..15288 2022.
Feng Wentai, Kuang Jinbo, Ji Zheng, and Xu Shuoyu. Multiple- inntiple-instance learning with efficient transformer for breast tumor image classification in bright challenge. In 2022 IEEE International Symposium Biomedical Imaging Challenges (ISBIC), pp. 1-5. IEEE, 2022.
Xiongwu Wu, Yidong Chen, Bernard R Brooks, and Yan A Su. The local maximum clustering method and its application in microarray gene expression data analysis. EURASIP Journal Advances in Signal Processing, 2004(1):1-11, 2004.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Jiawei Yang, Hanbo Chen, Yuan Liang, Junzhou Huang, Lei He, and Jianhua Yao. Concl: Con- cept contrastive learning for dense prediction pre- pre-training in pathology images. arXiv preprint arXiv:2207.06733, 2022.
Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
13

---

Under review as a conference paper at ICLR 2023
Yangen Zhan, Hao Bian, Yang Chen, Xiu Li, and Yongbing Zhang. Breast tumor image classifi- cation in bright challenge via multiple instance learning and deep transformers. 2022 IEEE International Symposium Biomedical Imaging Challenges (ISBIC), pp. 1-5. IEEE, 2022.
Jingwei Zhang, Xin Zhang, Ke Ma, Rajarsi Gupta, Joel Saltz, Maria Vakalopoulou, and Dimitris Samaras. Gigapixel whole-slide images classification using locally supervised learning. In In- ternational Conference on Medical Image Computing and Computer-Asiisted Intervention, pp- 192-201. Springer, 2022.
Chuang Zhu, Wenkai Chen, Ting Peng, Ying Wang, and Mulan Jin. Hard sample aware noise robust learning for histopathology image classification. IEEE Transactions on Medical Imaging. 41)): 881-894, 2021.
Konstantinos Cormpas-Petridie Rosa Noguera, Daniela Kolarevie Ivankovic, Ioannis Roxanis, Yann Jamin, and Yinyin Yuan. Superhistopath: a deep learning pipeline for mapping tumor hetero- geneity low-resolution whole-slide digital histopathology images. Frontiers oncology, 10: 586292, 2021.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
We provide additional details regarding the following in this section:
Dataset and Implementation details A.1
Additional experiments and results A.2
Additional ablation studies A.3
Effect of WSI magnification on DiRL
A.5
Token-level analysis A.4
Additional attention distributions and visualization A.6
Disentangled block from pathological point of view A.7
A.1 DATASET AND IMPLEMENTATION DETAILS
A.1.1 DATASETS
WSI Datasets: 1) TCGg-Lung: This dataset consists of 940 diagnostic digital slides from two sub- types of Lung cancer- Lung adenocarcinoma (LUAD) and Lung Squamous cell carcinoma (LUSC). We split the data into 748 training (391 LUAD, 357 LUSC) and 192 (96 LUAD, 96 LUSC) testing samples randomly. The WSI-crops (670K train, 150K test) extracted at 5x magnification. 2) TCGA-BRCA: This dataset consists 1034 diagnostic digital slides of two subtypes of Breast cancer Invasive ductal carcinoma (IDC) and Invasive lobular carcinoma (ILC). We split the data into 937 training (747 IDC, 190 ILC) and 97 (77 IDC, 20 ILC) testing samples randomly. The WSI-crops (790K train, 90K test) are extracted at 5x magnification. 3) TCGA-PAAD: This dataset consists of 168 diagnostic digital slides of two subtypes of Pancreatic cancer Classical and Basal. We split the data into 112 training (93 Classical, 19 Basal) and 56 (43 Classical, 13 Basal) testing samples randomly. The WSI-crops (113K train, 60K test) are extracted at 5x magnification. 4) BRIGHT: Comprises 703 (423 training. 80 validation. 200 testing) diagnostic digital slides. This dataset contains two sub-tasks: 3-class WSI classification and 6-class WSI classification tasks. For the first sub-task, the 3 classes are as follows Non cancerous (PB+UDH), Pre-cancerous or Atypi- cal (ADH+FEA), and Cancerous (DCIS+IC). For the second sub-task, the 6 classes are follows- Pathological Benign (PB), Usual Ductal Hyperplasia (UDH), Flat Epithelia Atypia (FEA), Atypical Ductal Hyperplasia (ADH), Ductal Carcinoma in Situ (DCIS), and Invasive Carcinoma (IC). The BRIGHT challenge contains train, validation, and test splits. Since the challenge is not active now, labels for the test validation set as our test set. Class-wise data split can be found here The WSI-crops (1 (1.24M train, 0.2M test) extracted at 10x magnification.
not available. Therefore, reported our results for this dataset its
Patch Datasets: 1) MHIST (Wei et al.] (02T): Consists of 3152 images of colon with tasks to classify the type of colorectal polyps into two types, benign and pre-cancerous. All the image resolutions are of 224 x 224 pixels. 2) Chaoyang (Zuu et al. 2021): Consists of 6160 patches of size 512 512 pixels from Colon cancer divided into four classes normal, serrated, adenocarcinoma, and adenoma. These patches are resized to 224 x 224 pixels in our experiments.
For these two patch datasets, we split their official training sets into a fold cross validation sets. We train on 4 folds, validate on fold and test on their official test sets. Thus, we report our results (accuracy and AUC) as a mean of fold cross validation trials.
A.1.2 IMPLEMENTATION DETAILS.
For all our experiments, 224 224 sized crops extracted from WSIs. We set the patch size for vision transformer input to = 16. Therefore, the number of tokens per WSI crop are n = 196. For ViT-Tiny (ViT-T), the embedding dimension d 192, whereas for ViT-small (ViT-S), 384.
Pre-training: For pre-training with DINO, we follow the hyper-parameter initialization from their source code (Caron et al. 202I). We use batch size of 256. In pre-training DiRL, we set the loss 0.1 weighting factors a = 0.5 and A2 4 whereas for DiRL without disentangle block (Cellback),
15

---

Under review as a conference paper at ICLR 2023
set A 0.5. We use two different projection heads of default output sizes (65536) (Caron et al., 2021) for region representations fc and fb, and four different projection heads of smaller output size (4096) for disentangled spatial interplay features. For pre-training with i) SimCLR, we adopted the implementation from Li et al. (021a)) with a batch size 512, ii) EsViT, implementation is adopted from Li et al. (202Th), and iii) SelfPatch, implementation is adopted from Yun et al. (2022). Note that pre-training is only performed on training samples of WSI datasets. For slide- level classification tasks, all models are pre-trained for 100 epochs on Lung and PAAD datasets, for 50 epochs on BRIGHT, and 30 epochs on BRCA. To study data-efficiency plot in Fig. 6 all the models are pre-trained for 50 epochs on the Lung dataset.
Multiple instance learning: We use DSMIL (L al. (021a) for slide-level classification throughout this study. For training DSMIL,, we use learning rate of 2e-1, and weight decay of 5e-2. Batch size is set to 1 to handle variable bag size for each WSI W. Other syper-parameters such as number of epochs, and train- validaaaiotioon split ratio are kept consistent with [i et al. (2021a). Note that training samples from the WSI datasets are split into train and validation for MIL training. We hope to
explore the impact of other MIL frameworks such as Ilse et al. (201 8); Shao et al. (202T): Lu et al.
2021b): Lerousseau et al. (2021); (0eennCChenne Chen al. (2022); Zhang et al. (002)) ); Pinckaers et al. (2020) on
our DiRL learned features in future.
Patch classification: In our experiments of fine-tuning for patch classification, an average pooling layer (for averaging the tokens) followed by a fully connected layer is placed on top of the pre- trained transformer-encoder backbone. For all experiments rate of 3e-4, weight decay of 1e -2, and batch size of 128. We train the network for 40 epochs and decay the learning rate by 0.1 at epoch 20 and epoch 30. For all experiments on the Chaoyang dataset, we use a learning rate of le-4, weight decay of le and batch size of 128. We train the network for 45 epochs and decay the learning rate by 0.1 at epoch 20, 30 and 40.
MHIST dataset, we use a learning
A.2 ADDITIONAL EXPERIMENTS AND RESULTS
Comparison with HIPT. HIPT Chen al. 2022 is recently proposed SSL framework tailored for digital pathology. We compare the performance of our DiRL-based models with HIPT on the datasets common both studies, i.e, Lung and BRCA. HIPT consists of eight transformer blocks of ViT-S followed by four blocks of ViT-tiny. For the model size to be comparable to HIPT parameter size, we adopted a similar model architecture as HIPT, termed with notation -H. Note that the two set of blocks in HIPT are trained in a hierarchical way, whereas we simply train all the 12 blocks together in an end-to-end manner. We remain consistent with their data split for prior-guided pre- training and MIL. training. On Lung datasets, Cellback shows relative improvement of 3% in both accuracy and AUC compared to HIPT. Whereas on BRCA, Cellback significantly outperforms HIPT with relative improvement of more than 9% in accuracy and 6% in AUC. Note that HIPT is pre- trained on whole The Genome Cancer Atlas (TCGA), consisting of 10678 WSIs from 33 cancer types. In contrast, we just pre- pre-trained for each dataset separately, thus using 10 times less WSIs
Table 4: Comparison with HIPT (Chen et al. 2022
Dataset Metric
Lung Acc
BRCA
AUC
Acc
AUC
HIPT
0.862
0.942
0.839
0.901
Cellback-H
0.888
0.971
0.917
0.958
Comparison with other methods in BRIGHT challenge: Here we compare Fl F1-scores of our models with a few papers published as part of the BRIGHT challenge (Wentai et al. (0022); Zhan et al. (202)); Marini et al. (2022)) termed as Method 1, Method 2, and Method 3 respectively, and the BRIGHT baseline presented by the challenge organizers in Brancati et al. (022 Note that BRIGHT challenge consists of WSI-level labels, well as 3000+ annotated ROIs with an average size 2000 x 2000 pixels. Using them for supervision can naturally boost the performance. BRIGHT baseline doesn't 't use annotated ROIs. Method 2 and Method 3 utilized the ROIs for training their feature encoder. In contrast, Method 1 presents two experiments, one with SSL on WSIs for feature encoder, and one with fully supervised training (FS) for training feature encoder. Following
16

---

Under review as a conference paper at ICLR 2023
this, for MIL., they experimented with both CLAM and Reformer for slide-level classification. For MIL framework in our experiments we adopted DSMIL througout this study for slide-level tasks, and DSMIL is more similar to CLAM than transformer- we included their experiments with CLAM. It can be observed that for methods not supervised with localized annotation ROI-level, Cellback performs best for both 3-class and 6-class classification tasks in terms of Fl-score. Compared to supervised counterpart, our Cellback still performs best for 6-class task, whereas just slightly lower than Method 2 and Method 3.
Reformer, for fair comparison
Table 5: Fl-score on BRIGHT validation set.
Dataset
BRIGHT (3)
BRIGHT (6)
ROIs
Metric
F1-score
Fl-score
BRIGHT baseline (Brancati al. 2021h)
0.580
0.390
X
Method (SSI. + CL.AM) (WeMta (Wentai et al. 2022)
0.642
0.412
X
Baseline-S
0.632
0,467
X
Cellback-S
0,663
0,494
X
DiRL-S
0.625
0.467
X
Method (FS CLAM) (Wentai et al.
2022)
0.691
0,453
V
Method 2 (Zhan et al. 2022
0.680
0.440
Method Marini 2022
0.650
0.450
A.3 ADDITIONAL ABLATION STUDIES
Baseline with additional layer. For fair comparison with DiRL pre-training which contains additional disentangle transformer block, we implement the Baseline ViT-S with an additional layer, ie. a model with 13 transformer blocks. We explored two versions of this model: 1) only one DINO projection head from the 13th layer (Baseline-S -13), 2) two DINO projection heads, one from the 12th layer and another from the 13th layer (Baseline-S-De sub-par to our proposed pre-training, confirming the importance of our domain- design choice in Cellback and DiRL over just adding more model parameters.
In Table both these models perform
Table 6: Effect of additional layer in Baseline
Dataset Metric
Lung
BRIGHT (3)
BRIGHT (6)
Acc
AUC
Acc
AUC
Acc
AUC
Baseline-S
0.911
0.967
0.637
0.835
0.475
0.787
Baseline-S- 13
0.916
0.963
0.625
0.808
0.462
0.766
Baseline-S 13*
0.916
0.966
0.637
0.850
0.462
0.762
Cellback-S
0.921
0.966
0.675
0.854
0.525
0.813
DiRL-S
0.927
0.973
0.650
0.852
0.500
0.799
Effect of cell segmentation pipelines: We evaluate DiRL framework with cell prior Pa generated from three different cell segmentation pipelines namely HistomiesTK, Cellpose, and Ver- Lung cancer subtyping task in Table 7 Histomic is a python API which provides handcrafted
approach for cell segmentation based on Wu al. (2004); Lowe (2004); Al-Kofahi al. (2009). The other two pipelines constitute powerful deep learning based models. Among the three, Hore Net >> Cellpose > HistomicTK in terms of segmentation performance (see Fig. 15 16). It can be observed that models trained with the near-perfect segmentation pipeline best in slide-level classification compared HistomicTK and Cellpose. It is noteworthy that for other two segmentation pipelines, our method still performs par with the vanilla DINO Baseline, except for Cellback model using HistomicTK for cell prior. This exception attributed to the poor segmentation quality by HistomicTT,, thus infusing noise in matching regions between the views in pre-trate tre-training. We believe that using Ho Ver-NNeet on other datasets will further boost their performance compared to using Cellpose in Table In Fig. 7 show the effect of cell segmentation pipelines on the re-ssarsificatin of attention. Consistently across all the three segmentation pipelines, Cell- back and DiRI. achieve better de-sparsification compared to the Baseline.
Net) performs the
17

---

Under review as a conference paper at ICLR 2023
Table 7: Effect of cell segmentation pipelines
Dataset
Lung
Segmentation
HistomicsTK
HoVer-Net
Cellpose
Metric
Acc
AUC
Ace
AUC
Acc
AUC
Cellback-T
0.890
0.960
0.906
0,964
0.901
0,960
DiRL-T
0.906
0.964
0.906
0.965
0.901
0.957
Cellback-S
0.901
0.971
0.921
0,966
0.911
0,969
DiRL-S
0.916
0.967
0.927
0.973
0.911
0.964
HoVer-Net
Cellpose
HistomicTK
Calica
Par Percorraate of tukent
Percerttaon tolore
Attentto
Figure 7: Attention distribution plot. Vertical lines separate three defined bins. The percentage values show the percentage of tokens with attention values in the desired bin (0.5-2) for Baseline, Cellback, and DiRL.
Adaptation of DiRL with other SSL frameworks: In this study so far we adopted DINO frame- work for the self-supervision of the WSIs. However, DiRL framework be incorporated with any self-supervised learning strategies. In Table [ we demonstrate the performance of our proposed Cellback and DiRL representations pre-trained with either BYOL (Grill al., 2020) SimCLR Chen et al. 2020) pipeline on the Lung dataset. We compare this adapted framework with Base- line models pre-trained with vanilla BYOL and vanilla SimCLR. Irrespective of SSL framework, Cellback and DiRI. consistently outperform the Baseline in both Accuracy and AUC.
Table 8: Pre-training DiRL with other SSL frameworks. All the results are reported for models pre-trained for 20 epochs.
Dataset Metric
Lung
SSL framework
Acc
AUC
Baseline-S
0.744
0.822
BYOL
Cellback-S
0.796
0.858
DiRL-S
0.802
0.866
Baseline-S
0.791
0.869
SimCLR
Cellback-S
0.807
0.893
DiRL-S
0.802
0.894
Baseline-S 0,802
0.891
DINO
Cellback-S
0.812
0.904
DiRL-S 0.823
0.904
Effect of MixUp in vanilla SSL: Another alternative to diversifying model attention would be to use stronger augmentation techniques. To test this, we evaluate the effect MixUp in pre-training a VIT with SSL. For applying MixUp in self-supervision, we adopted i-mix (Lee et al. 2020) in the DINO framework. In Table it can be observed, that this MixUp strategy improves the performance of the vanilla SSL due to its regularizing effects. However, this improvement is not attributed to the de-sparsification of the transformer attention weights (see Fig. g). The complementary nature of our proposed approach and stronger augmentation techniques will be explored in future work.
18

---

Under review as a conference paper at ICLR 2023
Table 9: Pre-training DINO with MixUp
Dataset
Lung
Metric
Acc
AUC
Baseline-S
0.911
0.967
Baseline-S w/ MixUp
0.916
0,972
Cellback-S
0.921
0.967
DiRL-S
0.927
0.973
Percentage of tokens
20-
Baseline Raseine Maup
Cellack
29.85%
DIRL
80 93%
5-
3.0
05
100
2.0
25
3.0
35
Attention values
Figure 8: Attention distribution plot. Vertical lines separate the three defined bins. The percentage values show the percentage of tokens with attention values in the desired bin (0.5-2) for Baseline, Baseline w/ MixUp, Cellback, and DiRL.
Multi-task adaptation vanilla SSL: An alternative approach using cell segmentation as a auxiliary task in self-supervised pre-training strategy-
prior could be to the segmentation Consequently, in this section, we evaluate the impact of pre- pre-training ViT with vanilla SSL jointly with segmentation related auxiliary task. In order avoid the use of a heavy decoder for segmen- tation, we instead design a new 'cell prediction task' to predict the number of cells present in each p X p patch of WSI-crop. A linear layer is applied top of the ViT encoder for this task.
The joint optimization of vanilla SSL with the cell prediction task could hypothetically force the model learn discriminative features from SSL and capture de-sparsification effeets from the su- pervised loss (Lsup). We adopted the cell prediction supervised loss Luup with the Baseline. We find that the model with Lup could outperform the baseline model at early epochs (accuracy of 0.828 vs. 0.802 in baseline, AUC of 0.920 vs. 0.891 in baseline). However, with later training epochs, the supervised loss does not augment the vanilla SSL pre-training (accuracy of 0.911 vs. 0.911 in baseline, AUC of 0.967 vs. 0.967 in baseline). Thus, the multi-task adaptation leads to better convergence at lower epochs but under-performs our DiRL pre-training strategy when trained for longer training schedule (100 epochs).
We analyzed the effect Lsup on the attention distribution in Fig. 9 reveals that although the supervised loss helps to de-sparsify attention to extent, the de-sparsification is still sub-par compared to Cellback and DiRL.
A.4 TOKEN-LEVEL ANALYSIS
Disentanglement of cellular and non-cellular regions. To investigate whether DiRL implicitly distills the information regarding cell and back tokens into pre-trained model, we generate t-SNE plot from cell tokens and back tokens before their average pooling in the transformer encoder. For both token types, 30K tokens are randomly sampled from WSI-crops in the test set. We then fit sep- arate 2-D Gaussians on the t-SNE points of cell tokens and back tokens (see Fig. 10), and measure the
divergence (JS) (Lin) 1991) and Bhattacharya distance (BD) (Bhattacharyyal 1943) between the two distributions. It can be observed in Table that DiRL Cellback > Base- line, in both and BD, which implies that DiRL can better separate the cell tokens from the back tokens, followed by Cellback, while the Baseline performs worst. This analysis provides us fur-
19

---

Under review as a conference paper at ICLR 2023
Percentage of tokens
sup
34.3%%
Collack
29.5%%
DILL
25
89 09.9%%
5-
0.0
0.5
1.0
1.5
2.0
2.5
3.0
35
Attention values
Figure 9: Attention distribution plot. Vertical lines separate the three defined bins. The percentage values show the percentage of tokens with attention values in the desired bin (0.5-2) for Baseline, Baseline w/ Lsups Cellback, and DiRL.
ther insights that model pre-trained with DiRL retains domain-speciin information on cellular vs. non-cellular regions.
Baseline
Cellback
DIRL
Figure 10: t-SNE plot of cell and back tokens.
Table 10: Separability of cell-back tokens. Jensen-Shannon Shanmon divergence (JS) and Bhattacharya distance (BD) metries are employed measure the overlap of distribution of cell and back tokens mapped on t-SNE plot in Fig. 10 Higher values mean better separability.
JS
BD
Baseline-S
1.04
0.26
Cellback-S
1.20
0.36
DiRL-S
1.30
0.43
A.5 EFFECT OF MAGNIFICATION
Fig. 14 illustrates WSI-crops extracted at 5x, and it's magnified view at 10x magnification tiled into four crops with same image size. The crops extracted at 5x contains four times more visual field compared to that of the crops extracted at 10x, though with less resolution. From Fig. 14 is evident that the crops from 5x contain visually diverse information, consisting of various structures such glands and cell clusters. Whereas crops from 10x are much less diverse, with crops domi- nantly consisting of either cellular or non-cellular regions. Consequently, co-occurence of both the regions is limited same size WSI-crops higher magnification like 10x compared to the ones at 5x. Limited co-occurrence of both regions in satisfactory quantity, could potentially lead to sub-optimal representation learning of spatial interplay patterns between different regions, We at- tribute Cellback's 's superior performance on BRIGHT dataset to this observation (more details rather than context). Hence, we conclude that utilizing the disentangle block is more favorable when op- erating lower magnification like 5x. Thus DiRL is suitable when downstream tasks require lower magnification to focus at diverse spatial structures such as glands, tumor infiltrating lymphocytes (TILs), and their spatial interplay in the tumor microenvironment. This could be possible reason
20

---

Under review as a conference paper at ICLR 2023
for DiRL being best for datasets processed at lower magnification (Lung, BRCA, PAAD, MHIST, Chaoyang) (see Table D
A.6 ADDITIONAL ATTENTION DISTRIBUTIONS AND VISUALIZATIONS
In Fig. illustrates the distribution of learned attention discussed in 4.2 for PAAD and BRIGHT datasets. Baseline model has around 35-40% tokens in lower range (0-0.5) and higher range (>2) sparse bins, while DiRL and Cellback contain less than 10% tokens in sparse bins. There observations on PAAD and BRIGHT are consistent with our observation of attention distribution on Lung and BRCA (refer to 4.2). Surprisingly, though the model pre-trained with ViV on PAAD dataset contains all the tokens in the desired bin with uniform attention values close to one. it per- forms worse for slide-level classification task compared to DiRL. Cellback, and even Baseline (see Table I]. This shows that although maximum encapsulation of tokens in the desired bin is helpful, to achieve optimum performance they still need to be well distributed across the desired bin which is achieved by our DiRL models.
PAAD
BRIGHT
Percentage of takens
50-
75.92%
57.46%
SeffPatch
Baseline
Percentage takens
66.35%
SelfPatch
Baseline
ESVIT
93.19%
ESVIT
40-
100.0%
Callbark
71.27%
Cellback
99 62%
90.98%
30-
97.37%
30.59%
23
0.0
3.3
Attention values
Attention values
Figure 11: Attention distribution plot on PAAD and BRIGHT dataset. The second bin (0.5-2) is the desired one. Here, the percentage values show the fraction of tokens with attention values in the desired range. Similar to the plots fig. the baseline method has a higher fraction of tokens in lower range and higher range sparse bins, which is not ideal for digital pathology applications.
In Fig. 12 we visualize the attention overlay from models pre- trainerid using Baseline vanilla DINO, and our proposed DiRL and Cellback. We mark various regions such as region dominated by tumor cell, immune cells. It can be observed that for Lung cancer, model predominantly focus on immune cell region, while ignoring much of the tumor cells region. Oppositely in BRCA samples, model attends to tumor cells region mainly, while ignoring the immune cells. This could result in loss of crucial information in encoding the representation by Baseline. In contrast, Cellback and DiRL are able to adequately attend each such region, thus encoding more informative representations.
So far, in qualitative attention analysis (see Fig. 3 and Fig. 12) we visualized the aggregated attention being given to the tokens WSI-crop level. In Fig. 13 we now visualize the interaction relationship between given query token with other tokens in the WST-crop. We illustrated how given query (marked with yellow bounding box) interacts with (attends to) other tokens by visualizing the row of an attention matrix corresponding to the query. Our findings in query-level interaction analysis is consistent with that of previously shown WSI-crop level attention analysis, ie, the query token sparsely attends to different tokens in baseline (vanilla SSL). In contrast, our pretraining alleviates this problem and densely interacts the query token with other tokens in the image.
A.7 DISENTANGLED BLOCK FROM PATHOLOGICAL POINT OF VIEW
In this study, we aim to model the interaction between the cellular (comprising various types of cells) and non-cellular regions (comprising stroma, smooth muscle region, fat, etc). In pathology, the interaction between various entities (nuclei, stroma, glands, etc.) has been found to have clini- cal significance (Saltz et al.] 2018 Diao al. 2021 Zormpas-Petridis et al. 2021). For modeling the interaction between cellular and non-cellular regions, we believe that disentangling both the regions followed by explicitly encoding the inter-intra region interaction is necessary. Otherwise
21

---

Under review as a conference paper at ICLR 2023
without disentanglement, encoding the inter-intra region interaction would not be precisely achiev- able, pertaining to the fact that average pooling in vision transformers could potentially dilute these token-level crucial signals. Future directions could delve into utilizing more refined entities (such as immune cells region, tumor regions, glands, necrotic region, and stroma) and quantifying their mutual interactions, thus better guiding the neural network to learn intricacies of digital pathology.
22

---

Under review as a conference paper at ICLR 2023
WSI-crop
Baseline
Cellback
DiRL
High
TCGA-LUNG
a.2
a.3
Low
TCGA-LUNG
b
b.2
b.3
TCGA-BRCA
TCGA TCGA-BRCA BRCA
d.
d.2
d33
TCGA-PAAD
e.2
e.3
Figure 12: Attention visualization. a and b shows an example 5x patches (size 224 x 224) from lung cancer, с and d shows an example 5x patch (size 224 224) from breast carcinoma BRCA, shows an example 5x patch (size 224 x 224) of a normal region from the pancreatic adenocarci- noma dataset. column 2, (*.1) shows the Baseline model attention maps of the patches column 1. In column 3, (* (*.2) shows the Cellback model attention maps of the patches in column 1. In column 4, (*.3) shows the DiRL attention maps of the patches column
In a, b, c, and d, the tumor regions have been annotated in white, lymphocytic regions in green and necrosis in yellow. In the first and second rows, the arrows indicate the tumor regions which were sparsely attended to by the baseline compared
models. Though the bascline model attends well to the lymphocytic regions in a. it fails to densely attend to the important tumor areas. In the third and fourth row, the arrows indicate the lymphocytic regions which were sparsely attended to by the baseline com- pared to our models. The tumor regions, however, exhibit high attention for all the models. In the fifth row, the baseline models sparsely attend many of the acinar tissue regions (blue regions). In contrast, our model diversifies the attention over all the acinar regions.
23

---

Under review as a conference paper at ICLR 2023
Baseline
Cellback
DiRL
Figure 13: Tokens Interaction Visualization. Query token is marked in yellow bounding box. The interaction of the query token with all other tokens in the WSI-crop is illustrated for Baseline, Cellback, and DiRL. can be observed that, in Cellback and DiRL. the query token interacts more densely to various tokens in the image compared to the Baseline.
24

---

Under review as a conference paper at ICLR 2023
5X
10X
10X
10X
10X
5X
10X
10X
10X
10X
Figure 14: Illustration of WSI-crops extracted at 5x and their magnified view at 10x magnification tiled into four images with same size as of WSI-crops at 5x. The crops at 5x contain diverse phenotypic information such cluster of cells, glands, and stroma. In contrast, the crops at 10x are often limited in phenotypic diversity these structures.
25

---

Under review as a conference paper at ICLR 2023
WSI-crop
HistomicsTK
Cellpose
HoVer-Net <
Figure 15: Cell segmentation performance. Illustrations WSI-crops, and the output of cell segmentation from HistomiesTK, Cellpose and Herer-Net. Black arrows denote normal cells/lymphocytes, and yellow arrows denote large tumorous cells. HistomicsTTK misses numer- ous tumorous cells, while adequately detecting the normal ones. The more powerful deep-learning pipelines, Cellpose and Ho Her-NNe, are able to capture both type of cells with greater precision.
26

---

Under review as a conference paper at ICLR 2023
WSI-crop
X
HistomicsTK
x
Cellpose
HoVer-Net
Figure 16: Illustrations of WSI-crops, and the output cell segmentation from HistomicsTK, Cell- pose, and Ho Ver-Net. Green arrow denotes blood cells, and yellow arrow denotes large tumorous cells. Note that the arrangemen/ddisriibutioo of blood cells is not implicated in tumor phenotyping. Both HK and Cellpose segment the blood cells, while Ver-Ne avoids them large extent. This example shows that Ho Ver-Net is preferable.
27


Reviews:
[{"review_id": "aSKWmKewLTf", "reviewer": "Reviewer_vgEj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a digital pathology representation learning method that uses a vision transformer. The method splits a given whole slide image (WSI) into multiple crops and the cropped images are represented by using multi-head self-attentions (MSAs): Each cropped image is split into patch images, the patches are encoded by MSAs, and an average pooling is applied to the feature vectors obtained by the MSAs as a feature vector of the crop. Different from conventional vision transformer, the proposed method separates the patches into two regions: A cell-region and a back-region. The former is a set of the patches with cell nuclei inside and the latter is a set of the patches without cell nuclei inside. The method computes six different feature vectors for each crop. Mixing the patches from both regions and using vision transformers, the proposed method obtains a feature vector for each of the two regions by applying average pooling to the resultant feature vectors of the patch images. Separating the patches for each region and applying vision transformer, the method obtains another feature vector for each region. This feature vector is obtained based on the context within each region. Other two feature vectors are obtained based on the context between the two regions, which is described by the attention matrix computed basically from pairs of two patches from different regions. It is shown that separating (disentangling) the patches into the two regions, one can diversify the strength of the attentions and the performance of the classifiers can be improved.\n", "summary_of_the_review": "Because of the weakness described above, the reviewer rates this paper just below the borderline. I am willing to rase the rating if it is appropriately explained the reason why the proposed method can be classified as a SSL method.", "strength_and_weaknesses": "Strength:\nAs the authors mention in the paper, WSIs have characteristics different from natural images. In pathology images, cell nuclei are spatially dispersed, and the features of each cell nucleus, the spatial patterns of the multiple cell nuclei and the cytoplasm textures can all contribute to pathological image diagnosis. The authors point out the importance of the divergence of the attentions and find that the proposed method can diversify the attentions. The diversified attention would describe not only the relationships between spatially close patches but also the relationships between more distant patches distributed in wider areas. The experimental results demonstrate that the proposed method improves the performance of classification significantly.\n\nWeakness:\nThe method employs a cell segmentation method that requires labeled data and hence the reviewer questions whether the proposed method can be classified as a self-supervised learning (SSL) method and whether the performance comparison with SSL methods is fair. The labeled data used in the training of the cell segmentation could implicitly teach that the nuclear regions would play important roles in the classification tasks.\n\nThe analysis of the obtained attentions is not satisfactory. The experimental results show that the attentions are diversified but the details of the obtained attentions between tokens are not indicated. As shown in the left image in Fig.5, each crop image would include pathologically different regions such as the necrosis regions and the tumor ones. The attention matrix can represent the relationships between the patch images. Did the attention matrices obtained by the proposed method and by other conventional methods represent the pathological difference of the regions? By disentangling the representation, did the representations obtained by the proposed method become more valid from a pathological point of view?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K8nU5EzKIph", "reviewer": "Reviewer_nwTd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mentioned an observation: the vanilla SSL (for example, DINO) usually learned sparse features, which is suitable for object-centric tasks but not good for non object-centric tasks. As a result, the paper proposed using domain-priors together with several pretext tasks to learn de-sparsification features. The experiments on several datasets (slide-level/patch-level classification tasks) show the effectiveness of the proposed method.", "summary_of_the_review": "This is an interesting paper to solve sparse feature problem in SSL for pathology images. Generally, I'm interested in the observation part and fine with the proposed method (I agree the proposed method can largely alleviate the problem). However, I donot think this paper touches the essence of the phenomenon, which I think is due to data and the task (and we can use certain data augmentation to generate better views and thus enforce the model to learn better \"dense\" features). ", "strength_and_weaknesses": "Strength:\n1. The observation is quite interesting and it makes sense.\n2. The proposed strategies (domain-priors, pretext tasks..) is simple but make sense to de-sparsification the attention map.\n3. The results on several datasets are promising. The authors indeed conduct many experiments and well describe the datasets and corresponding experimental results.\n\nWeakness:\n1. The paper only considers on SSL framework, what's the situation in other SSL frameworks?\n2. It is hard to apply the proposed method on other tasks, because we first need a well trained segmentation model which are not easily achieved in many cases. \n3. Suppose we have segmentation ground truth and train the segmentation model, mabye we can just design a multi-task (ssl+supervised segmentation) learning framework and then achieve the de-sparsification effects (segmentation force the network to learn details).\n4. The claim \"We believe our work opens exciting avenues toward utilizing biology-relevant concepts and enhancements in neural networks\" is weird, where do you use biology-relevant concepts and enhancement?\n\nConcerns:\n1. I can imagine that the SSL models (e.g., DINO) on imagenet classification tasks produces sparse attention features (object-centric). I think that's because the task (classify those images mainly need to focus on the object) and data (object-centric). However, if the data is non-object-centric (for example, pathology slides), then the task (if well trained) should be able to force the model to learn more dense features, since it is these dense features which can contribute to correct classification. If it is in such a case, do we still need the de-sparsification step?\n2. Maybe some augmentation which are well designed for non-object-centric data can better solve this problem. I believe the sparsification is due to the data and the task (classification), so if we use some kind of augmentation to generate the two views (considering the dense essence of the data), the SSL should be able to learn the de-sparsification features.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5T_z6H3aM9", "reviewer": "Reviewer_6NiW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed manipulating the attention learned during self-supervised learning toward more meaningful regions (cells against the background). Precomputed masks of cells are injected into the self-attention modules, i.e., combined with the self-attention before the softmax. Attentions are further divided into groups of cells, backgrounds, and cross-cell-backgrounds. Several public datasets are employed here for the experiments and evaluation. Superior results of the proposed method are reported in comparison to self-supervised learning baselines (pre-trained with DINO). ", "summary_of_the_review": "The overall quality of the presented work is good, though there are some flaws in the introduction of the method and experimental settings. Adjusting the attention for medical images is essential in addition to the standard self-attention-based pre-training. The authors proposed a novel way toward that direction.", "strength_and_weaknesses": "Strength:\n+ The manuscript is overall well-written and easy to follow\n+ The idea of injecting prior-based attention into the self-attention module is novel and seems effective, at least for pathological image-related applications.\n+ Many public datasets are employed here for the experiments\n\nWeakness:\n- I am concerned about the comparison study, considering only baselines are compared. If the proposed attention add-on should be regarded as a general approach, other self-supervised learning frameworks/networks should be considered in the comparison study as well. Otherwise, SOTA results on these utilized datasets could also be reported to get a sense of how the proposed framework generally works.\n- The definitions of M_self and M_cross are not well-introduced and discussed. I am unsure about the philosophy of setting the values to either 0 or minus infinite. Moreover, the M_self and M_cross are added to qk^T directly without considering the scale of these two vectors. Is it better to be a weighted sum?\n- How the accuracy of M_self and M_cross will affect the overall training performance should also be investigated and discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject