Lupus Nephritis Subtype Classification with only Slide Level Labels

Authors: amit  sharma
ekansh  chauhan
megha s uppin
liza  rajasekhar
c v jawahar
e  sharma
  chauhan
l  uppin
c v rajasekhar
p k vinod jawahar
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 10:47:23

Abstract: Lupus Nephritis classification has historically relied on labor-intensive and meticulous glomerular-level labeling of renal structures in whole slide images (WSIs). However, this approach presents a formidable challenge due to its tedious and resource-intensive nature, limiting its scalability and practicality in clinical settings. In response to this challenge, our work introduces a novel methodology that utilizes only slide-level labels, eliminating the need for granular glomerular-level labeling. A comprehensive multi-stained lupus nephritis digital histopathology WSI dataset was created from the Indian population, which is the largest of its kind. LupusNet, a deep learning MIL-based model, was developed for the sub- type classification of LN. The results underscore its effectiveness, achieving an AUC score of 91.0%, an F1-score of 77.3%, and an accuracy of 81.1% on our dataset in distinguishing membranous and diffused classes of LN.

Main paper: 
 introduction: 
 Lupus Nephritis (LN) is one of the most severe manifestations of systemic lupus erythematosus (SLE), an autoimmune disease, due to its potential for severe renal damage and the intricate diagnostic and classification process. The complex nature of this disease is worsened by the substantial-high inter and intra-observer variability in histopathological renal biopsies (Dasari et al., 2019) . As some classes of LN exhibit varying levels of aggressiveness, a precise classification of these classes becomes crucial in assessing fatality risks, predicting long-term prognosis, and determining a practical therapeutic approach.Deep learning has recently emerged as a powerful tool in medical AI and healthcare, revolutionizing various aspects of medicine, from diagnosis and treatment to drug discovery and patient monitoring (Rajkomar et al., 2018) . Digital pathology has significantly advanced due to its capacity to extract intricate patterns and features from complex medical data (Wu and Moeckel, 2023; Ahmed et al., 2022) . Improvements in image analysis have led to significant advancements in various aspects of renal pathology, including automated detection and classification of glomerular lesions (Sheehan and Korstanje, 2018; Ginley et al., 2019) , and identification of interstitial fibrosis (Zheng et al., 2021a) . Advanced imaging techniques and molecular analyses may assist, but standardization and consensus in interpretation remain ongoing challenges.Traditional LN classification follows a two-step process: first, identifying glomeruli types, then classifying LN based on these types, heavily dependent on detailed glomeruli annotations (Sheehan and Korstanje, 2018; Zheng et al., 2021b ). Yet, annotating glomeruli on large-scale WSIs is impractical in clinical settings due to their massive size and memory limitations, leading to patching and streaming solutions (Campanella et al., 2019; Pinckaers et al., 2020) . Previous studies mainly differentiated LN from non-LN, not addressing subtype classification (Wang et al., 2023) , which is complicated by similar glomerular types across subtypes and the unequal contribution of glomeruli to classification. (Cicalese et al., 2021) proposed an end-to-end LN subtype classification method, but it required manual segmentation on mice biopsies, not directly applicable to human samples due to differences in physiology and pathology.In contrast, our work simplifies this process by creating an end-to-end pipeline that does not necessitate reliance on glomeruli class labels at any intermediate stage. Multiple Instance Learning (MIL) has been extensively explored for other areas of digital histopathology (Campanella et al., 2019) , but not much has been reported or explored in renal pathology.While digital pathology has made strides, the LN classification research faces challenges such as access to the datasets and lack of consensus among medical professionals regarding its classification. In light of these considerations, the principal contributions of our work are as follows:• We focus on creating a valuable dataset of LN to drive research (computational and medical) in kidney diseases. This dataset, featuring multi-stained whole slide images, stands as one of the largest collections for lupus nephritis, a part of the consortium India Pathology Dataset (IPD)1 .• We also introduce a novel architecture, LupusNet, an explainable MIL-based model that significantly improves LN subtype classification by integrating Gated and Multi-Head Attention, underscoring the critical requirement to learn the morphological differences between LN classes 4 & 5.• To the best of our knowledge, we present the first end-to-end pipeline for LN subtype classification by relying only on slide-level labels, eliminating two-step methods that relied on glomeruli labels, easing clinical workload and facilitating practical integration.LupusNet 
data acquisition & description: 
 In this study, biopsy specimens of 166 patients (retrospective and prospective cases) in different subclasses (ranging from 1 to 6) of LN from the Nizam Institute of Medical Sciences (NIMS) in Hyderabad, India, were digitalized. A total of 540 WSIs were digitalized using the Morphle Optimus 6X Scanner, with each WSI captured at a maximum magnification of 40x and stored in the widely used TIFF format. Slide-level labels depicting subtype classes for each of the cases were also recorded. Within this repository of 540 WSIs, there are four distinct categories of stained images, specifically Hematoxylin and Eosin (H&E), Periodic Acid-Schiff (PAS), methenamine silver Periodic Acid-Schiff (mt-PAS), and silver methenamine Periodic Acid-Schiff (sm-PAS). In this dataset, LN classes 4 (diffused proliferated) and 5 (membranous) exhibited the highest representation, with 62 and 53 cases, respectively. Class 4 LN displays a varied glomerular appearance characterized by widespread inflammation, cellular proliferation, and diverse lesions, whereas class 5 LN demonstrates a uniform appearance due to immune complex deposition, resulting in a membranous pattern (Weening et al., 2004) . Figure 1 shows glomerulus samples from our dataset for each class. Consequently, our study focused primarily on observations and results for these two prominent LN class classifications using PAS-stained slides highlight carbohydrates, glycogen, and glycoproteins, aiding the identification of renal structures.This India region-specific dataset is created to support global collaboration in lupus nephritis research. It helps add diversity to the other existing cohort, offering insights into potential regional and ethnic variations in the disease. We aim to learn a function that can predict the presence or absence of a condition within a WSI based on its constituent patches. Mathematically, this problem can be defined as follows: We are provided with a dataset containing pairs of bag-labels {(X i , Y i )} D i=1 . Each X i represents a collection of instances (patches) within a bag, and Y i is the label assigned to that bag. Each bag X i contains a variable number of instances {x 1 , x 2 , . . . , x N } ∈ X i . These instances have labels {y 1 , y 2 , . . . , y N } with y n ∈ {0, 1}. However, the labels for individual Y i = 1, if ∃x n ∈ X i such that y n = 1 0, otherwiseOur methodology extends this formulation to multiple positive classes for subtype LN classification. Unlike lung, brain, and breast datasets, renal pathology primarily focuses on a limited region of interest, particularly the glomerular area, allowing us to use recurrent networks. Glomeruli play a pivotal role in various renal diseases, including LN. Instead of providing MIL with all WSI patches, we exclusively use glomerular patches, enhancing precision by avoiding potential noise. Recognizing the laborious labeling at the glomerular area, we aimed to eliminate the need for intermediate glomerular-level labels; thus, opting for weakly supervised approaches is an appropriate option.Our novel end-to-end MIL architecture for LN classification, LupusNet, works on raw glomerular patches extracted using a fine-tuned YOLOv4 model (Hemmatirad et al., 2023) , with two key components: (a) Feature Extractor (f ) and (b) Feature Aggregator (g), jointly trained. f transforms inputs into an information-rich feature space using a ResNet-50 network pre-trained on histopathology images (Kang et al., 2023) . We built on CLAM principles (Lu et al., 2021) , which utilizes gated attention pooling and instance-level clustering to distinguish positive from negative samples. Gated attention, however, cannot fully exploit the uniformity of class 5 lupus nephritis glomeruli, hindering its ability to achieve optimal efficacy in capturing its consistent patterns. We hypothesize that adding contextual information among all glomeruli patches will improve the performance. To address this, we integrate self-attention and Bi-LSTM into the MIL framework, enhancing contextual understanding among instances (patches) in a WSI.Suppose, in a WSI bag X, we have N glomerular patches, and the Feature Extractor f transforms each image x n ∈ R 224×224×3 into a h vector of dimension d ∈ R 1×d . For N such images, we obtain a matrix H ∈ R N ×d (eq: 1). Our feature aggregator can further LupusNet be divided into three branches: (1) Gated Attention Pooling, (2) Self-Attention + LSTM and (3) Instance-level Clustering. In Branch 1, the gated attention block assigns attention scores A g = {a g 1 , a g 2 , . . . , a g N } ∈ R 1×N to every instance (eq: 2), followed by instance-level clustering using A g as pseudo labels for confident instances (Branch 3).H = f (X; Θ) where H = {h 1 , h 2 , . . . , h N }(1)a g k = W T c (tanh(W a h T k ) ⊙ σ(W b h T k )) N j=1 W T c (tanh(W a h T j ) ⊙ σ(W b h T j ))(2)C g = N k=1 a g k h k (3)where W a , W b and W c are trainable parameters, a g k can be supposed as positive probability of instances. σ represents sigmoid function and ⊙ represents element-wise multiplication. C g is the output context vector of Branch 1 (eq: 3).In Branch 2, initially, H goes to MHA, yielding contextualized output among instances (A s ). Self-attention (eq: 4) enables context consideration between every instance pair, and the multi-head mechanism focuses on modeling various such contextual relationships and dependencies among instances. The attention scores obtained from different heads, n h is a total number of heads, are concatenated, and a linear transformation is applied to ensure that the resulting shape matches the input, resulting in R n×d (eq: 5). To further process this contextualized information, we employ LSTM, which uses gating mechanisms and outputs the hidden layer of the last time step R 1×d .EQUATIONwhere Q i = HW Q i , K = HW K i , and V = HW V i , for the i th head, are derived using trainable parameters W Q i , W K i , W V i , and W o linearly transforms the multi-head outputs. d k is used for scaling to prevent the dot product from becoming too large, and C s is the bi-LSTM processed output context vector from Branch 2 on A s . Furthermore, we use softmax normalized learnable parameters s 0 and s 1 to adaptively aggregate contributions from each pipeline's output. A scaling learnable parameter γ finetunes the overall merged output contribution, introducing an additional degree of freedom in the weighting process (eq: 6). Inspired by attention principles, this approach facilitates contextual understanding and dynamic weighting for effective information extraction from both branches. It draws parallels from a multiple-layer fusion of contextual embeddings in ELMO during downstream tasks (Peters et al., 2018) .logits = γ (s 0 C g + s 1 C s ) (6)After applying the adaptive aggregation method, a binary classifier with a single neuron and a sigmoid activation function is used to estimate the probabilities, y, of a slide being positive. Subsequently, binary cross-entropy loss is computed at the slide level (Branch 1 and 2), while Smooth SVM loss (Lu et al., 2021) is applied for instance-level clustering (Branch 3). The Smooth SVM loss, a generalization of traditional cross-entropy classification loss, accommodates diverse margin values and temperature scaling strategies, providing flexibility to mitigate overfitting. The rationale for choosing Smooth SVM loss lies in addressing potential noise in pseudo-labels, offering robustness in the presence of uncertainties. The total loss, as per Equation 7, is calculated as the weighted sum of both losses, where H ′ and A g′ are the subset of H and A g respectively, ŷ is the ground truth, and β is a hyper-parameter.J = β BCE(y, ŷ) + (1 -β) Smooth-SVM(H ′ , A g′ ) (7)3. Experiments and Results 
experiment setup: 
 For a robust evaluation of classification performance, we employed 10-fold cross-validation. All methods were implemented in PyTorch and trained on a single NVIDIA RTX 3080ti GPU. The patch size for the YOLOv4-based glom detector was set to 6000 × 6000, and the MIL training involved 50-200 epochs with early stopping. n h = 4, β = 0.8, a Bi-LSTM hidden dimension of 512, and Adam optimizer with lr = 1e4. Batch size is set to 1 for all models. Our code is available on GitHub2 . We established baselines using a pseudo-labeling approach for lack of detailed glomeruluslevel labels by assigning whole slide labels to all glomeruli and tested models like AlexNet, ResNet, and DenseNet, with ResNet-101 performing best (Table 1 ). These experiments underscored the challenge of label inconsistency among similar glomeruli in lupus classes 4 and 5, affecting model accuracy and emphasizing the need for alternative methods in the absence of precisely labeled datasets. Furthermore, we employed an end-to-end vanilla Vision Transformer (ViT) (Dosovitskiy et al., 2020) on glomeruli patches followed by weakly supervised max-pooling, average pooling, CLAM single-branched variant (CLAM-SB), DSMIL (Li et al., 2021) and Trans-MIL (Shao et al., 2021) and our proposed LupusNet on the in-house dataset. CLAM-SB results are presented for both scenarios, wherein we either input all the WSI patches or just the glomeruli patches. Pooling methods showed competitive performance, with max pooling achieving 81.00% AUC, 72.82% F1 score, and 73.33% accuracy, and average pooling resulting in 85.50% AUC, 76.98% F1 score, and 77.78% accuracy. The conclusive findings, as shown in Table 1 , demonstrate that LupusNet outperforms all baseline models. We can empirically observe a significant performance improvement when only glomeruli patches are provided, consequently reducing noise to the weakly supervised models. Additional observation showed LupusNet outperforming CLAM-SB (GP), by a significant F1-score improvement for class 5 LN (65.17% to 77.03%), highlighting its efficacy in distinguishing the two classes, reducing false positives and enhancing precision. In our ablation study, we methodically introduced various architectural components to evaluate their individual and combined effects on the model's performance. Beginning with a basic LSTM model as our starting point, we then integrated Gated Attention and Instance-level clustering. Each addition led to noticeable improvements in performance, as shown in Table 2 , with our final model, LupusNet, outperforming all other configurations. This step-by-step process helped us identify the specific contributions of each component to the model's overall effectiveness in classifying two LN classes. We further optimized LupusNet by adjusting the learning rates and the number of Multi-Head Attention (MHA) blocks (Figure 3 ).  
discussion and conclusion: 
 Our study has showcased the application of MIL for LN subtype classification, which uses only slide-level labels, eliminating the necessity for glomeruli-level labels. Our idea was to explore how weakly-supervised methods perform in this situation and propose a framework (LupusNet) to improve it. Although using transformer-based models seems like a natural choice for their advanced context sensitivity, their empirical efficacy was suboptimal due to the reduced regions of interest. However, we recognized the need for self-attention among glomeruli for context inclusion. Therefore, our work includes this aspect without increasing network complexity by using LSTM and MHA. Furthermore, the attention weights can be assessed to infer the contribution of each glomeruli in the final classification which can also help reduce the inter and intra-variability among pathologists. Additionally, it holds significance for researchers studying other diverse renal diseases beyond the specific focus on LN. It also contributes to renal pathology research by creating a digital whole slide image dataset. While LupusNet exhibits promising results, there are areas for potential improvement. Our future work involves improving glomeruli detection models and feature aggregators, which could extract even better contextual information from glomeruli. Data Availability Statement: The dataset generated and/or analyzed during the current study is available from the authors within the terms of the data use agreement and compliance with ethical and legal requirements (if any). 
 
 Appendix: Procedures in studies with human participants adhered to ethical standards set by institutional (NIMS) and/or national research committees (ICMR).https://hai.iiit.ac.in/ipd/Code: https://github.com/CancerDiag/LupusNet 


Decision: All reviewers found the proposed method to be novel and the results promising, therefore I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposed an attention-based multi-instance learning model, LupusNet, that allows classification of two glomerular types using only slide-wise labels. The results show that the proposed method outperformed others in instance-wise glomerular classification. The ablation study also provides a comprehensive comparison of the different contributions of the design.
main_review: main_review: I appreciate all the efforts from the authors during the rebuttal session, which improved the quality of the paper and clarified its scope by (1) clarifying the importance of slide-wise predictions for LN subtype classes and the contribution of the slide-wise labeled dataset, (2) providing additional experiments from other baseline methods, and (3) making the source code available for replication. I also appreciate the other reviewers for highlighting the contribution of the dataset as well as the innovation of the methods. I have increased the score for these efforts. I have increased the score for these efforts.

However, the discussion during the rebuttal revealed several limitations of the current stage of the work. The main concerns remaining are (1) the use of a small-scale private dataset, which poses the risk of cherry-picked results that may not fairly demonstrate the capability and generalizability of the models. If the authors aim to position this paper as a benchmark, additional datasets are encouraged to be included. (2) The explanation and interpretation of the results from the proposed methods may not fully showcase the models' capabilities. It is unclear whether the improved performance is due to the model design with correct knowledge learned from different gloms or simply due to increased complexity and parameters in the architectures. Therefore, some intermediate representations (glom-wise) of the models might help the audience to believe the functionality of the models. It should also be noted that achieving semi-quantified activity/chronicity scores for WSIs requires accuracy at the glomerular level, which is the main reason I questioned the motivation and contribution of the paper.

Overall, the quality of the paper has improved after the rebuttal, prompting me to increase the score. All the discussions and additional experiments during the rebuttal should be included in the final version of the paper if it is accepted.
strength_weakness: strength_weakness: 1. The motivation and clinical value of the method are insufficient. During data collection, instance-wise annotations were collected, and fully supervised learning methods have already well resolved the classification tasks. There is no clear technical question or clinical problem in the data collection and supervised learning classification. Why do the authors frame the question as weakly supervised learning, creating a non-existent barrier?
2. More state-of-the-art MIL-based methods should be compared to demonstrate the functionality and capability of the proposed method, besides CLAM, which was proposed 3 years ago.
3. More details of the pipeline are lacking, creating a barrier for pipeline replication.
questions: questions: Please further illustrate the motivation and contribution of the proposed method, as well as address the questions in the weaknesses and detailed comments sections.
paper_summary: paper_summary: The authors propose two main contributions: 1) a fairly large-scale dataset of Lupus Nephritis and 2) and end-to-end pipeline for LN subtype classification. The method utilizes attention mechanisms to enable slide-level labeling to be sufficient for training their network.
main_review: main_review: The new experiments are fantastic! Comparison with ViT and ViT in place of LSTM + MHSA are EXACTLY what this paper needed. I also appreciate the removal of the half-thought-out interpretation claims. I change my rating to a strong accept. I know there is still some hesitancy from the other reviewers, but as a consistent authors and reviewer of MIDL I think this work definitely meets the bar of publication for this venue. I appreciate the other reviewers concerns about the complicated nature of the method, which is somewhat unavoidable (see my strengths section about attention and the weakly supervised nature of the approach as well as the authors new experiments on a vanilla ViT), and concerns about the usefulness of a weakly labeled dataset (but honestly this is pretty standard for slide datasets in general). But I would strongly urge the AC to consider this work for publication. This is a fairly large scale dataset contribution and a fairly novel technique that outperforms ViT by a wide margin, that alone justifies publication easily.
strength_weakness: strength_weakness: 1. It is unclear why the authors combined an LSTM with a transformer style network. Then entire point of a transformer is the attention mechanism removes the need for a memory unit. The HOW is clearly explained in section 2, but the WHY seems unclear. See questions below. But this makes the method feel a bit like, "we threw everything but the kitchen sink" at it, it would be great to see this solved via a pure transformer.

2. The authors have a few great comparisons, ResNet-101, CLAM, and an LSTM. These latter two are particularly good ablations. However, it would have been really nice to see some other related works here, rather than just baselines. I understand since this is a custom dataset, this would be replicating their methods, which hopefully have open source code, but the authors reference previous two-stage approaches and having an easy reference point of how those perform would be very valuable. 

3. Figure 3 and section 3.3 claim to speak to the interpretability of the authors' approach. However, there is no interpretation provided. Just, here is a single test sample and some activations. Okay, cool. What does it mean? Are there consistent patters from one image to the next? There is no interpretation here. To make space for this (or just remove it, because as is is not worth keeping), I would recommend shortening the details in section 2.2. You don't need the equation for a bi-directional LSTM, people can go look that up because it's not novel to this work.
questions: questions: Why do the authors choose to combine an LSTM with their transformer based architecture. Is the memory unit attempting to solve a memory issue of some kind, where because you're patch sampling, you need some notion of other patches? Could this possibly be solved with some sort of sparse (long-context) style transformer instead of the complexity and inefficiency of an LSTM?
paper_summary: paper_summary: The authors present a dataset and method for lupus Nephritis subtype classification from WSIs, using a multiple instance learning paradigm. In particular, they show that predetection of glomeruli regions is really helpful for this task. The results indicate that the approach introduced by the authors outperforms standard MIL approaches such as CLAM.
main_review: main_review: I think the paper has seen significant advances in the rebuttal period and the authors were really responsive and reacted constructively to the criticisms. I also read the insightful comments of reviewer svAa and agree with the other reviewer and I am now also sure the paper meets the acceptance criteria for MIDL.
strength_weakness: strength_weakness: - The approach seems a bit complicated, especially in the combination of two long-range dependency modeling schemes (MHSA and LSTM). Could the authors explain why a more simple aggregation was not trialed? In particular, the ablation study suggests that the LSTM in itself features only marginal performance (AUC of 0.64, Test F1 of 0.56). 
- The results given by the authors only report single shot performance. However, given the size of the dataset, we could expect a quite significant data dependency between patients. I think that the authors should run a cross-validation on their data and report mean+std (or results across runs) for a more fair comparison across approaches. 
- The dataset is only available "upon reasonable request". If there is no distinct legal requirements against this, then I would strongly suggest to make the data available openly without preconditions, especially since it is listed as core contribution of the paper.
- The interpretability of the approach is questionable.
questions: questions: - How robust are the advances over the CLAM architecture (with glomeruli preselected)? 
- To which architectural contributions can these be attributed?
- Reasons why it is not possible to make the dataset available, or, better: a link to the publicly available dataset.

Comments: title: Reply to the rebuttal
Answer to the rebuttal
comment: Thank you for your detailed feedback and for highlighting opportunities for improvement in our paper. We appreciate the time you've invested in reviewing our work, and your insights have been invaluable in refining our study.

Initially, we were under the impression that revisions were to be submitted at the end of the discussion period, and were not aware that modifications could be made during the rebuttal phase as well. We apologize for this oversight. Following your suggestions, we have now revised our manuscript accordingly and believe these changes significantly enhance its quality and impact.

1. We have incorporated an additional ablation study that explores simpler aggregation methods, including mean and max pooling. These experiments have provided compelling evidence supporting our original hypothesis, confirming the superiority of our proposed method even when contrasted with these simpler approaches. This addition not only addresses your valid critique but also strengthens our argument by providing a more robust empirical foundation.

2. We've made a sample dataset accessible to all immediately after they create an account on the Data Foundation portal (https://datafoundation.iiit.ac.in/datasets), without the need for prior approvals. To access the full dataset, users simply need to request access post-sign-up, subject to agreement with the terms of use. We believe this strikes a balance between accessibility and responsible data management. Our dataset can be found under the "India Pathology Dataset (IPD)" section, with the name "Lupus Nephritis".

3. We have taken your feedback on the interpretability and explainability of our model, and we have decided to remove the interpretation section from our paper. This decision was made to avoid any misunderstanding and to ensure that our findings are presented as objectively as possible.

We hope that these revisions and explanations address your concerns adequately. We are eager to hear your thoughts on the updated manuscript and remain open to further suggestions. Our goal is to contribute meaningfully to our field, and your expert feedback is crucial in achieving this aim.
1. A fully supervised setting is an ideal situation. If this is expected, then there is no scope or need for MIL work. However, it can be seen that in the last few years, MIL has become increasingly relevant in the histopathology domain due to limitation in always obtaining pixel or patch-level annotation [ https://doi.org/10.1016/j.compmedimag.2024.102337 ]. In our work, we have showcased the application of MIL in renal histopathology, which earlier relied on obtaining annotation at the glomeruli level. Our idea is to see how MIL methods like CLAM work in this situation and propose a framework to improve it.  MIL framework is relevant since not all glomeruli contribute to the slide-level classification, and glomeruli classes are different than the LN subtype classes - which is usually not the case with other histopathology images where at least an instance (patch) belongs to the slide-level class directly [ https://doi.org/10.1016/j.eswa.2018.09.049 ].  \
Our solution is end-to-end without the need to do the intermediate task of predicting glomeruli labels (multiclass classification) and aggregating it for slide-level prediction. It can be argued that the obtained performance may not be sufficient for clinical diagnosis. An AUC of 91 and AAC of 81.11 in the LN subtype classification provide confidence for further improvements with more slide-level data. Further, this is the first work in the literature to do this classification, setting the benchmark for further improvement. \
Our work does not attempt to establish a one-to-one relationship between glomeruli classes and slide-level labels (LN subtype classes). However, the attention weights of each glomeruli can still be assessed to infer the contribution of each glomeruli in the final classification.

2. We beg to differ with this view that the dataset is of no use without instance labels. The generation of relevant medical datasets itself is of value given that they are not available in the first place. We have not seen any data on lupus available today of this size with various subtypes. Our work shows that it is of use by AUC of 91 and AAC of 81.11 in the LN subtype classification. We are only showing an alternate way to predict and are not claiming that it is superior to supervised learning.

3. In this work, we have chosen weakly-supervised learning since the glomeruli label is not available. It is not clear what the reviewer is suggesting to do under this setting to predict multi-class labels for evaluations. The attention map shows the relevant glomeruli, which are confirmed by pathologists to be well-known morphological features for that particular class. A supervised framework will be an idle solution, but we are dwelling on questioning how well MIL works in our data setting, benchmarking it, and improving it.

4. Please check out our code, which we have now made available on [GitHub](https://github.com/AmitSharma1127/LupusNet).
I appreciate all of the additional experiments and illustrations provided by the author during the rebuttal. Here are my comments on the rebuttal:

1. Motivation and Clinical Value: 

       Originally, I thought the proposed method could achieve instance-wise glom classification by only using slide-wise labels during training and evaluating the instance-wise glom classification performance in the testing stage. While several works have already addressed instance-wise glomeruli segmentation and classification, only providing the slide-wise label and training a model to give slide-wise predictions are insufficient and lack value for clinical research. It does not provide “precise clinical diagnosis” when the method only achieves slide-wise coarse prediction, reducing the precision of the prediction. The general application of MIL in WSI is due to the difficulty in obtaining pixel-level or patch-level annotations for disease diagnosis, while the target classification precision is also at the patient level or slide level (like CLAM). Even then, CLAM can also use attention scores to highlight disease regions on WSIs to evaluate the functionality and capability of their MIL model.  
       Therefore, not providing instance-wise classification on each glomerulus and evaluating the classification performance at the instance level further reduces the innovation and contribution of the proposed method. 

2. Clarification on Data Curation:

       As mentioned in #1, if the annotation is only slide-wise grading without instance-level labels, the dataset's contribution and clinical value are lacking, since previous studies have achieved glomerulus-wise annotation. On the other hand, all previous glomerulus-wise annotation datasets can directly generate slide-wise grading labels proposed in this paper.

3. Other MIL-based Methods Comparison:

       Since the dataset is very small and the prediction is at the slide level, the results may fluctuate between different epochs. Providing an instance-level prediction (by inputting only one glomerulus patch into the bag each time) can offer a more comprehensive evaluation of the models. Additionally, providing the results from a two-step glom-focus baseline method is necessary to understand the gap between supervised learning and weakly-supervised learning.

4. Pipeline Details:

       At the time of writing these comments, no code has been uploaded to GitHub, which remains a concern about the details of the code and replication.

5. Patch Size and Glomeruli Capture:

       I appreciate the clarification.

6. Class Distribution at Different Levels and Bag Size:

       I was thinking about the instance-wise (glomerular-wise) distribution of different classes and instance-wise labels in each bag.
I am a bit surprised to see that the authors opted to not revise their paper according to the comments. Please note that MIDL explicitly allows for revisions during the rebuttal phase.

**Regarding point 1:**

While I thank the authors for elaborating on their decisions, I uphold my main criticism, which is that the architecture is complex and we can’t clearly attribute parts of it to the success of the paper, due to the missing ablation study that also experiments with simpler aggregation methods. While I see the point that long-range dependencies are of importance and can be beneficial, this is for now just an assumption and not proven. Simpler aggregation mechanisms like majority voting or average pooling are surprisingly effective. To this end, it is especially unfortunate that the authors chose to run their ablation study always starting from the LSTM, and not also without the LSTM. Hence, the insights remain limited, unfortunately. 

**Regarding point 3:**

If the dataset is not made available with the publication on MIDL, it is not a contribution of the paper, and I would kindly ask the authors to not name this as such in the paper. It is, however, a pity, as the authors also point out comparison is hard without availability of the data.

**Regarding point 4:**

Better performance in one subclass does not make the model explainable or interpretable. I also still don’t understand what Figure 3 wants to convey, unfortunately. Furthermore, I think that an illustration based on a single, potentially cherry-picked, example is not really helpful to support the statements.
We would like to start by thanking you for your thoughtful feedback. We are encouraged to learn that you've found our work to be of significant importance. We would now like to address all your queries one by one.

1. **Choice of combining 2 long-range dependency modeling schemes and why a simple aggregation was not trialed:** \
The decision against a simpler aggregation method stems from our objective to address both the complexity and variability inherent in WSIs. While simpler aggregation methods can offer computational efficiency, they often fall short of capturing the detailed context and interdependencies between instances within a bag, which are crucial for the accurate classification of our task. \
The ablation study with LSTM was conducted to demonstrate the incremental value each component (LSTM and MHSA) adds to the model. This experiment underlines the LSTM's role in enhancing the model's ability to process sequential patterns, as well as MHSA's ability to emphasize the important ROIs, resulting in a more effective model.

2. **Cross-Validation of the data:** \
We have conducted all our experiments with 10-fold cross-validation, as mentioned in section 3.1. We would like to extend our sincere apologies for an oversight in our reporting which does not share the standard deviation. We acknowledge that this information is crucial for comprehensively understanding the variability and confidence in the performance metrics across different folds. The following are the results with respective mean and standard deviations for your reference. We will rectify this oversight in the next revision of our manuscript. 
Model | Input | Test AUC | Test F1 | Test ACC |
-|-|-|-|-|
ResNet-101 | GP | 52.88 ± 20.54 | 44.12 ± 23.01 | 53.23 ± 18.26
CLAM-SB | AP | 57.65 ± 18.00 | 52.22 ± 15.34 | 52.43 ± 11.26
CLAM-SB | GP  | 86.00 ± 14.78 | 72.80 ± 12.66 | 75.55 ± 10.48
**LupusNet** | **GP** | **91.00 ± 08.91** | **77.30 ± 06.80** | **81.11 ± 05.36**

3. **Dataset Availability** \
We appreciate your recommendation and understand the importance of open data access. We are currently working with our healthcare partners and the data foundation team to release the dataset in an appropriate manner, backed by the required infrastructure to host such a large image dataset. We will remove the phrase “upon reasonable request” in the next revision of our manuscript, as prompted by your reasoning. \
To access the dataset, users need to register on the portal and agree to the usage policy. There will be no preconditions or formalities except for a very minimalistic inquiry form, which is standard practice for online data repositories. This form will be designed to gather basic information about the researchers and their intended use of the data. This process ensures that the data is, in its true essence, open to use by anyone for academic purposes. By the time the portal gets ready, anyone interested can write to us to request access to the dataset.

4. **Interpretation Section:** \
The main advantage of our model lies in its enhanced ability to accurately identify the LN5 class, as mentioned in section 3.2 (by a significant F1-score improvement from 65.17% to 77.03%) which is characterized by uniform features across glomeruli. We observed that the gated attention mechanisms often fail to capture this uniformity as they only focus narrowly on glomeruli-level features. In contrast, our model employs a self-attention mechanism that assesses features across multiple glomeruli, hence improving the performance. Since the addition of branch 2 adds more information about different gloms, which branch 1 and 3 alone were not able to capture, hence this newly added information is responsible for improved performance and figure 3 is trying to convey the same. We can improve this explanation in our next revision, although we also understand the limited understanding and intuition the interpretation section conveys. We can remove this section if it promotes any confusion or offers less interpretability value to our overall claims and logic.

5. **Robustness of our model and the respective architectural contributions:** \
We have conducted all our experiments with 10-fold cross-validation, as mentioned in section 3.1. As there are no similar, openly available datasets, a direct measure of robustness is not possible.  We have also shared the standard deviations for all the results, which should further give insights into the model’s performance. \
We wanted to eliminate the 2-step process, which most of the previous works have adopted, a direct correlation was infeasible. The results, shared in section 4, reaffirm our understanding of the importance of multi-head attention and LSTM along with the already established gated attention and instance-level clustering. A combination of these 3 modules is the major contributor to the robustness and enhanced performance of the methodology as compared to the previous methods.
We would like to start by thanking you for your immensely thoughtful feedback. We are encouraged that you've found our work to be of significant importance and you've correctly highlighted the importance of this large dataset as well as the significance of our approach that is aimed at reducing the laborious human effort that goes on with the 2-stage approaches. We would now like to address all your queries one by one.

1. **Why did we combine LSTM with a transformer-style network?** \
We agree with the assessment that at the heart of transformers is the attention mechanism. The decision to combine LSTM + MHSA, however, was driven by a pragmatic assessment of our dataset's limitations, which is the small number of ROIs (glomeruli images), which are insufficient for data-hungry models like transformers. We experimented with a vanilla ViT alone, as well as employing ViT in place of the LSTM + MHSA block (branch 2) while keeping branch 1 (gated attention) and branch 3 (Instance level clustering) as they are. But, as shown in the results below, neither of these ideas worked for us. This is why we chose LSTM, which performed very well in solving our problem. We recognize the limitations of LSTM, and in our future work, we plan to explore better alternatives and design choices to enhance the performance even further.
| Model | AUC | F1 | Acc |
|-|-|-|-|
|Vanilla ViT (alone) | 67.00 ± 19.22 | 56.96 ± 23.58 | 62.22 ± 18.14 |
|ViT in place of LSTM + MHSA (Branch 2) | 73.00 ± 18.01 | 64.99 ± 13.76 | 66.67 ± 12.05 |
|**LupusNet** | **91.00 ± 08.91** | **77.30 ± 06.80** | **81.11 ± 05.36** |

2. **Choice of comparison models:** \
We are glad that you've found our comparisons and ablations to be useful. We also acknowledge the value of reporting the results on previous two-stage approaches for direct comparisons, however, since our curated dataset does not have any glomeruli-level labels, it was not possible for us to make this direct comparison. One of the primary motivations behind our work is to eliminate the current challenge of acquiring glomeruli-level annotations, a laborious task, thereby facilitating the construction of a model capable of achieving reliable performance utilizing only slide-level labels and eventually reducing the workload of clinicians. For detailed implementation, please check out our code which we have now made available on [Github](https://github.com/AmitSharma1127/LupusNet).

3. **Interpretation section** \
The main advantage of our model lies in its enhanced ability to accurately identify the LN5 class, as mentioned in section 3.2 (by a significant F1-score improvement from 65.17% to 77.03%) which is characterized by uniform features across glomeruli. We observed that the gated attention mechanisms often fail to capture this uniformity as they only focus narrowly on glomeruli-level features. In contrast, our model employs a self-attention mechanism that assesses features across multiple glomeruli, offering a more comprehensive analysis. Hence, other patches are required here. Since the addition of branch 2 adds more information about different gloms, which branches 1 and 3 alone were not able to capture, hence this newly added information is responsible for improved performance and Figure 3 tries to convey the same. We can improve this explanation in our next revision, although we also understand the limited understanding and intuition the interpretation section conveys. We can remove this section if it promotes any confusion or offers less interpretability value to our overall claims and logic.
We sincerely thank you for your feedback and review. We would now like to address each of your queries one by one.

1. **Motivation and Clinical Value**: \
We have mentioned our work's motivation and clinical relevance in Section 1's last three points, highlighting its importance for the Lupus Nephritis (LN) classification. Our methodology leverages only slide-level labels, avoiding the intensive task of glomeruli-level annotation. This strategy aids in efficiently classifying LN subtypes, potentially enabling quicker and more precise clinical diagnoses. Unlike previous studies that mainly depended on a two-step, glomeruli-focused annotation process, our work proves that effective LN subtype classification is possible using only slide-level labels. We believe this underlines the significance and clinical applicability of our research. We will add more clarity on this in the manuscript.

2. **Clarification on Data Curation:** \
It seems there was a misunderstanding regarding our data collection process. We did not curate instance-wise (glomeruli-level) annotations; we acquired slide-level labels only. This distinction is critical as obtaining instance-level annotations for whole slide images is prohibitively labor-intensive. Our approach addresses this challenge by leveraging weakly supervised learning to make efficient use of slide-level labels, a method already established and successful in the field. We will clarify this in our manuscript to prevent any further confusion.

3. **Other MIL-based methods comparison:** \
We acknowledge the importance of benchmarking against the latest advancements. However, it is important to note that there are no established baselines for this specific task. Our approach, incorporating gated and self-attention mechanisms, is designed to use the medical insights from Lupus Nephritis (LN) classes 4 and 5, expected to boost performance significantly. While we initially thought models like Vanilla ViT, DSMIL, and TransMIL might not compare well due to their attention mechanisms and transformers' limitations with limited data, we agree that including these comparisons will solidify our findings, prompted by your feedback. \
Below is a brief overview of the performance metrics comparing our method against other notable models, including a non-MIL-based model, Vanilla ViT (trained from scratch). We will add these results.  
| Model | AUC | F1 | Acc |
|-|-|-|-|
| Vanilla ViT | 67.00 ± 19.22  |56.96 ± 23.58  |62.22 ± 18.14 |
| DSMIL | 79.50 ± 16.35  |68.34 ± 17.98  |71.11 ± 16.08 |
| TransMIL | 54.50 ± 25.73  | 50.11 ± 24.17  |54.44 ± 22.08 |
| **LupusNet** | **91.00 ± 08.91**  | **77.30 ± 06.80**  | **81.11 ± 05.36**|


4. **More details of the pipeline are lacking, creating a barrier for pipeline replication**.  \
We value the feedback and acknowledge the need for transparency in replicating our work. Section 3.1 of our manuscript provides all essential details for reproduction and for detailed implementation, we’ll include the [Github repository link](https://github.com/AmitSharma1127/LupusNet) in the camera-ready version.  

5. **What is the size of the patches? How are glomeruli of different sizes captured in the same patch size?**  \
The glom detector uses 6000x6000 patches, while the MIL model's feature extractor processes 224x224x3 images, as outlined in sections 3.1 and 2.2, respectively. Images of glomeruli are resized to 224x224x3 for the extractor, which then produces a 1xd vector, with d being the feature dimension size, which is 2048 in our study. 

6. **What is the distribution of the two classes in the dataset at the patch level and slide level?** \
Section 2.1 details the dataset distribution, highlighting the class breakdown with 62 slides for LN4 and 53 for LN5. Our approach is independent of glomeruli count or type, as slide-level labels (LN4 or LN5) don't directly correspond to glomeruli-level distinctions (normal, sclerosed, hypercellular, etc.). A single WSI can contain various glomerulus types, hence the distribution of two classes at a patch level does not hold much value, which is why it's omitted. 

7. **What is the bag size? How does the bag size influence classification performance? How can it be ensured that the bag has sufficient glomeruli with the slide label?** \
Our method utilizes the features of multiple glomeruli to derive insights at the slide level, akin to how clinicians assess slides. In line with the Multiple Instance Learning (MIL) framework, the 'bag size'—the number of instances or glomeruli— isn’t a hyperparameter to play with. It should be noted that we haven't defined a strict minimum number of glomeruli that a WSI must contain for it to be processed. We extended the formulation of binary MIL to subtype MIL in which at least one glomerulus is mandatory for our analysis to proceed which contributes to a subtype class.