Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading
Authors: Qi Bi
Jingjun Yi
Hao Zheng
Wei Ji
Haolan Zhan
Yawen Huang
Yuexiang Li
Yefeng Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Medical Image Grading
Domain Generalization
Selective State Space
Gaussian Mixture Model
Primary area: machine_learning_for_healthcare
TL;DR: Propose a Severity-aware Recurrent Modeling, dubbed as Samba, for medical image grading problems on unseen target domains

Abstract: Disease grading is a crucial task in medical image analysis. Due to the continuous progression of diseases, i.e., the variability within the same level and the similarity between adjacent stages, accurate grading is highly challenging.
Furthermore, in real-world scenarios, models trained on limited source domain datasets should also be capable of handling data from unseen target domains.
Due to the cross-domain variants, the feature distribution between source and unseen target domains can be dramatically different, leading to a substantial  decrease in model performance.
To address these challenges in cross-domain disease grading, we propose a Severity-aware Recurrent Modeling (Samba) method in this paper.
As the core objective of most staging tasks is to identify the most severe lesions, which may only occupy a small portion of the image, we propose to encode image patches in a sequential and recurrent manner.
Specifically, a state space model is tailored to store and transport the severity information by hidden states.
Moreover, to mitigate the impact of cross-domain variants, an Expectation-Maximization (EM) based state recalibration mechanism is designed to map the patch embeddings into a more compact space.
We model the feature distributions of different lesions through the Gaussian Mixture Model (GMM) and reconstruct the intermediate features based on learnable severity bases.
Extensive experiments show the proposed Samba outperforms the VMamba baseline by an average accuracy of 23.5\%, 5.6\% and 4.1\% on the cross-domain grading of fatigue fracture, breast cancer and diabetic retinopathy, respectively. 
Source code is available at \url{https://github.com/BiQiWHU/Samba}.

Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading
Qi Bi Jingjun Yi?, Hao Zheng" 20-d Wei Ji Haolan Zhan', Yawen Huang?. Yuexiang Li Yefeng Zheng IN
Westlake University, China, 2Jarvis Research Center, Tencent Youtu Lab, China, Yale University, United States, *Monash University, Australia,
Guangxi Medical University, China
howzheng@tencent. com, yuexiang- www.iiieeee..... org zhengyefengOwestllkee edu. cn
Abstract
Disease grading is a crucial task in medical image analysis. Due to the continuous progression of diseases, i.e., the variability within the same level and the similarity between adjacent stages, accurate grading is highly challenging. Furthermore, in real-world scenarios, models trained on limited source domain datasets should also be capable of handling data from unseen target domains. Due to the rross-domain variants, the feature distribution between source and unseen target domains can be dramatically different, leading to substantial decrease in model performance. To address these challenges in cross-domain disease grading, we propose a Severity- aware Recurrent Modeling (Samba) method in this paper. As the core objective of most staging tasks is to identify the most severe lesions, which may only occupy a small portion of the image, propose to encode image patches in sequential and recurrent manner. Specifically, a state space model is tailored to store and transport the severity information by hidden states. Moreover, to mitigate the impact of cross-domain variants, an Expectation Expectation-Vaxximzztiiin (EM) based state recalibration mechanism is designed to map the patch embeddings into more compact space. We model the feature distributions of different lesions through the Gaussian Mixture Model (GMM) and reconstruct the intermediate features based on learnable severity bases. Extensive experiments show the proposed Samba outperforms the VMamba baseline by an average accuracy of 23.5%, 5.6% and 4. % on the cross-domain grading of fatigue fracture, breast cancer and diabetic retinopathy, respectively. Source code is available at wthhtps:/ htthtts://gittee //ggth//b.ttttttt www. Bii QHWHHWHHHUSSHtaS
1
Introduction
Disease grading aims to assess the severity level of disease pathological region from a medical image [46 31 52 50 6) It is more challenging than conventional deterministic classification with distinctive categories (e.g., cat vs. dog), owing to the inherent severity ambiguity within and between levels. This ambiguity arises because the progression of certain disease or pathological region is transitional, continuous and time-growing process (illustrated in Fig- [p)) On the one hand, different medical images within a same severity level can have rather different disease or pathological developments (shown in Fig. [p)) On the other hand, medical images among different severity levels can share similar patterns, as low-level lesions may persist throughout the discase's 's progression.
"Qi Bi is affiliated with University of Amsterdam. This research was conducted with Westlake University and Tencent Youtu Lab.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Diabetic Rotinopathy
Witii fithinllvvel Appearance D Discrepancy
Feature Space Illustrtion
Severe
Scharte
Setere
mild apdr
muderaate mpdr
severe updr
pdi
pdr
pdr
Breast Cancer
Cross- looss-level.o Appearance Similarity
mild
moderatt
pds
Trequed
O : mild npdr/ moderate npdr
A/O:
severe npdr/ pår
seciere
moderale
blue
/red
/prple
APTOS! DeepDR/ FCADR/ Messidor
(a) Continuous disease development
(b) Within-/cross--ene ambiguity (c) domain clustering challenge
Figure I: (a) The development of a disease or pathological region is continuous progress; (b) The continuous development apart from individual differences and style variation poses both within-level discrepancy (top) and cross-level similarity (bottom) on the medical image appearance; (c) These challenges can make the medical images from the same unseen domain, instead of those from the same grade level, to be clustered in the feature space.
The past decade has witnessed the rapid development of discase grading methods 330 34 4
owing
to the deep learning techniques (25] 24 28 26 57]. However, most of these methods were developed by experts from a specific clinic field (ophthalmology. gynecology, etc.). Furthermore, these models usually assume that the medical images used for training and inference are independently and identically distributed (i.i.d.). In practical clinical scenarios, grading model trained on number of medical images (source domain) is often required to handle images it has not encountered before (unseen target domains). Due to variations between patients, scanners, imaging parameters, clinic centers, etc., the feature distributions of the source domain and unseen target domains can be dramatically different [32] 68, 23 8). When the severity level of a disease is measured by the distribution of lesions, this cross-domase variance can lead the misdetection of crucial lesions, resulting in grading errors [4] [T]. Especially when the appearance of lesions is significantly affected by the style change, it may be observed that medical images from the same domain instead of from the same grade are clustered in the feature space (illustrated in Fig. Te). This suggests that the model has learned features with limited generalization ability.
Domain generalized disease grading learns models from only a source domain, but is expected to be applicable unseen target domains. The key to addressing this problem is accurately identifying the lesions that have decisive impact on grading 449 45). As the disease progresses, multiple lesions may coexist in the image, and the critical aspect of grading is identifying the most severe one among them. However, the most severe lesion may be localized in a small region in the image, exhibiting variable shapes, and being influenced by cross- challenges, this paper proposes a severity-aware recurrent modeling method (Samba). Samba encodes image patches in recurrent manner and recalibrates the state distributions based learnable bases.
style changes 4 MTT. To overcome these
In many disease grading scenarios, the decisive lesions only occupy a small portion of the total area. For instance, in retinal photographs, the affected blood vessels may only involve small section at the distal. Similarly, computed tomography (CT) or magnetic resonance imaging (MRI) seans, malignant tumors can also present as small lesions with a diameter less than 3 mm. These small lesions are casily influenced by style variations, which can lead to incorrect grading. Therefore, the model needs to pay sufficient attention to these detailed patches to classify them accurately. To address this issue, we treat the image patches as sequential data and encode them in a recurrent manner. This approach allows the information of decisive lesions to be stored in the hidden states and propagated to subsequent sequences. Furthermore, we adopt bidirectional encoding, enabling critical local information influence the overall representation. More specifically, we incorporate bidirectional Mamba [17] layer into the Samba, which supports -o-seuueee- transformation and efficiently selects data in an input-dependent manner.
The Mamba model achieves its selection mechanism by parameterizing the State Space Model (SSM) based on the input. While this selection mechanism [17 69 35) aids in identifying decisive lesions

---

and propagating critical information, these input-dependent parameters are also vulnerable to the influence of image style transformations. When the feature distribution is affected by cross-domain variations, both the update of hidden states and the gating mechanism are disrupted. To resolve this problem, we utilize learnable tokens to capture the lesion representations, which are then used as bases to map the feature embeddings into a more compact space. To preserve the semantic information within this process, we further employ the Expectation-Maxine by these bases to estimate the lesion feature distribution for each image and reconstruct the features accordingly. We refer to this process as EM-based state recalibration in this paper.
(EM) algorithm [14] initialized
Our contributions can be summarized as follows.
We develop a Severity- Recurrent Modeling, dubbed as Samba, for general disease grading within- and cross-domain medical images.
We propose encode the image patches in a recurrent manner accurately capture the decisive lesions and transport the critical information from local to global.
An EM-based state recalibration mechanism is designed to reduce the impacts of cross- domain variants by mapping the feature embeddings into a compact space.
Extensive experiments on three cross-domain disease grading benchmarks show the effec- tiveness of the Samba against the baseline.
2 Related Work
Domain generalization aims to learn model that can be generalized unseen target domains when only trained by the source domain, where the cross-domain feature distribution is usually not identical
65] A variety of machine learning techniques (e.g., discrepancy minimization 447 13 knowledge
ensemble [12]. uncertainty quantification
(39 53 optimal transport [16] 60) self-learning [51] (3).
frequency decoupling [59 g [0] and casual inference [37
38) have been proposed. In the medical imaging community, the effort of bridging the domain gap between training data and unseen inference data is so far mainly focused on medical image segmentation [32 68 23
g 58 and classification
[66] (54) These methods usually rely on either learning shape-invariat invariant representation or reaching pixel-wise consensus among the source domains. However, they are not especially devised to tackle the key challenge in cross-domain medical image grading, where the medical images from the same severity level instead of the same domain tend to cluster together.
Medical grading has also been studied. For Diabetic Retinopathy (DR) grading, many works
highlight the subtle local pathological regions to better discern different severity levels 30 34 44 6, 77. Similarly, grading models have also been developed for pulmonary nodules [46), fatigue fracture (3I. glioma
[52]. acne vulgaris [50], etc. However, most of the existing grading methods are task-specinf and assume the training and inference medical images are i.i.d., which is far from reality. Practically, a medical grading model is supposed to show reliable inference on unseen target domains that have different feature distribution from the source domain. To the best of our knowledge, only 4 and 11 made an initial investigation on learning domain generalized DR grading.
State Space Model (SSM) 27 contributes to a variety of fields such as robotics, navigation, and control theory, which is foundational scientific model. In the past few years, SSM has been adapted in the context of deep representation learning, and has shown great success in sequence modeling [19 20). More advanced SSM, exemplified by Mamba [17], not only shows stronger representation ability in long sequence modeling, but also exhibits linear scaling ability for long- sequence data. Built upon this, multiple Mamba variations (e.g., Vim (35 and VMamba [6 have shown effectiveness in the computer vision field. However, these methods mainly focus on enhancing the context representation from the image by exploiting the long-range dependencies. Instead, how to model the cross-level severity development from the medical image by SSM remains unexplored.
3 Methodology
3.1 Problem Definition & Framework Overview
For a given disease grading task, assume we have a number of medical images x and the corresponding severity-level labels y from K different domains, which is denoted D1 {(x11) , y |
3

---

point product point wise sum
stage 1
stage 2
slage 3
slage 4
level-1
Samba block
Samba block
Samba
Sambe
fn
state embooking estimeled state embedking tmining inference
block
block
level-2
Labels
fn
D
x2
x2
x9
x2
level-n
Sirget
Elddrrctional State Space Modeling
EMHhased State Recalibration
i
GMM
forward SSM
botrh nomm
layer
fireer layer
EMA
Innar loyer
severity base phe
( (1)
>
bockward SSM
E-Sepp
M-Stip
restimate state X
Figure 2: Framework of the proposed Severity-awan aware Recurrent Modeling (Samba) method. The patch embeddings pass through four encoding stages consisting of different number of severity-aware recurrent layers. Within each Samba block, the embeddings are first input to the bidirectional Mamba layers to store and transport the information about decisive lesions. After that, an EM-based state recalibration module models the feature distribution of lesions via a Gaussian Mixture Model with learnable severity bases. Morcover, the bases are re-estimated by the EM algorithm for each image and reconstruct the features finally.
(2) D2 ((xn)) Uh 1 DR = (<()) Un ) domain k. For the cross-domain disease grading problem, the objective is to learn a grading model Fe + y using images only from a source domain D1, which is supposed to generalize well on other unseen target domains D2, Dx. Following prior domain generalization works, each dataset regarded as a domain KR. as the samples in certain dataset are usually collected from the same clinical center by the same scanners and therefore share more similar feature distribution.
(K)
> Here N₂ denotes the number of images in
The overview of the proposed method is illustrated in Fig. The input image is first encoded into patch embeddings through a stem unit with 4 x 4 convolutional kernels, where the stem unit partitions the input image into patches. The patch embeddings further pass through four encoding stages. Each Samba block involves a certain number of severity-aware recurrent layers and there are downsampling layers between
consecutive blocks. Finally, a grading head consisting of an average pooling layer and linear layer generates the final prediction. Within the Samba block, the patch embeddings are first input to the bidirectional Mamba layers extract the information about decisive lesions. After that, EM-based state recalibration is applied to map the lesion representation into compact space by learnable bases.
3.2 Recurrent Patch Modeling by State Space Model
The core issue in most medical image disease grading scenarios is to identify the most severe lesion. However, due to the presence of lesions from different stages of the disease in the image, accurately capturing the most severe lesion is highly challenging. When the lesion occupies a large proportion of the image, the model only needs to extract stage-related features. In contrast, when the area of the critical lesion is small, the model needs to simultaneously locate the lesion and extract relevant features. This places higher demands the model's 's ability to handle local information. To address this issue, in this paper, we propose to encode the image patches in a recurrent manner. Specifically, the state space model is used process the sequential patch embeddings.
State Space Model. Let (t)) denote 1-D input signal. SSM maps it to the 1-D output signal y() by an intermediate N N-dimensional latent state u(t), given by
n' (1) Au(l) + (I)(), v(() = Cu(l)= + Dr((),
where E N denotes the state matrix, while E NXI C NN1 and E Nx1 denote the projection parameters. For deep sequential modeling, A, B. C and D are parameters that can be learned by gradient descent. The parameter D is omitted for exposition (i.e., D 0) as D=(1) can be regarded as skip connection and is easy to compute (19 20

---

Discretization. The structured state space 20 and Mamba [17] discretize the above continuous system so as be tailored for deep representation learning. There are usually two ways for dis- cretization, namely, linear recurrence and discrete convolution. For linear recurrence, instead of continuous function srt), a discrete sequence (zo, #1; ...)) ) is taken as input. Conceptually, we have a&k = .<(k). The state matrix A is approximated as A by the zero-order hold rule. The discrete SSM is sequencee-o--seuuence map Th: + yı, given by
uk = A + BTY:
A = eAA
(2)
yk Cuk:
B - AB,
C
Selective Scan Mechanism. Prior SSM methods usually focus the linear time-invariant scenario. Instead, the selective scan mechanism [17], which is the core of SSM operator in Mamba, learns the dynamism of weights from the input and is more aware of the context information.
The Mamba model is a suitable structure that aligns with our needs. When encoding the image patches as sequential data, once important lesion information is discovered, it can be stored in hidden states and propagated to subsequent sequences. Specifically, after sliding the image x E RR x3 into variety of patches, the input is formed as sequences of 2-D patches, each of which has spatial position of H/4 x W/4. Then, in each Samba block, the bi-directional state space modeling module has both feedforward and backforward SSM, where the selective scan mechanism allows to handle the patches in a recurrent manner. The input patches are traversed along two different scanning paths (horizontal and vertical), and each sequence is independently processed by the SSM. Subsequently, the results are merged to construct a 2D feature map as the final output.
By a bidirectional design, the severity information can be transported to each patch. The local-to- global transportation of severity information plays a vital role in the whole process, especially the selective mechanism. With the guidance of global severity awareness, the update of hidden states can selectively ignore information about low-level lesions, primarily preserving information about the most severe lesions. Specifically, to encode the 2D images, we follow the design of vision Mamba [69 which processes the input features in the forward and backward directions. As illustrated in Fig. the outputs are gated and added together, while there is a skip connection before input to the EMsed state recalibration module.
3.3
EM-based State Recalibration
Another core issue in cross-domain disease grading is the domain generalization ability of the model. Both the intermediate features and the input-dependent parameters in Mamba are affected by the cross-domain variance. To reduce the impact of domain shift, we aim to map the features into more compact space by feature recalibration. Specifically, the feature distribution of background and grading-related lesions is modeled by Gaussian Mixture Model (GMM) [2), given by
K
p(f)) = Σ Zn Ex),
where K is the total number of the Gaussian models, fr, is the feature embedding of the n-th patch in image æ, fa and Σ& denote the mean and covariance of the k-th Gaussian basis, respectively. For simplicity, we set Σg as the identity matrix J. For easy computation, the mixing coefficients of GMM are left out and the exponential inner dot kernel is used. After that, each Gaussian basis is represented by Me which is called severity base in this paper. These bases are learnable parameters capture the representation of lesions. In the recalibration process, instead of directly reconstructing the features based on the bases, we estimate the lesion distribution of each image which is initialized by the severity bases. This is to prevent the loss of useful information during the compression. Concretely, we adopt the EM algorithm to estimate the GMM of each image.
Within each iteration, we first estimate Znk in the E-step, which denotes the responsibility of the k-th basis to f,. Here we have < n < N and < < K. The posterior probability of f. given μ can be formulated as p(f((#)) = ICff.. n. (2)) by a kernel function K. Consequently, estimating the responsibility can be re- re-formulated into a more general form, given by
ΣK K(Ff.sH))
5

---

where for simplicity we directly use the exponential inner dot exp(f" μ4) as the kernel function.
Given the estimated Z' the severity base likelihood maximization, functioning the M-step, is realized by updating fa. As the bases are supposed to be aligned to the embedding space of each image, the weighted sum used update the bases, given by
2n n01
(5)
2 mu
mm=1
where refers to the t-th iteration and Np denotes the number of patch embeddings.
Assume that the E-step and M-step execute alternately for T. iterations and the convergence criterion has been reached [14], The final μ== and Z are used to recalibrate the image feature F, resulting in F. Here f { Te } and Z = {=ズØ) refer to the Gaussian basis and the responsibilities of all the patch embeddings from a sample, respectively. This process mathematically computed as
F Te.
(6)
Then, the recalibrated feature F is fed into the next Samba module. During this process, grading- related features mapped more compact space, while style differences introduced by image sources are partially removed. Consequently, the critical information transportation within the Mamba model can be more stable in unseen target domains.
To alleviate the potential unstable issue, moving averaging adapted update the bases 2t during the training process. After the T-th iteration, the generated ęe" is first averaged over mini-batch to get f²" Then, the update of yu" with momentum a E [0, 1] is given by
11. opp 0 + (1 - aa)
3.4 Theoretical Analysis
Consider the source domain D1 (( and certain unseen target domain Dj P(X (*)),
where l: = 2, ... K. Given a hypothesis h E H. according to the domain adaptation//eeerrlzzziii
bound theory 15 21, the relation between the target risk eD, (h) and the source risk eD, (h) can be modeled by a relation inequality, given by
CD. (h) S cD, (h) + dHAH (( +
min P(X))P(()))PPX))))
[lkD, (x) hDD (x)1),
where dHan (() P(((*)) denotes the distribution gap between the source domain and an unseen target domain, and the right-most term refers to minimal total risk over both domains. In other words, the risk of the proposed Samba on the target domain is bounded by the source domain.
4 Experiments
4.1 Datasets Evaluation Protocols
Cross-domain Fatigue Fracture Grading Benchmark [31 consists of a total number of 1,785 normal X-ray images and 940 X-ray images with fatigue fracture. They are collected from two hospitals with different types of sensors, which denote as Domain-1 and Domain-2, respectively. These fatigue fracture images were graded into four stages by three physicians according to the severity level. For simplicity, denote the grades (including the normal grade) from level-1 1 to level-5.
Cross-domain Breast Cancer Grading Benchmark consists of a total of 3644 H&E stained breast invasive ductal carcinoma pathological images from two domains ( The first domain contains 2,486 images under the 20x magnification (denoted Domain-1). The second domain contains 1,158 images under the 40x magnification (denoted as Domain-2). Different magnifications make the image appearance dramatically different. For each experiment setting, one is used the source
httrps://giihuu..co////A/// http://raast--cccoo-rrrrrraaaa

---

15
55
99
622
41
463
460
03.
527
267
249
187
340
24
22-
Reodio canber?
Protto Eetber
Protio DEnberT
116
10
70
56.
67.
ABC P5
25
76
Pepagazin
Prppggniin
Propggaiin
Figure 3: Impact of iteration number (top row) and severity base (bottom row) updating approaches on generalized medical grading performance. Evaluation metries AUC, ACC and F1 presented in percentage (%), from the first to the third column. Domain-l and Domain-2 in the Fatigue Fracture Grading Benchmark used as the source and unseen target domain, respectively.
domain and the other is used the unseen target domain. According to the severity of breast invasive ductal carcinoma, three grades, namely, rare, frequent and abundant, are annotated. For simplicity, we denote them from level-1 to level-3, respectively.
Cross-domain Diabetic Retinopathy Grading Benchmark consists of six DR retinal image datasets, namely, DeepDR (33), Messidor (1) IDRID [40, APTOS [3] FGADR [67]. and RLDR [49]. Following recent work [T]. the single-domain generalization protocol is adapted. Specifically, one of the above six datasets is used as the source domain, and all the rest datasets are used as unseen
target domains. Following [T], two extra large-scale datasets, DDR (2] and EyePACS [15] are used to enrich the source domain for each experiment setting. The development of DR is graded into five levels according to the severity, namely, normal, mild nonproliferative diabetic retinopathy (npdr), moderate npdr, severe npdr and pdr. For simplicity, we denote them from level-1 to level-5.
Under the single-domain generalization protocol, three most common evaluation metrics for grading are used, namely, area under the curve (AUC), accuracy (ACC), and F1-score (F1).
4.2 Results on Fatigue Fracture Grading Benchmark
We conduct extensive ablation experiments to study the impact of iteration number T and optimization of severity basis μ the unseen target domain. Images from the first clinical center (Domain-l) are used as the source domain, while images from the second clinical center (Domain-2) are used as the unseen target domain for testing. The vanilla Mamba 35 under the Empirical Risk Minimization (ERM) is the baseline.
Iteration Number T. In the EM algorithm, the iteration number T plays an important role, because implements the approximation by iteratively conduct the E and M step. Keeping other hyper- parameters and module designs the same, we report the results when the iteration number T of the EM algorithm varies from to 8. The top row of Fig. 3 shows how T impacts the AUC, ACC and FI -cooe on the unseen target domain. Notice that, the VMambaERRM bascline does not have EM-based state recalibration. Therefore, the performance T' does not reach the convergence criterion, and reduces the effectiveness of feature recalibration. Therefore, a clear performance decline on all the metrics is observed. On the other hand, when T too large, the representation ability saturates, resulting in little performance improvement, while wasting computation resources.
Vmamba-ERM is consistent to T. A too-small
Severity Base Update. We further study how different optimization approaches of the severity base H* impact the generalization performance on unseen target domain. We study three different settings. namely, no update, only back propogation, and moving average (Eq. 7.. The bottom row of Fig.
7

---

Table 1: Effectiveness of the proposed Samba on recurrent patch modeling, Domain-1 and Domain- in the Fatigue Fracture Grading Benchmark are used as the source and unseen target domain, re-
Table 2: Ablation study on each component. BSSM: Bi-directional State Space Modeling; ESR: EM-based State Recalibration. Experi- ments the Fatigue Fracture Grading Bench- mark. Domain-1 (x20)/Domain-2 (x40) is
spectively. Metries presented in percentage (%).
Method
1
A
F
used as source/target domain. Metries in per-
LSTM [22]
39.8
50.2
18.6
centage (%)
UR-LSTM [8
43.3
61.8
20.9
Components
Evaluation Metric
UR-GRU 18
45.7
65.1
22.4
VMamba BSSM
ESR
ACC
AUC
Fl
VIT 48
50.0
69.3
26.5
X
X
52.7
70.4
28.7
VMamha 69
52.7
70.4
28.7
X
57.9
72.1
33.6
Samba
76.2
81.5
45,8
76.2
81.5
45.8
Table 3: Category- performance and com- putational cost comparison between VMamba- ERM and the proposed Samba. Experiments conducted on the DG Breast Cancer Grading Benchmark. Domain-1 ( (x20)Domainn-2 (4)) is used source/target domain. Metrics in per-
Table 4: Impact of the number of components K
in GMM. Experiments are conducted on the DG Breast Cancer Grading Benchmark. Domain-1 (x 20)/Dmmai-22 (x40) is used as source/target
domain. Metrics presented in percentage (%).
K value
ACC
AUC 1
Fl F
centage (%).
16
58.6
70.0
56.0
LRM
Samba
Method
VMama T
Backbone
Compaation
37
55
Piza.
32.7
32.7
12.1
40.5
Diomain- as Soumae
1E5212
31.5
70.7
42,9
36.1
54.8
aeg.
2014
48
32
64
60.4
60.5
59.2
72.0
72.3
71.1
57.2
59.1
58.9
EXM
Samiea
VMani-S
11.
79
03.
63.4
47.1
26.7
00.5
71.5
30.1
43.7
30.1
56.1
96
60.4
72.2
58.8
LRR Sambu
VMama-R
100
19.6
112.4
112.4
27.
44.8
754
825
45.2
382
519
68.5
128
59.5
71.0
57.9
shows the results of the above three settings under a variety of iteration number T. Using moving average update the severity base ftg able to improve the performance substantially. It may be explained that the proposed state recalibration is differentiable, thereby enabling the application of back-propagation to update 140- However, the stability of the update cannot be guaranteed due the EM iterations. Moving average can update P4 to avoid collapse.
Effectiveness on Recurrent Patch Modeling. The proposed Samba realizes the recurrent patch modeling by harnessing the selective state space model. To demonstrate its effectiveness compared with other recurrent long-contex. based representation learning methods, we compare the proposed Samba with vanilla VMMmba [69]. Vision Transformer [48]. LSTM [22) and UR-LSTM (18). Table T shows that the proposed Samba has stronger generalization performance target domain, noticeably outperforming the second-best by 23.5%, 11.1 1.1% % and 17. 17.1% in terms of accuracy, AUC and Fi-score, respectively.
the unseen
Studies on Each Component. On top the VMamba baseline, two key components, namely, Bi-directional State Space Modeling (BSSM) and EM-based State Recalibration (ESR), are evaluated. The experiments are conducted on the DG Fatigue Fracture Grading Benchmark. Jomain-I/Domain-22 is used as the source/target domain, respectively. The results reported in Table 2] It is observed that BSSM contributes to an ACC, AUC and Fi improvement of 5.2%, 1.7% and 4.9%, respectively. ESR contributes to ACC, AUC and Fl improvement of 18.3%, 9.4% and 12.2%, respectively.
Ablation
4.3 Results on Breast Caneer Grading Benchmark
Grade-wise Improvement Analysis. We provide a break-down analysis on the grade-wise perfor- mance of the proposed Samba and the baseline, i.e., VMamba under the empirical risk minimization (ERM). Table 3 reports the performance. The proposed Samba shows a significant performance improvement on each grade level. Especially, on level-1, level-2 and level-3, the accuracy improve- ment over the ERM baseline 17.0%, 7.1% and 7.0%, respectively. Compared to VMamba-ERN baseline, the EM-based State Recalibration in Samba models the feature distribution of lesions via Gaussian Mixture Models with learnable severity bases, and re-estimates by E-M algorithm. The grading features are mapped
more compact space, and more stable unseen target domains.
Ablation Studies on the number of Gaussian components K. We study how the number of components K impacts the generalization ability the unseen target domain. By default K is set
8

---

to 64, and we further test the performance when K is set to 16, 32, 48 and 96, respectively. The results are reported in Table 4 When K is set to 64, the proposed Samba achieves the best grading performance, iee, 60.5%, 72.3% and 59. 1 1% in terms of ACC, AUC and F1-Score, respectively.
Scalability to Clinical Computational Pathology. The domain shift in the Cancer Grading Benchmark is from a machine learning perspective, and only handles the magnifica- tion difference. However, from clinical perspective, the computational pathology has to handle the domain shift from not only the magnification difference, but also the staining procedure. However, most existing clinical computational pathology datasets only support the classification task, i.e. separating tumor category from norall category, which is not strongly relevant to our grading task. Appendix [A.4] studies the performance of the proposed Samba along with the vanilla VMamba baseline on the CAMELYONI7 dataset I for clinical sanity check.
Breast
4.4 Results on Diabetic Retinopathy Grading Benchmark
Comparison with State-of-the-art. We compare the proposed Samba with methods from three primary categories: 1) generic domain generalization methods, including Mixup [61], MixStyle 64.. DDAIG [63] ATS [55]. Fishr 4T. and MDLT [56): 2) thee--of-the-art DR grading methods, which focus on DR grading without explicitly addressing domain generalization, including GREEN CABNet 21), Swin-Transformn
(3 and MiT 7 3) domain- domain--eneraaized DR grading methods, including DRGen [4] and GDRNet [II] Additionally, the vanilla Mamba (35) results under Empirical Risk Minimization (ERM) are provided as baseline reference. By default, the results are
cited directly from (II]
Table 5 presents comparison between Samba and existing methods within the context of single- domain generalization. Notably, for DG grading tasks, the metric of accuracy and Fi-score are more meaningful than AUC, as the AUC can be made artificially high due to the large amount of negative samples belonging to other stages. Therefore, we only involve ACC and FI for comparison.
The proposed Samba achieves substantial improvement over state-of-the-ard domain generalized DR grading methods. Especially, on APTOS, DeepDR, FGADR, Messidor and RLDR, it outperforms the second-best in terms of ACC and Fl by 5.2% and 2.2%, 27.2% and 5.7%, 60.7% and 31.3%, 6.7% and 1.3%, 28.3% and 4.7%, respectively. The significant improvement on FGDAR may be explained that it has different severity-level sample distribution than other datasets. The samples without DR (level-1) only occupy only 5.5% among all the training samples, which are far less than others (e.g., level-1 I samples occupy 49.3% in APTOS). Therefore, existing methods may overfit other severity levels and underfit level-1. In contrast, the selective scan mechanism of the Vmamba-ERM and Samba is robust to this severity distribution shift. The EM state re-calibration in Samba makes the feature space more compact, and improves the generalization.
Additionally, Samba shows marked improvement over the baseline VMmmba model under Empirical Risk Minimization. Especially, on APTOS, DeepDR, FGADR, IDRID, Messidor and RLDR, it outperforms the VMamba-ERM baseline in terms of ACC and F1 by 1.3% and 1. 1.7%, 2.2% and 2.1%, 3.0% and 1.6%, 3.7% and 2.6%, 7.3% and 2.7%, 7.4% and 3.4%, respectively. These results demonstrate its effectiveness in handling domain gap.
Understanding Recurrent Patch Modeling. We validate if the proposed recurrent patch modeling can store and transport the lesion information. An intuitive way is to inspect the correlation between the patch embeddings before and after the recurrent patch modeling. Therefore, we extract the patch embeddings before and after the fourth block. We compute the correlation matrix between the patch- patch-wise embeddings and visualize the results in Fig. 4 After processed by the Recurrent Patch Modeling module, more regions in the correlation matrix have higher responses. Specifically, after passing through certain high-response positions, the relevant information is transmitted to the subsequent patches in the forward direction. The high-response patches have grade-related lesions and the information is transported in the recurrent process.
1-SNE Visualization. To assess the generalization capacity of the proposed Samba, analyze the feature distribution across the source and unseen target domains using t-SNE visualization in Fig. 5 which compares the t-SNE plots of the ERM baseline (left) and the proposed Samba (right). The results indicate that, after the EM-based state feature calibration, the proposed Samba enables
https://camelyyonl7.uu wtw.:lllyyyooo...ddddd. challenge........ org/

---

Table 5: Performance comparison of the proposed Samba and existing domain generalized DR grading methods under the single-domain generalization protocol. Evaluation metrics include ACC and F1
(in percentage %). Top three results are highlighted as best second and third respectively.
Method
APTOS
DeepDR
FGADR
IDRID
Average
ACCT
FT7
ACC1
FI
ACC
FT
ACC1
FT
ACCT
FIT
ACCT
FIT
ACC1
FIT
RewNer-30 baxed:
Mixup
6
49.4
30.2
49.7
33.3
5.8
7.4
64.0
32.6
63.0
32.6
21.7
27.0
43.3
27.2
MixStyle
64
48.8
25.0
32.0
14.6
7.0
7.9
53.5
19.4
57.6
16.8
18.3
6.4
36.2
15.0
GREEN 34
52.6
33.3
44.6
31.1
5.7
6.9
60.7
33.0
54.5
33.1
31.9
27.8
41.7
27.5
CABNet
52.2
30.8
55.4
32.0
6.1
7.5
62.7
31.7
63.8
35.3
23.0
25,4
43.8
27.2
DDAIG
63
48.7
31.6
38.5
29.7
5.0
5.5
60.2
33.4
69.1
35.6
25.4
23.5
41.2
26.7
ATS 55
51.7
32.4
52.4
33.5
5.3
5.7
66.6
30.6
64.8
32.4
24.2
23.9
44.2
26.4
Fishr
41
61.7
31.0
61.0
30.1
6.0
7.2
48.0
30.6
52.0
33.8
19.3
21.3
41.3
25.7
MDI.T 56
53.3
32.4
50.2
33.7
7.1
7.8
61.7
32.4
58.9
34.1
29.0
30.0
43.4
28.4
DRGen
60.7
35.7
39.4
31.6
6.8
8.4
67.7
30.6
64.5
37.4
19.0
21.2
43.0
27.5
GDRNet
52.8
35.2
40.0
35.0
7.5
9.2
70.0
35.1
65.7
40.5
44.3
37.9
46.7
32.2
W7 based:
MIL-ViT 7
61.8
36.8
38.2
36.3
8.7
9.3
68.6
31.1
67.7
40.7
28.1
34,5
45.5
31.5
Swin-T E6
36.7
VMambu bused:
36.2
65.0
65.2
38.9
65.2
39.1
65.1
39.2
65.1
Samba (Ours)
65.9
37.9
67.2
40.7
68.2
40.5
68.9
41.7
72.4
41.8
72.6
42.6
69.2
40.9
Before
After
Raseline
Ours
gmeen) villon jaln troppppple tornal rild spdr moderate rpdr severn npdr 'prr 1010 ai 01 of Awt.AAPDDDDDD/O.AAA.. (DW.G/DDDDDDDDDD/Daaaaattaee
Figure 4: The correlation matrix of each patch
Figure 5: T-SNE visualization of the feature space from the ERM baseline (left), and the proposed Samba (right). APTOS is chosen the source domain and the rest datasets are used for as target domains.
embedding before and after processed by the current patch modeling in forward direction, denoted 'Before' and 'After' respectively. The higher correlation, the more red a cell is.
feature embeddings from different domains to achieve a more uniform distribution, thereby reducing the domain gap. This improved uniformity in the feature space suggests that Samba enhance generalization, contributing to better performance on unseen domains.
Conclusion
In this paper, aimed to tackle practical but challenging task, learning domain generalized medical image grading. We mainly focused on two issues: the identification decisive lesions and the impact caused by inter-domain differences. The proposed severity aware recurrent modeling adopts state space model to store and transport the severity information from local to global. To further mitigate the impact of cross-domain variants, EM--ased state recalibration was designed map the patch embeddings into a compact space. The proposed method can be used in a variety of disease grading scenarios, providing effective tool for automatic medical image analysis.
Limitation Discussion Broader Societal Impact. The feature distribution lesions modeled by the Gaussian mixture model and estimated by the Expectation -aaittiiin-aiiiizaiiiionn algorithm. However, when the training source domain has severe class imbalance, the estimated probability distribution by the proposed Samba may not necessarily reflect the domain-agnostic lesion distribution. Nevertheless, the proposed method can be combined with other techniques specifically designed for addressing class imbalance. The proposed method advances the versatility of automatic disease diagnosis, which benefits the human well beings. We do not envision negative societal impact.
Acknowledgments and Disclosure of Funding, This work was supported by the Science and Technology Major Project of Guangxi (AA22096030 and A A22096032), and National Key R&D Program of China under Grant (2000AAA0109500 and 2020AAA0009500),
10

---

References
[1] Michael D. Abråmoff, Yiyue Lou, Ali Erginay, and et al. Improved automated detection of diabetic retinopa- thy on a publicly available datasct through integration of deep learning. Investigative Ophthalmology Visual Science, 57(13):520-5526 2016.
[2] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiaoo Falk, and Ioannis Mitliagkas. Gener- alizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00004. 2019.
[3] APTOS. APTOS 2019 blindness detection, 2019.
URL https://www. www..a.c..........e.oooooooo../e aptos2019 blindness- deteecttonntnnnnnn Accessed on February 20, 2024.
[4] Mostafa Atwany and Muhammad Yaqub, DRGen: Domain generalization in diabetic retinopathy classifi-
cation. In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 635-644, Springer, 2022.
[5] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains, Machine Learning, 79:151-1777 2010.
[6] Qi Bi, Shuang Yu, Wei Ji. Cheng Bian, Lijun Gong, Hanruo Liu, Kai Ma, and Yefeng Zheng. Local-global dual perception based deep multiple instance learning for retinal discase classification. In Medical Image Computing and Computer Assisted Intervention, pages 55 64, 2021.
[7] Qi Bi. Xu Sun, Shuang Yu, Kai Ma, Cheng Bian, Munan Ning, Nanjun He, Yawen Huang, Yuexiang Li. Hanruo Liu, et al. ViT: A multiple instance vision Transformer for fundus image classification. Journal of Visual Communication and Image Representation, 97:103956, 2023.
[8] Qi Bi, Jingjun Yi, Hao Zheng, Wei Ji, Yawen Huang. Yuexiang Li, and Yefeng Zheng. Learning generalized medical image segmentation from decoupled feature queries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 210-818, 2024.
[9] Qi Bi, Shaodi You, and Theo Gevers. Learning generalized segmentation for foggy-scenes by bi-directional wavelet guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 801-809, 2024.
[10] Qi Bi, Shaodi You, and Theo Gevers. Generalized foggy-scene semantic segmentation by frequency decoupling. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-1399, 2024.
[1I] Haoxuan Che, Yuhan Cheng, Haibo Jin, and Hao Chen. Towards generalizable diabetic retinopathy grading in unseen domains. In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 430-440. Springer, 2023.
[12] Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi- Ming Ma. Explore and exploit the diverse knowledge in model zoo domain generalization. In International Conference on Machine Learning, pages 4623-4640, 2023.
[13] Rui Dai, Yonggang Zhang, Zhen Fang, Bo Han, and Xinmei Tian. Moderately distributional exploration for domain generalization. In International Conference on Machine Learning, pages 6786-681 2023.
[14] Arthur P Dempster, Nan Laird, and Donald B Rubin. Maximum likelihood from incomplete data the EM algorithm. Journal of the Royal Statistical Society: Series B (Meehoabllggcaa)) 39(1)1112 1977.
[15] EyePACS. Kaggle EyePACS dataset, Accessed 20 February 2024. URL httpps://papersutthcodd com/ http///aattttaagleeeeee eyepacs
[16] Milena Gazdieva, Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Extremal domain translation with neural optimal transport. Advances in Neural Information Processing Systems, 36, 2023.
[17] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0 00752, 2023.
[18] Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffmann and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Learning, pages 3800-3809, 2020.
[19] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous saatinuouu--imeeeee models with linear state space layers. Advances in Neural Information Processing Systems, 14:572-5855, 2021.

---

[20] Albert Gu. Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Represenlations, 2022.
[21] An He, Tianyi Li, Ning Li, Ke Wang. and Huazhu Fu. CABNet:
Category attention block for imbalanced diabetic retinopathy grading. IEEE Transactions on Medical Imaging, 40(1):143-153, 2020.
[22] Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural Computation, 983):1735-1780, 1997.
[23] Shishuai Hu, Zehui Liao, Jianpeng Zhang, and Yong Xia. Domain and content adaptive convolution based multi-source domain generalization for medical image segmentation. IEEE Transactions on Medical Imaging, 42(1):233- 244, 2023.
[24] Wei Ji, Jingjing Li, Shuang Yu, Miao Zhang, Yongri Piao. Shunyu Yao. Qi Bi, Kai Ma, Yefeng Zheng. Huchuan Lu, et al. Calibrated rgb-d salient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9471-9488 2021.
[25] Wei Ji, Shuang Yu, Junde Wu, Kai Ma, Cheng Bian, Qi Bi, Jingjing Li, Hanruo Liu, Li Cheng, and Yefeng Zheng. Learning calibrated medical image segmentation via multi- agreement modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12341- 12351, 2021.
[26] Wei Ji, Jingjing Li, Qi Bi, Tingwei Liu, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of sam on different real-world applications. Machine Intelligence Research, 21:617-660. 2024.
[27] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
[28] Jingjing Li, Wei Ji, Qi Bi, Cheng Yan, Miao Zhang, Yongri Piao, Huchuan Lu, et al. Joint semantic mining for weakly supervised rgh-d salient object detection. Advances in Neural Information Processing Systems, 34:11945-11959 2021.
[29] Tianyi Li, Yang Gao, Kai Wang. Shuai Guo, Hong Liu, and Hong Kang. Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences, 501:511-522, 2019.
[30] Xiaomeng Li. Xiaowei Hu, Lequan Yu, Lei Zhu, Chi- Wing Fu, and Pheng- Ann Heng. CANet: cross-diseas attention network for joint diabetic retinopathy and diahetic macular edema grading. IEEE Transactions on Medical Imaging, 39(5):1483-149) 2019.
[31] Yucxiang Li, Yanping Wang, Guang Lin, Yawen Huang, Jingxin Liu, Yi Lin, Dong Wei, Qirui Zhang, Kai Ma, Zhiqiang Zhang, et al. Triplet- branch network with contrastive prior-knowledge embedding for discase grading, Artificial Intelligence in Medicine, page 102801, 2024.
[32] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Am Ann Heng. FedDG: Federated domain generaliza- tion on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1013-1023, 2021.
[33] Rui Liu, Xintao Wang, Qi Wu, Lin Dai, Xiaohui Fang, and et al. DeepDRID: Diabetic retinopathy-graaleee and image quality estimation challenge. Patterns, 300512, 2022.
[34] Shujian Liu, Libui Gong, Kai Ma, and Yalin Zheng. GREEN: A graph residual re-ranking network for grading diabetic retinopathy. In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 585-594, 2020.
[35] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166 2024.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference computer vision, pages 10012-10022, 2021.
[37] Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain gencralization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8046-8056, 2022.
[38] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pages 7313-7324, 2021.
12

---

[39] Xi Peng. Fengchun Qiao, and Long Zhao. Out-of-domain generalization from single source: An uncertainty quantification approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 (3):1775-1787, 2022.
[40] Piyush Porwal, Sanket Pachade, Rakesh Kamble, Milind Kokare, Ganesh Deshmukh, and et al. Indian diabetic retinopathy image dataset (IDRiD); A datahase for diabetic retinopathy screening research. Data, 3(3):25, 2018.
[41] Ali Rame, Cedric Dancette, and Matthicu Cord. FishR: Invariant gradient variances for out-of titto---ditributionn gencralization. In International Conference on Machine Learning, pages 18347-18377, 2022.
[42] Sylvia Richardson and Peter J Green, On Bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Stotistical Society Series B: Statistical Methodology, 59(4):731-792, 1997.
[43] Lianghe Shi and Weiwei Liu. Adversarial self- saff-training improves robustness and generalization for gradual domain adaptation. Advances in Neural Information Processing Systems, 36, 2023.
[44] Ruoxian Song. Peng Cao, Jinzhu Yang, Dazhe Zhao, and Osmar Zaiane. A domain adaptation multi- instance learning for diabetic retinopathy grading on retinal images. In IEEE International Conference on Bioinformatics and Biomedicine, pages 743-750, 2020.
[45] Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, and Yongdong Zhang. Lesion-aware Transformers for diahetic retinopathy grading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10938-10947, 2021.
[46] Wenqiang Tang, Zhouwang Yang, and Yanzhi Song, Disease-gradee grading networks with ordinal regularization for medical imaging. Neurocomputing. 545: 126245, 2023.
[47] Peifeng Tong, Wu Su, He Li, Jialin Ding, 7han Haoxiang, and Song Xi Chen. Distribution free domain generalization. In International Conference on Machine Learning, pages 34369-34378, 2023.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 30, 2017.
[49] Qiang Wei, Xiang Li, Weiyang Yu, Xiang Zhang, Yu Zhang, Bin Hu, Biao Mo, Dawei Gong, Nan Chen, Dian Ding, and et al. Learn to segment retinal lesions and beyond. In International Conference on Pattern Recognition, pages 7403-7410. IEEE, 2021.
[50] Xiaoping Wu, Ni Wen, Jie Liang, Yu- Kun Lai, Dongyu She, Ming Ming Cheng, and Jufeng Yang. Joint acne image grading and counting via label distribution learning. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 10642-10651, 2019.
[51] Zhenbang Wu, Huaxiu Yao, David Liebovitz, and Jimeng Sun. An iterative self-learning framework for medical domain generalization. Advances in Neural Information Processing Syxtems, 36, 2023.
[52] Angi Xiao, Biluo Shen, Xiaojing Shi, Zhe Zhang, Zeyu Zhang, Jie Tian, Nan Ji, and Zhenhua Hu. Intraoperative glioma grading using neural architecture search and multi-modal imaging. IEFE Transactions 2022.
on Medical Imaging, 41 (100
[53] Mixue Xie, Shuang Li, Rui Zhang, and Chi Harold Liu. Dirichlet-based uncertainty calibration for active domain adaptation. In International Conference on Learning Representations, 2023.
[54] Geng-Xin Xu, Chen Liu, Jun Liu, Zhongxiang Ding, Feng Shi, Man Guo, Wei Zhao, Xiaoming Li, Ying Wei, Yaozong Gao, et al. Cross-site severity assessment of COVID-19 from CT images via domain adaptation. IEFE Transactions on Medical Imaging, 41(1):88-102, 2021.
[55] Fan-E Yang. Yu-Chiang Cheng, Zhan-Yuan Shiau, and Yu-Chiang Frank Wang. Adversarial teacher-student representation learning for domain generalization. Advances in Neural Information Processing Systems, volume 34, pages 19448 19460, 2021.
[56] Yue Yang, Hao Wang, and Dina Katabi. On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. In European Conference on Computer Vision, pages 57-75. Springer, 2022.
[57] Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang. Temporal cue guided video highlight detection with low-rank audio-visur fusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7950-7959, 2021.
13

---

[58] Jingjun Yi, Qi Bi. Hao Zheng. Haolan Zhan, Wei Ji. Yawen Huang, Shaoxin Li. Yuexiang Li, Yefeng Zheng, and Feiyue Huang. Hallucinated style distillation for single domain generalization in medical image segmentation. Internationa Conference on Medical Image Computing and Computer-Assisted Intervention, pages 438 448, 2024.
[59] Jingjun Yi, Qi Bi, Hao Zheng, Haolan Zhan. Wei Ji, Yawen Huang, Yuexiang Li, and Yefeng Zheng. tokens for domain generalized semantic segmentation. ACM Multimedia
Learning pneetral-decomperr 2024, 2024.
[60] Zhongqi Yue, Qianru Sun, and Hanwang Zhang. Make the in UDA matter: Invariant consistency learning for unsupervised domain adaptation. Advances in Neural Information Processing Systems, 36, 2023.
[61] Hongyi Zhang, Moustapha Cisse. Yann N. Dauphin, and David Lapez-Paz. Mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[62] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and paitern recognition, pages 2921-2922 2016.
[63] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain adversarall image generation for domain generalisation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 13025 13032, 2020.
[64] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with MixStyle. In International Conference on Learning Representations, 2021.
[65] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(4):4396 4415, 2022.
[66] Yi Zhou, Lei Huang, Tao Zhou, and Ling Shao. CCT-net: category-invarian cross-domain transfer for medical single-to- multiple disease diagnosis. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8260 8270, 2021.
[67] Yufan Zhou, Bin Wang, Lei Huang, Shuguang Cui, and Ling Shao. benchmark for studying diabetic retinopathy: Segmentation, grading, and transfcrability. IEEE Transactions on Medical Imaging. 40(3): 818-822 2020.
[68] Ziqi Zhou, Lei Qi, and Yinghuan Shi. Generalizable medical image segmentation via random amplitude mixup and domain-specific image restoration. In European Conference on Computer Vision, pages 420-436, 2022.
[69] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation lcarning with bidirectional state space model. arXiv preprint MoXiv2401.009417. 2024.
14

---

A Appendix supplemental material
A.1 Severity Base Normalization
During the iteration steps f = 1...... .T, the severity base Pa may not deviate too much from each other, which otherwise can lead to collapse when back propagation. We study the scenarios when no normalization, L-1 normalization and L-2 normalization are used on these severity basis. Fig. shows the results of the above three settings when under variety of iteration number T on the unseen target domain. L-2 normalization achieves the best performance on all the metrics, especially when T becomes larger.
(1)
462
43-
04
683
41.5
422
400
750
76-
12.8ia (tums)
Li-Nax
Ni (Dar))
LL-Narx
122hkam (Dun)
Li-Nar
777
No-
Ne-om
No Norr
Eanaian nanber
bortion namber
boaaaan camber
Figure 6: Impact of severity base normalization on generalized medical grading performance, Evalu- ation metries AUC, ACC and Fl are presented in percentage (%). Domain-1 I and Domain-2 in the Fatigue Fracture Grading Benchmark are used as the source and unseen target domain, respectively. Better zoom in to view.
A.2
Understanding Recurrent Patch Modeling
Fig. 4 in the main text only visualizes the correlation between the patch embeddings before and after the recurrent patch modeling when trained on the APTOS dataset. Here we further demonstrate results from FGADR, IDRID, Messidor and RLDR. They are visualized in Fig. Za. b, e and d, respectively. On all these datasets. can observe common pattem. After processed by the Recurrent Patch Modeling module, more cells in the correlation matrix have higher response. Usually, handful of the patches inside the image have grade-related lesions. After the processing of our module, the information of these grade-related related lesions is transported to other patches. allows the model to perceive more global-wise representation. Consequently, grade-related lesion information are activated, and more cells are highly responded in the correlation matrix.
patches that contain the
A.3 Visualize and Understand the Severity Level
We model the relation between patch embedding from SSM and severity level by drawing inspiration from the class activation map (CAM) mechanism 62). Specifically, we take the patch embeddings from the last Samba block as input to generate the per-level severity activation patterns. Then, the activated severity patterns are displayed
the original images. We use FGADR the unseen target domain. The results are shown Fig. S where the activated patches are highlighted in blue boxes. From the first the fifth row, the samples from level-1 to level-5 are provided accordingly. From the first to the fifth column, the patch activation maps from level-1 to level-5 are displayed. Notice that, level-1 refers to the normal scenario, each sample has activations on level-1, meaning some patches are normal.
A.4 Application Computational Pathology Classification
It is important to note that the domain shift in the Breast Cancer Grading Benchmark is technically from machine learning perspective, and only handles the domain shift from the magnification difference. However, from clinical perspective, the computational pathology has to handle the domain shift not only from the magnification difference, but also from the staining procedure. Therefore, it is beneficial to test if the hypothesis works in real-world computational pathology scenario. However, a bottleneck that, most clinical computational pathology dataset so far conducts the classification task, i.e., separating tumor category from normal category. Therefore, in this
15

---

Before
Aftar
Before
Aftar
(a) RGADR
(b) IDRI
Before
After
Before
Aftar
Messidor
(d) RLDR
Figure 7: The correlation matrix of each patch embedding before and after processed by the recurrent patch modeling in the forward direction, denoted 'Before' and Affter, respectively. The higher correlation, the more red a cell is.
Table Impact the number of components K GMM on tumor classification performance from unseen target domains. Experiments are conducted the CAMELYONIZ, Domain-1 I is used as source domain. The rest four are used as unseen target domains. Metrics presented percentage (%)
Domain-2
Domain-3
Domain-4
Domain-5
K value
ACC
AUC
F1
ACC
AUC
F1
ACC
AUC
Fl
ACC
AUC
Fl
16
81.80
92.09
79.96
79.45
88.96
76.60
83,92
95.38
82.10
75.84
81.99
74.16
32
82.68
93.81
80.95
80.39
91.28
77.97
84.38
96.20
82.96
76.53
82.46
74.92
48
84.06
94.25
81.83
81.84
92.60
78.49
85.87
97.16
83.87
77.69
83.85
75.04
64
84.59
95.67
83.10
82.48
93.32
79.85
86.50
97.82
85.13
78.64
84.70
75.32
96
84,27
95.48
82.53
82.06
92.90
79.44
86.16
97.45
84,80
78.
84.28
74,86
128
83.65
94.70
81.97
81.50
92.41
78.36
85.47
96.90
84.62
77.38
83.66
74.14
subsection, the proposed Samba along with the vanilla VMamba baseline are benchmarked on the CAMELYON17 dataset for the cross-domain computational pathology classification task.
The first experiment is the impact of the number of components K in GMM, where ACC, AUC and F1 are used evaluation metric. The results are reported Table T By default K is set to 64, and we further test the performance when K is set to 16, 32, 48 and 96, respectively. When K is set to 64, the proposed Samba achieves the best grading performance. This observation is consistent performance on Cross-domain Breast Cancer Grading Benchmark, where number of 64 Gaussians achieves the optimal performance.
The second experiment is to analyze the trade off between the classification performance and the baseline model. Both the VMamba-ERM and the proposed Samba are tested, where only accuracy used as the evaluation metric. The results are reported in Table 7 The trend is the same as the trend on the Breast Cancer Grading Benchmark. Using Samba on each type of the VMmmba backbone shows a clear performance improvement on unseen domains.
A.5 Attention Maps on Unseen Domains by Samba
Fig. 9 and Fig. I0 show some attention maps the Samba on unseen retinal images. The proposed Samba is able to model the recurrent relation among patches. Therefore, the activation regions can general cover the lesions and more robust the domain shift.
hattpe://camelyon11..e wwwp:/yyoooo....ddddo. challenge........ .rg/
16

---

Level-1
Level-2
Level-3
Level-4
Level-5
Level-1
Level-2
Level-3
Level-4
Level-5
Figure 8: Per-severity activation map of the proposed Samba. From the first to fifth column are the activated patches from level-1 level-5, highlighted in blue boxes. From the first to the fifth the samples with an annotation from level-1 to level-5. FGADR is the unseen target domain.
Table Classification performance comparison between VMamba-ERM and the proposed Samba. Experiments are conducted on the CAMELYONI7 dataset for cross-domain tumor classification. Domain- is used the source domain, while the four used as unseen target domains. Metries in percentage (%).
Domain- Source
Method
Backbone
Domain-2
Domain-3
Domain-4
Domain-5
avg.
ERM
70.08
67.29
72.96
63.16
68.37
VMama-T
Samba
78.74
76.15
80.06
71.05
76.50
ERM
72.86
69.50
75.08
65.72
70.79
VMama-S
Samba
81.01
78.96
82.75
73.88
79.15
ERM
76.23
74.17
79.53
69.87
74.95
VMama-B
Samba
84.59
82.48
86.50
78.64
83.05

---

Figure 9: Attention maps of the proposed Samba on retinal images from unseen domains.
Figure 10: Attention maps of the proposed Samba on retinal images from unseen domains.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper propose Severity-aware Recurrent Modeling (Samba) for medical image grading problems on unseen target domains.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: At the end of the conclusion section, the limitation of the proposed method has been discussed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification:
The theory assumptions are in the methodology section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The model realization and implementation details are provided in the experi- mental section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The datasets are public. The source code will be publicly available up on publication.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppblligggddpeeeeeeeeeee Coaee//ddeeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings and details are provided in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The evaluation protocols of these grading datasets do not require report the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computation resources and details are discussed in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors have read the code of ethics. The experiments are all on public datasets without obeying the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
At the end of the conclusion section, the broader impacts have been discussed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The asserts used in this paper are all public available for academic researches.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "Ic1O3Ssh4l", "reviewer": "Reviewer_Wyiu", "summary": "The authors propose a model which they call Samba, for Severity-aware recurrent modelling, which is a method designed for cross-domain medical image grading. They introduce several challenges in medical grading, namely the difficulty models encounter in generalising to unseen domains, as well as the existence of ambiguity in lesion severity grading. The model is comprised of two main blocks: recurrent bidirectional Vision Mamba layers and a Expectation-Maximisation State Recalibration (EMSB) module which consists of learnable tokens which capture lesion representations, and which are then used as bases to map to more compact feature embeddings. These are then used to initialise an Expectation-Maximisation (EM) algorithm and the lesion feature distribution is estimated using Gaussian Mixture Models (GMMs) for each image. The Mamba layers treat the image patches as sequential data, with the rational being the relevant information will be propagated through the hidden states. The EM module models the feature distribution of lesions using the GMMs to try and eliminate domain shift. Samba is applied to three benchmarks, where images are separated between a source domain and a target domain: Diabetic Retinopathy fundus images (DR), Fatigue Fracture X-rays and Breast Cancer histopathology images. The model is trained on the source domain and applied to the target domains. Ablation is carried out, comparing Samba to different baseline and SOTA models, as well as looking at iteration number and severity base update methods. The results obtained show Samba generally outperforms existing methods to some degree. The authors also provide a theoretical analysis on the generalisation risk bound in the Appendix.", "questions": "Line 44 - I don't fully understand this sentence. Maybe you could rephrase?\n\nLine 54 - at the distal of what? \n\nLine 123 - what's a stem unit?\n\nFigure 2 - I think you need to describe the Samba blocks in more detail - here it looks like the bi-directional SSM and EM recalibration occurs in the downsampling block. Is this a mistake? What'a VSS block? \nWhat's the difference between the training and inference arrows in the figure?\n\nFigure 3 - Iteration Number T - What models are you comparing yours against? What is Vmamba-ERM and what is SOSS?? Why is Vmamba showing a constant trend across iteration number T? What are the different columns? \n\nLine 235 - why does the number of iterations play an important role in the EM algorithm? You should introduce this beforehand and the ablation study should answer this question.\n\nLine 237 - do you mean Figure 3 instead of Figure 6 here? - Figure 6 shows the effect of regularisation techniques and is situated in the appendix. \n\nLine 244 - why are these these methods appropriate to test different optimisation techniques here? Please explain this. \n\nLine 252 - these are percentage points, not percentage of improvement. \n\nTable 1- Why is Table 1 showing three F1 scores highlighted in brighter green? and two blocks of yellow and light green? Is this a mistake? Also what do these colours represent? \n\nTable 3 - OK, why do you think Samba is doing better there?\n\nLine 283 - percent points, not percents...\n\nTable 4 - why do you think Samba and Vmamba-ERM are doing so well on FGADR compared to other methods?\n\nA.3 - the text needs a rewrite to make sense I think. Please check this. \n\nFigure 8/9 - Do you have any ground truths available to compare the attention maps shown in Figure 8 and 9?", "limitations": "The authors point out some limitations of their work with regards to class imbalance. I have pointed out other perceived limitations in the Weakness section above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper gives a good introduction to the challenges inherent in grading disease severity in the medical imaging domain. It tackles an important problem, as we know current algorithms often fail at generalising across domain. Integrating Vision Mamba layers with EM-based recalibration of image features is a nice contribution designed to tackle this challenge, which is motivated by logical reasoning about the properties of medical images. The method is comprehensively evaluates on three different benchmark datasets, which are specifically designed to test domain generalisation of algorithms. Furthermore, the results suggests that Samba is able to generalise to unseen domains better than other established methods. There is also ablation provided on several aspect of the model (number of components K, iteration T and update on $\\mu$), as well as t-SNE plots showing how the model embeddings cluster better by severity grading and not data provenance, as well as attention maps of DR images. Overall, the paper combines ideas from current SOTA in deep learning (mamba), statistical modelling (GMM-EM) and properties of medical images.", "weaknesses": "In my opinion the paper's main weaknesses are the following:\n\n1 - Unclear explanation of the EM-based State Recalibration module\n\nI think the introduction to the EM-based State Recalibration module in this paper is too brief, perhaps assuming reader familiarity, and does not provide intuition as to why using GMMs + EM makes sense in this context or is more suitable to this task than other potential approaches. The implementation details of how the the GMM \\& EM is integrated into the overall Samba architecture are not clearly or fully presented. The initialisation strategy for the EM is unclear to me: the paper mentions using learnable severity bases to initialise EM, but doesn't explain how these bases are learned or why this initialisation is most beneficial. It also doesn't discuss the convergence criteria used for the EM and although it shows ablation into how many iterations are best, this comes without proper introduction in the background or methods sections. The same goes for the number of Gaussian basis used: you show ablation on this, but don't introduce this in background or methods sections. Finally, although some equations are presented outlining the E- and M-steps, variables are not always defined and the choice of kernel function seems entirely ad-hoc and unjustified. In particular\n\nYou define your GMM as:\n\n$$p(f_n) = \\sum_{k=1}^K z_{nk} \\mathcal{N}(f_n|\\mu_k, \\Sigma_k),$$ \n\nbut $z_{nk}$ typically represents responsibilities which are estimated in the E-step, not given as part of the model definition, where $\\pi_k$ would represent the mixing coefficients of the GMM. Then you derive the E-step as:\n\n$$z_{nk} = \\frac{\\mathcal{K}(f_n, \\mu_k)}{\\sum_{i=1}^K \\mathcal{K}(f_n, \\mu_i)}$$\n\nwhereas a standard formulation would be \n\n$$z_{nk} = \\frac{\\pi_k \\mathcal{N}(f_n|\\mu_k, \\Sigma_k)}{\\sum_{i=1}^K \\pi_i \\mathcal{N}(f_n|\\mu_i, \\Sigma_i)}.$$\n\nHowever, you don't justify the use of the arbitrary kernel function $\\mathcal{K}$ (exponential inner dot $\\mathrm{exp}(f^T\\mu)$ over standard Gaussian probabilities. Could you explain this in more detail? Then you don't explicitly define what $Z^t$ represents, nor what the relationship is with $z_nk$. Likewise with $F$, you do not explicitly define what it represents, but I will assume you mean the feature matrix of the input image. How does $F$ relate to equation (5)? How does this derive into equation (6)? The recalibration step $\\tilde F$ is not well explained in terms of matrix operations or dimensions. This lack of clarity makes it difficult to interpret the method in terms of a well-established optimisation algorithm, which is an important issue as explaining the transformations applied to the feature representations is crucial to understanding Samba. Overall, while the general idea of using GMM-EM for feature recalibration is interesting, the mathematical formulation presented in the paper has inconsistencies and deviations from standard GMM-EM that I don't think are well-justified and which I would like to see more fully explained and motivated. \n\n2 - Unclear structure of results and lack of clear baseline comparisons\n\nWhile the paper compares three datasets to various baseline and baseline methods, each dataset is treated separately, so it's not always clear the results shown extend to the other datasets, if the comparisons are fair or how the baselines were implemented. I feel like the main results are the ones presented for the Diabetic Retinopathy dataset, yet these are presented last. I think the results section would read better and make more sense if these were presented first. However, one of my main issues here is that you only mention your main baseline comparison method on p.7, but you compare against in Figure 3 without even mentioning it. This should be properly introduced and you should explain why VMamba under empirical risk minimisation (VMamba-ERM) is a good baseline model against which to compare. You also need to explain how Samba differs from VMamba-ERM and what advantages these differences bring to the task that the other doesn't provide. Finally, you don't provide ablation on Samba itself, showing the individual contribution of the Mamba layers vs EMBS layers. With regards to the theoretical analysis on the generalisation risk bound, it feels disconnected to the rest of the paper and not particularly relevant to the stated aims. Is there some way to tie it more or explain why its important? Finally, I think an Appendix section on how the datasets were preprocessed and showing the hyperparameters employed for all the other models presented in the results sections would enrich this paper. There are other things which I find unclear, which I have put below in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Unclear explanation. **R**:\n\n1) After proceeded by BSSM, each patch embedding $\\boldsymbol{f}_n$ is projected by a linear layer to get the state embedding $\\boldsymbol{x}_n$. The severity base $\\boldsymbol{\\mu}_n$ is modeled as a mixture of Gaussian from $\\boldsymbol{x}_n$.\nThe E-M algorithm iteratively approximates $\\boldsymbol{\\mu}_n$ to $\\boldsymbol{x}_n$.\nAfter convergence, the embedding is fed into the rest module.\n2) As introduced in L194-196, moving average is adapted to update the $\\boldsymbol{\\mu}^{0}$, which is used as the initial Gaussian parameter in each EM process.\n3) The E-M empirically implements 3 iterations.\n4) We will define the number of Gaussian kernels $K$ in L171.\n5) We will mention that the mixing coefficients of GMM are left out for simplicity and easy computation and the exponential inner dot kernel is used. \n6) In the $t$-th iteration, $\\mathbf{Z}^t$: the responsibilities of all the patch embeddings from a sample, where $\\mathbf{Z}^t=\\{z_{nk}^t\\}$. \n7) Typo. L185, remove $\\mathbf{F}$.\n8) In Eq.6, $\\mathbf{\\mu}$: the Gaussian basis of all the patch embeddings from a sample, where $\\boldsymbol{\\mu}=\\{\\mu_{k}\\}$. $\\mathbf{\\widetilde{F}}$: the entire image feature from all the patch embeddings.\n\n**Q2**: Structure.  **R**: \n\n1) To clarify, as this paper handles cross-domain medical grading, each dataset we use has more than one domains.\nThe evaluation of domain generalization is evaluated inside each dataset, not across different datasets.\nTab.4: pls refer to general response.\n\n2) We will put Table 4 and the subsection first. \n\n3) We will metion before Fig.3. Especially, \nThe comparison is made between the proposed Samba and the vanilla VMamba as baseline.\nThe VMamba baseline is trained under the empirical risk minimization (ERM), which we denote as Vmamba-ERM.\nNotice that, training under ERM is a common baseline for domain generalization.\nOn top of Vmaba-ERM, the advancement of Samba is two modules EMSR and BSSM.\n\n4) Per-component ablation: in general response.\n\n5) We will put the theoretical analysis to the supplementary, making room for details of the method and ablation study.\n\n6) Details of data pre-processing and hyper-parameter settings on each dataset will be in Appendix.\n\n**Q3**: rephase L44. **R**: *Domain generalized disease grading aims to learn a model that can be well generalized to unseen target domains, when the model is only trained on the source domain data.\nPractically, the feature distribution between the source and unseen target domains usually varies.*\n\n**Q4**: L54.\n**R**: *at the distal of the blood vessels.* \n\n**Q5**: Stem unit?\n**R**: A stem unit partitions the input image into patches.\n\n**Q6**: Typo in Fig.2. **R**: \n\n1) Both modules should occur in the Samba block. Will correct.\n\n2) Typo. 'VSS block'->'Samba block'.\n\n3) Training and inference input images from different domains, marked by red and green arrows, respectively. We will distinguish the source and target domains (located at the upper-left of Fig.2) using different colors.\n\n**Q7**: Fig.3. compare what? What's Vmamba-ERM\\&SOSS? Why Vmamba is constant across T? Different columns? **R**: \n\n1) The comparison is made between the proposed Samba and the vanilla VMamba as baseline, which is trained under the empirical risk minimization (ERM) and denoted as Vmamba-ERM.\n\n2) Typo. 'SOSS' -> 'Samba'.\n\n3) VMamba-ERM does not have EM-based State Recalibration. EMSR is parameterized by T.\nTherefore, the performance of Vmamba-ERM is consistent to T.\n\n4) From column 1 to 3, it reports the AUC, ACC and F1 metric.\n\n**Q8**: Why iteration number?\n**R**: E-M algorithm implements the approximation by iteratively conduct the E and M step [12].\nA small iteration number does not reach the convergence criterion, which results in an poor approximation.\nAfter a too-large iteration number, the approximation may already reach the optimal, adding unnecessary computational cost and training time. We will introduce \\& clarify.\n\n**Q9**: Line 237.\n**R**: Typo. Fig.6->Fig.3. \n\n**Q10**: L244-explain why proper? \n**R**: The process of the proposed state recalibration is differentiable, thereby enabling the application of back-propagation to update $\\boldsymbol{\\mu}^0$. However, the stability of the update cannot be guaranteed due to the EM iterations. Therefore, we adopt moving average to update $\\boldsymbol{\\mu}^0$ to avoid collapse.\n\n**Q11\\&14**: L252&283 - typos on percentage points.\n**R**: will correct. \n\n**Q12**: Tab.1\n**R**: Brighter green, yellow and light green intend to highlight the best, second and third top performance. We will correct. \n\n**Q13**: Tab.3, why Samba better?\n**R**: Compared to VMamba-ERM baseline, the EM-based State Recalibration in Samba models the feature distribution of lesions via Gaussian Mixture Model with learnable severity bases, and re-estimates by E-M algorithm. The grading-related features are mapped to a more compact space. It can be more stable in unseen target domains. \n\n**Q15**: Why much better on FGADR?\n**R**: FGDAR has a different severity-level sample distribution than other datasets. \nThe samples without DR (level-1) only occupy only 5.5\\% among all the training samples, which is far less than others (e.g., level-1 samples occupy 49.3\\% in APTOS).\nTherefore, other methods may over-fit other severity levels and under-fit level-1.\nIn contrast, the selective scan mechanism of the Vmamba-ERM and Samba allows the this severity distribution shift.\nThe EM state re-calibration in Samba makes the feature space more compact, and improves the generalization.\n\n**Q17**: Fig.8/9. ground truth?\n**R:** As the datasets are for grading, we respectfully ask for the reviewer's understanding, there are no available fine-grained ground truth (e.g., pixel, boxes).\n\nFinally, should you have further questions or suggestions, we are happy to address during the discussion stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NjrBWKn27a", "reviewer": "Reviewer_LrSF", "summary": "The paper introduces a new method for disease grading for both within and cross-domain medical images. Three different imaging modalities have been used for experimentation including Retinal, X-ray, and H&E images. In terms of methodological novelty, the authors propose to encode image patches in a recurrent manner to capture informative lesions. Further, they use an EM-based recalibration method to reduce the cross-domain variance by compacting the feature space. Overall, their experiments shows that the method proposed is superior to the baselines.", "questions": "I am curious if the authors can elaborate more on this conclusion made in page 15: \"After processed by the Recurrent Patch Modeling module, more regions in the correlation The high-response patches have grade-related lesions and the information is transported in the recurrent process matrix have higher response.\"", "limitations": "It has been justified properly.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and has a strong theoretical base with detailed explanations. Besides the quantitative evaluations, the authors have conducted some qualitative experiments to show attention maps on retinal images. Also, the idea seems novel along with introducing new modules to tackle issues present in medical images datasets.", "weaknesses": "There are a few major issues with the experimental design:\n1. The Cross-domain Breast Cancer Grading Benchmark is not a well-defined cross-domain problem in computational pathology. Firstly, this is not a widely used dataset in the field. Secondly, the images come from the same center yet with different scanning magnifications. It is not common to use images from two different magnifications as the \"domains\". The cross-domain problem in computational pathology is when the images are actually from two different centers and with two different staining protocols (images might be scanned in varying magnifications). An example of such a problem can be found here: https://camelyon16.grand-challenge.org/Data/\n\n2. In Table 4, the authors have reported the benchmark from [8]. It has not been mentioned whether the proposed model was trained on the exact same seed, device, and cross-validation folds. If any of these are different, the comparison is not fair. Instead, I'd suggest the authors benchmark these with the same exact setting and report the results.\n\n3. For highly imbalanced datasets, especially medical data, it is proper to report Balanced Accuracy to show the performance of the model on the rare classes. Both ACC and AUC fail to represent the rare classes. This is important for comparison as results in Table 1 show that F1 is significantly lower than ACC and AUC. Therefore, Balanced accuracy should be reported for all the benchmarks.\n\n4. There are quite a few modules in different parts of the model that have not been studied in an ablation study. These modules are the bach norm, linear layer, etc. in the EM-based recalibration. Adding the ablation study for these modules can enhance the work :)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:  Is Cross-domain Breast Cancer Grading Benchmark proper? 1) not widely used. 2) It is not common to use images from two different magnifications as the \"domains\". 3) Evaluation on CAMELYON17, from different centers and different staining.\n\n**R**: Thanks for your valuable comment, so that we could have a chance to clarify some important aspects.\n\n1) Following your suggestion, we take the five individual domains marked by the staining protocols in CAMELYON17 for further experiments.\nWe use Domain-1 as the source domain, and reports its performance on the rest four unseen target domains, denoted from Domain-2 to Domain-5.\nThe results are reported as follows.\nIt is observed that, the proposed Samba still shows a significant performance improvement on all the four unseen target domains compared with the baseline VMamba-ERM.\nThe results will be included when revision.\n\nTable R2: Generalization performance comparison between Samba and VMamba-ERM baseline.\n| Method | Domain-2 | Domain-3 | Domain-4 | Domain-5| avg.| \n|----------|----------|----------|----------|----------|----------|\n| VMamba-ERM  |  76.23  | 74.17 | 79.53 | 69.87 | 74.95 | \n| Samba |  **84.59**  | **82.48**  | **86.50** | **78.64** |  **83.05**  | \n||||||\n\n2) We also inspect in our dataset, if the samples vary in staining and if they have domain gap from a rigorous machine learning perspective.\nAs shown in Fig.R2 a (attached in the 1-pg PDF), the samples from Domain-1 ($\\times$20) and Domain-2 ($\\times$40) are not only different in magnifications, but also varied in staining.\nBesides, we use t-SNE visualization to inspect the samples' feature distribution from Domain-1 and Domain-2, displayed in Fig. R2 b (attached in the 1-pg PDF), respectively.\nIt can be observed that, samples from Domain-1 (marked in X) and Domain-2 (marked in O) have clear distance in the feature space. Besides, samples of the same severity level but from different domains are not clustered together. It indicates that the two domains in this dataset have the so-called domain gap and are suitable to benchmark the generalization capablity. \n\n3) In real-world scenarios, some hospitals may use magnification factors that are not present in the training set. So testing the cross-magnification generalization ability by the Cross-domain Breast Cancer Grading Benchmarkalso still has practical significance.\n\n**Q2**: Are evaluation in Table 4 same and fair?\n\n**R**: We would like to clarify that, the proposed Samba is evaluated under all the default settings of [8] for fair evaluation.\nSpecially, the batch size is 16, the training terminates after 100 epochs, the initial learning rate is $10^{-3}$ with a weight decay of $5\\times10^{−4}$ and a momentum value of 0.9.\nWe will explicitly mention these details. \n\n**Q3**: 1) Proper to report Balanced Accuracy, e.g., Table 1. 2) Balanced accuracy for all the benchmarks.\n\n**R**:\n1) We adapt the balance accuracy metric (denoted as BACC), which is the mean of Sensitivity and Specificity, on DG Fatigue Fracture Grading benchmark (Table.R3) and DG Breast Cancer Grading Benchmark (Table.R4).\nThe results are reported as follows.\nIt can be seen that, the proposed Samba still outperforms the rest methods in term of the balance accuracy, indicating its effectiveness on rare classes. \n\nTable R3: Effectiveness of the proposed Samba on recurrent patch modeling under BACC metric. Domain-1 and Domain-2 in the Fatigue Fracture Grading\nBenchmark are used as the source and unseen target domain.\n| Method | ACC | AUC | F1 | BACC | |\n|----------|----------|----------|----------|----------|----------|\n| LSTM | 39.8 | 50.2 |18.6 | 25.6 |\n| UR-LSTM | 43.3 | 61.8 | 20.9 | 25.2 |\n| UR-GRU | 45.7 | 65.1 | 22.4 | 27.1 |\n| ViT  | 50.0 | 69.3 | 26.5  | 30.9 |\n| VMamba-ERM  | 52.7 | 70.4 | 28.7 | 34.0  | \n| Samba  |  **76.2** | **81.5** | **45.8** | **52.2**  | \n||||||\n\nTable R4: Effectiveness of the proposed Samba than baseline under BACC metric. Experiments on DG Breast Cancer Grading Benchmark.\n| Method | Backbone | ACC | BACC | ||\n|----------|----------|----------|----------|----------|----------|\n| ERM | VMamba-T | 40.4 | 18.7 |  |\n| Samba | VMamba-T | **54.8** | **24.5** |  |\n| ERM | VMamba-S | 50.1 | 20.6 |  |\n| Samba  | VMamba-S | **56.1** | **27.9**  |  |\n| ERM  | VMamba-B | 54.9 | 25.6 |  | \n| Samba |  VMamba-B | **60.5** | **29.8** |  | \n||||||\n\n2) We would like to raise the reviewer's attention that, despite that the severity level is highly imbalanced in grading problems especially in DR grading, ACC, F1 and AUC metrics are both acknowledged as the commonly-used evaluation metrics [4,6,8,19,29,36,50,51,53,54,55]. Therefore, we directly adapt these metrics and existing evaluation protocols for fair evaluation in Table 4. \nBesides, balanced accuracy probes a model's performance very similar as F1-score does, while we already use F1-score.\n\n**Q4**: Ablation on mdules.\n\n**R**: We provide an ablation study on each module of the proposed Samba, which is also mentioned by Reviewer\\#Wyiu and b4sx.\nSpecifically, on top of the VMamba baseline, two components, namely, Bi-directional State Space Modeling (BSSM) and EM-based State Recalibration (ESR), are added.\nThe experiments are conducted on the DG Fatigue Fracture Grading Benchmark. The results are reported in Table~R1 in the attached 1-pg PDF file.\nIt is observed that BSSM contributes to an ACC, AUC and F1 improvement of 5.2\\%, 1.7\\% and 4.9\\%, respectively.\nESR contributes to an ACC, AUC and F1 improvement of 18.3\\%, 9.4\\% and 12.2\\%, respectively.\nMeanwhile, we respectively ask for the reviewer's understanding that our work focuses on innovatively introducing BSSM and ESR on top of the vanilla elements (e.g., layer, norm) in VMamba without modify. The ablation of vanilla elements may beyond the scope of this work.\n\n**Q5**: Elaborate more on text in p15.\n\n**R**: Please refer to the general response.\n\nShould you have further suggestions, we are glad to address during the discussion stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GeAePQlD0j", "reviewer": "Reviewer_b4sx", "summary": "The authors introduce a new method named Severity-aware Recurrent Modeling (Samba) for disease grading in medical imaging. Specifically, they propose encoding image patches in a recurrent manner to accurately capture decisive lesions and transmit critical information from local to global contexts. Additionally, an Expectation-Maximization (EM) based state recalibration mechanism is designed to map feature embeddings into a compact space, thereby reducing the impacts of cross-domain variations.", "questions": "1.\tAmong the compared methods mentioned in Table 4, some ViT-based and ResNet-50-based methods are not specifically designed for domain adaptation tasks. Is this comparison fair?\n2.\tThe article states: \"The Mamba model is a suitable structure that aligns with our needs. Guided by global severity awareness, the update of hidden states can selectively ignore information about low-level lesions, primarily preserving information about the most severe lesions.\" Can you explain how SSM selects severe lesion information? A theoretical justification should be provided, as this is a point of concern.\n3.\tIn the comparative experiments with other SOTA methods, are all the source domains supplemented with the two additional large-scale datasets DDR [24] and EyePACS [13]?", "limitations": "The authors have addressed the limitations of the current work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed severity-aware recurrent modeling uses a state space model to store and transmit severity information from local to global, which is valuable for the classification of medical images with small lesion areas.\n2.\tFor domain adaptation tasks, an EM-based state recalibration mechanism was also proposed and its effectiveness was validated in experiments.\n3.\tThe interpretability visualization of the experimental results is excellent.", "weaknesses": "1.\tThe significance of using the Mamba architecture in cross-domain tasks is uncertain; in fact, restricting the Mamba module may lead to excessive specialization.\n2.\tThe ablation experiments regarding the specific structure of Samba are not sufficiently comprehensive.\n3.\tThe open-source code is not explicitly provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Significance of using Mamba in cross-domain tasks; restricting Mamba module may lead to excessive specialization.\n\n**R**: Thanks for your valuable comments, so that we could have a chance to clarify the generalization and universality of the proposed Samba. Specially,\n\n(1) The feature distribution shift between the source domain and unseen target domains affects not only the intermediate features but also the selective scan mechanism of Mamba, which poses a clear performance drop of Mamba on unseen target domains.\nTo address issue, the proposed method not only introduces both forward and backward directions to primarily preserve information about the most severe lesions, but also introduces a EM-based State Recalibration mechanism to compact the feature space so that the feature distribution is less sensitive to the domain shift.\n\n(2) From the experimental side, the proposed Samba shows a significant unseen domain performance improvement than the VMamba baseline on three different modalities, varying from fundus images, X-ray images, and pathological images, respectively.\nMoreover, as discussed in the paper, the recurrent encoding of image patches also contributes to cross-domain disease grading. In Tab.4, even the VMamba baseline outperforms CNN- and ViT-based methods.\n\nWe will enrich these discussions accordingly. \n\n**Q2**: Ablation studies regarding the specific structure of Samba.\n\n**R:**\nWe provide an ablation study on the specific structure of Samba.\nSpecifically, two components, namely, Bi-directional State Space Modeling (BSSM) and EM-based State Recalibration (ESR),  are incorporated into the VMamba baseline.\nThe experiments are conducted on the DG Fatigue Fracture Grading Benchmark, and the results are reported in Table R1 in the attached PDF.\nIt is observed that BSSM contributes to an ACC, AUC and F1 improvement of 5.2\\%, 1.7\\% and 4.9\\%, respectively.\nESR contributes to an ACC, AUC and F1 improvement of 18.3\\%, 9.4\\% and 12.2\\%, respectively.\n\n**Q3**: The open-source code is not explicitly provided.\n\n**R**: Thanks for your valuable suggestion. Owing to the company regulation, the source code will be made available after publication.\nWe respectively ask for the reviewer's understanding on this restriction. \n\n**Q4**: Among the compared methods mentioned in Table 4, some ViT-based and ResNet-50-based methods are not specifically designed for domain adaptation tasks. Is this comparison fair?\n\n**R**: Thanks for your value comments, so that we could have a chance to clarify the compared methods.\nWe first compare six methods for domain generalization tasks, namely, [36,50,51,53,55,56].\nThen, we compare two DR grading methods under the domain generalization setting, namely [4,8].\nAll these eight methods are plenty for comparison under the context of domain generalization. \nThe rest four CNN or ViT based methods [6,19,29,31] do not have domain generalization property, and are involved just for boarder comparison.\nBesides, these methods are implemented under the emperical risk minimization (ERM) setting, which is the same as the VMamba baseline.\nThis demonstrates the effectiveness of Mamba structure in the cross-domain grading tasks.\n\n**Q5**: Can you explain how SSM selects severe lesion information? \n\n**R:**\nThanks for your valuable suggestion.\n\nWe model the relation between patch embedding from SSM and severity level by drawing inspiration from the class activation map (CAM) mechanism [a].\nSpecifically, give an image $\\boldsymbol{x}$, assume that $\\boldsymbol{\\tilde{f}}_k(i)$ denote the activation of unit $k$ from the last Samba block at the patch $i$.\nThen, for unit $k$, after implementing a global average pooling (GAP), the feature embedding $F^k$ is $\\sum_i f_k(i)$.\n\nThus, for a certain severity level $c$, the input to the softmax $S_c$ is $\\sum_k w_k^c F_k$, where $w_k^c$ is the weight corresponding to severity level $c$ for unit $k$.  \nHere $w_k^c$ indicates the importance of $F_k$ for severity level $c$.\nFinally, the output of the softmax for severity level $c$, $P_c$ is computed as\n$$\nP_c = \\frac{{\\rm exp} (S_c)}{\\sum_c {\\rm exp} (S_c)}.\n$$\nBy plugging $F_k = P_i f_k(i)$ into the severity level score $S_c$, we obtain\n$$\nS_c = \\sum_k w_k^c \\sum_i f_k(i) = \\sum_i \\sum_k w_k^c f_k(i).\n$$\n\nThen, the severity activation map $M_c$ for severity level $c$ is defined as follows, where the activation of each patch $i$ is computed as\n$$\nM_c(i) = \\sum_k w_k^c f_k(i).\n$$\n\nWe take the patch embeddings from the last Samba block as input, so as to visualize the per-level severity activation patterns.\nThen, the activated severity patterns are displayed on the original images.\nWe use FGADR as the unseen target domain. \n\nThe results are shown in Fig.~R1 in the attached 1-pg PDF, where the activated patches are highlighted in blue boxes.\nFrom the first to the fifth row, the samples from level-1 to level-5 are provided accordingly.\nFrom the first to the fifth column, the patch activation map from level-1 to level-5 is generated by the aforementioned methods.\nNotice that, as level-1 refers to the normal scenario, each sample has activations on level-1, meaning some patches are normal.\n\n[a] Zhou, Bolei, et al. \"Learning deep features for discriminative localization.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n\n**Q6**: In the comparative experiments with other SOTA methods, are all the source domains supplemented with the two additional large-scale datasets DDR [24] and EyePACS [13]?\n\n**R:** Yes, as all the performance of the state-of-the-art methods reported in [8] is supplemented with the two additional large-scale datasets DDR [24] and EyePACS [13], for fair evaluation, we also supplement both datasets when training.\nWe will clarify it in the main text accordingly.\n\nFinally, should you have further suggestions and comments, we are glad to incorporate during the discussion stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81kVhrqHfT", "reviewer": "Reviewer_hFq3", "summary": "Accurate disease grading in medical image analysis is challenging due to the variability within disease levels and the similarity between adjacent stages. Additionally, models must handle data from unseen target domains, where differences in feature distribution can significantly reduce performance. To address these issues, this paper proposes the Severity-aware Recurrent Modeling (Samba) method, which encodes image patches sequentially to capture severity information and employs an Expectation-Maximization based recalibration mechanism to handle cross-domain variations. The method also uses a Gaussian Mixture Model to model feature distributions and reconstructs intermediate features using learnable severity bases.", "questions": "- Adding specific numbers and metrics to the abstract would be beneficial. Currently, lines 18-19 are quite vague.\n- On page 7, the authors likely intend to refer to Figure 3 for the results, rather than Figure 6.\n- In Figure 5, the T-SNE plot is difficult to interpret due to small icons. Could the datasets be represented by icons and the severity by color for better qualitative evaluation?", "limitations": "The authors addressed the limitations of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addresses a significant problem in medical image analysis where different grading can appear differently in the data.\n- Different diseases and imaging modalities have been evaluated.\n- Use of publicly available datasets.\n- Code will be made publicly available upon acceptance.", "weaknesses": "- Some parts need more detailed explanation. Although space is limited, certain sections assume a high level of pre-knowledge. For example, more explanation on Samba would be useful.\n- For me it is not really clear how the patches are used in a recurrent manner and what their sequence should be. Can you please elaborate more on the recurrent part. \n- There is no study demonstrating that the model specifically attends to patches related to severity. Is there a way to illustrate this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Some parts need more detailed explanation. Although space is limited, certain sections assume a high level of pre-knowledge. For example, more explanation on Samba would be useful.\n\n**R**: Thanks for your valuable feedback.\nWe will provide more explanations of the proposed Samba, such as:\n1) After proceeded by BSSM, the feature embedding of each patch embedding $\\boldsymbol{f}_n$ is firstly projected into the latent space by a linear layer to get the target state embedding $\\boldsymbol{x}_n$. The severity base $\\boldsymbol{\\mu}_n$ is modeled as a mixture of Gaussian from $\\boldsymbol{x}_n$.\nThen, the E-M algorithm iteratively approximates $\\boldsymbol{\\mu}_n$ to $\\boldsymbol{x}_n$.\nAfter convergence, the embedding is fed into the rest module.\n2) Each Gaussian is initialized by the Kaiming initialization.\n3) The E-M empirically implements 3 iterations, according to the observation in Fig.3.\n4) The number of Gaussian kernels $K$ in L171 will be defined when first introducing.\n5) The specialization of Samba for cross-domain tasks, especially,\nthe feature distribution shift between the source domain and unseen target domains affects not only the intermediate features but also the selective scan mechanism of Mamba, which poses a\nclear performance drop of Mamba on unseen target domains. To address issue, the proposed method not only introduces both forward and backward directions to primarily preserve information about the most severe lesions, but also introduces a EM-based State Recalibration mechanism to compact the feature space so that the feature distribution is less sensitive to the domain shift.\n\n**Q2**: It is not really clear how the patches are used in a recurrent manner and what their sequence should be. Can you please elaborate more on the recurrent part?\n\n**R:** Thanks for your valuable feedback.\nAfter sliding the image $\\mathbf{I} \\in \\mathbb{R}^{H \\times W \\times 3}$ into a variety of patches, the input is formed as a sequences of 2-D patches, each of which has a spatial position of $H/4 \\times W/4$.\nThen, in each Samba block, the Bi-directional State Space Modeling module has both feedforward and backforward SSM, where the selective scan mechanism allows to handle the patches in a recurrent manner.\nSpecifically, the 2-D selective scan mechanism in both components is directly inherited from [15].\nThe input patches are traversed along two different scanning paths (horizontal and vertical), and each sequence is independently processed by the SSM.\nSubsequently, the results are merged to construct a 2D feature map as the final output.\n\nWe will enrich these details accordingly.\n\n**Q3**: There is no study demonstrating that the model specifically attends to patches related to severity. Is there a way to illustrate this?\n\n**R**: Thanks for your valuable suggestion.\nWe model the relation between patch embedding from SSM and severity level by drawing inspiration from the class activation map (CAM) mechanism [a].\nWe take the patch embeddings from the last Samba block as input, so as to generate the per-level severity activation patterns.\nThen, the activated severity patterns are displayed on the original images.\nWe use FGADR as the unseen target domain. \nThe results are shown in Fig.~R1, where the activated patches are highlighted in blue boxes.\nFrom the first to the fifth row, the samples from level-1 to level-5 are provided accordingly.\nFrom the first to the fifth column, the patch activation map from level-1 to level-5 is generated by the aforementioned methods.\nNotice that, as level-1 refers to the normal scenario, each sample has activations on level-1, meaning some patches are normal.\n\n[a] Zhou, Bolei, et al. \"Learning deep features for discriminative localization.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n\n**Q4**: Adding specific numbers and metrics to the abstract would be beneficial. Currently, lines 18-19 are quite vague.\n\n**R**: Thanks for your valuable suggestion.\nWe will accordingly add the following in the introduction:\n*Extensive experiments show that the proposed Samba outperforms the VMamba baseline by an average accuracy of 23.5\\%, 5.6\\% and 4.1\\% on the cross-domain grading of fatigue fracture, breast cancer and diabetic retinopathy, respectively.*\n\n**Q5**: On page 7, the authors likely intend to refer to Figure 3 for the results, rather than Figure 6.\n\n**R**: Sorry for the typo. We will correct it from Fig.6 to Fig.3. \n\n**Q6**: In Fig.5, the T-SNE plot is difficult to interpret due to small icons. Could the datasets be represented by icons and the severity by color for better qualitative evaluation?\n\n**R:** Thanks for your valuable suggestion, and we have modified it accordingly.\nPlease refer to Fig.~R3 in the attached PDF file for reference.\n\nFinally, should you have further suggestions and comments, we are glad to incorporate during the discussion stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
General Response:

We thank the reviewers for their time and constructive suggestions, and are glad that the reviewers unanimously give appreciation in a few points:

- Technique contribution (**Wyiu**: integrating Vision Mamba layers with EM-based recalibration of image features is a nice contribution; **LrSF**: the idea seems novel; **b4sx**: use state space model to store and transmit severity information from local to global; **hFq3**: significant problem.)

- Extensive evaluation \& visualization (**Wyiu**: comprehensively evaluation; **LrSF**: quantitative evaluations and some qualitative experiments; **b4sx**: The interpretability visualization of the experimental results is excellent; **hFq3**: different diseases and imaging modalities.)

- Motivation/Task significance (**Wyiu**: generalising across domain; **LrSF**: has a strong theoretical base with detailed explanation;  **hFq3**: significant problem.)

However, there are also some major concerns as follows, in which we clarify as follows.

- Lack of per-component ablation study (**Wyiu** comment\#2; **LrSF** comment\#4; **b4sx** comment\#2).

**R**: We further provide an ablation study on each module of the proposed Samba.
Specifically, on top of the VMamba baseline, two key components, namely, Bi-directional State Space Modeling (BSSM) and EM-based State Recalibration (ESR), are evaluated.
The experiments are conducted on the DG Fatigue Fracture Grading Benchmark, and the results are reported in Table~R1 (attached in the 1-pg PDF file).
It is observed that BSSM contributes to an ACC, AUC and F1 improvement of 5.2\%, 1.7\% and 4.9\%, respectively.
ESR contributes to an ACC, AUC and F1 improvement of 18.3\%, 9.4\% and 12.2\%, respectively.

- Path to visualize and understand the severity level (**b4sx** Comment\#5; **hFq3**: comment\#3).

**R**: We model the relation between patch embedding from SSM and severity level by drawing inspiration from the class activation map (CAM) mechanism [a].
Specifically, we take the patch embeddings from the last Samba block as input to generate the per-level severity activation patterns.
Then, the activated severity patterns are displayed on the original images.
We use FGADR as the unseen target domain. 
The results are shown in Fig.~R1 (attached in the 1-pg PDF file), where the activated patches are highlighted in blue boxes.
From the first to the fifth row, the samples from level-1 to level-5 are provided accordingly.
From the first to the fifth column, the patch activation map from level-1 to level-5 is generated by the aforementioned methods.
Notice that, as level-1 refers to the normal scenario, each sample has activations on level-1, meaning some patches are normal.

[a] Zhou, Bolei, et al. "Learning deep features for discriminative localization." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

- Clarify if the state-of-the-art comparison (Table 4) is fair (**Wyiu** comment\#2; **LrSF** comment\#2; **b4sx** Comment\#4).

**Wyiu**, **LrSF**, **b4sx**: For Table 4, the results of the state-of-the-art methods are directly cited from [8], and the evaluation protocols of the proposed method follow all the default settings as [8] does.

**LrSF**: Fair mplementation details. Specially, the batch size is 16, the training terminates after 100 epochs, the initial learning rate is $10^{-3}$ with a weight decay of $5\times10^{−4}$ and a momentum value of 0.9.

- A.3. The text needs a rewrite (**Wyiu** comment\#16; **LrSF** comment\#5).

**R:** The sentences have been re-written as:
*After processed by the Recurrent Patch Modeling module, more cells in the correlation matrix have higher response. 
Usually, a handful of the patches inside the image have grade-related lesions.
After the processing of our module, the information of these grade-related lesions is transported to other patches. It allows the model to perceive a more global-wise representation.
Consequently, more patches that contain the grade-related lesion information are activated, and more cells are highly responded in the correlation matrix.*

We will explicitly mention these details when revision. 

We hope our clarification could help to make a more informed evaluation to our work.
In the following individual response, we provide answers to each raised weakness/question.

Best regards,

Authors

Meta-review: The paper was reviewed by four experts. Although there was some disagreement about the novelties and clarity of the paper before the rebuttal and discussion stages, the authors were able to address most of the concerns. All reviewers recommend acceptance of the paper.  The authors are advised to address the requested clarifications in the final version of the paper.
Decision: Accept (poster)