Domain Generalization with Small Data
Authors: Kecheng Chen
Elena Gal
Hong Yan
Haoliang Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: domain generalization
small data
healthcare
medical image

Abstract: In this work, we propose to tackle the problem of domain generalization in the context of insufficient samples. Instead of extracting latent feature embeddings based on deterministic models, we propose to learn a domain-invariant representation based on the probabilistic framework by mapping each data point into probabilistic embeddings. Specifically, we first extend empirical maximum mean discrepancy (MMD) to a novel probabilistic MMD that can measure the discrepancy between mixture distributions (i.e., source domains) consisted of a serial of latent distributions rather than latent points. Moreover, instead of imposing the contrastive semantic alignment (CSA) loss based on pairs of latent points, a novel probabilistic CSA loss encourages positive probabilistic embedding pairs to be closer while pulling other negative ones apart. Benefiting from the learned representation captured by probabilistic models, our proposed method can marriage the measurement on the distribution over distributions (i.e., the global perspective alignment) and the distribution-based contrastive semantic alignment (i.e., the local perspective alignment). Extensive experimental results on three challenging medical datasets show the effectiveness of our proposed method in the context of insufficient data compared with state-of-the-art baseline methods.

Under review as a conference paper at ICLR 2023
DOMAIN GENERALIZATION WITH SMALL DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
In this work, we propose to tackle the problem of domain generalization in the con- text of insufficient samples. Instead of extracting latent feature embeddings based on deterministic models, we propose to learn a based on the probabilistic framework by mapping each data point into probabilistic embeddings. Specifically, we first extend empirical maximum mean discrepancy (MMD) to novel probabilistic MMD that can measure the discrepancy between mixture distributions (i.e., source domains) consisted of a serial of latent distri- butions rather than latent points. Moreover, instead of imposing the contrastive semantic alignment (CSA) loss based on pairs of latent points, a novel probabilistic CSA loss encourages positive probabilistic embedding pairs to be closer while pulling other negative ones apart. Benefiting from the learned representation cap- tured by probabilistic models, our proposed method can marriage the measurement on the distribution over distributions (i.e., the global perspective alignment) and the distribution-bart alignment). Extensive experimental results on three challenging medical datasets show the effectiveness of our proposed method in the context of insufficient data compared with oate--oftthe-arr baseline methods.
representation
contrastive semantic alignment (i.c., the local perspective
INTRODUCTION
Nowadays, we have witnessed a lot of successes via imposing machine learning techniques in a variety of tasks related computer vision and natural language processing, such as face recognition Li al. (202bb). object detection Zaidi et al. (2022), and speech recognition Mridha et al. (002).. Despite many achievements so far, the videly-adopted assumption for most existing methods, i.e., it is identically and independently distributed between training and testing data, may not always hold actual applications Zhou et al. (2022): Liu al. (2022). the real-world scenario, it is quite common that the distribution between training and testing data may be different, owing to sophisticated environments. For example, resulting from the differences of device vendor and staining method, acquired histopathological images of breast cancer from different healtheare centers exist significant domain gaps (a.ka.,, domain shift, see Figure| for more details), which may lead to the catastrophic deterioration of the performance Qi et al. (2020). To address this issue, domain generalization (DG) is developed to learn a model from multiple related yet different domains (a.k.a., source domains) that is able generalize well on unseen testing domain (a.k.a., target domain).
Recently, researchers proposed quite a few domain generalization approaches, such as data augmenta- tion with randomization Yue al. (2019), data generalization with stylization Verma et al. (2019): Zhou et al. (2021), meta learning Li et al. 2018a); Kim et al. (202T)-brr -based training schemes, among which representation learning-basee based methods are one of the most popular ones. These representation learning-based methods Balaji et al. (2019) aim to learn Nomain-invarian
feature representation. To be specific, if the diserepancy between source domains in feature space can be minimized, the model is expected to be better generalize well on unseen target domain, owning to learned domain- invariant and transferable feature representation Ben- David et al. 20066.. For instance, an classical contrastive semantic alignment (CSA) loss proposed by (Motiaan et al. 2017) was to encourage positive sample pairs (with same label) from different domains closer while pulling other negative pairs (with different labels) apart. (Dou et al., 2019) introduced the CSA loss which jointly considers local class alignment loss (for point-wise domain alignment) and global class alignment loss (for distribution-wis alignment).

---

Under review as a conference paper at ICLR 2023
kK7
Epuhelumn
INC
Fysheliar
vat
Pythiium
Strnno
Sturea
Storea
Figure 1: Histopathological image examples of breast cancer tissue from three different healthcare institutes, including NKI with 626 images. IHC with 645 images, and VGH with 1324 images. There are two different tissue types, including epithelium and stroma. Obvious domain gaps (e.g., the density of tissue and the staining color) can be observed.
Despite the progress being achieved so far, it should be noted that a reliable contrastive semantic loss with piint-wise (or local) perspective usually requires sufficient samples on source domains such that diverse sample-to-sample pairs can be constructed Sohn (0166); Khosla et al. (020). For example, (Khosla et al. 2020) proposed supervised contrastive semantic loss with considerable volume of batch size on large-scale datasets such that decent performance can be guaranteed. (Ya
et al. 2022) also emphasized the importance of the number of sample-to-sample pairs influenced by data sizes for contrastive- baseed loss on DG problem. Meanwhile, in the eye of distribution-wise (or global) alignment between domains Dou al. (019)), a consistent distribution measurement (e.g.. Kullback-eeilerr (KL) divergence) thcoretically relies on sufficient samples for the distribution estimation as discussed by (Bu et al. 2018). However, these sufficient samples from multiple source domains may not always be available and accessible, especially for the medical imaging data, due to potential ethic, privacy and proprietorship risks. It therefore necessary to develop reliable and effective contrastive semantic alignments with local and global perspectives in the context of insufficient samples (a.k.a., small-data scenario) from source domains, in order to achieve better domain-invarian representations.
In this paper; we propose to learn domain-invariant representation from multiple source domains to tackle the domain generalization problem in the context of insufficient samples. Instead of ex- tracting latent embeddings (i.c., latent points) based on deterministic models (c.g., convolutional neural networks, CNNs), propose leverage a probabilistic framework endowed by variational Bayesian inference to map each data point into probabilistic embeddings (i.e., the latent distribution) for domain generalization. Specifically, by following the domain-invariin (distribution-wis) perspective, we propose to extend empirical maximum mean discrepancy (MMD) novel probabilistic MMD (P-MMD) that can empirically measure the discrepancy between mix- ture distributions (a.k.a,, distributions over distributions), consisted of a serial of latent distributions rather than latent points. From local perspective, instead of imposing the CSA loss based on pairs of latent points, a novel probabilistic contrastive semantic alignment (P-CSA) loss with kernel mean embedding is proposed to encourage positive probabilistic embedding pairs closer while pulling other negative ones apart. Extensive experimental results on three challenging medical imaging classi- fication tasks, including epithelium stroma classification on insufficient histopathological images, imbalanced-class based skin lesion classification, and spinal cord gray matter segmentation, show that our proposed method can achieve better cross-domain performance in the context of insufficient data compared with sfute-of-the-as baseline methods.
learning from global
2 RELATED WORKS
2.1 DOMAIN GENERALIZATION AND ITS APPLICATION IN MEDICAL IMAGE CLASSIFICATION
Existing DG methods can be generally categorized into three different streams, namely data aug- mentation/generation Yue et al. ) Graves (201) Zhou et al. (202)) meta-learning Li et al. (0118)); Kim et al. (2021) and feature representation learning Li et al. (2018b); Gong et al. (2019); Xiao et al. (0211. Among these methods, feature representation learning, which aims explore
invariant feature information that can be shared across domains, demonstrates to be a widely adopted
method for the problem of DG. For feature representation learning-based DG method, Li et al.
2

---

Under review as a conference paper at ICLR 2023
(0188) proposed to conduct multi-domain alignment in latent space via multi-domain MMD distance. Gong et al. (2019) leveraged adversarial training to eliminate the domain discrepancy such that domain-invarre
be learned in manifold space. Due to the varieties of imaging protocol (e.g., the choice of image solution for MRI image), device vendors (e.g., Philips or Siemens CT scanners), and patient populations (the race and age group), the acquired imaging data from different medical sites may exist significant domain shift problem Liu et al. (2021). Dou al. (2019) proposed a meta-learning framework perform local and global semantic alignment for medical image classification. Similar design is also adopted by Li et al. (2022a) for tissue image classification. Qi et al. (2020) utilized the curriculum learning scheme transfer the knowledge for histopathological images classification. Li et al. (2020a) combined the data augmentation and domain alignment to achieve decent performance on multiple medical data classification tasks. However, these methods may not focus on learning domain-invarian representation on insufficient sumples from source domains. This scenario may widely encounter in clinical environments because 1) The cost of annotated data by experienced professionals are typically prohibitive, leading to the lack of samples in size and diversity Yoon et al. (2019). 2) For rare diseases (e.g., glioblastoma and lymphoma), the size of data is usually small Lee et al. (2022) 3) medical imaging data strictly insufficient in most cases due to potential ethic and privacy-preserving concerns Li et al. (2020b).
representation
2.2 PROBABILISTIC NEURAL NETWORKS
Compared with deterministic models, probabilistic neural networks turns to learn a distribution over model parameters, which can integrate the uncertainty in predictive modeling Kingma et al. (2015); Gal & Ghahramani (2016). When the data insufficient, probabilistic models usually can achieve better generalized performance due to its probabilistic property (as an implicit regularization) Blundell al. (2015). In the context of insufficient samples, Bayesian neural network Neal (2012) (BNN) with variational inference, a representative probabilistic model, not only can improve predictive accuracy as a classifier Wilson Izmailov] (2020), but also can build up the quality of low-dimensional embeddings of insufficient data Mallick et al. (202I), which is crucial motivation for this paper. Meanwhile, modern analytical approximation techniques (e.g., Variational inference Blei et al. (2017), empirical Bayes Krishhan et al. (2020)) can efficiently infer the posterior distribution of model parameters with stochastic gradient descent method, which can integrate BNN with deterministic DNN conveniently.
In Xiao et al. (202T) the authors proposed to consider the uncertainty of a generalizable model based on BNN, where the distances of positive probabilistic embedding pairs and class distribution are minimized via KI. measure. Despite the effectiveness, the dissimilar pairs (i.e., negative pairs) are ignored, which may not benefit feature representation learning. Moreover, they only focused on sample similarity while the distribution information is ignored. Instead, our proposed method comprehensively considers both positive and negative probabilistic embedding pairs via novel distribution- contrastive semantic loss.
3 METHOD
3.1 PRELIMINARY
Assume that there are K domains from different collected environments. The samples in each domain can be represented as X; = {xx,, ,Xt,, ).
where / E N+ {1,.... ".....KK. X1, E dx denotes a sample with the d dimension vector in the 1-th domain. n, is the total number of samples in the I-th domain. The corresponding labels of samples in each domain can be denoted as Y,
{yr,," ... Yt., where yı, E mx1 is the form of one-hot encoding with m classes in total. For the setting of domain generalization, the source domain data represented as (XS, Y Ji=1. can be available in the training phase only, whereas the target domain data, denoted by XT are only seen in test phase.
K
Here, we provide: framework that can learn better domain-invarian representation when there is insufficient source domain data. The probabilistic neural network is imposed to enable high-quality and powerful feature representation in the context of insufficient samples. To effectively perform global perspective alignment, a novel probabilistic MMD is proposed to empirically measure the discrepancy between distributions over distributions based on reproducing kernel Hilbert space. We
3

---

Under review as a conference paper at ICLR 2023
also propose probabilistic contrastive semantic aliggment to adapt probabilistic embeddings with local perspective. The details of our proposed method are discussed as below.
3.2 PROBABILISTIC EMBEDDING OF INSUFFICIENT DATA
Compared with deterministic models, the probabilistic models can learn distribution over model weights, which has shown a better capacity to represent latent embeddings Mallick et al.] (202T) under insufficient sample scenario, which is key motivation for this work. Here, Bayesian neural network (BNN) Blei et al. (2017) is utilized to extract the low-dimensional embeddings from high- dimensional inputs. By feeding the inputs into BNN with parameter W p(W). , the samples X = { Xt.,, } of each domain can be represented by set of probabilistic embeddings (i.e., latent distributions), i.e., (..,p(Zi) ( = {p(z)x,, W).- P(Z W)} where W ~ p(W) is sampled stochastically. The variational inference is used to approximate the posterior distribution of W with the evidence lower bound (ELBO) (more details can be found in appendix). By using Monte Carlo (MC) estimators with T stochastic sampling operations from the W, the predictive distribution of each p(z x) can be unbiased approximation. The number of MC samples and the corresponding issue of computational efficiency is discussed in A.6
3.3 DISTRIBUTION ALIGNMENT VIA PROBABILISTIC MAXIMUM MEAN DISCREPANCY
In this section, we introduce an approach learning perspective by minimizing the discrepancy among domains. Among various distribution distance metrics, Maximum Mean Discrepancy (MMD) widely adopted Long et al. (2017): Li et al. (2018b) which aims to measure the distance between two probability distributions in non-parametric manner. Specifically, assume that latent embeddings Z from two unknown distributions P. and Pr. The probability measure P can be mapped into a reproducing kernel Hilbert space (RKHS) H as a element by setting,
representation from a global
Zi,,,) } and ( ... Zt., } are drawn
/ k(z,,) Ex-p(k(2,,)) )]
μP (02-p(0(2))
where a reproducing kernel k: X' x + I and corresponding feature map Ø X + H are defined. Let the kernel k is characteristic such that the map P + μo is injective. In this case the MMD can be defined as the distance ||w. - (uP, l| in H between mean embeddings and it can be used as measure of distance between the distributions IP; and PP, |Borgwardt et al. (2006); Gretton et al. (2012). The explicit computation of MMD can be derived by unbiased empirical estimation mean map Gretton et al. (2012). i.e.,
MMD P.)2 ||pe, pr, 11.c
(2)
The idea of using MMD for domain generalization has been explored in several works (e.g.. Li et al. (2018b); Hu et al. (2020)).
In the probabilistic framework, instead of the individual latent embeddings Zi we have latent probabilistic embeddings 1I, := p(z|xI, W),.. For source domain Dr, we have the associated distribution over distributions F.- = (II,, ... .II.,, J. For this scenario, we propose to extend the existing point-based empirical MMD estimate to a distribution- bused empirical probability MMD (P-MMD) estimate. P-MMD utilizes empirical estimation by kernels on distributions to measure the discrepancy between mixture distributions IP, and P., under the probabilistic framework.
Specifically, we first represent latent probabilistic embeddings as elements in RKHS H. using the kernel k. that we call a level-/ kernel in the sequel, e.g., JATTI : Eg-11, [o(z)] = Ez- [k:2,, -), which is an analogous way to the Eq. (I) The kernel mean embedding ALLL, can be regarded as new feature map for a variety of tasks Yoshikawa et al. (2014). Here, to on distributions, we introduce level-2 kernel K Muandet et al. (2012). Consider a level-1 kernel on H and its reproducing kernel Hilbert space (RKHS) H.g Define K as
non-linear learning
K II,)) k(Pn, : Muu.) (v/Mm,, ), (b,,,, 3)
where K and its explicit form on kernel mean embeddings are p.d. kernels Berlinet & Thomas Agnan (20IT). We define the probabilistic MMD (P-MMD) empirical estimation method using the 1evel-2 kernel K:
4

---

where my and my are determined by sampling times T. The kernel mean embedding using the level-1 kernel creates distributions (P.)....., (P(NN) represented by the samples (μTT. .. /MTT,,_ } for I .. respectively in the RKHS H4. The underlying strategy of P-MMD to apply the Under review as a conference paper at ICLR 2023
1
1 nt Σ ==1
P-MMD(P,, P.)²
(pn,
K (1 11.) +
K (Wlg
2 TUTT
a
(4)
In this work the level-1 and level-2 kernels. 1 and K, are both Gaussian RBF kernel due to its impressive performance limited amount of distribution data Muandet et al. (2012). Namely, K can be represented
KGau (Tl,, ITl,)) w(11I1 MMlle))
exp
Ipm, -MITe, IB:...
exp
(HU, pm,, mm)) 2(1U,,, "M Mu,,),. (um,, Mille ymw))
A
exp(
k(ai,, mim.
k(ZI, 2t,))
2 mi
mB
kizzy 2ig)
(5)
classic MMD to these distributions (with respect to the kernel k) To access the effect that the minimization of P-MMD has on the original latent probability distributions across different domains we recall the following:
Theorem Muandett et al. 2012)). Let P1...., .. be probability distributions and P N ΣN P. Then the distributional variance given by tV Ellpe u4ll is 0 if ... P--- The upper bound of the distributional variance can be written
Corollary (Liet al. (0188)).
Σ MMD(P.,P,)
K2
ISIJSK
In our setting Theorem I and Corollary Jalon with the fact that a characteristic kernel imply the following
Corollary 2. K² EnsiJsk P-MMD(P,, P2) iff all moments of latent distributions II associ-
ated to points of domain for L,..., N are distributed identically across domains.
Following Corollary
2 we define the following loss function:
Lglabaa
Σ ISUJKK
P-MMD(P.PP...
(6)
K2
Corollary 2 implies that as 6 tends to 0 so does the distance between the distributions of means, variances and higher moments of the distributions IT, associated to points of different domains.
Remark. In the ablation study (see Appendix), we compare the P-MMD approach to simply taking the mean (i.c.. first moment) of latent probabilistic embeddings 11, i.c. taking 11, mn [1x)- and then minimizing the associated "vanilla" MMD. Although this scheme is more computational- efficient over proposed method, throws away most information about high-level statistics as discussed by (Muandet et al. 2017). We verify empirically that our approach leads to better performance across the domains. The visualized computation of P-MM is shown in Figure [
Although we focus on the scenario of insufficient samples, the computational consumption from Eq. and Eq. (5) may be still prohibitive the calculation of MMD distance between distributions

---

Under review as a conference paper at ICLR 2023
Denate
Leadil Kenn
RKTRS P
"INDD
Kein W)
Figure 2: visualized computational process for probabilistic MMD (P-MMD) on two source domains. The same color for each sample in different domains denotes the same label.
scale at least quadratically with the increasing of sample size (especially for image segmentation task), i.e., (nn²) in domain. Here, by following the linear statistic theory of MMD, the unbiased estimate can be derived by drawing pairs from two domains with replacement, i.c., P-MMD((P, P,), so
(I M + K (1 12) ) K (I 11 K 2 11.2.) where as- suming n, for simplicity. (Borgwardt et al. 2006) gives proof's about the unbiased property of the linear statistic of MMD and shows that statistic power does not be sacrificed too much.
R2
3.4 PROBABILISTIC CONTRASTIVE SEMANTIC ALIGNMENT
To learn domain-invariant representation from local perspective, a popular idea is to encourage positive pairs with same label closer while pulling other negative ones with different labels apart Motian et al. (2017); Dou et al. (2019). These methods usually measure the Euclidean distance between samples in the embedding space. However, this scheme may not satisfy our probabilistic framework due to its probabilistic embeddings. To this end, we propose probabilistic contrastive semantic alignment (P-CSA) loss that can utilize the empirical MMD to measure the discrepancy between probabilistic embeddings. The proposed P-CAS loss Liocat consists of two components, including the positive probabilistic contrastive loss and negative probabilistic contrastive loss. The former aims to minimize the distance between intra-class distributions from different domains, i.e.,
Crpna
MMD(ILL, 11,) 2 2 Σ s (Ma(zn:)) 1
7 Σ (Me
S.L.y Yạ
where (- denotes the embedding network of metric learning, which will contribute to learn the distance between features better Dou et al. (2019). Then, by introducing a distance margin & (can guarantee appropriate repulsion range), the negative probabilistic contrastive loss is denoted by
max MMD(I,,, TT
locat
2
=
0
T
à (Me(zn)))
T
Σ 2 (Me
)) 1.s.t.y A
(8)
Model Training. Our proposed framework consists of three modules, a BNN-based probabilistic extractor Qs. BNN-based classifier Cw, and metric network Me ). For the Qs. we only add a Bayesian layer with ReLU layer on the bottom of a pretrained deterministic model (e.g.. ResNet18 by removing connecco layers) by following (Xiao et al.) (021). For the C.. a Bayesian layer also introduced to adapt the classification on insufficient sample better. The structure of Me the same as (Dou et al. 2019). The images X' = (x1,) conduct 7 stochastic forward passes on the Qs and C, by MC sampling obtain probabilistic predicts {Ø), YT where the outputs (i.e., probabilistic embeddings) of Qs serve as the inputs for the calculations of Cylobol and Llocat. The final predicts (0/, are the expectation of (tt, I The total objectives can be summarized as below, (QL (1 (Q))) + KK (Co (Cw)ll((C[)] (LL Stocall + B B2Lytobels (9)
Crotal = Coo 1.5
where (ūr, Vt.) is the cross-entropy loss with ground-truth VI and its estimation ف The second and third terms aim to learn variational distribution qo(-) approximate the Bayesian posterior distribution on the weights, while minimizing the KL divergence with its prior distribution p(-). The first three terms refer variational Bayesian inference with ELBO Blei al. (017).

---

Under review as a conference paper at ICLR 2023
Table 1: Domain generalization results on the skin lesion classification task. The average value and standard deviation are reported by running each method with five times. Each column denotes cross-domain task. For example, in the second column, we use DMF dataset denotes as the target domain and the remaining datasets as the source domains.
Method
DMF
DTP
MSK
PH2
SON
UDA
AVG
DeepAll
0,2492 10.0127
0.5680.10.0181
0.6674100.083
0.8000110:0167
0.8613110:1996
0.6264110:0112
0.6287
MASP Dou et
2019
0.2692:0.0116
0.5678.10:0361
0.681510000122
0.7833-0:0101
0.920410.0227
0.6538-1.0016
0.6460
0.2793--0.024
0.6007:40:0177
0.6967:0.0011
0.8167+0.0000
0.9272+000117
0.6978-00.0182
0.6697
SWAD
Cho al.
0,3582 +0002344
0.5491 +0.0231
0.6842 00.016
0,9167 +0.0121
0.9824 +0.0012
0.7240 +00025
0.7024
BDIL Xiao al
0.2985-0.0452
0.6204+0:0212
0.7059+0.0145
0.8967+000006
019860+0.0199
0.7219+000024
0.7049
DNA Chu
0.3532 1000133
0.5581
10.0178
0,7120
=0.0194
0,9333 10 0645
0.9851 20.0032
0.7314
1:000111
0.7122
-000266
0.5739 10.0147
0.6935
-0.0165
0,8833
+0.0231
0.9841 20.0008
0.7201
1:000121
0.7063
0.3781+0.0116
0.120000:0155
0.7276 -0.000
0.9416+0:0103
0.9889=0.0041
0.7486 +0.0123
Table 2: Experiment results of Epithelium Stroma Classification in Histopathological Images. Each column denotes a cross-domain task. For example, in the second column, we use IHC dataset denotes as the target domain and the remaining datasets as the source domains.
Method
IHC
NKI
VGH
Average (%)
DeepAll
73.29 +0133
70.60
#0.15
+0.11
74.48
MASF Dou et al.
2019)
80.45++0.10
76.10-+0.11
84.44-0.12
80.33
SWAD Cha er al.
2021
79.74+0.15
74.84+0.11
84.29-t0.12
79.62
BDIL Xiao
2021)
85.56+0..12
71.89+0.14
85.90=0.188
81.05
DNA Chu et aL (2022)
83.93+-0.18
73.94+-0.15
85.57+0.17
81.14
DSU Li et al 2022e1
81.56++0.14
72.47+0.12
83.94+0.16
79.32
Ours (in this paper)
88.82++0.09
76.71=00110
86.92=#0.14
84.06
4 EXPERIMENTS AND ANALYSES
4.1 SKIN LESION CLASSIFICATION
Here, we first perform the skin lesion classification task to explore the generalization performance of our proposed method. 7 public skin lesion datasets] aree utilized, including HAM10000, UDA, SON, DMF, MSK, D7P, and PH2. There are seven classes of skin lesions. The challenges of this task refer to two aspects. 1) The diversity of samples: The lesion locations (e.g., on the leg and the thigh), patients' characteristics (e.g., the skin age and complexion) and the imaging vendors are different among domains, significant domain shifts thus not be ignored. 2) Insufficient samples: The acquired data not only are restricted by the total number of samples in some domains (e.g.. UDA and PH2 have only 601 and 200 samples, respectively) but also suffer from the limitation of inter-class imbalance (e.g., SON dataset only has a class of lesion). We follow experimental settings in (Li et al. 2020a), where each dataset is randomly split into 50% training set, 30% test set, and 20% validate set, respectively. The pretrained ResNet1 8 network is used as the backbone for all methods.
Results. Some competitive domain generalization approaches are introduced for comparison, includ- ing MASF Dou et al. (2019). BDIL Xiao et al. (202T), LDDG Li et al. (2020a) SWAD Cha et al. (202T). DNA Chu et al. (2022). and DSU et al. (2022c). "DeepAll" denotes the model that is trained directly without any domain generalization strategy in all subsequent tasks. We turn their hyperparameters in a wide range. Note that our proposed method, DSU, DNA and BDIL are based on SWAD framework. The accuracy results of each cross- domain task are shown in Table [ One has some observations as following. First, all methods achieve consistent improvements compared over DeepAll model. Second, we observe that MASF and LDDG with deterministic models may not explicitly adapt to the scenario of insufficient samples. In contrast, our proposed method and other baseline methods impose respective schemes to relieve the impact of insufficient samples, which leads to obvious improvements. Benefiting from probabilistic framework as an implicit regularization, our proposed method and BDIL can learn distribution over weights, which can handle insufficient samples flexibly. However, it can be observed that additional invariant classifier learning on BDIL may cause negative effects (see the results on DMF) on challenging data, which may be reasonable as the explicit alignment on the classifier with high error probability can lead to negative transfer (take more uncertainty). The lack of explicit domain- invaarint representations for DNA and DSU may be difficult to address significant domain shifts compared with our proposed method.
7

---

Under review as a conference paper at ICLR 2023
Table 3: The average results of spinal cord GM segmentation on 4 domain generalization tasks.
MASF
LDGG
KDGG
DSU
Method
DeepAll
Dou al. (2019)
et al. (2020a
Wang et al. (202T]
Lieta al. (2022c
Ours
DSC
0.7425
0.7710
0,788T
0,7886
0.7921
0.7957
CC t
-11.4
23,52
34,86
33.43
34.65
35.76
JI f
0.6160
0.6502
0.6667
0.6667
0.6775
0.6828
TPR
0.7667
0,7803
0.8058
0.8075
0,8225
0.8260
ASD
0.5265
0.5505
0.4076
0.3553
0.4362
0.3356
Table 4: Ablation study each component of our proposed method for spinal cord gray matter segmentation task (where "site2" is as the target domain). The model on the first row denotes the basic Unet model.
100
Bachboe
Rieen
Beessan Laven
Local Algenee
Aligineett Clesifer
Globd
Bexeden
DSC
OC
TPR
ASD
0.7235
2621
0.589
01100
DO09
Figure 3: The loss curve of iteration skin le- sion and epothelial-strome classficaiton tasks. (a) Global alignment loss (b) Local alignment loss.
0.7044 47.19
0.995 0.113
0.0022
038268 57.52 0.7067
0/116
0.0001
03364 (0.72 07195 0/8167 1.0485
037771
0157 0.221) 03152 B10600
035455
63.78
0.7389
0.5481
1.6401
4.2 EPITHELIUM STROMA CLASSIFICATION
The epithelium--tromaa ratio can reflect the prognostic status of the tumor, especially for the breast cancer. A key step, therefore, is to recognize which tissues are epithelial or stromal in histopatho- logical images. The obvious domain gaps can be observed from Figure [ Meanwhile, it is much difficult to collect massive number of histopathological images from different sites due to the privacy, Here, three histopathological image datasets collected from different medical institutes are used for comparison, where the NKI and VGH datasets only have 671 and 615 images, respectively. We follow the research in Qi et al. (020) to extract epithelial or stromal patches from histopathological images, in order to balance the number of images among datasets. Then, IHC, NKI, and VGH datasets have 1342, 1230, and 1376 patches, respectively, which is still insufficient for training. We utilize the DomainBed benchmark Gulrajani & Lopez-Pazi (2020) for fair comparison, where each dataset in source domain is randomly split into 80% training set, 20% validate set. The testing is on overall target domain. The pretrained ResNet18 is adopted by all methods as backbone.
Results. We compare our proposed method with recent DG models, including MASF, SWAD, BDIL, DNA, and DUS. By turning the hyperparameter of baseline methods in wide range, the classification accuracy on each target domain (the remaining is as the source domain) is reported in Table Some observe that the type of weight averaging
observations can be summarized as following. First, method (e.g.. SWAD and DNA) is effective for this challenging out-of-domain task. However, due to the lack of explicit domain alignment, the obvious domain shifts may not be fully addressed via weighted ensemble learning, leading to the limitation of the performance. Second, BDIL not only adopts two-level alignments on feature extractor and classifier, but also obtains further improvements by probabilistic framework. However, one can observe that BDIL has similar performance drop (in the skin lesion classification) on the challenging task (i.e., the NKI task). The best performance achieved by our proposed method thus shows the effectiveness of our proposed method. DSU has the poorest performance among all baseline methods, which may be reasonable as the straightforward domain randomization in feature space may not be powerful for eliminating obvious domain shifts.
4.3 SPINAL CORD GRAY MATTER SEGMENTATION
The spinal cord Gray Matter (GM) segmentation Challenge Dataset is used here, where the acquired magnetic resonance imaging (MRI) data are collected from four healthcare centers, and acquisition manufactures and imaging protocols are variable. The challenges of insufficient sample are from two aspects. 1) The number of slices in some sites is relatively small (e.g., sitel and site2 have only 30 and 113 slices). 2) The number of target pixels is small, as GM area only a very small area in overall slice. We follow the training protocols used in Li et al. (2020a) for all methods.
8

---

Under review as a conference paper at ICLR 2023
Table 5: Experiment results of PACS multi-domain classification task based on ResNet50. Each
column denotes a cross-domain task. For example, in the third column, we use Art dataset denotes as the target domain and the remaining datasets as the source domains.
Method
Reference
Art
Cartoon
Photo
Sketch
Average (%)
RSC Huang el al. 2020)
ECCV 2020
78.9
76.9
94.1
76.8
81.7
1,2A-OT 7 hou
2020)
ECCV 2020
83.3
78.2
96.2
73.6
82.8
MatchDG Mahajan Jan et CC22)
ICML 2020
81.2
80.4
96.8
77.2
83.9
pAdaIN Nuriel et 1021D
CVPR 2021
81.7
76.6
96.3
75.1
82.5
MixStyle Zhou et ala 2021
ICLR 2021
86.8
79.0
96.6
78.5
85.2
SagNet Nam at (021)
CVPR 2021
87.4
80.7
97.1
80.0
86.3
SWAD DEADiChace.o 1021
NeurIPS 2021
89.3
83.4
97.3
82.5
88.1
DNA Chu ala (2022)
ICML 2022
89.8
83.4
97.7
82.6
88.4
Bayesian
89.4
83.5
97.3
82.3
88.
Ours
90.2
85.2
98.7
83.6
89.4
Results. Here, four domain generalization approaches are utilized for comparison, including MASF, LDDG al. (2020a). KDDG Wang et al. (202I), and DSU. To qualitatively evaluate the segmenta- tion results, 5 complementary metrics are introduced from statistical and distance-based perspectives, respectively. The average results on four domain generalization tasks are illustrated in Table The detailed evaluation results for each domain can be found in Appendix. First, the performance of segmentation results among all methods achieve improvements with an obvious margin compared with DeepAll. Second, suffering from insufficient samples in some domains, LDDG and KDDG with deterministic models may not model these uncertainties explicitly. In contrast, our proposed method and DSU can generally obtained the best and second best results.
4.4 ADDITIONAL RESULTS
Ablation study each component of proposed method. We interested in the effectiveness of each component of our proposed method. The results can be shown in Table First, we observe that better performance can be achieved by introducing probabilistic layer compared with the results that using Unet, which reflects the superiority of probabilistic models. Secondly, we observe that by either introducing local or global alignment for performance can be achieved compared with the results of only using probabilistic layer, which shows the effectiveness of introduced probabilistic feature regularization term. Last but not the least, by imposing the domain-invaree invariant learning with both local and global views, the performances are further improved, which justifies the effectiveness our proposed method by jointly considering local and global alignment.
information learning, better
Effectiveness of domain-invariant loss. We are also interested impacts of on different tasks. The results can be shown in Figure 3 As we can observe, for the skin lesion (on DMF) and epithelium-steee saitheliumsttrmma (on IHC) classification tasks, the loss curves with iterations reflect the global discrepancy converges faster than local discrepancy, while the more challenging cross-domain task converges more slowly on global alignment.
losses
Results on DG Benchmark. While our proposed method is designed for the context of insufficient data, it can also be applied to the setting of conventional DG problem generalization. Here, we introduce three DG benchmarks, namely PACS, OfficeHome and VLCS, for further comparison. Compared with some large-scale benchmarks (e.g.. DomainNet and Wilds), these two datasets are more appropriate to explore the effectiveness of different DG models under the scenario of insufficient samples. We report the results on PACS in Table 5 We compare our proposed method with some state- of-the-art DG methods. To be fair, all methods adopt a same backbone, i.c., the pretrained ResNet50. "Bayesian" model does not have any alignment compared with our model. As we can see, our proposed method outperforms recent methods, such as DNA and SWAD. Specifically, our proposed method surpasses the gradient operation-based method (e.g.. RSC). Although data generation methods (e.g.. MixStyle) can effectively tackle the insufficient sample problem via additional generative samples, the lack of effective domain inveriinivaraant learning may hamper the improvement of the performance.
CONCLUSION
In this work, we address the domain generalization problem in the context of insufficient data from source domains. Benefiting from the learned representation captured by probabilistic models, our proposed method marriage the measurement the distribution over distributions by level-2 kernel and probabilistic contrastive semantic alignment. Extensive experiments on challenging medical image tasks indicate the effectiveness of our proposed method.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We believe that there is no ethics issue in our work. The reasons are provided as follow. First, no personal private information exists our adopted medical imaging data. Second, access these medical imaging data is feasible by signing an agreement form with the provider or download datasets directly from our given website link in the main content. Third, our submission, we focus on the problem of domain generalization instead of long-tai//mballen tagg///aiiimaanceeeeeeeeeeeeeeee data classification. We therefore follow the previous work on these medical imaging datasets.
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- cralization using meta-rogularization. Advances in neural information processing systems, 31, 2018.
Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 6500-6508, 2019.
Shai Ben- John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces probability and statistics. Springer Science & Business Media, 2011.
Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):
46-100, 2021.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859 8771 2017.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):c4--e5) 2006.
Yuheng Bu, Shaofeng Zou, Yingbin Liang, and Venugopal V Veeravalli. Estimation of kl divergence:
Optimal minimax rate. IEEE Transactions on Information Theory, 64(4))2648-2674, 2018.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Scunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-200 224188, 2021.
Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei. Dna: Domain generalization with diversified neural averaging. In International Conference on Machine Learning, pp. 4010-4034. PMLR, 2022.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via -adel-ag learning of semantic features. Advances in Neural Information Processing Systems, 32, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep lcarning. In international conference on machine learning, pp. 1050-1059,
PMLR, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain. Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-ad adversaril training of neural networks. The journal of machine learning research, (7(1):2096-22) 2016.
10

---

Under review as a conference paper at ICLR 2023
Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. Dlow: Domain flow for adaptation and generalization. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2019.
Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 2020.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In Uncertainty in Artificial Intelligence, pp. 292-302. PMLR, 2020.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In European Conference on Computer Vision, pp. 124-140. Springer, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 8661-18673, 2020.
Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Self-halanced learning for domain generalization. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 779-783, IEEE, 2021.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizzttoon trick. Advances in newral information processing systems, 28, 2015.
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Specifying weight priors in bayesian deep neural networks with empirical bayes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4477-4484, 2020.
Ranganath Krishnan, Pi Esposito, and Mahesh Subedar. Bayesian-torch: Bayesian neural network lay- for uncertainty estimation, January 2022. URL
5908307
David Krueger, Ethan Caballero, Joern- Jacohsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Junghwan Lee, Cong Liu, Junyoung Kim, Zhchuan Chen, Yingcheng Sun, James R Rogers, Wendy Chung, and Chunhua Weng. Deep learning for rare disease: A scoping review. medRxiv. 2022.
Chenxin Li, Xin Lin, Yijin Mao. Wei Lin, Qi Qi. Xinghao Ding, Yue Huang, Dong Liang, and Yizhou Yu. Domain generalization on medical imaging classification using episodic training with task augmentation. Computers in Biology and Medicine, 141:105144, 2022a.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference artificial intelligence. volume 32, 2018a
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 201 8b.
Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang, Tie-Qiang Li, and Alex Kot. Domain generaliza- tion for medical imaging classification with dinear-dependency regularization. Advances in Neural Information Processing Systems, 33:3118-3129 2020a.
Menghan Li, Bin Huang, and Guohui Tian. A comprehensive survey on 3d face recognition methods. Engineering Applications of Artificial Intelligence, 110: 100669 2022b.
11

---

Under review as a conference paper at ICLR 2023
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and LINGYU DUAN. Uncertainty modeling for distrii-iitiitttinnn generalization. In International Conference on Learning Repre- sentations, 2022c. bttts:::////eetrteee
net www.teeeeeeeintieeeeeeeee forum?id=SHN LHyZGO
Xiaoxiao Li, Yufeng Gu, Nicha Dvomek, Lawrence H Staib, Pamela Ventola, and James Duncan. Multi-site fmri analysis using pressevgi federated learning and domain adaptation: Abide results. Medical Image Analysis, 10176,,, 2020b.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 1980-29883 2017.
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pp.
1013-1023, 2021.
Xiaofeng Liu, Chaehwa Yoo, Fangxu Xing, Hyejin Oh, Georges El Fakhri, Je- Won Kang. Jonghye Woo, al. Deep unsupervised domain adaptation: A review of recent advances and perspectives. APSIPA Transactions on Signal and Information Processing, 11(1), 2022.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. International conference on machine learning, pp. 2208-2217. PMLR, 2017.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pp. 7313-7324, PMLR, 2021.
Ankur Mallick, Chaitanya Dwivedi, Bhavya Kailkhura, Gauri Joshi, and T Yong-Jin Han. Deep ker- nels with probabilistic embeddings for small-data learning. In Uncertainty in Artificial Intelligence, pp. 918-928. PMLR, 2021.
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pp. 57 5715-5725, 2017.
Muhammad F Mridha, Abu Quwsar Ohi, Md Abdul Hamid, and Muhammad Mostafa Monowar, study on the challenges and opportunities of speech recognition for bengali language. Artificial Intelligence Review, 55(4):3431-3455 2022.
Krikamol Muandet, Kenji Fukumizu, Francesco Dinuzzo, and Bernhard Schölkopf. Learning from distributions via support measure machines. Advances in neural information processing systems, 25, 2012.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends@ in Machine Learning,
2017.
Hyconseob Nam. HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8690-8699, 2021.
Radford M. Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Oren Nuriel, Sagie Benaim, and Lior Wolf. Permuted adain: Reducing the bias towards global statistics in image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9482-9491. 2021.
Qi Qi, Xin Lin, Chaoqi Chen, Weiping Xie, Yue Huang, Xinghao Ding, Xiaoqing Liu, and Yizhou Yu. Curriculum feature alignment domain adaptation for epithelium-sttrmma classification in histopathological images. IEEE Journal of Biomedical and Health Informatics, 113-)21 2020.
12

---

Under review as a conference paper at ICLR 2023
Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. Latent independent excitation for generalizable sensor- based cross-person activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11921-11929, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /91 I. /,0731, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450. Springer, 2016.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi. Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 6438-6447. PMLR, 09-15 Jun 2019. URL https:/ httphhsss//prcccednnngg.oooeeeee .algggsmll...... mmlr.press//97/ veera 1 9a. html
Yufei Wang. Haoliang Li, Lap-pui Chau, and Alex Kot. Embracing the dark knowledge: Do- main generalization using regularized knowledge distillation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2595-2604, 2021.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
Zehao Xiao, Jiayi Shen, Xiantong Zhen, Ling Shao, and Cees Snoek. A bit more bayesian: Domain- invariant learning with uncertainty. In International Conference on Machine Learning, pp. 11351- 11361. PMLR, 2021.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7097-7107, 2022.
Chris Yoon, Ghassan Hamarneh, and Rafeef Garbi. Generalizable feature learning in the presence of data bias and domain class imbalance with application to skin lesion classification. In International Conference on Medical Image Computing and Compuler-Asiiste Intervention, pp. 365-373.
Springer, 2019.
Yuya Yoshikawa, Tomoharu Iwata, and Hiroshi Sawada. Latent support measure machines for bag-of-words data classification. Advances in neural information processing systems, 27, 2014.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentllli, Kurt Keutzer, and Boqing Gong, Domain randomization and pyramid consistency: Simulation-to generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 2100-21 2019.
Syed Sahil Abbas Zaidi, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Asghar, and Brian Lee. A survey of modern deep learning based object detection models. Digital Signal Processing, pp. 103514, 2022.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. European conference on computer vision, pp. 561-578. Springer, 2020.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2 N00.::2000000008, 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS OF BAYESIAN NEURAL NETWORK
For our proposed method, the Bayesian layer refers to the probabilistic extractor Qo and the proba- bilistic classifier C.. Here, a simple and convenient PyTorch library, namely Bayesian Torch Krishnan al. (2022), is utilized construct the Bayesian neural network. The log evidence lower bound (ELBO) cost function, i.e.,
C {= golog(ulo, w)dw (KT.gg(w))p(()))
(10)
can be calculated automatically. By using AnyesianTorch, arbitrary deterministic models can be converted into the Bayesian layers easily. In this paper, mean-field variational inference (MFVI) Graves (2011) is adopted, where the parameters of the model are characterized by fully factorized Gaussian distribution endowed by variational parameters μ and 0, i.e.,
(11) By using stochastic gradient descent method with ELBO cost, the variational distribution go(w) as the approximation of the posterior distribution, and corresponding parameters (μ and Ø) and can be learned conveniently.
go(w) = N (wwlpy (wlls,o).
For the settings of Bayesian layer, we follow the model priors with empirical Bayes using DNN (MOPED) method for the parameter settings of weights prior, each weight is sampled from the Gaussian distribution independently Krishnan et al. (2020).
w N(WDNN 8/WDNN));
(12)
where WDNN denotes the mean of prior distribution from the maximum likelihood estimates of weights from deterministic deep neural network. 8, a hyperparameter, set to the initial perturbation factor for the percentage of the pretrained deterministic weight values. The variational layer is modeled using reparameterization trick. The MOPED can realize better training convergence for complex models Krishnan al. (2020), which is beneficial to our proposed method. In this paper, follow the setting in (Krishnan et al. 2020) to set the initial perturbation factor d for the weight to 0.1.
A.2 EXPERIMENTAL DETAILS OF SKIN LESION CLASSIHCATION
Dataset Details. There are seven classes of skin lesions, including melanoma (mel), melanocytic nevus (nv), dermato broma (df), basal cell carcinoma (bec) benign keratosis (bkl), vascular lesion (vasc), and actinic keratosis (akiec). The public skin lesion datasets suffer from an insufficient data problem from some (certain) source domains. For example, the PH2 and UDA datasets only have 200 and 601 skin lesion images, respectively. The number of images for each domain can be found in Table 9 More details of datasets can be found in Yoon et al. (2019). For inputs, all images resized into 224 x 224 for all methods.
Implementation Details. The pretrained ResNet18 is introduced as the backbone for all methods. For our proposed method, the structure of Bayesian layer in probabilistic extractor Q is a fully- connected based Bayesian neural network with 512 x 512. Note that the DSU, BDIL, DNA, and our proposed method are constructed based on SWAD framework. The hyperparameters of SWAD follow default settings Cha al. (02TT. The DSU can be regarded as the uncertainty version of SWAD with ResNet18. The structure of Bayesian layer in probabilistic classifier C. is also a fully-connecte based Bayesian neural network with 512 x 7. The construction of Ca is the same as that of T Due to the class imbalance problem, the focal loss Lin et al. (2017) as the classification objective is introduced for all methods.
During training, our proposed method is optimized by Adam optimizer with 5 x 10-5 learning rate. The training steps are 2000. For each step, we randomly sample from each training source domain with 32 samples to construct the mini-batch. To evaluate the testing set, the training process is stopped according to the validation loss computed by SWAD on the validation set. The hyperparameters are also selected in wide range the validation set. For the probabilistic MMD, level-1 and leve-2 kernels are the Gaussian RBF kernels by following (Muandet et al.) 2012). The kernel bandwidth is empirically set to 1 for all kernels. For the probabilistic CSA loss, the distance margin & is set
14

---

Under review as a conference paper at ICLR 2023
Table 6: Domain generalization results on gray matter segmentation task. For the DSC, CC, TPR. and JI, the higher the better. For the ASD, the lower the better.
(a) DespAll
(b) KDDG
source
target
DSC
CC
J1
TPR
ASD
source
DSC
CC
J1
TPR
ASD
2,3,4
1
0,8560
65.34
0.7520
0.8746
0.0809
2,3,4
1
0.8745
20.75
0.7795
0.8949
0.0539
1.3.4
2
0.7323
26.21
0.5789
0.8109
0.0992
1,3,4
0.8229
56.71
0.6997
0.8226
0.0490
1,2,4
0.5041
-209
0.3504
0,4926
1.8661
1,2,4
0.5676
0.3866
0.5904
1,2805
1,2,3
0.8775
71.92
0.7827
0.8888
0.0599
1,2,3
0.8894
75.06
0.8011
0.9222
0.0377
Average
0.7425
-1.4
0.6160
0.7667
0.5265
Average
0.7886
34.86
0.6667
0.8075
0.3553
LDDG
farget
DSC
CC
JI
TPR
ASD
target
DSC
CC
J1
TPR
ASD
2.3,4
0.8502
64.22
0.7415
0.8903
0.2274
2.3,4
0.8708
69.29
0.7753
0.8978
0,0411
1,3,4
2
0.8111
53.04
0.6844
0.8161
0.0826
1,3.4
0.8364
60:58
0.7199
0.8485
0.0416
1.2.4
0.5285
-99.3
0.3665
0,5155
1.854
1,2.4
0.5543
-71.6
0.3889
0.5923
1.5187
1,2,3
0.8938
76.14
0.8083
0.8991
0.0366
1,2,3
0.8910
75.46
0.8039
0.8844
0.0289
Avorage
0,7710
23.52
0.6502
0,7803
0.5505
Avorage
0.7881
33.43
0.6720
0.8058
0.4076
saurce
target
target
0.8739
70.32
0.7794
0,9210
0.0793
0.8786
71.57
0.7873
0.9293
0.0422
1,3,4
2
0.8474
63.58
0.7367
0.8502
0.0494
1,3.4
2
0.8485
63.78
0.7389
0.8401
0.0401
1,2,4
3
0.5574
-70.4
0,3923
0.6097
1.5049
1,2,4
3
0.5634
-68.0
0.3992
0.6103
1.2239
0.8018
0.0415
0.8921
Average
0.7921
34.65
0.6775
0,8225
0.4362
Average
0.7987
35.76
0.6828
0.8260
0.3356
to 1. For the Lloeat and the Lytobal, the B3 and B2 are 0.1 and 0.7, respectively. By balancing the performance and computational efficiency, T, the number of Monte Carlo sampling in each Bayesian layer, is 10.
A.3 EXPERIMENTAL DETAILS OF EPITHELIUM STROMA CLASSIFICATION
Dataset Details. There are two types of basic tissues, i.e., the epithelium and the stroma. Due to the differences of the scanner, the staining type, and the population, the color of the background and the morphological structure among different histopathological image datasets are diverse. The number of images for each domain can be found in Table 9 The extract epithelial or stromal patches are resized into 224 x 224.
Implementation Details. The pretrained ResNet18 utilized as the backbone for all methods. The basic model framework and corresponding parameters for proposed method similar with the settings mentioned in A.2 The DSU, BDII, DNA, and our proposed method are constructed based on SWAD framework with DomainBed benchmark. The classification objective is the cross-entropy loss with softmax function.
During training, our proposed method is optimized by Adam optimizer with 5 10-5 learning rate. The training steps are 4000. The holdout fraction rate for DomainBed is set to 0.2 for all methods such that the hyperparameters can be selected in a wide range on the validation set. The 31 and 83 are 0.1 and 0.7 for the Llocat and the Lytobal, respectively. Other hyperparameters are the same as the settings mentioned in A.2
A.4 EXPERIMENTAL DETAILS OF SPINAL CORD GRAY MATTER SEGMENTATION
Dataset Details. The spinal cord gray matter (GM) segmentation is an emergent task that can be utilized to predict disability (as a biomarker) via evaluating the atrophy of GM area. The acquired magnetic resonance imaging (MRI) data are collected from four healthcare centers (including "sitel", "site3". and "site4"), where acquisition manufacturers (including Philips Achieva. Siemens Trio, and Siemens Skyra) and imaging protocols (lead to the difference in the resolution of the voxel) variable. The number of images for each domain can be found in Table g By following (Li et al.
15

---

Under review as a conference paper at ICLR 2023
Table Out-of-domain accuracies (%) ) on OfficeHome based on ResNet50.
Algorithm
Art
Clipart
Product
Real
Avg
Mixstyle Zhou et al.
(2021)
51.1
53.2
68.2
69.2
60.4
RSC Huang et al. 2020)
60.7
51.4
74.8
75.1
65.5
DANN Ganin et al. 20066
59.9
53.0
73.6
76.9
65.9
GroupDRO Sagawa et
(2099
60.4
52.7
75.0
76.0
66.0
MTL Blanchard ct al
61.5
52.4
74.9
76.8
66.4
VREx Krueger ct al.
2021
60.7
53.0
75.3
76.6
66.4
MLDG Balaji ct al. (2018)
61.5
53.2
75.0
77.5
66.8
SagNet Qian al.
2021
63.4
54.8
75.8
78.3
68.1
CORAL Sun & Saenko
2016
65.3
54.4
76.5
78.4
68.7
SWAD Cha al. 2021
66.1
57.7
78.4
80.2
70.6
DNA hu ct al.
20/2)
67.7
57.7
78.9
80.5
71.2
Bayesian
67.0
58.0
79.3
80.4
Ours
68.2
58.9
80.2
80.7
72.0
Table 8: Out-of-domain
a accuracies () on VLCS based ResNet50. wom.
-
Algorithm
Avg
Mixstyle Zhou et
2021
28.3.
64.8
72J
74.3
ZZ4
RSC Huang 2020
97.9
62.5
72.3
75.6
72.1
DANN Gamin el al 20166)
99.0
65.1
73.1
.72.2
78.6
GroupDRO Sagawa ct
2019
97.3
63.4
69.5
76.7
76.7
MTL Blanchard et al (001)
97.8
64.3
71.5
75.3
72.2
VREx Krueger et al. (2221
98.4.
64.4
74.1
76.2
78.3
MLDG Balaji
(2018
97.4
65.2
71.0
75.3
77.2
SagNet Qian et all (021))
27,9.
64.5.
714.
22.5
77.8
CORAL Sun
See Saenko 2016
98.3
66.1
73.4
77.5
78.8
SWAD Cha el al. 2021
98.8
63.3
75.3
79.2
79.1
DNA Chu et al.
98.8
63.6
74.1
79.5
79.0
Ours
98.9
63.4
75.8
79.8
79.5
2018b), the 3D MRI data are split into 2D slices in axial view. Then, these obtained 2D slices are centered cropped to 160 160 and randomly cropped to 144 x 144 for training.
Implementation Details. The 2D-Unet Ronneberger al. (015) is leveraged as the backbone for all methods. For our proposed method, probabilistic extractor Qo constructed by two Bayesian-basee 1 1x convolutional layers. The input and output channels in the first convolutional layer are both 64. After a ReLU layer, the input and output channels the second convolutional laye are 64 and 1, respectively. The Bayesian Torcc Torch enable to convert ordinary convolutional layer into Bayesian convolutional neural network easily. The Bayesian neural network adopts MFVI to approximate the posterior distribution of weights. The parameters of Bayesian layer are the same as aforementioned settings. The structure of Bayesian layer in probabilistic classifier C. is a Bayesian- convolutional layers. The input and output channels 64 and respectively. The construction of C. is the same as that of Tg. Here, all methods adopt a two-stage scheme for coarse- to---to-ffie -infine segmentation, as used in (Li et al. 2020a). Specifically, first conduct preliminary segmentation to obtain the spinal cord area from the original 2D slice. Then, we perform elaborative segmentation on obtained spinal cord results to derive gray matter results.
1 x 1
Here, the settings of most hyperparameters follow (Li et al. 2020a), where the Adam optimizer is utilized with learning rate as 10-4, weight decay as I X 10 We randomly select slices from each source domain to construct the mini-batch. All models where the learning rate will be decreased each 80 epochs with a factor of 10. Other hyperparameters such kernel function, kernel bandwidth and distance margin similar with the settings in skin lesion classification and epithelium-strom classification. The segmentation can regarded as the pixel-level classification. For the Liocat and Ligibbll, follow (Motiian et al., 2017) to randomly
trained with 200 epochs,
16

---

Under review as a conference paper at ICLR 2023
Table 9: The details of adopted datasets
Task
Datasets (Domains) and Corresponding Size
Number of Class
10051 DMF 1212; DTP M55.5M3
Skin Lesion Classification
UDA PH 9251
Epithelium Stroma Classification
NKI: 671; 1376
VGH 615
Spnal Cord GM Segementation
sitel: 30; site2: 113: site:: 246; site4: 122
2 levx)l
PACS
Art: 2048; Cartoon 2344, Photo: 1670; Sketch:3929
A total of around 15500 images for 4 domains.
Ofti
65
Cliprllii Product, and Real with ground 3897 per dormin)
VOC2007 3376; LabelMe (L) 2656;
VLCS
SUND9K)) C1ltech111 (C) 1415
sample some positive and negative pairs from two domains such that the computational efficiency can be improved significantly. Here, we randomly sample 400 positive and negative pixel pairs from two domains in a mini-batch for the computation of Ltocat, respective. By leveraging selected pixels of domain in Ltocat. we further utilize these pixels to calculate the Lglohat, which may induce a more accurate measurement owing the balanced class distribution, as well as reducing computational cost. For the Llocal and Lylobal, the 31 and the B3 are set to 0.01 and 0.001.
Result Analysis. Dice Similarity Coefficient (DSC), Jaceard Index (JI), and Conformity Coefficient (CC) are used to measure the accuracy of obtained segmentation results. Besides, True Positive Rate (TPR) and Average Surface Distance (ASD) are introduced as complementary evaluations from statistical and distance-based perspectives. The experimental results are shown in Table Gin in details. As we can see, our proposed method can achieve best or second-best performance in all task. For average results, our proposed method and DSU roughly achieve the best and second-best performance, especially in the DSC, JI. and TPR, which may be reasonable. Specifically, DSU introduce the multivariate Gaussian distribution of feature statistics for the uncertainty of the feature. Our proposed method not only can model the uncertainty by the introduction of Bayesian neural network, but also can learn distribution bbased Nomain-invariinn representations in latent feature space.
A.5 EXPERIMENTAL SETTINGS AND ADDITIONAL RESULTS ON BENCHMARKS
Besides PACS benchmark dataset, we further validate the effectiveness of our proposed method on two popular benchmark datasets, including OfficeHome (has 15588 samples with 65 classes from four domains) and VLCS (has 10729 samples with 5 classes from four domains). The number of images for each domain can be found in Table We adopt pretrained ResNet50 as the backbone for all benchmarks. The structure of the overall framework is similar with the model mentioned in lesion skin classification. Our proposed method as well as baseline methods are all based on DomainBed, where the holdout fraction (the proportion of validation set) rate for DomainBed is set to 0.2 for all methods. A domain is the target domain and the remaining domains are the source domain for training. The testing is on the overall data of a target domain.
Here, our proposed method optimized by Adam optimizer with learning rate as x 10-5, The batch size for each source domain is 32. The training steps are set to 20000 for PACS and OfficeHome, and 2000 for VLCS. By following the SWAD framework, the training process will be stopped for our proposed method when the validation loss increases significantly. The hyperparameters are selected wide range on the validation set. For the Liocat and Leglobul, the 31 and the 82 are set to 0.1 and for all benchmark datasets. Other hyperparameters such as kernel function, kernel bandwidth and distance margin are similar with the settings mentioned before.
The experimental results OfficeHome and VLCS can be shown in Table 7 and Table g As we can see, our proposed method achieves better performance compared with state-of-thee art methods, such as SWAD and DNA. Compared with proposed method has a significant improvement due to the introduction of probabilistic framework. Meanwhile, the model, namely "Bayesian" Table 7 can be regarded as a probabilistic version of
based approaches (c.g., DANN), our
17

---

Under review as a conference paper at ICLR 2023
76.5
76
ACDES)
74-
73
Numbar Monta Carlo samples
Figure 4: The performance of our proposed model the NKI task of Epithelium Stroma classification with different Monte Carlo samples T.
77
Embeddinn
Kernel Mean Embedding
ACC/S)
75
74
Number Monte samples
(a)
-+-
Mean Embedding
76
Embedding
AC025)
75
10
20
Number Mante Carlo samples
(b)
Figure 5: The performance comparison between mean embedding method and kernel mean embed- ding method with different Monte Carlo samples T. For each sub-figure, we use only one alignment operation. (a) Local alignment. Mean Embedding: The mean embedding operation with Euclidean distance is utilized between probabilistic embedding pairs. Kernel Mean Embedding: The kernel mean embedding with MMD distance is utilized between probabilistic embedding pairs. (b) Global alignment. Mean Embedding: The mean embedding operation with MMD distance is utilized between domains (as distributions). Kernel Mean Embedding: The kernel mean embedding with P-MMD distance utilized between domains (as distributions over distributions).
SWAD via replacing deterministic layers with Bayesian layers. Interestingly, compared with SWAD, a significant improvement can be obtained by Bayesian model (which does not have any alignment compared with our model), which shows the effectiveness of probabilistic framework on insufficient data. Our proposed method also outperforms the data augmentation-basee approach (e.g., Mixstyle).
A.6 ADDITIONAL ANALYSIS
First, it is much important to balance the number of Monte Carlos samples and the computational efficiency. On the one hand, the property of probabilistic embeddings can be affected by the Monte Carlos sampling. On the other hand, too many Monte Carlos samples may suffer from the heavy computational cost. (Xiao et al., (02T) suggested that the distributional property and computational cost are both acceptable for the computation of the KI. divergence when the number of Monte Carlos
18

---

Under review as a conference paper at ICLR 2023
samples is chosen appropriately, the practical performance for our proposed method need to be explored. We conduct the experiments on the NKI task of Epithelium-Stree -Srroaa classification with different Monte Carlo samples T.
The results are shown in Figure + As we can sec, if the number of Monte Carlo samples is too small, it is difficult to capture the property of distribution for probabilistic embeddings. As the increase of T, there an obvious improvement for our proposed method. Interestingly, the performance is gradually saturated. As a result, by balancing the number of Monte Carlos samples and the computational efficiency, the number of Monte Carlos samples T in each Bayesian layer is set to 10.
Kernel Mean Embedding (level-2 kernel) v.s. Mean Embedding. Second, we explore the effect of different schemes for probabilistic embeddings. A straightforward method is first to represent probabilistic embeddings with the expectation (i.e., first moment), which is called as the Mean Embedding. Then, probabilistic embedding can be regarded as a latent point, and the MMD can be leveraged to measure the discrepancy between distributions consisted of latent points.
For the mean embedding baaggd--bbddda- Cglobal, the computational process of this scheme for MMD distance can be formulated as
Ry Σ n
T1e ne Σ
1
MMD(P,,P))
(E
(13)
The Eq. 13) can be further constructed a global alignment loss Lgiobal- For the local alignment loss Clocat, the Euclidean distance can be used compute the distance between latent points, which is similar with original CAS loss in (Motiian et al. 2017). For the positive pairs with the same label, the mean embedding- based positive contrastive loss Llocat can be represented as
1 T
Local 2
E Me ( )))
(Zn;
E (e }))
s.l.yn Yu:
(14)
T
where (- denotes the embedding network metric learning. For the negative pairs with the different labels, the negative contrastive loss is denoted by
1 T
1
T Σ
Gloat
max 00,
E Me (zn.))
E Me bz II)
1.s.t.y + Yg.
(15)
As a result, a mean embedding- based contrastive loss with the view of local alignment can be calculated as
Liocal
-loeat + Cloeal
(16)
Instead, can observe from Figure 2 that our proposed method induces a level-2 kernel-based MMD with empirical estimation for probabilistic embeddings. Specifically, our proposed scheme can preserve higher moments of probabilistic embedding via nonlinear level-1 kernel (see the fourth component in Figure 2 Moreover, by introducing level-2 kernel, the similarities between probabilistic embeddings also can be measured based on their own moment information (see the last component in Figure 2). Benefiting from these virtues, the proposed probabilistic MMD can accurately capture the discrepancy between mixture distributions via an extended empirical MMD fashion.
Here, we validate the effectiveness of different schemes on the NKI task of Epithelium Stroma classification in each aligned view. The experimental settings are similar for different methods. The experimental results can be found in Figure 5 As we can see, our proposed method achieves consistent improvements in each alignment method with different Monte Carlo samples, which may be reasonable as the kernel mean representation can preserve many statistical components due to the injective property. Second, when the number of MC samples is 10, can observe obvious margin global alignment, which refers to the computation between mixture distributions. ~
Finally, we also interested in the performance of the proposed method under challenging small data scenarios compared with baseline methods. As a result, we conduct two kinds of experiments with different conditions skin lesion classification, including fixed number of samples per class
19

---

Under review as a conference paper at ICLR 2023
Table 10: Fixed number of samples per class in each source domain.
Number sample
DeepAll
DSU
BDIL
DNA?
Ours
per class for each source domain
40
0,5399 =0,056
0,6145
+0.0175
0.5897
#0 0029
0,5412
01.013
0.6368 +0 0074
30
0,5309 0200
0.5458
+0.011
0.5762
+0 01.0101
9.5132
0229
0.6138 + 0291
0,5044 #0.0129
0.5243
1:001.3
0.5573
10.0011
0,5048 0.0088
0.6037
10.0121
Table 11: Fixed fraction of samples in each source domain
Fraction of sample
DeepAll
DSU
BDIL
DNA
Ours
for, each source domain
100%
0.6674 #0,0112
0.6935
0121
0.7059
+0.0284
0.7121
#0.014]
0.7276.=0.012 =0,0133
80%
0.6614 0.0013
0.6717
0009
0.0002
0.6591
+0,0022
0.6975
+ 0030036
60%
0.6249 +1202
0,6299
00.0114
0.6468
+ 0.00
0.6149
+0.0112
0,6641
+ +0.0114
40%
0,5911 #0.0215
0,6188
+0.054]
0.6491
#0.017
0.6065
=0.0111
0.6579=0.0055 00.0057
in each source domain and a fixed fraction of samples in each source domain. We choose MSK dataset as the target domain and the remaining domains as the source domains.
Fixed number of samples per class in each source domain. Specifically, we randomly draw T samples from each class in a source domain to represent this domain for training. Here, we set T to 20,30, and 40, respectively, in different experiments. The experimental settings are the same as the descriptions in A.2 except for the training steps as 600.
The results can be found in Table 10 As can see, proposed method achieved the best performance among all settings compared with all baseline methods. Mcanwhile, it seems that the seyesian-basee DG approaches (c.g., our proposed method and BIDL) have better performance compared with other methods, which is reasonable as the BNN can be adaptive to the small data scenario well. Especially, our proposed method has around 5% improvements compared with the second-bes, best method when T' is set to smaller, i.e., 20.
Fixed fraction of samples in each source domain. Specifically, we randomly draw C% samples from the training samples of each source domain to represent this domain for training. For example, D7P dataset has 963 training samples originally. The total number of samples for this domain is 40% x 963 = 385 for training when C/ is set to 40%. Here, we set C' to 40%, 60%, and 80%, respectively, as separately different experiments. Note that C' can not be set too small (as the number of samples in some classes of some domains is very limited.), otherwise the batch size can not be uniform. We can observe this kind of setting is challenging as the total number of samples for each domain is gradually small. The results can be found in Table 11 We can observe from Table IT that our proposed method also achieves a relatively stable and better performance compared with baseline methods, as the decrease of fraction of samples in each source domain.
A.7 DISCUSSIONS
In this paper, the definition of small data is based on a specific task, including the difficulty of prediction, the quality of on-hand images, the number of source domains, and so on. Small data scenarios may be a relative concept. Specifically, a small data scenario not only can represent the number of training examples is small among all source domains compared with some large volumes of datasets but also can reflect the number of training examples is relatively smaller in some (certain) domains. On the abovementioned conditions, it may be difficult to ensure reliable contrastive semantic loss with point-wise (or local) alignment and distribution wise (or global) alignment because both of them require sufficient samples among source domains. Our proposed method aims improve
20

---

Under review as a conference paper at ICLR 2023
performance over the abovementioned small data scenarios, which is significant motivation for this work.
21


Reviews:
[{"review_id": "sPUqHPYsgTQ", "reviewer": "Reviewer_s3JK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The problem of domain generalization in the context of small samples is discussed. This is particularly important in digital pathology.\nNovel probabilistic extensions of classical MMD and contrastive semantic alignment loss provide additional features. Evaluations on standard pathology datasets show reasonable performance improvements.", "summary_of_the_review": "This paper addresses an important challenge in digital pathology, domain generalization when sample size is small. The proposed approach is solid with good experimental results and reasonable improvements over SOTA.", "strength_and_weaknesses": "Strengths: Domain generalization for small sample cases is an important problem, especially in digital pathology. Probabilistic extensions of MMD and contrastive semantic loss make the paper more interesting. The domain of digital pathology is ideal for testing the efficacy of the proposed approach. Experimental results are convincing.\nWeaknesses: The authors are not the first to come up with a measure to evaluate the discrepancy between mixture distributions (i.e., source domains). Please refer to Y. Balaji, R. Chellappa and S. Feizi, “Normalized Wasserstein for Mixture Distributions with Applications in Adversarial Learning and Domain Adaptation”, Proc. Intl. Conf. on Computer Vision, Seoul, South Korea, Oct. 2019 for a related work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "jL2gymnqKMH", "reviewer": "Reviewer_6Mag", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with the problem of domain generalization in the case that we have only insufficient source domain examples and proposes a new loss function called probabilistic maximum mean discrepancy (p-MMD). The proposed loss function introduces a kernel function \\kappa for kernel means of latent embeddings and applies a standard MMD for the kernel \\kappa. If the kernel \\kappa and a kernel k for latent embeddings z are both Gaussian, the proposed loss function can be computed by the kernel k.", "summary_of_the_review": "I am not so positive about this paper due to the weaknesses presented in this review, namely (1) the mismatch between the targeted problem and the proposed method and (2) the lack of significant experimental conditions. I believe that the proposed probabilistic MMD would be useful for some tasks. I strongly recommend the authors reconsider problems for which the proposed method is suitable.\n\nAfter author feedback: I acknowledge the feedback; however, I would keep the original review score.\n- I agree with Reviewer BRdo in terms of the redesign of experiments.\n- Significant experimental conditions should be clearly presented in the main content. Or, some pointers to the appendices should be presented in the main content.", "strength_and_weaknesses": "[Strength]\n\nS1. The problem dealt with in this paper is significant. Domain generalization is one of the fundamental problems in machine learning and computer vision, and in some cases, we cannot obtain sufficient training examples for training machine learning models for this purpose. Technically solid methods with thorough experimental evaluations will draw attention from a broad range of researchers and engineers.\n\nS2. The use of level-2 kernels for extending the standard MMD is interesting. The proposed p-MMD might be useful for other tasks emphasizing probabilistic distributions.\n\n[Weakness]\nW1. The proposed method lacks \"why\". I understand that the proposed method extends the standard MMD by introducing a kernel function for kernel means and it seems to be novel as far as I know. However, I am unsure why the proposed method can boost the performance on few-resource domain generalization.\n\nW2. The current main content lacks sufficient experimental conditions. For example,\n- What is the source / target domain data in the experiment presented in Section 4.1? --> This makes the task setting totally unclear.\n- How many training examples are available in the experiment presented in Section 4.1? --> I could not understand whether the current experimental setting can be regarded as \"few resources\" or not.\n- How many competitors include the Bayesian framework in them? --> I could not understand which contributes to the performance improvement, Bayesian neural networks or probabilistic MMD.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fsh66Y0ZR69", "reviewer": "Reviewer_BRdo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As the title suggests, this paper explores domain generalization for small data through a probabilistic maximum mean discrepancy (MMD) approach. For this purpose a domain-invariant representation is being learned from multiple domains (sources) where the assumption is that these domains inherently have insufficient samples (a.k.a small data). For the training and evaluation purpose the paper explores their claims in the medical image processing domain (skin lesion classification, epithelium-stroma classification, spinal cord gray matter segmentation) where such data limits are more clear.  They also studied and ablated each component of the proposed method including but not limited to the effectiveness of domain-invariant loss, use of global vs. local alignment, use of Bayesian layer and such. ", "summary_of_the_review": "Although I appreciate that the paper presents various experiments, the evaluations and numbers do not support the claim of the paper around small data (no significant improvement in dataset with less data), also the evaluation and experiment protocol can improve drastically (i.e. controlled sampled size). Moreover, clarity and reproducibility of paper can be improved by reorganizing some sections and presenting experimental protocol upfront and in a concise manner. \n\nAfter Rebuttal: As most of my concerns has been answered, I increased my score. \n", "strength_and_weaknesses": "**Strength:**\n- The paper in general is very well-motivated and the target problem is relevant to both the ML and Medical AI community.\n- They evaluate and ablate the method in multiple settings. \n\n**Weakness:** (see below for more details) \n- Paper can benefit from the redesign of experiment to better support and showcase the claim of the paper around small data.\n- Some details around experimental protocol and details are either missing or not presented concretely. \n- Results are marginal and I cannot observe significant improvement specifically for small datasets. \n\n**Suggestion to improve paper:**\n- One of my main concerns is around the evaluation protocol of the paper. For each application, we are dealing with multiple datasets and dataset size ranges. For example skin lesion classification task includes 7 public dataset with different dataset size as mentioned in the paper, for example UDA and PH2 includes 601 examples and 200 examples consequently. What has not been mentioned in the paper is that for example HAM1000  includes 10015 dermascopic images and more critical details around other dataset are also missing. In this experimental setting paper claims the method improves performance for small data setup where we are dealing both with the domination danger of larger dataset and also limits and boundaries of improvement is not clear.  For paper claims, I am more interested in the question of how many samples is enough to reach the baseline performance? So coming up with unification is very important, picking up the smallest dataset size, and analyzing everything in the fixed dataset size or fraction (e.g. performance with 10 samples, vs. performance with 200 samples). Such experiments are standard in representation learning literature. Does the larger dataset dominate the learned space?\n- Even in the current setting I would expect to see significant improvement given the proposed method for SON, UDA and PH2 but all I am seeing is marginal improvement given standard deviation on five runs.  With this experimental protocol at best, the paper is a domain generalization paper for medical image processing, and any claim around small data should be addressed and assessed concisely. \n- More importantly, as the paper mentioned some of at least skin lesion dataset suffers from class imbalance, long tail issue, and effect of any method should get analyzed per class and fairness consideration should be addressed. \n\n**Minor Editorial:**\n- Improving tables and image captions can help with the quality and readability of the tables. (e.g. bold and underline, number of runs to obtain std for table 2, do they run significant tests, if so, highlighting significant improvements, and more.)\n- Figure 3 is very small, moving some results to the appendix does not hurt the paper or using alternative visualizations such as bar graphs help with the quality of results.\n- Last paragraph of related work, page 3, “we noticed …” can benefit from a rewriting. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1skN9olMgVU", "reviewer": "Reviewer_RmYa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to learn a domain-invariant representation based on the probabilistic framework by mapping each data point into probabilistic embeddings. For this, a probabilistic MMD and a probabilistic CSA loss are proposed.", "summary_of_the_review": "Overall the paper seems to propose a novel approach with promising results. The quality of the paper can be improved by addressing potential shortcomings and an even more thorough evaluation on other benchmark datasets.", "strength_and_weaknesses": "The paper proposes some new methodology and shows promising results on medical datasets. But it is missing a discussion on potential drawbacks of this approach such as e.g. slower inference time. It is also missing comparison against baselines on datasets such as VLCS, TerraIncognita and DomainNet. While some of these might not showcase the strength of the proposed approach, but the expectation should be that the approach performs similarly on these compared to existing baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject