Domain-Generalizable Multiple-Domain Clustering
Authors: Amit Rozner
Barak Battash
Lior Wolf
Ofir Lindenbaum
Venue: Accepted by TMLR ()

Abstract: This work generalizes the problem of unsupervised domain generalization to the case in which no labeled samples are available (completely unsupervised). We are given unlabeled samples from multiple source domains, and we aim to learn a shared predictor that assigns examples to semantically related clusters. Evaluation is done by predicting cluster assignments in previously unseen domains. Towards this goal, we propose a two-stage training framework: (1) self-supervised pre-training for extracting domain invariant semantic features. (2) multi-head cluster prediction with pseudo labels, which rely on both the feature space and cluster head prediction, further leveraging a novel prediction-based label smoothing scheme.
We demonstrate empirically that our model is more accurate than baselines that require fine-tuning using samples from the target domain or some level of supervision. Our code is available at \url{https://github.com/AmitRozner/domain-generalizable-multiple-domain-clustering}.

Published in Transactions Machine Learning Research (02/2024)
Domainn-enerallizbbe Mulmiple-Domain Clustering
Amit Rozner"
Faculty of Engineering, Bar Ran University
Equal contribution
Barak Battash"
Facully of Engineering, Bar Ban Universily
Equal contrinution
Lior Wolf
School of Computer Science, Tel Aviv University
Ofir Lindenbaum
ofirlin (irlin@gmail. com
Faculty of Engineering, Bar Ran University
Reviewed on OpenReview: https: //https:
// openreview. net/ forum? id= O9RUANP
Abstract
This work generalizes the problem of unsupervised domain generalization to the case in which no labeled samples are available (completely unsupervised). We are given unlabeled samples from multiple source domains, and we aim to learn shared predictor that assigns examples to semantically related clusters. Eval- uation is done by predicting cluster assignments in previously unseen domains. Towards this goal, we propose a two-stage training framework: (1) self-supervised pre-training for extracting domain invariant semantic featurcs. (2) multi-head cluster prediction with pseudo labels, which rely on both the feature space and cluster head prediction, further leveraging novel prediction- ing scheme. We demonstrate empirically that our model is more accurate than baselines that require fine-ti fine-tuning using samples from the target domain some level of supervision. Our code is available at lhttps://githnn com/ Ami tRozner/ domain Seraiin-gneralline wwt.:gaeaaaaaaleeliiiiaaaaaeaaea
label smooth-
1 Introduction
Clustering high-dinensional measurements accurately is crucial for effectively analyzing scientific data. In recent years, deep learning based frameworks have shown significant improvements when compared to classical clustering models, such as K-means (Lloyd 1982) or Spectral Clustering (Ng ct al. 2001). However, the observations are collected from multiple domains, classical and deep learning clustering models may group samples based on donain-specific information, such as style, rather than their semantic content. This problem becomes even more difficult if samples from the target domain not available during training. This situation may arise due to privacy or

---

Published in Transactions Machine Learning Research (02/2024)
computational considerations and has several applications, such as face clustering on edge devices (Caldarola et al. 2021p, medical image clustering (Irshaid et al. 2022), and more.
To overcome this challenge, we develop a predictor called f that can categorize a given sample to its respective cluster, regardless of its domain. We then evaluate f on an unseen target domain where no samples were seen during training. This task requires solving the problem of multiple-domain clustering along with domain generalization. The combination of these two problems is highly valuable, especially
scientific discovery in fields such as biomedicine, where individuals have different distributions (Zhao et al. 2021). Accurately predicting new domains highly valuable in 2022. While previous studies have tackled similar problems, they rely partial supervision require access to target domain samples during training
these fields (Bac et al.) 2022 Farhadian et al.
(Zhang et al. 2021: Harary et al. 2022 Gopalan 2017
Menapace et al. 2020).
Formally, we are given a dataset 5 with N samples from d domains, where each domain has N₁, Vi {I.....d df samples. We know which sample belongs to each domain, and a classifier f trained to map every sample in one of K groups. For evaluation purposes, set of labelod examples used, and the K groups are assigned lo set of ground truth labels using best- matching method, the Hungarian method (Kuhn 1955 clarify, without further training, f is applied to each sample of T, and the predicted labels are then compared to the ground truth labels of the samples in T.
unseen single-domain dataset T. To
We propose a two-phase learning approach to tackle the challenge of clustering data from multiple domains while ensuring domain generalization. the first phase, called the preiraining slage, we use contrastive loss with several complementary elements to encourage learning domain-agnostic representations. We achieve this by using style augmentations that reduce stylistic variations and an adversarial domain loss that eliminates the influence of domain-specific content. We also introduce special queue procedure and domain balancing lo stabilize the multi-domain training scheme. In the second phase, we present several novel components to learn multi-domain clustering assignments. These include (i) a new pscudo-label selection procedure that uses a style-augmented common domain to identify reliable labels, (ii) a prediction-base mode collapse to highly populated clusters and can overcome bad initialization, and (iii) a multiple clustering head prediction and filtration procedure to stabilize the clustering step in the presence of nigh-dimensional data many categories.
label smoothing procedure that prevents
We have tested our method on various multi-domain datasets and have demonstrated its superior- ity over several strong baselines. Our framework is comparable to or better than state-of-the-art clustering methods but does not require any labeled data or adaptation to samples from the target domain. By removing these two requirements, our model opens the door to new applications where labeled data unavailable or access to target samples is infcasible due privacy constraints. Fur- thermore, our ablation studies confirm the importance of each proposed component in contributing to the model's generalization capabilities.
2 Background and related work
Sclf-supervised lcarning (SSL) is a technique for extracting usefuul representation from unla- belled data using a pretext. task. In computer vision, many methods rely on a contrastive loss; for example, SimCLR. (Chen al.) 2020a) uses paired augmentations, creating feature representations that are projected and trained to maximize agrecment. The seminal SSL framework MoCo (He
2

---

Published in Transactions Machine Learning Research (02/2024)
et 2020) introduced a queue of negative samples and a momentum-moving average encoder to improve the contrastive learning framework.
Deep clustering aims to exploit the strength of neural networks (NN) feature extractors to identify representation that better preserves cluster structures using unlabelled data. DeepCluster Caron et al. 2018) is a method that learns useful features and assigns them to clusters by applying 6-means to the extracted features and training a NN to predict the cluster assignments. ]i et al. (2018) use content-preserving image transformations to create pairs of samples with shared semantic information. Then, they train NN to maximize the mutual information between the image pairs in a probabilistic data representation. Recently, Semantic Pseudo-Labeling for Image Clustering (Niu et al., 2022) has obtained state-of-the-art results several benchmarks. This method is an iterative deep clustering approach that relies on self-supervision and pscudo-labeling. First, self- supervision is performed using a contrastive loss to learn informative fealures. Then, protolype pseudo-labeling is created to avoid common annotainntsssons in pseudo- labeling techniques.
Domain generalization (DG) is technique that involves incorporating knowledge from multiple labeled source domains into model that can perform well on unseen target domains. [Cha al.
(2021) have developed dense and overfit-aware stochastic weight sampling strategy, which seeks a flat minimum and has been found to reduce the domain gap and prevent overfitting, Zhou et al. (2021) have proposed method that aims to mix the styles of different image domains by using a probabilistic convex combination between instance-level feature statistics of early CNN layers. Li et al. (2021) uses augmentations to mitigate the domain gap by perturbing the feature embedding with Gaussian noise during training. Zeng et al. (2023) have focused on "evolving domain generalization" with additional information of the time, which helps with the class label prediction in the target domain.
Unsupervised domain generalization (UDG) was recently presented by Zhang el al. (2021 Harary et al. (2022). UDG is related to our goal but requires some supervision. Specifically, UDG involves: unsupervised training on a set of source domains, then fitting a classificr using a small number of labeled images. Ultimately, the model is evaluated on a set of larget domains that were unseen during training. Toward this goal, Zhang al. (2021) suggested ignoring domain-related features by selecting negative samples from a queue based on their similarity the positive sample domain. Harary et al. (2022) recently presented BrAD, which involves self-supervised pre-training on multiple source domains in shared ch-lik domain. To fine-tune classifier, they used varying amounts of labeled samples from source domains, In contrast, our research does not use class labels for either source or target domains during model training. Additionally, we assume no knowledge about corresponding images between domains but only that all target classes appear in each domain.
Unsupervised Clustering under Domain Shift (UCDS) presented by Menapace et al. (2020) discusses task of clustering samples from multiple source domains and adapting the model to the target domain using multiple unlabeled samples from that domain. The authors propose optimizing an information--heerrttee loss, along with domain- aaligmmtt layers, to achieve this goal. Figure | shows different approaches for multi-domain image classification. Although the UCDS setting is similar to ours, we aim to design a model that can predict cluster assignments on multiple source domains and generalize new unseon domains without any further tuning or adaptation. This is the first work that solves this task without using any labels or samples from the larget domain, which is
3

---

Published in Transactions Machine Learning Research (02/2024)
a significant advantage in real- settings where we often don't know that a domain shift occurred can't access samples from the test domain due to privacy computational considerations.
Source Domains
Target Domain
Training Data Required
Task
Source
Target
D1
D2
D3
DA
LABELLED
LABELLED
UDA
LABELLED
UNLABELLED
DG
LABELLED
UDG
LIMITED
UCDS
UNLABELLED
UNLABELLED
Ours
UNLABELLED
Figure 1: Different tasks for multi-domain image classification or clustering. Domain adaptation (DA) deals with adapting model from one domain to another and requires the availability of labeled samples in the source and target domain labels for training the model (Csurka et al. 2017]. Unsupervised domain adaptation (UDA) alleviates the need for labeled target samples and is trained using labeled source samples and unlabeled samples from the target domain (Piva et al. 2023). Domain generalization (DG) aims to create a model that generalizes across multiple source and unseen target domains. The training procedure is similar UDA but does not require access to samples (unlabeled) from the target domain during training (Roy ct al. 2019 Matsuura & Harada, 2020 Unsupervised domain generalization (UDG) does not assume access target samples but relies only limited amount of labeled data in the source domain (Harary et al. 2022 Zhang ct al. 2022) Unsupervised clustering under domain shift (UCDS) does not rely on labeled data but requires access to samples from the source and target domains for training (Menapace et al. 2020 Our fully unsupervised approach can generalize to the target domain without access Lo its samples during training.
3 Method
Our approach involves two phases. In the first phase, we train a feature extraction model, denoted as fe, in a self-supervised manner on data from various source domains. This phase helps to bridge the gap between differrnt domains by extracting semantically related features u; E R', where e represents the embedding dimension. In the second phase, we focus on training clustering head, denoted as fer while the weights of the feature extraction model Se are kept frozen. Our cluster predictions are then based on the composition of and fe, denoted as f f fe-
The feature extraction process has several pre- pre-training stages that aim to learn features that capture semantic information from each image while suppressing the style content. In order to train a cluster predictor that is invariant to the domain, we suggest the following components: data augmentation for training the clustering head, generating reliable pseudo-labels, using a clustering head training loss, smnothing based on predictions, and using multiple clustering heads. A detailed explanation of each of these steps can be found in the following subscctions.
4

---

Published in Transactions Machine Learning Research (02/2024)
3.1 Pre-training
Our pre-training strategy generalizes MoCoV2 CCen et al., 2020b for multiple domains by in- troducing four components that enable learning robust domain-agnostic embeddings. First, we introduce style transfer augmentations, and using contrastive loss, we learn a representation in- variant to stylistic variations across domains. We further introduce adversarial loss to eliminate domsin-specific information at the distribution level. To stabilize the multi-domain contrastive training process, we introduce domain-specific queues, each tailored to a distinct domain. This pro- vents pushing apart representations from different domains and encourages learning content rather than domain-specific features. Additionally, we address domain imbalance by implementing a vir- tual balancing approach mitigate the impact of population differences across domains. These pre-training components are detailed below, followed by description of the contrastive learning framework used here.
1. Style transfer augmentation: We propose incorporating style augmentations into our contrastive loss to attenuate the influence of "style" on our extracted representation. Specifically, with prob- ability Pst we perform a style transfer augmentation ST(x;). This example is then fed into a contrastive loss with 2ạ that learns a mapping such that (z;) and x; are indistinguishable. To augment with varying styles domains, we use Huang Belongie (2017); this allows us to enhance the ability of our feature extractor to ignore domain-specific features.
2. Adversarial loss: We add complementary step to remove domain-speeifie information at the distribution level. We achieve this using an adversarial domain loss (Ganin & Lempitsky 2015). Specifically, we train our feature extractor fe to learn features that are not influenced by domain identity. To do this, we 'fool" a domain classifier fa by updating the weights of fe using a constant dL12, This technique is called gradient reversal layer.
Aa 2 0 and the gradient of fa denoted as 88a This encourages the feature extractor to remove the domain-specific information. We note that the label of the domain is generally available for free; for instance, in medical applications, this could refer to the imaging technology.
-peaiic-peccfiic queues: contrastive training, negative samples are stored in queues. To avoid pushing apart the representations of different domains, we use multiple domain-specife queues (Harary et al. 2022). Specifically, we introduce domain-specific queues Q = [Q1, Q2, ... Qa) each with size N₂ for each of the d domains. The negative examples u, are drawn from the same domain as the positive sample, which makes the discrimination more challenging and encourages the model to focus on the content rather than the domain. Positive samples u, are stored in the relevant domain queue for later use as negative examples.
Domain balancing:
The populations of samples from different domains may differ significantly. Such domain imbalance cause poor generalization since the model might predominantly learn from the heavily populated domain. To address this issue, we employ virtual balancing approach by sampling domain- balanced mati-batches. This strategy helps mitigate the domain imbalance, enhancing the model's ability to generalize more effectively across the different domains.
Pretraining overview We are given an input. batch of images x = [x1, 22, 22, Fn) BBxCxwxX where C, W, H represent color, width and height dimensions. Each image 2 is trans- formed twice. First, using strong augmentation xt = S(x;). Second, the image is transformed
5

---

Published in Transactions Machine Learning Research (02/2024)
Domain head fd
Feature
fe
Features
reversal
Projection head
proj
Posltve keys stored
domain specific queue
Contrastive loss
Negative examples queue
Domain negative
Negative koys are taken from the came domain the positive examples
Domain negative queue
The proposed pre-training procedure. Each image is transformed using strong aug- mentations or style transfer augmentation. The features u" (strong augmentation) and u (style augmentation) extracted using fe- Then identity of each sample, minimizing the domain loss Líai feature extractor fe to fool the domain head in an adversarial fashion. The contrastive loss f is minimized based on the output of the projection head u², utt, and u (negative samples).
Figure 2:
domain head fa to classify the domain gradient reversal to update the
using:
(T.)),
w.p Pst.
2 = S(xx) w.p 1 Pst.
Where (Tx;)) replaces the style of 2, with another domain's style with probability Pat, otherwise, a strong augmentation S(x;) is applied to generate the positive sample. Both of and xft are passed through the feature extractor fe to create the embeddings uf = P(fe(z; 01), Op), uf (zfe ()), where P and 0p the projection head and its weights respectively. 0f μ6 + (1 0) and 6, = pd + (1 11---p1 4)), are the moving average versions of 0, and A, respectively. Finally, negative samples u, Vi [0, N₂] are sampled from a domain queue of the same domain dr, as Z4. We use the following contrastive loss:
exp((u")) wtt) INz exp((u))Tu]) u²) + exp((ue)Tuut)
[ -log
where u² [uf, u...., l Bure at [ust, un u T transformed input batch x², a.
R Rae, are the embeddings for the
To further remove domain-specific information, we use an additional domain adversarial term using a cross-entropy loss: Cia = (fa((") Hence, our complete objective is
Cr. Lspras + L
6

---

Published in Transactions Machine Learning Research (02/2024)
Feature extractor
Features
Clustering heads
( Joi
Select representitives
Wed
Pseuco label
CE loss
Figure 3: Clustering head training. The image is passed through the feature extractor Le in its original (u), strongly augmented (u"), and BCD form (uhrd), The weights of fe are frozen and used to produce the features. Reprosentatives are selected from the original image features based on the clustering head's predictions over the BCD images. The class representatives are used as pseudo labels for the CE loss.
The contrastive loss term and anveraaad loss term help learn features that are related to the content. and invariant the domain.
3.2 Clustering predictor head
The pre-training phase provides a solid and robust feature extractor fe on which clustering head fe can be trained. The clustering head should be able to separate multi-domain data into groups samples with similar semantic content while ignoring the cross-domain distribution shifts. We do not assume that samples from the test domain available during training, and want to design a clustering head that generalize to new unseen domains at inference time.
To obtain this goal, we present a clustering head training scheme for multi-domain assignments. First, we bridge domain gaps using Basic Common Domain (BCD), transforming images into a sketch-like domain that retains object identity while reducing biases from color or style. The clustering head then trained on strong augmentations and style-transferred images. Next, we generate reliable pseudo-labels based on the clustering head's predictions in the BCD (logits) (yylzhed) fe f=(x/od), and the semantic features (embeddings) age. To enhance clustering stability, we introduce multiple heads and select the most reliable ones by evaluating their diversification. Moreover, we have presented a prediction- ing technique that adjusts label uncertainties based on past prediction statistics, promoting robust lcarning across domains. Our holistic strategy ensures effective clustcring in high-dimensional image data while addressing the challenges of domain gaps and instability inherent in the clustering pro- cess. The steps involved in this method are detailed below, and the training procedure is illustrated in Figure 3
f(()) of the original im-
label smooth-
7

---

Published in Transactions Machine Learning Research (02/2024)
Data augmentation for clustering head training: We leverage a BCD to mitigate the gap between several domains. The BCD is designed to maintain the content of cach sample while removing domain-related information. A sketch-like domain can be considered a suitable BCD for image data with varying color and texture domains. Transforming an image to a sketch domain keeps high-level features such as object identity while decreasing the bias induced by color or style- preserving features. The chustering head is trained by sampling a batch of images from all source domains. Each sample F2 passes through fe in three different versions. Based on the original image, and using two transformations: strong augmentation of Ex;),, and style transfer to our BCD whcd = C(r.). Where C() represents a style transfer (see the Pre-training section) of the input image ry to an image with aketch-like style (Huang & Belongie 2017 |. E( is defined as:
C(x,)) w.p PatPied
ST(I;), w.p Pat - (Peec),,,),
((x;) w.p Pst.
The BCD transformed and the original images are used to define the pseudo labels, while the strong angmentations are used to train the clustering head fe, as detailed below.
2. Reliable pseudo label generation: First, features are extracted from the original image: u fe(2,, 01). Then, logits are extracted in the BCD based on: Hyyl=bed = f(abed = Se fa(abed). The top 7 := 2K samples are chosen from (yy 2/(d) as the set of most confident samples of class k based on the clustcring head's score samples from the BCD. Thus, the selected samples are denoted
B
as follows:
M₁ = {uili E arglop Vi E
(2)
where ("kc NY is a vector of indexes that chooses the most confident samples, based on their corresponding BCD score bee) Mr is set of y embedding vectors. Using Mr,
the center of class is determined by:
Gk
1
Σ di-
(3)
M.EME
One can calculate the similarity between each sample and each of the centers: sim; = (G 7). a
Gx_. Where G2 = [ and ū = [a] are the normalized feature and center vectors, respectively. Specifically, sim, R represents how close each sample in the batch is the center of cluster k.
Samples that are closest to the center are used as pseudo labels for cluster k; thus, the set of strongly augmented data samples with pseudo labels is formulated as follows:
2. = fa (srgtop(ssima)),
(4)
Where x (ssttoprsssmmm) means indexing x using (sito)) i.e., choosing the samples that will be used as pseudo-labels for class k using the similarity in the embedding domain. While Ma are chosen based on the predictions in the logit space, 2, are pscudo labels basod on information from both the semantic space Eqs. 3 and in the logits space Eq. 2 Using only the logit values to infer the pseudo labels results in poor cluster assignments, as demonstrated in our ablation study. The entire set of representatives and pscudo-label pairs will be denoted as 2.
8

---

Published in Transactions Machine Learning Research (02/2024)
3. Clustering head training loss: In the second phase, the clustering head fe trained using 2 minimize cross-entropy loss using the pseudo labels. A batch of samples and pscudo labels (1 Vpt) E 2, are propagated through fc, and fe, and the objective can be formulated as:
C B Lce (fe o fe(Tpt), Upt).
(5)
During this training phase, domain balancing is used, as detailed in our pre-training section.
4. Prediction-based lobel smoothing: Label smoothing (Szegedy et al., 2016) prevents classifier from being overconfident, which may lead the classifier to collapse to specific classes and neglect the rest. Using past prediction statistics, we present an improved label smoothing that varies the smoothing intensity between clusters. We perform stronger smoothing for frequently predicted clusters and vice versa. Specifically, let f(x) E {0,1)* K be the onc-hot prediction of sample æ, and we denote the empirical density of predictions over the batch Heur E R We further compule the exponential moving average of prediction density H is constant for balancing between history and current statistics. We perform our cluster-specific prediction-basee based label smoothing using H. Precisely, for cluster & E K we define the smoothed pseudo-label as Tn [k] max(1 H[k]. 8min), ) where 8min is the minimal smoothing value, and the values of the remaining clusters are spread uniformly to sum up to
H + (1 a)H with (0, 1)
Multiple clustering heads: Clustering is inherently unstable, espocially when dealing with many classes high-dimensional datasets. Several authors have proposed using feature selection (Solorio- Fernández et al. 2020 Shaham et al. 2022 Lindenbaum et al. 2021) to improve clustering capa- bilitics by removing nuisance features in tabular data. We are interested in stabilizing clustering performance on diverse digh-dimensional image data. Therefore, we propose training multiple clus- tering heads simultancously and selecting a reliable head based on an unsupervised criterion. This allows us to handle many categories and overcome the instability that stems from random weight initialization. For more details about the source of randomization between heads, please see Ap- pendix B33 The number of clustering heads is denoted h, hence the objective in Eq. L can now be formulated as the average of the h head specific losses:
h
C
Bh. Σ Ciale o S Se(App)
(6)
Next, we define the diversification of head as:
du, = (argmarne
(7)
First, argmaz reduces the prediction (y)2z) of the j-th head to cluster index. Next, the operator |-l, which counts the number of distinct cluster predictions in the set of predictions by head j; this process is performed over the entire dataset without any parameter update.
Due to high variability in the training procedure between heads, some are better than others; we leverage this variability by keeping only the most diversified heads (MDH). Two MDH are chosen out of clustering heads based on higher du, values compared the other heads. The heads with lower duy are discarded, and we replace the weights of the non-MDH with a linear combination of the two MDH weights. Mathematically speaking, let us define 02 the weights of the i-th head,
9

---

Published in Transactions Machine Learning Research (02/2024)
and let us assume that j, k are the MDH indices; hence the weights of the non-MDH heads are overridden in the following manner:
02. r202. + +r,02, Vi + k,j-
(8)
Where rkk 2023
U(0 and ri rk. Inspired by recent weight averaging works (Jain et, al. Yin et al. 2023 Wortsman et al.) 2022), this removes the influence of non-diverse heads and maintains some degree of variability for the following optimization steps.
In cases where there is equality in dry wetween several heads, which results in more than two MDHs, we limit the number of MDHS to five. The rationale behind this limitation can be elucidated through the following illustrative scenario, w.l.o.g,, assume that the first head does not predict one class, and the other heads do not predict five classes; if the number of MDH kept not limited, the advantage of the first head is not utilized. Since all heads will perform poorly in the carly training phase, MDH selection is initiated after few epochs. Furthermore, to allow the heads to make gradual learning, the process repeats every epochs.
4 Experiments
Experiments are conducted using four datasets commonly used to evaluate domain generalization methods. Representative images from several datasets and domains appear in Appendix C Of fice31 dataset (Sacnko et al.) 2010) consists of images collected from three domains: Amazon, Webcam, and DSLRR, with 2817, 795, and 498 images, respectively. The dataset includes 31 dif- ferent classes shared across all domains. The samples consist of objects commonly encountered in an office setting. PACS dataset (Li al., 2017) consists of four domains: Sketch, Cartoon, Photo, and Artpainting with 3929, 2344, 1670, and 2048 images, respectively. It includes seven different classes, which are shared across all domains. Officehome dataset (Venkateswara et al. 2017) contains four domains: Art, Product, Realworld, and Clipart, with 2427, 4439, 4357, and 4365 images, respectively. It includes 65 different classes, which are shared across all domains. The large number of domains and classes makes the task challenging. In particular, since we aim to cluster the data without access to labeled observations. Existing state-of-thc-art results on this data (Menapace et al., 2020) corroborate this claim. DomainNet dataset (Peng et al., 2019), consists of six domains: Real, Painting, Sketch, Clipart, Infograph, and Quickdraw. It includes about 0.6 million images (48K 172K per domain) distributed among 345 classes.
Baselines To evaluate the capabilities of our model, we focus on the following scheme: train the model using unlabelled source domains, then evaluate our model on the unseen and unla- belled target domain. We compare our approach to several recently proposed deep learning-baee clustering models for multi-domain data. Our implementation details appear in Appendix [..1
When evaluating Office31 and Officehome datasets, we compare with ACIDS (Menapace al. 2020 Invariant Information Clustering for Unsupervised Image Classification and Segmentation (IIC) (Ji et al., 2018), and DeepCluster |Caron et al.) 2018 Importantly, they were trained directly on the target domain before predicting the clusters. We further compare two variations of IIC, specifically, IIC-Merge involves training IIC on all domains, including the target domain; TIC+DIAL: IIC, which contains a domain-specific batch norm layer jointly trained on all domains. We further compare to continuous domain adaptation (Continuous DA) (Mancini al, 2019 which trains on d domains, then adapted and tested on the target domain. Note that all the former
10

---

Published in Transactions Machine Learning Research (02/2024)
Method
Target fine-tuned
Supervision
D,W
A,W
A. D
W
Avg
DeepCluster Caron et al. 2018
19.6
18.7
18.9
19.1
IIC (Ji et al. 2018
31.9
34.0
37.0
34.3
IIC-Merge Ji ct 2018
29.1
36.1
33.5
32.9
IIC DIAL J1
2018
28.1
35.3
30.9
31.4
Continuous DA Mancini et al.
2019
20.5
28.8
30.6
26.6
ACIDS Menapace
2020
33.4
36.1
37.5
35.6
K-means Lloyd 1982
14.9
21.3
20.8
Table 1: Accuracy results on the Office31 dataset (31 classes) upon all three domain combinations,
each of the letters A, W, represent the domains Amazon, Webcam, and DSLR, respectively. The notation X,Y - Z, means the model was trained on X, domain and tested on the Z domain. Target fine-tuned means the method was trained or adapted the test domain. In K-means, we first pre-trained the M2coV2 model and trained K-mcans top of its embeddings.
Mothod
Target fine-tuned
Supervision
C,P.R
A,P,R -
A.C,R
A,C,P -
Avg
DeepCluster Caron et 2018
8.9
11.1
16.9
13.3
12.6
IIC [Ji
12.0
15.2
22.5
15.9
16.4
1CC-Merge Ji 2018
11.3
13.1
16.2
12.1
13.3
+ DIAT. 2118
15.4
12.8
13.0
Contimnons D3 Minicnn
2019
10.2
11.5
13,0
11.7
11.6
ACIDS Menapace et al M221
V
12,0
16.2
23.9
15.7
17.0
K-means Mloyd 1982
9.1
11.3
13.8
10.6
11.2
Ours
20.8
26.2
27.7
27.2
25.5
Table 2: Results on Officehome dataset (65 classes) upon all four domain combinations, cach of the letters A, P. R, C represent the domains Art, Product, Realworld, and Clipart, respectively. The notation W, W, - Z, means the model was trained on W,X,Y domain and tested on the domain. Target fine-tuned means the method was trained or adapted to the test domain. In K K-means, we pre-train the MocoV2 model and then train K K-means on top of its embeddings.
basclines compared with our work were trainod on the target domain. We added another bascline, training MoCoV2 on all the source domains and fitting the K-means clustering algorithm the target domain.
On the PACS dataset, we also compared to BrAD Harary et al. 2022 with various amounts of source domain labels. This comparison is very challenging as we do not use any class labels. Evaluation on DomainNet (Peng et al. 2019) was done following the protocol for UDG introduced Zhang et al. (2021). Our method does not utilize labels compared to other methods, which use 1% source domain labels.
Results Table I depicts the results all three domain combinations of the Office31 dataset. Our method outperforms the current state-of-the-art (SOTA) by large margin on both DSLR and Webcam as target domains, even without adaptation to the target domain. However, our method's performance on the Amazon domain is inferior to the current SOTA, which may be due to limited source domain data. The target finc-tuned method, which relics on unsupervised fine-tuning on the target domain, uses 317% more data for their training scheme. On average, our method performs better by 10.1% than methods that use the target domain for adaptation and 31.1% over baselines with the same conditions.
11

---

Published in Transactions Machine Learning Research (02/2024)
Results on the Officehome dataset can be seen in Table 2 This dataset is more challenging than the former and consists of four domains. Our method outperforms the basclines all four domain combinations and is better average by 47.1% than the previous SOTA.
On the PACS dataset (Table 3 we compare to target fine-tuned and limited source domain label methods. Our method outperforms the current SOTA on 3 out of larget domains for the target fine-tuned case. Our method achieves slightly lower results on the fourth domain (Sketch) than the current target fine-tuned SOTA. This can be explained by a large amount of additional Sketch domain data (65%), exploited by baselines fine-tuned on the target domain. On average, our method outperforms all baselines that do not require any level of supervision. When comparing the two variations of BrAD with 1% source domain labels, achieve superior results all domains. Overall, our method outperforms BrAD-KNN, even though BrAD-KNN uses 10% of the source domain labels.
Table ! illustrates the results on DomainNet dataset (Peng et al, 2019 Our method is compared different SOTA UDG methods that use 1% of source domain labels. Although our method does not any class labels, it oulperforms all previous schemes across all domains by large margin.
Ablation study We conducted two ablation studies to evaluate model's performance. The first study explores different variations of our model, while the second study focuses on the proposed
Method
Target fine-tuned
Supervision
C,P.S
A,P,8 +0
A,C,S P A,C,P 8 Avg
DeepChuster Caron ot al. 2018
22.2
24.4
27.9
27.1
25.4
IC [Ji
39.6
49.1
TIC-Marge J]
2018
32.2
33.2
56-4
30.4
38.1
HC
2013
30.2
30.5
30.7
30.7
35.3
Condimupus DA (Matiiin
35.2
39.1
ACIDS (Menapeoe et ml
2120
42.1
44.5
644
51.1
50.5
K-meuns
Lloyd
[182
17.7
18.5
21.1
22.4
19.9
Ours
51.8
BrAD)
Harary al.
2022
1%
33.6
43.5
61.8
36.4
43.8
BrAD
KNN [Hamary ct m
2122
1%
35.5
38.1
55.0
34.1
40.7
BrAD
Haraay al.
2022
5%
11.4
50.9
65.2
50.7
52.0
BrAD
KNN (Hamrry
2022
5%
39.1
45.4
38.7
46.1
47.3
B:AD
Hararr el al.
2022
10%
11.2
50.0
72.2
55.7
55.5
BrAD-KNN Harary
2022
45.3
50.0
51.1
Table 3: Results on PACS dataset (7 classes) upon all four domain combinations, each letter
A, P, S, C' represents the domains: Art painting, Photo, Sketch, and Cartoon, respectively. The notation W,X," X,Y
Z, means the model was trained W, X, Y domain and tested on the Z domain. Target finc-tuned means the method was trained or adapted to the test domain. In K- means, we first pre-trained the M2 model and trained K-means on top of its embeddings.
Method
Supervision
clipart, infograph, quickdraw painting
painting, real, sketch inlograph
Avg
real
skrtch
clipart
quickdraw
DIUL Zhann
1%
14.5
10.6
DIMEA Yang 2I2
1%
20.2
30.8
20.0
26.5
15.5
15.5
21.1
BrAD-KNN Harary Al
2022
1%
16.9
22.3
25.7
40.7
14.0
21.3
23.5
BrAD
Harary et 2022
1%
20.0
25.1
31.7
47.3
16.9
23.7
27.5
Ours
27.0
29.0
Table 4: Accuracy results on NomainNet dataset (Peng et al. 2019) upon two source domain combinations. The notation X,Y Y + Z, means the model was trained on X, domain and tested the Z domain. Without using any labels, results outperform other SOTA UDG methods, which use 1% of source domain labels.
12

---

Published in Transactions Machine Learning Research (02/2024)
Table 5: Ablation study on the PACS dataset. The letters A, P. S, and C represent the domains Art painting, Photo, Sketch, and Cartoon, respec- tively. The arrow notation similar to other tables. We denote NA cases in which some of the classes were not pre- dicted by the model, which makes cal- culating clustoring accuracy unavail- able.
Method
C.P.S
P.P. +C C
A,C.S
A,C,P
Avg
->
P P
S
Logits only
NA
NA
NA
NA
NA
Plain pre-training
25.0
22.7
NA
No style trausfer
40.6
37.2
50.2
45.8
43.5
No domain head
40.2
41.5
58.6
42.8
45.8
No smoothing
48.2
44.5
56.8
46.6
49.0
Label smoothing
46.3
44.4
66.6
49.0
51.6
PB label smoothing
47.3
45.4
66.6
51.8
prediction-assed (BP) label smoothing. In the first study, we used the PACS datasct and tested four variations of our model. The first variant, called "Plain pre-training', used a standard MoCoV2 feature extractor, followed by training the clustering heads. In the second variation, we omitted the style transfer augmentation during pre-training and clustering head training. The third variation, 'no domain head,' exchuded the domain head and its adversarial loss from the entire training procedure. The fourth variation removed the label smoothing, using a smoothing value of 1. The results presented in Table 5 showed that model performed better than all its ablated versions. This suggests that all the proposed components contribute to our ability to generalize to unscen domains. In cases where not all clusters were predicted, and thus, no clustering accuracy could be calculated, the result was denoted as NA.
In the second ablation study, we examined three variations of label smoothing: no smoothing, standard label smoothing, and prediction- label smoothing. We evaluated our model's per- formance the PACS, Officchome, and Office31 datasets. Our results showed that the proposed prediction-based label smoothing improved clustering capabilities across all evaluated datasets. We presented the results on PACS in Table while the results on the other datasets are included in Appendix B.4| We excluded the results indicating the "single head' ablated version of our model because its prediction often did not cover all clusters.
Hyperparameter stability We use PACS and Officehome datasets to evaluate our model's sen- sitivity to hyperparameters. We test three hyperparameters and present our results in Appendix B.5 suggesting that our method is nearly insensitive to slight variations in the hyperparametens.
5 Discussion
We have developed novel framework for clustering that is fully unsupervised and can be applied to multiple domains. Our approach does not rely on class labels from either the source or target domains, which is a significant advantage. Moreover, our method does not require adaptation to the target domain, making it more capable of generalizing to new, previously unseen domains. We have compared our approach to cxisting basclines and found that it outperforms them. Furthermore, we plan to exlend our model to other modalities, such as audio and text, and apply to other unsupervised learning tasks, such as feature selection or anomaly detection. We are currently working on theoretical analysis of our results. We believe that framework
Conclusion:
13

---

Published in Transactions Machine Learning Research (02/2024)
has great potential advance the unsupervised multi-domain regime and can be used for future research.
Limitations We should note that our method has certain limitations when handling datascts with large number of classes. However, one possible solution to mitigale this issue is weak supervision. Moreover, we acknowledge that our current implementation cannot accommodate non- overlapping classes between the training and tosting subsets. This presents an interosting question for future research that needs to be addressed.
References
Sungwoo Bae, Kwon Joong Na, Jaemoon Koh, Dong Soo Lee, Hongyoon Choi, and Young Tae Kim. Celldart: cell type inference by domain adaptation of single-cell and spatial transcriptomic data. Nucleic acids research, 50(10):e57 e57, 2022.
Debora Caldarola, Massimílianoo Mancini, Fabio Galasso, Marco Ciccone, Emanuele Rodolå, and Barbara Caputo. Cluster-driven graph federated learning over multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2749-2758, 2021.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathhij Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Junbum Cha, Sanghyuk Chun, Kyungiae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advonces in Neural Information Processing Systems, 34:22405- -24418,, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint orXiv:2003. 04297, 2020b.
Gabriela Csurka et al. Domain adaptation in computer vision applications, volume 2. Springer, 2017.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced scarch space. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognition workshops, pp. 702 702-703, 2020.
Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout, 2017. URL htttps//aaxiiiii www.aarwwv..oroaaaaaaaaaee httggaab//770... 04550
Shelli Farhadian, Ofir Lindenbaum, Jun Zhao, Michael J Corley, Yunju Im, Hannah Walsh, Alyssa Vecchio, Rolando Garcia-Milian, Jennifer Chiarella, Michelle Chintanaphol, et al. Hiv viral transcription and immune perturbations in the ens of people with hiv despite art. JCL
insight, 7(13), 2022.
14

---

Published in Transactions Machine Learning Research (02/2024)
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pp. 1180-1189. PMLR, 2015.
Raghuraman Gopalan. Image clustering under domain shift. In 2017 IEEE Third International Conference on Multimedia Big Data (BigMM), pp. 74-77. IEEE, 2017.
Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Abu-Hussein, Elad Amrani, Roei Herzig, Amit Alfassy, Raja Giryes, Hilde Kuehne, et al. Unsupervised domain generalization by learning a bridge across domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5280-5290, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuporvised visual reprosentation learning. In Proceedings of the IEEE/CVF conference on compuler vision and paltern recognilion, pp. 9729 9738, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEFE international conference on computer vision, pp. 1501-1510 2017.
Lina Irshaid, Jonathan Bleiberg, Ethan Weinberger, James Garritano, Rory Shallis, Jonathan Patsenker, Ofir Lindenbaum, Yuval Kluger, Samuel G Katz, and Mina L Xu. Histopathologic and machine deep learning criteria to predict lymphoma transformation in bone marrow biopsies. Archives of Pathology & Laboratory Medicine, 146(2):182 193, 2022.
Samyak Jain, Sravanti Addepalli, Pawan Kumar Sahu, Priyam Dey, and R Venkatesh Babu. Dart: wwversfy-aggrregae--epe training improves generalization of neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 16048-16059, 2023.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information distillation for unsupervised image segmentation and clustering. arXiv preprint 10X::1887..0653, 2(3):8, 2018
Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, 2(1 2):83 97, March 1955. doi: 10.1002 (0.1002/na.3800020100
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the TEFE international conference on computer vision, pp. 5542 5550, 2017.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple fea- ture augmentation for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1886-8895, 2021.
Ofir Lindenbaum, Uri Shaham, Erez Peterfreund, Jonathan Svirsky, Nicolas Casey, and Yuval Kluger. Differentiable unsupervised feature selection based on a gated laplacian. Advances in Neural Information Processing Systems, 34:1530-1542. 2021.
15

---

Published in Transactions Machine Learning Research (02/2024)
Stuart P. Lloyd. Least squares quantization in pem. IEEE Trans. Inf. Theory, 28(2):129 136, 1982.
URL http:://bllp.nn// wtwp:pppp..tttrrr..com ht//://///.///////////////////1/
Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Adagraph: Unifying predictive and continuous domain adaptation through graphs. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6568 6577, 2019.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. Proceedings of the AAAI Conference on Arlificial Intelligence, volume 34, pp. 11749 11756, 2020.
Willi Menapace, Stéphane Lathuilière, and Elisa Rieci. Learning cluster under domain shift. In European Conference on Computer Vision, pp. 736 752. Springer, 2020.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), Aduances in Neural Information Processing Systems, volume 14. MIT Press, 2001. URL.
neurigss..urr.p...... cc/paper/2001/
Chuang Niu, Hongming Shan, and Ge Wang. SPICE: Semantic pscudo-laboling for image clustering.
IEEE Transactions on Image Processing, 31:7264 7278, 2022. doi: (0.1109/tip:2022.332
URL https://doi.or: org/ 10. 11w.%/211iippppp 2022. 32212900
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Fabrizio J Piva, Daan de Geus, and Gijs Dubbelman. Empirical generalization study: Unsupervised domain adaptation vs. domain generalization methods for semantic segmentation in the wild. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 499- 508, 2023.
Subhankar Roy, Aliaksandr Siarohin, Enver Sangineto, Samuel Rota Bulo, Nicu Sebe, and Elisa Ricci. Unsupervised domain adaptation using feature-whitenin and consensus loss. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9471-9480, 2019.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision, 2010.
Uri Shaham, Ofir Lindenbaum, Jonathan Svirsky, and Yuval Kluger. Deep unsupervised feature selection by discarding muisance and correlated features. Neural Networks, 152:34-43, 2022.
16

---

Published in Transactions Machine Learning Research (02/2024)
Saúl Ferrrie-err,,dee; J Ariel Perrasco-Ochoke and José Feo unsupervised feature selection methods. Artificial Intelligence Review, 53(())907-948, 2020.
A review of
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Goorgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European conference on computer vision, pp. 268 285. Springer, 2020.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborly, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018 5027, 2017.
Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. Cross-domain contrastive learning for unsupervised domain adaptation. IEEE Transactions Multimedia, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965 23998. PMLR, 2022.
Haiyang Yang, Shixiang Tang, Meilin Chen, Yizhou Wang, Feng Zhu, Lei Bai, Rui Zhao, and Wanli Ouyang. Domain invariant masked antocncoders for sclf-supervised learning from multi-domains. In Compuler Vision ECCV 2022: 17th European Conference, Tel Aviu, Isruel, Oclober 23 27, 2022, Proceedings, Part XXXI, pp. 151-168. Springer, 2022.
Lu Yin, Shiwci Liu, Mong Fang, Tianjin Huang, Vlado Menkovski, and Mykola Pochenizkiy. Lottery pools: Winning more by interpolating lickets without increasing training or inference cost. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10945-1099)) 2023.
Qiuhao Zeng, Wei Wang, Fan Zhou, Charles Ling, and Boyu Wang. Foresee what you will learn: Data augmentation for domain generalization non-stationary environments. arXiv preprint arXim:2301.0884 2023.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards Unsupervised Domain Generalization. arXiv e-prints, art. arXiv:2107.062: .02199, July 2021.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Domain- irrelevant representation learning for unsupervised domain generalization. arXiv preprint arXio:2107.06219 2(3):4, 2021.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards unsupervised domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4910-490099 2022.
17

---

Published in Transactions Machine Learning Research (02/2024)
Jun Zhao, Ariel Jaffe, Henry Li, Ofrr Lindenbaum, Esen Sefik, Ruaidhrí Jackson, Xiuyuan Cheng, Richard A Flavell, and Yuval Kluger. Detection of differentially abundant cell subpopulations in serna-seq data. Proceedings of the National Academy of Sciences, 118(22)::2100028111 2021.
Kaiyang Zhou, Yougxin Yaug, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104.0000 2021.
18

---

Published in Transactions Machine Learning Research (02/2024)
A Appendix
B Additional implementation details and results
B.1
Implementation details
Our work is implemented on an NVIDIA RTX 3080 GPU using PyTorrc (Paszke et al. 2019 We use blank Resnet18 (He et al. 2016) as our feature extractor to align with the baseline Menapace
ct al. 2020 Harary al., 2022 Wang al. 2022) The models in the first phase were trained using batch size of 8 and train the model for 500 epochs. To train the clustering head, use the same optimizer with batches of size 256 for 100 epochs for Office31 and Officchome datasets and 50 epochs for the PACS dataset. This difference is due to the small number of classes in the PACS dataset, which enables the model to converge much faster. To create style transfer augmentations, we use a pre-trained AdaIN model (Huang & Belongie, 2017). The most diversified head selection mechanism initiates at epoch 30 and repeated every 10 epochs. For more information on the head selection mechanism, see the Multiple Clustering Heads section in the main text of our work. An important regularization for diversified training is label smoothing (Szegedy et al., 2016). Using pscudo-labels, a high ratio of mislabeled samples; label smoothing helps prevent, the model from predicting the training samples too confidently. The ablation study shows empirical evidence of the importance label smoothing in our task.
SGD with momentum 0.9 and weight decay le 4. We
assume
B.2 Implementation of strong augmentations
In the pre-training phase 3.1 we perform style transfer strong angmentations for each batch. When a batch is chosen to undergo strong augmenaations, we use multiple augmentations which a chained together, creating the strong augmentation. Each augmentation component chosen by a random probability. Those components include: random resized crop with a usage probability of p 0.8, color jitter (p 0.8), and parameters brightness=0.44 contrast=0.4, saturation=0.4, hue=0.1. Additional components Gaussian blur (p 0.5). Strong augmentation for the clustering head training 3.2 was done with the same strategies used in SCAN (Van Gansbeke et al. 2020 Cutout (DeVries & Taylor 2017]
random grayscale (p 0.2), horizontal flip (p 0.5), and
and four randomly transformations from Rugmentment CCbuuk et al. 2020).
B.3 Source of randomization between heads
Each head weight is initialized randomly using Py (Paszke et al.; 2019 default initialization. Since the classifier prediction determines the pseudo labels, each head will be trained using different pseudo labels; this variability will keep propagating as training proceeds.
B.4 Additional ablation studies
This section presents the results of an ablation study performed on three datasets. Tables 6 show results of our method without any smoothing, with label smoothing, and using prediction- based label smoothing. In most cases, our proposed prediction-basee label smoothing is the best- performing method, illustrating contribution to the overall method.
19

---

Published in Transactions Machine Learning Research (02/2024)
Table 6: Different label smoothing techniques ablation study results on the Officehome dataset Venkateswara et al. 2017
Method
C,P,R+A
A,P,R-C A,C,R-P A,C,P
Avg
smoothing
25.2
27.1
25.0
Label smoothing
20.8
25.5
27.9
25.6
25.0
PB label smoothing
20.8
26.2
27.7
27.2
25.5
Table 7: Different label smoothing techniques ablation study results on the PACS dataset (Li et al.
2017
Method
C,P,8 A
A,P,S -C
A,C,- P A,C,P -
Avg
No smoothing
48.2
44.5
56.8
46.6
49.0
Label smoothing
46.3
44.4
66.6
49.0
51.6
PB label smoothing
47.3
45.4
66.6
48.0
51.8
Table 8: Different label smoothing techniques ablation study results on the Office31 dataset Saenko
et al.
2010).
Method
D, W +
A
A, -
W
A, W.W
+
D
Avg
No smoothing
24.0
50.0
47.4
40.4
Label smoothing
23.1
49.2
45.2
39.2
PB label smoothing
24.1
50.1
47.7
40.6
20

---

Published in Transactions Machine Learning Research (02/2024)
B.5 Sensitivity hyperparameters
We perform a hyperparameter stability sensitivity study on different hyperparaneters. We show that significant changes in hyperparameters do not lead to major changes in the performance of our proposed method. Specifically, we show deviation of less than 6% in performance for all tested hyperparameter changes (see Table
Table 9: Hyperparameter stability results-pat is the probability of doing style augmentation on the input data. Phod the probability of transforming the input sample the BCD, and #MDH the number of cluster heads to keep during training.
Dataset
Variation
Hyper- pepee-parameter
Base value
Deviated value
Base Acc.
Deviated Acc:
Deviation
PACS
A,P, +
Phot
0.2
0.3
44.7
44.1
1.34%
PACS
A,P,S +
Pood
44.7
46.1
3.13%
PACS
A,P,S C
Pst
0.2
44.7
45.0
0.67%
PACS
A,P,S -CC
Pet
0.3
0.4
44.7
45.6
2.01%
PACS
A,P,5-C
MDHs
5
3
44.7
46.2
3.35%
PACS
A,P,S C
MDHs
44.7
46.6
4.25%
PACS
A,C,S
Phon
0.2
0.3
66.6
64.8
2.70%
PACS
A,C.5
Phot
0.2
0.4
66.6
66.2
0.60%
PACS
A,C.5
Pst
0.3
0.2
66.6
02.8
5.71%
PACS
Pst
0.3
0.4
66.6
63.6
4.50%
PACS
A,C,S
MDHs
66.6
63.1
5.26%
PACS
A,C,S
MDHs
66.6
65.3
1.95%
Officehome
A,C,R
Phod
27.7
28.5
2.89%
Officchomo
A,C,R
Phot
26.8
3.25%
Officehome
A,C,R P
Pst
0.3
0.2
27.7
27.9
0.72%
Officehome
A,C, R - P
0.3
0.4
27.7
27.8
0.36%
Officehome
A,C, P
Pst
0.3
0.4
27.7
27.8
0.36%
Officehome
A,C,R
MDHs
27.7
28.3
2.16%
Officehome
A,C,R P
MDHs
27.7
27.6
0.36%
Officehome
A,C,P
Pood
0.2
0.3
27.2
27.4
0.74%
Officchome
A,C,P R
Phod
0.2
0.4
27.2
27.4
0.74%
Officchome
A,C,P
Pst
27.2
27.0
0.74%
Officehome
A,C,P
Pst
0.3
27.2
25.6
5.88%
Officehome
A,C,
MDHs
3
27.2
27.6
1.47%
Officehome
A,C,P
MDHs
27.2
27.7
1.84%
21

---

Published in Transactions Machine Learning Research (02/2024)
C
Sample images
Figure 4
presents example images from the original domains and our BCD. Next, In Figure 5 present sample images from datasets used in our paper.
we
Art
BCD
3
4
6
Product
BCD
12
2
9
3
6
9
Clipart
BCD
10
10
3
3
8
4
7
6
5
7
6
6
Real World
BCD
11
12
9
3
9
3
8
6
Figure 4: Sample images from BCD domain for Officehome dataset Venkateswara et al. 2017 The right and left columns show the original image and its BCD transform.
22

---

Published in Transactions Machine Learning Research (02/2024))
Sample images from each dataset
Art Painting
Cartoon
Photo
Sketch
PACS
Amazon
Webcam
DSLR
Office 31
Art
Clipart
Product
Real World
Officehome
Figure 5: Sample images from the datasets used in the paper.
23


Reviews:
[{"review_id": "7pFpxnZiEh", "reviewer": "Reviewer_Q7oG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The proposed method is simple and easy to follow.\n- The experimental results are good.\n\nWeaknesses\n- The technique of this work lacks novelty. Although the performance seems to be good, their proposed algorithm is like a stack of existing tricks, such as style augmentation, label smoothing, and adversarial domain adaptation.\n- According to the method part, the proposed method aims to eliminate the domain differences. Therefore, given a mixed domain of multiple different domains, would this method be still effective?\n- The writting has to be further improved. For example, the grammar and the format of reference have some issues. But the writing problem is not limited to what I've pointed out.\n- Some important baselines are missing, such as SCAN and DBSCAN.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the multi-source domain generalized clustering,  where no class labels are used for either source or target domains. This setting is interesting and difficult. They propose a two-stage training framework: (1) self-supervised pre-training for extracting domain invariant semantic features. (2) multi-head cluster prediction with pseudo labels, further leveraging a novel prediction-based label smoothing scheme. The experimental results are good.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Wrtting quality.\n- More basline methods.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xXXtAYeB2R", "reviewer": "Reviewer_rk48", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper is clearly written and easy to follow. I can quickly get the main idea of this paper.\n2.\tThe problem is novel and challenging. Current Domain Generalization is usually conducted under full supervision. But in this paper, the authors study the purely unsupervised setting.\n3.\tThe experimental results are good. For example, in Office-31 W->D and OfficeHome, the performance is improved for over 10%.\n\nWeaknesses:\n1.\tThere is not theoretical guarantee of the proposed method. We cannot be sure that under such framework a machine learning model could be trained to converge to a certain point.\n2.\tThe essential idea of this paper is not novel. There are several similar works using style augmentation that are also close to the field of domain generalization, such as:\n- Zhou et al., Domain Generalization with MixStyle.\n- Li et al., A Simple Feature Augmentation for Domain Generalization. \n- Huang et al., Harnessing Out-Of-Distribution Examples via Augmenting Content and Style.\n- Zeng et al., Foresee What You Will Learn: Data Augmentation for Domain Generalization in Non-stationary Environment. \nMoreover, there is no discussion of these related works.\n3.\tThere are too many tricks introduced as a methodology contribution which makes the proposed framework very incompact and not easy to implement. Moreover, there is no code submitted for examination of the reproducibility.\n4.\tSome of the compared methods are outdated. State-of-the-art performed methods such as SWAD: Domain Generalization by Seeking Flat Minima are not included as a baseline method.\n5.\tThere is no computational complexity discussed in the experiments. How much memory is required to store all the negative queues? Is it still applicable when the framework is conducted on large-scale datasets such as LAION-400-milion?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the Domain Generalization problem, where only unlabeled data are given without any supervised knowledge. To solve this problem, a two-stage framework is proposed: First, the authors use self-supervised learning conduct representation learning to obtain style consistency. Then, using clustering with pseudo labels, the label prediction could be effectively inferred. By introducing several other training tricks, such as domain balancing, data augmentation, reliable pseudo labels, etc, the authors claim that the performance can be further refined. After evaluating on the effectiveness of the proposed method, the authors show on several typical domain generalization datasets, such as Office-31, PACS, VLCS, and DomainNet, the domain generalization result indeed can be largely improved by the proposed framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "No ethical concerns. Limitions are discussed in the appendix.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nyX5VPUgU4", "reviewer": "Reviewer_y9f5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. A potentially new task setting. The explored task setting differs from previous studies and holds practical significance.\n2. The proposed method consists of two training stages, which involve learning domain-invariant representations and cluster prediction, respectively.\n\n\nWeaknesses:\n\n1. The technical novelty appears to be somewhat limited. Several techniques, such as style transfer augmentation, domain adversarial training, and domain-specific queues, are referenced from other papers. As a result, the proposed method incorporates many existing techniques from other methods, and the true technical contribution is unclear.\n2. The definition of UDA, DG, and UDG is unclear. To the best of my knowledge, UDA and DG are different, with UDA additionally utilizing unlabeled target data. However, this distinction is not depicted in Figure 1.\n3. Pre-trained baseline models like CLIP have demonstrated strong generalization capabilities across domains. Therefore, I am curious about the performance comparison between the proposed method and these pre-trained models. Such a comparison could further validate the effectiveness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\nThe paper presents a novel task setting in which no labeled samples are available in the source domain, and the target domain is unseen during the training stage.\nTo address this task, the paper introduces a two-stage training framework. The first stage involves self-supervised pre-training, while the second stage focuses on multi-head cluster prediction. Experimental evaluations are conducted on standard benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "Nan.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the problem setting “interesting and difficult” (Reviewer Q7oG), “novel and challenging” (Reviewer rk48), and holds practical significance (Reviewer y9f5). The experimental results are “good” (Reviewer Q7oG, Reviewer rk48), the proposed method is “simple and easy to follow” (Reviewer Q7oG). During the rebuttal, the authors have made modifications and clarifications to the paper, and presented a comparison to an additional baseline that the reviewers mentioned (with their method comparing favourably). I therefore believe the paper meets the bar of acceptance at TMLR.

For the camera ready, I strongly encourage the authors to include qualitative analysis such as feature visualizations that can aid in interpreting the strong performance of the proposed method and strengthen the paper.
Decision: Accept (poster)